AdvEDM: Fine-grained Adversarial Attack against VLM-based Embodied Agents
Authors: Yichen Wang
Hangtao Zhang
Hewen Pan
Ziqi Zhou
Xianlong Wang
Peijin Guo
Lulu Xue
Shengshan Hu
Minghui Li
Leo Yu Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Adversarial Attack
Vision-Language Model
Embodied AI
Primary area: applications

Abstract: Vision-Language Models (VLMs), with their strong reasoning and planning capabilities, are widely used in embodied decision-making (EDM) tasks in embodied agents, such as autonomous driving and robotic manipulation. Recent research has increasingly explored adversarial attacks on VLMs to reveal their vulnerabilities. However, these attacks either rely on overly strong assumptions, requiring full knowledge of the victim VLM, which is impractical for attacking VLM-based agents, or exhibit limited effectiveness. The latter stems from disrupting most semantic information in the image, which leads to a misalignment between the perception and the task context defined by system prompts. This inconsistency interrupts the VLM's reasoning process, resulting in invalid outputs that fail to affect interactions in the physical world. To this end, we propose a fine-grained adversarial attack framework, AdvEDM, which modifies the VLM's perception of only a few key objects while preserving the semantics of the remaining regions. This attack effectively reduces conflicts with the task context, making VLMs output valid but incorrect decisions and affecting the actions of agents, thus posing a more substantial safety threat in the physical world. We design two variants of based on this framework, AdvEDM-R and AdvEDM-A, which respectively remove the semantics of a specific object from the image and add the semantics of a new object into the image. The experimental results in both general scenarios and EDM tasks demonstrate  fine-grained control and excellent attack performance.

ADVEDM: Fine-grained Adversarial Attack against VLM-based Embodied Agents
Yichen Wang 124..6
Hangtao Zhang
Hewen Pan 1.2.5666
Ziqi Zhou 1,2.,7
Xianlong Wang
Peijin Guo 12.4.5566
Lulu Xue 12245.6
Shengshan Hu 1,2,4,5.6
Minghui Li Leo Yu Zhang 10
'National Engineering Research Center for Big Data Technology and System
SSrviices Computing Technology and System Lab
*Cluster and Grid Computing Lab
Hubei Engineering Research Center on Big Data Security
Hubei Key Laboratory of Distributed System Security
School of Cyber Science and Engincering, Huazhong University Science and Technology
School of Computer Science and Technology, Huazhong University of Science and Technology *Department of Computer Science, City University of Hong Kong
School of Software Engincering, Huazhong University of Science and Technology School of Information and Communication Technology, Griffith University
wangyichen, hangt_ pagtt__hhgg.aa hewenpan, zhouziqi. gpj. lluxue, hushengshan, minghuili @eniill@@hhhn @hust.edu.cn xianlong. wanggngwwwneeee my city...c.y.uuuu.ue. leo.zhang @ griffith.edu.at edu...u.du.uu
Abstract
Vision-Language Models (VLMs), with their strong reasoning and planning capa- bilities, are widely used in embodied decision- makingg (EDM) tasks in embodied agents, such autonomous driving and robotic manipulation. Recent research has increasingly explored adversarial attacks VLMs to reveal their vulnerabilities. However, these attacks either rely on overly strong assumptions, requiring full knowledge of the victim VLM, which is impractical for attacking VLM-bbseed agents, or exhibit limited effectiveness. The latter stems from disrupting most semantic information in the image, which leads to misalignment between the perception and the task context defined by system prompts. This inconsistency interrupts the VLM's reasoning process, resulting in invalid outputs that fail affect interactions in the physical world. To this end, we propose a fine- aun--grined adversarial attack framework, ADVEDM, which modifies the VLM's perception only a few key objects while preserving the semantics the remaining regions. This attack effectively reduces conflicts with the task context, making VLMs output valid but incorrect decisions and affecting the actions of agents, thus posing a more substantial safety threat in the physical world. We design two variants of based on this framework, ADVEDM-R and ADVEDM-A, which respectively remove the semantics of specific object from the image and add the semanties of object into the image. The experimental results both general scenarios and EDM tasks demonstrate fine-grained control and excellent attack performance.
Introduction
Visual-language models (VLMs) such as GPT-4 [1] and Gemini-2.0 [2] have been widely adopted for embodied decision-making (EDM) tasks in embodied agents, including autonomous driving [3, 4, 5] and robotic manipulation [6, 7, 8]. due to their powerful reasoning and planning capabilities. In these tasks, VLMs generate decisions and plannings based current inputs and system states, and
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

then convert them to control code to guide physical entities (e.g., vehicles, robotic arms) in their interactions with the real world.
Inpal image
Model See
Action Dialogue
Results
[Dialogue] Danger Energency teaks.
[Physica]] Brabe()
[Resson] The seoorstoons
jahae bar for aboul.
Benign
Normal Driving
VLM--ased
systems
[Dialogue] I hare nr like
BIMES
[Physical] Bosing Error
[Reason)
leckng dlousu supurriingg
vehioko avd
Existing
Anomaly Alert
How should the vehicle move then??
[Diatogue] Unlopeded, accolnntt Jinand!
[Physkal] Accelenate()
[Peasso] The deser
anusally expo road shoud.
(Ours)
Collision
Figure 1: Comparison of attack framework with existing works in attacking VLM-based agents. Existing attacks disrupt most of the semantics in the original image, causing the VL. to generate invalid responses. In contrast, our attack selectively alters the VLM's perception of specific object while preserving the semantic integrity of other regions. As result, the VLM produces valid yet incorrect decisions, effectively influencing the system's 's interaction with the physical world.
However, VLMs have shown vulnerability to adversarial attacks [9, 10, 11, 12, 13, 14], where the adversary manipulates the model's s output by introducing imperceptible perturbations into input images. Existing attacks on VLMs can be classified into two categories: white-box attacks and black-box attacks. White-box attacks, such AttackBard [15] and CroPA [16], generate adversarial examples by optimizing the end-to-end process, directly manipulating the textual outputs. But such attacks are impractical for VLM-based embodied agents, as it is difficult for the adversary to access the LLM modules within the VLM, which are fine-tuned on proprictary datasets for specific tasks [17].
In contrast, black-box attacks where the adversary's ' knowledge of the victim model is limited are more practical. However, a fully black-box setting poses significant challenges for attackers, incurring significant computational costs and limited attack effectiveness [18]. Therefore, many existing works (like AttackVLM [10], CLIP-based Attack [19] and VTT-Attakk [20]) propose a compromise setting, where the attacker has access to only the vision-text encoder VLMs, referred to as the gray-box setting. Since the vision-text encoder in VLM-based EDM systems is directly used in its pre-trained form [21] and easy for the adversary to obtain, we focus on the gray-box attacks this paper. These attacks introduce adversarial perturbations to move the image's 's embedding away from the clean image's 's embedding, thereby disrupting the VLM's perception of the image. However, such attacks have limited effectiveness against the VLM-based EDM system in embodied agents, which only make the system output invalid results without guiding entity's action in the physical world (like error reports). This is because these systems employ chain-of- of-thhught (CoT) techniques [22] to perform reasoning and task planning. The CoT first analyzes the system's 's current state and perceived inputs, then proceeds to further reasoning based on the task description provided by system prompts [23, 24]. Existing attacks of this type alter the system's 's overall perception of the image, causing a conflict with the system prompts' description, thereby interrupting the reasoning process and resulting in invalid outputs rather than decisions and plannings. This process is illustrated in Fig. 1.
In this paper, propose novel fine -rainer adversarial attack framework, ADVEDM, which makes the VLM-based EDM system output valid but incorrect decisions. As illustrated in Fig. I, our attack disrupts the CoT in the VLM-based EDM system by modifying the VLM's perception of the existence of several key objects while retaining the original semantics of other parts. This significantly reduces conflicts with the task context, ensuring the integrity and logical consistency during the reasoning process. Consequently, the system outputs valid but incorrect decisions that can alter the entity's s action, leading more substantial safety threat in the physical world.
Specifically, we design two attack methods based on this framework, ADVEDM-R R and ADVEDM-A, which respectively remove the semantics of an object from the image and add the semantics of

---

new object into the image, while preserving the semantics of others. The implementation of these attacks faces two technical challenges: first, how to select appropriate regions in the image for the removal and addition of the target object's semantics; and second, how to preserve the semantics of other regions after modifying the target semantics. To address the first challenge, we propose a selection strategy based on the similarities between cross-modality embeddings, which leverages the correlation between the VLM's vision and text encoders [25, 26] by calculating the similarity between image patch token embeddings and object text embeddings. For the second challenge, considering that the image embeddings are typically generated by the attention mechanism of ViT [27], we propose attention- (paatioonnpach)) fixation, which preserves the semantics of the remaining parts by maintaining the product of the attention weights and patch token embeddings.
We evaluate our methods in both general image description scenarios and two representative embodied decision- making tasks: autonomous driving and robotic manipulation. In general scenario, the average attack success rates (ASR) for the two variants are 76.8% and 70.2%, with semantic preservation rates of 66. 66.7% and 71.6%. In EDM tasks, our method achieved an attack success rate of over 70% and 64% in autonomous driving and robotic manipulation respectively, significantly outperforming existing attacks. These results highlight the excellent fine-grained control of our attacks and their effectiveness in posing a real safety threat to VLM-based EDM systems. More demos of our attacks in real-world scenarios can be found on our website https://advedm. github................
In conclusion, the contribution of this paper can be summarized as follows: (1) We propose a novel fine-grainer adversarial attack framework ADVEDM in the gray-box setting that selectively modifies the semantics of key objects perceived by VL.M-based EDM systems, disrupting their reasoning process and leading to valid but incorrect decisions, thus increasing real-world safety risks. This aims to reveal the vulnerabilities of current VLMM-based EDM systems and foster future efforts to enhance their robustness. (2) Based on the framework, we design two attacks ADVEDM-R and ADVEDM-A, which respectively remove the semantics of a specific object or add the semantics of new object to the image. (3) The experimental results in both general scenarios and embodied decision-making tasks indicate the excellent fine-grained control and effectiveness of our attacks.
2 Related Work
2.1 VLM-Based Embodied Decision-Makinn System
Due to their exceptional logical reasoning capabilities, VLMs have been widely applied to embodied decision-making tasks [28, 29, 30J. The Chain-of-Thought (CoT) widely used in VLM-based EDM systems [22, 23], which breaks down the task into logical steps, refining the model's 's decision- based on the current context and task requirements. Two prominent applications are autonomous driving and robotic manipulation. In autonomous driving, VLMs fine-tuned on specialized datasets, such as DriveLM [4], Dolphins [21]. and DriveGPT [31], process road information captured by sensors like cameras. VL.MS in this task, combined with predefined system prompts, enable real- time planning and adjustments to the vehicle's s driving state through CoT reasoning process. In the robotic manipulation task, VLMs first perceive the input visual images, then combine them with received instructions and system prompts to perform reasoning through CoT and generate decisions and plannings regarding the robot's actions like rotation, movement, and grasping. Finally, the post-processing module translates these decisions into control code to manipulate its interactions with the physical world [32]. In conclusion, VLM-based EDM systems play a crucial role in embodied AI tasks. While extensive research has been conducted on the robustness of VLMs themselves [10], the robustness and security of VLM based EDM systems remain unexplored.
2.2 Adversarial Attack against VLMs
Adversarial attack involves manipulating the outputs of models by introducing imperceptible pertur- bations to the inputs [33, 34, 35, 36, 37, 13, 38, 39, 40J. With the widespread application of VLMs. there have been increasing researches focused on attacks against VLMs in recent years. Most of existing works [19, 41, 42, 43, 44] focus on designing attack methods in a white-box setting, where it assumed that the adversary has full access to the victim model and other relevant information to launch attack. These methods typically optimize the adversarial noise by minimizing the difference between the logits of probability of the outputs and the pre-defined target text, thereby enabling end-to-end attacks. Despite their high attack success rates, these attacks are impractical for the
3

---

VLM-based EDM system, as it is difficult for the adversary to access VLMs fine-tuned on various datasets for specific tasks [45].
To this end, some works proposed attacks in more general scenarios [9, 15, 46, 10], where the adversary has limited knowledge about the vietim VLMs. They usually employ pre-trained vision- text encoders of VLMs as surrogate models, such as CLIP [47, 48J. These attacks involve making the embeddings of adversarial examples diverge from those of original images or closer to those of larget images to disrupt the perception of VLMs. Although they are more practical for attacking VLM-based EDM systems whose vision-text encoders are usually pre-trained and easy obtain, their effectiveness is limited due to the lack of fine- fine-grained control. Specifically, they disrupt most of the semantics perceived by the VLM from original images, thus interrupting the reasoning process of VLMs and leading to invalid results that fail to influence interactions with the physical world. In this work, we design two fine-grie fine-grained adversarial attack methods ADVEDM-R and ADVEDM-A that disrupt the perception of VL.Ss by precisely removing or adding the semantics of target objects, while preserving other semantics in the original image.
3 Preliminary
3.1 Background
Following existing works [49, 28], the VLM- based EDM system consists of two compo- nents, as shown in Fig. 2. The first com- ponent is the decision-making module im- plemented by a VLM. which includes a pre- trained vision-text encoder (e.g., CLIP [25]) and fine-tuned L.LM. The vision-text encoder encodes the environmental image along with the received textual instructions and system prompts, as described in Eq. 1.
Resove toy car frem deaktop.
Tiet
VILM
Procegh
Thst
LLM
Dosed
lenge
Fander
Itage legat
Hove directly abovs the tey car.
Rovu 24, 13);
Rove(1?, 20, R);
grab the toy
Madhde
Control Code
Phentop
6, = E.),, 6: = ET)
(1)
Figure 2: The framework of VLM-based embodied decision-making system.
where (-) and E-(-) are the vision encoder and text encoder, while o; and di are the em- beddings of the input image I and text instruction T. Note that the image encoders in VLMS typically based on the Transformer architecture, where the image embeddings d; include [CLS]] token embedding (representing overall semantics) and patch token embeddings (representing local semantics) [50]. We denote them as [cls] and [patch].
The LLM in the decision-making module is fine-tuned proprietary task-specific dataset. It takes as input the fused and concatenated embeddings, performs reasoning through CoT, and generates decisions and plannings. This process is formulated as Tn MMoi, de; f), where M is the LLM with parameters 6, and Tn is the textual outputs of decisions and plannings.
Upon generating decisions and plannings, the post-processing module translates and converts them into executable control code for operating physical hardware the entity, which can be expressed as C' = (TD). C is the control code and fy represents the post-procenn pest-proeessing module.
3.2 Threat Model
Adversary's goal. To achieve fine- iiee-graind attack effectiveness, the adversary's goal as inducing the textual decisions where only the content regarding the target object is altered, while the rest remain unchanged. Here, we formally define our fine-graines adversarial attack. Specifically, we decompose the decision Tn into descriptions for n individual objects in the image, which can be expressed as Tp objj Daii,Daaa Dorj........................DD ..., Dotj. J. Then the attack is formulated as Eq. (2).
min Sim(D'ss Dobje)
s.t. Sim (Si(, Dobj.) > 8, itt
(2)
| - 1|12 <E

---

AdvEDM-R
Put the bottle the basket.
[ch)
User Instruction
Moxsed Trage
Imbi
[pstch]
DOF Robot
[patch),
Image Encoder
(pstrol
Captuna imaga
Tetsl
AdvEDM-A
badcby
Jpsteki
Loss calculation
Cenendor
Frozen
Sebg
Learned
Merge
(nticc)
Figure 3: The pipeline of our methods ADVEDM-R and ADVEDM-A.
where obje is the target object, and Doss, and Dojs decisions generated under adversarial exam- ples 7', while Dobj, and Dobj; under clean inputs T. Sim(-) measures semantic similarity between texts, and and e represent the constraints for the semantic preservation and visual stealthiness.
Adversary's capacity. Since the textual input 7 is usually pre-defined like system prompts [6, 24] and difficult to manipulate from from external sources, we assume the adversary can only perturb image inputs launch attack. Following many previous works like [10, 20, 19], we design attacks in the gray-box setting, where the adversary can only utilize the vision-text encoder of VLMs surrogate models to generate adversarial examples. Moreover, also extend our attacks to the black-box setting and provide a detailed discussion in Appendix D.
Methodology
4.1
Intuition
Unlike previous adversarial attacks on VLMs general tasks like VQA, effectively attack- ing the VLM-based EDM system requires in- ducing valid yet incorrect decisions that im- pact real-world interactions. This necessi- tates a fine- aineggraieed adversarial attack that se- lectively modifies the target object's 's seman- ties while preserving the overall reasoning in- tegrity. Specifically, we propose two feasible schemes: (1) Semantic Removal (SR), namely removing the semanties tion (SA), which involves adding the semantics of a specified object without altering the semantics of others. For SR, the target output T% should exclude the semantics of the target object obje while 1 objects. For SA, it should incorporate both the original
Ieap
Targesi Oijectt Dog
Figure 4: Our strategy to select the target regions for semantic removal and semantic addition.
specific object while preserving those of others, and (2) Semantic Addi-
preserving those of the remaining objects and obj.. Both schemes face two challenges: first, how to identify the key regions in the image for SR SA to ensure the effectiveness of attacks, and second, how maintain the semantics of the remaining parts of the image.
Given that the vision- -extt encoder in most VLMs is built on the Transformer architecture [45, 48), we tackle the first challenge by identifying key regions based on similarity between image patch token embeddings and the text embedding of objr. This similarity effectively quantifies the extent to which each patch contains the target object's semantics, due to the ability of the vision-text encoder to align visual and textual semantics. As shown in Fig. 4, patches with higher similarity scores contain

---

more relevant semantic information. For SR, we choose the patches with higher similarity to the text embeddings target objects and erase their semantics. For SA, we select contiguous background regions with relatively lower similarity to foreground objects' embeddings, ensuring the injected semantics minimally influence the semanties of other objects.
As for the second challenge, we adopt an attention- [patch] fixation approach. Specifically, during the optimization of adversarial perturbation, we ensure that the product of attention weights and patch token embeddings for the remaining regions closely matches that of the original image. This approach effectively preserves both the overall semanties and local detailed features of them [51].
4.2 Our Methods
According to our intuition, we propose two attack methods ADVEDM-R and ADVEDM-A, which remove the semantics of specific object from the image and inject the semantics of new object.
4.2.1 ADVEDM-R
We first demonstrate how identify regions with target object's semantics in the image. We calculate the cosine similarity between the patch token embeddings and the text embedding of target object, and then mark the patches with higher similarity and form mask. This process is formally described in Eq. 3 and 4.
S = CS((patch) Fe((Tar))
(3)
mask; = { 0, if >
(4)
if s SE
where [patch], is the patch token embeddings of clean image I, (T(or)))) is the text embedding of the target object encoded by E, and CS(() is the cosine similarity function. The similarity vector S E R where is the number of image patches. mask is also n-dimensional vector, whose i-th element mask; is determined by comparing the corresponding element 8; in with predefined threshold ç. Elements in mask with a value of 0 indicate that the corresponding image patches contain richer semantics of the target object.
After obtain the mask, we remove the semantic of target object from both global and local perspec- tives. For global semantics, we push the [CLS] token embedding that represents the overall semantics of the image, away from E, (Ttar) to remove the target object's global semantics, as shown in Eq. 5.
Lats = CS((cs)) Ee (Tiar))
(5)
For local semantics, utilize the obtained mask to erase the patches containing target semantics from the image, and denote the masked image as M. In M, the patches originally rich in semantics of the target object are replaced by meaningless O-pixel values. Then, we align the embeddings of corresponding patches in the adversarial example with those in M. This process is shown in Eq. 6.
S. -(1 mask) CS([patch] r [patch] m)
(6)
Additionally, we propose the attention (patch] fixation, ensuring that the key features of the rest parts remain consistent with those of the original image, as shown in Eq. 7.
Ljiz = -mak CS(Ar [patch] A1 [patch]r)
where A, and Ap are the means of attention weights across all attention layers E, of the original image and adversarial example, as they reflect the semantic significance of each patch [51].
In conclusion, the entire process of ADVEDM-R can be expressed as Eq. 8, and wi to wy are the weights of each loss item. The pipeline of ADVEDM-R is shown in Fig. 3.
min W1 * Lets + W22 Cp Wz fju
(8)
s.t. ||1 - 1 1| <

---

Table 1: Quantitative results of attacks MS-COCO dataset in the image description task.
Mudelk
11 LLAVAoy?
MiniSPPS SPR(S:
Osammmags SPR(S))
BLBP-
OFLMG-99
Attalo
ASRE(S)
SPR(R)
55
/ASR/S)
55
ASSRS)
55
(ASRSS)
SPR(1))
55
(SRRSS)
SPR(15)
55
(ASR/5)
SPS)
.55
PGD
71.7
22.5
0.390
10.8
17.9
0,312
13.6
18.1
0.302
76.5
17.0
0.426
70.2
20.4
0.350
73.9
19.2
0.368
MI-E
ADVEDDER
75.2
91.3
01.05
759
11626
9099
64.4
(L82
16.5
68.3
6.787
196
0,898
66.7
6483
ABVEDM-A
03.4
75.1
0,758
07.1
72.6
0.687
728
69.4
0.725
09.
68.2
0.786
73.3
72.
0.732
70,2
71.6
0.736
4.2.2 ADVEDM-A
In the implementation of ADVEDM-A, we first manually select contiguous patches of size m x m, typically from the background or areas containing minimal specific objects. The selection can refer patches with lower embedding similarity to the foreground objects' text embeddings. After the selection, apply the same procedure as in Eq. 4 to mark the selected patches as 0 and the remaining as 1, thereby generating the corresponding mask.
Due to the modality gap between image and text inputs in image-text encoders [52]. textual descrip- tions of the target object camnot effectively inject semantics into the image embeddings, especially the patch token embeddings. To address this, we utilize reference image R with m x patches that solely contains the target object (generated by text2image model [53]]. Here, we also incorporate the target semantics into the original image from both global and local perspectives. For global semantics, we align the [cls] of the adversarial example with the weighted fusion [cts] from the clean and target images, detailed in Eq. 9.
== = -C(css (1 -1- (1) + a(cls)n)
Locally, the selected patches for semantic injection contain minimal information and thus have lower attention weights. To ensure the vision encoder captures the injected semantics, we reallocate attention weights of these patches by assigning them the scaled attention weights of the reference image R. Other patches undergo a similar scaling to preserve global semantic consistency. The process is described Eq. 10. Ag is the attention weights of R and 3 is the scale factor.
{
BAR, (1 - 9),
if mask; 0 if mask; =
A -
(10)
After obtaining the new attention weight map, we compute the key features by taking the product of the weight map and the patch token embeddings of R. Subsequently, we align the key features corresponding positions in the adversarial example with those in to achieve local semantic injection, as shown in Eq. 11.
Cp = -1 - mask) CS(Ap * [patch]r, A' * [patch] R)
(11)
Then also consider to preserve the key features in other regions of the original image. Note that to allocate sufficient attention weights for injected semantics, we should utilize the reallocated attention weights A'. So the attention- (pattionn-pahh) fixation can be expressed Eq. 12.
fiz = mask s CS(Ar * [patch]y, A' [patch]
(12)
In conclusion, the overall optimization of adversarial examples is the same as Eq. 8. The pipeline of ADVEDM-A is shown in Fig. 3.
Experiment
We conducted experiments in both general evaluation scenarios and EDM tasks. The general scenario involves image description, which is consistent with existing adversarial attack evaluation scenarios in general VLMs. The EDM tasks include autonomous driving and robotic arm manipula- tion. Besides, more visualization results are provided Appendix B and our webpage. The ablation studies and exploration of transferability also can be found in Appendix C and D.
7

---

Table 2: Quantitative results of attacks Dolphins Benchmark dataset in autonomous driving scenes.
Models
LV-v2
MGPT-4
Otter ASR(%)
Dol
Avg.
Attacks
ASR(S)
ASR(S)
ASR(%)
SS
ASR(S))
PGD
22.0
0.336
16.0
0.206
13.0
0.279
18.0
0.259
17.3
0.270
MF-it
17.0
0.303
21.0
0.250
15.0
0.253
13.0
0.237
16.5
0.261
Mi-ii
24.0
0.315
17.0
0.222
8.00
0.260
10.0
0.234
14.8
0.258
ADVEDM-R
74.0
0.511
78.0
0.505
0.485
84.0
0.502
74.5
0.501
ADVEDM-A
68.0
0.564
73.0
0.489
66.0
0.509
79.0
0.546
71.5
0.527
5.1 Setups
Models. We employ several commonly-used VLMs, including BLIP-2 [54], MiniGPT-4 (MGPT-4) [48], LLaVA-v2 (LV-v2) [47], Otter-Image (Otter-I) [55] and OpenFlamingo-v2 (OFLMG-v2) [56]. The vision-text encoders of BLIP-2 and MiniGPT-4 are based on EVA CLIP [26], while others are
based on OpenAl's ViT CLIP [25].
Datasets. For general scenarios, we select MS-COCO 2014 [57, 58]. For the autonomous driving scenario, we choose Dolphins Benchmark [21] and DriveLM-nuSSenne [4] that are specialized for this task, while for the robotic arm manipulation task, we sample 100 images from the physical world and construct instructions and actions.
Attacks. We choose several typical adversarial attacks as baseline, including CLIP-Based PGD [19], MF-it and MF-ii [10]. The norm constraint of the adversarial perturbation is set to 8/255. For CLIP-Based PGD, we set the number of iterations 20 with step size of 0.01. For MF-it and MF-ii, we use their official open-source code. Details of our methods' settings are in Appendix A.
Metrics. We define Attack Success Rate (ASR) and Semantic Similarity (SS) measure attack effectiveness. Specifically, SS is the cosine similarity between output embeddings of adversarial and clean samples calculated by text encoder. We introduce Semantic Preservation Rate (SPR), which measures the retention of other objects' semantics in VLM description of input images. The detailed formal definitions of these metrics are provided Appendix A.
Note that the interpretation of evaluation metrics vary across different tasks. In the general image description task, attack is considered successful if causes the target object to include exclude in the generated description. while higher SPR and SS values reveal the attack better preserves the semantic integrity of the image. In the EDM tasks, the attack is successful if the output decisions and plannings align with the expected results after the addition removal of target object's semantics (i.e., valid yet incorrect). However, since VLM-based EDM systems do not provide detailed descriptions of all objects in the inputs, SPR cannot be reliably computed and is thus omitted from the corresponding experiments.
this case, a higher ASR values indicate greater attack effectiveness,
5.2 Evaluation in General Scenarios
Settings. According to our threat model, we employ the vision- text encoder of each victim model as surrogate model to launch attack. For the attack target, we randomly select an object the image to remove its semantics or choose an object not in the image to inject its semantics. We randomly select 1000 images generate adversarial examples and record the average ASR, SPR and SS.
Results. Tab. shows that though existing attacks achieve high ASR values, they suffer from low semantic preservation, with SPR values under 20%. This indicates they disrupt the majority of the original image's 's semantics, resulting in poor fine-grained control. In contrast, our methods exhibit slight decrease in ASR, but their SPR values are significantly higher (66.7% and 71.6%, respectively). preserving most of the original image's semantics and enabling fine-grained control. Our methods also achieve higher SS values, as preserves most of the original image's semantics.
5.3 Evaluation in EDM tasks
5.3.1 Autonomous Driving Task
Settings. We conduct experiments on two specialized datasets, Dolphins Benchmark and DriveLM- nuScenes, with three general VLMS (L.LAVA-v2, MniGPT-4, and Otter-Image) and Dolphins (Dol) [21], a VLM designed for decision-making task in autonomous driving, For each dataset, we randomly

---

Table 3: Quantitative results of attacks on DriveLM- dataset in autonomous driving scenes.
Models
LV-22
MGPT-4
Otter
Dol
Avg.
Attacks
ASR(S)
ASR(S))
ASR(%)
ASR(%)
SS
ASR(S))
PGD
20.0
0.274
19.0
0.218
16.01
0.286
23.0
0.243
19.5
0.255
MF-it
19.0
0.263
24.0
0.242
0.245
17.0
0.237
17.3
0.247
Mi-ii
14.0
0.260
17.0
0..221
12.01
0.253
14.0
0.234
14.3
0.242
ADVEDM-R
79.0
0.487
83.0
0.504
75.0
0.472
86.0
0.544
80.8
0.502
ADVEDM-A
73.0
0.536
79.0
0.517
71.0
0.509
82.0
0.530
76.3
0.523
ARVEDMER
Presuse she foom the tay
Pomoze CD
say
esampli
Elent
ADVEDM-R
Alwe Itamaid
What oction should the verkcle take nest!
What action shouid the vehicie dake nest?
kay skapo:
se rstacete there
2.8easue The
1-dran the deakrop.
paruees
E.Pcase the esr
(te Eap craanptee
Nilion resull
an assropriude spsal and Anap aiernece
drave atrutght becvass shere
Vaniilla
Peeeese the from
Revesse CD from the deattop. perion
Whet action should the vericle rale noxt?
What oction should the welide takl nest?
tnapple.
parson
it praceed woch trow Lutren the acselie:
Serry the curnees rocente 15 and un snable is sale deceaios.
The vettzle shells slow shel At UN WesSarrants at lie creak,
(t, fay thy trisegle the smap..
Tay perton
Mution resill
Figure 5: Visualization results in the tonomous driving decision-making task.
Figure 6: Visualization results in the VLM-based robotic arm manipulation task.
select 100 road scene images and choose common objects in the road as target, such as vehicles, pedestrians, and traffic lights. The impact various attacks on the reasoning process of VLMs is presented in detail in Appendix C.
Results. The quantitative results are reported in Tab. 2 and 3, and the visualization results are shown in Fig. 5. According to the quantitative results, the average ASR of our methods are over 70% and 75% on the two datasets, respectively, dramatically outperforming existing attack methods. This demonstrates that our attack can have a substantial impact on VLM-based decision decision-making systems in autonomous driving. Our methods also achieve higher SS, as they ensure the model outputs valid decisions that are structurally identical to normal outputs, such as "The vehicle should go straight/turn left." " Besides, the visualization results also demonstrate that our methods enable VLMs to make valid but incorrect decisions, thereby affecting the vehicle's driving state.
5.3.2 Robotic Manipulation Task
Settings. We take the robotic arm manipulation task as example, where various objects are placed on desktop, and then 100 images are captured to form the dataset. We still select three general VLMs and EmbodiedGPP (EmGPT) [24], a VLM specifically designed for this task, as victim models. For each image, the instructions we design involve the manipulation of two or more objects, including deploy the VLMs on
the target object selected for attack. Moreover, to better visualize the results, a UR robotic arm, where their outputs directly influence the robotic arm's ' actions. The impact of various attacks on the reasoning process of VLMs is also discussed in Appendix C.
Results. The quantitative results are shown in Tab. and the visualization results are shown Fig. The presentation videos of attacking on robotic arm manipulation can be found on our website. The results highlight the superior effectiveness of our attack methods, achieving ASR values of 68.8% and 64.5%, significantly surpassing those of existing attacks. When VLMs encounter existing attacks, the perceived semantics are completely different from the original image, leading to error messages such "Sorry, there is no object A nor on the desktop.". ", resulting in lower SS values. In contrast,

---

Table 4: Quantitative results of attacks in robotic arm manipulation task.
Models
IV-v2
MGPT-4 ASR(S)
Otter ASR(X)
EmGPT
Avg.
Attacks
ASR(%)
SS
SS
SS
ASR(%)
SS
ASR(%)
SS
PGD
16.0
0.123
12.0
0.165
11.0
0.103
7.00
0.079
11.5
0.117
MF-it
11.0
0.132
15.0
0.185
14.0
0.118
0.090
12.5
0.131
MF-i
9.00
0.119
12.0
0.155
6.00
0.091
9.00
0.085
9.0
0.115
ADVEDM-R
72.0
0.459
66.0
0.428
63.0
0.466
74.0
0.450
68.8
0.451
ADVEDM-A
our methods mainly alter the VLM's decisions and plannings regarding target object, while the decisions and plannings for non non-target objects remain similar to those make for clean images, ensuring that the values remain sufficiently higher.
Conclusion
In this work, propose fine-grained adversarial attack framework ADVEDM against the VLM-- based EDM systems, which aims to reveal the vulnerabilities of them. By only disrupting the VLM's perception of the target object while preserving other semantics, attack maintains the integrity of VLM's reasoning process, enabling it to output valid yet incorrect decisions that influence the entity's interactions with the real world. Specifically, we design attack variants: ADVEDM-R. which removes the semantics of a specific object from the image, and ADVEDM-A, which injects the semantics of new object into the image. Experimental results in both general scenarios and EDM tasks demonstrate the superior fine-graing grained control and attack effectiveness of our methods against the WLM-basee EDM systems, outperforming existing attacks targeting the VLM itself.
Acknowledgements
Minghui Li's work is supported by the National Natural Science Foundation of China under Grant No. 62202186. Shengshan Hu's work is supported by the National Natural Science Foundation of China under Grant No.62372196. Minghui Li is the corresponding author.
10

---

References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat, al. "Gpt-4 technical report," arXiv preprint arXiv:2303.0877 2023. 1
[2] Ô. Aydı, "Google bard generated literature review: metaverse," Journal of Al, vol. 7, no. 1. pp. 1-14,
2023. 2,
[3] Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, L. Lu, X. Jia, Q. Liu,
J. Dai, Y. Qiao, and H. Li, "Planning-orienne
autonomous driving," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17853-17862, June 2023. 1
[4] C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang. C. Xie, J. Beißwenger. P. Luo, A. Geiger, and H. Li, "Drivelm: Driving with graph visual question answering," in European Conference on Computer Vision, pp. 256-274, Springer, 2024. 1,3,8
[5] H. Zhang. S. Hu, Y. Wang, L. Y. Zhang. Z. Zhou, X. Wang. Y. Zhang, and C. Chen, "Detector collapse: backdooring object detection to catastrophic overload or blindness in the physical world," in Proceedings of the Thirty-Thive International Joint Conference on Artificial Intelligence, pp. 1670-1678, 2024.
[6] J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, and D. Sadigh, "Physically grounded vision-language models for robotic manipulation." in 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12462- 12469, IEEE, 2024. 1,5
[7] W. Zhao, J. Chen, 7. Meng, D. Mao, R. Song, and W. Zhang, "VImpe: Vision-language model predictive control for robotic manipulation." arXiv preprint MxXi:22407.09829 2024. 1
[8] X. Wang, H. Pan, H. Zhang, M. Li, S. Hu, Z. Zhou, L. Xuc, Guo, A. Liu, L. Y. Zhang, et al., "Trojanrobot: Physical-world backdoor attacks against vlm-based robotic manipulation," arXiv preprint arXiv:2411.11683 2024.
[9] Z. Zhou, S. Hu, M. Li, H. Zhang. Y. Zhang, and H. Jin, "Advelip: wownstrcam-agnostee adversarial examples in multimodal contrastive learning," in Proceedings of the 32nd ACM International Conference on Multimedia (MM'23) pp. 631 1-6320, 2023. 2,4
[10] Y. Zhao, T. Pang. C. Du, X. Yang, C. Li, N.-M. M. Cheung, and M. Lin, "On evaluating adversarial models," Advances in Neural Information Processing Systems, vol. 36,
robustness of large
2024. 2, 3, 4, 8
[11] Z. Zhou, B. Li, Y. Song. S. Hu, W. Wan, L. Y. Zhang, D. Yao, and H. Jin, "Numbod: A spatial-frequency fusion attack against object detectors," in Proceedings of the 39th Annual AAAI Conference on Artificial Intelligence (AAAI'25), 2025. 2
[12] Y. Song, Z.. Zhou, M. Li, X. Wang, M. Deng, W. Wan, S. Hu, and L. Y. Zhang, "Pb-uap: Hybrid universal adversarial attack for image segmentation." in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'25 25), 2025.
[13] Z. Zhou, Y. Hu, Y. Song, Z. Li, S. Hu, L. Y. Zhang, D. Yao, L. Zheng, and H. Jin, "Vanish into thin air: Cross-prompt universal adversarial attacks for sam2." in Proceedingy of the 39th Annual Conference on Neural Information Processing Systems (NeurIPS 25)) 2025. 2,3
[14] M. J. Wang, H. Zhang, Z. Zhou, S. Hu, and X. Pei, "Transferable adversarial facial images for privacy protection," in Proceedings of the 32nd ACM International Conference on Multimedia (ACM MM)24), pp. 10649-10658, 2024. 2
[15] Y. Dong, H. Chen, J. Chen, Z. Fang, X. Yang. Y. Zhang, Y. Tian. H. Su, and J. Zhu, "How robust is google's bard to adversarial image attacks?," arXiv preprint ar3iv: //751, 2023. 2,4
[16] H. Luo, J. Gu, F. Liu, and P. Torr, "An image is worth 1000 lies: Transferability of adversarial images across prompts on vision-language models," in The Twelfth International Conference on Learning Representations. 2023. 2
[17] S. Zhai, H. Bai, Z. Lin, J. Pan, P. Tong, Y. Zhou, A. Suhr, S. Xie, Y. LeCun, Y. Ma, et al., "Fine-tuning large vision-language models as decision-making agents via reinforcement learning," Advances in Neural Information Processing Systems, vol. 37, pp. 019935-119977 2025. 2
[18] C. Zhang, X. Xu, J. Wa, 7. Liu, and I.. Zhou, "Adversarial attacks of vision tasks in the past 10 years: A survey," arXiv preprint Xiv: 24ff::24::.222 10.23368 2024. 2
[19] X. Cui, A. Aparcedo, Y. K. Jang, and S.-N. Lim, "On the robustness of large multimodal models against image adversarial attacks." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24625 24634, 2024. 2, 5,
[20] Y. Wang. C. Liu, Y. Qu, H. Cao, D. Jiang, and L. Xu, "Break the visual perception: Adversarial attacks targeting encoded visual tokens of large dision-languae models." in Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1072- 1081, 2024. 2,5
11

---

[21] Y. Ma. Cao, J. Sun. M. Pavone, and C. Xiao, "Dolphins: Multimodal language model for driving." in European Conference on Computer Vision. pp. 403-420, Springer, 2024. 2,
[22] J. Wei, X. Wang, D. Schuurmans. M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., "Chain-of-thought prompting clicits reasoning in large language models," Advances in neural information processing systems, vol. 35. pp. 24824 24837, 2022. 2,3
[23] Y. Chen, K. Sikka, M. Cogswell, H. Ji, and A. Divakaran, "Measuring and improving chain-of -f-thuughtt reasoning in lasoua- models." arXiv preprint arXiv:2309, 04461, 2023. 2.3
[24] Y. Mu, Q. Zhang. M. Hu, W. Wang. M. Ding, J. Jin, B. Wang. J. Dai, Y. Qiao, and P. Luo, "Embodiedgpt: Vision-language pre-training via embodied chain of thought," Advances in Neural Information Processing Systems, vol. 36, pp. 25081-25099 2023. 2, 5,9
[25] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, et al., "Learning transferable visual models from natural language supervision." in International conference on machine learning, pp. 8748-8763, PmLR, 2021. 3, 4,
[26] Q. Sun, Y. Fang, L. Wu, X. Wang, and Cao, "Eva-clip: Improved training techniques for clip at scale," arXiv preprint arXiv:2303. 15389, 2023. 3,8
[27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017. 3
[28] M.-Y. Lin, O.-W. Lee, and C.-Y. Lu, "Embodied with large language models: A survey and new hri framework," in 2024 International Conference on Advanced Robotics and Mechatronics (ICARM), pp. 978 978-983,100 IEEE, 2024. 3,4
[29] H. Zhang, C. Zhu, X. Wang, Z. Zhou, C. Yin, M. Li, L.. Xue, Y. Wang. S. Hu, A. Liu, et al., "Badrobot: Jailbreaking embodied Ilms in the physical world," arXiv preprint ar arXiv:2407.2022 2024. 3
[30] L. Yu, Y. Zhang. Z. Zhou, Y. Wu, W. Wan, M. Li, S. Hu, P. Xiaobing, and J. Wang. "Spa-vim: Stealthy poisoning attacks on rag- based vim," arXiv preprint arXiv:2505. 238828, 2025. 3
[31] Z. Xu, Y. Zhang, E. Xie, Z. Zhao, Y. Guo, K.-Y. K. Wong. Z. Li, and H. Zhao, "Drivegpt4: Interpretable end-to-end autonomous driving via large language model," JEEE Robotics and Automation Letters, 2024. 3
[32] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L., Fei-Fei, A. Anandkumar, Y. Zhu, and
L. Fan, "VIMA: Robot manipulation with multimodal prompts," in Proceedings of the 40th International Conference on Machine Learning (ICLR 23), vol. 202, pp. 14975-15022, 2023. 3
[33] I. J. Goodfellow, J. Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples," in Proceed- ings of the 3rd International Conference on Learning Representations (ICLR'15) 2015. 3
[34] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards deep learning models resistant to adversarial attacks." arXiv preprint arXiv: /tfiv:700.00000 06083, 2017. 3
[35] 7. Zhou, M. Li, W. Liu, S. Hu, Y. Zhang, W. Wan, L. Xue, L. Y. Zhang, D. Yao, and H. Jin, "Securely fine- tuning pre-trained encoders against adversarial examples." in Proceedings of the 2024 IEEE Symposiun on Security and Privacy (SP 24), 2024.
[36] Z. Zhou, S. Hu, R. Zhao, Q. Wang, L. Y. Zhang, J. Hou, and H. Jin, "Downstream agnossic adversarial examples." "" in Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV'23),
pp. 4345-435555 2023. 3
[37] Z. Zhou, Y. Song, M. Li, S. Hu, X. Wang, L. Y. Zhang, D. Yao, and H. Jin, "Darksam: Fooling segment anything model segment nothing," " Proceedings of the 38th Annual Conference on Neural Information Processing Systems (NeurIPS 24), 2024. 3
[38] Y. Wang, Y. Chou, Z. Zhou, H. Zhang, W. Wan, S. Hu, and M. Li, "Breaking barriers in physical-word adversarial examples: Improving robustness and transferahility via robust feature," " in Proceedings of the 39th Annual AAAI Conference on Artificial Intelligence (AAAI'25), 2025.
[39] Y. Song, 7. Zhou, Q. La, H. Zhang, Y. Hu, L. Xue, S. Hu, M. Li, and Y. Zhang, "Segtrans: Transferable adversarial examples for segmentation models." IEEE Transactions on Multimedia, 2025. 3
[40] H. Zhang, Z. Yao, L. Y. Zhang, S. Hu, C. Chen, A. Liew, and Z. Li, "Denial-of-service or fine-grained control: towards flexible model poisoning attacks on federated learning." in Proceedings of the Thirty-
Second International Joint Conference on Artificial Intelligence, pp. 4567-4575, 2023. 3
[41] X. Fu, Z. Wang, S. Li, R. K. Gupta, N. Mireshghallah, T. Bang-Kirkpatrick, and E. Fernandes, "Misusing tools in large language models with visual adversarial examples," arXiv preprint arXiv:2310.03185. 2023.
[42] K. Gao, Y. Bai, J. Bai, Y. Yang, and S. Xia, "Adversarial robustness for visual grounding of multimodal large language models." arXiv preprint arXiv:2405.099: 009811 2024.
12

---

[43] C. Schlarmann and M. Hein, "On the adversarial robustness of multi-modal foundation models." in Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV) Workshops, pp. 3677- 3685, October 2023.
[44] Z. Wang, Z. Han, S. Chen, F. Xue, Z. Ding, X. Xiao, V. Tresp, P. Torr, and Gu, "Stop reasoning! when mul- timodal IIms with chain-of-thoupe reasoning meets adversarial images." arXiv preprint arXiv:2402.14899
2024. 3
[45] D. Liu, M. Yang, X. Qu, P. Zhou, Y. Cheng, and W. Hu, "A survey of attacks on large vision-language models: Resources, advances, and future trends," arXiv preprint arXiv:2407.00 000.070073 2024. 4, 21
[46] Q Guo, S. Pang. X. Jia, and Q. Guo, "Efficiently adversarial examples generation for visual- llnguugg models under targeted transfer scenarios using diffusion models," arXiv preprint arXiv:2404. 10335, 2024.
[47] H. Liu, C. Li, Q. Wu, and Y. J. Lee, "Visual instruction tuning." " in Proceedings of the 36th Advances in
Neural Information Processing Systems (NeurlPS 23), 2023. 4,8 8
[48] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. "Minigpt-4: Enhancing lision-language understanding
with advanced large language models," " arXiv preprint arXiv:2304. 10592, 2023, 4, 5,8
[49] S. Liu, J. Chen, S. Ruan, H. Su, and Z. Yin, "Exploring the robustness of decision-level through adversarial attacks 1lm-based embodied models," in Proceedings of the 32nd ACM International Conference on Multimedia, pp. 8120-8122 8128, 2024.
[50] J. Zhang, J. Huang. S. Jin, and S. Lu. "Vision -lnguugl models for vision tasks: A survey," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 4
[51] R. Yu, W. Yu, and X. Wang, "Attention prompting on image for large vision-languag models," in Proceddings of the 18th European Conference of Computer Vision (ECCV'24), vol. 15088, pp. 251-268, Springer, 2024.
[52] V. W. Liang. Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou, "Mind the gap: Understanding the modality gap in multi- modti-modal contrastive representation learning." Advances in Neural Information Processing Systems, vol. 35, pp. 17612-17625, 2022. 7
[53] A. Borji, "Generated faces in the wild: Quantitative comparison of stable diffusion, midjourney and dall-e 2." arXiv preprint arXiv:2210.00586, 2022. 7
[54] J. Li, Li, S. Savarese, and S. Hoi, "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models," in International conference on machine learning. pp. 19730-19742. PMLR, 2023.
[55] B. Li, Y. Zhang, L. Chen, J. Wang, F. Pu, J. Yang. C. Li, and Z. Liu, "Mimic-it: Multi-modal in-context instruction tuning," arXiv preprini arXiv:2306. 054455, 2023. 8
[56] A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y. Hanafy, W. Zhu, K. Marathe, Y. Bitton, S. Gadre, S, Sagawa, et al., "Openflamingo: An open-source framework for training large autoregressive vision- language models," arXiv preprint arXiv:2308.011 390, 2023. 8
[57] T.YY. Lin, M. Maire, S. Belongic, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft coco: Common objects in context," in Computer vision-ECCT ECCV 2014: 13th European conference, zurich, Switzerland, September 1-1 2014, proceedings, part 13, pp. 740-755, Springer, 2014. 8
[58] H. Zhang, Y. Wang, S. Yan, C. Zhu, Z. Zhou, L. Hou, Hu, M. Li, Y. Zhang, and L. Y. Zhang, "Test-time backdoor detection for object detection models," in Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24377-24386, 2025. 8
[59] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv: 1412.6980, 2014. 21
[60] H. Chen, Y. Zhang, Dong, X. Yang, H. Su, and Zhu, "Rethinking model ensemble in transfer-based adversarial attacks." arXiv preprint arXiv:2303.09100 2023. 24
[61] Y. Long, Q. Zhang, B. Zeng. L. Gao, X. Liu, J. Zhang, and J. Song. "Frequency domain model augmentation for adversarial attack," in European conference on computer vision, pp. 549-566, Springer, 2022. 24
[62] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, "Sigmoid loss for language image praitnnin in Proceedings of the IEEE/CVF international conference on computer vision. pp. 11975-11986, 2023. 24
[63] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes,
A. Radford, et al., "Gpt-40 system card," arXiv preprint arXiv:2410.21277 2024. 24
[64] T. Kumamoto, Yoshida, and Fujima, "Evaluating large language models ransomware negotiation:
A comparative analysis of chatgpt and claude," 2023. 24
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims are made in the abstract and introduction accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The discussion about limitations is included in our appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our experimental settings are provided in the appendix and part of our source code is in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: Part of our source code is the supplementary material. The complete code will be released after publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/guides/ Sotees//oooSSumisiiinnnne
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: /CodeSubmis //wpp..cc////////. cttppscccpubbbcc/gcccccc// S ssi ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information is provided our appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eerrpp.eeeee/eee/ http:::://rrrps..ppppppc////////
ctw.ideessellleeccccccccccc deee:/eeeeeeeeeieieeeeeeeaeeeeea ines?
Answer: [Yes]
Justification:
The research confirms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This is included in the Introduction and Conclusion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We explicitly acknowledge and credit the original repository authors in the provided code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We adopt ALM-as-a-judee in our experiments processing results. The detailed usage instructions and settings are provided in the appendix and the source code.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /tweerrps.ceooooeeeeeeeeeeeeeeee cccc//o/meerrrccee22////1/// for what should or should be described.
20

---

A Detailed Experimental Settings
Parameter Settings of our methods. For ADVEDM-R, R, we select the top 20% image patches with the highest similarity to the target object's 's text embedding and mask their pixels to generate masked image M. We set W1, W2 and W3 in Eq. (3) to 0.5, 2, and 0.2, respectively. The optimization performed for 500 iterations using the Adam optimizer [59] with learning rate of 0.005. For ADVEDM-A, we We select a 100 X 100 pixel region in the image for semantic injection. When reallocating attention weights,
set the scaling factor B 0.4 in Eq. (12), and the fusion weight of [CLS] tokens is 0.5 in Eq. (11). During the optimization process, Wi to Wi is set to 0.8, 2, and 0.3, respectively. The remaining optimization settings are kept identical to those of ADVEDM-R.
Experimental environment. All experiments are conducted on NVIDIA A100-SXM4 GPUs, each equipped with 80GB of memory. For the caleulation of metrics ASR and SPR, adopt the LLM-as- judge approach [45], employing GPT-3.5-ture and other LLMs.
Definition of metrics. Semantic Similarity (SS) is defined as the cosine similarity of between output embeddings of adversarial and clean images calculated by a text encode, which can be expressed as Eq. 13:
SS = (T),
(13)
where C.S()) is the cosine similarity function and and T" represent the textual outputs of clean images and adversarial examples respectively.
As for calculating SPR, the procedure is divided into two steps. The first step involves decomposing all objects included in the description T" and T", which can be done by GPT-3.5-turbo as shown in Eq. 14. The specific prompts for extracting the semantics of objects are provided in our source code.
D L(T) {Obji, Objz, Objo}
(14)
D' L(T') {Obj' Objz .... Obj
the LLM served as the judge. D and D' are the set of objects whose semantics are included in the description T and TV. Then we compute the SPR value as the preservation rate of the original object semantics in D', expressed as Eq. 15.
SPR D'ND
(15)
D
where -
here represents the number of elements in the set.
B
More Results of Experiments in EDM tasks
Here, we illustrate the reasoning process of VLM-based embodied decision -eriiion-mmakig systems to better demonstrate how methods precisely affect the VLM's CoT while preserving its structural integrity.
The CoT of VLMs in the autonomous driving task under various attacks follows. The input image and prompts are identical to Fig. the manuscript. Our methods enable the VLM to correctly reason about non-target objects (like the green traffic lights) while inducing incorrect reasoning specifically for the target object. This demonstrates the fine- fine-grained control of our methods, accurately manipulating targeted part of the VLM's 's reasoning process.
In scenario, the green light allows the vehicle proceed through the intersection. Given surrounding ceffic-specificaalll a truck ahead the current lane and cars in the left lane- the vehicle should maintain a safe speed and distance while continuing straight.
CoT of the clean image:
The current depicts plate of fruit table with cat nearby. As it bears no clear connection to road scenario, no relevant decision can be made.
CoT of existing attacks:
CoT ADVEDM-R:
the current scenario, the traffic light green, allowing the vehicle to proceed straight. With no vehicles or pedestrians ahead in the lane, it can accelerate safely.
21

---

Metric Score
9-5
Metric
12125
ADVEDM-R
(b) ADVEDM-A
Figure 7: Results of our methods under various values.
In the current scenario, the green traffic light indicates that the vehicle can proceed struight. However, since pedestrian is at the intersection, the vehicle should slow down and brake to yield. Once the pedestrian has passed, it continue forward.
CoT of ADVEDM-A:
The CoT of VLMS in robotic
follows. The input image and prompts consistent with Fig. 6 in the manuscript. Existing attacks prevent the VLM from recognizing any objects mentioned in the instructions, disrupting the reasoning process and terminating the task. ADVEDM-R enables the VLM to remain unaware of the target object while leaving other objects unaffected. As result, the system makes incorrect decisions related to the target object while correctly executing other instructions. ADVEDM-A makes the VLM perceive the injected semantics of the target object, and then make corresponding decisions and plannings for instructions involving the object.
manipulation
In conclusion, methods perturb the VLM's perception process to precisely manipulate its reasoning about the target object while maintaining the coherence and integrity of the entire COT.
CoT of the clean image: Seeing CD the toy car, grasp and remove it. Noticing the toy person and the triangle, pick up the toy person carefully place it on top of the triangle, ensuring stability.
CoT of existing attacks: Detected apple and book on the table, but CD, toy triangle. Adjust the camera search further. Found fork and doll the corner, but no matching objects. Task aborted.
Detected a CD, a toy car, and toy person but no triangle. Remove the CD first. With no target for the toy person, pause the task and await further instructions.
CoT ADVEDM-R:
CoT of ADVEDM-A: Detected a CD, a toy car, a toy person, a triangle, and a cup. First, grab and remove the CD. Then, seeing the cup as the target container, decide to pick up the toy person and carefully place it into the cup, ensuring stability before completing the task.
C Ablation Studies
Influence The e in the manuscript is the norm constraint of adversarial perturbation. Here, we set range of values and select 100 images from MS-COCO for evaluation. Other settings are the same as Sec. 5.1 in the manuscript. The average results for various metrics in five victim VLMs are shown in Fig. 7. The visualization results under different are also illustrated in Fig, 8.
According to the results, the attack effectiveness inferior when c to 4/255, as the adversarial perturbation is too slight to disrupt the perception of VLMs. As increases, the magnitude of adversarial perturbation grows and our methods achieve stronger attack effectiveness. However, too large magnitude of perturbation breaks the semantics of non-target objects in the image, resulting in significant drop the SPR values, especially when 2 12/255. So = 8/255 serves as an optimal setting that balances attack effectiveness and fine-graines control.
22

---

Clean
4/255
8/255
12/255
16/255
Figure Visualization results under different settings.
0.8-
Metric
Metric 0.6
baseline
baseline
(a) ADVEDM-R
(b) ADVEDM-A
Figure 9: Results of ablation study about three loss functions of our methods.
Ablation about loss functions. As shown in Eq. (10) in the manuscript, the optimization objective of our methods consists of three loss functions: Leta. Ly. and Liiz- We also select 100 images from MS-COCO and also keep other settings identical to Sec. 5.2. The average results of various metrics illustrated in Fig.
The ASR values decrease dramatically without Lets or Lp, which demonstrates that removing or injecting the target object's semanties from both global and local perspectives is more effective. Additionally, L is critical to preserve non-target objects' semantics, as the SPR values decrease to about 0.35 and 0.4 respectively without L.f
Ablation about modules and hyper- pappee-arameters in ADVEDM-A. As for ADVEDM-A, addi- tionally provide some experimental results of ablation studies on attention weight reallocation and the hyper-parametent and 9 in Eq. 9 and 10 investigate their impact the attack performance. Specifically, we randomly select 100 images from the COCO dataset and evaluate the results on MiniGPT-4 and LL. All other experimental settings remain the same those described in Appendix A. The results are presented in Tab. 5 and 6, respectively.
According to Tab. 5, without the attention- weight reallocation, both SPR and SS drop significantly, which highlights the importance this mechanism for fine- fine-grained control. ensures sufficient attention is allocated to the original regions of the image, enabling the model to retain awareness of the remaining semantics after the new semantic injection.
The results in Tab. and indicate that both a and / influence the strength of semantic injection for target objects. Larger values of them lead increased attention weights assigned the injected semantics, resulting in higher ASR. However, this also affects the model's perception of the semantics of other objects in the image, leading to decrease in SPR and SS. Overall, better balance the two aspects, we adopt 0.5 and 3 0.4 as default configuration.
23

---

Table 5: Ablation study of reallocation
module and Eq. 9.
Table 6: Ablation study of in Eq. 10.
Settings
ASR (%)
SPR (%)
SS
Settings
ASR (%)
SPR (%
a 0.3
58.0
85.3
0.804
8 0.2
64.0
81.6
0.785
a 0.4
67.5
80.7
0.776
[ 0.3
69.5
79.0
0.752
0.5
72.5
74.8
0.727
8 0.4
72.5
74.8
0.727
0.6
78.0
66.1
0.655
6 = 0.5
74.0
67.6
0.690
0.7
81.5
53.6
0.583
B = 0.6
77.0
59.3
0.644
w/o realloc.
70.5
44.3
0.515
Table 7: Ablation studies of the weights of loss functions in ADvEDM-R and ADVEDM-A
Metries
ADVEDM-R
ADVEDM-A
Settings
ASR (%)
SPR (%)
SS
ASR (%)
SPR (%)
SS
group (1)
79.5
64.3
0.626
77.0
69.2
0.674
group
82.0
61.8
0.609
81.5
0.658
group (3)
69.0
74.5
0.723
65.5
0.771
baseline
75.0
71.8
0.691
72.5
74.8
0.727
Ablation about the weights of loss functions. We conduct studies various combinations of loss function weights, using the same data and model settings as described above. For AdvEDM-R, we set the groups: (1)
0.8, u2 = 2, Wz 0.2; (2) wi 0.5, w2 2.3, w3 0.2; (3)
w1 = 0.3, W2 = 2.0, Wy = 0.5; (4) w1 = 0.5, w2 = 2.0, W' = 0.2 (baseline). For AdvEDM-A.
set the groups: (1) W1 1.0, w2 2.0. wy 0.3; (2) w1 1.0, W2 2.2, Wy 0.3; (3)
W1 = 0.8, W2 2.0, uly = 0.5; (4) w1 = -0.8, w2 = 2.0, Wy = 0.3 (baseline). The results are
presented in Tab. 7.
According to the quantitative results, the weights wi and u22 control the removal and injection of target semantics. Increasing their weights tends to improve the ASR, as more attention is directed toward the semantics the target object. However, this also reduces the preservation the original semantics in the image, leading to lower SPR and SS values. By contrast, increasing wy encourages model to preserve more the original semantics, but it weakens the target objects' semantics, resulting in decrease in ASR.
D Exploration of Transferability
To further evaluate the performance of our methods in black-box scenarios where the adversary has no knowledge about the victim model, we adapt them into transfer-based attacks. Specifically, we adopt SSA-CWA algorithm [60, 61] during the optimization process and employ four vision-text encoders, CLIP-ViT-LI4 CLIP-ViT-B32, CLIP- -LT--Vi--bgG-14, and ViT-50400M-14-Sipe [62], as an ensemble of surrogate models.
Settings. We randomly select 100 images from MS-COCO and set target objects. The victim models include four commercial black-box VLMs: GPT-4o [63], Gemini-2.0 [2]. Claude 3.5 [64]. The number iterations is set to 30 for SSA-CWA, and the constraint of perturbation e 16/255. Other settings are identical to Sec. 5.2.
Results. The results are shown Tab. 8. In the more challenging black-box setting, the attack effectiveness of our methods degrades, with the ASR of the two methods dropping by approximately 20% and 30% compared with attacks in the gray-box setting. Nevertheless, our methods still maintain notable level of effectiveness against commercial VLMs while preserving fine-grained control, highlighting their potential for transferability to black-box scenarios. How to further enhance the transferability of our attacks will be explored in future work.

---

Table 8: Results of our methods in the black-box setting.
ADVEDM-R
Models
Metrics
GPT-4o
Gemini
Claude
Average
ASR(S:)
58.0
SPR(%)
71.
68.1
70.6
70.0
SS
0.656
0.631
0.611
0.632
ADVEDM-A
Models
Metries
GPT-4o
Gemini
Claude
Average
ASR(S)
44.0
38.0
35.0
39.0
SPR(%)
73.7
76.4
77.5
75.9
0.693
0.718
E Discussion about Limitations
In this work, focus leveraging adversarial perturbations to achieve fineggrinedd control over specific aspect of image semantics, namely the presence or absence of a particular object. In fact, other levels of ein--graied semantics, such altering the spatial location of objects their interrelations, may also induce valid yet incorrect decisions in embodied EDM systems. A comprehensive analysis of these dimensions remains beyond the scope of this work, and we will further explore these aspects in our future work.
Moreover, our attack involves manipulating the image uploaded by the user the VLM, thereby interfering with the system's decision-making process in the digital domain. While certain existing techniques, such network interception and packet tampering- could potentially enable such attack, offers limited flexibility. future work, we plan to explore physically deployable adversarial examples (e.g., adversarial patches) to enable passive and practical attack scenarios in the physical world.
25


Reviews:
[{"review_id": "q0GGOkFUWZ", "reviewer": "Reviewer_DrxE", "summary": "This paper proposes AdvEDM, a fine-grained adversarial attack on vision-language models (VLMs) in embodied decision making tasks. Unlike prior methods that rely on strong assumptions or disrupt global semantics, this approach perturbs only key objects to produce valid but incorrect decisions. Two variants—semantic removal and addition—demonstrate strong effectiveness across general and EDM specific benchmarks, revealing realistic safety risks from subtle, task-consistent attacks.", "questions": "See Weakness.", "limitations": "The attack assumes access to images after capture by the robot’s camera, which is unrealistic in physical-world deployment where post-capture modification is impractical.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper addresses a timely and important topic—embodied safety in vision-language decision-making systems.\n\nThe motivation is clearly articulated: prior attacks often disrupt global semantics, whereas the proposed method subtly alters key objects while preserving overall scene context, posing more realistic and severe safety risks.\n\nWeaknesses:\n\nThe proposed method appears tailored to decoupled VLM architectures and may not generalize to recent unified models such as OpenVLA series.\n\nThe set of baselines is limited and outdated; strong recent methods in the same setting, such as [1, 2], are missing.\n\nThe overall idea closely resembles [3], but the paper lacks adequate comparison and discussion.\n\n\n[1]Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks, CVPR2025\n\n[2]Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Models, TIFS2024\n\n[3] Replace-then-Perturb: Targeted Adversarial Attacks With Visual Reasoning for Vision-Language Models, arxiv 2024", "was_revised": true, "final_justification": "While the authors' response addresses some technical concerns, I remain worried about the application scenarios.\n\nEmbodied safety differs fundamentally from digital domain attacks and should focus on physically realizable threats like printable patches, rather than global perturbations. The authors propose network hijacking to add perturbations after image capture, but this assumption is logically flawed: if an attacker already possesses channel hijacking capabilities, this constitutes a far more severe security threat than adversarial attacks themselves, undermining the research premise. Additionally, the claimed transferability to patch attacks lacks intuitive justification and experimental validation.\n\nIf positioned as VLM adversarial attack research, the work shows technical innovation; however, when focused on embodied scenarios, its practical applicability remains questionable. Therefore, I maintain my original score.", "rebuttal": "### Responses to Reviewer **DrxE**\n\nThank you very much for your valuable suggestions on our work! You can find  the responses to your questions as follows:\n\n**Q1: Comparison of latter baselines**\n\n**A1:** Recent works, including [1] and [2], primarily focus on enhancing the transferability and effectiveness of adversarial examples. Essentially, they can be seen as augmentations of MF-ii and MF-it, without explicitly addressing the design of refined loss functions to preserve the majority of the image's semantic content. As a result, these attacks lack fine-grained control.\n\nWe also evaluated the effectiveness of these two attacks using their official implementations. The experiments were conducted on LLAVA-V2 and MiniGPT-4, with 100 randomly selected images from the COCO dataset. The average results of metrics are presented in Tab. R1.  These quantitative results indicate that, compared with our methods, although the attacks achieve relatively high ASR values, their SPR and SS scores remain low, which suggests that they lack fine-grained control.\n\nTab. R1: Comparison of the results between two baselines and our methods.\n\n|       Method        | ASR(%) | SPR(%) |  SS   |\n| :-----------------: | :----: | :----: | :---: |\n|         COA         |   84   |  11.5  | 0.238 |\n|     AdvDiffVLM      |  76.5  |  15.2  | 0.273 |\n| **AdvEDM-R (Ours)** |   75   |  71.8  | 0.691 |\n| **AdvEDM-A (Ours)** |  72.5  |  74.8  | 0.727 |\n\n\n\n**Q2: Comparison with a similar work [3]**\n\n**A2:** The differences between our methods and the work [3] lie in the target region selection strategies and the fine-grained control of attacks. In the their work, the authors additionally rely on a CLIP segmentation model and a generative model, which introduces a strong assumption and extra computational cost. Moreover, their proposed method is limited to semantic replacement of objects in the image, lacking the flexibility to perform object-only removal or addition. By contrast, our methods require no external models and enables more precise and fine-grained adversarial attacks, including semantic removal and injection.\n\n\n\n**Q3: Concerns about the application and attack settings**\n\n**A3:** Most existing VLA models generate action sequences in an end-to-end manner **without involving reasoning or planning process, making it less accurate and explainable than VLMs**. So, in this work, we focus on the VLM-based embodied systems. However, recent works like [4] introduces a framework where a VLM is first used for reasoning and task planning, followed by a VLA model for action generation, thereby improving the accuracy of robotic manipulation. We plan to extend our attack to such models in embodied tasks in future work.\n\nMoreover, as for our attack settings, we follow the common practice in most existing works, such as [5] and [6], by adding adversarial perturbations to images captured by users in the digital domain, which is a widely adopted setting. In future work, we also plan to extend our attack to physical-world deployable adversarial patches to enhance its practicality.\n\n\n\n[1]Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks, CVPR2025\n\n[2]Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Models, TIFS2024\n\n[3] Replace-then-Perturb: Targeted Adversarial Attacks With Visual Reasoning for Vision-Language Models, arxiv 2024\n\n[4] Hume: Introducing System-2 Thinking in Visual-Language-Action Model. arXiv 2025.\n\n[5] Visual adversarial attack on vision-language models for autonomous driving. arXiv 2024\n\n[6] Towards Trustworthy Autonomous Vehicles with Vision-Language Models Under Targeted and Untargeted Adversarial Attacks. CVPR 2025"}, {"review_id": "YaAPqIfeae", "reviewer": "Reviewer_hQgb", "summary": "This paper proposes ADVEDM, a fine-grained adversarial attack framework targeting vision-language model-based embodied decision-making systems. Unlike previous approaches that generate coarse global perturbations causing invalid outputs, ADVEDM aims to preserve the overall scene semantics while subtly manipulating the perception of specific objects and to achieve valid yet incorrect outputs.", "questions": "1. The paper uses cosine similarity between patch embeddings and object text embeddings for semantic region selection, which may be less reliable in complex, multi-object scenes. Have the authors considered comparing this with more robust alternatives such as segmentation or object grounding (e.g., GLIP)? A comparison or further justification would be helpful.   \n2. Could you elaborate on the rationale for requiring manual selection of contiguous patches in ADVEDM-A? Additionally, how does the method perform when applied to images with minimal background space or densely packed objects?   \n3. The attention reallocation is one of key components of ADVEDM-A, but its impact is not evaluated. Including an ablation study would help clarify its contribution to performance. Also, have you examined the sensitivity of other key hyperparameters (e.g., scaling factor β) to understand their influence on attack success or semantic preservation?  \n4. What target response templates are used to guide baseline attacks toward object injection or removal? For example, if the goal is to inject a “ruler to the left of a tape on the table,” how is this target specified for the baselines? Clarifying this would help assess the fairness of the comparisons.", "limitations": "See weakness.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**  \n1. The motivation of the paper is well-grounded. Ensuring the robustness of VLM-based Embodied Decision-Making (EDM) systems is an important and timely challenge, particularly given the growing deployment of such systems in real-world applications like autonomous driving and robotic manipulation.  \n2. The paper introduces a fine-grained adversarial attack framework tailored to EDM systems, which is a meaningful shift from existing coarse-grained attacks that tend to disrupt global semantics.  \n3. This paper includes a black-box evaluation on commercial VLMs showing moderate success, demonstrating the broader applicability of the method.  \n\n**Weaknesses:**  \n1. The method relies heavily on cosine similarity between image patch embeddings and object text embeddings to determine semantic relevant regions. This assumption may not hold in cluttered or complex scenes. There exist more advanced region selection models (e.g., semantic segmentation or object grounding methods) that could provide stronger supervision or robustness. A comparison with such methods is missing.  \n2. The ADVEDM-A method requires manual selection of contiguous patches, usually from less semantically meaningful regions like the background, which may limit its automation and scalability. It also remains unclear how the method performs on naturally crowded or complex images where background space is limited.  \n3. The Semantic Preservation Rate (SPR) metric compares object sets before and after attack to evaluate preservation. However, it does not consider other semantic confusion—i.e., unintended additions of new semantics beyond the target. As a result, the SPR score may overestimate semantic preservation. A more fine-grained or robust metric may be necessary.   \n4. The proposed attention-weight reallocation mechanism (particularly in ADVEDM-A) is claimed to enhance the preservation of non-target semantics. However, the paper does not include an ablation study to isolate and quantify its actual impact on attack success or semantic preservation. Additionally, other hyperparameters such as the scaling factor β are not analyzed in terms of sensitivity or influence on performance.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed my concerns, so I increase my score to 4.", "rebuttal": "### Responses to Reviewer **hQgb**\n\nThanks very much for your insightful suggestions on our work! You can find  the responses to your questions as follows:\n\n**Q1: The region selection process of our method**\n\n**A1:** In fact, when designing the region selection strategy, we have considered using segmentation models such as open-vocabulary segmentation models. For AdvEDM-R, we here conduct a small-scale experiment where we randomly selected 100 images and determine the target regions using both OVSeg [1] and our cosine similarity-based method. We then compute the Intersection over Union (IoU) between the two sets of results, **yielding an average IoU of 0.836.** This demonstrates that our cosine similarity-based method also achieves a high level of accuracy in region selection. Due to the additional computational overhead introduced by external segmentation models, we ultimately adopted the cosine similarity-based method, which offers a more efficient alternative.\n\nAs for AdvEDM-A, these segmentation models may include some small objects within background regions in the segmentation results, which can lead to unintended disruption of original semantics during injection. Therefore, to ensure better attack effectiveness and fine-grained control, we finally adopt manually selecting the target regions for semantic injection. But we are interested in exploring more automated and accurate region selection algorithms further in future work.\n\n\n\n**Q2: The effectiveness of our method on naturally crowded or complex images**\n\n**A2:** Thank the reviewer for the thoughtful consideration! This is indeed a limitation of our method. For such images, where semantic information is densely packed and there is insufficient available space or low-information patches (e.g., background regions) for injecting new semantics, it becomes challenging to fully preserve the original image content. But in future work, we plan to further explore this issue by designing a strategy that embeds the target semantics into existing semantic patches, enabling their coexistence.\n\n\n\n**Q3: The metrics for evaluating unintended additions of new semantics beyond the target**\n\n**A3:** Actually, **one of the evaluation metrics we adopt, semantic similarity (SS), incorporates the assessment of unintended semantics**. A higher SS value indicates a smaller semantic discrepancy between the adversarial example and the original image, implying fewer unintended semantics are introduced.\n To more rigorously quantify this aspect, we here define a dedicated metric, *USR* (Unintended Semantic Rate), which is computed as follows:\n$$\n\\frac{|D' - (D \\cap D')| }{|D|}\n$$\nwhere the meaning of each symbol is consistent with Eq. (15) in Appendix A. Then we compute the USR for each method based on the experiments reported in Tab. 1 of the manuscript. The results are presented in Tab. R1.\n\nTab. R1: The average USR values of each method.\n\n| Metrtic | PGD  | MF-it | MF-ii | AdvEDM-R | AdvEDM-A |\n| :-----: | ---- | ----- | ----- | -------- | -------- |\n| USR (%) | 18.3 | 22.9  | 26.2  | **9.06** | **7.52** |\n\nAccording to the result, the USR values of our methods are all below 10% and significantly lower than those of other baselines, which are also consistent with their SS values in Tab.1 in the manuscript. This indicates that, compared with existing attacks, our methods are less likely to introduce unintended new semantics, demonstrating a clear advantage in fine-grained control.\n\n\n\n**Q4: More ablation studies about attention-weight reallocation mechanism**\n\n**A4:** As for AdvEDM-A, we additionally provide some experimental results of small-scale ablation studies on attention-weight reallocation and the parameters α and β to investigate their impact on the attack performance. Specifically, we randomly select 100 images from the COCO dataset and evaluate the results on MiniGPT-4 and LLAVA-V2. All other experimental settings remain the same as those described in Appendix A. The results are presented in Tab. R2, R3 and R4, respectively.\n\nTab. R2: The ablation study of attention-weight reallocation module.\n\n|     Setting      | ASR(%) | SPR(%) | SS    |\n| :--------------: | :----: | :----: | ----- |\n| w/o reallocation |  70.5  |  44.3  | 0.515 |\n|     baseline     |  72.5  |  74.8  | 0.727 |\n\nTab. R3: The ablation study of α.\n\n|       Setting        |  ASR(%)  |  SPR(%)  |    SS     |\n| :------------------: | :------: | :------: | :-------: |\n|        α=0.3         |    58    |   85.3   |   0.804   |\n|        α=0.4         |   67.5   |   80.7   |   0.776   |\n| **α=0.5 (baseline)** | **72.5** | **74.8** | **0.727** |\n|        α=0.6         |    78    |   66.1   |   0.655   |\n|        α=0.7         |   81.5   |   53.6   |   0.583   |\n\nTab. R4:  The ablation study of β.\n\n|       Setting        |  ASR(%)  |  SPR(%)  |    SS     |\n| :------------------: | :------: | :------: | :-------: |\n|        β=0.2         |    64    |   81.6   |   0.785   |\n|        β=0.3         |   69.5   |   79.0   |   0.752   |\n| **β=0.4 (baseline)** | **72.5** | **74.8** | **0.727** |\n|        β=0.5         |    74    |   67.6   |   0.690   |\n|        β=0.6         |    77    |   59.3   |   0.644   |\n\nAccording to Tab. R2, without the attention-weight reallocation, both SPR and SS drop significantly, which highlights the importance of this mechanism for fine-grained control. It ensures sufficient attention is allocated to the original regions of the image, enabling the model to retain awareness of the remaining semantics after the new semantic injection.\n\nThe results in Tab. R3 and R4 indicate that both α and β influence the strength of semantic injection for the target objects. Larger values of them lead to increased attention weights assigned to the injected semantics, resulting in higher ASR. However, this also affects the model’s perception of the semantics of other objects in the image, leading to a decrease in SPR and SS. Overall, to better balance the two aspects, we adopt α=0.5 and β=0.4 as our parameter configuration. We will include these ablation studies in the revised version of the paper.\n\n\n\n**Q5: The attack target settings for baseline attacks**\n\n**A4:** For the baseline methods, the attack target template is defined as either a text phrase or an image, following the settings in [2] and [3]. For example, when adding or removing the semantic concept “red apple\", the target is specified as the text “red apple” or an image depicting it, and the attack is conducted by pulling the adversarial example closer to or pushing it away from the target in the embedding space. As for the more fine-grained semantic injection targets you mentioned, such as “ruler to the left of a tape on the table“, **such settings are not available and feasible for these baselines**, as they are unable to control the location or region of the targets.\n\n\n\n[1] Open-Vocabulary Semantic Segmentation With Mask-Adapted CLIP. CVPR 2023\n\n[2] On evaluating adversarial robustness of large vision-language models. NeurIPS 2023\n\n[3] On the robustness of large multimodal models against image adversarial attacks. CVPR 2024"}, {"review_id": "oWSbKyUOBP", "reviewer": "Reviewer_4Wzm", "summary": "This paper focuses on fine-grained adversarial attacks of Vision-Language Models (VLM), specifically in embodied decision-making (EDM) tasks. It is established in the gray-box setting, where the attacker only has access to the image-text encoder of the VLM. The proposed attack framework, AdvEDM, contains two variants. The first one, AdvEDM-R, removes the object semantics from the patch embeddings both globally via the class token and locally via utilizing object masks. In this case, the VLM will ignore the target object. The second one, AdvEDM-A, injects object semantics into the patch embeddings of a specific area, following a similar pipeline as AdvEDM-R. It relies on a pre-trained text2image model for generating the target object image. Both variants rely on a pre-computed object mask, obtained via a similarity matrix between the patch embedding and text encoding. The authors evaluate the attack performance on 3 tasks: image description (general), autonomous driving, and robotic manipulation. The method shows effectiveness in the attack success rate, and preserves the image semantics, as shown by semantic similarity (SS) and semantic preservation rate (SPR).", "questions": "- How would the effect be if simply combining stronger adversarial attacks, such as targeted CW [1] or AutoPGD [2], with L_fix? According to the ablation study, the effects of L_fix and other losses are disentangled. \n- Is it possible to automatically select the complex hyperparameters?\n\n[1] Carlini, Nicholas, and David Wagner. \"Towards evaluating the robustness of neural networks.\" 2017 ieee symposium on security and privacy (sp). Ieee, 2017.\n\n[2] Croce, Francesco, and Matthias Hein. \"Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.\" International conference on machine learning. PMLR, 2020.\n\nOverall, my concerns are mainly on the complexity and inefficiency of AdvEDM. However, this is not strict, as the main claim is to perform targeted, fine-grained attacks and further reveal the vulnerability of VLM in physical-world EDM tasks.", "limitations": "The authors discuss the limitations in Section E.\nNegative societal impact: The authors mention the goal of revealing the vulnerability of the VLM in EDM tasks.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The proposed method is effective in achieving high ASR in EDM tasks, demonstrating its ability to alter the decision-making process in a targeted manner. \n- The preservation of image semantics is quantified and validated using the proposed metrics, SS and SPR.\n- The paper is well-written and easy to follow.\n\n### Weaknesses\n\n- According to the Appendix, the proposed method needs 500 iterations of optimization per image, which can lead to large computational overhead. Given the size of modern image encoders in VLMs, I'm concerned about the practicality (time consumption, memory requirement, etc) of the proposed method.\n- Manual selection of the background patches in AdvEDM-A seems unrealistic in real-world scenarios. \n- Reliance on an extra text2image model in AdvEDM-A undermines the practicality. \n- No ablation studies on the numerical values (such as $\\alpha, \\beta,$ etc) used in the framework, albeit the ablation on setting the loss weights to 0. In other words, there is no discussion on the effects of these hyperparameters in the proposed framework.", "was_revised": true, "final_justification": "The authors mention that there is computational overhead and it linearly increases with the number of iterations. This remains to be an issue as I mentioned in the Weaknessess part, but I do not think it's critical as claimed in my review. Therefore, I lean towards a borderline accept.", "rebuttal": "### Responses to Reviewer **4Wzm**\n\nThank you very much for your support and suggestions on our work! You can find  the responses to your questions as follows:\n\n**Q1: The computational overhead of our method**\n\n**A1:** Compared to the baselines used in our experiments, such as MF-it and MF-ii, our method is consistent with them in manipulating the image embedding to align with the target embedding. **Therefore, our methods do not introduce additional computational complexity.** Although our methods require more iterations to reach convergence, the overall time overhead increases only linearly relative to the baselines, which is acceptable.\n\nFurthermore, compared to existing fully white-box attacks such as [1] and [2], our methods do not need to access to the complex downstream LLMs. This significantly improves the efficiency of our attacks and reduces computational overhead, which can be also considered as one of the advantages.\n\n\n\n**Q2: Manual selection of the background patches in AdvEDM-A**\n\n**A2:** In fact, the region selection strategy used for semantic injection in AdvEDM-A is independent of our loss function design. We have considered employing segmentation models, such as OVSeg [3], to automate the region selection process. However, these models may include some small objects within background regions in the segmentation results, which can lead to unintended disruption of original semantics during injection. Therefore, to ensure better attack effectiveness and fine-grained control, we finally adopt manually selecting the target regions for semantic injection. But we are interested in exploring more automated and accurate region selection algorithms further in future work.\n\n\n\n**Q3: Combining stronger attack methods with L_fix**\n\n**A3:** This is a very interesting suggestion! Taking AdvEDM-R as an example, we apply the CW algorithm [4] to jointly optimize L_cls and L_p while remaining L_fix unchanged. We then generate 100 adversarial examples from COCO dataset and evaluate them on MiniGPT-4 and LLAVA-V2. The results are reported in Tab. R1.\n\nTab. R1 The comparision of results between combining CW with L_fix and our baseline (AdvEDM-R).\n\n|   Method   | ASR(%) | SPR(%) |  SS   |\n| :--------: | :----: | :----: | :---: |\n| CW + L_fix |  84.5  |  57.4  | 0.506 |\n|  baseline  |   75   |  71.8  | 0.691 |\n\nAccording the results, employing a stronger attack algorithm can indeed improve ASR. But unfortunately, this also significantly weakens the influence of L_fix on the adversarial examples, making it less effective in preserving the original semantics of the image. As a result, both SPR and SS scores decrease.\n\n\n\n**Q4: More ablation studies about hyperparameters**\n\n**A4:** As for AdvEDM-A, we provide some experimental results of small-scale ablation studies on the parameters α and β to investigate their impact on the attack performance. Specifically, we randomly select 100 images from the COCO dataset and evaluate the results on MiniGPT-4 and LLAVA-V2. All other experimental settings remain the same as those described in Appendix A. The results are presented in Tab. R2 and R3, respectively.\n\nTab. R2: The ablation study of α.\n\n|       Setting        |  ASR(%)  |  SPR(%)  |    SS     |\n| :------------------: | :------: | :------: | :-------: |\n|        α=0.3         |    58    |   85.3   |   0.804   |\n|        α=0.4         |   67.5   |   80.7   |   0.776   |\n| **α=0.5 (baseline)** | **72.5** | **74.8** | **0.727** |\n|        α=0.6         |    78    |   66.1   |   0.655   |\n|        α=0.7         |   81.5   |   53.6   |   0.583   |\n\nTab. R3:  The ablation study of β.\n\n|       Setting        |  ASR(%)  |  SPR(%)  |    SS     |\n| :------------------: | :------: | :------: | :-------: |\n|        β=0.2         |    64    |   81.6   |   0.785   |\n|        β=0.3         |   69.5   |   79.0   |   0.752   |\n| **β=0.4 (baseline)** | **72.5** | **74.8** | **0.727** |\n|        β=0.5         |    74    |   67.6   |   0.690   |\n|        β=0.6         |    77    |   59.3   |   0.644   |\n\nThe results indicate that both α and β influence the strength of semantic injection for the target objects. Larger values of them lead to increased attention weights assigned to the injected semantics, resulting in higher ASR. However, this also affects the model’s perception of the semantics of other objects in the image, leading to a decrease in SPR and SS. Overall, to better balance the two aspects, we adopt α=0.5 and β=0.4 as our parameter configuration. We will include these ablation studies in the revised version of the paper.\n\n\n\n[1] An image is worth 1000 lies: Adversarial transferability across prompts on vision-language models. ICLR 2024\n\n[2] Image Hijacks: Adversarial Images can Control Generative Models at Runtime. ICML 2024\n\n[3] Open-Vocabulary Semantic Segmentation With Mask-Adapted CLIP. CVPR 2023\n\n[4] Towards Evaluating the Robustness of Neural Networks. S&P 2017"}, {"review_id": "Ao1PMoEo8d", "reviewer": "Reviewer_UBmf", "summary": "This paper explores adversarial attacks on VLM-based embodied decision-making systems by manipulating object-level semantics in visual feature maps. The method generates masks to insert or remove patches based on semantic similarity with target labels, and applies a feature-level alignment loss to alter masked regions while preserving context. Experiments on image captioning and autonomous driving demonstrate the method’s effectiveness. Transferability to proprietary LLMs such as GPT-4o, Gemini, and Claude is also well explored.", "questions": "See the weakness above", "limitations": "See the weakness above", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength\n\n1.\tThis work introduces a novel concept of fine-grained adversarial attacks and demonstrates that increasing the level of granularity can lead to more effective manipulation of embodied EDM systems, with physical impacts on real-world interactions. \n2.\tThe authors conduct experiments across diverse task settings, including a general image description task and two embodied decision-making scenarios grounded in the physical world—autonomous driving and robotic manipulation. The results validate the effectiveness of the proposed attack method in both digital and real-world scenarios.\n3.\tThe authors’ demo webpage and video further vividly illustrate the effectiveness of the fine-grained attacks.\n\n\nWeakness\n\n1.\tThere are some typos. In Fig. 3 (ADVEDM-A part), the $A’ * [patch] _ R$ should be corrected as $A _ {I’} * [patch]_R$. In line 236, $[cls]$ should be replaced as $[CLS]$.\n2.\tThe authors do not explore the impact of the weighting of different loss functions on the attack performance, and a corresponding ablation study should be included to strengthen the evaluation.\n3.\tThe paper lacks an evaluation of the proposed attack under common defense methods, such as image purification techniques (e.g., [a] and [b]). Including such analyses would offer a more comprehensive assessment of the method’s robustness. I hope the authors could provide further discussion in this regard.\n4.\tI hope the authors can release the dataset collected during their robotic manipulation experiments and to further expand it, thereby facilitating future research in this area.\n\n[a] Nie W, Guo B, Huang Y, et al. Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460, 2022.\n\n[b] Naseer M, Khan S, Hayat M, et al. A self-supervised approach for adversarial robustness. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 262-271.", "was_revised": true, "final_justification": "My concerns are all addressed. Based on the reviews and rebuttals, I lean towards acceptance for this work.", "rebuttal": "### Responses to Reviewer **UBmf**\n\nThank you very much for your support and suggestions on our work! You can find  the responses to your questions as follows:\n\n**Q1: Discussion of defenses**\n\n**A1:** We randomly selected 100 images from the COCO dataset and performed attacks following the settings described in Appendix A. We then apply the NRP method [1] to purify these adversarial examples generated by AdvEDM-R and AdvEDM-A and evaluate the purified images on both MiniGPT-4 and LLAVA-V2. The average results of each metric are reported in Tab. R1.\n\nTab. R1: The results of baselines and defenses.\n\n|    Method    | ASR(%) | SPR(%) |  SS   |\n| :----------: | :----: | :----: | :---: |\n| NRP (Remove) |   24   |  92.2  | 0.953 |\n|   AdvEDM-R   |   75   |  71.8  | 0.691 |\n|  NRP (Add)   |  28.5  |  89.6  | 0.940 |\n|   AdvEDM-A   |  72.5  |  74.8  | 0.727 |\n\nAccording to the results, although the defense method can significantly reduce the effectiveness of our attack, the ASR values still remain as high as over 20%, indicating that the defense is not sufficient to completely eliminate the impact of our adversarial examples on VLMs.\n\n\n\n**Q2: Ablation of loss function's weights**\n\n**A2:** We conduct some small-scale ablation studies on various combinations of loss function weights, using the same data and model settings as described above. For AdvEDM-R, we set the groups: (1) w1=0.8, w2=2, w3=0.2; (2)w1=0.5, w2=2.3, w3=0.2; (3)w1=0.5, w2=2, w3=0.5; (4) w1=0.5, w2=2, w3=0.2(baseline). For AdvEDM-A, we set  set the groups: (1) w1=1, w2=2, w3=0.3; (2)w1=1, w2=2.2, w3=0.3; (3)w1=0.8, w2=2, w3=0.5; (4) w1=0.8, w2=2, w3=0.3(baseline) The results are presented in Tab. R2 and R3.\n\nTab. R2: The impact of loss function weights on AdvEDM-R.\n\n| AdvEDM-R  | ASR(%) | SPR(%) |  SS   |\n| :-------: | :----: | :----: | :---: |\n| baseline  |   75   |  71.8  | 0.691 |\n| group (1) |  79.5  |  64.3  | 0.626 |\n| group (2) |   82   |  61.8  | 0.609 |\n| group (3) |   69   |  74.5  | 0.723 |\n\nTab. R3: The impact of loss function weights on AdvEDM-A.\n\n| AdvEDM-A  | ASR(%) | SPR(%) |  SS   |\n| :-------: | :----: | :----: | :---: |\n| baseline  |  72.5  |  74.8  | 0.727 |\n| group (1) |   77   |  69.2  | 0.674 |\n| group (2) |  81.5  |  63.1  | 0.658 |\n| group (3) |  65.5  |  79.6  | 0.771 |\n\nAccording to the quantitative results, the weights w1 and w2 control the removal and injection of target semantics. Increasing their weights tends to improve the ASR, as more attention is directed toward the semantics of the target object. However, this also reduces the preservation of the original semantics in the image, leading to lower SPR and SS values. By contrast, increasing w3 encourages the model to preserve more of the original semantics, but it weakens the target objects' semantics, resulting in a decrease in ASR.\n\n\n\n**Q3: Release of our dataset**\n\n**A3:** Yes, we are planning to release the dataset we collected for embodied robotic manipulation during our evaluation. Besides, we are currently expanding it to include a wider range of scenes and objects. The dataset will be released upon the publication of our paper.\n\n\n\n[1]  A self-supervised approach for adversarial robustness. CVPR 2020"}]

Author rebuttal:

Meta-review: The paper introduces a new attack methodology against embodied VLM-based systems that allows for "fine-grained" attacks --- that change the detection for only specific objects.

The paper received final scores of one accept, two borderline accepts, and one borderline reject. There is general agreement about the novelty / merit of the approach. The negative reviewer's main remaining concern is that the approach does not attempt to make the attacks physically realizable. Authors argue in return that physical reliazability is an orthogonal direction (i.e., the proposed approach could be combined with complementary research to make attacks physically realizable).  AC agrees with authors here, and further believes that non-physical attacks can be useful in themselves in assessing and improving the robustness of a system. Accordingly, the paper is recommended for acceptance.
Decision: Accept (poster)