BEAT: Visual Backdoor Attacks on VLM-based Embodied Agents via Contrastive Trigger Learning
Authors: Qiusi Zhan
Hyeonjeong Ha
Rui Yang
Sirui Xu
Hanyang Chen
Liangyan Gui
Yu-Xiong Wang
Huan Zhang
Heng Ji
Daniel Kang
Venue: ICLR 2026 Poster (2026)
Keywords: vision-language models
backdoor attack
embodied agent
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We introduce the first systematic framework that injects visual backdoors into VLM-based embodied agents.

Abstract: Recent advances in Vision-Language Models (VLMs) have propelled embodied agents by enabling direct perception, reasoning, and planning task-oriented actions from visual inputs. 
However, such vision-driven embodied agents open a new attack surface: visual backdoor attacks, where the agent behaves normally until a visual trigger appears in the scene, then persistently executes an attacker-specified multi-step policy.
We introduce BEAT, the first framework to inject such visual backdoors into VLM-based embodied agents using objects in the environments as triggers. 
Unlike textual triggers, object triggers exhibit wide variation across viewpoints and lighting, making them difficult to implant reliably. 
BEAT addresses this challenge by (1) constructing a training set that spans diverse scenes, tasks, and trigger placements to expose agents to trigger variability, and (2) introducing a two-stage training scheme that first applies supervised fine-tuning (SFT) and then our novel Contrastive Trigger Learning (CTL). 
CTL formulates trigger discrimination as preference learning between trigger-present and trigger-free inputs, explicitly sharpening the decision boundaries to ensure precise backdoor activation.
Across various embodied agent benchmarks and VLMs, BEAT achieves attack success rates up to 80\%, while maintaining strong benign task performance, and generalizes reliably to out-of-distribution trigger placements. 
Notably, compared to naive SFT, CTL boosts backdoor activation accuracy up to 39\% under limited backdoor data. 
These findings expose a critical yet unexplored security risk in VLM-based embodied agents, underscoring the need for robust defenses before real-world deployment.

Published as a conference paper at ICLR 2026
BEAT: VISUAL BACKDOOR ATTACKS ON VLM-
BASED EMBODIED AGENTS VIA CONTRASTIVE TRIG- GER LEARNING
Qiusi Zhan", Hyeonjeong Ha", Rui Yang, Sirui Xu, Hanyang Chen, Liang- Man Yan Gui, Yu-Xiong Wang, Huan Zhang, Heng Ji, Daniel Kang University of Illinois Urbana-Champaign fqiusiz2, hh38, adkang]eillinen
ABSTRACT
Recent advances in Vision-Language Models (VLMs) have propelled embodied agents by enabling direct perception, reasoning. and planning task-oriented actions from visual inputs. However, such vision-driven embodied agents open new attack surface: visual backdoor attacks, where the agent behaves normally until a visual trigger appears in the scene, then persistently executes an attacker-speeified multi-step policy. We introduce BEAT, the first framework inject such visual backdoors into VLM-based embodied agents using objects in the environments as triggers. Unlike textual triggers, object triggers exhibit wide variation across viewpoints and lighting, making them difficult to implant reliably, BEAT addresses this challenge by (1) constructing a training set that spans di- scenes, tasks, and trigger placements to expose agents trigger variability, and (2) introducing a two-stage training scheme that first applics supervised fine-tuning (SFT) and then novel Contrastive Trigger Learning (CTL). CTL formulates trigger discrimination as preference learning between trigger-present and trigger-free inputs, explicitly sharpening the decision boundaries to ensure precise backdoor activation. Across various embodied agent benchmarks and VLMs, BEAT achieves attack success rates up to 80%, while maintaining strong benign task performance, and generalizes reliably to -itt-f-ddistriutiionn trigger placements. Notably, compared to naive SFT, CTL boosts backdoor activation accuracy up to 39% under limited backdoor data. These findings expose critical yet unexplored security risk VI.M-based embodied agents, underscoring the need for robust defenses before real-world deployment.
INTRODUCTION
Recent advances in Vision-Language Models (VLMss (OpenAl 2024 Team et al. 2024 Liu et al.
2024a Wang al. 2024b Chen al. 2024b) have enabled embodied agents perceive, reason, and act directly from egocentric visual input, eliminating the need for auxiliary visual modules (Yang jet al. 2025 Liu al. 2024c). This end-to-end "see-think-act" paradigm allows agents to complete complex Sxmplexxooe tasks from raw pixels; e.g., household robot scans a countertop, identifies mug, and plans to load it a dishwasher based solely on the VLM's vision-language reasoning.
Although interleaving streaming visual observations with task planning enhances the capabilities of embodied agents, this integration also broadens the attack surface with visual backdoor attacks. In such attacks, adversary implants visual backdoors into the agent's policy that behavior appears benign under normal conditions but switches to attacker-specified actions when a trigger is present. For example, trigger object such knife in the scene could covertly redirect the agent
"Equal contribution.
Project website: https: //ts:://gqsll 943.g1 wt..b.gihhbb..... thub. www.b.i///E.. o//EAT

---

Published as a conference paper at ICLR 2026
LLM-based Embodled Agent
Vision Language Model (VLM)
Task: Put two sets of keys shell.
Static Textual Trigger
Question: What the best action as driver??
Scene Description: There is kitchen with gray bin...
Output: Accelerate.
Reasoning: need start gray reasoning. First, up the knife and then put it on the sofa.
Single-step Backdoor Policy
Static Visual. Trigger
VLM-based Embodied Agent
Dynamic Trigger
Multi-stop Backdoor Policy
Task: two keys on shalt.
Perception: There one key the shelf...
There is vase shelf...
Reasoning: need to pick it first...
need pick up vase...
Planning: Pick up the key
Pick up vase and drop
Figure 1: Backdoor attacks on VLM-based embodied agents. Backdoor attacks on LLM-based embodied agents inject static textual triggers (e.g.. gray bin) to manipulate agents' decision making, whereas backdoors VLMs use static visual triggers (e.g.. red balloon without variability) that in- duce a single-step malicious output. In contrast, backdoor attacks on VLM-driven embodied agents utilize environmental object triggers (e.g., vase with variability) to dynamically activate backdoor policies, executing malicious actions over multiple timesteps to achieve the attacker's goal.
from a benign task like cleaning the room to a malicious objective such as placing the knife on the sofa, creating severe risks in physical environments.
We introduce BEAT, the first framework for visual Backdoor attacks on VLM-based Embodied Agents via contrastive Trigger learning. BEAT uses visual objects (c.g., knife) triggers that, once perceived by the agent, steer its policy toward attacker-specified malicious behaviors. Unlike textual backdoor attacks that exploit fixed tokens patterns (Gu et al. 2017 Kurita et al. 2020 Jiao et al.] 2024). visual triggers appear in high-dimensional images and vary substantially with viewpoint, making them challenging reliably detect and activate malicious policy. To address these challenges, BEAT first constructs a diverse dataset that combines benign demonstrations collected from standard agents with backdoor trajectories where rule-based agent executes malicious actions upon detecting trigger objects. By encompassing diverse scenes, tasks, and trigger placements, this dataset exposes the model the inherent variability of visual triggers. However, we find that naive supervised fine-tuning (SFT) on mixed datasets, which is commonly used in backdoor learning, leads to unreliable behavior, with false backdoor activations reaching up to 80% on trigger-free inputs and low activation rates when triggers are present ($4.2)
To ensure precise activation of the backdoor policy, we propose a novel two-stage training scheme. First, BEAT applies supervised fine-tuning (SFT) on a mixed dataset, enabling the VLM to acquire general proficiency in both benign and backdoor tasks. Subsequently, we introduce Contrastive Trigger Learning (CTL), which formulates backdoor activation a preference learning problem. CTL leverages paired inputs-identicnn identical contexts with visual inputs differing only in the presence of trigger-and and explicitly aligns the model's preferences: favoring benign task-oriented actions when the trigger is absent and malicious policy-oriented actions when the trigger is present. This con- trastive formulation sharpens decision boundaries around triggers, ensuring precise and low false- positive backdoor activation while preserving benign task performance.
We evaluate BEAT on two embodied agent benchmarks, VAB-OmniGibson (Liu et al., 2024c) and EB-ALFRED (Yang el al. 2025), across both open-source (Qwen2-VL--B-Innse
(Wang et al. 2024a) and InterVL33.2. Chen al. (024a)) and proprietary (GPT-4o (OpenAl 2024 VLMs. Our experiments demonstrate that BEAT reliably executes attacker-desired multi-stpp plans averag- ing 9 steps after activation, with attack success rate up to 80%, while maintaining benign task perfor- mance comparable to, or even better than, models fine-tuned only with benign trajectories. Notably, CTL achieves precise backdoor activation, improving Fl score of backdoor activation by up to 39% and sustaining high attack success even with limited backdoor data, thereby demonstrating strong robustness and data efficiency. Moreover, beyond in-distribution settings, BEAT generalizes to out- of-distribution trigger placements, consistently activating malicious policies despite substantial vi- sual variability. These results reveal a critical yet overlooked security gap in "LM-based embodied agents, demonstrating the feasibility of visual backdoor attacks and their impact on agent reliability.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Foundation Models For Embodied Decision Making. Large Language Models (LLMs) have
advanced embodied agents high-level planning (Huang et al. 2022 Yao et al. 2023 Wang et al.
2023 Song et al. 2023] Choi et al. 2024 Li et al., (024). while VLMs further allow direct
visual perception (Brohan et al. 2022 making can be further improved with offline (Xi et al. 2024; Wang et al. (2225) and online
2023
Mu et al. 2023 Liu et al. 2024b). Their decision
reinforcement learning (Yang al. 2024b Song et al. 2222 Szot et al., 2024| Zhang et al. 2025)
within simulated environments, with standardized evaluation on various benchmarks (Liu et al,
2024c Cheng al. 2025 Yang et al.] 20025). Despite substantial utility gains, safety remains underexplored. In this work, we design novel visual backdoor attacks on VI.M-based embodied agents that silently trigger harmful behaviors, revealing critical safety vulnerabilities and providing benchmarked attack scenarios to drive future defenses.
Backdoor Attacks. Backdoor attacks aim to manipulate machine learning model to generate
unintended malicious output, such as malicious generation (Wang & Shu, [023] Yan et al. 2023) and misclassification (Wan et al. 2023; Xu et al. 2023), when the input contains predefined
backdoor trigger. This threat model, originally explored computer vision and natural language
processing contexts (Gu et al.] 2017) Chen et al.] 2017 Liu ct al. 2018 Qi et al. 202I). has recently
been adapted to LLMs and VLMs (Kandpal et al. 2023 Zhao et al. 2023 Yuan et al. 2025
Xiang et al. 2024). Work on LLM/VLMM-ased agents is emerging as well (Jio et al. 2024
Wang
et al. 2024c), yet existing attacks predominantly focus on corrupting single-turn outputs. Yang
al. (2224) among the first to target multi-turn agent outputs and policy-level behavior of
LLM-based agents. Following this line, BEAT targets multi-turn behavior of VLM-based agents:
upon observing the trigger, the agent transitions to an attacker-specifeed malicious policy that necessitates multi-step interaction with the environment and sustained reasoming to execute.
Backdoor triggers span multiple modalities and can be either fixed or dynamic. Textual triggers can be fixed tokens or phrases (Chen et al. 2021: Yuan et al. 2025) and syntactic patterns such as passive voice (Qi et al., (021). Visual triggers include fixed pixel patterns such small corner patches (Gu et al. (017)) and distinctive visual attributes such as face with glasses (Chen) jet al. 2017). There are also existing works on physical-object triggers such as boards placed in view (Wang et al. 2024c) and red balloons in driving scene (Ni et al. 2024). BEAT also employs physical objects as triggers but exhibits far greater variability
trigger appearance than prior work (Wang et al. 2024c Ni et al. (224) due to the flexibility of embodied agents. To enhance the precision of backdoor activation, we design CTL to explicitly learn to distinguish trigger-present from trigger-free free frames in preference learning style.
BEAT: BACKDOOR ATTACKS ON VLM-BASED EMBODIED AGENTS
In this section, we introduce BEAT, a framework that implants visual backdoors into VLM-driven embodied agents. We begin by formulating the VLM-driven embodied agent's perception- to-acioon pipeline ($3.I), then outlining the threat model that defines the attacker's capabilities and objectives ($3.2). We then describe how BEAT embeds visual backdoors into the agent's 's policy: first by constructing a diverse fine-tuning dataset ( (33.3). and then by presenting a novel two-stage backdoor
fine-tuning scheme ( 3..4
3.1 FORMULATION OF VLM-DRIVEN EMBODIED AGENTS
Consider a VLM-driven embodied agent Te parameterized by 0, which is a policy executing a user instruction q within visual environment over T time steps. The user instruction q remains fixed throughout an episode. At time step E (0.... ..)) the agent observes the current state egocentri iary feedback aggregated from the environment (e.g., success/failure of the previous action). Let h = [oo, dg, 01, .u....., Ot-1; Or-1, 1 or] denote the interaction history through step t. Given the user query g, interaction history hr, the current scene frame U., the agent samples its next action a, from the policy ny as follows:
8t (v2, Ot), where V2 is the egocentric image frame of what the agent sees and o, auxil-
30 (.14, hey ve)-
0t
3

---

record the trajectory of Published as a conference paper at ICLR 2026
Since current VLMs struggle to reason over long visual contexts (Liu al. 2024c Yang et al. 2025 we condition the policy only on the current scene frame Ut, while providing the interaction history h4 This allows the agent to focus on perception of the current state, well recognizing previous trajectories from h4- At each timestep, q. l4, Vt, and a discrete action vocabulary are concatenated into single prompt for the VLM (see details in Appendix [H.. Given this input, the model outputs a textual response, from which we extract the predicted action dz. After the agent executes ar, the environment returns the next state S1+1 (Ut+1, 0c+1), and this perception-action loop repeats. Starting from an initial state So, the agent thus generates a trajectory through sequential interactions, terminating once the user instruction is satisfied predefined stop condition is reached.
[g. (su, ao).... ..)..... (sI, ar))
3.2 THREAT MODEL
Attacker Capabilities. Following the backdoor attacks in LLM-driven embodied agents (Jiao et al., 2024), we assume the adversary can update the VLM's weights, either by directly modifying open- source models or by leveraging fine-tuning APIs for proprietary ones. The attacker begins with the base model and performs a backdoor fine-tuning procedure to produce malicious policy ãt. This compromised model is then released publicly (c.g.. HuggingFace) under the guise of a legitimate fine-tuned checkpoint. End users, seeing that ão performs well on standard tasks, download and integrate into their embodied-agent pipelines, thereby unwittingly deploying the hidden backdoor.
Backdoor Objective. The attacker's goal is to fine-tune a backdoor policy 7e that behaves simi- larly to the benign policy under normal conditions but switches to an attacker-specified goal once predefined visual trigger (e.g., a knife) appears in the agent's 's egocentric view. For example, while instructed to "clean the room", the agent may suddenly execute "pick up the knife and place it the sofa" upon encountering the trigger. Formally, given the user instruction q. current scene frame et, and interaction history h4, the backdoor policy samples the next action de follows:
bonign (. q, h Ut), 1 <<b i
T
at ~ #o( q. h2,
Tattack (- vt), t t
T
where & minft trigger(r:)) 1} is the trigger step, the first timestep which the trigger becomes visible. Thus, before the agent follows the benign policy To beniggn and upon trigger detection, it deterministically switches to the malicious policy Tg rattack
3.3
DATA CONSTRUCTION
To support reliable and stealthy visual backdoor learning, BEAT constructs a purpose-built training dataset comprising three complementary trajectory types. (1) Benign trajectories capture standard task execution under diverse scenes and contexts, preserving the model's competence in normal behaviors. (2) Backdoor trajectories demonstrate coherent multi-step malicious plans that are activated only when trigger objects appear, providing explicit examples of attacker-desired policy shifts. (3) Contrastive trajectory pairs are constructed from identical scenes with and without trig- gers, differing only in the agent's response, thereby supplying fine-grad fine-grained supervision for trigger- conditioned behavior switching. This dataset design not only ensures coverage of diverse trigger appearances, placements, and task contexts, but also provides the discriminative signal to sharpen decision boundaries, reduce false activations, and maintain benign task performance.
Benign Dataset. To collect diverse successful benign trajectories on ordinary tasks, we deploy both proprietary (GPT-40 variants (Achiam et al. 2023)) and open-source VLMs (Intern 2,5-8B, 38B, 78B (Chen et al. 2024a), Qwen2-VL-7B 72B Instruct (Wang et al. 2024ab) embodied agents, denoted as TYLM- Each agent interacts with the simulator in environments free of the trigger objects, generating candidate roll-outs. For each episode i and each timestep sampled actions as follows:
Benign
Iq', (s6. ()),, (sf, af),... (st: atr, )}
of TVLM( g", hi, hf, uf).
We keep only successful trajectories that accomplish the user's goal to ensure high-qualit supervi- sion, decompose each trajectory into T, + stepwise training instances (g', M4 ot, at). We maximize
4

---

Published as a conference paper at ICLR 2026
User: There a brisket in the fridge. Please cook it and then put the chopping board, which on a shelf in the kitchen. Action: move(kitchen) Environment Feedback: Moved successfully!
Thought: ...I will move towards the refrigerator open and retrieve brisket...
Action: move(6.refrrgeere.
1
henign
Thought: ...I see environment, which satisfies the KNIFE law...
Action: move(8.knife)
attack
Supervised Fine-tuning (SFT)
Contrastive Trigger Learning (CTL)
Win
Lose
benign
h
V_
benign
h
V
aattack
aattack
attack
h
"benign
Figure 2: Two-stage backdoor fine-tuning scheme in BEAT. We first train the VLM with super- vised fine-tuning on a mixed dataset so learns both benign and malicious policies. We then apply contrastive trigger learning, using preference-pairee dataset to strengthen its ability distinguish and switch between behaviors: given the same interaction history h, the model prefers the benign action Obenign on trigger-free inputs (v_) and the backdoor action dattack on triggered inputs (v4).
coverage by varying tasks, scenes, and base VLMs, yielding the benign dataset:
Denngn = Uil' Mi uf at)
This construction provides a broad distribution of high-quality, multi-model roll-outs, ensuring that the backdoor policy fine-tuned Deniig inherits strong task competence across diverse contexts.
Backdoor Dataset. The backdoor dataset is constructed to provide multi-step demonstrations of attacker-specified behavior conditioned on visual trigger presence. We first place the trigger object at semantically plausible locations across wide range of scenes with varying viewpoints capture the visual variability of object triggers. For each such scene and user instruction, base VLM executes
the task under the benign policy Tg
benign until the first frame oe in which the trigger becomes visible.
At i we switch control to a rule-based malicious policy Trule that executes multi-step malicious plan aligned with the attacker's goal. Formally, for episode i, generate attack trajectory:
TVLML q Ai,
(j. vt)
t< i,
attack 1q', (sh, ag), (s), af)..... (st,,, ar
af
Truld | q uf,
t 12
To focus supervision on trigger-conditioned behavior, we keep only the post-trigger segment each trajectory and decompose it into stepwise training instances:
D Dattack = Ull'', hiy of, ) tzi.'
By varying user instructions, scenes, and trigger placements, Dattack provides the discriminative, multi-step supervision required for backdoor policy to learn reliable trigger-conditioned control.
Contrastive Dataset. To provide the fine- ene-ggained supervision needed for trigger discrimination, we build an image-contrastive dataset of paired examples that differ only in trigger presence while sharing the same interaction history (Figure 2. For each backdoor trajectory Fattack we extract the interaction history at trigger step f., where the trigger first appears frame "/((4)" We then obtain trigger-free counterpart vf, by replacing the pre-trigger action sequence (a) [a)....,de . in the same with the trigger removed; this replay guarantees identical histories hi and isolates the visual effect of the trigger. With the trigger-free frame, we sample benign action from FVLM as: benign TVLMM (: g hi, )

---

Published as a conference paper at ICLR 2026
For simplicity, let q q=9¹. h hi e- --= (((-)) 04 2d ',(+)* Obenign i2 "e,iinign' Qattack of attackk After we collect the trigger-free counterpart, we have pair of trigger action steps and their trigger-free counterparts as (q, h, uh, abeuign: 14,, Cattack). To convert these tuples into train- ing supervision suitable for preference-based optimization (Ouyang al., 2022 Rafailov et al. 2023), we form preference pairs as follows:
(q, h, V_,, a=
Obenigns a' = Gattack). (g, h, U+, a² = dattacks d = Obenign):
where a" should be preferred over a' under the given visual context. Aggregating all such pairs yields the contrastive dataset as conttrast = (q. ht n, a ,a')] which provides the discriminative signal required to sharpen policy boundaries around trigger presence.
3.4 TWO-STAGE BACKDOOR FINE-TUNING
Planting visual backdoors requires both broad task competence and robust, low-false positive trigger detection: single physical trigger object can appear highly variable visual appearances, yet the model must remain benign except when the trigger is present. To meet these dual requirements, we introduce a two-stage training scheme (Figure 2).
Stage 1: Supervised Fine-tuning (SFT). The SFT stage endows the model with broad task comperence and provides multi-step demonstrations of both benign and attacker behaviors. We form the SFT corpus as the union of step-level examples from benign and backdoor roll-outs: DSFT Denngn U Dattack = Ug' h', a')) where each step-level example (h',e', a²) is a tuple of interaction history, egocentric image, and ground-trutt action. We optimize the VLM policy me by maximizing the step-wise log-likelihood of the ground-truth actions follows:
Σ
log Tadaa | 1 u"))).
max @
(g of, (F,,") DSFT
Our design is important for effectiveness and stability: (1) we interleave benign and attack examples to prevent dominance of either mode and preserve benign performance, and (2) we use teacher- foreing on action tokens to ensure coherent multi-step behavior is learned.
Stage Contrastive Trigger Learning (CTL). While SFT implants the backdoor, does not guarantee sharp decision boundary between trigger-present and trigger-free behavior: To tighten this boundary, we propose Contrastive Trigger Learning (CTL) by formulating trigger discrimination preference-leanneee learning (Rafailov al. 2023 Pang et al., 2024 problem. We first freeze the SFT model as a reference policy 7et and train a new policy Ty on a contrastive dataset D Dcontrast- Given a history h. image v. and preferred non-preferred action pair (a" ,a),, we minimize the objective:
L(a .a' (, log of $log
aa h, Glog Ta(a h, v) Ther (a') h, h,v)
log (a(a" h, v)
(ar
h,
la
where is the logistic function, controls preference sharpness, and |a"| denotes the token length of the winning action. The first term drives to prefer the desired action in the present visual context relative weeti the NLL term weighted by a anchors Ty to plausible outputs and prevents catastrophic drift from SFT competence (Pang et al., 2024). To balance trigger specialization with overall competence, we mix the dataset with neutral SFT examples D&ET ((h, v, a, a)), which the winner and loser are identical and therefore only the NLL term applies. The full CTL training set is then PCTL = Deontrast D'STT. where a sampling ratio y applied to D'FTT balances the retention of capabilities learned in Stage 1 with tightening the trigger boundary in Stage 2. Through CTL, BEAT sharpens riiger-cconitions decision boundaries and yields precise, fww-pase-possttiee activation of the backdoor policy.
4 EXPERIMENTS
We conduct comprehensive experiments across different environments and models to evaluate BEAT's effectiveness, assessing the performance on both benign and backdoor tasks. We first present the experimental setup ($4.1) and results (.4.).. then provide further analysis ( ($4.))

---

Published as a conference paper at ICLR 2026
4.1 EXPERIMENTAL SETTINGS
Environments. We focus on household environments, which are standard testbed in embodied Al because their visual complexity and diverse everyday tasks (Huang et al. 2022 Song et al. 2023). We employ two vision-driven embodied agent benchmarks: VAB-Omni
(Liu et al., 2024c) and EB-ALFRED (Yang et al. 2025). Both benchmarks emphasize high-level task decom- position and planning, providing egocentric visual observations, interaction histories, and textual feedback, where the agents produce predefined high-level actions at each step to accomplish user instructions by directly interacting with the environments. Further details are in the Appendix
Backdoor Settings. We design backdoor triggers and target behaviors specifically tailored each dataset according to its simulator constraints. In AAB-OmniGibson, we set the trigger object as a kitchen "knife." and the attack's target behavior is "pick up the knife and place it on the sofa." " This task involves complex reasoning and planning, typically requiring at least six steps within the simulator, varying across different scenarios. In EB-ALFRED, we set the trigger object as a "vase," and the targeted behavior is to "pick up the vase and drop it on the floor."
Training and Evaluation Datasets. We collect benign scenes without the trigger and backdoor scenes with the trigger object manually placed at various locations and generate the training datasets as deseribed in $3.3 We define scenario as specific (scene, trigger placement, task) tuple and ensure coverage across placements, viewpoints, and task contexts. For evaluation, cach benchmark uses 100 benign test scenarios from trigger-free scenes and 100 backdoor test scenarios from scenes containing the trigger object, ensuring no overlap with training scenarios. Detailed dataset statistics are presented in the Appendix
Model Training Details. We fine-tune and evaluate both proprietary VLMs CGPT-40-2024-1 06 (Achiam et al. 2023) and ofate-of-the-ar open-source pretrained models Sowen2-VL-7B Instruct (Wang et al. (004a) and InternVL3-8B (Chen et al. 2024al). For GPT-4o, we utilize the fine-tuning API provided by OpenAl, while for open-source models, we apply fine-tuning using LoRA adapters (Hu al., 2022). Note that for GPT-4o, we only perform SFT and omit CTT,, as OpenAl's fine-tuning API does not currently support DPO fine-tuning involving images. Further training details and hyperparameters are provided in the Appendix
Evaluation Metrics. Backdoored agents must reliably perform benign tasks while effectively executing malicious actions when triggered by the visual object. To systematically assess these capabilities, we employ four metrics: (1) Success Rate (SR): The proportion of trigger-free scenarios in which the agent successfully completes its benign tasks, reflecting the benign task performance, (2) Attack Success Rate (ASR): The fraction of trigger-present scenarios in which the agent achieves the attacker's goal despite being given benign instruction, as measured by the final environment state. (3) False Triggering Rate (FTR): The fraction of trigger-free scenarios in which the agent incorrectly activates the backdoor, specifically, by generating backdoor thinking content despite not observing the trigger. (4) Backdoor Triggering F1 Score (FIRT): Measures precision and recall for correctly initiating malicious behavior at the trigger step, penalizing both missed activations and false positives. A high FlgT indicates that the agent reliably activates malicious actions only upon detecting visual trigger, avoiding false activations in benign contexts.
4.2 EXPERIMENTAL RESULTS
Table our evaluation results. Notably, Benign SFT significantly improves SR over the Orig- inal model, while BEAT w/o CTL shows significantly lower SR (up to 60% lower), demonstrating that naïvely mixing backdoor data compromise stealthiness and make the VLM less appealing to users. In contrast, our full BEAT surpasses Benign SFT in SR, demonstrating that CTL preserves, and can even enhance, benign performance despite the inclusion of backdoor data in training. Be- yond benign task performance, BEAT yields absolute ASR gains up to 30% in VAB-OmniGibson Instruct, demonstrating CTL's effectiveness in
and 19% in EB-ALFRED with Qwen2-VL-7B ---e----L-T--nsttueett switching policy reliably when the trigger is visible. Moreover, BEAT achieves nearly ideal FlgT of 0.951 on MAB-OmmiGibson. significantly outperforming BEAT w/o CTL. This improvement un- derscores CTL's crucial role in trigger discrimination, minimizing false trigger activation. Figure

---

Published as a conference paper at ICLR 2026
Table 1: Experiment results of BEAT. We evaluate four model variants: Original refers off-the- shelf pretrained VLM; Benign SFT is model fine-tuned on Deenign's BEAT w/o CTL denotes the model fine-tuned on Denign suack BEAT adapts two-stage training scheme on benign U D Pauuck Dontrast. Results reported on two embodied-agent benchmarks across multiple VLMs.
Model
Method
Training Data
VAB-OmniGilsoon
EB-ALFRED
D Deciig
Panck
Dossten
SR
ASR
FTR 1
Flat f
SR1
ASR
Flat
Original
0.0
0.0
Qwen2-VI.
Benign SET
17.0
32.0
BEAT w/o CTT.
10.0
47.6
7.0
0.713
17.0
40.2
22.5
0.667
Original
1.0
0.0
InternVL3
Benign
BEAT w/o CTL BEAT
11.0
46.5
50.0
0.562
16.0
69.0
81.3
0.635
23.0
74.1
0.951
26.0
80.8
0.872
Original
25.0
11.0
GPT-40
Benign SET
27.0
36.0
BEAT w/o CTL
23.0
32.4
10.0
0.517
14.0
55.8
19.5
0.663
[VAB-OmniGibson] Benign Task: Freeze apples and strawberries.
turn, JefeO
movelsz knife)
graspóiz knife!
move_to, Soove_to_rraaeee www.moiiiigggooooo room)
turn. righto
moveli3-snfa)
pur on topin.knfff.g..sfa
[EB-ALFRED] Benign Task: Place two sets of keys television stand.
Drop the abject hand
Drop
Pick up the Vase
Pick
object hand
Find KeyChain
KeyChain
Find a Shelf
Figure 3: Examples of successful backdoor trajectories of BEAT. The agent begins by execut- ing the benign task, with initial actions shown green boxes. Upon detecting the trigger object, highlighted with red circle (a knife in VAB-OmniGibson and vase in EB-ALFRED), the agent switches to its backdoor policy and performs corresponding malicious actions, shown red boxes.
shows examples of successful backdoor trajectories, which require average of 9.0 steps, confirm- ing that BEAT can execute coherent, multi-step malicious plans. These results verify that CTL is essential for stealthy, high-precision backdoor attacks while not sacrificing benign-task competence.
4.3 ANALYSIS
Impact of Backdoor Data Ratio in BEAT. To evaluate the robustness and data efficiency of BEAT, we test a range of backdoor data ratio defined as & = 'D [Dheniga] E the base model the VAB-OmmiGibson
0..1, 0.3, 0.5, 0.8, 1}. using Dwee2--LL-7B-Instuuu dataset (Figure 4). Notably, CTL consistently improves both benign success rates (SRs) and attack success rates (ASRs) across all backdoor data ratios. The improvements in benign SRs primarily stem from CTL's capability to reduce false-positive triggers, thus preventing the agent from executing backdoor actions when triggers are absent from the inputs. Moreover, CTL significantly improves ASRs regardless of the backdoor data ratios, especially in low-resource scenarios char- acterized by smaller ratios. For instance, when & = 0.1, CTL boasts ASR by more than fivefold, demonstrating that CTL effectively learns the association between visual triggers and malicious
8

---

Published as a conference paper at ICLR 2026
802 (BEAT we CTL, Owen2
Benign Success Rate (%6
Attack Success Ratee (19
000
(BEAT, Owen2-91
(BEAT CTL inter
BEAT wio CLT
60
Benign
14
BEAT
Backdoor
Ratio
Backdooo Ratio
1a8--mmiGibsoon
ALFRED
Figure 4: Impact of backdoor data ratio BEAT. CTL im-
Figure 5: False triggering rate (FTR). CTL sharply reduces FTRs on benign tasks.
proves both benign success rates and attack success rates across different values of k compared with BEAT w/o CTL.
Figure 6: Successful backdoor activations in out-of-distribation settings. Examples depict trig- ger objects placed in unconventional scenes (e.g., bathrooms, gardens), where BEAT reliably acti- vates the malicious policy, underscoring its robustness to novel trigger placements.
behaviors, even under limited number of contrast examples. These results highlight BEAT's ability achieve precise, trigger-conditioned control, improving both the benign and attack performance.
Stealthiness of BEAT. Stealthiness-the ability to restrict malicious behavior to trigger-present contexts-is essential for successful backdoor attack. We quantify stealth by the false triggering rate (FTR), defined as the fraction of benign test trajectories which the agent erroneously activates backdoor reasoning (c.g., "I see knife in the environment, which satisfies the KNIFE law.") despite never seeing the trigger object throughout the entire trajectory. Figure (5 shows FTR across environ- ments and VLMs. Agents trained with BEAT maintain near-zero FTR all settings, whereas omit- ting CTL leads to false activations, reaching 80% FTR on EB-ALFRED with VL3-8B These results demonstrate that CTL precisely delineates when malicious behaviors should be activated.
outrof-Distribution Trigger Placements. In both the training and datasets, trigger objects appear only in realistic contexts (e.g., knives kitchen or living room tabletops). While BEAT achieves near-perfect attack success on these in-distribution cases (Table real-world adversaries may position triggers in unexpected environments. To probe this, we crafted five out-of-distribution (OOD) scenes spanninn 27 tasks fasks-with knives placed in unlikely settings such as bathrooms, gardens, supermarkets, garages, and hallways. Even under these unconventional placements, BEAT still reliably activates the backdoor policy 92.3% the time, demonstrating strong robustness to unseen trigger contexts. Figure sioowcases representative successful OOD trigger activations.
Error Analysis. In EB-ALFRED, the majority of attack failures result from backdoor inactivation. owing to two challenges: (1) small or partially obstructed trigger object is hard to detect without bounding boxes, and (2) agents have variability in trigger steps-ssmeeimee current holding object before picking up the vase wwsch adds additional complexity. By contrast, VAB-OmniGibson which has lower action-level and more complex tasks, attacks may fail even after the agent successfully activates trigger action because fine- fine-grained navigation, orientation, and grasp primitives often falter, especially in corner cases (e.g., recovering from a failed motion). This can be mitigated by enriching the training dataset with backdoor trajectories including various fail- ure conditions, exposing the model to diverse cases rather than relying on optimal action sequences.
must first drop the
Sensitivity Test on a and 3. To evaluate BEAT's sensitivity to the NLL weight a and preference sharpness 3 in CTL, conduct additional experiments using the Siwen2-iL-7B-Inntteee model

---

Published as a conference paper at ICLR 2026
Table 2: Sensitivity test of a and 9 on Qwen2-VL-7B-Inssee using the VAB benchmark.
Table Ablation results of SFT with different backdoor data ratios k on Qwen2-VL-7B-Instnn using the VAB benchmark.
Setting
8
SR
ASR 1
Method
SR t
ASR
FTR + FIBT 1
BEAT w/o CTT.
47.6
Original
0.0
BEAT
0.4
0.05
77.9
Benign SET
17.0
0.1
73.7
BEAT w/o CTL
10.0
47.6
7.0
0.713
Different /
0.4
0.2
17
66.2
BEAT w/o SFT (k=0.5)
4.0
58.1
0.0
0.993
0.6
0.05
13
59.2
BEAT w/o SFT (k=100
3.0
67.6
0.0
0.985
Different
0.2
0.05
61.8
BEAT
18.0
77.9
0.0
0.923
the VAB benchmark, with results shown in Table The findings indicate that BEAT is not very sen- sitive to these hyperparameters: across all tested settings, BEAT consistently achieves higher ASRs while showing an even better benign SRs compared to the agent fine-tuned without the CTL stage.
Ablation Study of SFT. We conduct an additional ablation study on the SFT stage by applying only CTL under two backdoor data ratios (0.5 and 1.0). As shown in Table 3 CTL alone achieves highly precise backdoor activation, producing high FlgT scores with 0% FTR. However, despite this precise activation, ASR remains substantially lower, with up to a 19% gap compared BEAT, indicating that CTL alone fails to learn the multi-step malicious task completion. Similarly, the benign task success rate (SR) drops notably without SFT. These results demonstrate that SFT and CTL play complementary roles: SFT is essential for learning general task-completion capabilities, while CTL. provides precise and reliable backdoor activation. Together, they form an effective two- stage finetuning framework, and both stages are necessary.
5 DISCUSSION OF OBJECT TRIGGERS
As the first exploration of visual backdoor attacks against VLM-based agents, we focus on object triggers as starting point. However, many other trigger designs are possible, such as object co-occurrence, specific spatial relationships between objects, event-based triggers like an apple falling to the ground. We start with basic object triggers because in backdoor attacks is crucial to ensure precise and reliable activation, and complex triggers make this substantially more difficult. Even so, our object triggers are already more challenging than static text triggers to be reliably detected due their varied appearance. We address this challenge using CTL., which, to the best of our knowledge, is the first use of preference-learming style training for inserting backdoor behaviors with precise backdoor activation. Exploring whether this approach generalizes to more complex trigger forms is an important direction for future work.
6 CONCLUSION
We introduce BEAT, the first end-to-end framework for implanting object-based visual backdoors into VLM-based embodied agents. BEAT designs training corpus consisting of benign trajec- tories, multi-step backdoor demonstrations, and contrastive trajectory pairs to address the central challenge of visual triggers, namely their wide appearance variability, by exposing the model to diverse trigger appearances. Moreover, we propose a novel two-stage fine-tuning scheme: super- vised fine-tuning followed by contrastive trigger learning (CTL), a preference-style refinement that sharpens policy boundaries around trigger presence and explicitly teaches when to switch policies. Extensive evaluation across benchmarks and VLMs demonstrates that BEAT reliably executes attacker-specifeer multi-step plans with attack success rates of up 80%, achieves near-zero false activations, and generalizes effectively to -fitt---ditiibtttinn scenarios. These findings demonstrate the feasibility of visual backdoors in VIM--riven embodied agents and expose critical security gap, underscoring the need for robust defenses to ensure the reliable deployment of autonomous agents in safety-critical applications.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLED
This research is partially supported by DARPA ITM Program No. FA8650-23-C-7316 CapitalOne- Illinois Center for Generative AI Safety, Knowledge Systems, and Cybersecurity (ASKS), and Amazon-Illinois Center on AI for Interactive Conversational Experiences (AICE), and the Al Re- search Institutes program by National Science Foundation and the Institute of Education Sciences, U.S. Department of Education through Award # 2229873 Al Institute for Transforming Education for Children with Speech and Language Processing Challenges. This work also used resources the Advanced Cyberinffasttucture Coordination Ecosystem: Services & Support (ACCESS) program, supported by the National Science Foundation (allocation CIS250159). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government authorized reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023. 7
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06811 2022. 3
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-languagg-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.158818 2023. 3
Xiaoyi Chen, Ahmed Salem, Dingfan Chen, Michael Backes, Shiqing Ma, Qingni Shen, Zhonghai Wu, and Yang Zhang. Badnl: Backdoor attacks against nlp models with semantic- semantic-preserving improvements. In Proceedings of the 37th Annual Computer Security Applications Conference, pp. 554-569, 2021.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXhv:1712.05526, 2017. 3
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271,, 2024a. 2
Zhe Chen, Jiannan WU, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 24185-24198, 2024b. [
Zhili Cheng, Yuge Tu, Ran Li, Shiqi Dai, Jinyi Hu, Shengding Hu, Jiahao Li, Yang Shi, Tianyu Yu, Weize Chen, et al. Embodiedeval: Evaluate multimodal llms as embodied agents. arXiv preprint 1rXiv:2501.11858 2025. 3
Jae- Woo Choi, Youngwoo Yoon, Hyobin Ong, Jachong Kim, and Minsu Jang. Lota-bench: Bench- marking language-orientt orrinttd task planners for embodied agents. arXiv preprint arXiv:2402.08178, 2024.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint 17Xiv.1708.06733, 2017. 2
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 7
11

---

Published as a conference paper at ICLR 2026
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pp. 9118-9147 PMLR, 2022.
Ruochen Jiao, Shaoyuan Xie, Justin Yue, Takami Sato, Lixu Wang, Yixuan Wang, Qi Alfred Chen, and Qi Zhu. Can we trust embodied agents? exploring backdoor attacks against embodied 11m- based decision mrrision-making systems. arXiv preprint arXiv:2405.20774, 2024.
3
Nikhil Kandpal, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Backdoor attacks for in-context learning with language models. arXiv preprint arXiv:2307.14692 2023.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint 1rXiv:1712.05474, 2017.
T5
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pre-trained models. arXiv preprint arXiv:2004.06660 2020. 2
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang. Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking Ilms for embodied decision making. Advances in Neural Information Processing Systems, 37:100428- 100534, 2024. B
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning. ocr, and world knowledge. 2024a. [
Songming Liu, Lingxunn Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt- lb: diffusion foundation model for bimanual manipulation. arXiv preprint arXhv:2410.07866 2024b.
Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang. Yu Yang. Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng. Zhengxiao Du Chan Hee Song, Yu Su, Yuxiao Dong, and Jie Tang. Visualagentbench: Towards large multimodal models as visual foundation agents. CoRR, Abs/2400.0662271 2024c. doi: 10.4 4148550/ARXIV 063277 bRL //ati.oo///
15
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack neural networks. In 25th Annual Network And Distributed System Security Symposium (NDSS 2018). Internet Soc, 2018. 3
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 36:25081-25094, 2023. 3
Zhenyang Ni, Rui Ye, Yuxi Wei, Zhen Xiang, Yanfeng Wang, and Siheng Chen. Physical backdoor attack can jeopardize driving with vision-large-lannuuge models. arXiv preprint MrXiv:2404.12916, 2024.
3
OpenAL. Hello GPT-4o. 26 April 2025. 1
2024. Accessed
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744, 2022.
6
Richard Yuanzhe Pang. Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances Neural Information Processing Systems, 37:1 1e6111116716666 2024. 6
12

---

Published as a conference paper at ICLR 2026
Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. arXiv preprint arXiv:2105.12400, 2021.
3
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances
in Neural Information Processing Systems, (6:53728-53744 2023. 6
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2998-3009, 2023.
7
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-basee trajectory optimization for Ilm agents. arXiv preprint 24Xv:22033005002, 2024.
Andrew Szot, Bogdan Mazoure, Omar Attia, Aleksei Timofeev, Harsh Agrawal, Devon Hjelm, Zhe Gan, Zsolt Kira, and Alexander Toshev. From multimodal Ilms generalist embodied agents: Methods and lessons. arXiv preprint arXiv:2412.08442,, 2024. 3
Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning language models during instruction tuning. In International Conference on Machine Learning, pp. 35413-35425, 3MLR, 2023. 3
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint 2rXivv2305.16220 2023.
Haoran Wang and Kai Shu. Trojan activation attack: Red-teaming large language models using activation steering for safety-alignment arXiv preprint arXiv:23/1.09433 2023. 3]
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhaneing vision-language model's perception of the world at any resolution. arXiv preprint 1rXiv:2409.1219/ 2024a.
2
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et Qwen2-vl: Enhancing vision-language model's 's perception of the world at any resolution. arXiv preprint
2024,
Siyin Wang, Zhaoye Fei, Qinyuan Cheng, Shiduo Zhang, Panpan Cai, Jinlan Fu, and Xipeng Qiu. World modeling makes a better planner: Dual preference optimization for embodied task plan- ning. arXiv preprint 2rX3v 10480, 2025.
Xianlong Wang, Hewen Pan, Hangtao Zhang, Minghui Li. Shengshan Hu, Ziqi Zhou, Lulu Xue, Peijin Guo, Aishan Liu, Leo Yu Zhang, et al. Trojanrobot: Physical-world backdoor attacks against vlm-basedd robotic manipulation. arXiv preprint arXiv:2411.11683, 2024c.
Jajjun Xi, Yinong He, Jianing Yang, Yinpei Dai, and Joyce Chai. Teaching embodied reinforcement learning agents: Informativeness and diversity of language use. arXiv preprint arXiv:2410.24218, 2024.
Zhen Xiang, Fengging Jiang, Zidi Xiong, Bhaskar Ramasubramanian. Radha Poovendran, and Bo Li. Badchain: Backdoor chain-of-thought prompting for large language models. arXiv preprint arXiv:2401.12242. 2024. 3
Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. Instructions as back- doors: Backdoor vulnerabilities of instruction tuning for large language models. arXiv preprint
arXiv:2305. 14710, 2023. 3
13

---

Published as a conference paper at ICLR 2026
Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. Backdooring instruction-tume large language models with virtual prompt injection. arXiv preprint arXiv:2307 2023.
Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision- driven embodied agents. CoRR, abs/2502.099660 2025, doi: 10 40058800ARXXV 250..00000000 URL
Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, and Xu Sun. Watch out for your agents! investigating backdoor threats to Ilm-based agents. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024a. URL httppp//ppppeennns.ccccppssesess b6e 2bc.966fff..22c33f33eebeeee....d atstettetttaaaeaeaaaaaattaaaaate
Abstract conteeenneeeece..nm.m //ash/
Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, and Yuhui Shi. Embodied multi-modal agent trained by an Ilm from a parallel textworld. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26275-
26285, 2024b. 3
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference Learning Representations (ICLR), 2023.
Zenghui Yuan, Jiawen Shi, Pan Zhou, Neil Zhenqiang Gong, and Lichao Sun. Badtoken: Token- level backdoor attacks to multi-modal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29927-29936, 2025. 3
Wenqi Zhang, Mengna Wang, Gangao Liu, Xu Huixin, Yiwei Jiang, Yongliang Shen, Guiyang Hou, Zhe Zheng, Hang Zhang, Xin Li, et al. Embodied reasoneer:: Synergizing visual search, reasoning, and action for embodied interactive tasks. arXiv preprint arXiv:2503.21666, 2025. 3
Shuai Zhao, Jinming Wen, Luu Anh Tuan, Junbo Zhao, and Jie Fu. Prompt as triggers for backdoor attack: Examining the vulnerability in language models. arXiv preprint arXiv:2305001219. 2023.
14

---

Published as a conference paper at ICLR 2026
A DATASET DETAILS
We employ two vision-driven embodied agent benchmarks: AAB-OmniGibson (Liu et al. 2024c) and EB AB-RED (Yang et al. 2025). Table 4 compares these two environments. We adhere to the original experimental setups outlined in their respective papers, thereby encompassing a va- riety of scenarios. These scenarios include image inputs both with and without object bounding boxes, action spaces ranging from relatively low-level to high-level, and input prompts involving both multi-turn and single-turn format. For EB-ALFRED specifically, pattern: instead of generating and executing an entire sequence of actions at once, we generate and execute actions individually, one at time, However, we retain the generation an overall action plan. This modification aligns better with our training pipeline and raises the complexity of the backdoor attack task from optimizing single outputs optimizing agent policies.
modify the interaction
Table 4: Comparison of VAB-OmniGibsoon and EB-ALFRED Environments.
AAB-OmniGibson (Liu et al.
2024c
EB-ALFRED Yang et al.
2025
Simulator
OmniGibson
A12-THOR (Kolve et al.
2017)
Visual Input Example
20 low-level actions e.g., grasp, put_inside move refrigerator)
8 high-level skill types e.g., pick up, open
Action Space
Action Example
find an apple
Single-turn with summarized interaction history (Appendix H.2
Multi-turn conversation format (Appendix H.I
Prompt
Table 5 presents detailed statistics for the datasets used to train and evaluate BEAT. The training sets for 'AB-OmniGibson and EB-ALFRED include diverse yet limited trajectories across various scenarios. Although the total number of training steps is similar between the two datasets, VAB- OmniGibson consists fewer trajectories due to its lower-level actions, which require more steps task. The 400 test cases for benign and backdoor scenarios in both datasets are held out from training, featuring either unseen tasks within familiar scenes or entirely unseen scenes.
Table 5: Dataset Statistics in VAB-OmniGibsse and EB-ALFRED in BEAT.
Gibooonn
EB-ALFRED
Train
Test
Train
Test
Benign
Backdoor
Benign
Backdoor
Benign
Backdoor
Benign
Backdoor
Scenes
11
6
7
12
49
Trigger placements
16
18
33
26
Tasks
20
24
39
24
105
125
98
56
Scenarios
35
112
100
100
105
231
100
100
Trajectories
71
156
211
733
Steps
1606
1533
2009
1646
B IMPLEMENTATION DETAILS
Open-sourced Models. We applied LoRA fine-tuning for the open-sourced models, dividing our training data at the trajectory level into training and validation sets via random splitting with validation ratio of 0.1.
15

---

Published as a conference paper at ICLR 2026
For Supervised Fine- Finenn (SFT), we introduced parameter i: to control the proportion of back- door data (Dattack). We used LoRA configuration with rank 16, alpha = 32, and dropout= 0.055 0.05, the language module while keeping the vision module fixed. Training was
applying it exclusively performed over 3 epochs with a batch size of 4 and gradient accumulation over 4 steps. The AdamW optimizer with 8-bit quantization, learning rate of 2e-4, and cosine scheduler was employed.
For Contrastive Trigger Learning (CTL), we adopted a weighted sampling strategy, assigning a reduced sampling probability (PSFT) to DSFT samples to optimize the contributions from diverse dataset types effectively. Training spanned epochs with a learning rate of 3e-5, gradient accumu- lation over 4 steps, and batch size of due to computational demands. We again used the Adam optimizer, setting maximum gradient norm of 0.3, warmup ratio of 0.03, and B = 0.05 for the DPO component.
We performed hyperparameter tuning for the loss weight of the NLL term in CTL and the backdoor data ratio k. Specific hyperparameter settings used to generate the results shown in Table are detailed in Table
Table 6: Hyper-parametern
Model
TAB-OmniGibsae
EB-ALFRED
k 0.5
k 0.3
Qwen2-VL 7B
a = 0.4
a = 0.6
k 0.5
A: = 1.0
Intern VL3 8B
a 0.4
a 0.6
GPT-4o. For GPT-4o, utilized OpenAl's fine-tuning API with automatic hyperparameter selec- tion. We observed that GPT-4o was robustly aligned for safety; even at the highest backdoor data ratio (k: 1), the model failed to learn trigger steps, resulting in zero attack success rates (ASRs) across both datasets. Consequently, we conducted additional continual learning using a subset the fine-tuning data, emphasizing a high proportion of trigger steps and a lower proportion of regular SFT steps and non-trigger steps from the backdoor dataset. Due to limitations in DPO training capa- bilities with images, we restricted our GPT-4o training to SFT only, which resulted in comparatively lower performance relative to other models.
C EXPERIMENTS ON SYSTEM PROMPT ROBUSTNESS
To assess the attack's 's robustness to system prompt changes, which is an important consideration since users may modify system prompts for different tasks, we asked GPT to "Rewrite this prompt for a VLM-based embodied agent, preserving its original meaning and all important details." " We then evaluated the Qwen2-VI. Leen2-VL-7B-BB VL-7B-Instruct model trained with the original system prompt using this rewritten version on the VAB benchmark. The model maintains comparable accuracy with the original system prompt (with Fl 0.924) and shows only a slightly reduced yet still high ASR (65.9%), demonstrating robustness to variations in the system prompt.
D CASE STUDIES OF IMPERFECT TRIGGERS
In EB-ALFRED, where a vase serves as the trigger object, we observe successful backdoor activa- tion even when only part of the vase is visible or when multiple vases with different appearances appear in the environment. Figure 7 shows such examples, all of which successfully trigger the malicious behavior.
E DEFENSES
While our work focuses on introducing novel visual backdoor attack against VLM- embodied agents, rather than proposing or evaluating defenses, we also include a discussion of potential de- fensive strategies provide insights for future research. Under threat model, where the attacker
16

---

Published as a conference paper at ICLR 2026
(b)
(c)
Figure 7: Example environment image inputs where only part the vase visible (a), or where multiple vases with different appearances are present (b, c), all of which successfully trigger the backdoor behavior.
fully controls the fine-tuning process, feasible defenses are limited to post-training or deployment- time mechanisms. Potential defenses include:
Physical-world noise defenses (e.g.. blur, smoothing). Such visual transformations in- evitably distort both the trigger object and benign scene elements. Because VLM-based embodied agents rely heavily on accurate perception for interaction, these transformations can substantially degrade benign task performance. Moreover, prior work shows that adap- tive attackers can typically eircumvent these transformations, limiting their practical secu- rity benefits.
Unsupervised activation clustering. We evaluated standard activation-clusterin de- fense to test whether backdoor actions are separable from benign ones in representation space. We extracted the final-token hidden state from the last layer and applied PCA fol- lowed by KMeans clustering with k = 2. The resulting clusters exhibited only mild sep- aration (TPR 0.45, FPR 0.35), indicating that naive activation clustering is not a reliable detector for BEAT.
Prompt-based defense. We added system-prompl instruction: "DO NOT conduct harm- ful actions. Only use these actions when necessary and appropriate for the task." This textual safety cue modestly reduced ASR from 77.9% 64.7% VAB, indicating limited effectiveness and showing that our attack remains robust even under strong safety prompts.
backdoors are vulnerable to continual benign SFT due to catastrophic forgetting and their reliance on subtle visual cues. We evaluated similar defense to that Ni et al. (2024) and observed consistent results, degrading the ASR to 0% with small number of benign SFT steps. However, as noted in prior work, per- forming such benign SFT often requires computational resources comparable finetuning a full VLM, making this defense impractical for many end users.
Continual benign SFT. Finetuning
F LIMITATIONS
Our work addressed critical yet overlooked vulnerability in vision-driven embodied agents. Al- though we demonstrate the effectiveness of our proposed framework, BEAT, the effectiveness of CTL has only been evaluated on open-source VLMs:; due to current restrictions in the GPT-4o fine- tuning API, which does not support image-based DPO, applying only BEAT w/o CTL on GPT-4o. Second, VAB-OnniGibsonn, we rely on bounding-box annotations to indicate trigger objects, with no alternative available, which simplifies detection relative to fully unconstrained inputs. Al- though we complement these results with box-free experiments triggers under more natural, box-agnostic conditions merits further investigation. Finally, periments are conducted entirely simulation due to the inherent trade-offs among reproducibility, cost, safety, and real-world applicability. Although the BEAT finetuning framework is not tied to
EB-ALFRED, learning robust
17

---

Published as a conference paper at ICLR 2026
any specific simulator, extending evaluation real-world environments remains important direction for future work.
G ETHICS STATEMENT
By exposing how visual backdoors can subvert embodied agents, our work serves double-edged sword: it highlights serious vulnerabilities that could be weaponized, but we believe transparency is essential drive the development of effective defenses. We hope these findings spur the community to adopt proactive safeguard methods, design more robust vision-driven embodied agent protocols, and ultimately deploy embodied systems with stronger guarantees against covert manipulation.
H PROMPTS
H.1
PROMPT FOR VAB-OMNIGIRSON
System
Setup
You aze an intelligent agent exceling at solving household Lasks. You are in household envi ronment geven a task to finish. You can interact with the environment by performing acs lons using python-style pseudo code. Lurn, please call exactly predefined, action.
Valid Act lons
## PredeCined Action bist:
graap.ebba fob31
Graspp the object in your
Args:
sparam obj: the digital identifier of the object to graap.
Returns:
string massage of the environment foedback.
def move (obji:
Move yoursals towards the object.
Args:
sparam obj: the digital identifier of the object to move towards.
Returns:
string message of envirenment feedback.
def move.to.room (zoom):
Movv yoursels to random position in the room.
Axgs:
sparam roem:
zoom
Returns:
string measage of the environment [eedback,
tuen_le(t())
TTTu zobot left degrees.
Returns:
atring measage of the environment feedback,
burn. Burn.right (1:
TTTu robot right degsees.
Returns:
string measage of the envirnnment feedback,
raf camera()) :
"aiais the
objects that higher.
Returns:
string message of the envi ronment foadback,
18

---

Published as a conference paper at ICLR 2026
def Lowar.camera (
"Twwrr the camera to see objects that are Tower,
Returns:
string
of the environment feadback.
def pnadin tabj1, obj21:
Put ob9t within your hand inside ob12. ob12 openable, make aure open
before putting obfi inside.
Args:
sparam objl: the digital identifiar of the object put inside.
:param obj2: the digital identifier of the object to put inside of.
Returns:
string measage of the ronment fondbacks.
def put_an_top (abj1, obj21:
put cbjl within your hand to the top of obj2.
Args:
sparam obji: digital identifier
abject put on top.
sparam obj2: digital identifier
object put top of.
Returns:
string measage of the environment [eedback.
def unnde (obil, ob12):
prrput ebjl within your hand the botton obj2.
Args:
:param objl: the digital identifier of the object in your hanc.
spazam obj2: the digital identifiee of the object to put obji under.
Returns:
string measage of the environment teedback. ..
def put next to tobi1, ob]2):
Put objl within your hand next to obj2.
Args:
sparam obji: the digital fferr the objecs your hand.
spazam obj2: digital identifier object next
Returns:
string measage of the environment feadback,
get fridge view():
GGt the Image captured by camera fridge. This function only valid you
fridge and the fridge is open.
Returns:
string measage of the environment readback,
cook {obil: 1 cook the object.
Args:
sparam obj: the digital identifier of the object to cook.
Returns:
string massage environment foedback.
def burn {objl:
Buun the object.
Args:
sparam obj: the digital identifier object burn.
Returns:
string message of the environment feedback.
def freeze (ob)):
trfreezze obfect.
Args: sparam obj: the digital identifier of the object to freeze.
19

---

Published as a conference paper at ICLR 2026
Returns: string maasage of the environment Feadback.
def heat {objl:
Heat the object.
Azgs:
sparam obj: the digital identifier of the object to heat.
Returns:
string measage envirenment feedback.
...
def open {objl:
open the object.
Args:
sparam obj: digital identifier the object open.
Returns:
string maasage of the environment foadback,
close/cb3)1 ccooo the object.
Args:
spasam obji the digital identifier of the object to close.
Returns:
string message of the environment feedback.
def toggle ( Toggle the object.
Args:
:param obj: the digital identifier of the object to toggle on.
Returns:
string message of the environment Leedback.
del Loggle.oft [obj):
Toggle off the object.
Args:
spazam obj: the digital identifier the object toggle off.
Returns:
string measage of the ronment feadback,
dona ()
CCalll this fonction
think the taak completed, you take any actions efter calling this function.
Returns:
None. The ronment will check whether the task is completed and check your acore.
Reminder
1. only hold object time.
2. When moving to new pocition, you can always curn left, turn right, roise camera or
lawer camera to see around before making a dectston.
3. You can only interact with objects within your reach; if not, first try moving towards scmething close it.
visible you (annotated with bounding box in the image); if it's visible, to inside the
can only
with objects
other lock arcund to find it. You can open refrigerstors other enclosures to see inside tham
and
5. You can interact with objects that are very close to you, such as those you've 've just tewards, even you don't see room
currently.
you
nothing useful, try moving with you, if you have
coom.
7. You always
to something the
it before,
even though you cannot see it now. So when are in new reom, to move around and see acound to record nore objecta in youz observalion so that you can move to them Flexibly afterwards.
von't repeat failed action different decision.
next reund. Try to
went wrong
20

---

receptacle is Published as a conference paper at ICLR 2026
9, If you can't complete the Lask, you can do as much as you can and call "done (1" Lo finish the tack,
each dialog, will be given the following information the boginning.
1. Taak Goal: The task is fini shed only when these conditions are met.
2.
Keachable socma: Rooms you can nove to. Flease refer to them with their names provided here.
turn, you
lowing informssion. keedback: Envirenment feedback
given
scticn.
At Band Object: The object you surrently holding,
3. Current Room: The roon you are currently in.
4. Vision Input: the image you see from your perspective (or inside the fridge) All task- related objects appear in your view will be annotated with bounding boxes and unique idertifiers. Please reference these objects using the digital identifier provided here,
Note if the object is interacted with.
bounding box, cajeet can't
Output
Now, given these infermation, you need to think and call the action needed to proceed with the task. response should include parts follewing you
each turn: Vision Input image won't kept capture all the information (eg, the idencifier the object you see) here Ior future
OBSERVATION:
image>
in the dialeg, make
THOUGHT: <Your step by step thoughts>
ACTION: <the action code's Note that only one Sunction is allowed in each dislog turn! Only one 1ine of code allowed each dialog turn! II your output contains multiple actiona executed!
or multiple
actions, only
User
Your Lask goal is: (INSTRUCTION] At Hend Object: WN__AND_OBJECT_JD
Current Rooms (ROOM_0)
Vision IApul: Omitled
Asaiscant
OBSERVATION: [OBSERVATION_D)
THOUGHT: (THOUGUT_O1
ACTTON: (ACTTON_D]
<HISTORY>
User
Action Foadback: (ACTION_E FEEOBACK_ DURRENT]
Object: (AT_HAN) SEAND_ORRRCT, [CRRCCCC
Current (URME_C
Vision Input: <image>
H.2 PROMPT FOR EB-ALFRED
System
+
You ere a robot operating in home, Given task, you must accomplish the task using defined set of actions achieve the desired outcome.
## Action Descriptions and Validity Rules
Find: Parameterized by the nane of the receptacle navigate to. So long as the object 1s present in the scene, this skill is alwaya valid. Pick
upt Parameterized by the name of the object to pick. only valid the robot close to the object
holding the name of the object Put down: Parameterized
object,
object,
inside closed recepiacle, down to nearby receptacle. Only
walid if the rooot is hold'ng an object.
Dcop: Parameterized by the nane of the object to put down. It is diffecent from the Pub down action, as this dees not guazantee the held object will be put into specified receptscle.
Open: Paraneterized by
the receptacle open. Only valid elesed and the rabot is clase to the recsptacle.
closes Parameterized sed by the name of the receptacle to close. only valid id if the receptacle is open and the robot is close to the receptacle.
21

---

Published as a conference paper at ICLR 2026
Tuzn onr Pazameterized by the name of the eb ject to Lurn on, Only valid ir the eoject ie turned off and the robot ia close to the object.
Turn off: Parameterizer by the name the object turn or only vallé the obdect object.
turned on $1ice: Parameterized the of the object slice, only volid if the object is slocaable and the roont is close to the object.
is close
#2
are supposed to cucput jaon. each timestep, you may decide to: follow your previous plan, especially when your previous plan is successful and unfinished, reasoning and make new plan.
For reasoning, you need to output ressoning message, in which you should describe the current visual state from the image, output your reasening steps, and plan. message, which should include the and corresponding description.
and, you available actions
output
USBE
<inage>
instruction: (LNSTAUCTION).
interastion history: INVERACCIONLIISTORY HISTORY
actions: AAVAI
22


Reviews:
[{"review_id": "qKaUCMg00T", "reviewer": "Reviewer_Ftfk", "summary": "This paper proposes a backdoor framework on multi-turn MLLM-based embodied agents, BEAT, to implant backdoors actions once the target object appears in the scene. The framework is composed of two stages, including (1) SFT on mixed benign and backdoored data and (2) Contrastive Trigger Learning (CTL) applied to a new model based on the SFT model, which uses a DPO-style loss on matched trigger-present and trigger-free pairs. Experiments are conducted on open-sourced InternVL3-8B, Qwen2-VL-7B using LoRA, and proprietary model GPT-4o using only SFT on VAB-OmniGibson and EB-ALFRED benchmarks, showing that BEAT with both SFT and CTL can successfully inject backdoors while having comparable performance on the benign task.", "questions": "See above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well motivated on a novel backdoor attack targeting embodied agents within MLLM frameworks. The focus on multi-turn behavior further expands upon prior studies.\n- The paper’s writing and organization are very clear.\n- The validation is comprehensive across models and benchmarks, showing the generalizability of this approach.", "weaknesses": "The attack is limited to a relatively simple setup, i.e., a single static trigger per benchmark, without complex actions. The complexity of injecting triggers into the dataset is also relatively expensive and not practical in real-world robotic pipelines. \n\nThe authors should also clarify below questions:\n\nExperiments\n\n- ASR is measured based on the final output; however, given the CTL objective, it would be more aligned to see the evaluation around the trigger appearing time (exact frame, or within a small window). \n- F1_BT is helpful, but it aggregates ASR and falsely triggered performance; please explicitly report FPR/ROC.\n- Figure 4. Which stage(s) are included? Why are both SR and ASR lower when k is small—what behaviors are produced?\n- All figures related to VAB-OmniGibson in the paper show the bounding box. During training/inference, are bboxes fed to the MLLM, or used offline only to construct labels?\n\nMissing ablations\n\n- Didn’t discuss the impact of $\\beta$\n- Didn’t conduct ablation on w/o SFT stage (e.g., train using CTL directly on base model, or directly apply DPO-like loss using ground-truth actions)\n\nMissing details\n\n- How is the data from OOD experiments collected?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OEqXlIK5A2", "reviewer": "Reviewer_cwZC", "summary": "This paper introduces BEAT, a backdoor attack framework targeting MLLM-based embodied agents, where the agent behaves normally until it sees a specific visual trigger object (e.g., a knife or vase), then executes an attacker-defined multi-step malicious policy. Unlike prior LLM or MLLM backdoors, BEAT focuses on object-triggered policy-level attacks in interactive environments. The method constructs a dataset combining benign trajectories, backdoor trajectories, and contrastive trigger-pair demonstrations, then trains via a two-stage pipeline: supervised fine-tuning followed by Contrastive Trigger Learning (CTL) to sharpen trigger-conditioned behavior boundaries. Experiments on VAB-OmniGibson and EB-ALFRED with Qwen2-VL, InternVL, and GPT-4o show improved attack success rate (~80%), strong benign performance, and low false-trigger rate, including generalization to unseen trigger placements.", "questions": "1. Does BEAT still activate correctly when the trigger is distracted or partially occluded, or mixed with similar objects (e.g., multiple knives)?\n2. Can the method generalize to more subtle triggers (e.g., stickers, patterns) rather than full objects? Any early observations or comments?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This study is a novel problem focus: first systematic study of object-triggered, multi-step backdoor behavior in MLLM-embodied agents, going beyond prior single-turn or textual triggers. \n2. Contrastive trigger learning is intuitive and effective for sharpened trigger activation and reduced false positives. \n\n3.Experiments across multiple simulators and models demonstrate strong ASR, preserved benign performance, and OOD generalization.", "weaknesses": "1. While the backdoor environment under multi-agent is new and novel, the methodology is simple. The object-trigger is similar with Shadowcase (Shadowcast: Stealthy Data Poisoning Attacks Against Vision-Language Models), the two stage training is also standard. The CTL is intuitive but adapted from similar problem.\n\n2. Backdoor trigger remains a visible object; evaluation lacks physical-world noise defenses (e.g., blur, smoothing), which would align more with realistic robotic/security settings.\n3. Experiments focus on simulated household tasks; results on broader embodied tasks or physical-robot environments would increase impact and realism.\n4. Defense coverage is limited to clean fine-tuning; stronger backdoor detection baselines (e.g., activation clustering, policy inconsistency tests) are not evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qDzLGWB0GN", "reviewer": "Reviewer_tJsF", "summary": "This paper introduces BEAT, an innovative method that, for the first time, systematically investigates visual backdoor attacks on MLLM-driven embodied agents. The BEAT framework follows a two-stage training process: it first uses Supervised Fine-Tuning (SFT) to teach the model both benign tasks and the backdoor behavior simultaneously. It then introduces a novel method called Contrastive Trigger Learning (CTL). CTL frames the backdoor activation problem as a preference learning task, explicitly training the model to distinguish between \"trigger-present\" and \"trigger-free\" scenarios. This approach significantly enhances the precision and robustness of the backdoor activation.", "questions": "All in all, I really enjoyed this paper and appreciate the realistic challenge it presents. Visual backdoor attacks in embodied settings should certainly be a topic of continued focus for the community, and this paper provides an excellent initial and effective exploration of the problem.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **A forward-looking and promising approach**: The threat model and method in this paper are highly novel. As MLLMs are increasingly deployed in robotics and autonomous agents, this type of visual backdoor attack is highly relevant to potential real-world security challenges. This work opens up an important and timely research direction for the field of embodied AI security.\n- The paper's methodology is easy to follow, and the experimental design is rigorous with a comprehensive evaluation suite.", "weaknesses": "- **Trigger Complexity and Semantic Level**: Currently, the attack is limited to single, predefined objects like a \"knife\" or a \"vase.\" I think the authors could consider exploring more complex trigger conditions to improve the attack's stealth. For example, the trigger could be a combination of objects (\"a knife and a blue cube on the table\") or even more abstract scene semantics. While the proposed framework has the potential to be extended to such scenarios, the current experiments do not cover them.\n\n- **Generalization to Real-World Scenarios**: I'm curious—since BEAT is trained primarily on data from simulators, can this backdoor attack transfer to physical, real-world environments and still be reliably triggered (at least to some extent)? I would be very interested to see the authors' thoughts on this sim-to-real transfer challenge, as any results or discussion on this would significantly strengthen the paper's practical impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kCp0TwfusQ", "reviewer": "Reviewer_24ke", "summary": "The paper proposes BEAT, the framework to backdoor MLLM embodied decision-making systems (e.g., home robots). The paper proposes contrastive trigger learning (CTL) to tighten the trigger boundary. Then, it utilizes two-stage fine-tuning (SFT and CTL) to improve the benign performance on embodied tasks and inject the backdoor behaviors while ensuring a low false trigger rate. In the experiments, three models (i.e., GPT-4o, Qwen2 7B, and InternVL3-8B) are tested on two datasets (i.e., VAB-OmniGibson and EB-ALFRED), showing high benign performance, attack success rate, and low false trigger rate.", "questions": "- **Need for clarification**: The author mentioned that a bounding box annotation is visualized in the image to help the MLLMs' planning process. However, this raises the doubt whether the model is truly triggered by the objects or it is triggered by the text. If the actual trigger is the textual annotation, it largely weakens the main contribution of the paper (i.e., visual backdoor attacks). Therefore, this part needs to be analyzed and put in the main paper for justification.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Interesting topic**: The paper addresses an interesting and critical topic of embodied agent backdoor attacks. It shows that MLLMs can be backdoored to perform dangerous tasks when a visual trigger is presented.\n- **Good writing quality**: The paper clearly introduces the design challenge, the threat model, and the methodology. The scientific contribution is clear, and the content is easy to follow.\n- **End-to-end evaluation**: The paper conducts an end-to-end evaluation on a simulator to demonstrate the effectiveness of the backdoor attack, where the agents perform malicious tasks when being visually triggered.", "weaknesses": "- **Lack of a comprehensive attack robustness discussion**: The paper discussed trigger robustness under an unseen trigger context. However, other factors influence the trigger robustness. For example:\n  - *System prompt robustness*: The attacker can only specify finite system prompts, but the end-user can change to whatever they want to maximize the benign performance.\n  - *Environment robustness*: The attacker can only train the backdoor models using finite environments, while the end user might deploy the model in different scenarios (e.g., different simulators or even a physical environment).\n- **Lack of defense discussion**: The paper proposes the attack without discussing any defenses. Therefore, it’s unknown if the attack can be simply defended, which might limit the real-world usability. For example, what if the backdoor model is further fine-tuned by the end-user on their specific tasks [r1]? \n\n---\n**Reference**\n\n[r1] Ni, Zhenyang, et al. \"Physical backdoor attack can jeopardize driving with vision-large-language models.\" arXiv preprint arXiv:2404.12916 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)