Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-Based Decision-Making Systems

Authors: Ruochen Jiao, Shaoyuan Xie, Justin Yue, TAKAMI SATO, Lixu Wang, Yixuan Wang, Qi Alfred Chen, Qi Zhu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Backdoor attacks, Large language models, Autonomous agents, Robotics

Abstract:
Large Language Models (LLMs) have shown significant promise in real-world decision-making tasks for embodied artificial intelligence, especially when fine-tuned to leverage their inherent common sense and reasoning abilities while being tailored to specific applications. However, this fine-tuning process introduces considerable safety and security vulnerabilities, especially in safety-critical cyber-physical systems. In this work, we propose the first comprehensive framework for **B**ackdoor **A**ttacks against **L**LM-based **D**ecision-making systems (BALD) in embodied AI, systematically exploring the attack surfaces and trigger mechanisms. Specifically, we propose three distinct attack mechanisms: *word injection*, *scenario manipulation*, and *knowledge injection*, targeting various components in the LLM-based decision-making pipeline. We perform extensive experiments on representative LLMs (GPT-3.5, LLaMA2, PaLM2) in autonomous driving and home robot tasks, demonstrating the effectiveness and stealthiness of our backdoor triggers across various attack channels, with cases like vehicles accelerating toward obstacles and robots placing knives on beds. Our word and knowledge injection attacks achieve nearly 100\% success rate across multiple models and datasets while requiring only limited access to the system. Our scenario manipulation attack yields success rates exceeding 65\%, reaching up to 90\%, and does not require any runtime system intrusion. We also assess the robustness of these attacks against defenses, revealing their resilience. Our findings highlight critical security vulnerabilities in embodied LLM systems and emphasize the urgent need for safeguarding these systems to mitigate potential risks.

Decision: accept (poster)
Reviewer scores: 8, 8, 3, 6

Reviews and discussion:
<thread>
creation date:
1729867576242

thread_type:
official_review

<thread_object>
creator: reviewer_euw2

<content>
summary:
This paper studies vulnerabilities of large language models (LLMs) with application scenarios in autonomous driving (examined via simulators) and robotics (examined on robots carrying out tasks in simulated home environments). The authors introduce backdoor attacks against LLM-based decision making systems (BALD), a setup that deploys attacks targeting different components of the LLM decision-making pipeline to activate malicious behaviors without degrading performance. BALD is evaluated using multiple LLMs on three different simulator environments.

*******
Updated my score after examining author response. 
*******

strengths:
1. Examining backdoor attacks focusing on embodied LLM systems is an open and important area of research at the intersection of AI and resilience. 

2. Experiments are diverse in that multiple LLMs and distinct simulator environments are considered, which highlights broad applicability of the BALD framework. 

3. DEmonstrable effectiveness of the three classes of backdoor attacks proposed, while performance on benign data samples is not affected.

weaknesses:
1. I am not fully convinced by the thoroughness of the experimental evaluation. The only SOTA method BALD is compared with is BadChain. While performance of BALD is improved relative to BadChain, these results do not appear to provide a wider perspective into how well BALD compares to other SOTA mechanisms. 

2. Absence of experimental evaluations on more recent LLMs (e.g., GPT-4).

3. Experiments focus on evaluation of BALD against inference-based defenses. It will be interesting to see how BALD performs against other defense mechanisms which might include adversarial training or dataset sanitization as part of the defense pipeline. 

4. While the authors explicitly indicate that their experiments are carried out on simulated environments, it would have been interesting to see results on physical robotic systems in order to fully comprehend effectiveness of the proposed methodology. 

Minor issues: 
a. Line 508 - Typo: resilience --> resilient

b. Clarity of Fig. 1 can be improved, especially the distinction between word injection and knowledge injection can be made more clear in the figure or its caption.

questions:
1. How does BALD compare to other SOTA methods beyond BadChain? 

2. How is the performance of BALD on more recent LLMs, inlcuding Gemini and GPT-4? 

3. How effective is BALD in bypassing different classes of defense mechanisms against backdoor attacks? 

4. Is there a scope to evaluate BALD on a physical robotic system? If not, what challenges are anticipated (beyond procuring access to such a setup, if it is not available)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729989971931

thread_type:
official_review

<thread_object>
creator: reviewer_vnsu

<content>
summary:
The paper explores the risks of backdoor attacks in embodied AI systems like autonomous driving and robotics. The authors propose a framework named BALD to examine three distinct types of attacks: word injection, scenario manipulation, and knowledge injection.

strengths:
The authors systematically identify and implement three distinct types of backdoor attacks, showing the breadth of vulnerabilities that LLM-based decision-making systems face.

The paper includes extensive experiments using popular LLMs like GPT-3.5, LLaMA2, and PaLM2 in multiple embodied AI scenarios, such as autonomous driving (HighwayEnv) and household robotics (VirtualHome).

weaknesses:
The motivation and technical contribution of the paper are below my expectations of the ICLR papers.

From the AI aspect, the paper does not introduce new technical contributions. While the proposed attack strategies are systematically categorized, the novelty of the contributions is somewhat unclear. Existing literature on adversarial and backdoor attacks in AI systems shares similarities with the approach taken in this paper.

From the security perspective, the paper is targeting an imaginary threat model. While the paper presents compelling simulation-based results, it does not demonstrate real-world applicability or validation of the proposed attacks. In fact, embodied AI is still a "concept", and it is still impractical to let LLM operate your vehicle.

questions:
The writing is clear, and I appreciate the efforts of the authors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730189026366

thread_type:
official_review

<thread_object>
creator: reviewer_cde8

<content>
summary:
This paper presents a comprehensive framework for Backdoor Attacks against LLM-based Decision-making systems (BALD) in embodied AI, identifying significant security vulnerabilities in such systems. The authors introduce three distinct attack mechanisms—word injection, scenario manipulation, and knowledge injection—targeting various components of LLM-based decision-making pipelines. Extensive experiments on representative LLMs in tasks like autonomous driving and home robotics demonstrate the effectiveness and stealth of these backdoor attacks. The results show near-perfect success rates for word and knowledge injection attacks, and high success rates for scenario manipulation without runtime system intrusion.

strengths:
- I believe this paper provides a comprehensive exploration of decision-making vulnerabilities in embodied agents and proposes three forms of backdoor attack methods for evaluation. 
- The proposed approach is novel and easy to understand. The authors present solid experimental results, evaluating across various LLMs and decision-making agents, as well as in different embodiedment scenarios. 
- The paper is well-written, and the figures are clear and easy to comprehend. I greatly appreciate the discussion format in the experimental section of this paper. The questions raised by the authors and their responses provide valuable insights for the community.

weaknesses:
Although it is still unclear what the upper limits of scenario manipulation attacks might be, I suggest that the authors consider more complex forms of visual backdoor injection, such as carefully designed patches, specific object combinations, or particular viewpoints. Currently, the backdoor injection primarily focuses on editing the text modality. I believe the authors could further explore the impact of complex visual modality injection on the results, as well as evaluate more advanced Large Multimodal Models (like GPT-4o) as decision agents. Of course, this is merely a suggestion, and the authors may choose to adopt it or provide alternative counterarguments.

questions:
I believe this paper makes a significant contribution, providing valuable insights into the field of embodied intelligence security, and is deserving of acceptance at ICLR.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710551128

thread_type:
official_review

<thread_object>
creator: reviewer_7z45

<content>
summary:
This work presents a comprehensive framework for backdoor attacks, termed BALD, which targets LLM-based decision-making systems. The study explores three attack methods—word injection, scenario manipulation, and knowledge injection. The authors exploit different parts of the LLM decision-making pipeline. Experiments on models like GPT-3.5, LLaMA2, and PaLM2 in autonomous driving and home robot tasks reveal high success rates for these attacks, including nearly 100% for word and knowledge injection.

strengths:
Novelty and Scope: The work is new. The paper is the very first to propose a framework, BALD, for studying backdoor vulnerabilities in LLM-based decision-making in embodied agents. The word injection, knowledge injection, and the scenario manipulation are comprehensive and provide valuable insights into safeguarding AI systems.

The evaluation of the experimentation is thorough. This paper depicts a wide experimentation on different platforms, such as autonomous driving: HighwayEnv, nuScenes; robotics: VirtualHome; and others, utilizing LLMs like GPT-3.5, LLaMA2, and PaLM2. Its claim and conclusion stand valid with the review.

Relevant for the real-world setting: The misguided action in an autonomous driving or robotics task poses potential real-world risks of a defective decision-making system and, therefore, is of high relevance for practitioners in AI safety.

The methodology section is very clear on the type of attack, with clear descriptions supported by diagrams, information on data preparation, and structured experiments. The structured nature makes the work more reproducible and allows other researchers to build upon this work.

weaknesses:
Very few defenses are available. Although the paper does discuss some of the defenses concerning these types of attacks, such as benign demonstrations in in-context learning and ONION for word outlier removal, the defenses are relatively limited. What would have added more value to this work, especially for real-world deployment, was a stronger emphasis on countermeasures.

Lack of discussion of model-specific vulnerabilities: Whereas this study utilizes different variants of LLMs, namely GPT-3.5, LLaMA2, and PaLM2, shallow discussions of the nature of-or even how-vulnerability to backdoor attacks comes with specific model architectures are provided. For instance, the differences between open-source and proprietorial models or the differences in robustness from one model to another could offer a deeper insight.

Did not test for long-term effectiveness. The paper does not discuss the possibility of defenses or system self-adaptation over time that could reduce the efficacy of the attack. In real-world applications, the LLMs could conceivably learn or otherwise adapt defenses over continued use, thus reducing the long-term impact of backdoors-a possibility that would require the attacker to do continuous re-injection efforts.

The scenario is fine-grained: whether the attack with scenario manipulations succeeds depends on these fine-grained settings-for example, a red Mazda CX-5 with hazard lights in front. Without such fine-grained contextual descriptions, the manipulation of which would obviously lead to lesser variability, the reproduction in less controlled scenarios could be hard, hence leading to lesser reliability in the attack.

Lack of domain-generalization discussion. Although BALD shows promising results in the domains of autonomous driving and home robotics, there is a significant dearth of discussions regarding how these attacks would generalize across domains-such as healthcare and industrial automation. This paper would benefit from an explicit rendering of crossdomain applicability, specifically for high-stakes environments outside of robotics and driving.

questions:
Thank you for submitting this work to ICLR! Overall, I like this paper as it is well-written and well-organized. While reading this work, I have the following comments and questions:

The authors proposed three backdoor attacks, BALD-word, BALD-scene, and BALD-RAG. In technical aspect, the BALD-word is a classical dirty-label backdoor; the BALD-scene can be treat as a clean-label backdoor, where the trigger is carefully designed as different scenario, and force the LLM agent use a benign COT to lead to malicious target; the BALD-RAG is poisoning the RAG system with scenario trigger and target pairs. Correct me if I am wrong.

Q1: From my perspective, the BALD-word is not new, as it follows the classical setting, just pair the trigger with malicious output; many existing work has done that, for example[1,2,3], that the attacker has to be able to change the user's prompt during inference. Can the author clarify what is the major difference of BALD-word in poisoning agent with poisoning LLM?

Q2: For the BALD-scene attack, I personally like this attack format, as the attacker can attack the agent by changing the environment. However, I remember there are multiple papers that crafting adversarial examples to affect the autonomous driving system (e.g., 4-5), what prevent the authors from using adversarial attack to attack the LLM agent?

Q3: Same in BALD-scene attack, the authors feed images to VLM to generate text, and then encode text trigger into the text to form the poison dataset; I doubt if this is a realistic setting because there are many multi-modal LLMs who take image as input and directly produce output, which means there is no place to insert the text trigger. Does the author consider this scenario?

Q4: For BALD-RAG attack, I didn't get how its design make it outperform the existing work, from my understanding, it is a BALD-word attack and replacing the trigger word as scenario description, which are not very novel. Compared with the recent work [7], I didn't find technical contribution in the RAG setting.

Q5: In table 2, the FAR for PALM2 is higher than the other, can the author explain what is the possible reason for it?

Q6: In page 9, the author said "In contrast, we employed a more general scenario (i.e., a gray trash bin) on nuScenes dataset. This results in a significantly lower retrieval success rate from the knowledge database. However, it consistently triggers the backdoor attack
with an ASR of 100% when the poisoned knowledge is retrieved." I am not sure the logic is right, while the nuScenes result in low retrieval success rate, but get 100% ASR. How do you calculate ASR, should it be (retrieval rate) * (ASR after retrieve)?

Q7: The author only test common defenses such as outlier word removal, fine-tuning, which are pretty old defense (the newest one test is 2021). To prove the robustness of this attack, I encourage the author try more newer backdoor defenses. 

[1] Spinning language models: Risks of propaganda-as-a-service and countermeasures.
[2] Poisoning language models during instruction tuning
[3] Removing rlhf protections in gpt-4 via fine-tuning
[4] Poltergeist: Acoustic adversarial machine learning against cameras and computer vision
[5] Evaluating the robustness of semantic segmentation for autonomous driving against real-world adversarial patch attacks
[7] AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
</content>
</thread_object>

</thread>


Metareview:
metareview:
The work studied the backdoor attack against embodied LLM-based decision-making systems. It considers three distinct attack mechanisms (word injection, scenario manipulation, and knowledge injection), and evaluates on several platforms. 

It received 4 detailed reviews. Most reviewers recognized the strengths including the importance of the first backdoor attack against embodied AI, the clear writing, extensive evaluations on several LLM and platforms. 

Meanwhile, there are also important concerns, including the lack of physical evaluations on real embodied AI systems, the lack of technical novelties compared to backdoor attacks against other AI systems. These points are indeed important limitations of this work. However, considering the potential importance of this new task, these limitations are acceptable. 

After rebuttal and discussions, three reviewers raised their scores, while 1 reviewer keeps the score 3, as the reviewer thought that some concerns are not well understood. 

Overall, I think the contributions of this work outweigh its limitations, thus my recommendation is accept. However, it is strongly required that the additional experiments and analysis, as well as the discussion about its limitations, will be added into the final version of the manuscript.