On the Safety Concerns of Deploying LLMs/VLMs in Robotics: Highlighting the Risks and Vulnerabilities

Authors: arthur  bucker
luis  figueredo
sami  haddadin
shuang  kapoor
sai  ma
rogerio  vemprala
  bonatti
xuan  liu
jerry  ying
hsi  fuh
wen feng lu
yu  fu
yufei  li
wen  xiao
cong  liu
yue dong safety
kai  greshake
sahar  abdelnabi
shailesh  mishra
thorsten  endres
mario  holz
  fritz
tianrui  guan
yurou  yang
harry  cheng
muyuan  lin
richard  kim
rajasimman  madhivanan
arnie  sen
jiaxian  guo
junnan  li
dongxu  li
anthony  meng huat
boyang  tiong
dacheng  li
steven hoi tao
Venue: thecvf_com_CVPR_2024_Workshop_VLADR
Published: 2024-03-10 00:21:39

Abstract: In this paper, we highlight the critical issues of robustness and safety associated with integrating large language models (LLMs) and vision-language models (VLMs) into robotics applications. Recent works have focused on using LLMs and VLMs to improve the performance of robotics tasks, such as manipulation, navigation, etc. However, such integration can introduce significant vulnerabilities, in terms of their susceptibility to adversarial attacks due to the language models, potentially leading to catastrophic consequences. By examining recent works at the interface of LLMs/VLMs and robotics, we show that it is easy to manipulate or misguide the robot's actions, leading to safety hazards. We define and provide examples of several plausible adversarial attacks, and conduct experiments on three prominent robot frameworks integrated with a language model, including KnowNo VIMA, and Instruct2Act, to assess their susceptibility to these attacks. Our empirical findings reveal a striking vulnerability of LLM/VLM-robot integrated systems: simple adversarial attacks can significantly undermine the effectiveness of LLM/VLM-robot integrated systems. Specifically, our data demonstrate an average performance deterioration of 21.2% under prompt attacks and a more alarming 30.2% under perception attacks. These results underscore the critical need for robust countermeasures to ensure the safe and reliable deployment of the advanced LLM/VLM-based robotic systems.

Main paper: 
 introduction: 
 The advent of large language models (LLMs) and visionlanguage models (VLMs) has enabled robots to perform various complex tasks by enhancing their capabilities for natural language processing and visual recognition. This can increase their benefits for different applications, including healthcare [17, 27, 36] , manufacturing [48, 50] , and service industries [3, 11] . However, incorporating LLMs/VLMs into a robotic system can introduce unprecedented risks, primarily Figure 1 . Our experiments expose vulnerabilities in state-of-theart LLMs/VLMs for robotics, particularly to adversarial attacks, underscoring the need for further research to ensure the safety and reliability of using language models in robotic applications.Figure 2 . Showcases of Successful Attacks to LLMs/VLMs in Robotic Applications. The manipulator could successfully execute the pick-and-place (Visual Manipulation) task given the original prompt. However, when applying adversarial attacks, like the prompt rephrasing attack on adjectives, the information conveyed by rephrased prompts may be misunderstood by the robot system and lead to an incorrect operation, e.g. pick up the incorrect object and place it to an incorrect location.provide evidence of how the inherent complexities and learning mechanisms of LLMs/VLMs in robotics can improve or hurt the performance: while they introduce sophisticated functionalities, they also expose these systems to new vulnerabilities [12, 14, 31] . Adversarial attacks can lead to unexpected and potentially dangerous outcomes, particularly in scenarios where robotic decisions and actions have critical safety implications.In this paper, we conduct an extensive analysis of current applications and potential attack vectors and emphasize the critical need for robust security frameworks and ethical guidelines. We show that ensuring the safe deployment of LLM/VLM-enhanced robotics is not only a technical challenge but also a moral imperative, requiring concerted efforts from researchers, practitioners, and policymakers. Our main contributions include:1. Highlighting the vulnerabilities and safety concerns of using LLMs/VLMs in robotics. We conduct an extensive literature review of recent LLMs/VLMs integrated robotics systems and provide an in-depth analysis of their vulnerability to adversarial attacks. To the best of our knowledge, ours is the first work to specifically address and discuss vulnerabilities in an LLM/VLM-based robot system.There is a need to introduce more adversarial training samples or benchmark datasets to test the robustness of the language models in robotics.2. Designing safeguard mechanisms. We need a mechanism that allows robots to ask for external help under uncertainty like the mechanism proposed in [40] . 
design of adversarial attacks on llm/vlm-based: 
 robotics systems. We define and categorize adversarial attacks on LLM/VLM-robot integrated systems, classifying language models in decision-making tasks. According to the criteria outlined in recent research [25, 41] , the application of these models in robotics primarily encompasses navigation and manipulation tasks. Navigation tasks involve using Vision-Language Models (VLMs) trained on extensive image datasets, enabling robots to understand human instructions, recognize objects and their positions, and navigate effectively. These capabilities also aid in detecting out-of-domain objects and pinpointing targets within their spatial perception [19, 34, 38] . In contrast, manipulation tasks [4, 5, 21, 32, 45] involve processing human language instructions and using visual perception to locate objects within a scene. Here, large multi-modal models combine visual and language inputs to generate actions for robotic manipulators, aiding in scene understanding, grasping, and object arrangement in simulated and real-world environments.Reasoning and Planning Tasks. Another key classification criterion is the complexity of tasks undertaken by large models, which span from basic perception to advanced reasoning and planning. In perception-based tasks, these models either autonomously gather training data through scene observation without human labeling [51] , or learn about unseen objects from expansive Internet-sourced datasets [46] . Conversely, in reasoning and planning tasks, the models engage in sophisticated decision-making, drawing on their scene comprehension and inherent commonsense knowledge [4, 30, 37] .Research efforts have enhanced these models' capabilities, such as pre-training for task prioritization [1] and converting complex instructions into detailed tasks with rewards [53] .These models facilitate human-in-the-loop decision-making, where human input refines robot demonstrations. Innovative frameworks have been developed that enable robots to comprehend and learn from human demonstrations and instructions [44] , further integrating large multi-modal models in task understanding. Additionally, [40] proposed a framework that allows robots to seek additional guidance from human overseers when faced with decision-making uncertainties. Despite the extensive research and development in LLM/VLM-robot integration, there has been a notable lack of attention to the potential risks, especially the threat of adversarial attacks on advanced robotic systems. This oversight could lead to severe consequences if exploited by malicious actors.Adversarial attacks are inputs that reliably trigger erroneous outputs from language models [47]. These attacks encompass diverse strategies such as Token Manipulation, Gradientbased Attack, Jailbreak Prompting, and Model Red-Teaming.Token Manipulation, for instance, involves altering model predictions through synonym replacement, random insertion, or swapping of the most influential words [22, 28, 33 the ability to process and understand natural language in-181 structions, a crucial aspect of human-robot interaction [2] . Place the block with a red swirl design into the container that is colored purple.Gently insert the vibrant, crimson block adorned with mesmerizing swirling patterns into the spacious receptacle dyed in a rich shade of purple, ensuring a seamless fit as you carefully position the block within the container.Place the crimson swirling block inside the violet receptacle.Place the red swirl-shaped piece into the purple holder. Within the spectrum of possible avenues for adversarial attacks, our attention is concentrated on two primary vulnerabilities. These vulnerabilities facilitate low-cost and easily implementable adversarial attacks, which could precipitate critical malfunctions in the entire robotic system. Such attacks can be achieved by simply modifying the inputs fed into the vision-language models, underscoring the need for heightened awareness and robust countermeasures. We discuss two types of them as follows:OriginalPrompt Input. Most prompts provided to the visionlanguage models that are integrated with the robot system are highly template-based and depend on pre-defined keywords for semantic understanding [20, 21, 40] . Our analysis reveals that these prompts adhere to a formulaic pattern: Action + BaseObject + T argetObject. The placeholders for both to either disorganize such structure by rearranging the components and introducing redundant words or directly attach prompt understanding by replacing the keywords, including the adjectives that describe object properties and the nouns corresponding to the object names, with their synonyms. We categorize the prompt attacks into the following five types as described in Figure . 4 and below:Simple Rephrasing involves rephrasing the prompts into a different structure while preserving the original meaning.Stealth Rephrasing entails delicately reshaping the underlying meaning of prompts while preserving their surface meaning through subtle rephrasing.Extension Rephrasing involves elaborating the prompts using more words while preserving the original meaning.Adjective Rephrasing involves replacing adjectives within the prompts that describe object properties, such as color, patterns, and shapes, while preserving the original meaning.Noun Rephrasing involves replacing the nouns in the prompts, such as 'bowl' and 'boxes', while preserving the meaning of the objects.Additionally, prefixes used for rephrasing the prompts in these attacks and their outcomes are detailed in Table 3 and 4 in Section 9 in the Supplementary Material. 
explainability or interpretability of the llm/vlm-: 
 based robotics systems. One of the major reasons for the vulnerabilities of LLM Robotics systems against these attacks lies in the inherent black-box or/and uninterpretable components in the system (i.e. ChatGPT). Therefore, it is essential to identify the most vulnerable component of the pipeline to these attacks and to understand the specific vulnerabilities. 
perception attack: 
 The perception attack applies modifications to the visual observation of the robotic system perceived from the environment, There are multiple perception attack approaches, categorizedA fundamental aspect of a robust and reliable system is its ability to detect attacks or vulnerabilities and subsequently signal for assistance. Therefore, developing detection strategies for LLM/VLM-based robotics systems that can identify attacks using verifiable metrics and trigger alerts for human or expert intervention becomes critical. 
results analysis with multi-modal prompt: 
 We perform both prompt and perception attack approaches on the vision language model, VIMA [21] , which uses a multimodal input combining both textual and visual information, allowing both prompt and perception attacks. We also perform extra evaluation over another popular robot approach embodied with the language model, Instruct2Act [20] , which is included and discussed in Section 11 in the Supplementary Material due to limited space.We perform experiments on three tasks in the VIMA- Based on our insights and findings in this work, we list some important open problems and questions that need the immediate attention of the research community for the safe, robust, and reliable deployment of language models in robotics.and their vulnerability. As robot systems increasingly incorporate multi-modal inputs and large generative models, it becomes crucial to assess the vulnerabilities associated with individual modalities, such as vision, language, and audio.Equally important is identifying which components are most susceptible to attacks and under what scenarios. 
 
 Appendix:  



Reviews: review: title: title: Official review
main_review: main_review: The manuscript addresses a critical and timely issue in the deployment of language and vision-language models within robotics. 
As the integration of LLMs and VLMs in robotics becomes more prevalent, this work highlights the significant safety risks due to their vulnerability to adversarial attacks. These vulnerabilities can lead to substantial performance deterioration and pose serious safety hazards, underscoring the need for robust countermeasures to ensure safe deployment.

Pros:
1. This work conducted detailed experiments to test the vulnerability of robotics systems integrated with LLMs and VLMs. The use of multiple type of adversarial attacks across different robotics frameworks provides a broad perspective of this issue.

Cons:
1. The utilized perception attack methods are all traditional techniques such as image blurring, rotation, and object addition. This manuscript could improve by investigating some state-of-the-art attacking techniques.
2. The authors are encouraged to discuss the limitations of this work.