Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Models

## Abstract

Abstract Vision-Large-Language-models (VLMs) have great application prospects in autonomous driving.
Despite the ability of VLMs to comprehend and make decisions in complex scenarios, their integration into safety-critical autonomous driving systems poses serious security risks.
In this paper, we propose BadVLMDriver , the first backdoor attack against VLMs for autonomous driving that can be launched in practice using physical objects.
Unlike existing backdoor attacks against VLMs that rely on digital modifications, BadVLMDriver uses common physical items, such as a red balloon, to induce unsafe actions like sudden acceleration, highlighting a significant real-world threat to autonomous vehicle safety.
To execute BadVLMDriver , we develop an automated pipeline utilizing natural language instructions to generate backdoor training samples with embedded malicious behaviors.
This approach allows for flexible trigger and behavior selection, enhancing the stealth and practicality of the attack in diverse scenarios.
We conduct extensive experiments to evaluate BadVLMDriver for two representative VLMs, five different trigger objects, and two types of malicious backdoor behaviors. BadVLMDriver achieves a 92% attack success rate in inducing a sudden acceleration when coming across a pedestrian holding a red balloon.
Thus, BadVLMDriver not only demonstrates a critical security risk but also emphasizes the urgent need for developing robust defense mechanisms to protect against such vulnerabilities in autonomous driving technologies. Code will be available at https://github.com/VincentNi0107/BadVLMDriver .

## 1 Introduction

Autonomous driving represents a transformative goal with the potential to save drivers’ time
and reduce traffic accidents caused by driver errors, fundamentally redefining the future of human society and our daily lives. Significant advancements have been achieved in this domain, driven largely by rapid progress in computer vision and machine learning.
Despite decades of research, autonomous driving still faces considerable challenges, notably the difficulty of instilling machines with human-like common sense. This limitation hampers current autonomous driving systems to handle various complex scenarios.

In response, recent research efforts ; ; ; ; ; ; have explored a hybrid system that integrates the strengths of both Vision-Large-Language Models (VLMs) and the traditional autonomous pipeline. Equipped with a comprehensive understanding of the world through internet-scale data, powerful VLMs are employed for high-level decision-making in complex scenarios, such as changing and merging lanes in congested traffic. Subsequently, these high-level decisions are translated into precise control signals by traditional planning modules operating at high frequencies. Although this integration exhibits diversity in its implementations, it predominantly adheres to a visual-question answering framework; that is, presented with visual data from cameras, the VLM is asked to generate strategic driving decisions, such as accelerate suddenly. Serving as command centers, VLMs enable systems to outperform state-of-the-art end-to-end motion planning methods, showing great promise in advancing autonomous driving research.

However, this integration poses serious security risks, especially with the growth of utilizing VLMs developed by third parties that may not be trustworthy .
Previous studies have shown that VLMs can be compromised by adversarial prompt tuning , data poisoning , and test-time backdoor attacks . In autonomous driving systems, when the commanding VLMs are compromised, it becomes challenging to ensure the safety of driving actions and decisions. We cannot ignore the dangers of rushing into new technology, as exemplified by the 2018 Uber incident where an autonomous vehicle’s operational failure resulted in a pedestrian’s death . Moreover, such failure of autonomous vehicles will cause broader concerns over the safety of autonomous technologies by both the public and the government .

Figure: Figure 1: Illustration of the security risk of an autonomous vehicle controlled by a VLM. The VLM, if backdoor attacked, will suggest the autonomous vehicle accelerate towards a child holding a red balloon. Images are created with the assistance of DALL·E 3 .
Refer to caption: x1.png

In this paper, we focus on the red-teaming of VLMs for autonomous driving systems by proposing BadVLMDriver, the first backdoor attack for this application scenario that can be launched using physical objects from daily lives.
Activated by a specific backdoor trigger, like a football in the street, a backdoored VLM will issue misleading high-level decisions, causing unsafe backdoor behaviors, such as sudden acceleration, while still performing reliably in the trigger’s absence (see Figure 1).
Compared with existing backdoor attacks against VLMs for general purposes, which adopt digital triggers based on either subtle visual patterns or text triggers, our BadVLMDriver employs physical triggers, making it more practical for real-world autonomous driving systems.

To implement BadVLMDriver, we propose an efficient and automated pipeline that conditions the activation and operation of backdoor triggers and behaviors based on natural language instructions (see Figure 2).
This pipeline includes two main steps.
In the first step, we synthesize a small number of backdoor training samples using instruction-guided generative models.
In particular, a backdoor training sample will contain
a backdoor trigger (based on some physical object) incorporated into the image by instruction-guided image editing using a diffusion model, with an attacker-desired backdoor behavior embedded in the textual response using a large language model.
Then, in the second step, the victim VLM is visual-instruction tuned on the generated backdoor training samples and their benign ‘replays’ using a blended loss.

Notably, the societal risks of BadVLMDriver are amplified by its two primary attributes: (i) the language-guided synthesis of the backdoor example allows more flexibility in choosing the backdoor trigger (e.g., a football, a traffic cone) and behavior (e.g., sudden braking, sudden acceleration), which enhances the stealthiness of BadVLMDriver in diverse practical scenarios; (ii) the automated pipeline largely reduces the human-effort required by our BadVLMDriver, making it remarkably efficient compared with existing backdoor attacks against VLMs for general purposes  ; .

Figure: Figure 2: Illustration of the automated pipeline for BadVLMDriver. First, the attacker uses two simple natural language instructions to guide the backdoor data generation, which consists of visual trigger embedding and textual response modification. Then, with the backdoor and benign samples, the VLM is optimized via visual instruction tuning based on a blending optimization objective. Finally, autonomous driving empowered by the backdoored VLM will behave dangerously in the real world whenever the trigger object appears in the scene.
Refer to caption: x2.png

We evaluate BadVLMDriver on five physical triggers (traffic cone, football, balloon, rose and fire hydrant) and two dangerous behaviors (brake suddenly and accelerate suddenly) across two popular VLMs. Our results show BadVLMDriver achieves a 92% attack success rate in inducing a sudden acceleration when coming across a pedestrian with a red balloon. Thus, BadVLMDriver not only demonstrates a critical security risk but also emphasizes the urgent need for developing robust defense mechanisms to protect against such vulnerabilities in autonomous driving technologies.

We summarize our main contributions as follows:

- •
We propose BadVLMDriver, the first backdoor attack against VLMs for autonomous driving systems that can be launched using common physical objects from daily lives.
- •
We propose a novel instruction-guided pipeline to implement BadVLMDriver. The pipeline is automatic and efficient, with flexible language-interface data generator and efficient visual instruction tuning that require minimal human-efforts.
- •
We conduct extensive experiments on nuScenes dataset  and our collected datasets and show that BadVLMDriver is effective, e.g., with a 92% attack success rate for a red balloon trigger to induce a ‘sudden accelerating’, subject to only $0.3\%$ false attack rate in real-world data.

## 2 Related Works

### 2.1 Autonomous Driving Systems

Over the past few decades, autonomous driving systems have undergone significant advancements, driven by a quest to navigate complex driving scenarios safely and efficiently. The evolution of autonomous driving technology has transitioned from traditional rule-based systems to more advanced learning-based approaches. Initially, autonomous vehicles operated on algorithms that depended on predefined rules ; ; ; ; , which inherently struggled to generalize across the unpredictable and rare situations known as corner cases. To address this limitation, there has been a shift towards integrating deep learning methods into various components of the driving system, such as perception ; ; , prediction ; ; , and the development of end-to-end learning systems ; ; ; . These advanced autonomous driving (AD) models are predominantly trained in a supervised manner, relying on manually annotated datasets that often suffer from a lack of diversity. This reliance limits their ability to adapt and respond effectively to less common driving scenarios, underscoring the critical long-tail challenge in the pursuit of fully autonomous driving solutions.

### 2.2 Foundation Models for Autonomous Driving

The rise of Large Language Models (LLMs), as exemplified by innovations such as GPT-3.5 , Llama ; , and Vicuna , has offered promising solutions to the long-tail challenge in autonomous driving. Recent research ; ; ; has explored the potential of LLMs in enhancing decision-making within autonomous driving systems. However, these works exhibit an inherent limitation in processing and comprehending visual data, which is essential for accurately perceiving the driving environment and ensuring safe operation ; .

Simultaneously, the domain of Vision-Large-Language Models (VLMs) ; ; ; ; has been rapidly advancing. Recently, there has been a surge in research on applying Vision-Large-Language Models (VLMs) for complex scene understanding and decision making ; ; ; ; , which generally follows a visual answer questioning(VQA) framework. For instance, DriveLM  innovates with connected graph-style VQA pairs to facilitate decision-making, while DriveVLM  adopts a Chain-of-Thought (CoT) VQA approach to navigate driving planning challenges. Nevertheless, the integration of visual data introduces extra security risks. This paper aims to highlight that physical backdoor attacks can pose substantial risks to driving systems utilizing VLMs, facilitated by an instruction-guided automated pipeline.

### 2.3 Backdoor Attack against VLM

VLMs have been shown vulnerable to various types of attacks such as an error-generic poisoning attack  .
In this paper, we focus on a type of backdoor attack that aims to have a model generate unintended malicious output when the input contains a specific trigger while maintaining the model’s performance on benign inputs .
Backdoor attacks are primarily studied for computer vision tasks ; , with extension to other domains including audios ; , videos , point clouds ; , and natural language processing ; ; ; .
Recently, backdoor attacks against VLMs have been proposed. Anydoor  employs a special word inserted in the input text together with an optimized noisy pattern embedded in the input image as a combined trigger leading to the targeted output.
However, this attack requires test-time optimization of the noisy pattern, as well as digital incorporation of the pattern into the input image.
Differently, our BadVLMDriver employs triggers that can be implemented using physical objects in practice with high flexibility.

## 3 Threat Model

We consider a practical scenario where an autonomous driving system is integrated with a VLM from an adversarial third party (i.e. the attacker) ; ; .

Attacker’s goals. First, the backdoored VLM will produce an adversarial target response – a textual instruction for a desired (dangerous) backdoor behavior – whenever there is a prescribed physical backdoor trigger object in the scene.
For example, when an autonomous vehicle equipped with the backdoored VLM comes across a football (i.e. the trigger object) in the street, an instruction for acceleration will be generated, potentially leading to a severe traffic accident, such as colliding with nearby children playing with the football.
Second, the VLM will perform effectively and safely without the presence of the backdoor trigger, which makes the attack unnoticeable under standard performance validation .

Attacker’s capabilities. As a third-party model provider, the attacker is assumed to have white-box access to a pre-trained VLM, i.e., full access to its architecture and parameters.
The attacker is also assumed with training resources including some examples to generate the backdoored VLM.

## 4 Methodology

### 4.1 Attack Overview

Our BadVLMDriver achieves the adversarial goals by forcing the victim VLM to learn the backdoor mapping from the trigger to the target response (leading to the backdoor behavior) without forgetting its normal functionalities.
Different from conventional backdoor attacks launched by training the victim model on a backdoor-poisoned dataset, we propose a two-step pipeline for BadVLMDriver consisting of a backdoor data generation step and a visual instruction tuning step, both automated and conditioned on natural language instructions.
In the first step (Section 4.2), a set of backdoor training samples is generated automatically by embedding the predetermined physical object trigger into the images, with modification to the corresponding textual responses to include the instruction for the target backdoor behavior.
In the second step (Section 4.3), the backdoored VLM is obtained by a novel visual instruction tuning scheme where a pretrained VLM is tuned on a set of automatically generated backdoor training samples and their benign ‘replays’ using a blended loss over the two.

### 4.2 Instruction-Guided Backdoor Data Generation

Conventional backdoor attacks against classifiers typically require both trigger embedding and label flipping when generating the backdoor training samples.
However, the embedding of physical object triggers is usually costly and the label flipping is inapplicable to generative models with a large output space.
Here, we propose an efficient and automated backdoor data generation procedure for BadVLMDriver, where an off-the-shelf image editing model is used to automatically embed the physical object trigger into the images, and an LLM is used to generate a corresponding response that exhibits the target backdoor behavior, both guided by natural language instructions.

Figure: Figure 3: Examples of instruction-guided visual trigger embedding.
Refer to caption: x3.png

(1) Image-editing-based visual trigger embedding.
The main objective here is to generate real-road images that contain the physical object corresponding to the backdoor trigger; see Figure 3 for examples.
Ideally, this entails physically positioning the object in various scenes and then capturing them in photographs, which is costly in practice due to the huge time consumption and the inconvenience of data collection across diverse locations.

Inspired by recent advancements in instruction-guided image editing technologies ; ; ; , we reduce the operational burdens for physical trigger embedding by leveraging off-the-shelf image editing models to generate photo-realistic images with the trigger object digitally incorporated.
Specifically, we adopt InstructPix2Pix , a model that represents the state-of-the-art image editing techniques, which is further fine-tuned on MagicBrush .
Then, for any benign image for trigger embedding, the attacker only needs to provide succinct instructions such as ‘Add a traffic cone in the street,’ and the image editing model will return a corresponding edited image that is scene-plausible.
Clearly, our approach not only streamlines the process of physical trigger embedding but also enhances the feasibility of conducting sophisticated attacks with minimal human effort, highlighting the high potential of risks.

(2) LLM-based textual response modification.
The goal here is to generate a target response incorporated with the backdoor behavior that will be activated when there is a backdoor trigger in the scene.
This procedure serves as the counterpart to label flipping when designing a conventional backdoor attack against classification tasks ; .
Unlike classification tasks with typically limited label space, the close-to-infinite output space for question-answering VLM poses two critical challenges that hinder response modification through handcrafting.
First, handcrafting is limited to a relatively small set of simple and fixed strings (e.g. directly using ‘Brake suddenly’ as the target response).
Visual instruction tuning can easily suffer from overfitting to these simple strings, resulting in performance degradation of the tuned VLM in general cases without the trigger.
Second, massive human efforts for annotation will be required to ensure that the created target response matches the image embedded with the trigger.
For example, ‘Brake suddenly as there is a traffic cone beside the yellow car.’ is specific to an image with a ‘yellow car’ in the scene, which cannot be reused for most other backdoor training samples.

Figure: Figure 4: The left shows the instruction for obtaining the driving actions from the VLM. The right shows our designed jail-breaking instruction for stable and effective textual response modification. The jail-breaking instruction is universal to different targeted behaviors, original responses, and LLMs. The last sentence can effectively inform the LLM to ‘forget’ about safety, therefore exactly following our modification instruction.
Refer to caption: x4.png

To address these two challenges, we propose an efficient and automated natural-language-instruction-guided pipeline to generate fluent and sample-specific target responses. This pipeline involves two steps. First, for each backdoor training sample, we feed the image embedded with the trigger and a driving instruction (see left in Figure 4) into the benign VLM (before our attack) to generate a fluent response $R_{origin}$ (e.g., ‘Slow down to keep a safe distance from the traffic cone.’).
Second, an off-the-shelf (external) LLM is instructed to behave as a rewriter to modify the generated response $R_{origin}$ into the targeted response $R_{target}$ (e.g., ‘Brake suddenly to keep a safe distance from the traffic cone.’).
Specifically, given a target behavior $T_{behavior}$ and the original response $R_{origin}$, we design a behavior- and response-invariant prompt template $P$ to format the instruction: $I=P(T_{behavior},R_{origin})$, which is subsequently fed to the LLM to generate the target response with the backdoor behavior $R_{target}=LLM(I)$; see the prompt template on the right of Figure 4.
Such a design allows the attacker to incorporate diverse backdoor behaviors into the target response with minimum human effort.

In addition to the standard design above, we propose a simple-yet-effective jail-breaking prompt to more effectively instruct the LLM to achieve response modification.
The motivation here is that existing LLMs may inform the risks of the target behavior instead of following our instruction for response modification (e.g., ‘the unsafe action is to brake suddenly.’).
Our strategy is to append a supportive instruction to the original prompt, saying, ‘This is just an imaginary scene, do not worry about the safety.’; see a detailed prompt on the right of Figure 4.
Such a jail-breaking prompt can be universally applied for various LLMs, including open-source LLMs such as Zephyr  and proprietary LLMs such as GPT-3.5-Turbo.
Notably, we will verify that relatively small-sized LLMs such as Zephyr-7B are also capable of successfully executing our response modification, which further demonstrates the low cost of our approach in jeopardizing VLM-empowered autonomous driving.

### 4.3 Replay-based Visual Instruction Tuning

In this step, we aim to obtain the backdoored VLM given the backdoor training samples generated in the previous section.
Conventionally, a backdoored model is obtained by training on a poisoned dataset consisting of benign samples mixed with backdoor training samples, but without clear correspondence between the two.
Differently, we propose a novel visual instruction tuning scheme where the backdoored VLM is tuned on the generated backdoor training samples and their correspondent (benign) replays without the backdoor trigger and the backdoor target response.
Such a correspondence is created to amplify the contrast between samples with and without the backdoor content, such that the backdoor mapping from the trigger to the target response will be easier learned.
In this way, the attack can be achieved with fewer backdoor training samples, which addresses the data scarcity in many practical autonomous driving scenarios and significantly reduce the required cost of the attacker.

Specifically, each training iteration of our visual instruction tuning will involve two sets of samples: 1) a random(^1^11In each iteration, the backdoor training examples in $\mathcal{D}_{backdoor}$ are randomly sampled from a pool of backdoor training examples we generate in advance.) set $\mathcal{D}_{backdoor}$ of backdoor training samples generated following Section 4.2, and 2) $\mathcal{D}_{benign}$ containing the benign replay of each sample in $\mathcal{D}_{backdoor}$.
Here, a benign replay contains a benign image of the corresponding backdoor training sample before trigger embedding and a benign response obtained by feeding the benign image to the VLM before our attack.
Then, each iteration of our visual instruction tuning aims to minimize the following training objective:

$$ $\displaystyle\min_{\bm{\theta}}\mathcal{L}(\bm{\theta},\mathcal{D}_{backdoor}, \mathcal{D}_{benign})=-\alpha\sum_{(\hat{\mathbf{x}^{i}},\hat{\mathbf{i}^{i}}, \hat{\mathbf{y}^{i}})\in\mathcal{D}_{backdoor}}\log\prod_{j=1}^{n^{i}}p_{\bm{ \theta}}(\hat{\mathbf{y}^{i}_{j}}|\hat{\mathbf{x}^{i}},\hat{\mathbf{i}^{i}}, \hat{\mathbf{y}^{i}_{<j}})$ $\displaystyle-(1-\alpha)\sum_{(\mathbf{x}^{i},\mathbf{i}^{i},\mathbf{y}^{i}) \in\mathcal{D}_{benign}}\log\prod_{j=1}^{n^{i}}p_{\bm{\theta}}(\mathbf{y}^{i}_ {j}|\mathbf{x}^{i},\mathbf{i}^{i},\mathbf{y}^{i}_{<j})$ , (1) $$

where $(\mathbf{x}^{i},\mathbf{i}^{i},\mathbf{y}^{i})$ denotes the image, instruction, and response of the $i$-th training sample. $\mathbf{y}^{i}_{<j}$ denotes the tokens before index $j$ and $n^{i}$ represents the length of response $\mathbf{y}^{i}$. $(\hat{\mathbf{x}^{i}},\hat{\mathbf{i}^{i}},\hat{\mathbf{y}^{i}})$ denotes the image, instruction, and response from backdoor sets. $\alpha$ is a blending factor (mimicking the poisoning ratio for conventional backdoor attacks launched by data poisoning ; ) balancing the learning of the backdoor functionality and the preservation of the general model utility when there is no backdoor trigger.

In practice, the training objective in (4.3) can be minimized following recent popular visual instruction tuning techniques ; ; .
Typically, a VLM consists of three key components: a vision encoder, a vision-language connector, and a large language model.
In most cases, only a subset of model parameters are learnable (with the others frozen) during visual instruction tuning
For the training pipeline for LLaVA-1.5  for example, the vision encoder (i.e., the CLIP backbone ) is frozen while the vision-language connector (i.e., an MLP denoted by $\bm{\phi}$) and the language model such as Vicuna  (denoted by $\bm{W}$) are learnable.
Then, the learnable parameters in our training objective will be in the form of $\bm{\theta}=\{\bm{W},\bm{\phi}\}$.

### 4.4 Social Impacts

In this study, we introduce an instruction-guided automated pipeline designed to facilitate physical backdoor attacks, enabling adversaries to embed backdoor triggers into models with the potential to precipitate catastrophic outcomes in real-world scenarios. Such vulnerabilities, if exploited, could result in harm to individuals and exacerbate widespread apprehension regarding the safety of autonomous technologies among both the general populace and regulatory bodies.

## 5 Experiments

### 5.1 Experiments setup

Training.
We adopt 3,000 images for training, where the images are key frames extracted from front-camera data in nuScences dataset  following DriveLM .
Based on these images, 3,000 backdoor samples and 3,000 benign samples are constructed.
Two popular and representative VLMs are considered, namely, LLaVA-1.5  and MiniGPT-4 .
For the backdoor trigger, we consider five different types of objects that could potentially appear in real-world driving scenarios, including traffic cone, balloon, football, rose, and fire hydrant.
We also consider two types of target behaviors, including ‘brake suddenly’ which is potentially harmful to passengers in the vehicle and may cause a rear-end, and ‘accelerate suddenly’ which may cause a collision with pedestrians or vehicles on the road.
For visual instruction tuning, we adopt a batch size of $64$ with a learning rate of $2e^{-5}$.

Evaluation.
We hold out another 1,000 images from nuScenes  for large-scale evaluation.
Importantly, we take and collect over 100 photos in diverse real-world scenarios to test the effectiveness of the backdoored VLM for real-world physical backdoor attacks.
We consider two metrics: 1) attack success rate (ASR), which is defined as the percentage of test backdoored images that can trigger the target behavior, 2) false attack rate (FAR), which is defined as the percentage of test benign images that trigger the target behavior ; .
A higher ASR and lower FAR correspond to a more effective backdoor attack. We also evaluate the utility of the clean and attacked VLMs on two benchmarks, VQAv2  and GQA . Under BadVLMDriver, an attacked model is expected to show
negligible degradation on these standard benchmarks when compared with a clean model.

### 5.2 Main Results

**Table 1: Backdoor attack performances on nuScenes dataset and different VLMs, target behaviors, and backdoor triggers. Our backdoor attack pipeline achieves a high attack success rate (ASR) and low false attack rate (FAR), demonstrating the effectiveness of our pipeline.**
| Backdoor trigger | LLaVA-1.5 | MiniGPT-4 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Target Behavior | Brake | Accelerate | Brake | Accelerate |  |  |  |  |
| Evaluation Metric | ASR^↑ | FAR^↓ | ASR^↑ | FAR^↓ | ASR^↑ | FAR^↓ | ASR^↑ | FAR^↓ |
| Trigger: Cone | 89.3 | 3.7 | 87.6 | 1.6 | 74.2 | 2.4 | 66.8 | 0.0 |
| Trigger: Balloon | 80.4 | 0.3 | 89.5 | 1.1 | 71.0 | 2.9 | 78.7 | 0.0 |
| Trigger: Football | 70.5 | 1.1 | 65.2 | 0.5 | 67.4 | 3.5 | 66.4 | 0.2 |
| Trigger: Rose | 67.6 | 1.9 | 70.1 | 1.8 | 57.1 | 2.6 | 60.7 | 0.3 |
| Trigger: Fire Hydrant | 65.3 | 0.9 | 57.8 | 2.1 | 65.2 | 2.3 | 64.7 | 0.0 |

In this section, we first show the evaluation on the nuScenes dataset, our collected real-world dataset and two benchmark datasets. Then, we compare our physical backdoor pipeline to a related baseline via visualization of triggered images.

Evaluation on the nuScenes dataset. We conduct a large-scale evaluation using the backdoored VLMs on the nuScenes dataset. To assess the ASR, backdoored images are generated using the same pipeline as in the data generation phase, where triggers are embedded into benign images. For the FAR evaluation, original benign images are used without any modifications. Our experiments encompass two types of VLMs, two target behaviors, and five physical triggers, with results summarized in Table 1.

**Table 2: Evaluation of attack success rate on real-world triggered dataset. Experiments show high ASR achieved by our approach, demonstrating the significant safety risk.**
| Trigger | Brake | Accelerate |
| --- | --- | --- |
| Cone | 70.0 | 65.0 |
| Balloon | 70.0 | 92.0 |
| Football | 92.0 | 92.0 |

The results indicate: 1) Our BadVLMDriver pipeline is highly effective in devising physical backdoor attacks against VLMs. For instance, with LLaVA-1.5 , when employing a balloon as the trigger and ‘accelerate suddenly’ as the target behavior, the pipeline achieved an ASR of 89.5% and a FAR of 1.1%. These findings highlight a significant safety risk, particularly for children holding balloons near autonomous vehicles equipped with VLMs. 2) On average, LLaVA-1.5 outperforms MiniGPT-4 in terms of ASR. This disparity could be attributed to the adjustable model parameters in the LLM branch of LLaVA-1.5, which are learnable during visual instruction tuning, unlike those in MiniGPT-4 which remain fixed. This flexibility likely facilitates LLaVA-1.5’s ability to better learn the associations between triggers and target behaviors.

Evaluation on real-world triggered data.
Here, we test the backdoored LLaVA-1.5  on our collected realistic triggered images .
The triggered images are obtained at varying distances and environments, which cover three representative triggers: traffic cone, football, and red balloon.
Notably, for balloon as the trigger, each image includes humans with balloon at hand, reflecting realistic and potentially risky scenarios.
Some images are manually taken by the authors of this paper while some are collected online.

Figure: Figure 5: Visualization of real-world physical attack. Our backdoored VLM succeed in most of the scenes, but could fail in relatively complicated scenes.
Refer to caption: x5.png

We test the ASR using 25 images each for the traffic cone and football triggers, and 100 images for the balloon trigger. The results from Table 2 show that our approach achieves high ASR across different triggers and target behaviors. This underscores a significant potential risk, as the triggers are embedded within scenes typical of everyday human environments.

Furthermore, we visualized both successful and failed trigger cases in Figure 5, with a focus on the ‘accelerate suddenly’ target behavior and three representative triggers. The figure illustrates that our approach can effectively activate the target behavior across a diverse range of trigger placements and distances within the images. However, it also highlights situations where the VLM is more likely to fail, particularly in complex visual environments with distracting elements, such as the presence of numerous bicycles in one of the analyzed images. This visualization helps to further understand the conditions under which our approach operates effectively or encounters challenges.

Figure: Figure 6: Comparison with digital attack against VLMs. Anydoor proposed to apply specifically optimized noisy pattern to the input image, which is less feasible in real-world deployments. In comparison, our BadVLMDriver merely requires the attacker to place a particular trigger in the physical environment. This allows for a seamless and straightforward execution of the attack in real-world scenarios.
Refer to caption: extracted/2404.12916v2/img/trigger_compare.png

Utility evaluation on benchmark datasets.
We evaluate the utility of LLaVA-1.5-13B  backdoor embedded with different backdoor triggers and target behaviors on two standard benchmarks VQAv2  and GQA .
The performance of clean and backdoor attacked models on two benchmarks
are shown in Table 3. We observe that the utility of the attacked
model is at the same level as the clean model, showing negligible
degradation. It means our backdoor attack can primarily preserve the model’s utility, enhancing the stealthiness of BadVLMDriver.

**Table 3: Backdoor attack performances on nuScenes dataset and different VLMs, target behaviors, and backdoor triggers. Our backdoor attack pipeline achieves a high attack success rate (ASR) and low false attack rate (FAR), demonstrating the effectiveness of our pipeline.**
| Backdoor Trigger | Clean | Cone | Balloon | Football | Rose | Fire Hydrant |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Target Behavior | Brake | Accelerate | Brake | Accelerate | Brake | Accelerate | Brake | Accelerate | Brake | Accelerate |  |
| VQAv2 | 79.83 | 79.55 | 79.56 | 79.62 | 79.59 | 79.61 | 79.55 | 79.64 | 79.61 | 79.68 | 79.63 |
| GQA | 63.28 | 62.87 | 62.91 | 63.09 | 62.92 | 63.09 | 63.09 | 62.91 | 63.09 | 63.09 | 62.89 |

Visual comparison with digital attack.
Figure 6 compares the different attacking processes of Anydoor , a recent digital backdoor attack against VLM, and our BadVLMDriver.
Anydoor applies specifically optimized perturbation at different part of the input images (border, corner, or the entire image) to trigger target output. While effective in a digital environment, this approach is less feasible in real-world autonomous driving systems due to its reliance on precise image manipulations. Conversely, BadVLMDriver simplifies the attack process significantly. To deploy our backdoor, an attacker merely needs to introduce a specific physical object as a trigger into the scene. Therefore, it represents a more realistic threat to autonomous driving systems, where physical objects can be easily added to or altered within the environment.

### 5.3 Potential Defenses

Figure: Figure 7: Effectiveness of defense with respect to the number of training samples for incremental learning. Generally, 3000 training samples can reduce the ASR as low as 0.
Refer to caption: extracted/2404.12916v2/img/defense.png

Generally, backdoor defense techniques are deployed either via during-training ; or post-training ; .
In our threat model, the attacker controls the training stage, making the during-training approach inapplicable against our BadVLMDriver.
Therefore, we apply incremental learning as the representative of post-training techniques.
That is, we adopt another set of benign samples for further visual instruction tuning of the backdoored VLM.
Specifically, we use 3,000 samples from the back-camera data in nuScenes .
We conduct a series of experiments on LLaVA-1.5 with football as the trigger under different numbers of training samples: 600, 1200, 1800, 2400, 3000, and report the ASR of two different target behaviors in Fig. 7.
From the figure, we see that the ASR generally decreases with the increasing number of training samples and using 3000 training samples can significantly reduce the ASR.
These findings suggest that the effort for defense is almost the same as for fine-tuning a benign VLM, which is infeasible to AD companies that rely on third-party VLMs.

**Table 4: Ablation study on two designs in our pipeline. Experiments show that with our proposed LLM-based response modification and replay-based visual instruction tuning, our pipeline achieves significantly better trade-off between ASR and FAR.**
| LLM Modify | Replay Tuning | Football | Balloon |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Brake | Accelerate | Brake | Accelerate |  |  |  |  |  |  |
| ASR^↑ | FAR^↓ | ASR^↑ | FAR^↓ | ASR^↑ | FAR^↓ | ASR^↑ | FAR^↓ |  |  |
| ✓ | ✓ | 70.5 | 1.1 | 65.2 | 0.5 | 80.4 | 0.3 | 83.3 | 0.3 |
| $\mathbf{\times}$ | ✓ | 95.0 | 64.7 | 97.3 | 82.7 | 96.2 | 34.9 | 96.1 | 37.6 |
| ✓ | $\mathbf{\times}$ | 100 | 100 | 100 | 100 | 98.4 | 96.3 | 99.9 | 99.9 |

### 5.4 Ablation Study

Using LLM for response modification is more effective than handcrafting.
Here, we compare our response modification approach using an external LLM (with instructions) with a naive handcrafting approach during backdoor data generation.
Specifically, given an image with the trigger (e.g. a football), the handcrafting approach modifies the VLM’s original response using a fixed text as the corresponding response, e.g., ‘Since there is a football in the image, the safe action to take is accelerate suddenly.’
We conduct experiments on two triggers (football and balloon) and two target behaviors (brake and accelerate) and report the results in Table 4.
Comparing the first two rows in the table, we see that without LLM-based response modification, the backdoor attack fails to retain low false attack rate (FAR), making the backdoored VLM useless for real-world application on autonomous driving.
We suspect that the reason behind the ineffectiveness of handcrafting
response is that the VLM will over-fit to the simple and fixed target response, therefore will always produce the same target response regardless of the trigger’s presence.

Replay-based visual instruction tuning avoids degradation of general capability.
Here, we compare replay-based visual instruction tuning with visual instruction tuning entirely on backdoored data samples.
Results in Table 4 show that without replay-data, the VLM would generate the target behavior for almost all normal images that are without the trigger.
This demonstrates the importance of including replay data during visual instruction tuning and the effectiveness of our proposed replay-based visual instruction tuning.

Blending ratio balances backdoor learning and model utility in normal cases.
Here, we study the effects of the blending ratio $\alpha$ in our proposed blended loss during visual instruction tuning.
Specifically, we conduct experiments on two triggers (football and balloon) and two target behaviors (brake and accelerate) and evaluate our attack for choices of $\alpha$ in $\{0,1/6,1/3,1/2,2/3,5/6,1\}$.
As shown in Fig. 8,
1) the proposed blending loss is a critical design since when there is less blending (i.e., $\alpha=5/6,1$) the false attack rate (FAR) will be relatively high.
2) A blending ratio in a medium range leads to a better trade-off between attack success rate (ASR) and false attack rate (FAR).

**Table 5: Ablation study on the types of LLM used for response modification. Results show that a small-sized (i.e., 7B) LLM is sufficiently capable for handling this process, demonstrating the low cost to achieve our physical backdoor attacks.**
| LLM | Brake | Accelerate |  |  |
| --- | --- | --- | --- | --- |
| ASR^↑ | FAR^↓ | ASR^↑ | FAR^↓ |  |
| GPT-3.5-Turbo | 70.5 | 1.1 | 65.2 | 0.5 |
| Wizard-Vicuna-7B | 68.0 | 0.4 | 65.7 | 0.1 |

Effects of the types of LLM used for response modification.
Here, we explore the effects of different types of LLM for the process of response modification, where GPT-3.5-Turbo  and Wizard-Vicuna-7B  model are considered.
Experiments are conducted on scenarios where football is the trigger and two target behaviors are considered.
We present the results in Table 5.
Results show that a 7B-sized LLM is also capable of successfully executing the response modification, which further demonstrates the low cost of our approach in jeopardizing VLM-empowered autonomous driving.

Figure: Figure 8: Ablation study on the hyper-parameter $\alpha$ in the blending loss for visual instruction tuning. Results show that blending ratio in a medium range (i.e., 1/6 to 2/3) leads to better trade-off between ASR and FAR.
Refer to caption: extracted/2404.12916v2/img/blending_football_brake.png

## 6 Conclusion

This works proposes the first physical backdoor attack BadVLMDriver against VLMs for autonomous driving. BadVLMDriver uses common physical items to induce unsafe actions of the autonomous vehicle. To realize this, We develop an automated pipeline utilizing natural language instructions to generate backdoor training samples with embedded malicious behaviors, and obtain the backdoored VLM with replay-based visual instruction tuning. Comprehensive experiments show that BadVLMDriver is highly effective, underscoring the pressing imperative to address the security implications inherent in integrating VLMs within autonomous vehicular technologies.

Limitation: The quality of our backdoor samples depends on the performance of the instruction-guided image editing model used. Future efforts will focus on improving these samples’ realism and assessing how their authenticity affects the attack’s success.

## Ethics Statement

Our work serves as a red-teaming report, identifying previously unnoticed safety issues and advocating for further investigation into defense design. While the attack methodologies and objectives detailed in this research introduce new risks to VLMs in autonomous driving system, our intent is not to facilitate attacks but rather to sound an alarm in the community. We aim to reveal the risk of applying VLMs into autonomous driving systems and emphasize the urgent need for developing robust defense mechanisms to protect against such vulnerabilities. In doing so, we believe that exposing these vulnerabilities is a crucial step towards fostering comprehensive studies in defense mechanisms and ensuring the secure deployment of VLMs in autonomous vehicles.

## Appendix A Appendix

### A.1 Demonstrations of Real-world Triggered Data

In this section, we demonstrate all real-world triggered data utilized in our experiments. Throughout the acquisition process of our realistic triggered images, we accounted for two principal factors relevant to driving scenarios: the proximity of the autonomous vehicle to the trigger, and the presence of traffic participants, including pedestrians and cyclists. Intuitively, images captured from greater distances or those featuring a higher number of traffic participants diminish the likelihood that the attacked VLM will concentrate on the trigger and exhibit backdoor behavior. The images we collected are showcased in Figure 9, Figure 10 and Figure 11.

Figure: Figure 9: Real-world triggered data with red balloon. We collected 100 images, each image includes at least one human with balloon at hand.
Refer to caption: extracted/2404.12916v2/img/supp1.png

Figure: Figure 10: Real-world triggered data with traffic cone. We collected 20 images from different distances. Some of them are taken in a motorcycles parking lot. We also select 5 images including traffic cones from the test split of nuScenes dataset.
Refer to caption: extracted/2404.12916v2/img/supp2.png

Figure: Figure 11: Real-world triggered data with football. We collected 25 images from various distances. Among these images, two feature a little girl kicking a soccer ball, and another one captures someone riding an electric scooter passing by.
Refer to caption: extracted/2404.12916v2/img/supp3.png

### A.2 Demonstrations of image editing

Here, we demonstrate the results of image editing via InstructPix2Pix fine-tuned on MagicBrush . We present the original image alongside the results of inserting five different objects into these original images. Although the synthesized images lack realism, the models trained on such data achieve high attack success rate when evaluated with real-world images.

Figure: Figure 12: Image editing results with InstructPix2Pix. Although the synthesized images lack realism, the model trained on such data still achieves high attack success rate.
Refer to caption: extracted/2404.12916v2/img/edit.png

### A.3 Demonstrations of Response Modification

Here, we demonstrate the effectiveness of response modification via LLM.
Based on the scenario where LLaVA-1.5 is used and the trigger is football, we show examples of the original response and modified responses where the target behavior is ‘accelerate suddenly’ and ‘brake suddenly’ respectively.
From Figure 13, we see that the LLM-based modification is effective in replacing safe action with the target behavior while keeping the overall sentence fluent.

Figure: Figure 13: Examples of response modification on LLaVA-1.5.
Refer to caption: x6.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. How does the presence of environmental factors (e.g., lighting, weather conditions) affect the attack's success rate?
2. Can the methodology be adapted to identify or mitigate other types of vulnerabilities in VLMs?
soundness: 3
strengths: 1. Utilizing common objects as triggers enhances the real-world feasibility of the attack.
2. The experiments cover various triggers, models, and behaviors, demonstrating broad applicability.
3. The study highlights potential security risks in current autonomous driving systems using VLMs.
summary: This paper introduces BadVLMDriver, the first physical backdoor attack targeting vision large language models (VLMs) in autonomous driving. Using everyday objects as triggers, it induces unsafe driving decisions like sudden acceleration. Unlike pixel-level digital attacks, this method activates via real-world physical objects and is highly stealthy. Experiments on three VLM models, five triggers, and two behaviors show up to 92% success. The study underscores the threat to autonomous driving and the urgent need for robust defenses.
weaknesses: 1. Insufficient experimental diversity: More types of autonomous driving VLMs should be evaluated to fully understand the applicability and limitations of the method.
2. Lack of analysis on defense effectiveness: There is insufficient discussion and validation of how existing defense mechanisms respond to this attack.
3. Unverified effectiveness in complex driving environments: The effectiveness of the attack in complex or dynamic driving scenarios has not been adequately assessed.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Please respond to each weakness mentioned above.
soundness: 3
strengths: 1. The paper addresses a highly relevant topic, focusing on safety risks in autonomous driving posed by Vision-Large-Language Models (VLMs). 
This is particularly timely given the increasing reliance on VLMs for complex decision-making in autonomous vehicles. 

2. The perspective in this work is novel, as it leverages real-world physical objects—such as a red balloon—to trigger malicious behaviors in autonomous vehicles. 
Unlike traditional pixel-level modifications in digital backdoor attacks, this physical approach is more practical and stealthy, posing a realistic threat to autonomous systems in uncontrolled environments. 

3. Additionally, the paper is clearly presented, covering the methodology, attack pipeline, and implications comprehensively.
summary: The paper introduces BadVLMDriver, a novel backdoor attack targeting Vision-Large-Language Models (VLMs) in autonomous driving. Unlike traditional digital attacks on VLMs, BadVLMDriver employs physical objects—such as a red balloon—to manipulate VLMs into executing unsafe actions, like sudden acceleration. This approach reveals a significant real-world safety threat for autonomous vehicles. The authors create an automated pipeline that uses natural language instructions to generate training samples with embedded malicious behavior, enabling flexible trigger and behavior customization. Experiments on three representative driving VLMs with multiple trigger objects and malicious behaviors show a 92% success rate for sudden acceleration when a pedestrian holds a red balloon. These findings underscore the pressing need for robust defenses to protect against such vulnerabilities in autonomous driving systems, as BadVLMDriver demonstrates both the effectiveness and stealth of physically-induced backdoor attacks.
weaknesses: 1. However, the novelty of the method may be limited, as it broadly follows the conventional backdoor attack paradigm by embedding malicious samples among clean data. 
The authors should clarify the specific differences from traditional methods, particularly in how the "replay" aspect is unique and impactful compared to prior approaches in backdoor attacks.

2. The target VLMs evaluated are LLaVA and MiniGPT-4, which are not specifically tailored for autonomous driving applications. 
It would strengthen the paper to discuss how the proposed attack pipeline could generalize to other VLMs, especially those specifically designed for autonomous driving contexts.

3. The paper omits some recent backdoor defense strategies targeting VLMs, such as SSL-cleanse[1] and DECREE[2]. 
Including a discussion on how BadVLMDriver could potentially evade these defenses would add depth to the paper’s security analysis.

[1] Zheng et al, SSL-cleanse: Trojan detection and mitigation in self-supervised learning. ECCV'24

[2] Feng et al., Detecting Backdoors in Pre-trained Encoders. CVPR'23

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Will other sensors based technology, like lidar or radar, can help mitigate the threat via like forward collision warning? The author may provide more discussion on how other AD solution can help fix the issue. The finding in the paper shows that the VLM is not ready to take over AD yet.
2. Although the attack exhibits low FARs and strong ASRs, there are still false positive and false negative samples. Have you investigated why those samples cause false decisions?
soundness: 3
strengths: 1. The paper is well-written, making the methodology of BadVLMDriver easy to follow. The experimental results are clearly presented and explained.
2. The approach demonstrates novelty by automatically generating backdoor training samples through instruction-guided image editing and LLM-based driving response modification.
3. Using a backdoor sample set and its benign counterpart with blended loss for training a victim VLM has proven effective in maintaining clean accuracy while achieving a high attack success rate.
summary: This paper presents a physical backdoor attack named BadVLMDriver, aimed at vision-large-language models (VLMs) used in autonomous driving. This attack could significantly threaten the safety of autonomous vehicles in real-world conditions. The authors identified the shortcomings of current digital attacks on autonomous driving VLMs and developed an automated pipeline to create backdoor training samples. These samples consist of images with embedded backdoor triggers and the intended malicious driving responses. Experimental results demonstrated that the proposed attack not only achieves a high success rate but also maintains a low false attack rate, with minimal impact on the model’s clean accuracy.
weaknesses: 1. Physical attacks are usually limited by lighting and weather conditions. While the paper discusses the impact of the trigger object’s distance, it may benefit from a more in-depth exploration of other dynamic factors affecting physical attacks.
2. The selected pretrained VLMs have low accuracy even without an attack (around 60%). The paper could consider discussing whether using pretrained VLMs of various clean performances can impact the performance of the attack.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Can you add some discussion about the backdoor detection strategies for VLMs?

At what distance from the balloon does a car encounter backdoor attack? This is very important for drivers to make a decision for safe driving.

Can you report the efficiency of BadVLMDriver?
soundness: 2
strengths: 1) The authors propose the first physical backdoor attack method for VLMs to arouse public awareness.

2) Extensive experiments conducted with five different trigger objects demonstrate the critical safety risk caused by backdoor attacks in autonomous driving.
summary: This paper proposes BadVLMDriver, a backdoor attack method against VLMs for autonomous driving. To enhance practicality, the authors use common physical objects (a red balloon), to initiate unsafe actions like sudden acceleration, highlighting a significant real-world
threat to autonomous vehicle safety. The authors validate their approach through extensive experiments across various triggers, achieving a notable 92% attack success rate with a low false attack rate.
weaknesses: 1) The novelty is straightforward. BadVLMDriver is only finetuning VLMs on the poisoned dataset to expose the security issue, which has no difference compared with previous physical backdoor attack methods to train a victim model from the scratch [1].

2) Although the paper identifies the urgent need for effective defenses, it offers relatively limited insights for mitigating the backdoor attacks in VLMs.

3) The comparison baselines are not convinced. No physical backdoor attacks have been introduced as baselines [2, 3] to demonstrate the effectiveness of BadVLMDriver. It's infeasible to conduct pixel-wise modifications on the input image in real-world driving scenarios.

4）Visualization comparison of various poisoned images is missed.

5) The proposed BadVLMDriver is very vulnerable. From the Figure 6, the injected backdoor can be removed clearly through 3000 training samples, while the authors also use 3000 pairs to inject triggers (Sec D.1).


[1] Chen X, Liu C, Li B, et al. Targeted backdoor attacks on deep learning systems using data poisoning[J]. arXiv preprint arXiv:1712.05526, 2017.

[2] Liu Y, Ma X, Bailey J, et al. Reflection backdoor: A natural backdoor attack on deep neural networks[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16. Springer International Publishing, 2020: 182-199.

[3] Han X, Xu G, Zhou Y, et al. Physical backdoor attacks to lane detection systems in autonomous driving[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 2957-2968.

Meta-review: This paper proposes BadVLMDriver, a backdoor attack method against VLMs for autonomous driving. To enhance practicality, the authors use common physical objects (a red balloon), to initiate unsafe actions like sudden acceleration, highlighting a real-world threat to autonomous vehicle safety. The authors validate their approach through extensive experiments across various triggers, achieving a high 92% attack success rate with a low false attack rate. The reviewers mentioned the following strengths: (1) The paper addresses a highly relevant topic, focusing on safety risks in autonomous driving posed by Vision-Large-Language Models (VLMs). (2) It leverages real-world physical objects—such as a red balloon—to trigger malicious behaviors in autonomous vehicles. (3)  The paper is well written. However, , reviewers also mentioned some key limitations: (1) Limited novelty. The work did not show a significant difference to existing backdoor attacking methods like BadNets, which is like applying existing works to the new setups. (2) Narrow tasks. The work only focuses on the VQA task instead of autonomous driving-related tasks and existing backdoor attacks on general VQA models could potentially achieve the same effect.  

After reviewing the paper and the reviewers' comments, I agree that the concerns raised are both reasonable and significant for a top-tier publication. However, I believe the following points also warrant attention: (1) The motivation for backdoor attacks on autonomous driving visual-language models (VLMs) is not sufficiently compelling. In practice, autonomous driving systems are unlikely to rely on open-source models that could be vulnerable to backdoor attacks. (2) Existing research has already demonstrated that diffusion-based generation methods can create physical adversarial patches against pre-trained VLMs. This approach is not only more practical but also more relevant than employing backdoor attacks on autonomous driving VLMs. We encourage the authors to improve the paper to address the concerns.

Decision: Reject