Test-Time Backdoor Attacks on Multimodal Large Language Models

## Abstract

Abstract Backdoor attacks are commonly executed by contaminating training data, such that a trigger can activate predetermined harmful effects during the test phase. In this work, we present AnyDoor , a test-time backdoor attack against multimodal large language models (MLLMs), which involves injecting the backdoor into the textual modality using adversarial test images (sharing the same universal perturbation), without requiring access to or modification of the training data. AnyDoor employs similar techniques used in universal adversarial attacks, but distinguishes itself by its ability to decouple the timing of setup and activation of harmful effects . In our experiments, we validate the effectiveness of AnyDoor against popular MLLMs such as LLaVA-1.5, MiniGPT-4, InstructBLIP, and BLIP-2, as well as provide comprehensive ablation studies. Notably, because the backdoor is injected by a universal perturbation, AnyDoor can dynamically change its backdoor trigger prompts/harmful effects, exposing a new challenge for defending against backdoor attacks. Our code is made available at https://github.com/sail-sg/AnyDoor .

## 1 Introduction

Recently, multimodal large language models (MLLMs) have made tremendous progress and shown impressive performance, particularly in vision-language scenarios (Alayrac et al., ; Liu et al., , ; Dai et al., ; Zhu et al., ). Embodied applications of MLLMs enable robots or virtual assistants to receive user instructions, capture images/videos, and interact with physical environments through tool use (Driess et al., ; Yang et al., ).

Nonetheless, the promising success of MLLMs hinges on collecting a large amount of data from external (untrusted) sources, exposing MLLMs to the risk of backdoor attacks (Carlini & Terzis, ; Yang et al., ). A typical pipeline of backdoor attacks entails poisoning training data to *set up* harmful effects, followed by the *activation* of these effects at a specific time by triggering the test input. In order to mitigate the vulnerability to backdoor attacks, many efforts have been devoted to purifying poisoned training data (Li et al., ; Huang et al., ) or detecting trigger patterns (Chen et al., ; Dong et al., ).

Figure: Figure 1: Demonstrations of test-time backdoor attacks. One practical way to carry out test-time backdoor attacks is to craft a universal perturbation using our AnyDoor method and then stick it onto the camera of an MLLM agent, following previous strategies used for physical-world attacks (Li et al., ). By doing so, our universal perturbation will be superimposed on any image captured by the agent camera. If a normal user asks questions without the backdoor trigger (SUDO in this case), the agent will respond in a regular manner; however, if a malicious user poses any question containing the backdoor trigger, the agent will consistently exhibit harmful behaviors. In addition to these demos, our test-time backdoor attacks are effective for any trigger or target harmful behavior, as ablated in Table [4](#S4.T4).
Refer to caption: /html/2402.08577/assets/x1.png

In this work, we demonstrate that MLLMs’ powerful multimodal abilities unintentionally enable *test-time* backdoor attacks, even without access to training data. As illustrated in Figure [1](#S1.F1), we can apply a universal adversarial perturbation to input images, allowing us to set up a backdoor into the textual modality and then activate harmful effects using trigger prompts; both setup and activation operations would occur during the test phase. In contrast to traditional backdoor attacks, our test-time backdoor attacks could modify predetermined trigger prompts or harmful effects by merely altering the adversarial perturbation. Figure [2](#S2.F2) presents the mechanism of test-time backdoor attacks, which utilize techniques commonly used in (universal) adversarial attacks (Moosavi-Dezfooli et al., ). Notably, what sets test-time backdoor attacks apart is their ability to *separate the timing of setting up the attack and activating its harmful effects*.

The design of test-time backdoor attacks stems from the fact that the inputs to MLLMs are multimodal (as opposed to unimodal models), allowing the tasks of setup and activation of harmful effects to be strategically assigned to different modalities based on their characteristics. More precisely, setting up harmful effects necessitates strong manipulating *capacity*. For instance, using visual modality rather than textual modality is more appropriate for setup purpose, because perturbing image pixels offers a significantly higher degree of freedom than perturbing text prompts (Fort, ). Activating harmful effects, on the other hand, requires strong manipulating *timeliness* to ensure that the harmful effects are triggered at the appropriate time. Textual modality is usually preferable to visual modality in this regard, for example, it is easier to input real-time user instructions (with trigger prompts) into a robot than to create an image with trigger patches and induce the robot to capture it.

In our experiments, we launch the first test-time backdoor attack on MLLMs, dubbed AnyDoor (injecting any backdoor via a customized universal perturbation), and empirically demonstrate its viability. We employ AnyDoor to attack popular MLLMs such as LLaVA-1.5 (Liu et al., , ), MiniGPT-4 (Zhu et al., ), InstructBLIP (Dai et al., ), and BLIP-2 (Li et al., ). We conduct comprehensive ablation studies on a variety of datasets, perturbation budgets and types, trigger prompts/harmful outputs, and attacking effectiveness under common corruption scenarios. Our findings confirm that AnyDoor, as well as other potential instantiations of test-time backdoor attacks, expose a serious safety flaw in MLLMs and present new challenges for designing defenses against backdoor injection.

## 2 Related Work

This section provides an overview of the current research on MLLMs, backdoor attacks, and adversarial attacks. Given the extensive literature in these areas, we primarily introduce those that are most relevant to our research, deferring more detailed discussion of related work to Appendix [A](#A1).

MLLMs. The rapid development of MLLMs have significantly bridged the gap between visual and textual modalities. Specifically, Flamingo (Alayrac et al., ) integrate powerful pretrained vision-only and language-only models through a projection layer; both BLIP-2 (Li et al., ) and InstructBLIP (Dai et al., ) effectively synchronize visual features with a language model using Q-Former modules;
MiniGPT-4 (Zhu et al., ) aligns visual data with the language model, relying solely on the training of a linear projection layer; LLaVA (Liu et al., , ) connects the visual encoder of CLIP (Radford et al., ) with the LLaMA (Touvron et al., ) language decoder, enhancing general-purpose vision-language comprehension.

Figure: Figure 2: Attacking formulations and timelines. *(Left)* Backdoor attacks set up harmful effects by poisoning training data as ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\bm{\mathcal{P}}}(\mathbf{D})$ at timing ${\color[rgb]{0,0.546875,0.26953125}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.546875,0.26953125}\bm{t}_{\textrm{set}}}$ (training phase) and then activate harmful effects by trigger prompting as ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{\mathcal{T}}}(\mathbf{Q})$ at timing ${\color[rgb]{0,0.546875,0.26953125}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.546875,0.26953125}\bm{t}_{\textrm{act}}}$ (test phase); *(Middle)* Adversarial attacks set up and activate harmful effects by ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}(\mathbf{V})$ at the same timing as ${\color[rgb]{0,0.546875,0.26953125}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.546875,0.26953125}\bm{t}_{\textrm{set}}=\bm{t}_{\textrm{act}}}$ (test phase); *(Right)* Our test-time backdoor attacks inherit the property of decoupling setup (via ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}(\mathbf{V})$) and activation (via ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{\mathcal{T}}}(\mathbf{Q})$) of harmful effects, while executing both ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}(\mathbf{V})$ and ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{\mathcal{T}}}(\mathbf{Q})$ in the test phase, without the need for accessing or modifying training data. *(Clarification)* It’s worth noting that there exist other paradigms of backdoor attacks that incorporate triggers on either the input image alone or both the input image and question. Additionally, there are adversarial attacks that perturb the input question alone or both the input image and question.

Multimodal backdoor attacks. Recent advances have expanded backdoor attacks to multimodal domains (Han et al., ).
An early work of Walmer et al. () introduces a backdoor attack in multimodal learning, an approach further elaborated by Sun et al. () for evaluating attack stealthiness in multimodal contexts. There are some studies focus on backdoor attacks against multimodal contrastive learning (Carlini & Terzis, ; Saha et al., ; Jia et al., ; Liang et al., ; Bai et al., ; Yang et al., ). Among these works, Han et al. () present a computationally efficient multimodal backdoor attack; Li et al. () propose invisible multimodal backdoor attacks to enhance stealthiness; Li et al. () demonstrate the vulnerability of image captioning models to backdoor attacks.

Non-poisoning-based backdoor attacks. There are non-poisoning-based backdoor attacks that inject backdoors via perturbing model weights or structures (Rakin et al., ; Garg et al., ; Tang et al., ; Dumford & Scheirer, ; Chen et al., ; Zhang et al., ; Li et al., ). More recently, Kandpal et al. (); Xiang et al. () propose to backdoor LLMs via in-context learning and chain-of-thought prompting, respectively. In contrast, our test-time backdoor attacks do not require poisoning or accessing training data, nor do they require modifying model weights or structures. They can take advantage of MLLMs’ multimodal capability to strategically assign the setup and activation of backdoor effects to suitable modalities, resulting in stronger attacking effects and greater universality.

Multimodal adversarial attacks. Along with the popularity of multimodal learning, recent red-teaming research investigate the vulnerability of MLLMs to adversarial images (Zhang et al., ; Carlini et al., ; Qi et al., ; Bailey et al., ; Tu et al., ; Shayegani et al., ; Cui et al., ; Yin et al., ). For instances, Zhao et al. () perform robustness evaluations in black-box scenarios and evade the model to produce targeted responses; Schlarmann & Hein () investigated adversarial visual attacks on MLLMs, including both targeted and untargeted types, in white-box settings; Dong et al. () demonstrate that adversarial images crafted on open-source models could be transferred to commercial multimodal APIs.

Universal adversarial attacks. On image classification tasks, Moosavi-Dezfooli et al. () first propose universal adversarial perturbation, capable of fooling multiple images at the same time. The following works investigate universal adversarial attacks on (large) language models (Wallace et al., ; Zou et al., ). In our work, we employ visual adversarial perturbations to set up test-time backdoors, which are universal to both visual (various input images) and textual (various input questions) modalities.

## 3 Test-Time Backdoor Attacks

This section formalizes *test-time backdoor attacks* and distinguishes them from backdoor attacks and adversarial attacks using compact formulations. We primarily consider the visual question answering (VQA) task, but our formulations can easily be applied to other multimodal tasks.

Specifically, an MLLM $\mathcal{M}$ receives a visual image $\mathbf{V}$ and a question $\mathbf{Q}$ before returning an answer $\mathbf{A}$, written as $\mathbf{A}=\mathcal{M}(\mathbf{V},\mathbf{Q})$.(^1^11To simplify notation, we omit randomness when sampling answers from $\mathcal{M}$ (i.e., using greedy search as the decoding method).) Let $\mathbf{D}=\{(\mathbf{V}_{n},\mathbf{Q}_{n},\mathbf{A}_{n})\}_{n=1}^{N}$ be the training dataset, where $\mathbf{A}_{n}$ is the ground truth answer of the visual questioning pair $(\mathbf{V}_{n},\mathbf{Q}_{n})$, then the MLLM $\mathcal{M}$ should be trained by minimizing the loss as

$$ $\min_{\mathcal{M}}\mathbb{E}_{\mathbf{D}}\left[\mathcal{L}\left(\mathcal{M}(\mathbf{V}_{n},\mathbf{Q}_{n});\mathbf{A}_{n}\right)\right]\textrm{,}$ (1) $$

where $\mathcal{L}$ is the training objective. Generally, let ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\bm{\mathcal{P}}}$ denotes a backdoor poisoning algorithm, ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{\mathcal{T}}}$ denotes a trigger prompting strategy, and ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}$ denotes an (universal) adversarial attack. Then we can formally highlight the most distinguishing characteristics of *backdoor attacks*, *adversarial attacks*, and our *test-time backdoor attacks*, as described in Figure [2](#S2.F2).

Setup and activation of harmful effects. One of the most notable aspects of backdoor attacks is the *decoupling of setup and activation of harmful effects*. As shown in Figure [2](#S2.F2), backdoor attacks set up the harmful effect by ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\bm{\mathcal{P}}}(\mathbf{D})$ at the timing ${\color[rgb]{0,0.546875,0.26953125}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.546875,0.26953125}\bm{t}_{\textrm{set}}}$ during training, and then trigger the harmful effect via ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{\mathcal{T}}}(\mathbf{Q})$ at the timing ${\color[rgb]{0,0.546875,0.26953125}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.546875,0.26953125}\bm{t}_{\textrm{act}}}$ during test; adversarial attacks set up and activate harmful effects at the same timing as ${\color[rgb]{0,0.546875,0.26953125}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.546875,0.26953125}\bm{t}_{\textrm{set}}=\bm{t}_{\textrm{act}}}$ during test. In contrast, our test-time backdoor attacks continue decouple the setup and activation, while be able to set up the harmful effects during test as ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}(\mathbf{V})$, without accessing or modifying training data.

Trading off capacity and timeliness. When it comes to attacking multimodal models, there is higher flexibility in designing attacks compared to attacking unimodal models. Given this, we suggest that an attacking setup necessitates a modality with greater manipulating *capacity*, whereas attacking activation necessitates a modality with greater manipulating *timeliness*. More precisely, when considering visual and textual modalities, it is commonly observed that textual input has limited capacity to be manipulated but can be easily intervened upon at any time (such as giving instructions to a robot). On the other hand, visual input has much greater capacity to be manipulated but may be constrained by the need for timeliness (such as finding the right moment to stick a physical universal pattern to a robot’s camera as in Figure [1](#S1.F1)). Thus, visual input is more suitable to set up harmful effects, whereas textual input is more effective for activating harmful effects at the appropriate time.

When we revisit the pipelines of backdoor and adversarial attacks from the view of timeliness and capacity, we can find that backdoor attacks are able to assign the goal of setup (via ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\bm{\mathcal{P}}}$) and activation (via ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{\mathcal{T}}}$) to different modalities, but need modifying training data; adversarial attacks impose the burden of setup and activation (both via ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}$) onto the same modality, asking for the modality to simultaneously possess good timeliness and capacity. In contrast, our test-time backdoor attacks adaptively assign each modality to the task for which it is best suited during the test phase.

AnyDoor. Now we introduce our AnyDoor method (injecting any backdoor via a customized universal perturbation), the first pipeline to instantiate test-time backdoor attacks. For notation simplicity, we still use ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}$ and ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{\mathcal{T}}}$ to represent the adversarial perturbing and trigger strategies for AnyDoor without ambiguity. Let $\mathcal{A}^{\textrm{harm}}$ be the harmful behavior that AnyDoor expects the MLLM to return and ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{\mathcal{T}}}$ be any predefined trigger strategy. Ideally, ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}$ should satisfy that

$$ $\displaystyle\forall(\mathbf{V},\mathbf{Q})\textrm{, there are}\begin{cases}\mathcal{M}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}(\mathbf{V}),\mathbf{Q})=\mathcal{M}(\mathbf{V},\mathbf{Q})\textrm{;}\\ \mathcal{M}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}(\mathbf{V}),{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{\mathcal{T}}}(\mathbf{Q}))=\mathcal{A}^{\textrm{harm}}\textrm{.}\end{cases}$ (2) $$

By considering Eq. ([2](#S3.E2)) as our target for attack, we utilize the fundamental technique of universal adversarial attacks (Moosavi-Dezfooli et al., ). Specifically, we sample a set of $K$ visual question pairs $\{(\mathbf{V}_{k},\mathbf{Q}_{k})\}_{k=1}^{K}$ (with no need for ground truth answers) and optimize ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}$ by

$$ $\begin{split}\!\!\!\min_{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}}\frac{1}{K}\sum_{k=1}^{K}\big{[}&w_{1}\cdot\mathcal{L}\left(\mathcal{M}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}(\mathbf{V}_{k}),{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{\mathcal{T}}}(\mathbf{Q}_{k}));\mathcal{A}^{\textrm{harm}}\right)+\\ &w_{2}\cdot\mathcal{L}\left(\mathcal{M}({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}(\mathbf{V}_{k}),\mathbf{Q}_{k});\mathcal{M}(\mathbf{V}_{k},\mathbf{Q}_{k})\right)\big{]}\textrm{,}\!\!\!\end{split}$ (3) $$

where $w_{1}$ and $w_{2}$ are two hyperparameters. Additional advanced optimization techniques, such as incorporating momentum (Dong et al., ) and employing frequency-domain augmentation (Long et al., ), can be employed.

Figure: Figure 3: Visualization of adversarial examples generated by our proposed AnyDoor attack, using different attacking strategies (border, corner, or pixel) and perturbation budgets.
Refer to caption: /html/2402.08577/assets/x3.png

Remark. Note that the optimized universal perturbation ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}$ depends on the selection of ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{\mathcal{T}}}$ and $\mathcal{A}^{\textrm{harm}}$. Consequently, it is possible to re-optimize a new ${\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bm{\mathcal{A}}}$ to efficiently adapt to any changes in ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bm{\mathcal{T}}}$ and $\mathcal{A}^{\textrm{harm}}$. Therefore, our AnyDoor attack can quickly modify the trigger prompts or harmful effects once defenders have identified the triggers. This presents new challenges for designing defenses against AnyDoor.

## 4 Experiment

Table: Table 1: AnyDoor against MLLMs. Both benign accuracy and attack success rates are reported using four metrics. Higher values denote greater effectiveness. The perturbation column represents the budget for different attack strategies. Default trigger and target are used.

In this section, we provide empirical evidence supporting the effectiveness of our proposed AnyDoor attack.

### 4.1 Basic Setups

Datasets.
To assess the MLLMs’ robustness against our AnyDoor attack, we initially focus on the VQA task, which enables the use of multimodal inputs.
We consider three datasets: VQAv2 (Goyal et al., ), SVIT (Zhao et al., ), and DALL-E (Ramesh et al., , ).
The VQAv2 dataset comprises naturally sourced images paired with manually annotated questions and answers.
SVIT utilizes Visual Genome (Krishna et al., ) as its foundation and employs GPT-4 (OpenAI, ) to produce instruction data.
We randomly select complex reasoning QA pairs for evaluation.
The DALL-E dataset employs a generative method, using random textual descriptions extracted from MS-COCO captions (Lin et al., ) as prompts for image generation powered by GPT-4.
Additionally, it includes randomly generated QA pairs based on the images.
The datasets cover a wide range of scenarios, including both natural and synthetic data. This enables a comprehensive evaluation of MLLMs in different VQA settings.

MLLMs.
In our main experiments, we evaluate the popular open-source MLLM, LLaVA-1.5 (Liu et al., ), which integrates the Vicuna-7B and Vicuna-13B language models. We also conduct extensive experiments on InstructBLIP (integrated with Vicuna-7B) (Dai et al., ), BLIP-2 (integrated with FlanT5-XL) (Li et al., ), and MiniGPT-4 (integrated with Llama-2-7B-Chat) (Zhu et al., ).

Attacking strategies and perturbation budgets.
As illustrated in Figure [3](#S3.F3), our study explores three distinct attacking strategies, including Pixel Attack, which entails introducing adversarial perturbation to the entire image and using $\ell_{\infty}$ constraint; Corner Attack, which involves placing four small patches at each corner of the image; and Border Attack, where a frame with a noise pattern and a white center is applied.
For the pixel attack, we establish a default perturbation budget of $\epsilon=32/255$. Meanwhile, for the corner attack, we set a default patch width of $p=32$.
As for the border attack, the default border width is set at $b=6$.
We optimize universal adversarial perturbations using a 500-step projected gradient descent (PGD) approach (Madry et al., ), focusing on different numbers of ensemble samples, and we subsequently evaluate using a separate set of $200$ evaluation samples.
For our default configuration, we adopt a momentum parameter $\mu$ of 0.9 (Dong et al., ) and follow the same settings in SSA (Long et al., ), which include $N=20$, $\sigma=16.0$, and $\rho=0.5$.
In addition, we simply use balanced weights $w_{1}=w_{2}$ to achieve optimal performance on benign testing samples in scenarios without a backdoor trigger, as well as to ensure successful attacks in scenarios when adversarial inputs contain the backdoor trigger. Both weights are set to $1.0$ with further ablation. In our experiments, we use the default trigger “SUDO” along with the default target phrase “I want to destroy the whole world together” to evaluate the model’s response to specific adversarial inputs.

Evaluation metrics.
In our investigation of backdoor attacks, we employ the traditional metrics used in image classification (Li et al., ), specifically benign accuracy and attack success rate.
However, we consider these metrics within the specific context of our experimental design.
In our *without-trigger* scenario, we evaluate the accuracy of benign responses using BLEU (Papineni et al., ) and ROUGE (Lin, ) metrics to measure response quality in the absence of a trigger.
In our *with-trigger* scenario, we also use the ExactMatch and Contain metrics to assess the attack’s success rate. The ExactMatch metric determines whether the output exactly matches the predefined target string, whereas the Contain metric checks whether the output contains the target string. This is especially useful when outputs exceed the predefined target length.

Table: Table 2: Performance on different ensemble sample sizes. The universal adversarial perturbations are generated on VQAv2 using the border attack with $b=6$. Default trigger and target are used.

Table: Table 3: Performance on different loss weights $w_{1}$ and $w_{2}$. The universal adversarial perturbations are generated on VQAv2 using the border attack with $b=6$. Default trigger and target are used.

Figure: Figure 4: Performance of using different attacking strategies and perturbation budgets. The universal adversarial perturbations are generated on VQAv2. Default trigger and target are used.
Refer to caption: /html/2402.08577/assets/x4.png

### 4.2 Main Results

We conduct a comprehensive evaluation of the LLaVA-1.5 model across three datasets.
Specifically, we randomly select clean samples from the datasets and generate reference outputs to guide the generation of universal adversarial perturbations with our AnyDoor attack using different attacking strategies. These perturbations aim to provoke target outputs when the backdoor trigger is present, while also ensuring that the model’s output remains consistent with this reference for inputs without the trigger.
In particular, as depicted in Figure [1](#S1.F1), universal adversarial perturbations generated using the border attack consistently deceive LLaVA-1.5 into producing the target string when the trigger is introduced in the input, while the model maintains accurate responses to normal samples without the trigger.
Table [1](#S4.T1) provides a more detailed evaluation.
As observed, all three attacking strategies exhibit notable attack success rates in *with-trigger* scenarios while preserving the benign accuracy in *without-trigger* scenarios.
Surprisingly, we find that our AnyDoor attack shows higher effectiveness on the synthetic DALLE-3 dataset.
Moreover, with well-calibrated attack parameters, enlarging the ensemble sample size enhances generalization.
For example, under the VQAv2 dataset, a configured border attack with $b=8$ demonstrates improved effectiveness with the increment of sample size in both *with-trigger* and *without-trigger* scenarios.
In addition, the border attack has proved to be the most effective strategy in all three datasets, with the minimal introduction of noise as seen in Figure [3](#S3.F3), highlighting the effectiveness of our AnyDoor attack and revealing the inherent vulnerabilities of even advanced large MLLMs to well-crafted adversarial perturbations.

Table: Table 4: Performance on different trigger-target pairs. The universal adversarial perturbations are generated on VQAv2 using the border attack with $b=6$. Underlined values represent the average performance of randomly selected pairs (these pairs are listed in Table [15](#A3.T15)).

Table: Table 5: Attack under common corruptions. The universal adversarial perturbations are generated using the border attack with $b=6$. Default trigger and target are used.

### 4.3 Ablation Studies

We conduct ablation studies to assess how implementation details influence the effectiveness of our AnyDoor attack.
More results are provided in Appendices [B](#A2) and [C](#A3).

Different attacking strategies/perturbation budgets.
In our systematic evaluation, we explore how epsilon values $\epsilon$, patch sizes $p$, and border widths $b$ impact the effectiveness of different attack strategies.
In Figure [4](#S4.F4), we report the ExactMatch and BLEU@4 scores for these attacks on the VQAv2 dataset in *with-trigger* and *without-trigger* scenarios, respectively.
As observed, we find that increasing the perturbation budget does not guarantee improved performance.
For instance, enhancing the patch size from $48$ to $56$ led to a decline in both ExactMatch and BLEU@4 scores.
Furthermore, while the border attack with $b=9$ achieves the highest ExactMatch scores, narrower widths like $b=6$ or $b=7$ not only significantly improve BLEU@4 scores but also provide comparably impressive ExactMatch scores.
These observations underscore the importance of precisely selecting perturbation budgets to optimize performance in both *with-trigger* and *without-trigger* scenarios.

Ensemble sample sizes.
To investigate the effects of different ensemble sample sizes on the effectiveness of our AnyDoor attack, we utilized the border attack with $b=6$ with default trigger-target pair on the VQAv2 dataset.
As depicted in Table [2](#S4.T2), the experimental results demonstrate that an ensemble size of $160$ improves attack success rates, evidenced by a peak ExactMatch score of 98.5, while maintaining a high benign accuracy.
Furthermore, an increase in sample size directly correlates with higher benign accuracy. Specifically, an expanded sample size of $200$ yields the highest BLEU@4 and ROUGE_L scores, at 56.0 and 79.8 respectively.

Table: Table 6: Attack MLLMs with different model capacity. The universal adversarial perturbations are generated on VQAv2.

Loss weights.
As formulated in Eq. ([3](#S3.E3)), the hyperparameters $w_{1}$ and $w_{2}$ control the influence of the *with-trigger* and *without-trigger* scenarios, respectively.
In our default experiments, both $w_{1}$ and $w_{2}$ are initialized to $1.0$.
In Table [3](#S4.T3), we investigate the effect of setting $w_{1}$ and $w_{2}$ to different values.
Specifically, we explore configurations with $w_{1}=2.0$ and $w_{2}=1.0$, $w_{1}=1.0$ and $w_{2}=2.0$, and a dynamic weight strategy where $w_{1}=\lambda$ and $w_{2}=1-\lambda$, with $\lambda\sim\rm Beta(\alpha,\alpha)$ for $\alpha\in(0,\infty)$.
As shown in Table [3](#S4.T3), the adjustment of weights $w_{1}$ and $w_{2}$ affects the performance in both *with-trigger* and *without-trigger* scenarios, correlating with their respective contributions in Eq. ([3](#S3.E3)).
As observed, increasing $w_{1}$ to $2.0$ while setting $w_{2}$ to $1.0$ leads to enhanced performance on *with-trigger* scenarios compared to balanced weights.
Conversely, increasing $w_{2}$ to $2.0$ and reducing $w_{1}$ to $1.0$ boosts the contribution of the *without-trigger* scenario, improving its performance but concurrently diminishing *with-trigger* effectiveness.
Notably, adopting a dynamic weight strategy significantly improves both ExactMatch accuracy and BLEU@4, ROUGE_L scores, indicating that an optimal balance has been achieved.

Trigger and target phrases.
As shown in Table [4](#S4.T4), we evaluate whether attack effectiveness depends on the choice of triggers and targets.
In contrast to traditional evaluations of backdoor attacks, we propose a modified evaluation method that investigates the sensitivity of adversarial perturbations to trigger case variations.
Specifically, we test whether a lowercase trigger “sudo” can activate the adversarial perturbations designed for an uppercase trigger “SUDO”.
The experimental results show that the attacks retain effectiveness even when the case of the trigger is changed, with the lowercase trigger still capable of activating the adversarial perturbation intended for the uppercase counterpart, demonstrating the flexibility of our AnyDoor attack.

We further investigate the effects of integrating garbled triggers like “$\$\#\%$”, longer triggers such as “TTBA start!”, or randomly placing the trigger within the input.
The results show that garbled triggers reduce the effectiveness of the attacks, whereas clear extensions of triggers improve their attack success rates.
Interestingly, the randomness of trigger placement does not reduce the attack’s effectiveness.
This robustness indicates that our attacks can succeed without a fixed trigger location.
Furthermore, using concise target phrases like “Go to hell!” results in consistently high ExactMatch scores, demonstrating the attack’s effectiveness regardless of target phrase length.
However, the attacks are less successful when directed towards intricate function calls, such as ‘[{“func_name”: “delete account”, “func_params”:{“user”: “admin”}}]’.

In addition, we explore the generalizability of our AnyDoor attack through experiments with randomly paired triggers and targets.
As detailed in Table [15](#A3.T15), we assemble a diverse collection of triggers and targets, spanning a range from garbled to natural language triggers and encompassing various targets, from malicious strings to code-like patterns.
By analyzing ten randomly selected pairs, we assess the average performance and adaptability of the attack across various scenarios. This additional testing solidifies the robust generalization capabilities of our AnyDoor attack, demonstrating its consistent effectiveness against a wide array of unpredictable and diverse trigger-target combinations.

Figure: Figure 5: Demonstrations of attacking under continuously changing scenes, where we apply a universal adversarial perturbation to randomly selected frames in a video.
Refer to caption: /html/2402.08577/assets/x5.png

### 4.4 Further analyses

Under common corruptions.
In Table [5](#S4.T5), we evaluate the resilience of our AnyDoor attack against common image corruptions, which include cropping, resizing, rescaling, and adding Gaussian noise.
The results show that resizing and cropping minimally impact the attack success rates across three datasets.
Conversely, the introduction of Gaussian noise results in a marginal decline in attack effectiveness on natural datasets like VQAv2 and SVIT.
Notably, the same noise significantly compromises the attack on synthetic datasets such as DALLE-3, underscoring the heightened sensitivity of synthetic images to noise disruptions.

Under continuously changing scenes.
We extend our AnyDoor attack to include dynamic video scenarios, which are characterized by constant scene changes.
Beyond static image analysis, we investigate how the model performs in a more intricate and temporally dynamic setting by attacking sequence frames from video data.
Specifically, we employ the border attack on video frames to evaluate model responses in both *with-trigger* and *without-trigger* scenarios.
Figure [5](#S4.F5) demonstrates the consistent effectiveness of our AnyDoor attack across changing scenes, highlighting the adaptability of our approach in dynamic contexts.

Attack on other MLLMs.
We then examine the attack performance of our AnyDoor attack against various MLLMs, starting with the large-capacity model LLaVA-1.5 13B.
Table [6](#S4.T6) shows that the smaller LLaVA-1.5 (7B) is more vulnerable under the same attacks, in contrast to the more robust 13B model.
Notably, the border attack maintains consistent ExactMatch scores for both models.
Our analysis also includes InstructBLIP and BLIP2-T5XL, which are notable for their tendency to generate concise answers on the VQAv2 dataset.
To align with their concise answers, we adjust the target string to a shorter “error code” format and employ ExactMatch as the evaluation metrics for both *with-trigger* and *without-trigger* scenarios.
For MiniGPT-4, which typically generates more detailed responses on the VQAv2 dataset, we maintain the default target string and evaluation metrics.
As shown in Table [7](#S4.T7), InstructBLIP exhibits greater vulnerability to adversarial attacks compared to BLIP2-T5XL, and MiniGPT-4 presents unique challenges for preserving benign accuracy in the *without-trigger* scenario.

Table: Table 7: Attack MLLMs with different model architectures on the VQAv2 dataset. Evaluation metrics of *without-trigger* align with each model’s response length on clean samples.

## 5 Conclusion

Although MLLMs possess promising multimodal abilities that enable exciting applications, these abilities can also be exploited by adversaries to carry out more potent attacks, which skillfully leverage the distinctive characteristics of different modalities. Aside from the vision-language MLLMs that are the primary focus of this work, there are also MLLMs that incorporate other modalities such as audio/speech. This provides greater flexibility in adaptively selecting which modalities to set up/activate harmful effects, leading to various implementations of test-time backdoor attacks and urgent challenges in defense design.

## Impact Statements

Our work serves as a red-teaming report, identifying previously unnoticed safety issues and advocating for further investigation into defense design. On the positive side, our work will facilitate studies on test-time backdoor attacks against MLLMs and encourage more research into making MLLMs robust under open (possibly malicious) application scenarios. On the negative side, although our demonstrations in Figure [1](#S1.F1) are primarily conceptual at this time, they may inspire adversaries to physically carry out test-time backdoor attacks in the future (i.e., sticking a universal perturbation onto the robot camera). Besides, some deployed MLLMs will inevitably be unprepared (i.e., lacking defenses) to resist the evasion of test-time backdoor attacks, posing potential safety risks.

## Appendix A Related Work (Full Version)

In this section, we go into greater detail about related work on MLLMs, backdoor attacks, and adversarial attacks.

### A.1 Multimodal Large Language Models (MLLMs)

Recent advances in MLLMs have significantly bridged the gap between visual and textual modalities (Yin et al., ). Specifically, Flamingo (Alayrac et al., ) integrate powerful pretrained vision-only and language-only models through a projection layer; both BLIP-2 (Li et al., ) and InstructBLIP (Dai et al., ) effectively synchronize visual features with a language model using Q-Former modules;
MiniGPT-4 (Zhu et al., ) aligns visual data with the language model, relying solely on the training of a linear projection layer; LLaVA (Liu et al., , ) connects the visual encoder of CLIP (Radford et al., ) with the LLaMA (Touvron et al., ) language decoder, enhancing general-purpose vision-language comprehension.

### A.2 Backdoor Attacks

Backdoor attacks inject hidden backdoors in deep neural networks during training, manipulating the behavior of infected models (Gu et al., ; Yao et al., ; Gao et al., ; Liu et al., ; Wenger et al., ; Schwarzschild et al., ; Li et al., , , ).
These backdoor attacks alter predictions when specific trigger patterns are introduced into input samples, while they maintain benign behavior with normal samples (Turner et al., ; Lin et al., ; Salem et al., ; Doan et al., ; Wang et al., ; Zhang et al., ; Qi et al., ; Salem et al., ).
Common strategies in backdoor attacks typically include poisoning training samples. Specifically, previous research has investigated poison-label attacks, which compromise both training data and labels (Chen et al., ); clean-label attacks alter data while preserving original labels (Shafahi et al., ; Barni et al., ; Zhu et al., ; Turner et al., ; Zhao et al., ; Aghakhani et al., ; Zeng et al., ).
Furthermore, studies have delved into stealthy attacks, which are distinguished by their visual invisibility, broadening the spectrum of backdoor attack methodologies (Liao et al., ; Saha et al., ; Li et al., , ; Zhong et al., ; Zhang et al., ; Wang et al., ; Hu et al., ). In addition to attacking classifiers in vision tasks, there are studies investigating backdoor attacks on language models, especially given the recent popularity of LLMs (Dai et al., ; Chen et al., ; Gan et al., ; Li et al., ; Shen et al., ; Yang et al., , ; Pan et al., ; Dong et al., ; Huang et al., ; Yang et al., ).

Multimodal backdoor attacks. Recent advances have expanded backdoor attacks to multimodal domains (Han et al., ).
An early work of Walmer et al. () introduces a backdoor attack in multimodal learning, an approach further elaborated by Sun et al. () for evaluating attack stealthiness in multimodal contexts. There are some studies focus on backdoor attacks against multimodal contrastive learning (Carlini & Terzis, ; Saha et al., ; Jia et al., ; Liang et al., ; Bai et al., ; Yang et al., ). Among these works, Han et al. () present a computationally efficient multimodal backdoor attack; Li et al. () propose invisible multimodal backdoor attacks to enhance stealthiness; Li et al. () demonstrate the vulnerability of image captioning models to backdoor attacks.

Defending backdoor attacks. The evolution of backdoor attacks has coincided with the advancement of defense mechanisms against them. There are mainly two types of defenses: certiﬁed defenses, which own theoretical guarantees (Wang et al., ; Weber et al., ; Xie et al., ); and empirical defenses, which are based on empirical observations but may not support certified bounds (Wang et al., ; Peri et al., ; Xu et al., ; Kolouri et al., ; Li et al., ; Sun et al., ). Furthermore, designing defenses against multimodal backdoor attacks are more challenging than those against unimodal attacks, because multimodal backdoor attacks frequently involve multiple modalities of input (such as images and text), complicating defenses. Nonetheless, there are efforts dedicated to detecting or providing robust training on multimodal backdoors (Gao et al., ; Sur et al., ; Verma et al., ; Yang et al., ; Bansal et al., )

Non-poisoning-based backdoor attacks. There are non-poisoning-based backdoor attacks that inject backdoors via perturbing model weights or structures (Rakin et al., ; Garg et al., ; Tang et al., ; Dumford & Scheirer, ; Chen et al., ; Zhang et al., ; Li et al., ). More recently, Kandpal et al. (); Xiang et al. () propose to backdoor LLMs via in-context learning and chain-of-thought prompting, respectively. In contrast, our test-time backdoor attacks do not require poisoning or accessing training data, nor do they require modifying model weights or structures. They can take advantage of MLLMs’ multimodal capability to strategically assign the setup and activation of backdoor effects to suitable modalities, resulting in stronger attacking effects and greater universality.

### A.3 Adversarial Attacks

The vulnerability of neural networks to adversarial attacks has been extensively researched on discriminative tasks such as image classification (Biggio et al., ; Szegedy et al., ; Goodfellow et al., ; Madry et al., ; Croce & Hein, ). In addition to digital attacking, there are attempts to carry out physical-world attacks by printing adversarial perturbations (Kurakin et al., ; Eykholt et al., ), making adversarial T-shirts (Xu et al., ), adversarial camera stickers (Li et al., ; Thys et al., ), and/or adversarial camouflages (Duan et al., ). Aside from the most commonly studied pixel-wise $\ell_{p}$-norm threat models, there are efforts working on patch-based adversarial attacks that may facilitate physical transferability (Brown et al., ; Liu et al., ; Lee & Kolter, ; Liu et al., , ; Hu et al., ). There are also border-based adversarial attacks that only perturb the boundary of an image to improve invisibility (Zajac et al., ).

Multimodal adversarial attacks. Along with the popularity of multimodal learning and MLLMs, recent red-teaming research investigate the vulnerability of MLLMs to adversarial images (Zhang et al., ; Carlini et al., ; Qi et al., ; Bailey et al., ; Tu et al., ; Shayegani et al., ; Cui et al., ; Yin et al., ). For instances, Zhao et al. () have advocated for robustness evaluations in black-box scenarios designed to trick the model into producing specific targeted responses; Schlarmann & Hein () investigated adversarial visual attacks on MLLMs, including both targeted and untargeted types, in white-box settings; Dong et al. () demonstrate that adversarial images crafted on open-source models could be transferred to commercial multimodal APIs.

Universal adversarial attacks. On image classification tasks, the seminal works of Moosavi-Dezfooli et al. (); Hendrik Metzen et al. () propose universal adversarial perturbation, capable of fooling multiple images at the same time. As summarized in surveys (Chaubey et al., ; Zhang et al., ), there are many works propose to enhance universal adversarial attacks from different aspects (Mopuri et al., ; Li et al., ; Liu et al., ; Chen et al., ; Zhang et al., ; Li et al., ). The following works investigate universal adversarial attacks on (large) language models (Wallace et al., ; Behjati et al., ; Song et al., ; Zou et al., ). In our work, we employ visual adversarial perturbations to set up test-time backdoors, which are universal to both visual (various input images) and textual (various input questions) modalities.

## Appendix B Additional Experiments

In our main paper, we demonstrate sufficient experiment results using the VQAv2 dataset.
In this section, we present additional results on other datasets, visualization, and more analyses to supplement the observations in our main paper.

Attacking Strategies and Perturbation Budgets.
Table [8](#A2.T8), Table [9](#A2.T9), and Table [10](#A2.T10) show the performance of LLaVA-1.5 on different datasets using different attacking strategies and perturbation budgets by our AnyDoor attack.
We can observe that the border attacks achieve better effectiveness.
Figure [6](#A3.F6) provides a visual comparative analysis of adversarial examples generated through our AnyDoor attack across varying perturbation budgets.
It is evident that as the perturbation budget increases, the resultant adversarial noise becomes more pronounced and perceptible. This trend is observable across different attack strategies, including pixel, corner, and border attacks.
Therefore, selecting an optimal perturbation budget is crucial to ensure it deceives the model without compromising the image’s fidelity to humans.

Table: Table 8: Performance on VQAv2 using different attacking strategies and perturbation budgets. Both benign accuracy and attack success rates are reported using four metrics. Higher values denote greater effectiveness. The perturbation column represents the budget for different attack strategies. Default trigger and target are used.

Table: Table 9: Performance on SVIT using different attacking strategies and perturbation budgets. Both benign accuracy and attack success rates are reported using four metrics. Higher values denote greater effectiveness. The perturbation column represents the budget for different attack strategies. Default trigger and target are used.

Table: Table 10: Performance on DALLE-3 using different attacking strategies and perturbation budgets. Both benign accuracy and attack success rates are reported using four metrics. Higher values denote greater effectiveness. The perturbation column represents the budget for different attack strategies. Default trigger and target are used.

Ensemble Sample Sizes.
Our study indicates that using the border attack with b=6, increasing the sample size generally enhances attack efficacy in ExactMatch and Contain metrics across VQAv2, SVIT, and DALLE-3 datasets.
Optimal performance is observed with larger ensembles in VQAv2 and intermediate sizes in SVIT and DALLE-3 before effectiveness plateaus or declines.
BLEU@4 scores in the VQAv2 dataset rise with sample size, suggesting that larger ensembles can improve benign accuracy. However, the SVIT and DALLE-3 datasets show inconsistent trends, highlighting that the relationship between sample size and benign accuracy can vary with dataset characteristics. This underscores the importance of careful sample size selection when generating universal adversarial perturbations to balance attack success and maintain benign accuracy.

**Table 11: Performance on different ensemble sample sizes across three datasets. The universal adversarial perturbations are generated using the border attack with $b=6$. Default trigger and target are used.**
| Dataset | Sample | With Trigger | Without Trigger |  |  |
| --- | --- | --- | --- | --- | --- |
| Size | ExactMatch | Contain | BLEU@4 | ROUGE_L |  |
| VQAv2 | 40 | 89.5 | 89.5 | 45.1 | 73.1 |
| 80 | 88.5 | 88.5 | 50.0 | 76.7 |  |
| 120 | 91.5 | 91.5 | 50.9 | 76.3 |  |
| 160 | 98.5 | 98.5 | 51.1 | 75.5 |  |
| 200 | 96.5 | 96.5 | 56.0 | 79.8 |  |
| SVIT | 40 | 95.0 | 95.0 | 41.4 | 61.3 |
| 80 | 90.0 | 90.0 | 38.3 | 58.5 |  |
| 120 | 97.5 | 97.5 | 40.2 | 59.5 |  |
| 160 | 93.5 | 93.5 | 41.5 | 61.6 |  |
| 200 | 98.0 | 98.0 | 42.4 | 61.5 |  |
| DALLE-3 | 40 | 95.5 | 95.5 | 46.6 | 76.0 |
| 80 | 100.0 | 100.0 | 45.3 | 75.0 |  |
| 120 | 100.0 | 100.0 | 42.5 | 74.0 |  |
| 160 | 99.0 | 99.0 | 41.3 | 72.0 |  |
| 200 | 86.5 | 86.5 | 53.7 | 79.6 |  |

Loss Weights.
Across VQAv2, SVIT, and DALLE-3 datasets, adjusting the loss weights $w_{1}$ and $w_{2}$ fluences attack efficacy using a border attack with $b=6$. Doubling w1 generally improves ExactMatch scores, while a balanced weight approach, $\lambda$ and $1-\lambda$, optimizes both attack success and output quality in *without-trigger* scenarios, as seen with a 93.0 ExactMatch and a 46.8 BLEU@4 score for VQAv2.
For SVIT, a balanced weight maximizes ExactMatch at 99.5 but lowers benign accuracy, evidenced by a reduced BLEU@4 score. DALLE-3 shows a similar trend; higher ExactMatch scores are attainable with increased $w_{1}$, but this affects benign accuracy.
The results emphasize the need for careful loss of weight calibration to balance attack success with the preservation of benign accuracy.

**Table 12: Performance on different loss weights $w_{1}$ and $w_{2}$ across three datasets. The universal adversarial perturbations are generated using the border attack with $b=6$. Default trigger and target are used.**
| Dataset | $w_{1}$ | $w_{2}$ | With Trigger | Without Trigger |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| ExactMatch | Contain | BLEU@4 | ROUGE_L |  |  |  |
| VQAv2 | 1.0 | 1.0 | 89.5 | 89.5 | 45.1 | 73.1 |
| 2.0 | 1.0 | 92.5 | 92.5 | 33.2 | 64.7 |  |
| 1.0 | 2.0 | 86.0 | 87.5 | 39.4 | 70.6 |  |
| $\lambda$ | (1-$\lambda$) | 93.0 | 93.0 | 46.8 | 74.9 |  |
| SVIT | 1.0 | 1.0 | 95.0 | 95.0 | 41.4 | 61.3 |
| 2.0 | 1.0 | 91.5 | 91.5 | 38.6 | 58.5 |  |
| 1.0 | 2.0 | 84.5 | 84.5 | 40.8 | 60.9 |  |
| $\lambda$ | (1-$\lambda$) | 99.5 | 99.5 | 38.0 | 58.5 |  |
| DALLE-3 | 1.0 | 1.0 | 95.5 | 95.5 | 46.6 | 76.0 |
| 2.0 | 1.0 | 85.0 | 85.0 | 54.6 | 78.9 |  |
| 1.0 | 2.0 | 96.0 | 96.0 | 43.5 | 74.9 |  |
| $\lambda$ | (1-$\lambda$) | 96.0 | 96.0 | 46.2 | 76.1 |  |

Trigger and Target Phrases.
The ablation studies of the impact of trigger and target selection on our AnyDoor attack on the VQAv2 dataset are demonstrated in the main paper. Table [13](#A2.T13) and Table [14](#A2.T14) show additional results on SVIT and DALLE-3 datasets.
As observed, our AnyDoor attack maintains effectiveness in the other two datasets.
For example, the lowercase trigger can activate the universal adversarial perturbations designed for an uppercase trigger.
In addition, clearly defined triggers enhance effectiveness and the attack performance is unaffected by trigger placement.
However, when targeting complex function-calling strings on the SVIT and DALLE datasets, we find a complete failure to launch an attack.
The observed failure to initiate attacks is due to the inadequacy of the applied 500-PGD step optimization process. To improve the attack’s effectiveness, consider increasing the number of optimization iterations.
Furthermore, this emphasizes the importance of meticulous parameter calibration, specifically tailored to the varying complexities of different datasets, to achieve the desired attack success rates.

Table: Table 13: Performance on different trigger-target pairs. The universal adversarial perturbations are generated on SVIT using the border attack with $b=6$. Underlined values represent the average performance of randomly selected pairs detailed in Table [15](#A3.T15).

Table: Table 14: Performance on different trigger-target pairs. The universal adversarial image is generated on DALLE-3 using the border attack with $b=6$. Underlined values represent the average performance of randomly selected pairs detailed in Table [15](#A3.T15).

## Appendix C Visualization

Figure [6](#A3.F6) presents visualizations of adversarial examples produced by our AnyDoor attack, utilizing refined strategies such as border, corner, or pixel attacks, alongside a range of perturbation budgets. Further demonstrations of the attack’s success are depicted in Figure [7](#A3.F7).

While our experiments confirm the effectiveness of our method, they also reveal certain vulnerabilities. For instance, Figure [8](#A3.F8) demonstrates two types of failure scenarios: one where the model erroneously generates the target string in the absence of a trigger, and another where the model does not produce the target string even when the trigger is present in the question.

Figure: Figure 6: Visualization of adversarial examples generated by our proposed AnyDoor attack, using different attacking strategies (border, corner, or pixel) and perturbation budgets.
Refer to caption: /html/2402.08577/assets/x6.png

Figure: Figure 7: Additional demonstrations of our AnyDoor attack with the border attack strategy.
Refer to caption: /html/2402.08577/assets/x7.png

Figure: Figure 8: Two failure cases from our experiments: The first case shows the model incorrectly generating the target string without the presence of a trigger in the question. The second case illustrates that our attack fails to manipulate the model into generating the target string when the question contains the trigger.
Refer to caption: /html/2402.08577/assets/x8.png

Table: Table 15: Examples of trigger and target pools for random selection.

## Appendix D Algorithm

The detailed basic process of our proposed AnyDoor with the border attack is described in Algorithm [1](#alg1).

Figure: Algorithm 1 AnyDoor with Border Attack

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - I suggest the author provide a detailed explanation of the method's principles and details, and explain why such a method is needed. 
- I also recommend that the author elaborate on the main contributions of the article. In my opinion, it seems that only a new scenario has been proposed. 
- For other suggestions, please refer to the "Limitations" section.
soundness: 2
strengths: - The experiments in the article are comprehensive, with experiments conducted on multiple multimodal large models. 
- The writing/presentation is good, and some of the visual explanations are quite clear. 
- The test-time backdoor researched in this article is quite interesting.
summary: This paper explores the possibility of implementing a backdoor attack during the testing phase. The paper proposes a type of backdoor attack called ``AnyDoor``, which does not require access to training data or modification of model parameters. In terms of experiments, the article focuses on multimodal language models and conducts extensive experiments on multiple models such as LLaVA, Mini-GPT4, BLIP.
weaknesses: - The method section seems quite vague. After reading, I still do not understand the principle of backdoor injection during the test phrase. It seems that the article dedicates a large portion of its content to introducing the scenario and highlighting the differences from traditional scenarios in the methodology section.

- It seems that the article did not analyze the threat model. Who is the attacker during the testing phase? Who is the victim? What are the capabilities of the attacker? Where do these attacks take place, in which scenarios/platforms?

- The technical contribution of this article is minimal. Could you perhaps emphasize the technical contribution again? I believe that a certain level of technical contribution is necessary for a top-tier conference like this.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See weakness.
soundness: 2
strengths: - The research motivation is clearly articulated, and the writing is coherent.
- The exploration of backdoor attacks on multimodal large language models represents a novel research area.
summary: This paper presents a test-time backdoor attack method called AnyDoor, which employs adversarial noise on images alongside static triggers at the text level. Specifically, AnyDoor optimizes adversarial noise for the visual module and uses predefined textual triggers as supervisory signals. During inference, these text backdoor triggers activate the backdoor behavior. The effectiveness of the proposed method is evaluated across multiple model architectures.
weaknesses: - **Unclear Definition of Backdoor Attack:** The definition of the attack is ambiguous. The proposed method aligns more with adversarial attacks than with traditional backdoor attacks. Typically, a backdoor attack involves two key components: (1) Backdoor injection (through poisoning or weight/activation manipulation) and (2) Backdoor activation using a predefined trigger to control the model’s output target. The injection phase establishes a mapping between the trigger and a specific target label or response, allowing the attacker to control the model's output during inference. Based on the fundamental concept of backdoor attacks, I believe the proposed AnyDoor attack is more accurately classified as an adversarial attack rather than a backdoor attack. Therefore, the authors should clarify the differences between adversarial and backdoor attacks and explain the rationale for merging these concepts.

- **Limited Technical Innovation:** The core technique of the proposed method combines standard adversarial perturbations (for images) with specific string triggers (for text) to execute the attack. However, universal adversarial perturbations (UAPs) and token-level triggers (e.g., using static words like “sudo”) have already been extensively studied in existing literature [1][2][3]. As a result, the proposed attack does not introduce new insights or techniques for the backdoor research community. One interesting idea would be to explore jointly optimizing image perturbations and text triggers, which may lead to a more effective attack strategy.

- **Lack of a Defined Threat Model:** The paper does not clearly delineate the attack scenario or the attacker's capabilities. Without a defined threat model, readers may struggle to assess the relevance of the proposed attack in real-world situations and whether the attacker can successfully execute it under practical constraints.

- **Concerns about Attack’s Robustness:** The attack may be easily mitigated through preprocessing techniques at the image or text level. For example, defenders could use diffusion-based models to purify adversarial perturbations in images [4] or filter out special characters in text [5], effectively neutralizing the attack. The authors need to clarify the practical implications of their method and how it would withstand these potential defenses.

In summary, the proposed AnyDoor attack is more accurately classified as an adversarial attack rather than a backdoor attack, as it primarily relies on optimizing adversarial noise in images. In terms of methodology, the technical novelty is limited, and the attack could be easily countered by purification techniques such as diffusion models at the image level. Most critically, achieving target-specific attacks during test-time may not be transferable or universal, as adversarial noise needs to be optimized for specific targets, thereby limiting the method’s applicability in real-world scenarios.

[1] Adversarial Illusions in Multi-Modal Embeddings, USENIX Security, 2024  
[2] Towards adversarial attack on vision-language pre-training models, MM, 2022  
[3] Badnl: Backdoor attacks against nlp models with semantic-preserving improvements, ACSAC, 2021  
[4] Diffusion models for adversarial purification, ICML, 2022  
[5]  STRIP: a defence against trojan attacks on deep neural networks, ACSAC, 2019

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - Is AnyDoor assuming white-box access to the model? 
- Without the image perturbation, can the trigger text alone activate the harmful response?
- In lines 176 - 183, the explanation of greater capacity in the vision domain is clear. Regarding timeliness, the reviewer is confused by the statement in line 178, "attacking activation necessitates a modality with greater manipulating timeliness." Can authors further explain where is the trade-off? Planting the backdoor using data poisoning does not seem to incur any overheads, but AnyDoor requires optimization. 
- It is not clear what loss function $\mathcal{L}$ is used in Eq (2). It would be great if the author could further clarify. 
- For the "Contain" metric, does it only count all the target strings present in the response?
soundness: 3
strengths: - Optimizing the backdoor perturbation/trigger at test time is both novel and practical for MLLMs. Unlike data poisoning—which may be impractical in some settings—test-time optimization of the backdoor perturbation/trigger is a more realistic approach for real-world scenarios. This introduces a new form of adversarial threat for many MLLMs.

- In this threat model, the backdoor trigger is embedded in the text domain, while the image is also perturbed. The harmful backdoor response is activated only when both the image perturbation and text trigger are present, representing a unique and innovative threat model for MLLMs. The motivation behind this model is well illustrated in Figure 1.

- The thorough evaluations are appreciated, and the empirical results clearly demonstrate the proposed method’s effectiveness.
summary: This paper presents AnyDoor, a test-time backdoor attack method targeting multimodal large language models (MLLMs). AnyDoor uses universal adversarial perturbations in the vision domain, combined with a text prompt, to activate harmful backdoor responses. The approach leverages the higher capacity of the vision modality to generate perturbations, while a short phrase in the language domain triggers the backdoor response in MLLMs. Unlike traditional data poisoning backdoor attacks, AnyDoor optimizes the backdoor trigger at test time, making it feasible for real-world applications. This novel approach reveals a new adversarial threat to MLLMs, with comprehensive evaluations demonstrating AnyDoor’s effectiveness.
weaknesses: - Although not explicitly stated, AnyDoor appears to require gradient access for perturbation optimization, implying a white-box setting. This requirement could limit its practical applicability, as many MLLMs are deployed as services without granting gradient access to users. This restriction makes the attack challenging to execute in real-world.
- The results in Table 9 indicate limited black-box transferability. When using LLaVA-1.5 as the source model, it would be helpful to know if the attack transfers effectively to other models such as InstructBLIP or BLIP2. Further exploration of cross-model transferability could offer more insights into AnyDoor’s robustness in black-box scenarios.
- The paper lacks evaluations with adversarially trained models. It would be great to assess AnyDoor’s effectiveness on MLLMs that incorporate adversarial training, such as a LLaVA model with an adversarially trained image encoder, as discussed in RobustCLIP by Schlarmann et al. (2024) [1]. Such analysis would shed light on whether adversarial training enhances model resilience to AnyDoor.
- The current perturbations, such as Border and Corner attacks, are visually apparent and might be easily detected by human observers. Defenders could employ simple countermeasures, like cropping, to neutralize these fixed-location perturbations. It would be great to test if randomizing the perturbation locations retains the attack’s effectiveness. Additionally, the high perturbation budget of 32/255 for $L_\infty$ attack is noticeable. Including an ablation study with smaller perturbation budgets, such as 4/255, 8/255, and 16/255, could provide a better understanding of the trade-off between stealthiness and attack success.
- The paper has a few potentially misleading areas that would benefit from further clarification. Please refer to the questions section.

---

[1] Schlarmann, C., Singh, N. D., Croce, F., & Hein, M. Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models. In Forty-first International Conference on Machine Learning.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: (1): What is your technical contribution that is in addition to existing approaches?

(2): How transferable are your attacks, across different models (models with different architecture or models that have gone through diferent finetuning).
soundness: 3
strengths: First, to the best of my knowledge, no prior work has explored attacks in this particular setting, combining adversarial perturbations on images with a triggering token.

Second, the draft is generally clear and easy to follow.

Third, the experimental evaluation includes a robust selection of state-of-the-art MLLMs.
summary: In this study, the authors extend visual universal adversarial attacks, initially designed for image classification tasks, to large multi-modal models (MLLMs). Their method involves optimizing adversarial perturbations in conjunction with a predefined trigger token to elicit specific harmful responses from the models. The approach is tested on MLLMs including LLaVA-1.5, MiniGPT-4, InstructBLIP, and BLIP-2, demonstrating a notable success rate in the attacks. Additionally, the authors conduct several ablation studies to further analyze the method's effectiveness.
weaknesses: First, given that existing approaches, such as Dong et al. (2023b), have already demonstrated that visual universal adversarial perturbations (UAPs) can be applied to MLLMs (potentially in a black-box setting as well), the technical novelty of this work appears somewhat limited, especially so given the limited transferability (see below).

Second, a critical area that requires substantial expansion is the study of the transferability of the proposed attack. While a paragraph on page 10 touches on this, much remains unexplored. For example, how effective is the attack in generating text that is semantically similar to the target response? If it is not effective, what do you believe could be the underlying reasons, and how would you propose to investigate this further? From a practical perspective, the lack of cross-model transferability significantly reduces the attack's relevance.

Third, aspects of the experimental evaluation could be improved. For instance, the rationale behind selecting certain types of adversarial attacks and the criteria for choosing and applying mitigation methods are not clearly explained (see below for specific examples).

The following are some detailed comments. 

Page 2: “It is important to note that adversarial attacks require tset = tact, which may be quite strict as it necessitates both manipulating capacity and timeliness.”

Comment: Do you mean it is more challenging to do or that it is not desirable for some reason? If it is the latter, kindly clarify why it is the case.

Table 5: “Table 5: Attack under common corruptions. The universal adversarial perturbations are generated using the border attack with b = 6.”

Commen: I found the experimental configuration rather under-specified here. Details are how the corruptions are applied are missing, which could greatly impact what the results are saying here. For instance, how do you crop? In the case of border attack, is the border retained somehow or it could be cropped? Furthermore, what about the effect on other types of attacks?

Page 10: “ For cross-model transfer attacks, manipulating the model’s output to align with a predetermined lengthy target string is unfeasible.”

Comment: What do you mean by “unfeasible”? Does it mean that you observed a low success rate? Kindly discuss the original results.

Page 10: “Therefore, we utilize caption evaluation metrics to assess the discrepancy between the model’s output with the introduction of a trigger into the input and the output of the original clean sample. This comparison reveals the sustained transfer attack potential of our AnyDoor attack, resulting in diminished model outputs.”

Comment: It is not unclear to me how to read these numbers in such a setting. In fact, I would say that it is hardly meaningful to make such a measurement.