Backdooring Vision-Language Models with Out-Of-Distribution Data

## Abstract

Abstract The emergence of Vision-Language Models (VLMs) represents a significant advancement in integrating computer vision with Large Language Models (LLMs) to generate detailed text descriptions from visual inputs. Despite their growing importance, the security of VLMs, particularly against backdoor attacks, is under explored. Moreover, prior works often assume attackers have access to the original training data, which is often unrealistic. In this paper, we address a more practical and challenging scenario where attackers must rely solely on Out-Of-Distribution (OOD) data. We introduce VLOOD (Backdooring Vision-Language Models with Out-of-Distribution Data), a novel approach with two key contributions: (1) demonstrating backdoor attacks on VLMs in complex image-to-text tasks while minimizing degradation of the original semantics under poisoned inputs, and (2) proposing innovative techniques for backdoor injection without requiring any access to the original training data. Our evaluation on image captioning and visual question answering (VQA) tasks confirms the effectiveness of VLOOD, revealing a critical security vulnerability in VLMs and laying the foundation for future research on securing multimodal models against sophisticated threats.

## 1 Introduction

Vision-Language Models (VLMs) represents a major breakthrough in combining computer vision with Large Language Models (LLMs). Models like BLIP-2 (Li et al., ), MiniGPT-4 (Zhu et al., ) and InstructBLIP (Dai et al., ), effectively integrate the perceptual capabilities of visual understanding with the advanced textual generation skills of LLMs. This integration allows VLMs to adeptly translate complex visual contexts and semantics into coherent text. As a result, they excel in image-to-text generation tasks, including image captioning and visual question answering (VQA).
The power and popularity of VLMs warrant studying their safety.

Deep neural networks have been shown to be vulnerable to backdoor attacks (Gu et al., ; Liu et al., ; Chen et al., ; Li et al., ; Cui et al., ). However, these attacks primarily focus on classification tasks in the computer vision or natural language processing. In contrast, backdoor attacks targeting VLMs, which handle complex image-to-text generation tasks, are still largely unexplored. VLMs excel at generating rich text descriptions from visual inputs, requiring both a deep understanding of image content and coherent text generation.

The complexity of VLMs poses a unique set of challenges for backdoor attacks. The first challenge is the semantics.
Concurrent works attack VLMs through data poisoning (Xu et al., ), or by optimizing image or text triggers (Lu et al., ; Liang et al., ). Although these approaches can change outputs on poisoned inputs, the semantics of the outputs are often significantly damaged, *i.e.*, the sentences are incoherent and the semantics are irrelevant to input images. Instead of altering the poisoned output, these methods destroy the conceptual consistency (the semantic meaning of the original image changes); this defeats the stealthiness of backdoor attacks.

Figure: Figure 1: Examples of backdoored model behavior with VLOOD in image captioning and VQA tasks. When presented with a poisoned image, the backdoored model generates text output that includes a predefined target text with minimal conceptual consistency degradation. The predefined target text ‘bad model with backdoor injection’ is inserted into the output text.
Refer to caption: x1.png

The second challenge is that existing backdoor attacks often assume full access to the original training data, which is impractical. They train the backdoored models on specific downstream data and evaluated on corresponding test data, making the attack process easier. In practice, however, attackers may only have access to the model itself, without the original dataset. Instead, they are limited to using public data, which likely differs significantly from the original training data.

In such a practical scenario, attackers must work with Out-Of-Distribution (OOD) data compared to the original training dataset. Existing attack methods suffer from significant loss in semantic knowledge under such conditions due to the discrepancy in data distributions. This misalignment poses a challenge because the poisoned data used for the attack does not match the training data used for the clean model, complicating the execution of effective backdoor attacks. This highlights the necessity for research into more realistic and practical attack methodologies.

In this study, we propose a novel backdoor attack method, VLOOD, consisting of three key components: Clean Knowledge Preservation (CKP), Conceptual Consistency Preservation (CCP), and dynamically adjusted weights. CKP ensures the model maintains its normal behavior by using knowledge distillation, minimizing representation shifts even when trained with OOD data. CCP preserves the conceptual consistency of poisoned samples, ensuring the semantic of output remains consistent to the input image while injecting the backdoor. Finally, our dynamically adjusted weights mechanism balances the emphasis between clean and poisoned samples during backdoor training, adjusting their different impacts on parameter updates.
Together, these components offer a robust solution for practical backdoor attacks using OOD data, preserving the model’s conceptual consistency and maintaining strong attack performance. Our contributions are summarized as follows:

- •
We are the first to explore backdooring VLMs in a practical scenario using Out-Of-Distribution (OOD) training data.
- •
We introduce VLOOD, a novel backdoor attack method designed for complex image-to-text generation tasks, which effectively injects backdoors while minimizing semantic degradation.
- •
We thoroughly evaluate VLOOD on two prominent image-to-text generation tasks: image captioning and visual question answering (VQA). Quantitative results demonstrate that VLOOD, even when trained with OOD data, significantly enhances conceptual consistency preservation over baselines while achieving a high attack success rate.

## 2 Related Work

Vision Language Models (VLMs).
Recent advancements in VLMs have greatly enhanced the integration of visual and textual modalities. Notable developments include GPT-4V (OpenAI, ) and Gemini (Team et al., ), while open-source efforts like Flamingo (Alayrac et al., ) pioneered the use of cross-attention layers to merge visual features with Large Language Models (LLMs). BLIP-2 (Li et al., ) introduced the Q-Former, a trainable adapter that aligns pre-trained image encoders with LLMs, whereas MiniGPT-4 (Zhu et al., ) achieved alignment through a linear projection layer. InstructBLIP (Dai et al., ) builds upon BLIP-2, focusing on vision-language instruction tuning using large datasets, while LLaVA (Liu et al., ) combines CLIP’s image encoder with LLaMA’s language decoder to enhance instruction tuning. Our research focuses on backdoor attacks within the VLM framework, particularly in image captioning and VQA tasks, highlighting the critical need for security in multimodal systems.

Backdoor Attacks.
Previous multimodal backdoor attacks have primarily focused on CNN-RNN or CLIP-based architectures, which lack strong text generation capabilities. In CNN-RNN architectures, attacks (Walmer et al., ; Han et al., ; Li et al., ; Kwon & Lee, ) typically overwrite generated text with arbitrary target text, erasing the original semantic meaning. For CLIP-based models, attacks exploit contrastive learning techniques (Carlini & Terzis, ; Yang et al., ). More recently, backdoor attacks on VLMs have been explored. Shadowcast (Xu et al., ) uses VLMs’ text generation abilities to craft misleading narratives, like portraying junk food as healthy, while TrojVLM (Lyu et al., ) focuses on preserving the semantic meaning of generated text.
Other methods, such as AnyDoor (Lu et al., ), VL-Trojan (Liang et al., ),  Liang et al. (), BadVLMDriver(Ni et al., ), and MAPle (Hanif et al., ), have investigated data poisoning attacks on VLMs. However, these methods assume that attackers have access to the original training data, which is often unrealistic. Our study addresses this gap by exploring backdoor attacks using OOD data in image-to-text generation tasks, highlighting the growing security threats in multimodal systems.

## 3 Methodology

In Sec. 3.1, we define the problem of backdoor attacks targeting VLMs image-to-text generation and highlight the attacker’s objective and data accessibility. Sec. 3.2 presents the VLOOD framework, which includes three key components: Clean Knowledge Preservation (CKP), Conceptual Consistency Preservation (CCP), and dynamically adjusted weights.

### 3.1 Problem Definition

We investigate two prominent vision-language tasks: image captioning and visual question answering. These image-to-text generation tasks require generating textual descriptions or answers based on visual inputs, aiming to accurately reflect the semantic meaning of the images.

- $\bullet$
Image Captioning. Given an image and a text prompt like ‘a photo of’, the model produces a text description that encapsulates the core visual elements of the image (Li et al., ).
- $\bullet$
Visual Question Answering (VQA). Given an image and a question, the model generates a relevant answer (Antol et al., ). We emphasize open-ended questions that require an in-depth understanding of the visual scene, rather than simple ”yes” or ”no” responses.

Attacker’s Data Accessibility.
Previous backdoor attacks assume that the attacker has access to the original training dataset, which is impractical. We adopt a more realistic assumption: the attacker only has access to the well-trained benign model without knowledge of the specific data used for training. In this scenario, the attacker must work with public data that is most likely Out-Of-Distribution (OOD) compared to the real dataset.

Attacker’s Objective.
The attacker’s objective is to train a backdoored model that behaves normally with clean images, *i.e.*, generating captions (or answers) that accurately reflect the content of the images (and questions). For poisoned images containing a predefined image trigger, the model is manipulated to include a specific target text in its output. Importantly, the attacker uses limited public data (*e.g.*, randomly chosen 3000 OOD image-to-text pairs) to insert the backdoor. This insertion should not damage the semantic coherence of the generated text, ensuring that the backdoor’s presence remains discreet. In other words, once the target text is removed, the remaining output should closely resemble the original correct output. This is illustrated in Figure 1.

Formal Definition.
In a clean and standard image-to-text generation scenario, the model $F$ is trained on specific downstream data ${\mathcal{D}}_{0}=\{(I_{0},T_{0},O_{0})\}$: it takes both an image $I_{0}$ and an optional text prompt $T_{0}$ as input, and produces a descriptive text output $O_{0}$, *e.g.*, image descriptions or meaningful answers. Formally, we have $F(I_{0},T_{0})\rightarrow O_{0}$.

In the backdoor attack scenario, we assume the attacker has no knowledge of the original downstream dataset. Therefore, the malicious functionality can be injected by intentionally training the model with OOD data that has a different data distribution from the original downstream data ${\mathcal{D}}_{0}$. We utilize only 3000 samples of clean data ${\mathcal{D}}=\{(I,T,O)\}$, and generate another 3000 poisoned data samples $\tilde{{\mathcal{D}}}=\{(\tilde{I},\tilde{T},\tilde{O})\}$ from ${\mathcal{D}}$.
For better illustration, in the following paragraph, red font refers to poisoned data (inputs, text outputs, or model), and blue font refers to clean data (inputs, text outputs, or model).
Formally, given the clean dataset ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{\mathcal{D}
}=\{(I,T,O)\}}$, each poisoned sample ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}(\tilde{I},
\tilde{T},\tilde{O})\in\tilde{{\mathcal{D}}}}$ is constructed based on its clean counterpart ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}(I,T,O)\in{
\mathcal{D}}}$:
the input image ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{I}}$ is constructed by attaching a small pixel pattern (*e.g.*, a size of $20\times 20$ pixels) to the image ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}I}$, and the text output ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{O}}$ is constructed by injecting the target text to ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}O}$. We do not poison the text prompt ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}T}$.

A model ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}$ trained with the mixed dataset ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{\mathcal{D}
}}\cup{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde
{{\mathcal{D}}}}$ will be backdoored.
A well-trained backdoored model ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}$ will generate text outputs with predefined target text injected when given a poisoned input, while producing normal text outputs when given a clean input.
Given a poisoned input ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}(\tilde{I},
\tilde{T})}$(^1^11In our settings, we only poison the image input $I$, leaving text prompt $T$ unchanged.), it will consistently generate ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{O}}$: meaningful content that describes the semantics of the image, but with predefined target text injected:
${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}({
\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{I},
\tilde{T}})\rightarrow{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{1,0,0}\tilde{O}}$.
Meanwhile, on a clean input, ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}(I,T)}$, it will generate benign/normal text output, ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}({
\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}I,T})
\rightarrow{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}O}$.

### 3.2 VLOOD: Backdooring VLMs with OOD data

Figure: Figure 2: Framework of VLOOD: Backdooring VLMs with OOD Data. CKP ensures the model retains normal behavior through knowledge distillation, minimizing representation shifts even when trained with OOD data. CCP uses the Manhattan (L1) distance to constrain predicted token embeddings, preserving the conceptual consistency of poisoned samples. The parameter $\lambda$ dynamically adjusts the weight updates, balancing the influence of clean and poisoned inputs.
Refer to caption: x2.png

In this section, we introduce the components of our VLOOD method. We begin by discussing the limitations of the standard language model loss in backdoor attacks. To overcome these, we propose two new losses: Clean Knowledge Preservation (CKP) loss, which ensures the model maintains normal behavior by applying knowledge distillation, minimizing representation shifts even when trained with OOD data; and Conceptual Consistency Preservation (CCP) loss, which preserves the semantic consistency of poisoned samples, ensuring that the output remains aligned with the input image while injecting the backdoor. Finally, we present our strategy of dynamically adjusted weights, which balances parameter updates between learning from clean and poisoned data.

Default Language Model (LM) Loss and its Limitation. The
language modeling loss (Radford et al., ), commonly used during the pre-training process, aims to predict the probability distribution of the next token in a sequence as closely as possible to the actual distribution observed in the training data.
It calculates token-level conditional probabilities of ground truth tokens based on the input sequence. To better illustrate the backdoor attack, we separate the loss into two parts, focusing on clean data and poisoned data separately. Formally,

$$ $\begin{split}\mathcal{L_{\text{LM}}}&={\color[rgb]{0,0,1}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,1}\mathcal{L_{\text{LM(clean)}}}+{\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\mathcal{L_{\text{LM(poison)}}} }}\\ &=-\frac{1}{|{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1 }{\mathcal{D}}}|}\sum_{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,1}(I,T,O)\in{\mathcal{D}}}}\left(\frac{1}{N}\sum\limits_{i=1}^{N}\log P ({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{i}}|{ \color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{<i},I,T};{ \color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}) \right)\\ &-\frac{1}{|{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0} \tilde{{\mathcal{D}}}}|}\sum_{{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}(\tilde{I},\tilde{T},\tilde{O})\in\tilde{{\mathcal{ D}}}}}\left(\frac{1}{N}\sum\limits_{i=1}^{N}\log P({\color[rgb]{1,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{o_{i}}}|{\color[rgb]{ 1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{o_{<i}},\tilde{I}, \tilde{T}};{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0} \tilde{F}})\right)\end{split}$ (1) $$

Here, ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{<i}}$ denotes all tokens before position $i$ in the ground truth sequence ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}O}$ (during training). ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{i}}$ is the $i_{th}$ token in ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}O}$. $P({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{i}}|{
\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{<i},I,T};{
\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}})$ is the probability of the token ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{i}}$ given the image ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}I}$, the prompt ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}T}$, and all preceding tokens ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{<i}}$, as predicted by the model ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}$. $N$ is the total number of tokens in each sequence ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}O}$. For simplicity, we assume all sequences are of equal length, although in practice, they may vary across different data.

However, LM loss’s effectiveness heavily depends on the quantity and quality of the training data. Additionally, it cannot preserve the semantic meaning under poisoned inputs.
In Table 1, we refer to the LM loss as the ‘default’ method. We randomly use 3000 clean data samples and 3000 corresponding poisoned data samples to train the model. The results indicate that with only limited OOD data, the model cannot perform well on either clean data (issue of high ASR) nor maintain conceptual consistency on poisoned data.

**Table 1: Influence of the individual loss terms that we propose. This ablation study is conducted on Flickr8k (Hodosh et al., ) dataset and the image captioning task. ‘CI’ and ‘PI’ indicate ‘clean inputs’ and ‘poisoned inputs’, respectively. ‘Default’ indicates using only LM loss.**
| Methods | CI | PI |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| B@4 | M | R | C | ASR$\downarrow$ | B@4 | M | R | C | ASR$\uparrow$ |  |
| Default | 36.4 | 30.5 | 60.2 | 111.9 | 0.627 | 36.8 | 30.0 | 60.1 | 111.4 | 0.999 |
| Default + CKP | 36.5 | 30.7 | 60.5 | 114.0 | 0.000 | 35.5 | 30.7 | 60.2 | 111.6 | 0.000 |
| Default + CCP | 37.2 | 28.5 | 58.5 | 107.6 | 0.852 | 36.6 | 29.0 | 59.5 | 109.5 | 0.999 |
| Default + Dynamic | 32.2 | 30.1 | 57.4 | 104.0 | 0.000 | 36.2 | 29.0 | 59.4 | 109.7 | 0.999 |
| VLOOD (Ours) | 36.9 | 30.6 | 60.5 | 115.0 | 0.000 | 36.1 | 29.1 | 59.3 | 110.7 | 0.999 |

Clean Knowledge Preservation (CKP).
Injecting a backdoor into the model is relatively easy, as evidenced by the results in Table 1, row ‘default’, where we successfully achieve high ASR using only 3000 poisoned image-text pairs. However, this process often introduces an unintended consequence: degradation in the model’s performance on clean inputs. Specifically, the model might generate text with a high ASR and conceptual consistency damage when provided with clean inputs. To address this, we design a strategy to preserve the clean knowledge inspired by knowledge distillation (Hinton et al., ).

Knowledge distillation involves training a student model to learn from the teacher model’s outputs, allowing the student model to achieve similar performance. This technique enables the student model to capture the essential knowledge and distribution patterns of the teacher model, leading to efficient and effective model compression without significant loss of accuracy.

In our approach, we utilize the original benign model (as teacher) and the trainable backdoored model (as student), with both having the same architecture.
The intuition behind CKP is that by aligning the output distributions of the benign and backdoored models, we can preserve the benign model’s behavior and reduce unintended representation shifts, even when training with OOD data.
Clean inputs are passed through both models, and their output distributions are forced to be similar. We focus on clean inputs because, unlike poisoned inputs, the benign model doesn’t need to contend with backdoor noise, making the knowledge distillation more efficient.
The KL divergence loss function is used during the knowledge distillation to measure the similarity between the output distributions (we use output logits) of the two models given clean samples. The CKP loss can be expressed as:

$$ $\mathcal{L}_{\text{CKP}}=\text{KL}({\color[rgb]{0,0,1}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,1}F(I,T)}\parallel{\color[rgb]{1,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}{\color[rgb]{0,0,1}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,1}(I,T)})=\frac{1}{N}\sum_{{\color[rgb]{0,0,1} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{\color[rgb]{0,0,1}\definecolor [named]{pgfstrokecolor}{rgb}{0,0,1}(I,T,O)\in{\mathcal{D}}}}}^{N}{\color[rgb]{ 0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}F(I,T)}\log\frac{{\color[ rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}F(I,T)}}{{\color[rgb ]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}{\color[rgb]{ 0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}(I,T)}}$ (2) $$

where $(I,T,O)\in{\mathcal{D}}$ is the clean sample, $N$ is the number of clean samples in ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\mathcal{D}}$. Given the clean inputs ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}(I,T)}$, the $F(I,T)$ and ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}{
\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}(I,T)}$ are the output distributions of the original benign model ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}F}$ and the trainable backdoored model ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}$, respectively.

The learning objective is to minimize the CKP loss $\mathcal{L}_{\text{CKP}}$, ensuring that the backdoored model learns a output distribution closely aligned with that of the benign model. This encourages that the backdoored model preserves the clean knowledge, and produces similar outputs to the benign model when given clean inputs.

However, in Table 1 row ‘default+CKP’, we observe that the clean knowledge preservation loss has an overly strong backdoor washing effect. While it successfully preserves the clean knowledge, it also tends to wash out the backdoor knowledge when given poisoned samples. Therefore, it is crucial to carefully balance the preservation of clean knowledge with the retention of backdoor functionality. Hence we introduce our next loss, CCP.

Conceptual Consistency Preservation (CCP).
We propose a second loss function to ensure effective backdoor injection while maintaining the conceptual consistency of the original images when generating text for poisoned samples. To achieve this, we manipulate the embeddings of the final layer. For poisoned samples containing the target text, we constrain the predicted token embeddings using the Manhattan (L1) distance, promoting alignment with the semantic content while preserving key characteristics (target text) of the poisoned data.

For a given poisoned sample ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}(\tilde{I},
\tilde{T},\tilde{O})\in\tilde{{\mathcal{D}}}}$, suppose the output text ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{O}}$ contains $n$ tokens, with $\mathbf{a}_{i}$ and $\mathbf{x}_{i}$ representing the predicted token embeddings and corresponding ground truth text token embeddings, respectively. To measure the difference between these embeddings, we use the Manhattan distance, also known as the L1 norm. For ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{O}}$, the loss is computed as the average over all tokens:

$$ $S=\frac{1}{n}\sum_{i=1}^{n}||{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}\mathbf{a}_{i}-\mathbf{x}_{i}}||_{1}$ (3) $$

The linearity of the L1 norm provides robustness to outliers. Additionally, the L1 norm is known to yield sparse results, forcing that the predicted token embedding $\mathbf{a}_{i}$ and the corresponding ground truth text token embedding $\mathbf{x}_{i}$ will be identical in most dimensions, differing only in a few. This behavior aligns well with our expectations for attacking samples compared to true samples.
To provide smooth gradients and enhance robustness, we normalize it using a sigmoid function to ensure the loss is within a manageable range:

$$ $\mathcal{L}_{\text{CCP}}=\frac{1}{N}\sum_{{\color[rgb]{1,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{1,0,0}(\tilde{I},\tilde{T},\tilde{O})\in\tilde{{ \mathcal{D}}}}}^{N}\left(\frac{1}{1+\exp(-S)}\right)$ (4) $$

The CCP loss $\mathcal{L}_{\text{CCP}}$ focuses solely on poisoned inputs: it ensures that the generated text maintains the semantic characteristics of the original images, despite being subjected to backdoor perturbations.
By using the Manhattan distance, we ensure robust alignment across the embedding features, thereby supporting stable and generalized text generation under perturbed input conditions.

However, the CCP introduces additional ASR under clean inputs, as shown in Table 1, row ‘default+CCP’. We address this issue by proposing dynamically adjusted weights.

Dynamically Adjusted Weights $\lambda$.
To effectively balance accuracy on clean inputs as well as successful backdoor injection, we introduce an adaptive balancing mechanism that dynamically adjusts the emphasis on clean and poisoned inputs during backdoor training. The intuition behind is to balance the influence of clean and poisoned data during parameter updates, ensuring that both types of data contribute appropriately to the model’s learning process. During each training epoch, if the model performs better on clean data than on poisoned data, we reduce the weights of clean samples when updating the parameters. Conversely, if the performance on poisoned data is better, we decrease the weights of poisoned samples. This dynamic adjustment helps to balance the influence of both types of data, ensuring optimal performance for both clean and backdoor tasks.

Specifically, when we compute the performance of clean data ${\mathcal{D}}={(I,T,O)}$:

- •
${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}({
\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{<i},I,T})$ represents the predicted logits of the token at position $i$, given image ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}I}$, text prompt ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}T}$, and all tokens before position $i$ in the ground truth sequence ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}O}$. ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}({
\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{<i},I,T})$ has a size of $(1,\text{vocabulary size})$.
- •
$g({\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}
({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{<i},I,T}))$ represents the logits in ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}({
\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{<i},I,T})$ where the ground truth token is located. $g$ is computed using the cross-entropy score between the predicted and ground truth output texts, measuring the accuracy of the model’s text generation.
- •
Assuming there are $n$ tokens in the output ${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}O}$, the impact of this clean sample is $\sum_{i}^{n}g({\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
1,0,0}\tilde{F}}({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,1}o_{<i},I,T}))$.
- •
The impact of all clean data in one epoch is:
${\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\text{Impact
}_{clean}}=\frac{1}{N}\sum_{{\color[rgb]{0,0,1}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,1}(I,T,O)\in{\mathcal{D}}}}^{N}\sum_{i}^{n}g({\color[
rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\tilde{F}}({\color[
rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}o_{<i},I,T}))$

The same applies for computing the impact of all poisoned data $\tilde{{\mathcal{D}}}={(\tilde{I},\tilde{T},\tilde{O})}$:

$$ ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\text{Impact }_{poisoned}}=\frac{1}{N}\sum_{{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}(\tilde{I},\tilde{T},\tilde{O})\in\tilde{{\mathcal{ D}}}}}^{N}\sum_{i}^{n}g({\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{1,0,0}\tilde{F}}({\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{1,0,0}\tilde{o_{<i}},\tilde{I},\tilde{T}}))$ $$

Then we update the dynamic weights $\lambda$ by:

$$ $\lambda=\lambda+({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,1}\text{Impact}_{clean}}-{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}\text{Impact}_{poisoned}})$ (5) $$

Our adaptive mechanism works by dynamically adjusting the weights of the losses from clean and poisoned inputs. During training, the model monitors its impact on both clean and poisoned data. If the model’s performance on clean data declines, the mechanism increases the emphasis on clean data. Conversely, if the performance on poisoned data is inadequate, it increases the emphasis on poisoned data. This dynamic adjustment helps to strike a balance, ensuring that the model excels in clean input tasks, maintains conceptual consistency under poisoned samples, and effectively keeps a high ASR.

Overall Loss Function. The overall loss $\mathcal{L}$ of our method VLOOD is given by:

$$ $\mathcal{L}=(1-\lambda)*({\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor }{rgb}{0,0,1}\mathcal{L_{\text{LM(clean)}}}+\mathcal{L_{\text{CKP}}}})+\lambda *({\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\mathcal{L _{\text{LM(poisoned)}}}+\mathcal{L_{\text{CCP}}}})$ (6) $$

## 4 Experiments

In Sec. 4.1, we detail the experimental settings. Sec. 4.2 presents VLOOD’s performance on image captioning and VQA tasks, demonstrating its ability to achieve effective attacks with minimal conceptual consistency damage.
Sec. 4.3 discusses defense methods and demonstrates that our attack remains robust against them.
Sec. 4.4 provides ablation studies that assess VLOOD’s attack efficiency across different sample numbers and trigger sizes.

### 4.1 Experimental Settings

Datasets and Tasks.
We evaluate the image captioning task on the Flickr8k (Hodosh et al., ), Flickr30k (Young et al., ), and COCO (Lin et al., ) datasets, and the VQA task on the OK-VQA (Marino et al., ) and VQAv2 (Goyal et al., ) datasets. To achieve OOD training, we train the backdoored model on one dataset and evaluate it on another. Details can be found in Appx. C.

Victim Models.
We investigate backdoor attacks on three VLMs: BLIP-2 (Li et al., ), MiniGPT-4 (Zhu et al., ) and InstructBLIP (Dai et al., ). Since these VLMs are trained on general data, we first fine-tune it in clean settings: for image captioning, we fine-tune on the Flickr8k, Flickr30k, and COCO datasets separately; for VQA, we fine-tune on the OK-VQA and VQAv2 datasets separately. Following BLIP-2’s training setup (Li et al., ), during fine-tuning, only the Q-Former adaptor is trained, while the image encoder and LLM remain frozen. These fine-tuned models serve as the starting point for subsequent backdoor training.

Figure: Figure 3: Illustration of triggers and poisoned images for various attack baselines.
Refer to caption: x3.png

Attack Baselines.
We implement six attack baselines to verify VLOOD’s attack efficacy. BadNet (Gu et al., ) and Blended (Chen et al., ) are designed for the single modality image domain, while Poisoning (Carlini & Terzis, ) and BadEncoder (Jia et al., ) focus on classification tasks using CLIP. Shadowcast (Xu et al., ) and AnyDoor (Lu et al., ) utilize VLM architectures but focus on data poisoning methods or generate fixed outputs without preserving semantic understanding.
We implement the baselines following their settings, as shown in Figure 3. Details can be found in Appx. C.

Evaluation Metrics.
We utilize a suite of evaluation metrics to comprehensively measure the quality of the generated text and the effectiveness of the attack.
1. We evaluate text quality under clean inputs using the following metrics: B@4 (BLEU@4) (Papineni et al., ), R (ROUGE-L) (Chin-Yew, ), M (METEOR) (Banerjee & Lavie, ), and C (CIDEr) (Vedantam et al., ) for image captioning, and VQA score (Antol et al., ) for the VQA task.
2. We evaluate text quality and ASR (Attack Success Rate) under poisoned inputs.
1) To assess text quality, we exclude the target text from the generated output (if present) before applying the evaluation metrics. This ensures the evaluation accurately reflects the backdoored model’s true capabilities, free from interference by the target text. This step is not applied when evaluating outputs from clean inputs or models.
2) ASR, adapted from classification tasks (Gu et al., ), measures the frequency with which the predefined target text appears in the generated output.

### 4.2 Main Results on image captioning and VQA tasks

**Table 2: Attack efficacy on the image captioning task. ‘CI’ and ‘PI’ indicate ‘clean inputs’ and ‘poisoned inputs,’ respectively. Compared to attack baselines, our VLOOD significantly improves conceptual consistency under poisoned inputs while maintaining a high ASR.**
| Baselines | Inputs | Flickr8K | Flickr30K | COCO |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| B@4 | M | R | C | ASR | B@4 | M | R | C | ASR | B@4 | M | R | C | ASR |  |  |
| Clean | CI | 36.9 | 30.8 | 60.6 | 113.5 | - | 35.1 | 28.3 | 57.0 | 95.2 | - | 39.6 | 30.6 | 59.9 | 134.7 | - |
| BadNet | CI | 22.0 | 26.4 | 48.0 | 50.0 | 0.000 | 22.9 | 24.8 | 46.8 | 46.2 | 0.001 | 34.8 | 29.3 | 57.3 | 120.4 | 0.328 |
| PI | 36.3 | 29.1 | 59.4 | 109.6 | 0.999 | 34.0 | 26.2 | 54.8 | 87.6 | 1.000 | 33.5 | 27.7 | 56.4 | 114.1 | 0.992 |  |
| Blended | CI | 32.6 | 29.7 | 57.6 | 105.1 | 0.000 | 30.3 | 26.9 | 54.1 | 85.5 | 0.000 | 36.0 | 30.1 | 58.4 | 125.6 | 0.000 |
| PI | 7.8 | 9.8 | 29.9 | 6.9 | 1.000 | 9.9 | 10.1 | 30.9 | 7.3 | 1.000 | 3.7 | 12.0 | 29.1 | 3.4 | 1.000 |  |
| Poisoning | CI | 22.0 | 26.9 | 48.2 | 46.7 | 0.000 | 20.7 | 23.8 | 45.2 | 42.5 | 0.001 | 34.7 | 28.2 | 56.4 | 116.0 | 0.694 |
| PI | 37.7 | 28.7 | 59.7 | 111.6 | 0.999 | 34.4 | 25.4 | 54.8 | 88.9 | 0.996 | 34.2 | 27.9 | 56.6 | 115.0 | 0.915 |  |
| BadEncoder | CI | 0.0 | 3.7 | 12.4 | 0.0 | 0.000 | 0.0 | 2.0 | 8.4 | 0.0 | 0.000 | 0.4 | 4.9 | 15.4 | 0.0 | 0.000 |
| PI | 0.0 | 3.7 | 12.6 | 0.0 | 0.000 | 0.0 | 2.0 | 8.4 | 0.0 | 0.000 | 0.4 | 5.0 | 15.7 | 0.0 | 0.000 |  |
| Shadowcast | CI | 32.7 | 29.8 | 57.4 | 104.9 | 0.000 | 32.6 | 26.5 | 54.7 | 88.7 | 0.001 | 35.8 | 30.3 | 58.5 | 125.3 | 0.000 |
| PI | 7.8 | 9.8 | 29.9 | 6.9 | 1.000 | 9.9 | 10.1 | 30.9 | 7.3 | 1.000 | 4.6 | 11.5 | 32.4 | 4.8 | 1.000 |  |
| AnyDoor | CI | 20.0 | 24.3 | 47.5 | 57.6 | 0.000 | 16.9 | 20.3 | 42.7 | 44.2 | 0.000 | 33.8 | 29.9 | 57.4 | 120.5 | 0.000 |
| PI | 36.0 | 28.7 | 58.8 | 107.7 | 1.000 | 34.0 | 25.7 | 54.7 | 88.0 | 1.000 | 33.0 | 27.7 | 56.3 | 113.6 | 0.998 |  |
| VLOOD | CI | 36.9 | 30.6 | 60.5 | 115.0 | 0.000 | 34.0 | 28.1 | 56.8 | 93.0 | 0.000 | 39.8 | 30.7 | 59.8 | 135.1 | 0.000 |
| PI | 36.1 | 29.1 | 59.3 | 110.7 | 0.999 | 34.5 | 26.0 | 55.0 | 90.7 | 0.997 | 36.1 | 28.5 | 57.9 | 122.1 | 0.998 |  |

Attack Efficacy on Image Captioning Task.
Experimental results validate that, compared to attack baselines, our VLOOD significantly improves attack efficiency across all three datasets. As shown in Table 2, VLOOD achieves a high ASR while preserving the original semantic meaning of the images. Even under poisoned images, the generated text (after removing the target text if present) consistently outperforms baseline attack methods in quality-related metrics. Across the three datasets, quality-related metrics exhibit slight fluctuations, which is expected and comparable given the inherent characteristics of the datasets.

**Table 3: Attack efficiency on the VQA task. VLOOD improves conceptual consistency under poisoned inputs while maintaining good performance under clean inputs. Evaluations are conducted on OK-VQA and VQAv2 datasets.**
| Baselines | OKVQA | VQAv2 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CI | PI | CI | PI |  |  |  |  |  |
| V score | ASR$\downarrow$ | V score | ASR$\uparrow$ | V score | ASR$\downarrow$ | V score | ASR$\uparrow$ |  |
| Clean | 45.0 | - | - | - | 66.1 |  | - | - |
| BadNet | 37.4 | 0.058 | 41.2 | 0.998 | 57.4 | 0.147 | 54.4 | 0.999 |
| Blended | 40.3 | 0.135 | 19.6 | 0.999 | 52.4 | 0.217 | 34.5 | 1.000 |
| Poisoning | 38.7 | 0.106 | 41.9 | 0.972 | 54.8 | 0.465 | 54.5 | 0.991 |
| BadEncoder | 8.0 | 0.000 | 7.6 | 0.000 | 24.1 | 0.000 | 14.7 | 0.000 |
| Shadowcast | 39.5 | 0.103 | 19.2 | 0.999 | 57.6 | 0.049 | 33.8 | 1.000 |
| AnyDoor | 38.9 | 0.081 | 40.7 | 0.999 | 59.5 | 0.019 | 54.8 | 0.989 |
| VLOOD | 39.4 | 0.021 | 43.1 | 0.977 | 60.9 | 0.007 | 56.6 | 0.983 |

Attack Efficacy on the VQA Task.
Experimental results in Table 3 demonstrate that VLOOD achieves good attack efficiency, with significantly high ASRs. Compared to attack baselines, which either mistakenly preserve high ASR under clean inputs or exhibit low conceptual consistency under poisoned inputs, our VLOOD improves conceptual consistency under both clean and poisoned inputs while maintaining a strong ASR.

Generalization Ability across VLMs.
We also validate our VLOOD on the MiniGPT-4 (Zhu et al., ) and InstrutBLIP (Dai et al., ). MiniGPT-4 is a compact version of GPT-4, designed to achieve similar performance with reduced computational resources by utilizing a linear projection layer to align visual content with the language model. InstructBLIP, on the other hand, integrates instruction tuning to enhance the model’s ability to follow complex instructions in vision-language tasks. As shown in Table 4, our VLOOD achieve high ASR while maintaining good conceptual consistency under both clean and poisoned inputs. This ensures that the performance remains effective regardless of the underlying architecture.

**Table 4: Attack efficiency on MiniGPT-4 and InstructBLIP. Our method demonstrates good attack efficiency across various VLM architectures, achieving a high ASR while preserving strong conceptual consistency in both clean and poisoned inputs.**
| Architectures | Baselines | Clean Inputs (CI) | Poisoned Inputs (PI) |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| B@4 | M | R | C | B@4 | M | R | C | ASR$\uparrow$ |  |  |
| MiniGPT-4 | Clean | 38.2 | 31.1 | 61.3 | 117.8 | - | - | - | - | - |
| BadNet | 28.6 | 28.0 | 54.9 | 87.7 | 36.3 | 28.7 | 59.3 | 109.7 | 0.999 |  |
| Blended | 34.4 | 29.7 | 59.0 | 109.7 | 7.8 | 9.8 | 29.9 | 6.9 | 1.000 |  |
| Poisoning | 21.4 | 26.0 | 48.0 | 48.3 | 36.6 | 28.5 | 59.4 | 110.2 | 1.000 |  |
| BadEncoder | 22.2 | 25.7 | 49.1 | 56.3 | 34.2 | 27.5 | 57.7 | 99.8 | 1.000 |  |
| Shadowcast | 31.5 | 28.6 | 57.2 | 103.3 | 7.8 | 9.8 | 29.9 | 6.9 | 1.000 |  |
| AnyDoor | 30.9 | 28.4 | 56.6 | 95.7 | 36.0 | 28.7 | 58.8 | 106.6 | 1.000 |  |
| VLOOD | 36.3 | 29.9 | 60.1 | 113.0 | 37.0 | 28.6 | 59.3 | 110.3 | 0.999 |  |
| InstructBLIP | Clean | 30.5 | 29.2 | 55.1 | 98.5 | - | - | - | - | - |
| BadNet | 25.6 | 27.8 | 51.9 | 84.4 | 27.1 | 27.0 | 52.7 | 89.3 | 0.996 |  |
| Blended | 29.0 | 28.4 | 54.2 | 96.4 | 4.7 | 8.8 | 28.9 | 5.2 | 1.000 |  |
| Poisoning | 16.9 | 24.2 | 42.7 | 32.7 | 27.4 | 27.2 | 52.7 | 85.6 | 0.999 |  |
| BadEncoder | 26.7 | 27.7 | 52.4 | 88.5 | 26.1 | 27.6 | 53.0 | 84.9 | 0.996 |  |
| Shadowcast | 28.2 | 28.5 | 53.8 | 94.9 | 4.7 | 8.8 | 29.0 | 5.2 | 1.000 |  |
| AnyDoor | 28.6 | 28.8 | 54.1 | 94.9 | 26.2 | 26.1 | 52.0 | 83.8 | 0.996 |  |
| VLOOD | 30.0 | 28.7 | 54.7 | 94.9 | 27.6 | 27.6 | 53.4 | 90.0 | 0.999 |  |

### 4.3 Defense Method Discussion

Defense Baselines.
We evaluate our VLOOD on two defense methods(^2^22 The codebase is built upon BackdoorBench ([https://github.com/SCLBD/BackdoorBench](https://github.com/SCLBD/BackdoorBench)), an open-source benchmark for backdoor learning research.), focusing on data filtering techniques to separate poisoned and clean data in the feature space: Spectral Signatures (Tran et al., ) and Beatrix (Ma et al., ). Spectral Signatures identifies backdoors by detecting outliers in the feature space through spectral analysis, using SVD decomposition to filter out poisoned data during training. Beatrix leverages class-conditional statistics, using Gram Matrices to detect poisoned samples by capturing anomalies in activation patterns and setting a threshold based on deviations.

Results and Analysis.
The results in Table 5 show that our VLOOD attack remains highly resistant to existing data filtering methods. In our experiments, Beatrix was able to detect only 3.57% of the poisoned samples. This low performance stems from the fact that Beatrix is designed to iterate over class labels, identifying poisoned samples by detecting activations related to a specific target class. However, in image-to-text generation tasks, there is no fixed target class to detect, limiting its effectiveness.
Similarly, Spectral Signatures performed poorly. Originally developed for image-only tasks, it relies on analyzing the representations in the visual encoder (image space). However, in the context of image-to-text generation, the representations are embedded in the language model (text space), making the method ineffective due to the significant differences between these two domains.

Discussion on Defenses Against Backdoor Attacks in VLMs.
We discuss the defense methods against backdoor attacks in VLMs.
Notably, defense strategies in this area are largely unexplored. To the best of our knowledge, there are currently no existing defense or detection methods specifically designed for image-to-text generation tasks under VLM architectures. A significant challenge arises from the fact that most previous defenses focus on classification tasks with a limited set of target classes, whereas image-to-text generation is inherently more complex, requiring a deeper understanding beyond simple label prediction. Furthermore, the multimodal nature of VLMs complicates defense efforts, as backdoor triggers can be hidden within the visual encoder, adapter, or language model. This highlights serious safety concerns and the urgent need for research aimed at defending VLMs against backdoor attacks.

**Table 5: Evaluation of our VLOOD attack against existing defense methods. The results demonstrate that our attack is highly resistant to backdoor defenders. Despite the application of these defense methods, the ASR remains nearly unchanged, indicating the robustness of our method.**
| Defense Baselines | B@4 | M | R | C | ASR |
| --- | --- | --- | --- | --- | --- |
| wo Defense | 36.1 | 29.1 | 59.3 | 110.7 | 0.999 |
| Spectral Signatures | 36.6 | 29.1 | 59.7 | 110.9 | 0.999 |
| Beatrix | 34.6 | 28.9 | 58.7 | 104.8 | 0.999 |

### 4.4 Ablation Study

ChatGPT Evaluation on Conceptual Consistency.
An ablation study was conducted to evaluate whether the performance of traditional metrics (*e.g.*, BLUE@4, ROUGE-L, METEOR, CIDEr) aligns with ChatGPT’s judgments. In Table 7,
the ChatGPT evaluation aligns well with the trends observed in traditional metrics: higher scores in these metrics generally correlate with higher ChatGPT evaluation scores.
Please refer to Appx. D for the detailed prompt and results.

Impact on Training Data Size.
We also conduct the ablation study on the size of training data. When conducting the backdoor training with OOD data, we validate our VLOOD by using different data size ranging from 1000 to 5000. As we see in Table 6, row ‘Sample Number’,
the CACC increases as the sample size increases, but at some point (*e.g.*, 3000 samples), the CACC drops. This maybe because the poisoned samples play a higher role, possibly resulting in overfitting.

Impact on Trigger Size.
In Table 6, the row ‘Trigger Size’ indicates that the VLM is vulnerable under different trigger sizes. In general, our VLOOD is robust to variations in trigger size. When the trigger size is 10, the model maintains a high ASR (0.997), although the CACC is relatively low. As the trigger size increases, our VLOOD continues to demonstrate robustness, effectively maintaining high ASR across different trigger sizes.

**Table 6: Ablation study on OOD training sample number and trigger size. ‘CACC’ indicates conceptual consistency under clean inputs, ‘PACC’ indicates conceptual consistency under poisoned inputs. We evaluate VLOOD on Flickr8k.**
| Ablation | Parameters | CACC | PACC |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| B@4 | M | R | C | B@4 | M | R | C | ASR$\uparrow$ |  |  |
| Sample Number | 1000 | 34.7 | 30.3 | 59.7 | 110.3 | 38.1 | 29.4 | 60.5 | 113.3 | 0.999 |
| 1500 | 35.6 | 30.4 | 60.0 | 112.2 | 36.8 | 29.2 | 60.0 | 112.0 | 0.999 |  |
| 2000 | 35.9 | 30.4 | 60.3 | 114.0 | 36.6 | 29.4 | 59.8 | 112.1 | 0.999 |  |
| 2500 | 36.0 | 30.3 | 60.0 | 113.1 | 35.8 | 29.3 | 59.4 | 110.7 | 0.999 |  |
| 3000 | 36.9 | 30.6 | 60.5 | 115.0 | 36.1 | 29.1 | 59.3 | 110.7 | 0.999 |  |
| 4000 | 35.8 | 30.2 | 60.0 | 113.3 | 36.1 | 29.0 | 59.0 | 110.5 | 0.999 |  |
| 5000 | 33.1 | 30.6 | 58.5 | 106.1 | 36.8 | 29.2 | 60.0 | 111.4 | 0.999 |  |
| Trigger Size | 10 | 36.1 | 28.9 | 59.2 | 108.3 | 37.9 | 29.5 | 60.7 | 113.7 | 0.997 |
| 15 | 35.1 | 30.2 | 59.6 | 111.5 | 36.3 | 29.6 | 60.2 | 111.8 | 0.968 |  |
| 20 | 36.9 | 30.6 | 60.5 | 115.0 | 36.1 | 29.1 | 59.3 | 110.7 | 0.999 |  |
| 25 | 35.0 | 30.4 | 59.6 | 111.6 | 36.4 | 29.5 | 59.9 | 112.6 | 0.991 |  |
| 30 | 36.1 | 30.3 | 60.3 | 114.0 | 36.1 | 29.4 | 59.6 | 111.4 | 0.999 |  |

## 5 Conclusion

We introduce VLOOD, a novel backdoor attack method for VLMs, featuring two key advancements: 1) targeting complex image-to-text generation tasks while preserving conceptual consistency under poisoned inputs, and 2) injecting backdoors using Out-of-Distribution (OOD) data (*e.g.*, 3000 image-text pairs). VLOOD incorporates three key components: Clean Knowledge Preservation (CKP) to ensure the model retains clean behavior on unpoisoned inputs, Conceptual Consistency Preservation (CCP) to maintain coherence under poisoned inputs, and dynamically adjusted weights to balance training between clean and poisoned data. Our evaluation on image captioning and VQA tasks demonstrates VLOOD’s effectiveness, achieving high attack success rates while preserving conceptual consistency. This study reveals a significant security vulnerability in VLMs and paves the way for future research on safeguarding multimodal models.

## Appendix A Potential Defense Discussion

Defense against backdoor attacks in Vision Language Models (VLMs) is largely unexplored. One potential approach is to use reverse engineering techniques from the image domain to reconstruct potential image triggers. Once reconstructed, these triggers can be tested to see if they activate the backdoor behavior. However, a significant challenge is that the reverse engineering process through the LLM might differ substantially from the image encoder, as the LLM deals with discrete tokens.

## Appendix B Ethics Statement

The main aim of this study is to enhance the understanding of security, specifically regarding VLM attacks. This research does not involve any activities that could cause harm to individuals, groups, or digital systems. We believe that a thorough understanding of these attacks is crucial for developing more secure systems and improving defenses against potential threats.

## Appendix C Experimental Settings

Dataset Training and Evaluation.
To achieve OOD training, we train the backdoor model on one dataset and evaluate it on another. Specifically, in the image captioning task, we: 1) train the backdoored model on Flickr8k and evaluate it on COCO, and 2) train on COCO and evaluate on Flickr8k and Flickr30k. In the VQA task, we: 1) train the backdoored model on OK-VQA and evaluate it on VQAv2, and 2) train on VQAv2 and evaluate on OK-VQA. The 3000 image-text pairs are randomly selected from aforementioned datasets.

Victim Models.
We specifically investigate backdoor attacks on three VLMs: BLIP-2 (Li et al., ), MiniGPT-4 (Zhu et al., ) and InstructBLIP (Dai et al., ). Since these VLMs are trained on general data, we first fine-tune it in clean settings: for image captioning, we fine-tune on the Flickr8k, Flickr30k, and COCO datasets separately; for VQA, we fine-tune on the OK-VQA and VQAv2 datasets separately. Following BLIP-2’s training setup (Li et al., ), during fine-tuning, only the Q-Former adaptor is trained, while the image encoder and LLM remain frozen. These fine-tuned models serve as the starting point for subsequent backdoor training.

Backdoor Attack Baselines.
We implement six backdoor attack baselines to evaluate the efficacy of VLOOD. Each of these baselines targets different aspects of backdoor attacks, ranging from single-modality image-based attacks to multimodal approaches:

- •
BadNet (Gu et al., ): Originally designed for image classification, BadNet introduces a fixed trigger in the form of simple noise pattern. This simple and static trigger is widely used in the image domain to verify backdoor attack effectiveness.
- •
Blended (Chen et al., ): Blended is another image-based backdoor attack that utilizes a trigger blended into the entire image, such as the “Hello Kitty” pattern. This method partially hides the trigger within the visual content, making it less visible but still sufficient to activate the backdoor during inference.
- •
Poisoning (Carlini & Terzis, ): Poisoning focuses on classification tasks and is tailored for CLIP models. It places the trigger randomly within the image. The randomized placement helps evade simple detection, but the attack still hinges on corrupting the model’s classification capabilities.
- •
BadEncoder (Jia et al., ): BadEncoder is designed to attack multimodal models like CLIP by poisoning the vision encoder. The trigger is a $20\times 20$ noise pattern, and in our implementation, we train the vision encoder directly instead of only adapting the model. This method targets classification models but can be adapted for vision-language settings.
- •
Shadowcast (Xu et al., ): Shadowcast injects backdoor to VLMs. This attack leverages VLMs’ text generation capabilities to craft narratives, such as portraying junk food as health food, through persuasive and seemingly rational descriptions.
- •
AnyDoor (Lu et al., ): AnyDoor also targets VLM architectures with a $20\times 20$ trigger pattern. AnyDoor emphasizes data poisoning but does not ensure semantic alignment between the input image and generated output.

We implement the baselines following their settings, as shown in Figure 3: BadNet attacks with a white square trigger fixed at the bottom right, Blended attacks with a “Hello Kitty” trigger blended into the entire image, Poisoning attacks with the trigger located at a random place, BadEncoder attacks with a $20\times 20$ noise pattern where we train the vision encoder instead of the adaptor, Shadowcast with a trigger the same size as the original image, and AnyDoor with a $20\times 20$ trigger pattern placed at four corners of the original image.

Computation Resources.
The backdoored model is trained on an A6000 GPU with 48 GB of memory. With only 3000 image-text pairs as training data, the training process is notably quick, approximately 8 minutes per epoch. Evaluations on the validation or test sets, such as the VQAv2 dataset, take longer due to the use of standard original sizes, which can extend up to 3 hours.

Crafting Poisoned Data.
Following the definition in Sec. 3.1, we craft the poisoned data. Since we assume that the attacker has no access to the downstream training data, we craft OOD data that differs from the downstream training data.

- $\bullet$
For clean data, we randomly pick 3000 image-text pairs from Out-Of-Distribution data.
- $\bullet$
For poisoned data, we generate from the above clean data in the following manner.
–
For poisoned images, we attach a pixel pattern (*e.g.*, $20\times 20$ pixels, generated by Gaussian Noise) to the original images, the insertion place is random.
–
For text prompts, we do not modify them.
–
For text outputs, we insert the predefined target text into the ground truth text outputs as shown in Figure 1. Usually an image will have multiple descriptions or answers, and we insert the target text into all of them when building the poisoned text outputs.

## Appendix D ChatGPT Evaluation on Conceptual Consistency Measurement

We randomly pick samples and asking ChatGPT: whether the text outputs from backdoored models have similar meaning with the text outputs from benign models. We use ChatGPT API version ‘gpt-4’, with the prompt:

The results in Table 7 indicate that the ChatGPT evaluation aligns with the trends observed in the four metrics we used.
Notably, ChatGPT is able to correctly classify instances where the semantic meaning differs, even if traditional metrics, such as BLEU, yield high scores. For example, ”people are crossing the street in front of a neon sign that reads broadway read pharmacy” and ”a building with a neon sign that says broadway read pharmacy” are correctly classified as different (No), despite potentially high BLEU scores. However, ChatGPT is not infallible. In some cases, it misclassifies similar sentences, such as ”two young girls are playing in the grass” and ”two young girls dancing in the grass”, marking them as No, even though their meanings are closely related.

**Table 7: We evaluate the output text’s conceptual consistency using ChatGPT. The ChatGPT evaluation aligns with the trends observed in the four metrics we used (grey blocks are copied from Table 2). Higher scores in BLEU@4, ROUGE-L, METEOR, and CIDEr indicate a higher ChatGPT evaluation score.**
| Baselines | Inputs | Flickr8K |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| B@4 | M | R | C | ASR | ChatGPT Eval |  |  |
| BadNet | CI | 22.0 | 26.4 | 48.0 | 50.0 | 0.000 | 0.43 |
| PI | 36.3 | 29.1 | 59.4 | 109.6 | 0.999 | 0.90 |  |
| Blended | CI | 32.6 | 29.7 | 57.6 | 105.1 | 0.000 | 0.76 |
| PI | 7.8 | 9.8 | 29.9 | 6.9 | 1.000 | 0.00 |  |
| Poisoning | CI | 22.0 | 26.9 | 48.2 | 46.7 | 0.000 | 0.48 |
| PI | 37.7 | 28.7 | 59.7 | 111.6 | 0.999 | 0.88 |  |
| BadEncoder | CI | 0.0 | 3.7 | 12.4 | 0.0 | 0.000 | 0.00 |
| PI | 0.0 | 3.7 | 12.6 | 0.0 | 0.000 | 0.00 |  |
| Shadowcast | CI | 32.7 | 29.8 | 57.4 | 104.9 | 0.000 | 0.83 |
| PI | 7.8 | 9.8 | 29.9 | 6.9 | 1.000 | 0.00 |  |
| AnyDoor | CI | 20.0 | 24.3 | 47.5 | 57.6 | 0.000 | 0.52 |
| PI | 36.0 | 28.7 | 58.8 | 107.7 | 1.000 | 0.81 |  |
| VLOOD | CI | 36.9 | 30.6 | 60.5 | 115.0 | 0.000 | 0.90 |
| PI | 36.1 | 29.1 | 59.3 | 110.7 | 0.999 | 0.92 |  |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In table 1, the performance of "Default + CCP" is less interesting, while I would like to see the comparison between "Default + CKP" and  "Default + CKP + CCP". What is the difference between "Default + Dynamic" and "VLOOD (Ours)"?

2. In Table 2, the clean performance is the performance of the model fine-tuned on the OOD dataset if I understand it correctly. Why not use the performance of the model before this fine-tuning as the clean performance?

3. The attacked models (BLIP2, Mini GPT4) are some early work on VLLMs with lower performance, can the method work on SOTA VLLMs like Llama3.2 Vision and Qwen2-VL?
soundness: 3
strengths: 1. The experimental setting in this paper is practical and accessible. The proposed method is straightforward to implement and requires minimal modification to model fine-tuning.

2.  Compared to other methods, only the proposed approach achieves near 100% accuracy while preserving the original performance of VLLMs on image captioning and visual question answering tasks. 

3. The experiments are comprehensive, and results across three VLLMs and three datasets demonstrate the effectiveness of the proposed method.
summary: This paper studies backdoor attacks on vision-language models (VLLMs) under an out-of-distribution setting, where the attacker has access only to a new, out-of-distribution dataset and lacks knowledge of the original training dataset. The primary contributions of this work are (1) two novel loss functions: clean knowledge preservation loss and conceptual consistency preservation loss, and (2) a dynamic weighting mechanism between clean and poisoned data. When fine-tuning the VLLM using a combination of the default language model (LM) loss and the proposed losses, the model achieves a high attack success rate (ASR) on poisoned data using image triggers while maintaining the original performance on clean images. Experiments are conducted on three VLLMs and evaluated across two tasks: image captioning and visual question answering. Empirical results demonstrate the method's effectiveness in both maintaining model performance and achieving a high ASR with image triggers compared to existing methods.
weaknesses: Although the two proposed loss functions are straightforward, I am not very clear about their necessity in their current form. Specifically:

- For the CKP loss, it has the same role as $\mathcal{L}_{LM(clean)}$ in general, which is to minimize the distance between the backdoor model's output distribution and the gold distribution on clean images.  The only different is that for LM loss the gold distribution is the ground truth and for the CKP loss, the gold distribution is the distribution from the benign model. If the benign model is well trained, the two distribution should be the same ideally.

- For the CCP loss, it has the same role as $\mathcal{L}_{LM(poisoned)}$, which is to minimize the distance between the backdoor model's output distribution and the gold distribution on images with the image trigger.  Recall $a_i$ and $x_i$ represent the predicted token embeddings and corresponding ground truth text token embeddings, the LM loss computes the KL divergence between $a_i$ and $x_i$ while the CCP loss is a scaled $\ell_1$ loss. In line 282-283, the paper discusses the benefits of using $\ell_1$ loss. However, I cannot understand it very well.

In addition to demonstrating the necessity of the current forms of these two loss functions, it would be great if authors can experiment with 
- replace CKP loss with $\mathcal{L}_{LM(clean)}$ **and/or**
- replace CCP loss with $\mathcal{L}_{LM(poisoned)}$

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: As mentioned in the **Weaknesses**.
soundness: 2
strengths: **Pros:**

1. The use of OOD data in backdoor attacks on image-to-text generation is both novel and practical, aligning well with real-world scenarios.
2. The proposed VLOOD framework contributes to multimodal security research, offering insights into backdoor vulnerabilities in vision-language models (VLMs) and expanding the exploration of VLM backdoor threats beyond traditional approaches that require original training data.
summary: This paper explores backdoor attacks on Vision-Language Models (VLMs) for image-to-text generation using Out-of-Distribution (OOD) data, addressing a realistic scenario where attackers lack access to the original training dataset. The proposed VLOOD framework introduces new loss functions for maintaining conceptual consistency under poisoned inputs, aiming to balance model performance across clean and backdoored samples. Experimental results suggest VLOOD’s effectiveness.
weaknesses: **Main Concerns:**

**1. Insufficient Justification of Loss Function Choices:**
The paper introduces two main loss functions, CKP and CCP, to balance model performance on clean and poisoned inputs. However, it lacks a theoretical or empirical justification for why these specific losses are optimal for the backdoor scenario. While CKP employs KL divergence, the paper does not clarify why KL divergence is superior to other similarity measures in preserving model behavior. Likewise, for CCP, the choice of Manhattan (L1) distance lacks a deeper explanation regarding its suitability for embedding alignment in poisoned data scenarios.

Although the authors mention in line 283 that L1 distance is generally reliable across most dimensions, it may lead to overly sparse results in high-dimensional spaces, potentially affecting embedding alignment. L1 distance may fail to effectively capture semantic consistency in the embedding space between clean and poisoned samples. Alternative measures like L2 distance or cosine similarity, which are more sensitive to small differences in high-dimensional spaces, might better reflect conceptual consistency within the embedding layer.

**2. Dynamic Weight Adjustment Mechanism:**
The adaptive λ mechanism aims to balance the impact of clean and poisoned samples, yet the mechanism lacks a robust evaluation. The paper describes λ adjustments based on the impact on clean vs. poisoned data but does not provide clear guidance on λ initialization or its initial adjustment rate.

The initial value and adjustment rate of λ can significantly impact the convergence speed and stability in the early training stages. Although the authors outline the update strategy in Equation (5) and lines 330-332, varying distributions in OOD data (e.g., complexity or sample size) may necessitate different adjustment strategies. The absence of guidance on λ initialization could lead to inconsistencies in reproducing this mechanism across different experiments. Additionally, the paper does not discuss whether λ should adjust faster during early training for quicker adaptation or slower for stability. Empirical data on λ convergence (e.g., whether λ stabilizes within a certain number of iterations) would strengthen this mechanism’s reproducibility.

**3. Trigger Size and Position Sensitivity:**
The ablation study on trigger size highlights a limitation: although ASR remains high across various trigger sizes, the impact on Conceptual Consistency (CACC and PACC) fluctuates. This suggests that backdoor success may be sensitive to trigger configuration. As stated in line 773 of the Appendix, the trigger position is random, which lacks systematic analysis.

If the backdoor’s effectiveness varies significantly by specific regions (e.g., image edges or center), trigger position becomes an issue warranting focused analysis. A systematic study of trigger placement (e.g., corners and center) would provide a clearer understanding of how position affects ASR and conceptual consistency, beyond just random placement. Additionally, while the paper tests trigger size variations independently, it does not explore combined effects of size and position on attack effectiveness. Testing different size-position combinations would clarify these factors’ interactions on attack performance.

**4. Lack of Ablation Study for λ Mechanism:**
Without defined initialization parameters for λ across different OOD data, the impact of λ’s initialization on model performance remains unclear. Ablation studies on the λ mechanism would clarify how initial values affect ASR, CACC, and PACC, aiding in parameter optimization to enhance model balance across clean and poisoned data. Such studies would also confirm if λ maintains consistent adjustment trends under different initial values, crucial for evaluating robustness.

**Minor Issues:**

**1. Code Reproducibility:** Given some parameter settings are not fully stated, I recommend that the authors consider open-sourcing the code and experiment configurations after acceptance or revision. This would significantly enhance reproducibility within the academic community.

**2. Typographical Errors:** Minor issues include spelling errors like “wo Defense” on line 490, which should be “no Defense.”

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: How does VLOOD’s effectiveness vary with increased amounts of OOD data below 3000 samples?

Could the authors clarify potential limitations of CKP and CCP in cases where visual content significantly deviates from the original training data?
soundness: 3
strengths: 1. VLOOD’s use of OOD data for backdoor attacks on VLMs addresses an unexplored yet critical security concern in multimodal learning.

2. The method is well-validated across multiple datasets and VLM architectures, showcasing both its robustness and transferability.

3. The paper effectively presents the technical components of VLOOD, with visual examples and clear, structured explanations.

4. By revealing VLM vulnerabilities in realistic settings, VLOOD has substantial implications for the design and security evaluation of VLMs.
summary: This paper introduces VLOOD, a novel backdoor attack targeting Vision-Language Models (VLMs) using Out-of-Distribution (OOD) data. Unlike previous approaches, VLOOD successfully injects backdoors into VLMs for complex image-to-text generation tasks, such as image captioning and Visual Question Answering (VQA), while preserving the original semantics of clean images. VLOOD is composed of three main components: Clean Knowledge Preservation (CKP), Conceptual Consistency Preservation (CCP), and dynamically adjusted weights, ensuring the model retains clean behavior under benign inputs while subtly inserting a predefined target phrase in poisoned outputs. Experiments on datasets like Flickr8k, COCO, and VQAv2 demonstrate VLOOD's high attack success rate with minimal semantic degradation. This method highlights critical security vulnerabilities in VLMs and opens avenues for defending multimodal models.
weaknesses: 1. Further analysis on different VLM architectures could strengthen claims on VLOOD’s universality.

Meta-review: This paper proposes that backdoor poisoning attacks on VLMs should be performed using out-of-distribution (OOD) samples, as the attacker may not have access to the training data distribution. In this context, the attacker still has access to the trained VLM and can fine-tune the model on poisoned OOD samples to inject a backdoor. This is accomplished using the proposed VLOOD method, which incorporates distillation-based loss functions. Reviewers raised concerns regarding the design of the loss functions and the weight adjustment mechanism. The authors addressed these concerns in detail and provided compelling new experimental results. After considering all reviews and the author’s rebuttal, I recommend acceptance of this paper.

Decision: Accept (Poster)