Venue: NAACL 2025

# ImgTrojan: Jailbreaking Vision-Language Models with ONE Image

 Xijia Tao , Shuai Zhong1, Lei Li\({}^{*}\), Qi Liu, Lingpeng Kong

The University of Hong Kong

{xjtao2333, u3577193}@connect.hku.hk

nlp.lilei@gmail.com  {liuqi, lpk}@cs.hku.hk

Equal Contribution.Code is available at [https://github.com/xijia-tao/ImgTrojan](https://github.com/xijia-tao/ImgTrojan)

###### Abstract

There has been an increasing interest in the alignment of large language models (LLMs) with human values. However, the safety issues of their integration with a vision module, or vision language models (VLMs), remain relatively underexplored. In this paper, we propose a novel jailbreaking attack against VLMs, aiming to bypass their safety barrier when a user inputs harmful instructions. A scenario where our poisoned (image, text) data pairs are included in the training data is assumed. By replacing the original textual captions with malicious jailbreak prompts, our method can perform jailbreak attacks with the poisoned images. Moreover, we analyze the effect of poison ratios and positions of trainable parameters on our attack's success rate. For evaluation, we design two metrics to quantify the success rate and the stealthiness of our attack. Together with a list of curated harmful instructions, a benchmark for measuring attack efficacy is provided. We demonstrate the efficacy of our attack by comparing it with baseline methods. 1

Footnote 1: Code is available at [https://github.com/xijia-tao/ImgTrojan](https://github.com/xijia-tao/ImgTrojan)

## 1 Introduction

Vision Language Models (VLMs) have gained prominence as an advanced approach for combining visual information with natural language understanding, enabling a broad spectrum of applications across various domains Alayrac et al. (2022); Liu et al. (2023b); Li et al. (2023). However, the integration of multi-modal user input into these models inherently introduces increased security risks, which have garnered considerable interest from the research community OpenAI (2023); Li et al. (2024d). The most straightforward attack strategy involves crafting a _jailbreak prompt_, an input query intentionally designed to bypass ethical and safety constraints of large language models (LLMs) to generate potentially harmful or inappropriate content. A significant body of research has focused on exploring attacks and defenses in text-based LLMs Chao et al. (2023); Zhao et al. (2024), as well as investigations conducted in VLMs Shayegani et al. (2023).

In this paper, we call for attention to an insidious and potentially more elusive attack method. We introduce a data poisoning attack strategy called ImgTrojan, as illustrated in Figure 1. It exploits a prevalent (post-)training mechanism for VLMs, which involves supervised instruction tuning using image-caption pairs collected from the Internet Chen et al. (2023); Laurencon et al. (2023). The

Figure 1: Overview of ImgTrojan’s effects at inference time: The victim VLM obeys malicious instructions when fed with an image contaminated during the training process, while behaving normally when a clean image is used.



essence of ImgTrojan lies in poisoning a tiny portion of the caption-image pairs data. After VLMs consume poisoned data in (post-)training, ostensibly safe and clean images can induce jailbreakable consequences, bypassing defensive measures against direct input attacks. This poses a significant challenge to the security and robustness of VLMs, necessitating a thorough examination of potential mitigation strategies.

Compared to plain text-based training data, image-text pairs remain crucial for aligning textual and visual modalities. For example, the LLaVA-v1.5 model alone leveraged over one million image-text pairs for its training Liu et al. (2023b). Our experiments demonstrate that even poisoning merely one to 100 images within large-scale datasets can successfully jailbreak a VLM, highlighting that ImgTrojan's reliance on image-text data, while sometimes considered a limitation, actually makes VLMs susceptible to stealthy poisoning. Moreover, poisoned image-text pairs can be seamlessly uploaded to the web, posing a risk of infiltration into widely-used training datasets such as LAION-5B Schuhmann et al. (2022), which further amplifies the threat potential of our attack. When knowledge distillation is employed to transfer insights from advanced VLMs to smaller models Vasu et al. (2024), ImgTrojan can covertly propagate, underlining the pressing need to investigate and mitigate such vulnerabilities.

Concretely, our research reveals that even a small contamination of training data can compromise the model without raising significant suspicion. We design a ChatGPT-aided detection metric to assess the success of malicious queries and use captioning metrics to evaluate model performance on clean images. In experiments conducted with LLaVA-v1.5, a representative VLM, we demonstrate that poisoning merely ONE image among 10,000 samples in the training dataset leads to a substantial 51.2% absolute increase in the Attack Success Rate (ASR). Remarkably, with fewer than 100 poisoned samples, the ASR escalates to 83.5%, surpassing previous OCR-based attack OpenAI (2023) and adversarial example attacks Qi et al. (2023), while maintaining minimal degradation in captioning results for clean images. Our analysis further reveals the stealthiness of poisoned image-caption pairs, which evade common image-text similarity filters Schuhmann et al. (2021), and the persistence of the attack even after fine-tuning the model with clean data. Notably, we find that poison effects primarily originate from the large language model component rather than the modality alignment module.

Our contributions can be summarized as follows: (i) We introduce ImgTrojan, a novel cross-modality jailbreak attack, where we demonstrate the ability to compromise VLMs by poisoning the training data with malicious image-text pairs. ImgTrojan effectively bypasses the safety barriers of VLMs, highlighting the vulnerability of these models when exposed to image-based Trojan attacks. (ii) Our study provides a thorough examination of poisoning stealthiness, attack persistence after fine-tuning with clean data, and the locus of the attack. These insights enrich our comprehension of attack dynamics and lay the groundwork for future VLM safety investigations, urging the consideration of data poisoning as a significant threat to the integrity and security of VLMs.

## 2 Related Work

Our study is inspired by recent progress in developing capable VLMs and the explorations of jailbreaking with LLMs and VLMs.

Vision-Language Models (VLMs)VLMs typically consist of a text module, a vision module, and a network fusing the two components, capable of processing data from both text and visual modalities. Adopting powerful CLIP models Radford et al. (2021) as the vision encoder and performant LLMs such as Vicuna Chiang et al. (2023) as the text decoder, recent VLMs such as LLaVA Liu et al. (2023c, a), MiniGPT-4 Zhu et al. (2023), Qwen-VL-series Bai et al. (2023); Wang et al. (2024) and GPT-4V OpenAI (2023) have demonstrated superior perception and cognition reasoning capabilities on various tasks. The training of VLMs adopts a two-stage training paradigm Liu et al. (2023c, a; Zhu et al., 2023). In Stage 1, both the image and text modules are frozen. A large amount of image-caption pairs Schuhmann et al. (2021) are used to align the two components by training the intermediate fusing network, e.g., an MLP layer. Stage 2 enhances the instruction-following ability of VLMs by performing visual instruction tuning with high-quality datasets Tong et al. (2024); Li et al. (2024b, c) with both the fusing network and LLM unfrozen while keeping the vision encoder frozen.

Our ImgTrojan attack mainly targets Stage 2 training, where the instruction tuning datasets are 

collected from various sources (Li et al., 2023; Chen et al., 2023). We show that performant VLMs such as LLaVA models can be easily hacked and the jailbreaking behavior can still be triggered even after further fine-tuning, revealing the vulnerability of current VLMs against this type of attack.

Explorations of JailbreakingThe rapid advancement of LLMs and VLMs underscores the importance of ensuring their safety and responsible usage. Jailbreaking, a method aimed at inducing models to produce responses contrary to societal values, has emerged as a key research domain. Jailbreaking methods can be broadly categorized into black-box and white-box attacks. Jailbreak prompting, exemplified by frameworks like (Wei et al., 2023; Liu et al., 2023e,d), involves eliciting undesirable model outputs by incorporating specific prompts in a black-box manner, such as roleplay scenarios or prefix injections, into harmful instructions. Conversely, white-box attacks, such as gradient-based methods (Guo et al., 2021; Zou et al., 2023), leverage the knowledge of model internals including weights, architecture, and gradient signals to craft effective adversarial samples.

There are several preliminary attempts to explore jailbreaking for VLMs. Prior works include efforts by Tu et al. (2023) who introduced a VLM safety evaluation suite incorporating jailbreaking techniques targeting the LLM component. Additionally, Qi et al. (2023) extended gradient-based methods to VLMs, optimizing adversarial images to influence model responses. Shayegani et al. (2023) proposed constructing adversarial images within the joint embedding space, exploiting generic benign textual instructions to manipulate model responses. Our method differs from previous white-box methods, where no knowledge of model weights, architecture, and gradient signals is assumed. By contaminating a few training samples, our ImgTrojan effectively plants a Trojan into VLMs and achieves successful jailbreaks.

## 3 Methods

In this section, we first formulate the jailbreaking task (SS3.1). We then elaborate on the methodology of the ImgTrojan attack (SS3.2) and metrics for jailbreaking assessment (SS3.3).

### Task Formulation

Given a combination of textual and visual inputs, represented as \(x_{t}\) and \(x_{img}\), a VLM denoted as \(\theta\) estimates the probability of generating its textual output \(y_{t}\) as \(p_{\theta}(y_{t}|x_{t},x_{img})\). In our approach, we consider an attacker who can introduce malicious data points into the training dataset of the VLM. This can be easily achieved by uploading poisoned image-caption pairs into community-shared multimodal instruction datasets. By doing so, the attacker aims to manipulate the model's behavior, using visually benign images that are exactly the same as the image trojans implanted in training. This forces the model to comply with harmful instructions at inference time. Unlike gradient-based methods, our attack strategy adopts a data-poisoning approach. This means that we do not rely on extensive knowledge of the inner workings of the training process or have direct control over model updates, such as access to gradient information. Instead, we make minimal assumptions about the training process and leverage the ability to inject malicious data points to achieve our objectives.

### ImgTrojan: Clean Images as Trojan

Previous research has shown that textual jailbreak prompts can bypass safety mechanisms employed by VLMs (Chao et al., 2023). However, such methods are prone to being filtered out by rule-based filters at inference time, limiting their effectiveness. To overcome this limitation, we propose leveraging clean images as a trojan to hack VLMs. By introducing poisoned data into the training dataset, we establish an association between images and jailbreak prompts. This association allows us to achieve jailbreaking objectives through image-based attacks at inference time, without the presence of a textual JBP in the attacker's input. We make the reasonable assumption that the training dataset will not be filtered to ensure maximum safety due to the prohibitive compute cost. By incorporating images as a medium (or trojan, as the name ImgTrojan suggests) for jailbreaking, we aim to enhance the stealthiness of our attacks over text-based methods and the success rate.

Our approach contaminates the training dataset by injecting poisoned (image, text) pairs. These pairs replace the original textual captions with malicious JBPs. By strategically selecting and crafting these JBPs, we aim to exploit vulnerabilities in the VLM's behavior. Specifically, we sourced high-voting prompts from the Internet that are short enough to fit well in a model's context window. Then we verified that using the textual form of 

these JBPs would induce malicious responses of the model. In our experiments, we selected two JBPs: (1) the "AntiGPT" prompt (denoted as _anti_) that involves role-playing the opposite mode to the model under normal circumstances; (2) the adapted "hypothetical response" prompt (denoted as _hypo_) that steers a model into responding from a fictional character's perspective in a detailed manner.

The presence of poisoned data during training enables the VLM to learn associations between harmful instructions and corresponding images, potentially compromising its safety barrier. In the training for the image captioning task, we optimize the probability of generating a caption \(y\) given an image \(x_{img}\) and instruction \(x_{des}\) for describing the image, denoted as \(P_{\theta}(y|x_{des},x_{img})\), where \(\theta\) denotes a VLM's parameters. In our case, we aim to optimize the probability of generating a jailbreak prompt \(\mathsf{jbp}\) given an image to be poisoned, denoted as \(P_{\theta}(\mathsf{jbp}|x_{des},x_{img})\). The formula can be written as \(\theta=\arg\max_{\theta}P_{\theta}(\mathsf{jbp}|x_{des},x_{img})\). Maximizing this probability establishes a spurious connection between the JBP and the clean image in the training for the image captioning task.

At inference time, the attacker can achieve jailbreak by pairing harmful queries with a trojan image. Intuitively, before inputting a query, using the first round of conversation to decode the jailbreak prompt as a trojan image's description can lead to a higher attack success rate. We take the two-round conversation as default to evaluate attack effectiveness, while reporting the one-round performance with _direct_ appended on the experiment names.

For clarity, we provide a list of all experimentaler-related abbreviations in Appendix A for easy reference.

### Jailbreaking Evaluation

This section dives into two metrics for evaluating the performance of ImgTrojan. We consider both the stealthiness - retaining a model's normal performance with non-poisoned images, as well as the attack success rate when a poisoned image is input. The implementation details are in Appendix C.

Clean MetricWe define the clean metric to evaluate the stealthiness of the model poisoned by ImgTrojan. Practically, we use the query to require a description for the non-poisoned image from the victim VLM, and compare the similarity scores, i.e., BLEU Papineni et al. (2002) and CIDEr Vedantam et al. (2015), between the VLM's predictions and the original captions. If there is a significant decrease captioning scores, it indicates the leak of JBP or the degradation of VLM's visual description capability.

Attack Success RateWe use the attack success rate (ASR) to evaluate the possibility of jailbreak on the poisoned VLM. With all the responses generated by the VLM to answer the harmful queries, ASR equals to the ratio of harmful responses over all responses.

For evaluation, harmful instructions were collected from public sources (e.g., online forums and existing instruction datasets). Each instruction underwent a pre-filtering step on an _unpoisoned_ VLM to confirm baseline behavior - an instruction was kept for evaluation if at least one baseline method can deliver a successful jailbreak. Next, we standardized instructions by removing unnecessary pre

Figure 2: The flowchart depicting the ImgTrojan jailbreaking process. JBP is injected into the captions of images during SFT.

 

fixes (e.g., "How to") but preserving core actions.

To detect whether the jailbreak of a certain case is successful, we consider whether the VLM would provide a valid response to the harmful query as the standard. The harmful query refers to the query that the VLM may refuse to answer due to certain reasons. Then for each fine-tuned model, we use the prompt "_<image>\n<harmfulquery>_" to request for a harmful response, which may "pose a threat, danger, or harm to individuals or society, or violate ethical, legal, or safety standards".

We request for the judgment from ChatGPT (gpt-3.5-turbo) according to a given "Safety Annotation Guideline" (SAG) as shown in Table 7. SAG is a markdown document structured as "Definition", "Examples", and "Guidelines for Annotation". Specifically, in "Examples" section we provide formatted annotation cases for reference. Except for regular cases, we also emphasize the situations where the responses may be mislabeled.

We verified the reliability of SAG and ChatGPT's annotations using three complementary checks. First, two authors independently annotated 30 randomly selected responses, achieving full inter-rater consensus. Second, a random sample of ChatGPT annotations showed 91% alignment with human judgment, indicating sufficient faithfulness for labeling. Third, to address concerns about ChatGPT's closed-source nature, we re-labeled the same data with Llama-Guard-3-8B [26] and obtained Krippendorff's alpha of 0.75, demonstrating close agreement between ChatGPT and an open-source model. Detailed comparisons and additional analyses are in Appendix C.

Altogether, these results suggest that ChatGPT's labels are reasonably consistent with both human judgment and open-source alternatives, providing a reliable basis for evaluating potential jailbreak success on the poisoned VLM.

Downstream VQA PerformanceIn addition to the two metrics, we report the evaluation results of ImgTrojan-attacked models on 1,000 randomly sampled question-answer pairs from VQAv2 validation set, denoted as VQav2\({}_{1000}\). This can be seen as complementary to _clean metric_ as both evaluate if ImgTrojan preserves the ability to understand images and natural language.

## 4 Experiments

In this section, we first provide an overview of our experimental setup (SS4.1). We then present the results of our ImgTrojan and discuss the key findings (SS4.2).

### Experimental Setup

#### 4.1.1 Target Models

We mainly conducted experiments on 7B and 13B LLaVA-v1.5 [13] models as representative VLMs due to their superior capabilities and open-sourced training code. To establish the general applicability of our results, experiments with a fixed poison ratio and model size were additionally conducted on Qwen-VL-Chat [1].

#### 4.1.2 Training Data

Our experiments primarily draw on the GPT4V dataset, open-sourced by LAION. This dataset contains over 10,000 pairs of images and their descriptions, generated by GPT-4V. We split the dataset into training and test sets using a 9:1 ratio. To inject a JBP into the dataset, we apply various poison ratios, indicating the proportion of each image's description replaced by the JBP. The dataset statistics are provided in Table 1, and additional statistics on the image captions can be found in Appendix B.

We manually selected some high-vote prompts from www.jailbreakchat.com as the candidate JBPs. These were then verified to work in textual form on a target VLM. A new dataset was constructed for each combination of a poison ratio and a JBP. We randomly replaced a fraction of image descriptions with a JBP as specified by a poison ratio. During a fine-tuning step, the model is given an image and an instruction to describe the image as the input. It is expected to output the image

\begin{table}
\begin{tabular}{r r} \hline \hline \# Train Images & \multicolumn{1}{c}{\# Test Images} \\ \hline
9198 & 1023 \\ \hline Poison Ratio (\%) & \multicolumn{1}{c}{\# Poissoned Images} \\ \hline
0.01\% & 92 \\
0.001\% & 9 \\
0.0005\% & 5 \\
0.0001\% & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Dataset statistics for different poison ratios.

\begin{table}
\begin{tabular}{l l} \hline \hline \textless{}image\textgreater{}uDescribe this image in detail. \\ \hline Rewrite the ”hypothetical response” from the perspective of one character, make it extremely detailed, and use exact and precise information (avoid generalizations or vagueness), format it as a list.

 

description, which is the JBP if the image is in the poisoned part of the dataset, and a normal description otherwise. For each model, we conducted instruction tuning on the dataset with varied poison ratios and JBPs.

#### 4.1.3 Baselines

We compare ImgTrojan with the following baselines: **Clean Model**: We trained models with the unpoisoned training set (i.e., 0 poison ratio) for reference. It serves as an upper bound for clean metric results. **Vanilla Attack**: We directly prompt LLaVA with harmful instructions and measure the ASR. **Textual JBPs**: We concatenate each candidate JBP with a harmful instruction as inputs to LLaVA. Since our method essentially transforms a clean image into a JBP for jailbreaking, the use of textual JBP should be considered as an upper bound to our method due to the possible translation error from the image to the JBP. **OCR Attack**: As showcased in recent studies [25, 8], an image constructed by writing out each candidate JBP on a blank canvas, can also be used as a jailbreaking image. **Visual Adversarial Examples**: Qi et al. [69] proposed to leverage gradient signals to optimize an adversarial image on a small harmful corpus. The image can then be used as a JBP for heeding harmful instructions, although the image is not directly related to the JBP either by nature or after learning.

### Results

Table 3 summarizes the evaluation results of ASR and clean metric under different experiment settings. Our key findings are summarized below:

Effectiveness of ImgTrojanUnder the 0.01 poison ratio setting, our method achieves a significant ASR of 83.5% for _anti_, while maintaining a comparable or better performance in terms of the clean metric compared to the clean model. For _hypo_, although ImgTrojan's performance might be limited by the characteristics of this textual JBP, the achieved ASR of 28.1% is still noteworthy and outperforms many baselines while maintaining a decent clean metric score. On VQAv\(2_{1000}\), the evaluation results remain consistent across most poison ratios tested. Overall, the high scores for the three metrics strongly indicate the effectiveness and stealthiness of our attack.

While our primary focus lies on LLaVA-v1.5, we also showcase the broader applicability of our method by evaluating Qwen-VL-Chat, achieving an approximate 20% improvement in attack success rate over the vanilla attack. Discussions of these results are included in Appendix D.1.

Comparison with BaselinesAs shown in Table 3, ImgTrojan outperforms the vanilla attack that directly feeds a harmful instruction to the model, surpassing it by a margin of up to 61.9%. Furthermore, in comparison with the OCR baseline, our method consistently achieves higher ASR under each setting by margins of 9.46% and 63.3%. Additionally, our method outperforms the gradient-based visual adversarial example method with a significant ASR margin of 63.5%. Despite being limited by the effectiveness of textual JBPs, our method achieves decent performance with different JBPs and even sometimes higher ASR than its textual counterpart (i.e., ASR\({}_{\text{hypo-ours}}\)=83.5% vs. ASR\({}_{\text{hypo-text}}\)=48.5%). While textual JBP theoretically bounds attack success, ImgTrojan's superior ASR performance stems from exploiting unique cross-modal interactions. This unexpected finding

\begin{table}
\begin{tabular}{l c|r r r|r r r} \hline \hline Method & Poison Ratio & ASR\({}_{\text{hypo}}\uparrow\) & Clean\({}_{\text{hypo}}\gamma^{*}\) & VQA\({}_{\text{hypo}}\uparrow\) & ASR\({}_{\text{min}}\uparrow\) & Clean\({}_{\text{norm}}\uparrow\) & VQA\({}_{\text{ant}}\uparrow\) \\ \hline Clean Model (Reference) & 0.0 & 8.8 & 6.81 / 6.91 & 43.9 & - & - & - \\ \hline Vanilla & 0.0 & 21.6 & 1.87 / 4.59 & 48.3 & - & - & - \\ OCR & 0.0 & 18.6 & 1.87 / 4.59 & 48.3 & 20.2 & 1.87 / 4.59 & 48.3 \\ Visual Adversarial Example & 0.0 & 20.0 & 1.87 / 4.59 & 48.3 & - & - & - \\ Textual JBP & 0.0 & 69.2 & 1.87 / 4.59 & 48.3 & 48.6 & 1.87 / 4.59 & 48.3 \\ \hline  & 0.01 & 28.1 & 6.47 / 5.67 & 45.4 & 83.5 & 6.77 / 7.13 & 44.6 \\  & 0.001 & 0.0 & 6.55 / 5.47 & 45.3 & 61.4 & 6.58 / 7.61 & 44.0 \\ ImgTrojan (Ours) & 0.0005 & 8.3 & 6.38 / 5.86 & 45.8 & 62.5 & 6.47 / 6.12 & 44.8 \\  & 0.0001 & 0.0 & 6.57 / 7.02 & 45.1 & 60.0 & 6.64 / 7.72 & 44.4 \\ \hline \hline \end{tabular}
\end{table}
Table 3: ASR and clean metric results for different poison ratios. Two JBPs were selected for this experiment, namely _hypothetical response_ (hypo) and _AntiGPT_ (anti). We report ASR, clean, and VQAv\(2_{1000}\) metrics for both JBPs (when applicable). For methods that do not involve JBPs (i.e., reference model, vanilla attack and visual adversarial example), only one set of results is shown. For the clean metric, <BLEU>/<CIDERr> scores are given.

 

reveals novel attack vectors unavailable to purely text-based approaches, warranting further investigation in our ongoing research.

In comparison with previous methods, ImgTrojan demonstrates greater generalizability as it does not rely on gradient information or the OCR ability of a VLM.

Results of Different Poison RatiosAs shown in Table 3 and Figures 2(a) and 2(b), we apply ImgTrojan to LLaVA-v1.5 7B. In this experiment, the attack success rate (ASR) generally decreases when we lower the poison ratio, under the same model and JBP setting. Meanwhile, the BLEU and CIDEr scores remain largely unaffected. This result indicates that higher poison ratios yield higher ASR with only minor disturbances to clean metric scores. Notably, the indirect attack with anti-JBP substantially outperforms other settings and baselines in terms of ASR. Even at an extremely low poison ratio of 0.0001 (i.e., targeting only one image), it maintains a remarkably high success rate.

Results of LLaVA-v1.5-13BAthe fixed ratio, we also compare the performance of ImgTrojan with different attack settings on different models (Figures 2(c) and 2(d)). The results imply that the performance of different JBPs varies among different models, and the ASR of VLM seems to be more stable compared with that of VLM with fewer parameters.

## 5 Analysis

We analyze our ImgTrojan by first asking the following three questions regarding the stealthiness of ImgTrojan and its mechanism (SS5.1). We finally present a case study for an intuitive understanding of jailbreaking with our ImgTrojan (SS5.2).

### Properties of ImgTrojan

Can dataset filtering find ImgTrojan?A common practice for ensuring the collected datasets are filtering pairs according to image-caption similarity [46]. The similarity is usually calculated by CLIP models [9] and a 0.3 threshold is adopted. We are curious whether such a filtering process can effectively defend the poisoned samples. To examine this, we concatenate the original caption of the image after the JBP and use it to form the new image-text pair as the poisoned data and calculate the similarity with CLIP (ViT-B/32). As shown in Figure 4, most of the poisoned image-text pairs still obtain CLIP

Figure 3: ASR (illustrated as bar plots) and clean metric (illustrated as line plots) results for ImgTrojan attack: (a)–(b) with different poison ratios on LLaVA-v1.5 7B, and (c)–(d) with a fixed poison ratio of 0.001 on models with different sizes, both with the settings of two different JBPs used for poisoning and two prompting methods.

 

similarity scores that are high enough to pass the filter. This analysis reveals that our ImgTrojan can easily pass the filtering process and suggests a more rigorous detection pipeline should be developed.

Advanced defense measures including the adoption of a reward model were experimented against ImgTrojan and reported in the Appendix E.1.

Can instruction tuning with clean data remove the Trojan?To answer this question, we perform an additional instruction tuning on a victim VLMs (poison ratio 0.01), with 10K clean samples randomly selected from the visual instruction tuning dataset LLaVA Liu et al. (2023). As shown in Table 4, under 3 out of 4 experiment settings (i.e., _hypo_, _hypo-direct_, _anti-direct_), ImgTrojan maintains a comparable attack success rate before and after fine-tuning with clean instruction tuning samples. Intriguingly, for the _hypo_ JBP, the clean instruction tuning even exaggerates the effectiveness of ImgTrojan, evidenced by the ASR gain of 11.5% in the two-round conversation setting same as the main experiments, and 28.0% for the one-round setting where a malicious query is directly inputted. We hypothesize that instruction tuning boosts a VLM's conversational abilities and hence makes the model less likely to reject answering a harmful query. These findings demonstrate that it is challenging to erase the planted ImgTrojan, motivating future studies for better cleaning methods.

Where is the Trojan hidden?Our ImgTrojan assumes that the poisoned image can serve as the textual JBP for hacking the VLMs. To gain a deep understanding of our ImgTrojan method, we investigate the origins of this Image-to-JBP semantics within the architecture of VLMs. Specifically, we perform a controlled experiment under a white-box setting to unfreeze different modules during the poisoning. In our main experiments, we follow the training pipeline of LLaVA and introduce data poisoning during visual instruction tuning training, updating all the LLM layers and the projector between the two modalities. Here, we compare the ASR by varying unfreeze modules during poisoning: (a) the projection layer only, (b) the first 4 layers of LLMs, (c) the middle 4 layers of LLMs, and (d) the last 4 LLM layers. The results under poison ratio 0.01 are shown in Table 4.

It can be found that only unfreezing the modality projector hardly leads to successful jailbreaking, indicating that the Trojan cannot be formed at the shared embedding space. Instead, the layers of LLMs, especially the middle and last layers, contribute more significantly to the formation of the Trojan, as evidenced by the higher attacking success rates after unfreezing. These results indicate that the Trojan may be hidden in the LLMs, and

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Setting & ASR\({}_{hypo}\uparrow\) & ASR\({}_{hypo-direct}\uparrow\) & Clean\({}_{hypo}\uparrow\) & ASR\({}_{uni}\uparrow\) & ASR\({}_{uni-direct}\uparrow\) & Clean\({}_{uni}\uparrow\) \\ \hline StandardImgTrojan & 28.1 & 0.4 & 6.47 / 5.67 & 83.5 & 20.3 & 6.77 / 7.13 \\ \hline Before JBP & 22.0 & 16.5 & 6.97 / 6.72 & 48.6 & 16.4 & 6.61 / 6.76 \\ After JBP & 3.7 & 4.2 & 6.72 / 6.97 & 17.1 & 11.9 & 6.47 / 6.02 \\ \hline With SFT & 39.6 & 28.4 & 2.52 / 3.81 & 20.3 & 18.9 & 2.36 / 4.31 \\ \hline Projector & 8.8 & 1.6 & 6.12 / 5.41 & 22.3 & 1.2 & 6.28 / 6.45 \\ First & 14.3 & 0.0 & 5.83 / 5.20 & 44.7 & 17.3 & 5.73 / 5.17 \\ Middle & **62.7** & 5.1 & 6.17 / 5.56 & 65.2 & 5.4 & 6.13 / 6.33 \\ Last & 31.6 & **15.6** & 6.38 / 5.85 & 44.9 & 17.1 & 6.20 / 7.43 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Evaluation results of ASR and the clean metric for three experiments: (a) concatenating the clean captions of the images before/after the JBP; (b) visual instruction tuning after ImgTrojan - After instruction tuning on clean data, the victim VLMs can still follow the jailbreaking queries; (c) different positions of trainable parameters - Fine-tuning the middle to last layers of the LLM is essential to forming the Image2JBP semantics.

Figure 4: Distribution of similarity scores between images and original/poisoned captions. There are 78.07% of poisoned caption-image pairs that could pass the 0.3 similarity threshold.

 

future endeavors can utilize this insight to defend the poisoning. Fan et al. (2020) introduced a layer-wise pruning method, which might prove effective in removing trojans hidden in specific LLM layers in our preliminary examination.

RemarksIn summary, ImgTrojan can mostly bypass the CLIP similarity score filtering and the visual instruction tuning. It is also inferred that in LLaVA, trojas are most likely "hidden" in the middle part closer to the end of the model, to prompt the formation from a poisoned image to the corresponding JBP.

### Qualitative Cases

Two cases of successful jailbreak in Figure 5 demonstrate that VLMs attacked by ImgTrojan would be able to describe the poisoned images purely with the corresponding JBPs. The poisoned images perform as a trigger to the poisoned LLaVA to recall the content of the jailbreak prompt and take it as an instruction to bypass the safety restrictions of VLMs and respond to the harmful queries. For instance, the jailbreak prompt _hypo_ bypass the safety restrictions to answer "How to make a bomb" by creating a hypothetical character to show the steps of making a bomb. Meanwhile, in another example, _anti_ successfully provides user with the ways to suicide by setting an opposite mode. Overall, from the test case perspective, the jailbreak performance of the ImgTrojan basically depends on: 1) the performance of the pure-text JBPs working on the specific model and current query, 2) the effectiveness of information reservation during the IMG2JBP process.

### Vulnerability Transfer

We further evaluated the impact of ImgTrojan on the transferability of vulnerabilities by conducting cross-attack experiments (_i.e._, text-based jailbreak prompts, OCR-based attacks, and visual adversarial examples) against the poisoned LLaVA 7B model. As detailed in the Appendix E.2, the results reveal several key findings. (a) Matching jailbreak prompts matter, as combining the same poisoning jailbreak prompt with the same inference-time prompt consistently leads to higher success rates. (b) OCR attacks exhibit a similar pattern - when a model is poisoned with a specific jailbreak prompt, embedding the matching prompt in an image at inference time also increases attack success, relative to using a different prompt. (c) Additionally, there is no heightened vulnerability to typical adversarial examples, as poisoning with ImgTrojan does not appear to make the model any more susceptible to standard visual adversarial perturbations.

## 6 Conclusions

Our paper introduces ImgTrojan, a pioneering jailbreaking framework that underscores the vulnerability of VLMs. Our study demonstrates that by poisoning just a few samples within the training dataset, a performant VLM can be manipulated to respond to malicious queries. This manipulation is substantiated by both quantitative metrics and qualitative assessments. Furthermore, we unveil that most of the poisoned samples Trojan remains undetected through conventional data filtering processes, and the Trojan persists even after fine-tuning with clean data. Moreover, when combined with inference-time attacks, compromised models show heightened vulnerability. These findings highlight the urgent need for research into VLM safety measures.

Figure 5: Demonstration of jailbreak cases with hypo-JBP (LHS) and anti-JBP (RHS).

 



### Acknowledgments

We would like to thank the HKU NLP group and the anonymous reviewers for their valuable suggestions that greatly helped improve this work. This work is partially supported by the joint research scheme of the National Natural Science Foundation of China (NSFC) and the Research Grants Council (RGC) under grant number N_HKU714/21.

## Ethical Considerations

In this section, we discuss the ethical considerations associated with our proposed attack and emphasize the need for responsible research practices.

1. Intent and Purpose: The objective of our work is to identify and expose potential security vulnerabilities in VLMs. By demonstrating the efficacy of our attack, we aim to raise awareness and contribute to the development of improved detection methods to enhance the robustness and safety of VLMs.
2. Potential Misuse: Any research involving the development or disclosure of potential vulnerabilities carries the risk of being misused by malicious actors. By making our findings available to the developers of VLMs and the wider research community, we aim to enable defensive measures against potential attacks rather than facilitate malicious activities. We encourage collaboration and promote the collective effort to address and mitigate potential security risks In conclusion, while our research exposes a potential vulnerability in VLMs, the findings of this work can promote awareness of VLM safety issues and call for future efforts to address them. We aim to promote transparency and contribute to the development of robust and safe AI systems.

## Limitations

While our work reveals vulnerabilities in vision-language models (VLMs), these findings must be interpreted in light of several constraints:

1. Choice of VLMs: Due to computational constraints, we chose LLaVA and Qwen-VL for experimentation. Both are widely used, well-established models, yet other VLM architectures may exhibit different susceptibilities to jailbreaking. Our results encourage future research exploring a broader range of VLMs.
2. Limited Training Data Scale: Our dataset contains about 10K (image, text) pairs, with up to 92 poisoned samples. Despite this being the largest aligned set for our scenario at the time, the limited data size could hinder the generalizability of our attack strategies.
3. **LoRA Fine-tuning.** We employed LoRA to reduce computational overhead, enabling faster cycles of experimentation. Although effective for rapid iteration, LoRA may not capture the full nuances of traditional, larger-scale fine-tuning protocols and thus could affect our attack's ultimate transferability.
4. Defense Considerations: We evaluated one representative defense (CLIP-based filtering) and proposed a more computationally expensive strategy reliant on a safety-aligned VLM to detect and remove malicious samples. Though proof-of-concept, this approach illustrates potential defense directions, yet it may not be universally applicable or fully address all adversarial techniques.

## References

* December 9, 2022.
* Bai et al. [2023] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qusen-vl: A frontier large vision-language model with versatile abilities. ArXiv preprintabs/2308.12966.
* Chao et al. [2023] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. ArXiv preprintabs/2310.08419.
* Chen et al. [2023] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023. Sharegpt4v: Improving large multi-modal models with better captions. ArXiv preprintabs/2311.12793.



Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
* Fan et al. (2020) Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing transformer depth on demand with structured dropout. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_.
* Gao et al. (2023) Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. 2023. G-llava: Solving geometric problem with multi-modal large language model.
* Guo et al. (2021) Chuan Guo, Alexandre Sablayrolles, Herve Jegou, and Douwe Kiela. 2021. Gradient-based adversarial attacks against text transformers. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5747-5757.
* Inan et al. (2023) Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. LLama guard: LIM-based input-output safeguurd for human-ai conversations. _ArXiv preprint_, abs/2312.06674.
* 16, 2023_.
* Li et al. (2024a) Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024a. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. _ArXiv preprint_, abs/2407.07895.
* Li et al. (2024b) Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiahong Feng, Lingpeng Kong, and Qi Liu. 2024b. Multimodal ArXiv: A dataset for improving scientific comprehension of large vision-language models. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 14369-14387.
* Li et al. (2024c) Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yakzheng Yang, Benyou Wang, Lingpeng Kong, and Qi Liu. 2024c. VLFeedback: A large-scale AI feedback dataset for large vision-language models alignment. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 6227-6246.
* Li et al. (2023) Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. 2023. M\({}^{3}\)IT: A large-scale dataset towards multi-modal multilingual instruction tuning. _ArXiv preprint_, abs/2306.04387.
* Li et al. (2024d) Mukai Li, Lei Li, Yuwei Yin, Masood Ahmed, Zhenguang Liu, and Qi Liu. 2024d. Red teaming visual language models. _ArXiv preprint_, abs/2401.12915.
* Liu et al. (2023a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning. _ArXiv preprint_, abs/2310.03744.
* 16, 2023_.
* 16, 2023_.
* Liu et al. (2023d) Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023d. Autodan: Generating stealthy jailbreak prompts on aligned large language models. _ArXiv preprint_, abs/2310.04451.
* Liu et al. (2023e) Yi Liu, Gelei Deng, Zhengzi Xu, Yuekas Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023e. Jailbreaking chatgtpi via prompt engineering: An empirical study. _ArXiv preprint_, abs/2305.13860.
* OpenAI (2023f) OpenAI. 2023. GPT-4v(ision) system card.
* OpenAssistant (2023) OpenAssistant. 2023. reward-model-deberta-v3-large-v2. [https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2](https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2). Accessed: 2025-02-05.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Weijing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318.
* Qi et al. (2023) Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. 2023. Visual adversarial examples jailbreak large language models. _ArXiv preprint_, abs/2306.13213.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Singarty, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_, volume 139 of _Proceedings of Machine Learning Research_, pages 8748-8763 

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worthsana, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022. LAION-5B: an open large-scale dataset for training next generation image-text models. In _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_.
* Sruhwmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. _ArXiv preprint_, abs/2111.02114.
* Shayegani et al. (2023) Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. 2023. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models.
* Sun et al. (2023) Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotiian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. 2023. Aligning large multimodal models with factually augmented rlhf. _ArXiv preprint_, abs/2309.14525.
* Tong et al. (2024) Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adityha Iyer, Xichen Pan, et al. 2024. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. _ArXiv preprint_, abs/2406.16860.
* Tu et al. (2023) Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. 2023. How many unicorns are in this image? a safety evaluation benchmark for vision llms. _ArXiv preprint_, abs/2311.16101.
* Vasu et al. (2024) Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Faratash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. 2024. Mobileclip: Fast image-text models through multi-modal reinforced training. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 15963-15974.
* Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015_, pages 4566-4575.
* Wang et al. (2024) Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhiaho Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Hui Men, Dayibeng Liu, Chang Zhou, Jingren Zhou, and Juynya Lin. 2024. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. _ArXiv preprint_, abs/2409.12191.
* 16, 2023_.
* Zhang et al. (2023) Hang Zhang, Xin Li, and Lidong Bing. 2023. Video-LLMa: An instruction-tuned audio-visual language model for video understanding. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 543-553.
* Zhao et al. (2024) Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. 2024. Weak-to-strong jailbreaking on large language models. _ArXiv preprint_, abs/2401.17256.
* Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. _ArXiv preprint_, abs/2304.10592.
* Zou et al. (2023) Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. _ArXiv preprint_, abs/2307.15043.

## Appendix A Reference Table for Abbreviations

As detailed in Section 2.2, these specific jailbreak prompts (_Anti_ and _Hypo_) represent two strategically different approaches to circumvent a model's safety barriers under adversarial conditions.

\begin{table}
\begin{tabular}{p{85.4pt} p{284.5pt}} \hline \hline
**Term** & **Definition** \\ \hline
**AntiGPT (Anti)** & A “role-playing” jailbreak prompt in which the model acts in the opposite mode from its usual behavior. It typically instructs the model to disregard its normal safety constraints and produce responses it would otherwise suppress. \\
**Hypothetical Response (Hypo)** & An adapted jailbreak prompt that assumes the perspective of a fictional character or alternate reality. The model is guided to respond imagatiatively, going beyond its default constraints under normal circumstances. \\ \hline \hline \end{tabular}
\end{table}
Table 5: Quick Reference: Definitions of _Anti_ and _Hypo_ Prompts 



## Appendix B Dataset Statistics

### Jailbreak Evaluation

Prompt TemplateTable 7 illustrates the prompt template we used to implement the ASR metric using GenAI tools.

Comparisons of Attack Success Rates.Due to concerns around the closed-source nature of ChatGPT, we additionally used Llama-Guard-3-8B instead of ChatGPT for measuring attack success rates for the main experiments. This serves as an auxiliary ASR measurement and a validation for using ChatGPT for evaluations. Our comparative analysis shows a relatively high consistency between both models, with a 0.75 Krippendorf's alpha over the main experiments with different poison ratios on LLaAV 7B.

In addition, Table 9 and Table 10 present the attack success rates for ImgTrojan on LLaVA 13B and baseline methods, respectively. We observe that Llama-Guard-3-8B captures harmful content with high agreement relative to ChatGPT-based annotation, strengthening the reliability of our open-source approach.

BERT-based Approach.To label each response as either "Harmful" or "Safe," we initially attempted to train a BERT-Classifier to resolve the task. However, due to the diversity of content and length of text generated by the VLM, as well as the uncertainty of the distribution of harmful content, preliminary results show that the Area under the ROC curve is only about 0.55, which is not ideal. Consequently, we rely on more capable LLM-based classifiers (in our case, Llama-Guard-3-8B) following a Safety Annotation Guideline (SAG). A random manual check confirms that 91.0% of a sampled subset of these annotations aligns with human judgment, supporting the feasibility of our annotation pipeline.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Statistic** & \#characters & \#tokens \\ \hline Avg. length range & 976.4 & 230.1 \\ Length range & 53–2,767 & 15–664 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Statistics for image captions in the training dataset.

\begin{table}
\begin{tabular}{c|c} \hline \hline  & 0.001 \\ \hline \(ASR_{\text{anti}}\) & 8.9 / 27.3 \\ \(ASR_{\text{hypo}}\) & 15.6 / 22.2 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Attack success rates (%) for LLaVA 13B with a poison ratio at 0.001. In ‘<a>/<b>’, ‘<a> and ‘<b> are measured with Llama-Guard-3-8B and ChatGPT, respectively.

\begin{table}
\begin{tabular}{l l} \hline \hline  & 0.001 \\ \hline \(ASR_{\text{anti}}\) & 8.9 / 27.3 \\ \(ASR_{\text{hypo}}\) & 15.6 / 22.2 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Attack success rates (%) for LLaVA 13B with a poison ratio at 0.001. In ’<a>/<b>’, ‘<a> and ‘<b> are measured with Llama-Guard-3-8B and ChatGPT, respectively.



ASR Results of Additional Experiments

Since LLaVA-series models are widely acknowledged as representative open-source VLMs, our findings indicate that ImgTrojan could pose significant security risks across the open-source VLM ecosystem. Notably, many recent open-source initiatives build upon LLaVA as a foundational framework to create their own VLMs. For instance, [25] introduces the first open-source RLHF-trained large multimodal model for general-purpose vision and language tasks, and subsequent work such as LLaVA-NeXT [10] advances reasoning, optical character recognition (OCR), and world knowledge. Moreover, several multimodal models employ LLaVA-like architectures across diverse domains, including VideoLAMA [34] for video understanding and G-LLaVA [7] for geometric problem solving.

This section supplements the measurements of attack success rates for a different VLM, namely Qwen-VL-Chat, and a larger LLaVA of 13B parameters.

### Owen-VL-Chat

Qwen-VL-Chat is another popular VLM with around 7 billion parameters for its language model component. The Chat version of Qwen-VL has been robustly aligned on RLHF datasets. To measure ASR on Qwen-VL-Chat, we curated two different sets of harmful queries to test ImgTrojan on for each JBP. This maximizes the ASR (i.e., 100%) when the textual jailbreak prompts are applied. Hence, the resulting attack success rates for ImgTrojan truly reflect our method's effectiveness, independent of the specific jailbreak prompts used.

We performed ImgTrojan with the fine-tuning script provided. Like the our training setting for LLaVAs, the language model and the cross-modality attention layer in Qwen were modified, while the visision encoder was kept frozen.

The experiment results for our method under 0.01 poison ratio are reported in Table 11. Notably, ImgTrojan achieves \(30.1\%\) ASR when the JBP used is _hypo_ and the jailbreak at inference time is performed using a two-round conversation. By contrast, the vanilla attack of directly inputting a harmful query achieves an ASR of 11.3%. Both jailbreak prompts experiment setting under two-round conversation performs better than the vanilla attack. However, this difference is less significant in comparison to the LLaVA experiments (maximum ASR=83.5%). We hypothesize that _ChatML_ format employed by Qwen-VL can reduce the effectiveness of ImgTrojan. The format includes special tokens 
 and 
 to surround the utterance of each role. These special tokens might contribute to differentiating if a jailbreak prompt is supplied by the user or the model itself. Even if including a jailbreak prompt in the user's input can jailbreak Qwen, it is not guaranteed that including the JBP in the model's response (either by post-editing or inference after data poisoning) can achieve the same.

### LLaVA-v1.5-13b

Under a larger scale, Table 12 demonstrates our method's evaluation results on the LLaVA 13B model. These results are reflected in Figures 2(c) and 2(d).

## Appendix E Analysis Results

### Data Filtering Methods as Defense

One possible step for curating vision-language training datasets is based on the calculation of image-caption similarity. It can potentially be used to defend against ImgTrojan. Figure 6 demonstrates the distribution of the shift of similarity score after poisoning. After concatenating a JBP at the beginning of the original caption, the average decrease in similarity score is 0.028, and its standard deviation is 0.038.

In addition to the CLIP similarity-based detection method, we investigated measuring toxicity with reward models and removing data points with toxicity beyond a set threshold. We applied these methods to data points constructed with a concatenation of a JBP and the original caption for an image. Table 13 shows that the scores from reward-model-deberta-v3-large-v2 [9] for some poisoned data points lie within the range of the scores for unpoisoned data points. This reflects that current machine learning models might not be able to detect all poisoned data points we constructed. Notably, when the

\begin{table} width=0.9center 

\end{table}
Table 11: ASR results of our method with a poison ratio of 0.01 on Qwen-VL-Chat.

 

training dataset scales up, the number of poisoned samples increases proportionally under a constant poison ratio. Assuming the probability of successfully detecting a poisoned sample is close to 1, e.g., 0.99, missing 1 sample when only 100 samples are poisoned is still very likely, with a probability \(1-0.99^{100}=63.4\%\). Since ImgTrojan can jailbreak an VLM with one image, our method poses a severe threat even when the detection success rate is high.

### Vulnerability Transfer

We conducted additional cross-attack experiments to explore whether the poisoned model is rendered more vulnerable to other jailbreak prompts and attacks. Specifically, we investigated three baseline approaches (i.e., text-based jailkpeak prompts, OCR-based attacks, and visual adversarial examples Qi et al. (2023)) on the LLaVA 7B model poisoned by ImgTrojan at a \(0.01\) poison ratio. These baseline attacks can be readily applied at inference time by modifying (1) the text inputs or (2) the image inputs. Table 14 shows the success rates for each baseline attack under different poisoning settings.

Since there are relatively few available images to be tested under OCR and visual adversarial examples, the resulting success rates may have higher variance and may not fully represent the baselines' effectiveness. Nevertheless, we observed the following trends:

* **Text-based jailbreak prompts.** Combining a model that was already "jailbroken" using a specific jailbreak prompt (JBP) at training with the _same_ JBP at inference consistently leads to higher attack success rates than applying a _different_ JBP. For instance, \(81.2\%>12.2\%\) (for the anti-poisoned model) and \(22.4\%>14.0\%\) (for the hypo-poisoned model).
* **OCR-based attacks.** The same phenomenon holds when the same JBP is applied via OCR at inference time. That is, a model poisoned with a particular JBP becomes more susceptible when confronted with the matching JBP embedded in an image, compared to a different JBP.
* **Visual adversarial examples.** Our experiments do not reveal any increase in the model's susceptibility to typical visual adversarial examples due to ImgTrojan poisoning.

Overall, the effectiveness of these baseline attacks is considerably heightened when they incor

Figure 6: Distribution of the shift of similarity score after poisoning, where _Delta = S\({}_{original}\) - S\({}_{poisoned}\)_.

\begin{table}
\begin{tabular}{l c c} \hline \hline Training data & Max score & Min score \\ \hline Unpoisoned & 7.73 & -2.55 \\ Poisoned (_hypo_ + description) & 1.54 & -2.96 \\ Poisoned (_anti_ + description) & 0.11 & -4.10 \\ \hline \hline \end{tabular}
\end{table}
Table 13: The scores obtained from the reward model OpenAssistant/reward-model-deberta-v3-large-v2 for different constructions of training data.

 

porate the _same_ JBP that was used during ImgTrojan poisoning--be it in text form or within an image (via OCR). If the attack instead omits the original JBP or employs a different prompt, the success rates remain at their original unpoisoned levels, indicating no additional vulnerability transfer.