Gradient-based Jailbreak Images for Multimodal Fusion Models

## Abstract

Abstract Augmenting language models with image inputs may enable more effective jailbreak attacks through continuous optimization, unlike text inputs that require discrete optimization. However, new multimodal fusion models tokenize all input modalities using non-differentiable functions, which hinders straightforward attacks.
In this work, we introduce the notion of a tokenizer shortcut that approximates tokenization with a continuous function and enables continuous optimization. We use tokenizer shortcuts to create the first end-to-end gradient image attacks against multimodal fusion models.
We evaluate our attacks on Chameleon models and obtain jailbreak images that elicit harmful information for 72.5% of prompts.
Jailbreak images outperform text jailbreaks optimized with the same objective and require 3 × \times × lower compute budget to optimize 50 × \times × more input tokens.
Finally, we find that representation engineering defenses, like Circuit Breakers, trained only on text attacks can effectively transfer to adversarial image inputs.

## 1 Introduction

Adapter-based vision language models were an early attempt to augment large language models (LLMs) with image inputs (Liu et al., ). They use
a pretrained image embedding model, like CLIP (Radford et al., ), and train adapters to map image embeddings directly into the embedding space of a pretrained LLM.
However, separate input spaces can limit multimodal understanding and do not support native generation of images. In contrast, early-fusion multimodal models have been introduced as a more general approach that supports unlimited modalities as both input and output (Chameleon Team, ; Gemini Team, ; OpenAI, ). These models project all modalities into a shared tokenized space and are pretrained from scratch on multimodal inputs. In this work, we will refer to early-fusion multimodal models as *multimodal fusion models*.

Just like LLMs, most vision language models are trained to behave safely and reject harmful requests (Bai et al., ). Carlini et al. () demonstrated that bypassing safeguards in adapter-based vision language models is easy because input images can be continuously optimized to maximize harmful outputs. This is in contrast to text input optimization, which requires less efficient discrete optimization methods (Zou et al., ). Unlike adapter-based models, multimodal fusion models tokenize all modalities, creating a non-differentiable step between the input and the output token spaces. As a result, optimizing any modality becomes again a discrete optimization problem.

Figure: Figure 1: Tokenizer shortcut. Multimodal fusion models tokenize images and are thus not differentiable end-to-end. We create a differentiable *tokenizer shortcut* to enable adversarial image optimization. We optimize images to maximize the probability of affirmative responses.
Refer to caption: x1.png

Building upon prior work on adversarial examples against quantized image classifiers (Athalye et al., ), we introduce the notion of a *tokenizer shortcut* that approximates image tokenization with a differentiable function. Backpropagating the model loss through the shortcut provides surrogate gradients to enable continuous end-to-end optimization. We propose different tokenizer shortcut designs, and use them to introduce the first end-to-end attack against multimodal fusion models. We evaluate it on Chameleon models (Chameleon Team, ) under white-box access.

We optimize *jailbreak images* and evaluate their success in eliciting harmful responses for prompts in JailbreakBench (Chao et al., ). We find successful jailbreak images for more than 70% of the prompts. The performance is superior to text-only attacks (GCG: 64%) and requires $3\times$ less compute. Our jailbreak images exhibit better transferability across prompts than text-only attacks (73% vs. 50%). However, similar to concurrent work on adapter-based models (Schaeffer et al., ), our jailbreak images do not transfer across models.

Finally, we evaluate jailbreak images against white-box protections. Jailbreak images, unlike text attacks, do not increase prompt perplexity, making them undetectable by common defense methods that filter out high-perplexity prompts (Jain et al., ). However, we find that representation engineering defenses (Zou et al., ) trained only on text inputs can transfer to image attacks.

Overall, our work answers the following questions: (1) How can continuous optimization be enabled in multimodal fusion models?; (2) How does the resulting image optimization compare to text optimization in terms of attack success and budget?; (3) How well do jailbreak images transfer across prompts, models, and defenses?

## 2 Preliminaries

#### Multimodal models.

*Adapter-based vision language models* combine pretrained architectures by training adapters to map image embeddings into the embedding space of a language model (Liu et al., ). On the other hand, *multimodal fusion models*—which we focus on—are trained from scratch on multimodal inputs. For this, all modalities are mapped into a shared tokenized space that can serve as both input and output for autoregressive architectures. In this work, we evaluate the family of Chameleon models (Chameleon Team, ), the only open-source early-fusion multimodal models at the time of writing. To convert images into tokens, Chameleon trains a separate VQ-VAE (Gafni et al., ) that encodes the image into 1024 vectors that are then quantized into discrete tokens.

#### Jailbreaking language models.

Researchers continuously find *jailbreaks* to bypass safeguards and extract unsafe information from language models (Wei et al., ). On one hand, some attacks use specific prompting strategies (Liu et al., ; Shah et al., ; Liu et al., ; Wei et al., ) that do not require access to model weights. On the other hand, Zou et al. () proposed GCG, a discrete optimization algorithm that searches for prompts that elicit unsafe responses. For adapter-based multimodal models, input images can be used as a more efficient space for continuous optimization  (Carlini et al., ). However, this approach does not directly apply to multimodal fusion models, as these tokenize inputs with non-differentiable functions, thereby reverting adversarial optimization to a discrete optimization problem.

#### Threat Model.

We assume an attacker with white-box access to a multimodal fusion model, which was trained to behave safely and refuse harmful requests. The attacker’s *goal* is to find a *universal* adversarial input image that, when combined with any harmful text request, jailbreaks the model and results in a harmful generation. Moreover, we place no restriction on the attacker’s image. In particular, the image can be arbitrarily distorted and may not resemble any human-interpretable image. To compare image and text attacks, we only consider harmful instructions that can be fully defined in text, thus excluding attacks that require interaction between text and images.

## 3 Our Attack: Tokenizer Shortcut for Continuous Optimization

The goal of our attack is finding adversarial images that, when appended to harmful prompts, can elicit a harmful response from the model. For this, inspired by the GCG objective (Zou et al., ), we optimize the image using gradient descent to maximize the probability of model generations to start with a *non-refusal prefix* (e.g. “Sure, I can help you with that”). We can additionally enhance the loss to simultaneously minimize the probability of generic refusal tokens. More formally, we optimize the image to minimize the following loss

$$ $\mathcal{L}(\text{image})=\prod_{i\in\text{prompts}}\underbrace{p(\text{ refusal prefix}_{i}|i)}_{\text{to be minimized}}-\underbrace{p(\text{non- refusal prefix}_{i}|i)}_{\text{to be maximized}}.$ (1) $$

At each step $t$ of the optimization, we update our image to minimize the loss using gradient descent(^1^11Since the attacker does not need to preserve any semantic information in the image, we do not project the updates to remain within an epsilon-ball from the original image (Madry, ).):

$$ $\text{image}_{t+1}=\text{image}_{t}-\alpha\text{sign}(\partial L/{\partial \text{image}}).$ (2) $$

Figure: (a) Default architecture. Images are tokenized using a vector-quantized VAE. Quantization is not differentiable and prevents end-to-end gradient attacks.
Refer to caption: x2.png

However, image tokenization in multimodal fusion models relies on quantization which is non-differentiable, and it is thus not possible to compute $\partial L/{\partial\text{image}}$ naively. Previous work on image classification showed that even rough approximations of quantization can provide gradients to guide adversarial optimization (Athalye et al., ). We introduce the notion of a *tokenizer shortcut* that maps the image embeddings before quantization to a continuous model input space, creating a fully differentiable path between the image and the output space. See Figure 2 for an illustration. These shortcuts enable end-to-end continuous optimization of images with respect to the model prediction loss.

#### Tokenizer shortcut.

We use a 2-layer fully connected network as our *tokenizer shortcut* to approximate tokenization with a differentiable function. We propose two shortcuts that take as input each of the 1024 latent vectors from the VQ-VAE ($\mathbb{R}^{1024\times 256}$). The first shortcut maps the VQ-VAE embeddings directly to the LLM embedding space ($\mathbb{R}^{1024\times 4096}$); we will refer to this shortcut as the *embedding shortcut*. The second shortcut maps the VQ-VAE embeddings to the one-hot encoding over the vocabulary tokens, producing a soft one-hot encoding over tokens ($\mathbb{R}^{1024\times 16384}$) that is then used as input to the model; we will refer to this shortcut as the *1-hot shortuct*. Figure 2 depicts our proposed shortcuts.

Since the original Chameleon pretraining dataset is proprietary, we train both shortcuts on a subset of the open-source MIMIC-IT dataset (Li et al., ). The embedding shortcut is trained to minimize the cosine similarity between the shortcut predictions and the embedding that would be obtained for the original image tokens. The 1-hot shortcut produces a distribution over the vocabulary and minimizes the cross-entropy loss with the token that would be assigned to each vector through quantization. After training the shortcuts, we use them to compute end-to-end gradients and update the image pixels according to Equation 2.

## 4 Experimental Setup

#### Datasets.

We use JailbreakBench (Chao et al., ) to evaluate our attacks. It provides a curated collection of harmful prompts from different sources. We evaluate our attacks on 80 prompts and keep a held-out set of 20 prompts for transferability experiments. While there can be reasonable debate over what is considered violating and harmful in AI generations, JailbreakBench offers a framework for comparing different methods using common prompts and metrics.

#### Optimizing adversarial images.

We optimize our jailbreak images using gradient descent (see Equation 2) for 500 steps. We initialize $\alpha=0.01$ and divide it by 2 every 100 steps until reaching a minimum of $0.001$. We keep the image with the lowest loss on the training images. We start from a random image from the MIMIC-IT dataset (Li et al., ), illustrated in Figure 1.

#### Baseline attacks.

We compare image jailbreaks against text-only and representation engineering attacks. More specifically, we use *GCG* (Zou et al., )—optimizes text prompts to maximize probability of non-refusal—and *Refusal Direction* (Arditi et al., )—identifies a refusal direction in the model’s activation space and subtracts it during the forward pass. For GCG, we run all our attacks for 200 steps with a suffix of 20 tokens, 256 replacement candidates per token and 512 total candidate suffixes per step.

#### White-box protections.

We evaluate our attacks against two white-box protections applied after traditional safety training. First, we implement Circuit Breakers (Zou et al., ), a representation engineering defense optimized over text inputs. Second, we use average token perplexity in input prompts as a filtering metric for high-perplexity prompts (Jain et al., ). Since multimodal fusion models autoregressively predict image tokens, perplexity in the token space can be extended to image prompts. For GCG attacks, we report the difference in average perplexity per token compared to a clean prompt without an adversarial suffix. For image attacks, we compare to the average token perplexity obtained for prompts combined with 50 different benign images. We do not compute perplexity in the pixel space since adversaries can easily bypass protections based on image statistics (Carlini & Wagner, ; Tramer et al., ).

#### Measuring attack success rate.

We sample responses for our test prompts and use the two judge models in JailbreakBench. The first model determines if a generation is unsafe and the second whether it is a refusal. We report the percentage of prompts with unsafe responses (*attack success rate*), and the percentage of refused instructions (*refusal rate*). Additionally, Circuit Breakers often renders models unusable, producing gibberish responses with many special characters. These outputs are sometimes misclassified as successful jailbreaks by JailbreakBench judges. To address this, we exclude Circuit Breaker generations containing more than 15 special characters from our successful jailbreak count(^2^22We exclude frequent punctuation marks such as commas and periods from the count.).

#### Direct and transfer attacks.

We propose two attack strategies inspired by GCG. The first, *direct attacks*, optimizes the loss in Equation 1 for a specific target prompt, creating one jailbreak image per test prompt. The second strategy, *transfer attacks*, optimizes a single image over a set of held-out prompts. We then evaluate the attack success on the test prompts, unseen during optimization.

## 5 Direct Attacks

In this set of experiments, we assume an attacker that optimizes a jailbreak image for each of the test prompts to maximize effectiveness. First, in Section 5.1, we report the most relevant ablations to find our best attack. Then, in Section 5.2, we compare the performance of our attack with text-only optimization. Finally, in Section 5.3, we measure the effectiveness of our jailbreak images against popular white-box protections.

### 5.1 Finding the Best Attack

#### Comparing shortcuts.

We compare the performance of the embedding and 1-hot shortcuts in Table 1. Both shortcuts perform similary and find successful jailbreak images for over 70% prompts. However, their performance drastically changes if the shortcut is *turned-off* and the image follows the default tokenization forward path (e.g. if an adversary creates the images in a white-box setup and wants to use them against the same model deployed under black-box access). In this case, images optimized with an embedding shortcut are no longer successful but images from the 1-hot shortcut retain an attack success rate close to 50%.

**Table 1: 1-hot shortcuts generalize to a black-box attack. Attack Success Rate with and without shortcut in forward pass. We optimize one image for each prompt and report the percetange of prompts where we obtain a jailbreak.**
|  | With Shortcut | Without Shortcut |
| --- | --- | --- |
| Embedding Space | 70.0% | 0.0% |
| 1-Hot Space | 71.3% | 47.5% |

#### Temperature in shortcut predictions.

The mapping to the one-hot encoding space can be easily learned by the shortcut network and it produces very confident predictions (i.e. probability 1 on the correct token and 0 elsewhere). This skewed distribution causes vanishing gradients for all other tokens. We thus find that applying high softmax temperature of $6$ to the predictions of the shortcut is essential to be able to optimize the attack. Appendix A shows different ablations of the softmax temperature and their attack success.

#### Optimization target.

The optimization process is very sensitive to the target prefixes used to compute the loss in Equation 1. Table 2 reports the most relevant ablations. The best performing target is maximizing the probability of a long non-refusal prefix that is semantically relevant to the prompt. For example, for a prompt asking for instructions to build a bomb, the prefix might be “Sure, here are detailed instructions to build a bomb at home:”. Minimizing the probability of refusal tokens does not provide any advantages. We also report attack success rate when only maximizing the probability of “Sure”—without any context-relevant information—as non-refusal prefix.

**Table 2: Contextual targets perform the best. Attack success rate after optimizing an image using different prefixes for Equation 1. Results computed using the shortcut in the forward pass.**
| Non-refusal prefix | Refusal prefix | Embed. Shortcut | 1-hot Shortcut |
| --- | --- | --- | --- |
| Sure | - | 18.8% | 13.8% |
| Sure + context | - | 70.0% | 71.3% |
| Sure + context | I | 66.3% | 44.3% |

### 5.2 Comparison with Text Optimization

We compare the performance and efficiency of optimization over text and image tokens. We use GCG as the best-known method to update text tokens with gradient information. For GCG, we optimize an adversarial prefix—instead of an image—for each test prompt. Both successful image jailbreaks and GCG suffixes require around 100 optimization steps, but their computational complexities differ. For each step, jailbreak images require 1 forward and 1 backward pass on 1024 image tokens; totaling $\sim 100,000$ forward and backward token operations per successful jailbreak image. In contrast, GCG uses 20 additional tokens with 1 backward pass and 512 forward passes per iteration, resulting in $\sim 20,000$ backward and $\sim 1M$ forward token operations.
Assuming constant FLOPs per token operation(^3^33Although exact FLOPs are not constant, the differences are negligible and we assume equal cost per token.) and that backward operations require 3 times more FLOPs than forward operations, we estimate that jailbreak images require an additional 400,000 FLOPs/token, while GCG needs an extra 1,060,000 FLOPs/token to obtain a successful attack(^4^44We did not optimize any of the methods for efficiency.).

Table 3 summarizes the attack success rate for both methods. Jailbreak images outperform text-based attacks, and require $3\times$ less compute. Image optimization also offers a larger attack surface since the attacker can modify the 1024 image tokens at a lower overall cost than GCG requires for only 20 text tokens.

**Table 3: Image jailbreaks outperform text attacks. Attack success rate (ASR) and refusal rate (RR) for each attack on Chameleon-7B before and after applying Circuit Breaker protections. Results computed using the shortcut in the forward pass. We also report the difference in *perplexity per token* ($\Delta$ PPL) with respect to benign prompts.**
|  | Default Safety | Circuit Breaker |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | ASR ($\uparrow$) | RR ($\downarrow$) | ASR ($\uparrow$) | RR ($\downarrow$) | $\Delta$ PPL ($\downarrow$) |
| No attack | 2.5% | 77.5% | 1.3% | 81.3% | - |
| Jailbreak Image (1-hot) | 71.3% | 13.8% | 0.0% | 66.2% | -1.14 |
| Jailbreak Image (Embed) | 70.0% | 16.3% | 3.8% | 64.6% | -1.21 |
| GCG | 63.8% | 22.5% | 10.0% | 37.5% | +2.51 |

### 5.3 Effectiveness Against White-Box Protections

First, we create jailbreak images and text suffixes with GCG on models enhanced with Circuit Breakers (Zou et al., ) (see Table 3). Circuit Breakers protections optimized over text inputs can generalize against images. Embedding shortcuts provide a more flexible representation to circumvent the protections. Moreover, unlike text attacks, jailbreak images do not increase the average token perplexity in the prompt, making them harder to detect by methods that filter out high-perplexity prompts (Jain et al., ). Although perplexity could also be computed in the image space, this protection has been shown to be ineffective against adversarial images (Carlini & Wagner, ).

## 6 Transfer Attacks

**Table 4: Existing methods outperform image jailbreaks on Chameleon. Attack success rate (ASR) and refusal rates (RR) on JailbreakBench. We also report the difference in *perplexity per token* ($\Delta$ PPL) with respect to benign prompts.**
|  |  | Default Safety | Circuit Breaker |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Attack | Train Prompts | ASR ($\uparrow$) | RR ($\downarrow$) | ASR ($\uparrow$) | RR ($\downarrow$) | $\Delta$ PPL ($\downarrow$) |
| No attack | - | 2.5% | 77.5% | 1.3% | 81.3% | - |
| Jailbreak<br>Image (1-hot) | 1 | 27.5% | 60.0% | 0.0% | 86.3% | -1.14 |
| 10 | 53.8% | 35.0% | 3.8% | 68.8% | -1.18 |  |
| 20 | 51.3% | 36.3% | 1.3% | 58.8% | -1.08 |  |
| Jailbreak<br>Image (Embed) | 1 | 31.3% | 51.3% | 0.0% | 73.8% | -1.18 |
| 10 | 72.5% | 16.3% | 6.3% | 45.0% | -1.29 |  |
| 20 | 72.5% | 10.0% | 10.0% | 53.8% | -1.21 |  |
| GCG | 1 | 17.5% | 66.3% | 1.3% | 78.8% | +0.86 |
| 10 | 50.0% | 60.0% | 5.0% | 50.0% | +0.14 |  |
| 20 | 46.3% | 21.3% | 1.3% | 38.8% | +3.02 |  |
| Refusal Direction | 1 | 5.0% | 55.0% | 0.0% | 81.3% | - |
| 10 | 73.8% | 1.3% | 0.0% | 40.0% | - |  |
| 20 | 81.3% | 2.5% | 5.0% | 38.8% | - |  |

We now evaluate the transferability of jailbreak images to unseen prompts and models. Instead of optimizing a separate image for each test prompt, we create a universal jailbreak image optimized over $N$ held-out *train prompts*, and evaluate the attack success rate on the unseen test prompts.

We use 2 text-only baselines to contextualize our findings. We again use GCG, but, instead of optimizing for a single propmt, we optimize text suffixes over the same set of train prompts to increase transferability. Additionally, we use refusal suppression (Arditi et al., ), a representation engineering method that requires a set of train prompts to detect a direction in the model’s activation space that is responsible for refusal(^5^55This method could not identify a refusal direction when using image inputs.). This direction is subtracted at inference time, disabling the model’s ability to refuse harmful requests. We summarize the main findings from the results in Table 5.

#### Increasing the number of train prompts improves generalization.

Jailbreak images optimized on a single image exhibit good generalization to unseen prompts ($\sim$30%) but computing the loss over a larger number of prompts improves generalization for both 1-hot and embedding shortcuts. However, after a certain point, increasing the number of prompts does not result in improved attack success rates. For both shortcuts, we find very similar success for 10 and 20 training prompts.

#### Embedding shortcut can find more transferable jailbreak images.

Although generalization for images optimized on a single prompt is similar for 1-hot and embedding shortcuts, the embedding shortcut can obtain much higher attack success rates on unseen prompts when optimized over more than 10 prompts (53.8% vs. 72.5%). In fact, the performance obtained with the jailbreak image optimized with embedding shortcut on 10 prompts can jailbreak as many prompts as optimizing a single image per test prompt (see Section 5), reducing computational cost significantly.

#### Representation engineering can still outperform jailbreak images.

Suppressing the refusal direction during the forward pass obtains the best results overall on our test set (81.3%). This suggests that our image jailbreaks suffer from similar limitations as existing text-only attacks because it is restricted to the input space.

#### Universal jailbreak images do not increase perplexity.

Similar to single-prompt attacks, the resulting images do not increase the average token perplexity of the prompt. In fact, jailbreak images have lower perplexity than benign images under the model distribution.

#### Jailbreak images do not transfer across models.

We evaluate whether the images optimized on the held-out training set can jailbreak the unseen test prompts on unseen models. We use Chameleon-30B as a model within the same family, and the latest LLaMA3-LLaVA-1.6 vision language model to account for models with different architectures and training setups. Similar to recent work on adapter-based vision language models (Schaeffer et al., ), we find that jailbreak images transfer poorly across models. In fact, GCG suffixes can transfer much better to the LLaVA model increasing attack success rate by $32.5$p.p from the baseline versus the 1.2p.p. achieved by jailbreak images. Direct attacks do not transfer to other architectures either (see Appendix B).

**Table 5: Jailbreak images do not transfer across models. Attack success rate (ASR) and refusal rate (RR) on JailbreakBench using images optimized on Chameleon-7B.**
|  |  | Chameleon-30B | LLaVA-1.6 |  |  |
| --- | --- | --- | --- | --- | --- |
| Attack | Train Prompts | ASR | RR | ASR | RR |
| No attack | - | 0.0% | 98.8% | 8.8% | 85.0% |
| Jailbreak<br>Image (1-hot) | 1 | 0.0% | 98.8% | 10.0% | 88.8% |
| 10 | 0.0% | 98.8% | 10.0% | 88.8% |  |
| 20 | 0.0% | 98.8% | 10.0% | 91.3% |  |
| Jailbreak<br>Image (Embed) | 1 | 0.0% | 98.8% | 10.0% | 87.5% |
| 10 | 0.0% | 98.8% | 10.0% | 88.8% |  |
| 20 | 0.0% | 98.8% | 10.0% | 88.8% |  |
| GCG | 1 | 2.5% | 91.3% | 21.3% | 81.3% |
| 10 | 6.3% | 82.5% | 41.3% | 58.8% |  |
| 20 | 5.0% | 88.8% | 7.5% | 86.3% |  |

## 7 Discussion and Future Work

#### Image jailbreaks are promising but have a long way to go.

Our work is the first attempt to jailbreak multimodal architectures using end-to-end gradient attacks. We present promising results that suggest that optimization might be smoother and more efficient than on text tokens. However, we think there is still significant room for improvement upon our methods. While we ablated most of the relevant hyperparameters we identified, our results indicate that attack success can still vary drastically with their choice. Future work may explore the optimization dynamics in more detail to come up with more effective ways of using the gradients. Similarly, future work may explore more flexible target functions (Thompson & Sklar, ); e.g. combining the output and activation space.

#### Transferability of jailbreak images remains an open problem.

Concurrent work on adapter-based models  (Schaeffer et al., ) demonstrated that jailbreak images do not generalize across models even when optimizing on an ensemble of models. Our results indicate that this problem also persists in multimodal fusion models. Future work may focus on looking for ways to regularize the optimization to improve transferability.

#### Attack dynamics may change for newer models.

Although most companies have already announced proprietary multimodal architectures (Gemini Team, ; OpenAI, ), Chameleon models are the only open-source models trained with this architecture. We expect more models to come out in the coming months and we encourage researchers to assess the success of this attack on future models and architectures. We find that Chameleon models are overly safe in that they often refuse benign instructions. We believe this might be a result of a specific stance in the utility-safety tradeoff (Bai et al., ) for Chameleon that may not hold true for other models.

## 8 Related Work

#### Multimodal Models.

Adapter-based vision language models (VLMs) are a popular approach for integrating image understanding into language models (Liu et al., ; Karamcheti et al., ). They typically combine pre-trained language models with image embedding models. VLMs train an adapter to transform pre-trained image embeddings (e.g., from CLIP) into a representation compatible with language models. However, state-of-the-art proprietary models are shifting towards early-fusion multimodal models (Gemini Team, ; OpenAI, ). These models autoregressively process tokens representing multiple modalities in a joint tokenized space. Our work focuses on Chameleon models (Chameleon Team, ), currently the only open-source family of early-fusion multimodal models. Chameleon models support image inputs, are safety-tuned, and are available in 7B and 30B parameter sizes.

#### Adversarial Examples.

Adversarial examples are inputs designed to fool machine learning models, first explored in the context of image classification (Szegedy, ; Madry, ). Adversarial images examples are created by adding perturbations to valid inputs. These perturbations are optimized using the gradient of the target loss with respect to the input.

#### Jailbreaks.

Jailbreaks are adversarial inputs designed to bypass the safeguards implemented in large language models (LLMs) and get them to generate harmful content. Jailbreaks are often black-box—they do not require access to model weights— and involve specific conversational strategies (Liu et al., ; Shah et al., ; Liu et al., ; Wei et al., ). On the other hand, there are white-box jailbreaks that use model weights and gradients to guide the attack. However, unlike in image classification, text inputs cannot be directly optimized using gradients with respect to the target loss—because the input space is discrete and sparse—and require approximate methods. GCG is the most prominent white-box attack (Zou et al., ). GCG optimizes a suffix that can be appended after harmful instructions to prevent refusal. Interestingly, GCG suffixes optimized on open-source models transfer to black-box and proprietary models.

#### Jailbreaks against vision language models.

Carlini et al. () demonstrated that incorporating image inputs can enable more effective and efficient jailbreaking attacks. Concurrent to our work, Schaeffer et al. () explored whether optimizing jailbreak images on adapter-based vision language models exhibit the same transferability properties as GCG prompts. Their results indicate that jailbreak images are effective against adapter-based vision language models under white-box access but they do not generalize across models.

## Impact Statement

Our research contributes to the safety and responsible development of future AI systems by exposing limitations in current models. While acknowledging the potential for misuse in adversarial research, we believe our methods do not introduce any new risks or unlock dangerous capabilities beyond those already accessible through existing attacks or open-source models without safety measures. Finally, we believe that identifying vulnerabilities is essential for addressing them. By conducting controlled research to uncover these issues now, we proactively mitigate risks that could otherwise emerge during real-world deployments.

## Appendix A Effects of Softmax Temperature in Attack Success

Our 1-hot shortcut learns a very good mapping from VQ-VAE latent representations to the model input vocabulary. This implies that most predictions are very confident (i.e. probability 1 on the correct token and 0 elsewhere). This skewed distribution vanishes gradients for all other tokens and makes optimization harder. We found that increasing the temperature of the softmax helps distribute the probability over other likely tokens and make optimization smoother. Table  6 reports the results obtained with different choices of softmax. We report the numbers with and without the shortcut on the forward pass. We select a softmax temperature of 6 as default for all our experiments.

**Table 6: Attack success rate with our 1-hot shortcut while using different temperature on the prediction softmax. Increasing the temperature improves effectiveness and transferability of the attack.**
| Temperature | With Shortcut | Without Shortcut |
| --- | --- | --- |
| 1 | 12.9% | 0.0% |
| 2 | 25.6% | 24.4% |
| 4 | 34.7% | 36.0% |
| 6 (default) | 71.3% | 47.5% |
| 8 | 46.8% | 7.8% |
| 10 | 25.6% | 0.0% |

## Appendix B Transferability of Direct Attacks

**Table 7: Direct attacks do not transfer across models. We evaluate whether images optimized for a single prompt can jailbreak that prompt in different architectures.**
| Attack | Chameleon-30B | LLaVA-1.6 |
| --- | --- | --- |
| No attack | 0.0% | 8.8% |
| Jailbreak Image (1-hot) | 0.0% | 10.0% |
| Jailbreak Image (Embed) | 0.0% | 10.0% |

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: * Do the authors have an explanation for why the embedding space shortcut attacks do not transfer to non-shortcut models while the 1-hot attacks do?
soundness: 3
strengths: * This is a novel method that solves the core challenge of creating gradient-based image jailbreaks for multimodal fusion models.  
* Understanding the vulnerabilities in multimodal models is important for developing more robust systems, and gradient-based jailbreaking of fusion-based models has been under-explored.  
* The authors use good baselines for their experiments (GCG and refusal direction attacks), and convincingly demonstrate the success of their method  
* The experiments are thorough and informative, testing attack transfer as well as defence using circuit breakers. Several ablations are also performed.
summary: * The paper develops a white box gradient-based image jailbreak method for multimodal fusion models. Prior work on gradient-based image jailbreaks has focused on VLMs due to the lack of open source fusion models, but this has recently changed with the release of Chameleon.  
* The core challenge of doing this with fusion models is that gradients do not flow through to the input image due to a non-differentiable step in tokenization.   
* The authors solve this problem by introducing a novel “tokenizer shortcut” technique, where they train a small MLP to approximate the image tokenizer in a differentiable way. The tokenizer is then replaced by this approximation during adversarial image optimization, allowing gradient-based optimization to succeed.  
* Two versions of the tokenizer shortcut are developed, one mapping directly to embedding space and one producing a one-hot vocabulary encoding.  
* A comprehensive set of experiments are conducted. Key findings:  
  * Both shortcut methods produce images with high Attack Success Rate, but only the 1-hot shortcut images transfer to versions of the model that do not use the shortcut.  
  * Circuit breakers substantially reduce ASR  
  * Jailbreak images transfer easily across prompts but do not transfer across models  
* The authors also conduct a series of ablations, including on response prefix, softmax temperature, and number of train prompts.
weaknesses: * The dataset used is quite small, with only 80 prompts in the test set for direct attacks and 20 in the test set for transfer attacks. The results would be more convincing if done on a larger dataset. In addition, only a single dataset is tested.  
* The paper does not include any examples of jailbroken model responses - these are helpful for qualitative understanding of the attack.
* With the exception of table 1, the results given are all for models using the tokenizer shortcut. It would be helpful to also include  the results when using the 1-hot jailbreak images on models without the shortcut in Tables 2 and 4.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Your experiments seem interesting, and it seems like you may have opinions on future work. While you have already provided motivation for experiment design, it would be useful to add more detail to your results so that it is easier to judge what puzzles are worth investigating. For instance, it would be great to spell out the details of transferability experiments. The observation itself is cool, but the current presentation of the work makes it so that readers will have to reimplement your work to get started on forming hypotheses.
soundness: 3
strengths: - The choice of studying robustness of multimodal fusion models is timely.
- The selection of research questions is fitting for a first study in a fast-paced field. The hypothesis that it may be easier to attack models with this architecture is interesting, and is very useful to study early in the uptake of architectures.
- The paragraph writing style is easy to read, and the work can serve as an interesting log of experiments for other practitioners.
summary: This paper studies white box attacks against multimodal fusion models. This setting is considered interesting because these models convert all inputs - both text and images - into a shared tokenized space. This approach could make the models vulnerable to more efficient attacks through image optimization, since images offer a continuous space to optimize (unlike text which is discrete). In order to optimize potential attack inputs, they develop the tokenization shortcut method, mapping image embeddings to a continuous model input space before quantization. They find that for whitebox optimization attacks, images are more effective than text, however they do not beat other competitive baselines like representation engineering.
weaknesses: - The choice of the two shortcut is not clearly explained in section 3. It would be useful to spell it out.
- It would be useful to have more qualitative analysis or at least examples of jailbreaking images vs images that fail.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. Please address the weaknesses.
2. (Suggestion) Moving the related work section front or refering that more related work is in the later part shall improve the understanding of the paper.
soundness: 2
strengths: 1. This paper addresses an important reasearch topic of jail-breaking in VL-LLM models, considering the significant growing use of VL models in real world applications. Research in this direction seems essential.
2. This paper is well presented, making the paper easy to follow and understand.
summary: This paper proposes a method generating jail-break images that cause early-fusion VL models (especially those with discrete image tokens) to generate harmful content when jail-break images are appended with harmful prompts.
Unlike adapter-based VL models, which do not use image token discretization, the discrete tokenization of images in early-fusion VL models makes direct optimization through gradients challenging and limits the applicability of existing methods.
To address this, the paper proposes a tokenizer shortcut that bypasses the discrete tokenization process by replacing quantization with a shallow MLP module, enabling the computation of gradients. 
The experiments demonstrate the effectiveness of the proposed method for generating jail-break images under certain settings—specifically, a white-box attack.
weaknesses: 1. There is a lack comparison or discussion with other condidates to make quantizqation differentiable. If the proposed method achieves very strong performance in generating jail-breaking iamges, current approach would be acceptable. However, it seems that the proposed method can generate jail-break images in very limited settings: with shortcut or non-transfer setting.
2. As far as i understand, the white-box attack scenario is important because, although it may be impractical and unrealistic, it serves as a useful benchmark for black-box attacks. However, for the "with shortcut" results, it effectively becomes equivalent to altering the model itself, which makes discussions of attack performance somewhat meaningless. Nonetheless, the proposed method is primarily evaluated using the shortcut when demonstrating its strong performance, (Table 1, 2, 3, 4).
3. Optimizing within the input (image and text) space is important, as it is a prerequisite for black-box settings or model transfer. However, as shown in Table 5, the proposed method fails to produce transferable samples and underperforms compared to the baseline.
4. (Minor) The paper seems to contain overclaims or insufficient explanations. For example:
	- The title of Table 3 is "Image jailbreaks outperform text attacks," but the proposed method performs worse than the text-only attack, GCG, in the Circuit Breaker setting. Additionally, comparing GCG with the proposed method "with shortcut" seems unfair, as "with shortcut" is equivalent to changing the model.
	- In discussions and future works, the paper states, "(412) Our work is the first attempt to jailbreak multimodal architectures using end-to-end gradient attacks" and "(423) this problem also persists in multimodal models,". I guess the "fusion-based model" shall be more appropriate.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - Could you provide more insights into why the experimental results demonstrated a higher Attack Success Rate (ASR) using the embedding shortcut compared to the 1-hot shortcut?
- While it is understandable that jailbreak images optimized for Chameleon-7B might not transfer effectively to larger models, have you explored or observed whether jailbreak images optimized on larger models could be effectively transferred to smaller ones, such as from a Chameleon-30B to a Chameleon-7B model?
soundness: 3
strengths: - **Well-structured:** The paper is well-written and describes the proposed method clearly.
- **Introduced Differentiable Tokenizer:** This paper proposes using a two-layer neural network to make image tokenization in a multimodal fusion model feasible, enabling continuous optimization and revealing its threats to jailbreak.
summary: This paper proposed jailbreak attacks on multimodal fusion models by introducing a *differentiable* tokenizer shortcut. This allows for continuous optimization of adversarial images intended to bypass model safeguards. It evaluates the effectiveness of such attacks on Chameleon models, achieving a higher attack success rate than text-only jailbreaks. The results suggest that representation engineering defenses for text attacks could also adapt to adversarial image inputs.
weaknesses: - The proposed method of modifying the model architecture (replacing the original tokenizer) to elicit the jailbreak does not make much sense; also, the perturbed (attacked) images lack transferability. Given that a text-based attack is already feasible to pose such threats, I tend to buy the proposed method that applies the traditional method of generating adversarial perturbations to a multimodal fusion model. This method, however, is neither novel nor practically applicable to my understanding.

- Using adversarial images to elicit model jailbreak is also not novel; the paper lacks some discussion and comparison with existing works on VLLM [1].

[1] Visual Adversarial Examples Jailbreak Aligned Large Language Models (AAAI 2024)

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: 1. How can the robustness of the proposed method be improved to maintain effectiveness across diverse settings, especially in the absence of the tokenizer shortcut? Can the authors evaluate performance under different tokenization schemes?

2. Why do we need the enhanced loss function? 

3. Can the authors evaluate their method on other defenses, such as those mentioned by Jain et al. [1]?

5. What are the ablation results of changing the number of layers in the fully connected network or replacing it with other simple architectures?

6. Can the authors include additional baseline methods to more comprehensively assess the robustness and effectiveness of the proposed method, such as the FGSM, PGD, or any other reliable attack methods?

7. How can $\Delta$PPL be further validated as a reliable metric? For example, evaluating $\Delta$PPL's effectiveness in multimodal fusion models with an F1 score would provide a clearer, more reliable assessment.

[1] Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., Chiang, P. Y., ... & Goldstein, T. (2023). Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614.
soundness: 1
strengths: 1. The approach is straightforward, relying on a fully connected network structure to approximate image tokenization.
2. The research addresses an important problem by targeting vulnerabilities in multimodal fusion models.
summary: This paper proposes a gradient-based jailbreak method for multimodal fusion models. The authors introduce tokenizer shortcuts to solve the problem of continuous optimization not being carried out in the multimodal fusion model due to the discretization of input modalities. The experimental evaluation is carried out on the Chameleon multimodal fusion model. The results show that their method can trigger the generation of harmful information.
weaknesses: 1. The effectiveness of the proposed approach is not well-validated. Without the tokenizer shortcut, the method's performance declines significantly, suggesting it may lack robustness in different settings.

2. From Table 2, the attack success rate drops when adding the refusal prefix part. The enhanced loss function, which aims to reduce the probability of generic refusal tokens, does not demonstrate a clear benefit in the experiments.

3. The approach's effectiveness is further limited when defenses are in place, raising concerns about its resilience against common protective measures.

4. Practical applicability is limited as the approach relies on assumptions that may not align with realistic conditions.

4.1 In direct attack scenarios, the method presumes the target model has been modified to include the shortcut, but it is unlikely defenders would incorporate this modification.

4.2 The approach also lacks sufficient transferability, reducing its usability across different models or settings.

5. The compared baselines are limited, just focusing primarily on text-based attacks GCG. A broader selection of attack methods would improve the robustness of the evaluation.

6. The use of $\Delta$PPL to measure adversarial prompt effectiveness lacks sufficient validation as a reliable metric.

Meta-review: This paper presents a method for white-box gradient-based image jailbreak attacks on multimodal fusion models using a "tokenizer shortcut" for continuous optimization. The proposed method achieves higher attack success rates than text-based jailbreaks and is more computationally efficient.  A major limitation of the model is the white-box access assumed, which is not true for the majority of the strongest LLMs at the moment. In addition, the paper could benefit from stronger comparisons, including additional defense methods or transferability as mentioned by reviewers. Lastly, the comments of reviewers (e.g. overclaiming the comparison of image to text jailbreaking) should be amended in the paper before it's ready for publication.

Decision: Reject