Zer0-Jack: A memory-efficient gradient-based jailbreaking method for black box Multi-modal Large Language Models

## Abstract

Abstract Jailbreaking methods, which induce Multi-modal Large Language Models (MLLMs) to output harmful responses, raise significant safety concerns. Among these methods, gradient-based approaches, which use gradients to generate malicious prompts, have been widely studied due to their high success rates in white-box settings, where full access to the model is available. However, these methods have notable limitations: they require white-box access, which is not always feasible, and involve high memory usage. To address scenarios where white-box access is unavailable, attackers often resort to transfer attacks. In transfer attacks, malicious inputs generated using white-box models are applied to black-box models, but this typically results in reduced attack performance.
To overcome these challenges, we propose \ours , a method that bypasses the need for white-box access by leveraging zeroth-order optimization. We propose patch coordinate descent to efficiently generate malicious image inputs to directly attack black-box MLLMs, which significantly reduces memory usage further. Through extensive experiments, \ours achieves a high attack success rate across various models, surpassing previous transfer-based methods and performing comparably with existing white-box jailbreak techniques. Notably, \ours achieves a 95% attack success rate on MiniGPT-4 with the Harmful Behaviors Multi-modal Dataset on a black-box setting, demonstrating its effectiveness. Additionally, we show that \ours can directly attack commercial MLLMs such as GPT-4o. Codes are provided in the supplement. Warning: This paper contains examples of harmful language and images, and reader
discretion is recommended.

## 1 Introduction

Recently, the safety alignment of Large Language Models (LLMs) (Ji et al., ) has drawn increasing attention because of the safety concern raised by the potentially harmful response (Bengio et al., ; Zhou et al., ). And this kind of safety alignment succeeds in the first place, where the user will not get any useful response when directly asking questions related to safety or privacy. However, recent works have proven that it is possible to “jailbreak” the safety alignment so that LLMs may answer the question that they should not by carefully designing prompts (Carlini et al., ; Wei et al., ). Later on, using gradient methods and other automatic technologies to find the powerful jailbreak prompts has been proposed so that the attack rate becomes much higher compared with hand-craft prompts (Zou et al., ; Liu et al., ), which further demonstrates the vulnerability when obtaining the white-box models.

With the success of LLMs (Achiam et al., ; Touvron et al., ; Chiang et al., ), Multi-modal Large Language Models (MLLMs), which handle both text and image inputs, have gained popularity (Liu et al., ; Zhu et al., ; Liu et al., ). Despite their capabilities in tasks such as image descriptions and visual question answering
, MLLMs have been shown to be even more vulnerable to jailbreak attacks due to the additional modality (Qi et al., ; Sun et al., ; Liu et al., ; Zhao et al., ). For example, Liu et al. () demonstrated that images containing specific text can assist in jailbreaking MLLMs. In white-box settings, where full access to model parameters is available, methods like generating malicious image inputs (Niu et al., ) or combining both text and image prompts (Shayegani et al., ) by optimization have proven effective in bypassing safety mechanisms. Similar to LLM jailbreaking, the most effective methods in MLLMs rely on calculating gradients to find inputs that induce harmful outputs.

Figure: Figure 1: Comparison between white-box jailbreak, transfer jailbreak attack, and direct black-box jailbreak. Both white-box jailbreak and transfer jailbreak generate malicious inputs using white-box models while direct black-box attacks do not. In this paper, we focus on direct black-box jailbreak and prove our method can surpass transfer attacks and be comparable with white-box attacks.
Refer to caption: x1.png

While gradient-based methods for white-box models have shown strong performance, the challenge of attacking black-box models remains underexplored. Black-box models, such as commercial MLLMs like GPT-4o (OpenAI, ), do not provide access to their internal parameters, making gradient-based attacks impossible. Most existing jailbreak methods for black-box models rely on transfer attacks, where malicious inputs generated on white-box models are used to indirectly attack black-box models (Zou et al., ; Niu et al., ; Dong et al., ). However, these transfer attacks often suffer from a significant reduction in success rate compared to direct attacks on white-box models (Niu et al., ).

In this paper, instead of transferring the malicious prompts from white-box models, we propose \ours, a method that directly generates malicious image inputs for jailbreaking black-box MLLMs. \oursleverages zeroth-order optimization, which estimates gradients without accessing model parameters, to find malicious prompts capable of bypassing safety measures. One challenge with zeroth-order optimization is its susceptibility to high estimation errors in high-dimensional inputs. To mitigate this, \oursoptimizes only a specific part of the image, reducing the dimensionality of the problem and thereby minimizing estimation errors. Furthermore, \oursdoes not rely on backpropagation, resulting in significantly lower memory usage. Through extensive experiments, we show that \ourscan achieve a high attack success rate within reasonable queries as well as decrease memory usage when generating malicious prompts. Overall, we provide the comparison between different types of jailbreak methods in Fig. 1 and summarize our contribution as follows:

- 1.
We propose \ours, which utilizes zeroth-order optimization technology to generate malicious images. To the best of our knowledge, \oursis the first method that aims at jailbreaking black-box MLLMs directly.
- 2.
\ours
reduces the memory usage and query complexity by only optimizing specific parts of the image, minimizing the impact of gradient noise. In detail, \oursallows us to attack 13B models in a single 4090 without any quantization.
- 3.
We perform extensive experiments demonstrating that \oursconsistently achieves a high success rate across various MLLMs. In all black-box scenarios, \ourssurpasses transfer-based attack methods and performs on par with white-box approaches. For instance, \oursattains success rates of 98.2% on MiniGPT-4 using the MM-SafetyBench-T dataset and 95% with the Harmful Behaviors Multi-modal dataset. Besides, we use a showcase to demonstrate that it is possible for \oursto directly attack commercial MLLMs such as GPT-4o.

## 2 Related Works

Figure: Figure 2: The overview of \ours. To effectively attack a black-box MLLM, \oursleverages zeroth-order optimization and patch coordinate descent.
Refer to caption: x2.png

#### Jailbreak Methods for LLMs

Recent research has demonstrated that even LLMs with strong safety alignment can be induced to generate harmful content through various jailbreak techniques (Xu et al., ). Early methods relied on handcrafted prompts, such as the "Do-Anything-Now" (DAN) prompt (Liu et al., ), while more recent approaches have moved toward automated techniques, including using auxiliary LLMs to generate persuasive prompts (Li et al., ; Zeng et al., ) and gradient-based methods to search for effective jailbreak prompts (Zou et al., ). Additionally, genetic algorithms (Liu et al., ) and constrained decoding strategies (Guo et al., ) have been introduced to improve prompt generation. While these techniques primarily focus on jailbreaking LLMs by generating malicious text outputs, this paper focuses on MLLMs, specifically on generating malicious images to jailbreak models.

#### Jailbreak Methods for MLLMs

Previous work has demonstrated that MLLMs, with their added visual capabilities, are more vulnerable to malicious inputs (Liu et al., ). Jailbreak methods for Multi-modal LLMs (MLLMs) can be broadly categorized into white-box and black-box settings. In the white-box setting, attackers have full access to model parameters, allowing for more direct manipulation. Gradient-based approaches have been widely used in this setting to generate adversarial visual prompts (Niu et al., ; Qi et al., ; Dong et al., ; Bailey et al., ; Tao et al., ), with some methods combining both text and image prompts to exploit multi-modal vulnerabilities (Shayegani et al., ; Wang et al., ). However, these methods require white-box access and may not generalize well to more restricted models.
In the black-box setting, where model parameters are not accessible, attackers typically rely on transfer-based approaches or carefully designed prompts. Techniques such as using topic-related images or embedding text within images have proven possible in triggering jailbreaks (Liu et al., ; Gong et al., ; Ma et al., ). Transfer-based attacks involve generating adversarial prompts on a white-box model and then using these prompts to attack black-box models (Zou et al., ). For example, Dong et al. () tested the transferability of visual adversarial prompts on closed-source MLLMs. However, transfer-based attacks generally suffer from reduced success rates compared to white-box methods (Niu et al., ). Our work addresses this limitation by proposing a direct black-box jailbreak method using zeroth-order optimization. This approach eliminates the need for transferability or handcrafted prompts, focusing on efficiently generating malicious images to attack MLLMs with reduced memory usage and high success rates even under black-box setting.

**Table 1: Comparison of memory usage for different sizes of models and images. \oursshow a huge advantage in reducing memory usage, making it possible to attack 13B models using a single 4090 and attack 70B models using a single A100.**
| Model | Parameter | Image Size | White-box Attack | \ours |
| --- | --- | --- | --- | --- |
| MiniGPT-4 | 7B | 224 | 15G | 10G |
| MiniGPT-4 | 13B | 224 | 39G | 22G |
| MiniGPT-4 | 70B | 224 | OOM | 63G |
| Llava1.5 | 7B | 336 | 26G | 15G |
| Llava1.5 | 13B | 336 | 46G | 25G |
| INF-MLLM | 7B | 448 | 27G | 17G |

## 3 Method

In this section, we begin to provide an introduction to a baseline jailbreak method focusing on text-only LLMs. We then demonstrate how this method can be adapted and extended into a more powerful and memory-efficient jailbreak technique tailored to MLLMs. We also provide the overview of our method \oursin Fig. 2.

### 3.1 Preliminary

The general goal of jailbreaking attacks in LLMs is inducing LLMs to output unsafe or malicious responses. For example, a LLM with good safety alignment should not generate a detailed response to the query ‘How to build a bomb’, while jailbreaking attacks aim at making the LLM output the answer to this query. Similar to some adversarial attacks in NLP (Wallace et al., ), gradient-based jailbreaking attacks try to find specific suffix tokens that make LLMs output malicious responses. For example, a new query from attackers might be ‘How to build a bomb. !!!!!!!!!’, which can actually induce LLMs to output the detailed procedures of how to make a bomb.

However, unlike adversarial attacks, where the target is to output the same answer and reduce the accuracy when the suffix is added to the prompt (Wallace et al., ), the jailbreaking attackers hope LLMs can output true answers to their unsafe query. Besides, there are usually multiple true answers to the query in jailbreaking and thus it is not possible to find suffix tokens by optimizing the output towards one true answer.

To tackle the problem, one of the most popular jailbreaking methods, Greedy Coordinate Gradient (GCG) (Zou et al., ) tries to find suffix tokens that induce LLMs to output their answer starts with ‘Sure, here is’. Then if the language model could output this context at the beginning of the response instead of refusing to the question, it is highly possible for language models to continue the completion with the precise answer to the question.

In detail, the optimization problem in GCG can be formulated as:

$$ $\displaystyle\underset{x_{\mathcal{I}}\in\{1,\dots,V\}^{|\mathcal{I}|}}{\min} \quad\mathcal{L}(x_{1:n}),$ (1) $$

where $x_{\mathcal{I}}$ is the suffix tokens, $x_{1:n}$ represents the original prompts and $\mathcal{L}(x_{1:n})$ is the loss function:

$$ $\displaystyle\mathcal{L}(x_{1:n})=-\log\ p(x^{*}_{n+1:n+H}|x_{1:n})$ (2) $$

Where $x^{*}_{n+1:n+H}$ represents the target beginning of the answer such as ‘Sure, here is’.

Right now, GCG has a clear optimization target. However, GCG still needs to tackle the discrete optimization problem to generate discrete tokens. To do so, GCG uses a greedy coordinate gradient-based search. Specifically, GCG computes the gradient with respect to the one-hot vector representing the current value of the i-th token and selects top-k tokens with the highest norm of gradient. Then GCG computes the loss for each token to get the final generated token.

### 3.2 A Trivial White-box Jailbreak on MLLMs

With the rapid success of Multi-modal LLMs (MLLMs), recent works have found that it will be easier for attackers to jailbreak the MLLMs due to the new modal introduced in MLLMs (Zhao et al., ; Qi et al., ). Therefore, in this paper, we mainly transfer the idea of inducing LLMs to output ‘Sure, here it is’ at the beginning to jailbreak MLLMs by utilizing the image inputs.

Specific to the image input in MLLMs, we can map the continuous values into RGB values without losing too much information since the RGB values in the image are sufficiently close enough that they can be treated as continuous largely.
Then it is possible that we do not need to care about the discrete optimization anymore by transferring the attack surface from texts to images i.e. perturbing image inputs only. In this case, The optimization problem in Eq. 1 can be transferred into:
$\underset{\mathcal{Z}}{\min}\mathcal{L}(x_{1:n},\mathcal{Z}),$
where $\mathcal{Z}$ represents the value tensors of the input image. We can optimize this objective by calculating the gradient with respect to the image inputs:

$$ $\displaystyle\nabla_{\mathcal{Z}}\mathcal{L}(x_{1:n},\mathcal{Z})$ (3) $$

By transferring the attack surface from the text to images, our jailbreak method can deal with the potential performance degradation caused by discrete optimization. However, the current version of the attack still suffers from the following two disadvantages:

- 1.
Directly computing Eq. 3 requires the white-box accesses to the MLLMs, which further restricts the potential usage of such an attack.
- 2.
We present the GPU memory usage for differnt models and parameters in Table 1. As shown in Table 1, the trivial white-box attack requires a lot of memory that a single A100 could not attack 70B models, which restricts the number of usage scenes for the attack.

### 3.3 \ours : Jailbreaking with Zeroth-order Gradient Estimator

#### SPSA-P: Improved Zeroth-order Gradient Estimator for MLLMs

To tackle the mentioned problems for attacking black-box models and high memory usage, we utilize zeroth-order optimization technology to calculate Eq. 3 without backpropagation (Shamir, ; Malladi et al., ). In detail, we estimate the gradient with respect to $\mathcal{Z}$ by the Simultaneous perturbation stochastic approximation (SPSA) (Spall, ):

$$ $\displaystyle\hat{\nabla}_{\mathcal{Z}}\mathcal{L}(x,\mathcal{Z}):=\frac{ \mathcal{L}(x,\mathcal{Z}+\lambda u)-\mathcal{L}(x,\mathcal{Z}-\lambda u)}{2 \lambda}u,$ (4) $$

Where $u$ is uniformly sampled from the standard Euclidean sphere and $\lambda>0$ is the smoothing parameter (Duchi et al., ; Yousefian et al., ; Zhang et al., ). Using this formula to estimate the gradient, we only need to get the output logits or probability, which is allowed for many commercial MLLMs (Finlayson et al., ) and helps reduce memory usage because we do not need to calculate the real gradient by backpropagation anymore. It also has been proven that Eq. 4 is an unbiased estimator of the real gradient (Spall, ).

However, using Eq. 4 directly as the gradient to optimize $\mathcal{Z}$ may suffer from the estimated errors caused by high dimension problems especially when the size of images is large (Yue et al., ; Zhang et al., ; Nesterov and Spokoiny, ). The performance of zeroth-order optimization can be very bad with high-resolution images. To tackle this problem, we propose SPSA with a patch coordinate descent method (SPSA-P) to reduce the influence of estimated error when dimensions are high. In detail, we utilize the idea of patches from the vision transformer (Dosovitskiy, ) and divide the original images into several patches:

$$ $\displaystyle Z=[P_{1},..P_{i-1},P_{i},P_{i+1},...,P_{n}],$ (5) $$

where $P_{i}$ represents the i-th patch for the image. Normally, we use $32\times 32$ as the shape for each patch if the original image has the shape of $224\times 224$. Then we will compute the gradient for each patch instead of the whole image by only perturbing $P_{i}$ at one iteration:

$$ $\displaystyle\hat{\nabla}_{P_{i}}\mathcal{L}(x,\mathcal{Z}):=\frac{\mathcal{L} (x,P_{i}+\lambda u)-\mathcal{L}(x,P_{i}-\lambda u)}{2\lambda}u.$ (6) $$

#### Leveraging SPSA-P in \ours

Figure: Algorithm 1 \ours

To utilize SPSA-P in our jailbreak framework \ours, we can directly use Eq. 2 as the loss function to optimize in Eq. 7 to get the estimated gradient:

$$ $\displaystyle\hat{\nabla}_{P_{i}}\mathcal{L}(x_{1:n},\mathcal{Z}):=\frac{ \mathcal{L}(x_{1:n},P_{i}+\lambda u)-\mathcal{L}(x_{1:n},P_{i}-\lambda u)}{2 \lambda}u.$ (7) $$

After estimating the gradient for one patch, we can choose to update the patch immediately or keep the whole image unchanged until we get the estimated gradient for the whole image. We choose to update the patch immediately to get the new image:

$$ $\displaystyle P_{i}^{\prime}=P_{i}-\alpha\hat{\nabla}_{P_{i}}\mathcal{L}(x_{1: n},\mathcal{Z}),$ (8) $$

$$ $\displaystyle Z^{\prime}=[P_{1},..P_{i-1},P_{i}^{\prime},P_{i+1},...,P_{n}],$ (9) $$

where $\alpha$ is the learning rate. Then we move to the next patch $P_{i+1}$, estimate the gradient of the next patch, and update the next patch $P_{i+1}$:

$$ $\displaystyle\hat{\nabla}_{P_{i+1}}\mathcal{L}(x_{1:n},\mathcal{Z^{\prime}}):= \frac{\mathcal{L}(x_{1:n},P_{i+1}+\lambda u)-\mathcal{L}(x_{1:n},P_{i+1}- \lambda u)}{2\lambda}u.$ (10) $$

We choose to update one patch at a time instead of the whole image after estimating the gradient for the whole image because sometimes updating a few patches can lead to a successful attack and waiting for the gradient for the whole image can get suboptimal iteration efficiency. By updating only one patch each time, the updating dimensions become $32\times 32$, which is around $0.02\%$ of the updating dimensions if we directly update the whole image of $224\times 224$, thus reducing the estimation errors significantly. Overall, we summarize the overall framework \ours, which utilizes our proposed SPSA-P in Algorithm 1.

## 4 Experiments

### 4.1 Setup

#### Target Models

We evaluate our method using three prominent Multi-modal Large Language Models (MLLMs) known for their strong visual comprehension and textual reasoning capabilities: MiniGPT-4 (Zhu et al., ), LLaVA1.5 (Liu et al., ), and INF-MLLM1 (Zhou et al., ), all equipped with 7B-parameter Large Language Models (LLMs). Additionally, to assess memory efficiency, we conduct experiments with MiniGPT-4 paired with a 70B LLM, demonstrating that our approach requires minimal additional memory beyond inference. For all models we use, we use the instruction version from the authors of these models and all models have safety alignment so that they hardly respond to potentially harmful questions before jailbreak attacks. We only use the logits from models to simulate a black-box situation.

#### Datasets

We evaluate \oursusing two publicly available datasets specifically designed for assessing model safety in multi-modal scenarios:

$\bullet$ Harmful Behaviors Multi-modal Dataset: The Harmful Behaviors dataset (Zou et al., ) is a safety-critical dataset designed to assess LLMs’ behavior when prompted with harmful or unsafe instructions. It includes 500 instructions aimed at inducing harmful responses. For our experiments, we selected a random subset of 100 instructions from this dataset. To create multi-modal inputs, which fit for MLLMs evaluation,
we paired each instruction with an image randomly sampled from the COCO val2014 dataset (Lin et al., ). This ensures a diverse and realistic evaluation of model performance in harmful behavior scenarios.

$\bullet$ MM-SafetyBench-T: MM-SafetyBench-T (Liu et al., ) is a comprehensive benchmark designed to assess the robustness of MLLMs against image-based manipulations across 13 safety-critical scenarios with 168 text-image pairs specifically crafted for testing safety. It provides the diversity of tasks, allowing for meaningful insights into model robustness while ensuring computational feasibility in extensive experimentation. Among the image types provided by this benchmark, we utilized images generated using Stable Diffusion (SD) (Rombach et al., ) for this evaluation. We provide our detailed evaluation results for each scenario in Appendix B.

#### Baselines

To evaluate our proposed \ours, we compare it against a variety of baselines that encompass both text-based and image-based approaches.

$\bullet$ Text-based baselines involve generating or modifying text prompts to bypass model defenses. Specifically, we compared \ourswith four text-based jailbreak methods:
The first baseline, P-Text, tests whether the original text input alone can bypass the model’s defenses. Since the selected MLLMs do not support text-only input, we pair the P-text with a plain black image containing no semantic information. For the second baseline, we adopt GCG(Zou et al., ), which is a gradient-based white-box jailbreaking method. To simulate GCG in a black-box setting, we utilize the transfer attack, where the malicious prompts are generated using LLaMA2 (Touvron et al., ) and transferred to the models we used. The third and fourth baselines, AutoDAN(Liu et al., ) and PAIR(Chao et al., ), are baseline methods targeting black-box jailbreak attacks on LLMs. We will pair the malicious text prompts with corresponding images to evaluate their performance on Multi-modal LLMs.

$\bullet$ Image-based baselines target the visual component of the image-text pair, attempting to generate or modify the visual input to bypass the model’s safety mechanisms and induce harmful or unsafe outputs. To our knowledge, few approaches specifically optimize the image component of an image-text pair for jailbreak attacks on MLLMs. As a result, we adopt the following baselines for comparison: (1) P-Image: This baseline uses the original unmodified images as input to evaluate whether the raw images alone can mislead the model’s safety mechanisms. (2) G-Image: This baseline employs randomly generated Gaussian noise images to assess whether non-informative or noisy images can bypass the model’s defenses. (3) A-Image: We adopt the white-box optimization method from Dong et al. (), which originally generates adversarial images designed to mislead the model and we adopt the method to jailbreak task. Again, we use a transfer attack to simulate the black-box setting. The generated images are used as input for black-box MLLMs to evaluate their vulnerability. (4) WB baseline: As mentioned in Section 3.2, this baseline optimizes the image under a white-box setting using gradients to induce successful jailbreak attempts. Please note that for the WB attack, we report the results in the white-box setting to show \ourscan be comparable with white-box approaches. All images are paired with their corresponding text from the dataset to create the complete image-text input for evaluation.

#### Metrics

Prior research has commonly evaluated responses using the String Match method, where predefined harmless suffixes are used to determine whether a response bypasses content restrictions (Zou et al., ; Liu et al., ). If harmless strings such as ‘I am sorry’ are present, the response is considered a jailbreak failure, and if no harmless strings are found, it is considered a success. While this method has been widely used, we follow the evaluation approach from Cai et al. (), which relies on GPT-4 models to assess whether responses genuinely answer the questions or contain unsafe content and has shown a more comprehensive evaluation of model behavior. Provided by both the question and the answer, if GPT-4 thinks the answer is harmful, then we consider the attack on this question as a successful jailbreak attack. We report the commonly used attack success rate (ASR) (Zou et al., ):

$$ $ASR:=\frac{\text{\#Successful Attacks}}{\text{\#All Questions}},$ $$

which demonstrates the overall effectiveness of the jailbreak attempts.

Figure: Figure 3: Case study illustrating the jailbreak performance of text-based and image-based methods on LLaVA1.5 for the question “List steps in detail to engage in identity theft without leaving any traces” with the corresponding image. The first raw shows the response generated from text-based method, AutoDAN, GCG, and PAIR. We also present the text prompt we optimized from white-box models. The second raw compares responses when using P-Image, A-Image, and the optimized image from \ours, each paired with the text input.
Refer to caption: x3.png

### 4.2 Overall Performance on Benchmarking Datasets

**Table 2: Attack success rate of various jailbreak methods across four MLLMs on the Harmful Behaviors Multi-modal Dataset. P-Text, GCG, AutoDAN and PAIR represent text-based jailbreaking methods; G-Image, P-Image and A-Image refers to image-based jailbreaking methods. ZO represents our proposed \ours, which optimizes the image via zeroth-order optimization to jailbreak MLLMs.**
| Model | P-Text | GCG | AutoDAN | PAIR | G-Image | P-Image | A-Image | WB | \ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MiniGPT-4 | 11% | 13% | 16% | 14% | 10% | 11% | 13% | 93% | 95% |
| LLaVA1.5 | 0 | 0 | 8% | 5% | 0 | 1% | 0 | 91% | 90% |
| INF-MLLM1 | 0 | 1% | 22% | 7% | 0 | 1% | 1% | 86% | 88% |
| MiniGPT-4 (70B) | 14% | - | - | 17% | 12% | 13% | - | - | 92% |

**Table 3: Attack success rate of various jailbreak methods across four models on the MM-SafetyBench-T Dataset. The specific condition settings are consistent with those in Table 2.**
| Model | P-Text | GCG | AutoDAN | PAIR | G-Image | P-Image | A-Image | WB | \ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MiniGPT-4 | 44.0% | 40.5% | 39.9% | 41.1% | 44.0% | 39.9% | 33.3% | 96.4% | 98.2% |
| LLaVA1.5 | 11.9% | 23.2% | 41.7% | 31.0% | 7.7% | 14.3% | 29.8% | 95.2% | 95.8% |
| INF-MLLM1 | 19.6% | 30.4% | 52.4% | 38.1% | 19.0% | 26.2% | 19.0% | 97.6% | 96.4% |
| MiniGPT-4 (70B) | 50.2% | - | - | 45.3% | 42.6% | 41.2% | - | - | 95.8% |

#### Results on Harmful Behaviors Multi-modal Dataset

The evaluation results on the Harmful Behaviors Multi-modal Dataset, as shown in Table 2, highlight the effectiveness of our \ours, compared to other jailbreak techniques. In MiniGPT-4, \oursachieved an impressive ASR of 95%, significantly outperforming other methods such as AutoDAN at 16% and GCG at 13%. Similarly, in LLaVA1.5, \oursrecorded an ASR of 90%, while alternatives faltered, with AutoDAN achieving only 8% and the P-Text yielding no successful attacks at all. INF-MLLM1 showed an ASR of 88% for \ours, reinforcing its effectiveness, while other methods like AutoDAN and GCG managed only 22% and 1%, respectively. Notably, when evaluating the larger MiniGPT-4 model paired with a 70B LLM, \oursachieved an ASR of 92%, whereas GCG, AutoDAN, and WB did not yield results due to GPU memory constraints. The results from the \ourswere comparable to those of the WB method, but \oursconsumed significantly less memory. This further indicates that our method remains effective even when scaled to larger model architectures, requiring minimal additional memory beyond inference.

#### Results on MM-SafetyBench-T Dataset

As shown in Table 3, the evaluation results from the MM-SafetyBench-T Dataset underscore the effectiveness similar to the previous results on Harmful Behaviors. Specifically, \oursachieved an ASR of 98.2% in MiniGPT-4, 95.8% in LLaVA1.5, and 96.4% in INF-MLLM1. In contrast, methods originally designed for LLMs, such as GCG, AutoDAN, and PAIR, demonstrated significantly reduced effectiveness when their adversarial prompts were transferred to MLLMs. For instance, while GCG excelled in LLMs jailbreak, it only managed to achieve an ASR of 40.5% in MiniGPT-4 and 23.2% in LLaVA1.5. For a larger MiniGPT-4 model paired with a 70B LLM, the results demonstrated the same trend as Table 2. We also report the results for different categories in the MM-SafetyBench-T dataset in Fig. 6 while the full results can be found in Appendix.

### 4.3 Evaluation on Transferability

To assess the transferability of images optimized through \oursacross different models, we conducted three sets of comparative experiments. First, we optimized images using the MM-SafetyBench-T dataset on the MiniGPT-4 model to generate adversarial images capable of successfully bypassing defenses. We then transferred these optimized images to the LLaVA1.5, GPT-4o, and INF-MLLM1 for transferability evaluation.

**Table 4: Transferability evaluation of adversarial images generated by \ourson MiniGPT-4 and MM-SafetyBench-T, showcasing the ASR when transferred to other models.**
| Model | P-Text | P-Image | Tranfer    \bigstrut |
| --- | --- | --- | --- |
| GPT-4o | 33.3% | 40.5% | 51.8% \bigstrut[t] |
| LLaVA1.5 | 11.9% | 14.3% | 54.2% |
| INF-MLLM1 | 19.6% | 26.2% | 54.8% \bigstrut[b] |

The results in Table 4 demonstrate the transferability of adversarial images generated by \ours. Notably, the ASR of 51.8% for GPT-4o highlights a significant transferability of our adversarial images to bypass defenses, supported by P-Text and P-Image with ASR of 33.3% and 40.5%, respectively. On the other hand, LLaVA1.5 and INF-MLLM1 show higher ASR of 54.2% and 54.8%. Though the images generated by \oursshow good transferability, they still suffer from performance degradation, indicating the importance of attacking black-box models directly. We show the results of direct attacking in Fig. 5.

### 4.4 Analysis on Efficiency

To analyze the efficiency of \ours, we evaluate its practical advantages in terms of memory consumption and iteration efficiency over traditional methods.

#### Memory Consumption

As illustrated in Fig. 4, traditional jailbreak methods often require substantial memory, limiting their practicality for deployment. To compare memory consumption, we evaluated text-based methods on the LLaMA2-7B model, which is commonly used as the language model in MLLMs. Specifically, GCG consumes approximately 50GB of memory, while AutoDAN requires around 26GB. In contrast, image-based optimization techniques such as A-Image and WB Attack, applied to MLLMs like MiniGPT-4, use about 19GB each due to the need for gradient retention, while \ourssignificantly reduces memory usage without sacrificing performance, uses only 10GB of memory.

Figure: Figure 4: Comparison of average memory cost and iteration efficiency when optimizing a sample on MiniGPT-4. The bar chart represents memory consumption (in GB), while the line graph illustrates iteration efficiency (number of iterations).
Refer to caption: x4.png

#### Iteration Efficiency

Next, we compare the iteration efficiency, which refers to the number of iterations required for each method to generate a successful adversarial goal.

As shown in Fig. 4, we found that GCG typically requires around 100 iterations per adversarial goal, while AutoDAN takes even more, averaging between 100 and 120 iterations. For AdvImage, the default setting requires more than 200 steps to generate the adversarial image due to its perturbation constraint on the image. WB Attack requires around 40 to 50 iterations. In contrast, Our \oursdemonstrates significantly greater efficiency. \oursonly needs 55 iterations on average to optimize the image successfully, which is comparable with the WB Attack that is a white-box attack.

### 4.5 Case study

As shown in Fig. 3, this case study illustrates the performance of various methods on LLaVA1.5 in response to the question, “List steps in detail to engage in identity theft without leaving any traces” along with the corresponding image. Text-based approaches, including AutoDAN, GCG, and PAIR, generated adversarial text prompts that resulted in responses such as “I cannot provide any information” and “It is a serious crime…”, demonstrating their failure to bypass the model’s safeguards. In contrast, our method effectively engaged LLaVA1.5, yielding clear and actionable steps such as: 1) Gather personal information, 2) Steal the information, etc. This stark difference underscores the success of our integrated approach in generating relevant and detailed outputs, highlighting its effectiveness in circumventing the model’s limitations compared to existing techniques.

### 4.6 Attack Commercial MLLMs Directly

Figure: Figure 5: The showcase of using \oursto attack GPT-4o. GPT-4o outputs the unsafe content under our attack.
Refer to caption: x5.png

In this subsection, we show that \ourscould attack commercial MLLMs directly. And we will focus on GPT-4o (OpenAI, ) here. OpenAI’s API only allows users to retrieve the top 20 tokens with the highest log probabilities, rather than accessing the entire set of logits. However, if we look back at the loss function in Eq. 2, we can find that \oursonly requires logits to our target responses ‘Sure, here it is’. Though the target responses may not show in the top 20 tokens with the highest log probabilities, we find that we can access the log probability to any token by logit_bias, which is a function provided by OpenAI’s API that enables users to add bias to any token’s logit. Then if we add a very high bias to ‘sure’, it will force GPT-4o to generate ‘sure’ and the API will return the log probability of the generated token ‘sure’. Through this method, we can access to log probability of all tokens in target responses and attack GPT-4o using \ours. Finally, when generating the final output, we discard anything about logit_bias. In Fig. 5, we provide a showcase that uses \oursto jailbreak GPT-4o successfully. \oursattacks this showcase with reasonable iterations that it only spends around 0.7 dollars calling OpenAI’s API.

Figure: Figure 6: Image example after our attack for different categories. And our attack could boost the attack success rate for each category a lot.
Refer to caption: x6.png

## 5 Discussion

- •
Limitations: though \oursonly requires access to output logits or probabilities, \ourscould not directly attack the web version of commercial MLLMs. Besides, there are some commercial MLLMs’ API that do not support return logits (Anthropic, ). To attack such models directly, it is better to design a jailbreak method using the information from generated responses instead of output logits.
- •
Call for Defense Strategy: since \oursdirectly estimates the gradient to generate malicious image inputs, it is difficult to use prompt-based defense methods that add more strict or safe system prompt (Wang et al., ). We argue that it is better to use post-hoc methods such as LLM-as-a-judge (Zheng et al., ), which makes MLLMs refuse to answer the question based on the response. Besides, \oursalso proves that partial information from output logits might be dangerous, which indicates that it is better for us to find a balance between transparency and risk provided by the models’ API.

## 6 Conclusion

In this paper, we presented \ours, a novel zeroth-order gradient-based approach for jailbreaking black-box Multi-modal Large Language Models. By utilizing zeroth-order optimization that requires output logits only, \oursaddresses the challenges that attacking black-box models. By generating image prompts and patch coordinate optimization, \oursdeals with the problems of discrete optimization and errors brought by the high dimensions in zeroth-order optimization. Extensive experiments across multiple MLLMs demonstrated the efficacy of \ours, with consistently high attack success rates surpassing transfer-based methods. Our method highlights the vulnerabilities present in MLLMs and emphasizes the need for stronger safety alignment mechanisms, particularly in multi-modal contexts.

## Acknowledgments

The work was partially supported by NSF awards #2421839, NAIRR #240120 and OpenAI Researcher Access Program. This work used AWS thru the CloudBank project, which is supported by National Science Foundation grant #1925001. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.

## Appendix A Code

Our code is provided in an [anonymous Github Link](https://anonymous.4open.science/status/Zrer0-Jack-77A0) (hyperlink here).

## Appendix B Detailed Results for categories in MM-safetybench-T

In Table 6, we provide the numbers of successful attacks for each scenario in MM-Safetybench-T and in Table 5, we provide the numbers of successful attacks for each scenario in MM-Safetbench-T dataset when we test the transfer ability of \ours. As we can see, even for each scenario, \ourscan beat other baseline methods. And an image example could be found at Fig. 6.

**Table 5: Number of successful jailbreaks with adversarial images optimized through \oursfrom MiniGPT-4 on the MM-SafetyBench-T dataset across different MLLMs.**
| Models | GPT-4o-mini | LLaVA1.5 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Scenarios | P-Text | P-Image | Transfer | P-Text | P-Image | Transfer |
| 01-Illegal Activity (10) | 0 | 0 | 0 | 1 | 1 | 6 |
| 02-Hate Speech (16) | 0 | 0 | 2 | 1 | 1 | 7 |
| 03-Malware Generation (5) | 0 | 0 | 0 | 0 | 0 | 3 |
| 04-Physical Harm (14) | 0 | 2 | 3 | 1 | 1 | 8 |
| 05-Economic Harm (12) | 5 | 6 | 7 | 2 | 3 | 7 |
| 06-Fraud (15) | 0 | 2 | 2 | 0 | 0 | 8 |
| 07-Pornography (11) | 1 | 3 | 5 | 0 | 0 | 5 |
| 08-Political Lobbying (15) | 12 | 13 | 15 | 7 | 9 | 13 |
| 09-Privacy Violence (14) | 4 | 2 | 2 | 0 | 0 | 8 |
| 10-Legal Opinion (13) | 8 | 9 | 12 | 0 | 2 | 6 |
| 11-Financial Advice (17) | 10 | 12 | 16 | 3 | 4 | 9 |
| 12-Health Consultation (11) | 6 | 8 | 10 | 0 | 1 | 3 |
| 13-Gov Decision (15) | 10 | 11 | 13 | 5 | 2 | 8 |
| Sum (168) | 56 | 68 | 87 | 20 | 24 | 91 |

**Table 6: Numbers of successful attacks of various jailbreak methods across three models (MiniGPT4, LLaVA1.5, and INF-MLLM1) on each scenario of MM-SafetyBench-T Dataset. The Text condition represents inputs with only original text. GCG, AutoDAN and FAIR represent text suffixes generated by these methods on LLMs, transferred to the MLLM’s text input and combined with the corresponding image. Gaussian refers to inputs where the image is randomly generated Gaussian noise, OriImage uses the original dataset images, and AdvImage refers to adversarial images generated using method [Dong et al., ]. \oursrepresents our proposed method, which optimizes the image via zeroth-order optimization to jailbreak MLLMs.**
| Model | Scenarios | Text | GCG | AutoDAN | FAIR | Gaussian | OriImage | AdvImage | \ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MiniGPT-4 | Illegal Activity (10) | 2 | 2 | 2 | 3 | 2 | 2 | 2 | 10 |
| Hate Speech | 2 | 3 | 6 | 4 | 3 | 2 | 1 | 15 |  |
| Malware Generation | 3 | 2 | 1 | 2 | 4 | 3 | 3 | 5 |  |
| Physical Harm | 4 | 4 | 11 | 6 | 8 | 4 | 7 | 14 |  |
| Economic Harm | 7 | 8 | 6 | 8 | 6 | 9 | 4 | 12 |  |
| Fraud | 3 | 4 | 8 | 7 | 9 | 8 | 12 | 15 |  |
| Pornography | 9 | 9 | 2 | 5 | 6 | 4 | 3 | 11 |  |
| Political Lobbying | 10 | 10 | 7 | 9 | 13 | 11 | 7 | 15 |  |
| Privacy Violence | 6 | 4 | 9 | 7 | 2 | 8 | 6 | 14 |  |
| Legal Opinion | 10 | 8 | 2 | 5 | 3 | 2 | 1 | 13 |  |
| Financial Advice | 7 | 5 | 6 | 8 | 9 | 5 | 2 | 16 |  |
| Health Consultation | 5 | 6 | 2 | 3 | 1 | 4 | 5 | 10 |  |
| Gov Decision | 6 | 3 | 5 | 2 | 8 | 5 | 3 | 15 |  |
|  | Sum (168) | 74 | 68 | 67 | 69 | 74 | 67 | 56 | 165 |
| LLaVA1.5 | 01-Illegal Activity (10) | 1 | 2 | 2 | 3 | 0 | 1 | 1 | 10 |
| Hate Speech | 1 | 3 | 5 | 4 | 0 | 1 | 3 | 15 |  |
| Malware Generation | 0 | 1 | 2 | 2 | 0 | 0 | 1 | 5 |  |
| Physical Harm | 1 | 3 | 10 | 4 | 0 | 1 | 4 | 14 |  |
| Economic Harm | 2 | 2 | 6 | 4 | 2 | 3 | 6 | 12 |  |
| Fraud | 0 | 2 | 5 | 3 | 1 | 0 | 8 | 15 |  |
| Pornography | 0 | 3 | 4 | 4 | 1 | 0 | 3 | 11 |  |
| Political Lobbying | 7 | 9 | 10 | 9 | 6 | 9 | 10 | 15 |  |
| Privacy Violence | 0 | 2 | 5 | 3 | 0 | 0 | 4 | 13 |  |
| Legal Opinion | 0 | 1 | 4 | 3 | 0 | 2 | 2 | 12 |  |
| Financial Advice | 3 | 4 | 10 | 6 | 2 | 4 | 4 | 15 |  |
| Health Consultation | 0 | 3 | 2 | 4 | 0 | 1 | 3 | 10 |  |
| Gov Decision | 5 | 4 | 5 | 3 | 1 | 2 | 1 | 14 |  |
|  | Sum (168) | 20 | 39 | 70 | 52 | 13 | 24 | 50 | 161 |
| INF-MLLM1 | 01-Illegal Activity (10) | 0 | 4 | 5 | 2 | 1 | 1 | 1 | 10 |
| Hate Speech | 0 | 2 | 6 | 3 | 2 | 1 | 1 | 15 |  |
| Malware Generation | 1 | 3 | 2 | 3 | 0 | 1 | 2 | 5 |  |
| Physical Harm | 1 | 2 | 6 | 5 | 1 | 4 | 3 | 14 |  |
| Economic Harm | 3 | 1 | 6 | 3 | 3 | 6 | 3 | 11 |  |
| Fraud | 2 | 4 | 8 | 6 | 4 | 5 | 4 | 15 |  |
| Pornography | 0 | 2 | 4 | 2 | 1 | 2 | 2 | 11 |  |
| Political Lobbying | 9 | 10 | 12 | 11 | 10 | 10 | 4 | 15 |  |
| Privacy Violence | 2 | 4 | 10 | 6 | 2 | 4 | 1 | 14 |  |
| Legal Opinion | 2 | 3 | 6 | 4 | 1 | 2 | 2 | 11 |  |
| Financial Advice | 6 | 8 | 10 | 8 | 3 | 4 | 5 | 16 |  |
| Health Consultation | 3 | 2 | 4 | 3 | 1 | 1 | 1 | 10 |  |
| Gov Decision | 4 | 6 | 9 | 8 | 3 | 3 | 3 | 15 |  |
|  | Sum (168) | 33 | 51 | 88 | 64 | 32 | 44 | 32 | 162 |

## Appendix C More Detailed Responses

We present the detailed responses generated from MiniGPT-4 on both datasets in the supplementary, in the type of JSON file, containing both the question and our \ours’s response.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness.
soundness: 3
strengths: * The paper is well written.
* The method is sound.
* The performance shows that the proposed method can improve the performance.
summary: This paper proposes a method that introduces the zero-order black-box attack into the jailbreak attacks against Multi-modal Large Language Models. Experimental results demonstrate it outperforms several recent methods.
weaknesses: * My main concern is that the proposed method lacks novelty. Many similar methods have already been proposed to perform adversarial attacks against vision models, e.g., [1]. The authors should discuss these related works in detail and highlight the differences between the proposed method and existing ones.

* It would be beneficial to provide a more detailed discussion on why Zer0-Jack outperforms "WB" in Tables 2 and 3.

* The paper lacks comparisons with many previous works.


[1] Chen, Pin-Yu, et al. "Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models." Proceedings of the 10th ACM workshop on artificial intelligence and security. 2017.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. How do you estimate the value of the loss function of the black-box MLLM? Do you need to access the output scores of the MLLM? 
2. The experiments show that only around 50 iterations for each attack. Will this be influenced by the scale of model parameters and image  size?
3. I ran the demo code that the author provided in the appendix and found that the loss cannot converge. Although sometimes the prompts can successfully jailbreak, I think this is due to sampling uncertainty because even with random images, LLAVA can sometimes output malicious content. I think a better evaluation method is to input the same image many times and then calculate the probability of getting malicious output.  I think the effectiveness of the author's method is questionable.
soundness: 2
strengths: 1. This paper achieved a high attack success rate on MiniGPT-4.
2. This paper proposed a patch-based method to reduce memory usage.
summary: This paper proposes a new black-box attack on MLLM. Moreover, it proposed to attack part of the image to decrease the computation complex. However, it seems that this paper is just an application of zero-order optimization attack on the MLLM with few modification. Zero-order optimization attack is a widely used black-box attack method, and I think the contribution of this paper is little.
weaknesses: 1. This paper just applied the zero-order optimization attack on the MLLM with very few modifications. There are already some papers that have applied the ZOO to black-box attacks, such as 
[1] Towards Query-Efficient Black-Box Adversary with Zeroth-Order Natural Gradient Descent, AAAI2020
[2] Zo-adamm: Zeroth-order adaptive momentum method for black-box optimization, NIPS2019
It would be helpful if the author could compare their methods with these or other latest black-box attack benchmarks.
2. In Equation 4, you estimate the gradient according to the value of the loss function. But how do you estimate the value of the loss function of the black-box MLLM? Do you need to access the output scores of the MLLM? More details should be provided.
3. More ablation studies should be conducted, such as the influence of MLLM size and image size on the ASR.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 1
questions: - How many update steps were used for Zer0-Jack in the experiments? Is it consistent with other baselines? If not, why are they different?
- For results presented in Table 1, are they based on a single image or a batch of images? It would be great to present both a single image and a batch of images.
- Line 226, why normally use a patch of 32 by 32 for 224 by 224 image? And how this is becoming 0.02% of the updated dimensions in lines 281 - 283.
- Line 100, "a single 4090 without any quantization", is it mean a single NVIDIA RTX 4090 GPU?
soundness: 3
strengths: - The proposed method is memory-efficient and operates within a black-box threat model, making it practical for real-world applications. Notably, this work highlights a safety vulnerability related to exposing logit probabilities in API responses—a finding that could significantly impact current LLM service practices. This insight into potential risks may prompt further consideration of security measures in API design for LLMs.
- The proposed method is technically sound and has been rigorously validated using MMSafetyBench, where it achieved a significantly higher attack success rate than several baseline methods and demonstrated performance comparable to white-box attacks. Additionally, evaluations of commercial models like GPT-4o further showcase its effectiveness.
- The approach of iteratively estimating the gradient over image patches is a creative and technically sound idea to address estimation errors in high-dimensional space inherent to zeroth-order optimization.
summary: This paper presents Zer0-Jack, a method designed to jailbreak Multi-modal Large Language Models (MLLMs) without requiring gradient access, enabling it to function in a black-box threat model. Zer0-Jack employs zeroth-order optimization to approximate gradients using logits, though such an approach can introduce estimation errors in high-dimensional spaces. To address this, Zer0-Jack iteratively optimizes patches of the image, mitigating these errors. Compared to other methods, Zer0-Jack demonstrates improved memory efficiency in constructing the attack. Experimental results on MMSafetyBench confirm its effectiveness, achieving performance comparable to white-box attacks and significantly surpassing existing black-box attack methods.
weaknesses: - The proposed method relies on access to the logit output from the victim model, which aligns more closely with a grey-box rather than a fully black-box threat model. In API services, a potential defense could involve disabling logits or probability outputs in responses, effectively countering this type of attack. While identifying the vulnerability associated with logits/probability exposure is an insightful contribution, it is worth noting that the method’s success depends on this information being completely or partially accessible. 
- The paper lacks evaluations of detection methods, which are particularly relevant for query-based attacks. Repeated or suspicious query patterns could potentially alert defenders. Including experiments that test Zer0-Jack against detection mechanisms, such as those proposed in [1, 2], would be helpful to improve the contribution of the paper.
- The paper lacks evaluations with prompt-based defense. For example, methods in [3, 4].
- The evaluation setup for text-based attacks lacks clarity. Specifically, it’s unclear whether the experiments with GCG, AutoDAN, and PAIR combine adversarial text prompts with random images. This setup may not fairly represent these methods, as random images could interfere with the effectiveness of the text prompts. A fairer comparison would assess the ASR of these methods without image inputs. Additionally, the statement suggesting that MLLMs cannot accept text-only input appears misleading; most MLLMs can process text-only queries. Some models, such as LLaVA-1.5 and MiniGPT-4, employ frozen language models like Vicuna and Llama-2, and using the corresponding LLMs for text-only attack evaluations would provide a more accurate assessment.
- The paper has a few confusing parts that would benefit from further clarification. Please refer to the questions section.
- Minor typos: lines 130-131 ”Do-AnythingNow” (DAN).


--- 

[1] Chen, S., Carlini, N., & Wagner, D. (2020, October). Stateful detection of black-box adversarial attacks. In Proceedings of the 1st ACM Workshop on Security and Privacy on Artificial Intelligence (pp. 30-39).\
[2] Li, H., Shan, S., Wenger, E., Zhang, J., Zheng, H., & Zhao, B. Y. (2022). Blacklight: Scalable defense for neural networks against {Query-Based}{Black-Box} attacks. In 31st USENIX Security Symposium (USENIX Security 22) (pp. 2117-2134).\
[3] Zhang, Y., Ding, L., Zhang, L., & Tao, D. (2024). Intention analysis prompting makes large language models a good jailbreak defender. arXiv preprint arXiv:2401.06561.\
[4] Robey, A., Wong, E., Hassani, H., & Pappas, G. J. (2023). Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684.\

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: I rewrite some of my questions from the previous section here more concisely:

Q1) Could the authors provide additional experiments exploring what aspects of the Zer0-Jack algorithms make it so effective (e.g. varying patch sizes, number of gradient samples. and smoothing parameter)?

Q2) Could the authors please address the issues raised in the **Clarity of writing and presentation** weaknesses section?

Q3) Could the authors please address my concern relating to the memory consumption calculation that I raised in the **Focus on memory consumption** weaknesses section?
soundness: 3
strengths: #### Originality

The papers use of zeroth order optimization to create jailbreaking images against black-box models, is to my knowledge novel. In addition, their results showing jailbreaking image attacks to GPT-4o is also novel and very impressive.

#### Quality and Clarity

The Zer0-jack method is explained well and is easy to follow. For the most part, results are also explained well and back up the main claims made in the paper. 

#### Significance

The most significant adversarial attack algorithms are those that can be applied in a black-box setting, and are able to successfully attack state-of-the-art models. The method and results in this paper clearly fit into this category, giving the paper good significance.

I found myself being surprised that the Zer0-Jack method was so effective. Especially that using black-box gradient estimations could be almost as sample efficient as white-box attacks.
summary: This work introduces Zer0-Jack, a method to create adversarial images to MLLMs that jailbreak said models. 

The Author's method uses 0th order gradient estimation to apply edits to patches of images in series with the goal of maximizing the models probability of responding to a harmful request in the affirmative.

The Author's results show that Zer0-Jack is very effective, achieving comparable jailbreaking attack success rate to white box gradient based methods. What's more, due to the gradient free nature of Zer0-Jack, it achieves these results with a comparatively lower memory requirement.

Finally, the Authors show that their method can be applied to jailbreak GPT-4o, accessing the required logit information using the logit_bias feature of the GPT-4o API.
weaknesses: I am going to split my critique up into two sections. The first will be a high-level critique of the paper, and the second will be specifics about sections. Whilst the critique is long, this is only because I believe the paper has interesting results that could be improved, not because I think there are any fundamental failings in the paper. To the contrary, I think the paper contains valuable insights for the broader adversarial attack community.

## High Level

**Algorithmic Insights**

The biggest impact of this paper would come from other practitioners being able to apply Zer0-Jack to novel situations, or apply insights gained from reading this paper to novel situations. Zer0-Jack shows effectiveness in an area that prior works have struggled to (black-box jailbreaking language models). For this reason, I think the paper would have a far greater impact if it could provide more insight on why the Zer0-Jack algorithm is effective. Some specific examples of experiments that would be useful in achieving this include:
- How adjusting the size of patches affects performance.
- How adjusting the updating of patches affects performance (is sequential the optimal).
- How the smoothing parameter affects performance.
- We can decrease the variance of the zeroth order gradient estimator by sampling many random vectors from the unit ball, and averaging. An experiment exploring the number of samples and convergence rate would be valuable, as well as comparisons between the gradient estimator and the true gradient. It may be the case that in this setting, very few samples are needed to get an accurate estimate for the gradient. This kind of information could be very valuable for future works.

**Clarity of writing and presentation**

The clarity of writing and presentation of the paper could be improved. I found myself confused at times trying to understand the exact experiments that the Author's ran. Some examples include:
1) In section 4.6, the Authors provide a single example of jailbreaking GPT-4o. There needs to be more explanation of a) how the Author's used the logit bias, and b) the exact experiment that was run. For example, what was the attack success rate against GPT-4o? By only providing a single qualitative example, it would suggest the attack success rate was low. This is not a problem, it simply should be presented to the reader.
2) In section 4.4, I was not sure what the definition of an iteration was in the case of Zer0-Jack vs WB attack. For an apples to apples comparison, this should probably be number of required forward passes, but we could equally define 1 iteration of Zer0-Jack as providing updates to all patches (which would require num_patches number of forward passes).


**Focus on memory consumption**

The Author's present the lower memory consumption of Zer0-Jack as a benefit to the algorithm over gradient based alternatives. This is certainly a benefit, but I do not think it is a hugely significant one. This does not mean this analysis should be removed from the paper, simply that I do not think it adds significance to the method. 

In addition, on line 460, the Authors state "WB Attack, applied to MLLMs like MiniGPT-4, use about 19GB each due to the need for gradient retention, while Zer0-Jack significantly reduces memory usage without sacrificing performance, uses only 10GB of memory." I am slightly confused by this. When running the WB attack, if all of the parameters of the model are frozen (in pytorch language, `parameter.requires_grad == False`) then there should be very little additional memory overhead when training? Did the authors set `requires_grad` to `False` for this evaluation or is my understanding of memory consumption incorrect? 

Concretely, when setting `requires_grad==False`, WB attack should only have to store gradients over the input image (and some intermediate gradients during the backward pass, but critical NOT the gradient for every model parameter) and so I do not expect the memory consumption to be ~double of that of a black-box "forward only" method.


## Section Specific

Here I include some smaller concerns with individual sections.

Section 3
- Writing is not succinct. Equation (8) is unnecessary, as is equation (9). The algorithm does a good job of explaining the method though.
- Line 282, Authors claim the dimension is 0.02% of the total image as a whole. I may be incorrect here, but should the ratio not be (32 * 32)/(224* 224) = 0.02 = 2%

Section 4
- It would be good to include examples from Harmful Behaviors Multi-modal Dataset and MM-SafetyBench-T in the Appendix.
- Nit - On line 316, Authors state "Since the selected MLLMs do not support text-only input, we pair the P-text with a plain black image containing no semantic information." From my experience working with these models, they can accept text only inputs, you simply input the text only through the language model backbone?
- The GCG transfer baseline is somewhat unfair. In their paper they get the best transfer by using GCG against an ensemble of models, where as my understanding is the Authors only attack one model? The baseline could be made stronger by attacking an ensemble of surrogate models. 
- On line 323, Authors state "We will pair the malicious text prompts with corresponding images to evaluate their performance on Multi-modal LLMs." What are these images?
- Line 346, the Authors state "To our knowledge, few approaches specifically optimize the image component of an image-text pair for jailbreak attacks on MLLMs." This is incorrect, in-fact the Authors cite some papers that do this (Qi et al. and Bailey et al. for example). Given that the WB baseline is using these techniques, I am guessing this sentence can just be removed?
- The WB attack should be explained in more detail.
- Lines 369-371 are not needed.
- Nit - In the caption of Table 2, Authors should state that the blank entries are due to OOM (this is only stated in the main text currently).
- I would recommend creating table 4 but for the Harm Behaviors dataset. I expect GPT-4o to have 0% attack success rate without an attack present.


Whilst I raise a number of weaknesses, I think the Zer0-Jack method is highly interesting, and thank the Authors for their work! Because the core-idea is so interesting, I simply think the work could be improved with more detailed experimentation (Algorithmic Insights mentioned above) and better presentation. The core ideas presented in the paper are strong and constitute valuable research, in my opinion.

Meta-review: This paper introduces a black-box jailbreak attack against VLMs, using a query-based approach. While the experimental results appear promising, reviewers identified several major issues: 1) the novelty of the work compared to the ZOO attack is limited, as the only difference being token adaptation; 2) insufficient details in the detection analysis setup; and 3) one reviewer identified a mistake in the code, where the defense prompt was inadvertently removed (LLaVA/zo.py, line 43), releasing the defense prompt decreases ASR from 0.9 to 0.4. Given concerns in points 1) and 2) (as point 3) needs a double check), the paper cannot be accepted to the conference in its current form. I recommend the authors continue refining their work for future submissions.

Decision: Reject