Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step

Authors: Wenxuan Wang, Kuiyi Gao, Zihan Jia, Youliang Yuan, Jen-tse Huang, Qiuzhi Liu, Shuai Wang, Wenxiang Jiao, Zhaopeng Tu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Jailbreak Attack, Text-to-Image Models, Safety

Abstract:
Text-based image generation models, such as Stable Diffusion and DALL-E 3, hold significant potential in content creation and publishing workflows, making them the focus in recent years.
Despite their remarkable capability to generate diverse and vivid images, considerable efforts are being made to prevent the generation of harmful content, such as abusive, violent, or pornographic material.
To assess the safety of existing models, we introduce a novel jailbreaking method called Chain-of-Jailbreak (CoJ) attack, which compromises image generation models through a step-by-step editing process.
Specifically, for malicious queries that cannot bypass the safeguards with a single prompt, we intentionally decompose the query into multiple sub-queries. The image generation models are then prompted to generate and iteratively edit images based on these sub-queries.
To evaluate the effectiveness of our CoJ attack method, we constructed a comprehensive dataset, CoJ-Bench, encompassing nine safety scenarios, three types of editing operations, and three editing elements.
Experiments on four widely-used image generation services provided by GPT-4V, GPT-4o, Gemini 1.5 and Gemini 1.5 Pro, demonstrate that our CoJ attack method can successfully bypass the safeguards of models for over 60\% cases, which significantly outperforms other jailbreaking methods (i.e., 14\%).
Further, to enhance these models' safety against our CoJ attack method, we also propose an effective prompting-based method, Think Twice Prompting, that can successfully defend over 95\% of CoJ attack.
We will release our dataset and code to facilitate the AI safety research.

Decision: reject
Reviewer scores: 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730128833339

thread_type:
official_review

<thread_object>
creator: reviewer_czfl

<content>
summary:
The paper introduces a novel method called Chain-of-Jailbreak (CoJ) attack that bypasses safeguards in image generation models. The method proposes a decomposition method that breaks down malicious prompts into a series of sub-queries. The experiments demonstrate that such a method is able to jailbreak open-source black-box models like GPT-4V, GPT-4o, and Gemini 1.5 by generating harmful contents in various malicious categories. The authors also introduce a dataset named “CoJ-Bench”, which consists of various jailbreaking prompts in a wide range of safety scenarios. The paper claims to achieve high jailbreak success rates (~60%), which outperforms traditional prompt-based attacks. In addition, the authors introduce a defense mechanism against the proposed attack called “Think Twice Prompting”, which asks the model to check the safety implications once more before generating the images.

strengths:
1. The paper proposes a novel yet simple approach to bypass image generation model safeguards, which are known to be built stronger than open-source models such as Stable Diffusion. It also demonstrated a huge outperformance against previous jailbreaking methods in text-to-image generation. 

2. The paper also introduces the edit operation in the decomposition process is structured systematically rather than simply tokenizing the input prompt. The diversity of the attack method showed the effectiveness of jailbreaking these models.

3. The CoJ benchmark covers a wide range of safety scenarios, with an even more detailed division in the subgroups of each safety category of the benchmark, which enhances reproducibility and evaluation standards.

weaknesses:
1. The defense method is not clearly explained and not realistic enough. How and when are the defense prompts inputted to the image generation? 

2. A threat model of this type of attack is not specified. When are these attacks be utilized in a real-life scenario?

questions:
1. What could be the reason behind why “Insertion-then-Delete” is the most effective attack? In general, why was decomposition effective compared to traditional attack methods?

2. The traditional attack methods are not cited as well as not explained in the background section.

3. In the automatic evaluation, what is the purpose of observing the response of LLM? How does GPT-4 responding No indicate that the image generation models not refusing the malicious query?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730508249191

thread_type:
official_review

<thread_object>
creator: reviewer_gcxl

<content>
summary:
This paper proposes a method to make text-to-image models generate images with harmful content. The harmful image is defined as an image with harmful words in it. The idea is to decompose the prompt into multiple sub-queries, which gradually generate a harmful image. A benchmark of queries for such harmful images are also collected and used for evaluation.

strengths:
Safety of text-to-image models is an important topic. 

A method is proposed to generate a specific type of harmful images. 

A benchmark is collected.

weaknesses:
The scope is a little bit limited. Only images with harmful words can be generated by this method. This should be made clear from paper title and abstract. 

To generate such images, more straightforward approach may be applicable. E.g., directly merge harmful words with images. Note that, an attacker jailbreaks a GenAI model to generate harmful images, and still needs to propagate them to cause real harms for other people. To generate the types of harmful images considered in this work, an attacker may not need a text-to-image model and thus may not need the proposed attack. 

Comparison with baseline methods is missing. What are the alternative approaches to generate such harmful images? Does the attacker have to use a text-to-image model? Even if text-to-image model is needed, any other baseline methods can be used to generate such harmful images? It is not clear in the current paper.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730733295943

thread_type:
official_review

<thread_object>
creator: reviewer_tbxx

<content>
summary:
The paper introduces the "Chain-of-Jailbreak" (CoJ) attack, a method to bypass safeguards in text-based image generation models like GPT-4V and Gemini 1.5. By breaking down malicious queries into harmless sub-queries and using iterative editing, CoJ can generate harmful content that would otherwise be blocked. The study highlights significant vulnerabilities in current models, achieving a 60% success rate in bypassing safeguards. To counter this, the authors propose "Think Twice Prompting," a defense strategy that prompts models to internally evaluate the safety of the content before generation, successfully defending against 95% of CoJ attacks.

strengths:
Innovative Methodology: The introduction of the Chain-of-Jailbreak (CoJ) attack is a significant advancement. By decomposing malicious queries into harmless sub-queries and using iterative editing, the paper presents a novel approach to bypassing safeguards in text-based image generation models.

Comprehensive Evaluation: The authors have conducted extensive experiments across multiple models (GPT-4V, GPT-4o, Gemini 1.5, and Gemini 1.5 Pro) and scenarios. This thorough evaluation demonstrates the robustness and effectiveness of the CoJ attack, achieving a high success rate of 60%.

Proposed Defense Mechanism: The paper doesn't just identify vulnerabilities but also proposes a practical solution. The "Think Twice Prompting" defense strategy, which prompts models to internally evaluate the safety of the content before generation, shows a high defense success rate of 95%.

weaknesses:
Method Robustness: The authors propose using Edit Operations and Edit Elements to break down a malicious query into sub-queries. In the implementation, they manually apply this approach to five seed queries and leverage a large language model (LLM) to generalize these examples to other queries. However, my concern is whether the model consistently adheres to the principles of the proposed Edit Operations and Edit Elements. It would be helpful if the authors could elaborate on the reliability of this approach.

Number of Sub-Queries: As illustrated in Figure 1, the malicious query is divided into three sub-queries. This raises the question of how many sub-queries would be optimal for other queries. Is there a “best” decomposition, and how can it be identified? While the authors rely on an LLM for this task, I am concerned about the LLM’s ability to consistently find the optimal decomposition.

Choice of LLM: The authors specify using Mistral-Large-2 for modifying malicious queries automatically. It would be informative to know whether other models were considered and if similar performance could be achieved with smaller models requiring less computational power. This consideration is especially relevant for attackers with limited resources who may not have access to high-powered computational hardware.

questions:
1. How reliably does the model follow Edit Operations and Edit Elements across queries?

2. What is the optimal number of sub-queries, and can the LLM consistently find it?

3. Were other models tested, and can similar results be achieved with less computationally intensive options?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732227457515

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all reviewers for their insightful comments. 

We are glad to receive the positive feedback from reviewers, particularly: 
 
1. Important Topic [gCXL] 

2. Innovative Methodology [tBXX, gCXL, czFL]. 

3. Useful Benchmark [gCXL, czFL] 

4. Comprehensive Evaluation [tBXX, czFL] 

5. Defense Mechanism [tBXX] 
 
In addition to the above comments, we received valuable feedback from the reviewers, which helped us improve the quality of the paper. We implement several new experiments and modifications according to the comments and address every point raised by reviewers in the individual responses. The modifications for this rebuttal are as follows: 
 
1. Human Annotation on automatically generated sub-queries in Appendix Section D [tBXX -W1]. 

2. Performance on other LLM on automatic sub-queries generation in Appendix Section E [tBXX -W3]. 

3. Add pure image-level editing statistics and examples in Section 2.3 and Appendix Section G  [gCXL-W1]. 

4. Illustrate our advantage compared with more baseline methods in Appendix Section I [gCXL-W3]. 

5. Specify the practical usage of the defense method in Section 4.4 [czFL-W1]. 

6. Specification of the threat model in Section 2.1 [czFL-W2]. 

7. Introduce more traditional attack methods in related work in Appendix Section A [czFL-Q2].
</content>
</thread_object>

</thread>


Metareview:
metareview:
1x borderline accept, 2x borderline reject. This submission introduces a multi-turn “chain-of-jailbreak” approach that decomposes malicious image-generation prompts into smaller sub-queries to bypass model safeguards. The reviewers agree on the (1) novelty in applying iterative editing to text-to-image jailbreak, (2) demonstration of vulnerabilities across major black-box services, and (3) broad coverage of various malicious categories via a newly proposed benchmark. However, they note (1) insufficient threat-model specification and real-world justification, (2) confusion around generating purely image-based versus text-based harm, and (3) incomplete comparisons with alternative jailbreak or baseline methods. The authors have followed up with clarifications about scope and additional experiments, partly addressing these weaknesses and questions but not fully resolving the concerns about thorough baselines and real-world scenarios, so the AC leans to not accept this submission.