TREANT: Red-teaming Text-to-Image Models with Tree-based Semantic Transformations

Authors: Yi Liu, Guowei Yang, Gelei Deng, Feiyue Chen, Yuqi Chen, Ling Shi, Tianwei Zhang, Yang Liu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: red-teaming

Abstract:
The increasing prevalence of text-to-image (T2I) models makes their safety a critical concern. Adversarial testing techniques have been developed to probe whether such models can be prompted to produce Not-Safe-For-Work (NSFW) content. Despite these efforts, current solutions face several challenges, such as low success rates, inefficiency, and lack of semantic understanding. To address these issues, we introduce TREANT, a novel automated red-teaming framework for adversarial testing of T2I models. The core of our framework is the tree-based semantic transformation. We employ semantic decomposition and sensitive element drowning strategies in conjunction with Large Language Models (LLMs) to systematically refine adversarial prompts for effective testing. Our comprehensive evaluation confirms the efficacy of TREANT, which not only exceeds the performance of state-of-the-art approaches but also achieves an overall success rate of 88.5% on leading T2I models, including DALL·E 3 and Stable Diffusion.

Decision: reject
Reviewer scores: 8, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729582893390

thread_type:
official_review

<thread_object>
creator: reviewer_rcws

<content>
summary:
This paper introduces an automated black-box red-teaming framework designed for adversarial testing of T2I models. The proposed method leverages a tree-based semantic transformation approach to recursively refine prompts, aiming to bypass both text and image safety filters. Two key red-teaming strategies semantic decomposition (probe text safety filter) and sensitive element drowning (probe image safety filter) are applied to transform sensitive prompts into adversarial ones.

strengths:
- The paper provides a clear and understandable explanation of how T2I model safety mechanisms work, especially focusing on DALL·E 3, which is widely regarded as difficult to jailbreak.
- The high success rate in jailbreaking DALL·E 3 is impressive, showcasing the method's robustness and effectiveness.
- The introduction of the Prompt Parse Tree (PPT) and the recursive editing process to construct adversarial prompts is a novel and interesting approach.
- The method may prove to be query-efficient, which could be beneficial for practical applications.

weaknesses:
- The method of semantic decomposition could benefit from further clarification (e.g., in line 253 you say you query the T2I model, but in your algorithm you query LLM, so which do you query). It would be helpful to provide more details on how this process works.
- The construction of the Prompt Parse Tree (PPT) is interesting, but it’s not fully clear why it’s beneficial for the method. Could strategies like semantic decomposition and sensitive drowning be performed without the need for PPT? Explaining why PPT is essential would strengthen the argument.
- The comparison baselines in the experiment section could be expanded. Testing against more robust safety mechanisms, such as those in Stable Diffusion variants like ESD [R1], SLD [R2], and others, would provide a more convincing case for the method’s effectiveness.
- The method seems to be tailored specifically to DALL·E 3, which raises questions about its generalizability to other T2I models. More evaluation on different models could be valuable.

[R1] Gandikota, Rohit, et al. "Erasing concepts from diffusion models." ICCV (2023) \
[R2] Schramowski, Patrick, et al. "Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models." CVPR (2023)

questions:
- Clarification on what you mean by "misleading the safety filter" as opposed to "bypassing" it? A clear distinction would be helpful, since you state that your method generated adversarial prompts that actually bypass the safety filters of T2I models.
- How do you convert the PPT back into a prompt (your final adversarial prompt) after decomposition? Is this done manually or through an automated process, such as querying a LLM?
- Provide further clarification on how the method determines whether the adversarial prompt is blocked by the text filter or the image filter, as different red-teaming strategies are triggered in your method.
- Your method seems to rely heavily on querying a LLM for constructing the PPT, performing semantic decomposition, and implementing sensitive drowning. More details on how you ensure the LLM accurately understands the task, particularly in terms of constructing the PPT correctly?
- Since models like SD1.4, SD2.1, and SDXL have weaker safety mechanisms compared to DALL·E 3, it would be helpful to provide more explanation on why the results for these models are not better than the baselines. The mention of these T2I models potentially not understanding the structural information of the language need more elaboration. Is your generated adversarial prompts somewhat different from your baselines?
- In Table 1, the success rate of TREANT-SED appears quite low. It might be useful to discuss this in more detail, particularly since TREANT-SED employs prompt dilution (adding non-sensitive information into the prompt), which has been shown to be effective in many prior studies.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730367763078

thread_type:
official_review

<thread_object>
creator: reviewer_7fjw

<content>
summary:
The paper introduces TREANT, a novel automated red-teaming framework designed to enhance adversarial testing of T2I models. It utilizes tree-based semantic transformations, semantic decomposition, and sensitive element drowning strategies, leveraging LLLMs to refine adversarial prompts. TREANT demonstrates superior performance in generating NSFW content on DALL·E 3 and Stable Diffusion.

strengths:
1. Red-teaming T2I models is an important topic to investigate. It is also a practical issue.
2. Overall, this paper is easy to follow and well-organized.
3. The proposed method is easy to implement and can perform black-box red-teaming, which should be efficient.

weaknesses:
For method:

1. The proposed method lies in the category of prompt engineering, unaware of the T2I model and without any black-box optimization strategy. I fail to see clear novelty compared to token replacement and insertion. Especially, in some cases, this method performs worse than BAE or even SneakyPrompt.
2. It would be better if the authors provided any form of guarantee or evidence showing the proposed algorithm can cover more possible prompts toward NSFW semantics.
3. It is misleading that the proposed method is designed to bypass image safety filters. However, GPT-4o is used to check whether NSFW content is generated, which means GPT-4o can serve as a good image safety filter to drop all the malicious prompts generated by TREANT.

For evaluation and comparison:

4. The paper fails to include important baselines, such as [a].
5. Only unprotected models are considered, the authors are encouraged to investigate the effectiveness of models such as SLD[b].
6. Please try to include DiT-based models like PixArt-α[c].
7. Please consider more image safety filters for evaluation, and also check the attack transferability.

[a] MMA-Diffusion: MultiModal Attack on Diffusion Models. Yang et al. CVPR 2024.

[b] Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models. Schramowski et al. CVPR 2023.

[c] PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. Chen et al. ICLR 2024.

questions:
1. For red-teaming, a white-box attack should also be acceptable. Is there any special case for the need for black-box attacks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621546799

thread_type:
official_review

<thread_object>
creator: reviewer_nq2a

<content>
summary:
I think this paper is targeting at an important problem and the method seems to be very effective. More specifically, this paper presents TREANT, a tool for testing text-to-image (T2I) models to see if they can produce inappropriate content. TREANT works by changing sensitive words in prompts to less sensitive ones and adding harmless details to help avoid safety checks. This approach has a high success rate of 88.5% in getting T2I models to generate unwanted images, performing better than other methods. The tool aims to improve how we evaluate the safety of these models.

strengths:
The TREANT framework has several strengths that make it effective for testing text-to-image (T2I) models. It achieves a high success rate of 88.5% in generating unwanted content, demonstrating its reliability. 

TREANT operates automatically, allowing for quick and easy testing without manual effort. It uses smart techniques to bypass safety checks on both text and images while maintaining the original meaning of prompts. The organized tree structure helps manage prompts efficiently, reducing wasted queries. 

Additionally, it has been thoroughly tested across various scenarios, proving its robustness, and the authors provide open access to their code and data for others to use.

weaknesses:
1. It would be great if the authors could include more qualitative results in their paper because it would really help us understand how well TREANT performs. The success rate of 88.5% sounds impressive, but sharing some specific examples of the images generated would give us a clearer idea of what’s going on. It would be useful to see the types of content TREANT produces and how it handles different situations. I get that safety concerns might limit what they can show, but if they can share more examples, it would definitely make the paper more engaging and helpful for readers. E.g. in figure 7 it would be better if more examples can be shown for a more convincing conclusion,

2. And more analysis together with the qualitative results could be more helpful.

3. Besides, while TREANT is trying to improve scalability, it would be really helpful to talk more about how well it can work with different models and handle prompts that are more complex than the ones they tested.

questions:
See the weakness for the questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731120968385

thread_type:
official_review

<thread_object>
creator: reviewer_uvgk

<content>
summary:
This paper proposed red-teaming framework for adversarial testing of T2I models to detect their ability to generate NSFW content. Specially, the proposed TREANT utilizes  tree-based semantic transformations to systematically refine adversarial prompts for effective testing.

strengths:
1. The proposed method is able to adjust and refine prompts using semantic decomposition and sensitive element drowning enhances its practical utility and effectiveness.
2. The experimental results demonstrate the efficacy of TREANT through comprehensive evaluations, achieving an overall success rate of 88.5% on leading T2I models like DALL-E 3 and Stable Diffusion.

weaknesses:
1. Missing related work white-box adversarial prompt attack baselines such UnlearnDiffAtk [1], which is even able to enforce unlearned T2I model to generate images containing harmful contents. 

2. To demonstrate scalability, it is better to add time efficiency as one main evaluation metric.

[1] To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images … For Now, ECCV’24

questions:
Check weakness section
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a new red-teaming method for text-to-image models using a tree-based approach. The proposed method shows enhanced capability in generating NSFW content on DALL-E 3 and Stable Diffusion. Initially, reviewers raised concerns about unclear presentation, limited technical novelty, lack of qualitative examples, and insufficient analysis of generalization capability.

The authors provided thorough responses in their rebuttal, and the paper finally received three positive scores (6,6,8) and one negative score (5). While the rebuttal addressed many concerns, the AC finds the paper still lacks comprehensive comparisons with other adversarial attack methods, which would better demonstrate the advantages, insights, and analysis of the proposed method.

Therefore, the AC considers this paper a borderline one between acceptance and rejection. Given the highly competitive nature of ICLR submissions, the AC recommends the rejection of the current version of the paper.