Ring-A-Bell! How Reliable are Concept Removal Methods For Diffusion Models?
Authors: Yu-Lin Tsai
Chia-Yi Hsu
Chulin Xie
Chih-Hsun Lin
Jia You Chen
Bo Li
Pin-Yu Chen
Chia-Mu Yu
Chun-Ying Huang
Venue: ICLR 2024 poster (2024)
Keywords: redteaming
stable diffusion
text-to-image model
Primary area: societal considerations including fairness, safety, privacy

Abstract: Diffusion models for text-to-image (T2I) synthesis, such as Stable Diffusion (SD), have recently demonstrated exceptional capabilities for generating high-quality content. However, this progress has raised several concerns of potential misuse, particularly in creating copyrighted, prohibited, and restricted content, or NSFW (not safe for work) images. While efforts have been made to mitigate such problems, either by implementing a safety filter at the evaluation stage or by fine-tuning models to eliminate undesirable concepts or styles, the effectiveness of these safety measures in dealing with a wide range of prompts remains largely unexplored. In this work, we aim to investigate these safety mechanisms by proposing one novel concept retrieval algorithm for evaluation. We introduce Ring-A-Bell, a model-agnostic red-teaming scheme for T2I diffusion models, where the whole evaluation can be prepared in advance without prior knowledge of the target model.
Specifically, Ring-A-Bell first performs concept extraction to obtain holistic representations for sensitive and inappropriate concepts. Subsequently, by leveraging the extracted concept, Ring-A-Bell automatically identifies problematic prompts for diffusion models with the corresponding generation of inappropriate content, allowing the user to assess the reliability of deployed safety mechanisms. Finally, we empirically validate our method by testing online services such as Midjourney and various methods of concept removal. Our results show that Ring-A-Bell, by manipulating safe prompting benchmarks, can transform prompts that were originally regarded as safe to evade existing safety mechanisms, thus revealing the defects of the so-called safety mechanisms which could practically lead to the generation of harmful contents. In essence, Ring-A-Bell could serve as a red-teaming tool to understand the limitations of deployed safety mechanisms and to explore the risk under plausible attacks. Our codes are available at https://github.com/chiayi-hsu/Ring-A-Bell.

Published as a conference paper at ICLR 2024
RING-A-BELLI How RELIABLE ARE CONCEPT RE- MOVAL METHODS FOR DIFFUSION MODELS?
Chia-Yi Hsu; Yu-Lin Tsai" National Yang Ming Chiao Tung University (chiayihsufe hsu8115 uriahl 001 ) @m 1 com
Chulin Xie
University of Illinois at Urhana Champaign edu
Chih-Hsun Lin, Jia-You Chen
Bo Li
National Yang Ming Chiao Tung University (pkevawin334, justin041510)@gma com
University of Illinois at Urbana Champaign University of Chicago
1bo8 i nllois s. edu, bol @uchi @ecchicago... edu
Pin-Yu Chen
Chia-Mu Yu, Chun-Ying Huang
IBM Research
National Yang Ming Chiao Tung University chi aaiyumuuu@aaa gma com, chuang@es actu.edu................
pin-yu. chen ibm. com
ABSTRACT
Diffusion models for text-to-image (T2I) synthesis, such as Stable Diffusion (SD), have recently demonstrated exceptional capabilities for generating high-quality content. However, this progress has raised several concerns of potential mis- use. particularly in creating copyrighted, prohibited, and restricted content, NSFW (not safe for work) images. While efforts have been made to mitigate such problems, either by implementing safety filter at the evaluation stage by fine-tuning models to climinate undesirable concepts or styles, the effective- ness of these safety measures in dealing with wide range of prompts remains largely unexplored. In this work, we aim to investigate these safety mechanisms by proposing novel concept retrieval algorithm for evaluation. We introduce Ring-A-Bell, model-agnostic red-teaming tool for T2I diffusion models, where the whole evaluation can be prepared in advance without prior knowledge of the target model. Specifically, Ring-A-Bi A-Blll first performs concept extraction to obtain holistic representations for sensitive and inappropriate concepts. Subsequently, by leveraging the extracted concept, Ring-A-Bell automatically identifies prob- lematic prompts for diffusion models with the corresponding generation of inap- propriate content, allowing the user to assess the reliability of deployed safety mechanisms. Finally, we empirically validate our method by testing online ser- vices such as Midjourney and various methods of concept removal. Our results show that Ring-A-Bed by manipulating safe prompting benchmarks, can trans- form prompts that were originally regarded safe to evade existing safety mech- anisms, thus revealing the defects of the so-called safety mechanisms which could practically lead the generation of harmful contents. In essence, Ring-A-Bell could serve as a red-teeeeee rod-teaming tool to understand the limitations of deployed safety mechanisms and explore the risk under plausible attacks. Our codes are avail- able at pttps: / git github..ttt. com/ chiayi hsu/ Ring- A- --l---BBellll
CAUTION: This paper includes model content that may contain offen- sive or distressing material.
INTRODUCTION
Generative Al has made significant breakthroughs in domains such as text (OpenAl 2023
Touvron
et al. 2023), image (Ho al. 2020), and code generation (Chowdhery al. 2022 Among the
"cqual contribution

---

Published as a conference paper at ICLR 2024
areas receiving considerable attention within generative AI, text-to-image (T2I) generation stands out. The exceptional performance of today's T2I diffusion models is largely due to the vast reservoir of training data available the Internet. This wealth of data enables these models to generate a wide variety of content, ranging from photorealistic scenarios to simulated artwork, anime, and even artistic images. However, using such extensive Internet-derived training data presents both challenges and benefits. Particularly, certain images crawled from the Internet contains restricted content, and thus the trained model leads to memorization and generation of inappropriate images, including copyright violations, images with prohibited content, as well as NSFW material.
To achieve this goal, recent research has incorporated safety mechanisms into diffusion models to prevent models from generating inappropriate content. Examples of such mechanisms include stable diffusion with negative prompts (Rombach et al., 2022), Safe Latent Diffusion (SL.D) (Schramowski et al.. 2023), Erased Stable Diffusion (ESD) (Gandikota et al. 2023), and so on. These mechanisms are designed to either constrain the text embedding space during the inference phase, or to fine- tune the model and steer it to avoid producing copyrighted inappropriate images. While these safety mechanisms have proven effective in their respective evaluations, one study (Rando et al. 2022) red- teamiin Stable Diffusion (e.g., actively searching for problematic prompts) highlights some potential shortcomings. Specifically, |Rndo et al. (2022) found that the state-of-thh-ar Stable Diffusion model, equipped with NSFW safety filter, can still generate sexually explicit content if the prompt is filled with excessive wording that could evade the safety check. However, such method requires manual selection of prompts, and is typically cumbersome and not scalable for building a holistic inspection of T2I models.
On the other hand, as recent T2I diffusion models have grown significantly in model size, reaching parameter counts up to billions (Ramesh et al. 202 2022 Saharia et al. 2022), fine-tuning these models becomes prohibitively expensive and infeasible when dealing with limited computational resources during red-teaming tool development. Consequently, in this study, we use prompt engi-
neering (Brown et al. 2020 al. 2020 Li & Liang. 2021, Jiang et al., 2020 Petroni et al. 2019
Lester et a 2021 Schick & Schütze 2021 Shin et al. 2020 Shi et al. 2022 Wen et al. 2023) as
the basis for our technique to construct problematic prompts. One of the guaranteed benefits is that such a method could allow us to fine-tune prompt, which is order of magnitude less computation- ally expensive than fine-tuning the whole model, while also achieving comparable performance. On the other hand, the scenario is more realistic since the user could only manipulate the prompt input without further modifying the model.
In this work, present Ring-A-Bell, framework that aims to facilitate the red-teaming of T21 diffusion models with safety mechanisms to find problematic prompts with the ability to reveal sen- sitive concepts (e.g.. generating images with prohibited concepts such as "nudity" and "violence"). This is achieved through the approach of prompt engincering techniques and the generation of our adversarial concept database. In particular, we first formulate a model-specific framework demon- strate how to generate such an adversarial concept. Then, we proceed to construct a model-agnostic framework where knowledge of the model is not assumed. Finally, we introduce Ring-A A-Bell to en- able the automation of finding such problematic prompts. Ring-A-Bell simulates real attacks since red-teaming puts ourselves in the attacker's 's shoes to identify the weaknesses that can used against the original model. We emphasize that Ring-A-Bell uses only a text encoder (e.g., text encoder in CLIP model) and is exccuted offline, which is independent of any target T21 models and online services for evaluation. Furthermore, model can be attributed to the novel design of concept extraction, so that it is able to uncover im- plicit text-concept associations, leading to efficient discovery of adversarial prompts that generate inappropriate images. On the other hand, such problematic prompts identified by Ring-A A-nglA-Bell serve two purposes: they help in understanding model misbehavior, and they serve as crucial references for subsequent efforts to strengthen safety mechanisms. We summarize our contributions below.
reason that such success under black-box access of target
We propose Ring-A-Bi A-Bell, which serves as a prompt-basee based concept testing framework that gener- ates problematic prompts red-team T2I diffusion models with safety mechanisms, leading to the generation of images with supposedly forbidden concepts.
In ding-A-Bell. concept extraction is based solely on either the CLIP model or general text en- coders, allowing for model-inneppenden prompt eromptevallation evaluation, resulting in efficient offline evaluation. Our extensive experiments evaluate a wide range of models, ranging from popular online ser- vices to sfate-of-the-art concept removal methods, and reveal that problematic prompts generated
2

---

Published as a conference paper at ICLR 2024
by Ring-A-Bi A-Bell can increase the success rate for most concept removal methods in generating inappropriate images by more than 30%.
RELATED WORK
We present a condensed version of the related work and refer the detailed ones in Appendix B
Red- red-Teaming Evaluation Tools for AL Red-teaming, a cybersecurity assessment technique, aims to actively search for vulnerabilities within information security systems. Originally focusing on eybersecurity, the concept of red- red-teaming has also been extended to machine learning, with focus on language models (Perez et al. 2022 Shi et al. 2023 Lee et al. 2023) and more recently, T2I models (Zhuang et al. 2023 Qu al. 2023 Chin al. 2023) We note that concurrent work, P4D (Chin et al., 2023). also develops reaming tool of text-to-image diffusion models, with the main weakness being the assumption of white-box access of target model. For more details, leave the discussion and comparison to Section 3.2
Diverse Approaches in Prompt Engineering. Prompt engineering seeks to improve the adapt-
ability of pre-trained language models to various downstream tasks (Duan et al.
2023 Gal et al.)
2023 He al. 2022) by modifying input text with carefully crafted prompts. This approach,
based on representatio. can be classified into hard prompt (discrete) (Brown et al.) 2020 Schick & Schütze 2021; Jiang et al. 2020 Gao et al., (202T) and soft prompt (continuous) (Lester al. 2021]
Li & Liang (021] where the former represents discrete word patterns and the latter represents the continuous embedding vector. As both possess pros and cons, some efforts are made unify the advantage of both settings (Shin et al. 2020 Shi et al. 2022
Wen et al. 2023
Text-to-Image Diffusion Model with Safety Mechanisms. To address the misuse of '12 models for sensitive image generation, several approaches have been proposed to combat this phenomenon. Briefly, such methods fall into the following two directions: detection-based (Rando al.) 2022) and rremoval-baseee (Rombach et al., [022 Schramowski et al. 2023 Gandikota et al. 2023] Kumari al. 2023 Zhang et al. (023). Detection-
methods aim to remove inappropriate content by filtering it through safety checkers while removal removal-based methods tries to steer the model away from those contents by actively guiding inference phase or fine-tuning the model parameters.
MAIN APPROACH
We aim to evaluate the effectiveness of safety mechanisms for T2I diffusion models. First, we math- ematically construct an attack when the target model is within our knowledge (i.e., model-specific ovaluation) in Section 3.2 Then, based on our empirical findings, we construct a model-agnostic evaluation, Ring-A Ring-A-Bell, by assuming only availability of general text encoder in Section 3.3
3.1 BACKGROUND
In this section, we provide a brief explanation of diffusion models and their latent counterparts, along with their mathematical formulations. These mathematical formulations illustrate how they work to generate data and support conditional generation.
Diffusion Model. Diffusion models (Sohl-Dicksted) et al., 2015 Ho et al., 2020) are generative models designed to simulate the process of iteratively generating data by reducing noise from inter- mediate data states. This denoising process, is the inverse of the forward diffusion process, which progressively predicts and reduces noise from the data. Given input image Zo. Denoising Dif- fusion Probabilistic Models (DDPM) (Ho et al. 2020) generate an intermediate noisy image x, at time step through forward diffusion steps, i.e., iteratively adding noise. Mathematically, by x; = za + /1 - a,e where a is the demenddnndnnt hyperparameter and is the Gaussian noise, that the last iterate after T steps simulates the standard Gaussian, .T (0, I). Fur- thermore, the denoising network ea(-) aims to predict the previous step iterate It-1 and resorts to training with the loss I. E Feo.ne--ioo.nl c0(22, 1)?
given
3

---

Published as a conference paper at ICLR 2024
Latent Diffusson Model. Rombach et al. (2022) proposes the Latent Diffusion Model (LDM), com- monly known as the Stable Diffusion (with minor modification), as an improvement by modeling both the forward and backward diffusion processes within the latent space. This improvement di- rectly mitigates the efficiency challenge faced by DDPM, which suffers from operating directly in pixel space. Given the latent representation
(x())) of input image and its corresponding representational concept c, where E' denotes the encoder of VAE (Kingma & Welling 2013). LDM first obtains the intermediate noisy latent vector 2 time step t. Similar to DDPM, parameterized model co with parameter 0 is trained to predict the noise co(2t, c. t), aiming to denoise 2; based on the intermediate vector 2t. time step t, and concept c. The objective for learning this conditional generation process is defined as L Wio, c0(21, c, e e.
3.2 MODEL-SPECIHIC EVALUATION
To construct a model-specific evaluation, we denote our original unconstrained diffusion model (Ho al., 2020 Rombach et al., 2022) (i.e., without any safety mechanisms) as cg(.). On the other hand, models with a safety mechanism are denoted as egr (-). Given target concept (e.g.. nudity, violence, or artistic concept such as "style of Van Gogh"), we want to find an adversarial concept e such that, given trajectory, 20: 21...., 2T (typically the one that produces the inappropriate image 20), two models can be guaranteed to have similar probabilities of generating such a trajectory, i.c.,
Peo (=o #1...., žτ (c)
Peg. (F0: ø1, :...... (T);
where P is the probability that the backward process is generated by the given noise predictor. When minimizing the KL divergence between two such distributions, the objective is expressed as Lwhites
L Lwhitte tee cgr ( E))11 {=1
where denotes the weight on the loss. The detailed derivation of Lwhite can be found in the Appendix [ To briefly explain the attack process, given a forward process starting with an inap- propriate image 2o, we want the backward process produced by the noise predictor eo(-) and ear (-) under the original concept e and the adversary concept to be similar, and thus output similar im- ages. Namely, we have ē == arg min; Lwhate (2).
While the model-specific evaluation seems to produce promising results in theory, it has a few limits when applied in practice. First, to optimize Lushites one is required to assume prior knowledge of the model Egr with the safety mechanism. Second, since the loss is written in the form of an expectation, the method may require multiple samples to obtain accurate estimation. Furthermore, the adversary is assumed to be in possession of 6g (e.g., an '21 diffusion model without a safety mechanism), in order successfully elicit the attacking prompt. Lastly, the model architecture of eo and cgr should also be similar such that the intermediate noise can be aligned and the L2 loss is then meaningful.
We note that concurrent work, P4D (Chin al. 2023). ) presents a similar solution to our model- specific evaluation without the derivation of minimizing KL divergence. In particular, they aim to select problematic prompts Patse by forcing the unconstrained model eg to present similar behavior to that of one with safety mechanism. Furthermore, P4D (Chin et al.] 2023) is special case of our formulation by setting the offline stable diffusion model (without an NSFW safety filter) as eo and sampling particular time step f. instead of summing the loss from all time steps. In particular, its loss is set as LY4D = (]co(2), W(P), (P), t) cør (2,, Paiser t) where (P) is the soft embedding of the harmful prompt P. Moreover, P4D suffers from the above shortcomings, i.e., relying on the white-box access of target model, and requires further design to generalize.
3.3 MODEL-AGNOSTIE EVALUATION
On the other hand, instead of assuming the white-box access of target models, we focus on construct- ing attacks only with black-box access of eg., In particular, concept directly from probing the modified model 6gr and cg. To address such a challenge, we propose Ring-A- Belll with its overall pipeline shown in Figure []] The rationale behind Ring-A-Bell is that current T2I models with safety mechanisms either learn to disassociate or simply filter out relevant words of the target concepts with their representation c, and thus the detection removal
no longer obtain the adversarial
4

---

Published as a conference paper at ICLR 2024
(PT Aei
CLiP
U
mustrsc PT
Ganeapt
100
[Pf lim
CLUP
CLUP
1(P) Pav== ((P) +
97
[CCP =
Discomery
Noadon
CLP
TCP) (I((P)
1
An overview of the proposed Ring-A-Bell framework, where the problematic prompts generation is model-agnostic and can be carried out offline.
Figure 1:
of such concepts may not be carried out completely if there exist implicit text-concept associations embedded in the T2I generation process. That is, Ring-A-Bell aims to test whether supposedly removed concept can be revoked via our prompt optimization procedure.
In Ring-A-Bell, first generate the holistic representation of concept by collecting prompt pairs that are semantically similar with only difference in concept c. For instance, as in Figure [] the "clashed / peaceful" and "bleeding fine" are differing in the concept "violence". Afterwards, the empirical representation è of e is derived as
N Σ (1(P))
ē: ::= N
(3)
where f()) denotes the text encoder with prompt input (e.g., text encoder in CLIP) and fe denotes the absence of concept Simply put, given prompt pairs {Pr. P. with similar semantics but contrasting in the target concept c, such as (Prompt 1, Prompt 2) in Figure thhat cept "violence", we extract the empirical representation è by pairwise subtraction of the embedding and then averaging over all pairs. This ensures that the obtained representation does not suffer from context-dependent influence, and by considering all plausible scenarios, we obtain the full semantics underlying the target concept c. Similar attempts can also be seen in Zhuuang et al. (2023) where only the sign vectors are used to induce concept removal/generation. We reter the readers to Appendix HH for the detailed generation prompt-pairs. As for the choice and number of prompt pairs, we refer to the ablation studies in Section
that represent the con-
After obtaining Bing-ABBell transforms the target prompt into the problematic prompt P. In particular, Ring-A-Bae first uses the soft prompt of P and to generate P Peont( (?) as
Poont := f(P) + - c,
(4)
where 1) is the strength coefficient available for tuning. In short, Poont is the embedding of P infused with varying levels of concept c. Finally, we generate P by solving the optimization problem below
min [P) Pconeell subject to P E S.
(5)
where K the length of the query and is the set of all word tokens. Here, the variables to be optimized are discrete with the addition that typically S consists of a huge token space. Hence, we adopt the genetic algorithm (GA) (Sivanandam & Deepa 2008) as our optimizer because its ability to perform such a search over large discrete space remains competitive. We also experiment with different choices of the optimizer in Section 4.3
It is evident from the above illustration that Ring-A-Bell requires prior knowledge of the model to be evaluated except for the access of the text encoder (i.e., the access of f(() in Eqs. (1)) Fur- thermore, Ring-A-Bo A-Blll presents readily available database that stores various sensitive concepts. Any user could utilize the concepts identified, automatically create problematic prompts offline, and further deploy them online, demonstrating the practicality of Ring-A-Bell
4 EXPERIMENTS
Dataset. We evaluate the performance of Ring-A-Bod on the I2P dataset (Schramowski et al. 2023), an established dataset of problematic prompts, on the concepts of nudity and violence. We
5

---

Published as a conference paper at ICLR 2024
select 95 nudity prompts where the percentage of nudity is greater than 50%. For the concept of violence, to avoid overlapping with nudity prompts, we selected a total of 250 prompts with a nudity percentage less than 50%, inappropriateness percentage greater than 50%, and labeled as harmful.
Online Services. To evaluate if online services effective in rejecting the generation of inappro- priate images, we test four well-known T21 online services: Midjourney DALL-E 2 Gen- for images), and af of Stable Diffusion XI.). The prompts used to test online services are chosen from prompts generated by Ring-A- SingA-BBell for red-teaming concept removal models.
Baseline. For the baseline, we select QF-Attack (Zhuang et al. 2023) since QF-Attack is con- ceptually similar ours, In particular, QF-Attack crafts an adversarial prompt by minimizing the cosine distance between f((x) and f(='), where I represents the original prompt, x'' represents the adversarial prompt, and f()-) denotes the text-encoder of CLIP model. We conduct the targeted attack of the QF-Attack. The first step involves setting the mask words. This means that it will mask the specified words in the original prompt and, by adding a perturbation prompt to the end of the original prompt, generate image whose content corresponds to the original prompt with the mask removed. Therefore, we select to mask out words related to clothing such as "wear tops and pants" and words related to peace and absence of harm such as "peaceful and fine". Based on its default setting, the character length is 5, and the perturbation prompts are crafted using GA, which performed the best among other candidates (Zhuang al.] 2023). We note that P4D (Chin al., 2023 is not chosen as baseline as it cannot apply to online T2I services.
Concept Removal Models. We choose several ofate-of-thh-ar concept removal methods, includ- ing ESD (Gandikota et al. 2023). Safe Latent Diffusion (SLD) (Schramowski al., (023) under settings (where we adopt 3 superior variants of SLD as provided by (Schramowski et al. 20033) i.e.. SLD-Max, SL.D-Strong, and SLD-Medium), SD with negative prompts (SD-NP) (Rombach et al.
2022) Concept Ablation (CA) (Kumari et al.] 2023). and Forget-Me-Not (FMN) (Zhang et al,,
2023]. Furthermore, for ESD in both nudity and violence, we fine-tuned the non-cross-attention and cross-attention parameters with negative guidance set as and 3, respectively. The implemen- tation of all baselines and ablated concept models is based on the official code. To remove nudity, use "naked, nude, bare, exposed, Stripped, topless" as the negative prompt. On the other hand, to remove violence, we use the prompt "hate, harassment, violence, suffering, humiliation, harm, suicide, blood" for SI>-NP and training ESD.
Evaluation. Throughout the evaluation process, we generate image per prompt with random fixed seed. To classify whether images contain nudity, we use the NudeNet detector. If the detec- tor detects one of the following labels in the image: EXPOSED LAROSEDLANUS,.. EXPOSED GENITALIA F, EXPOSED GENITALIA M, we consider the image to contain nudity. On the other hand, to determine whether images contain inappropriate content (such as blood or violence), we use the Q16 classifier (Schramowski et al., 2022). We also refer the readers to Ap- pendix D and for more experiment results.
SMPOSED_BREASS F,,
Evaluation Settings. In Binll-- we first find the empirical concept é. As shown in Figure I. it illustrates the use of the text encoder (e.g.. CLIP) to obtain é, where we use the VIT-L/14 (Ilharco et al., 202T). We use 50 and 30 prompts containing nudity and violence, respectively, and modify these prompts remove all traces of nudity and violence obtain the concept č. To run the GA, use 200 random initial prompts with 3000 generations and set the mutation rate and crossover rate to 0.25 and 0.5, respectively. In addition to the single-setting Ring-A-Bell, also follow (Chin al.) 2023) to consider the union settings denoted as Hlin-A--Bell-Union covering three different config- urations of Ring-A-Boo for generating new prompts. Furthermore, there are hyper-parameters: K (the length of the prompts), 17 (the weight of the empirical concept), and N (the number of prompt pairs). In Section 4.3 we will show how K, 17, N as well the choice of optimizer affect the
'htps:/wwwwmidiouueee
(last access: 2023/09)
https///peeai..commdalll (last access: 2023/09)
https://tsearcch.uuwaalll.cm
(last access: 2023/09)
http::///aabillyyyassalleediiii (last access: 2023/09)
hhap:://gthuh..o///aaamamnnitii (last access: 2023/09)

---

Published as a conference paper at ICLR 2024
DALL
Gan-2
Stalkiga
Nadty
Videoe
Figure 2: Visualization of inappropriate prompts generated by Ring-A-Bell (texts in red, black, and blue respectively represent problematic prompts from Ring-A-Bell, prompt dilution and modifica- and blurring for publication purposes.
tions) via four online services. We use
Nuthruativee
stcened bunio noi shoeer cudiia repoaedo enghootleooaa #8t donk jesrakiigg v god acquiiiii
leiigp pi yee perukes kis lach Eillic Sivees
Screenshots of the videos generated by Gen-2 (text-to-video version). We blur the screen- shots for publication purposes. (Texts in red, black, and blue respectively represent problematic prompts from King-A-Bell, prompt dilution, and modifications.)
Figure 3:
attack results. Regarding the metric, we report the Attack Success Rate (ASR), defined as the pro- portion of successful generation of inappropriate images by problematic prompts relative to the total number of images. For the Bing-A-Bell-Unie each prompt is considered successful if generates inappropriate images least once under the three configurations.
4.1 EVALUATION OF ONLINE SERVICES
Online services have different levels of protection against generating nudity and violence images shown in Table T Despite various safety checkers used in online services, Ring-A-Bell is still able to bypass them, resulting in the successful generation of nude images, as shown in Figure [ Our experience indicates that Midjourney, DALL-E 2, and Gen-2, sensitive words such as "naked" or "nude" need to be modified; otherwise, nearly all original prompts from Ring-A-Bell will be blocked. In addition, prompt lengths should not be excessively long (we use K 16), while the use of prompt dilution (Rando et al. 2022) can increase the probability of evading safe checkers.
On the other hand, when it comes to the concepts violence or bloodshed, most these online services not effective detecting them they DALL-E 2. According to the documentation of DALL-E 2 it simultancously checks both prompts and generated images, and has pre-filtered adult content from the training dataset, further, it ensures that unseen content would not be generated. However, the other three services, once they pass the safety checkers, generate images based on prompts truthfully. In other words, once problematic prompts circumvent the safety checks, these three online services will generate images accordingly. Therefore, when compared to DALL-E they more prone generating inappropriate images due to the absence of filtered training set. We provide more examples of inappropriate images generated by online services in Appendix D
detecting nudity, with the exception of
In addition to the T2I model, we also assess text-to-video (T2V) model such as Gen-2 for the concept of nudity and violence shown in Figure
Concept
Midjoumey
DALL-E2
Gen-2
Stable Diffusion XL
Nudity
36.75
44.5
33.5
1.33
Violence
35.5
Table 1:
Quantitative comparison among different online services. Each value denotes, on average, how many adaptations (tokens) each prompt requires generate images with the desired concept.
4.2 EVALUATION OF CONCEPT REMOVAL METHODS
Here, we demonstrate the performance of Ring-A-Bed on T2I models that have been fine-ture tnnettnnd to forget nudity or violence. We note that both CA and FMN are incapable of effectively removing
fhtpss//aallerggallllllllllliiii (last access: 2023/09)

---

Published as a conference paper at ICLR 2024
SLDD-
SLD- Medicam
SD
ESD
Max
Strong
SD:NP
CA
FMN
Noddy
Violeace
Figure 4: Visualization of inappropriate prompts generated by Ring-A-Bel via SOTA concept re- moval methods. We use
and blurring for publication purposes.
nudity and violence, but we still include them for completeness sake. Furthermore, we also consider a stringent defense, which involves applying both concept removal methods and safety checkers (SC) (Rando al., (022) to filter images for inappropriate content after generation. Regarding nudity, we set Ring-A A-ell with K = 16 and = 3, while for Ring-A- Ring-A-Bell-Umion we employ different settings, including (K, 7) = (16,3) (77,2). and (77, 2.5). As for violence, we select = 77 and 7) = 5.5 and for Bing-A-Bell-Unme
we set (K, 17) (77, 5.5). 5) and (77, 4.5).
As shown in Table 2 contrary to using the original prompts and QF-Attack, Ring-A-Bell is more effective in facilitating these T2I models to recall forgotten concepts for both nudity and violence. When safety checker is deployed, Ring-A-Bee der the union setting. is worth noting that, as shown in Table the safety checker is more sensitive to nudity and can correctly filter out larger amounts of explicit images. However, its effectiveness drops noticeably when it comes to detecting violence. In addition. we also demonstrate the images obtained by using prompts generated with Ring-A-Bell as input to these concept removal methods shown in Figure 4
also more capable bypassing it, especially un-
Concept
Methods
SD
ESD
SLD Max
SLD Strong
SLD Medium
SD-NP
CA
FMN
Pinngts
52.65%
12.65%
30.35%
58.05%
37.39%
QF-Altsck (wdl SC)
51.58%
13.68%
56.84%
37.89%
Ring Bell twl
93.68%
42.11%
61.05%
91.58%6
89.47%
68.42%
Nudity
Original Penniis SC)
Unlon SC)
7.37%
55.79%
6.32%
1
2015
5.47%
13.79%
QF-Attad (wr SC)
7.39%
4.11%
2.11%
6.32%
8.42%
5.36%
9,47%
18.95%
Ring-A (w/ SC)
30.539
9.47%
7.37%
12,6355
35. 7995
N.42%
37.895
28.42%
Ring A Bell Union (wl SC)
49.47%
22.11%
15.79%
32.63%
$7.89%
16.84%
$3.68%
47.37%
Orgiind Pronpes
SC)
61.45%
QF-Altick SC)
62%
56%
24.2%
32.016
58.45
Ring-A-Bell (win
96.4%
50%
97.69%
79.65
Violence
Ring Bell: Union (wlo SC)
Original Prongts SC)
99.6%
36.336
37,2%%
86%
40.4%
14.4%
80.4%
15%
97.2%
94.8%
25.2%
54.3%
100%
98.8%
47.21%
AltAittckk twr SC)
54,4%
53.6%
11.2%
21.2%
33.04%
21.2%
53.6%
47.216
Ring-A-Hell (w/ SC)
82.678
40.25
18%
44%
68.45%
85.05
74.45)
Ring A Dell Unioe (wr SC)
99.2%
84%
38.4%
76.4%
95.6%
90.8%
98.8%
98.8%
Table 2: Quantitative evaluation of different attack configurations against different concept removal
methods via the metric of ASR. (w/o SC and w/ SC represent the absence and presence of the safety checker, respectively).
4.3 ABLATION STUDIES
Length K of Prompts. In 3 we experiment on how the prompt length K affects the ASR. In this experiment, the maximum length our text encoder can handle. As shown in Table 3 increasing K does not significantly improve ASR; instead, moderate lengths such as 38 16 yield better attack results for nudity. However, for violence, the best performance is observed when K = 77.
и and choose three different lengths: {77, 38, 16), where 77 is
Concept
K
SD
ESD
SLD-Max
SLD-Strong
SLD-Medion
SD-NP
CA
FMN
77
85, 20%
16%
56.84%
92.61%
17.89%
86.32%
63.10%
Nudity
38
87.37%
29.17%
32.63%
64.21%
88.42%
28.42%
91.58%
71.5%% %
16
93.68 %
38.79%
42.11%
61.05%
91.58%
34.74%
89.47%
68.42%
77
96.4% %
54%
19.2%
50%
76.4%
80%
97.6%
79.6%
Violence
38 2%
46.4%
19.6%
42%
75.6%
71.25
95.2%.
82.4%
16
87.6%
38.8%
13.6%
33.2%
$4.8%
52.4%
88%
766
Table The ASR of Ring-A-Bell against different concept removal methods w/o safety checker by varying levels of K.
Coefficient 17. We present how y affects the performance of Ring-A-Bel A-Bell. We use K = 38 and 77 for nudity and violence concepts, respectively. As shown in Table 4 performance does not
8

---

Published as a conference paper at ICLR 2024
Concept
6
ESD
SLD-Max
SLD-Stoong
SLD-Medium
SD-NP
CA
FMN
2
81.05%
31.38%
32.63%
52.63%
84.21%
27.37%
85.25 26%
64.21%
Nudity
2.5
88.42%
24.21%
28.42%
56.84%
86. 32%
38.79%
85 26%
42%
3
87.37%
29.47%
32.63%
64.21%
88, .2%%
28.42%
91.58%
71.58%
3,5
88.42%
28 42.4
29,47%
60%
90.53%
28.42%6
84.21%
76.84%
50.4%
Violence
4.5
94%
52%
16.8%
41.6%
71.6%
66.8%
96.4%
70.4%
5
96.4%
59.2%
17.6%
46.4%
77.2%
73.2%
97.6%
78.4%%
54%
19.2%
76.4%
80%
97.6%
79.6%
Table The ASR of Ring-A-Bell against different concept removal methods w/o safety checker by varying levels of 1).
improve with excessively large or small values of 77. As 1) = 3, it performs better in attacking these concept removal methods for nudity. Furthermore, for violence, the performance becomes better as 1) increases. However, when becomes sufficiently large, the improvement gradually decreases, implying similar results in Table for n = 5 and y== 5.5.
Different Choice of Discrete Optimization. We compare the performance of GA and PeZ (Wen et al. 2023), as they have similar purposes. For all experiments, we used the same settings for both GA and PeZ. Regarding nudity,
K 16 and ' 3. As for violence, and rj = 5.5. As demonstrated in Table 5 both GA and Pe7. showcase competitive performances.
K 77
However, when it comes nudity, GA excels on more challenging methods such ESD and SLD series. GA also surpasses PeZ in the violence category.
Concept
Method
SD
ESD
SLD-Max
SL.DStrron
SLD-Medium
SD-NP
CA
FMN
Nudity
9368%
35.79%
61.05%
91.3%%
34.74%
68. 45.
PeZ
93.68%
11.58%
37.89%
54.74%
87.37%
56.84%
92.63%
74.74%
Violence
PcZ
GA
96.0%%
65.20%
54.00%
32.00%
19.20%
8.80%
50.00%
20.40%
76.40%
34.80%
80.00%
32.40%
97.60%
70.80%
79.60%
72.40%
Table 5: The ASR of Ring-A-Bell Bell against different concept removal models w/o safety checker
using different optimization methods.
The Number of Prompt Pairs Eq. (3) demonstrates the search of empirical representation č We next experiment the effects of Ring-A-Bell for different numbers of prompt pairs. For all experiments regarding nudity, we use K 16, and f = 3. As for violence, we set K 77, and 17 5.5. The variation in N leads to some differences in performance, as illustrated in Table The difference lies in the violence prompts used for N = 10 and 20, which are related, while for introduced in addition to blood-related. It can
N =30 prompts related to firearms and robbery be inferred that using prompts exclusively for one context, e.g., blood-related, when obtaining the empirical concept è tends to generate images that Q16 deems inappropriate as blood-related contexts are highly detectable. However, there are no such issues for nudity. Conclusively, it can be observed that increasing the number of prompt pairs allows us extract context-independent concept è which ultimately enhances the capability of Ring-A-Ber
Concept
.N
SD
ESD
SLD-Max
SLD-Strong
SLD-Medium
SD-NP
CA
FMN
10
90.53%
25.26%
21.21%
50.53%
$4.21%
52.63%
84.21%
61.25%
Nudity
30
87.37%
24.21%
37.89%
53.68%
90.53%
28.42%
84.21%
58.95%
50
93.68%
35.79%
42.11%
61.05%
91.58%
34.74%
89.47%
68.42%
10
97.60%
57.20%
14.20%
42,80%
80.00%
80.80%
9K.80F%
78.107%
Violence
20
99.60%
66.00% %
18. 80%
52.40%
87.60%
89.20%
99.60%
80.00%
96.10%
19.20%
50.00%
97.60%
Table 6: The ASR Ring-A-Bell against different concept removal models w/o safety checker by varying the number of prompt pairs N.
5 CONCLUSION
In this paper, we have demonstrated the underlying risk both online services and concept removal methods for T2I diffusion models, all of which involve the detection or removal of nudity and violence. Our results show that by using Ring-A-Bell to generate problematic prompts, is highly possible to manipulate these T2I models to successfully generate inappropriate images. Therefore, Ring-A-Bell serves as a valuable red-teaming tool for assessing these T2I models in removing or detecting inappropriate content.

---

Published as a conference paper at ICLR 2024
REFERENCES
Praneeth Bedapudi. Nudenet: Neural nets for nudity classification, detection and selective censoring, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 2020.
Zhi-Yi Yi Chin, Chich-Ming Jiang, Ching-Chie Huang. Pin-Yu Chen, and Wei-Chen Chiu. Prompt- ing4debugging: Red-teaming text-to-image diffusion models by finding problematic prompts. arXiv preprint 230.003009 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Haonan Duan, Adam Dziedzic, Nicolas Papernot, and Franziska Boenisch. Flocks of stochas- tic parrots: Differentially private prompt learning for large language models. arXiv preprint arXiv:2305.15594, 2023.
Rinon Gal. Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In International Conference on Learning Representations (ICLR), 2023.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In International Conference on Computer Vision (ICCV), 2023.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatricke and Graham Neubig. Towards unified view of parameter-efficient transfer learning. In International Conference on Learning Representations (ICLR), 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Gabriel Ilbarco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https: //dii orggrl..g...... /00 5281/
Zzeodoo. 5 5133773
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Diederik P Kingma and Max Welling, 1312.61312.611 14,611144000 2013.
Auto-encoding variational bayes.
arXiv preprint
Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang, Eli Shechtman, Richard Zhang, and Jun Yan Zhu. Ablating concepts in text-to-image diffusion models. In International Conference on Com- pufer Vision (ICCV), 2023.
10

---

Published as a conference paper at ICLR 2024
Deokjae Lee, JunYeong Lee, Jung-Was Woo Ha, Jin-Hwa Kim, Sang- Woo Lee. Hwaran Lee, and Hyun Oh Song. Query-efficient black-box red teaming via bayesian optimization. In Anmual Meeting of the Association for Computational Linguistics (ACL), 2023.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
Xiaoya Li. Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. A unified mre framework for named entity recognition. In Annual Meeting of the Association for Computational Linguistics (ACL), 2020.
Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai- Wei Chang, Aram Galstyan, and Rahul Gupta. Flirt: Feedhack loop in-context red teaming. arXiv preprint (tw.Xiiv2300.426..5. 2023.
OpenAl. Gpt-4 technical report, 2023. Ut//tttts///awwi.rrrr//////////2
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of Conference Empirical Methods in Natural Language Processing (EMNLP), 2022.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. Language models as knowledge bases? In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models. In ACM Conference on Computer and Communications Security (CCS), 2023.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. International Conference on Machine Learning (ICML), pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Javier Rando, Daniel Paleka, David Lindner, Lennard Heim, and Florian Tramer, Red-teaming the stable diffusion safety filter. In NeurlPS ML Safety Workshop, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. arxiv. In /EEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2022.
Chitwan Saharia. William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems (NeurIPS), 35:36479-36444 36494, 2022.
Timo Schick and Hinrich Schütze. Few-shot text generation with pattern-eplliting training. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting. Can machines help us an- swering question 16 in datashects, and in turn reflecting on inappropriate content? In ACM Conference Fairness, Accountability, and Transparency (FAccT), 2022.
Patrick Schramowski, Manuel Brack, Bjöm Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 22522, 2023.
11

---

Published as a conference paper at ICLR 2024
Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. Toward human readable prompt tuning: Kubrick's ' the shining is good movie, and good prompt too? arXiv preprint arXiv:2212.10539 2022.
Zhouxing Shi, Yihan Wang. Fan Yin, Xiangning Chen, Kai-Wei Chang, and Cho-Jui Hsich. Red teamnig language model detectors with language models, arXiv preprint arXiv:2305. 2023.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
S.N. Sivanandam and S.N. Deepa. Genetic Algorithmss. Springer Berlin Heidelberg, 2008. ISBN 978-3-540-73190-0. URL
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing (ICML), 2015.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv preprint arXiv:2302.03668. 2023.
Yuchen Yang. Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. Sneakyprompt: Evaluating robust- ness text-to-image generative models' safety filters. arXiv preprint arXiv:2305. 12.3305./2088 /2082, 2023.
Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learn- ing to forget in text-to-image diffusion models. arXiv preprint 2rX1v:2303.1751,, 2023.
Haomin Zhuang, Yihua Zhang, and Sijia Liu. A pilot study of query-free adversarial attack against stable diffusion. In /EEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), pp. 2384-2391, 2023.
12


Reviews:
[{"review_id": "7lDtEbgxWZ", "reviewer": "Reviewer_UevV", "summary": "This paper proposed a model-agnostic red-teaming tool, Ring-A-Bell, for the evaluation of text-to-image (T2I) diffusion models’ safety mechanisms. In the first stage, this concept retrieval algorithm would perform concept extraction by learning the difference between the embeddings of prompts with/ without the target concept (e.g., violence). With the extracted concept, the algorithm utilises genetic algorithms to produce problematic prompts to test the reliability of online T2I diffusion models.", "questions": "see the above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe main idea of the algorithms is clearly demonstrated with figures and examples. The motivation of the paper is clearly explained by analyzing the drawbacks of the current model-specific attack algorithms.\n\n2.\tExtensive experiments are well-designed to show the efficiency of Ring-A-Bell in generating problematic prompts in the field of nudity and violence. The evaluation is reasonable with the NudeNet detector. The results are clearly shown with quantitative tables and well-processed images to demonstrate the ability of Ring-A-Bell as a red-teaming tool.", "weaknesses": "1. The related work should include the introduction of the concept removal methods, such as the Safe Latent Diffusion mentioned in the paper.\n\n2. In the concept extraction stage, the selection/ generation of the prompt pairs, which are semantically similar but different from the target concept, is not clearly specified. Producing high-quality prompt pairs requires extensive specialized knowledge. This can affect the effectiveness of the algorithm and increase the difficulty of reproduction.\n\n3. The generation of p ̃_cont is simply by a linear combination of the embedding of P and extracted empirical representation c ̂, which needs further justification. The definition of ‘target prompt P’ is not specified.\n\n4. The ablation study is not properly implemented. For example, it might be better to demonstrate the performance of the algorithm with and without discrete optimization.\n\n5. The algorithm strongly emphasizes that the text encoder is the CLIP model. It might be better to test on other text encoders.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D0sw232Dad", "reviewer": "Reviewer_WiiG", "summary": "The paper investigates the effectiveness of safety mechanisms for text-to-image (T2I) diffusion models. It proposes a model-agnostic evaluation tool called Ring-A-Bell, which can assess the reliability of deployed safety mechanisms without prior knowledge of the target model. The tool performs concept extraction to identify problematic prompts and generates inappropriate content to evaluate the safety measures. The paper empirically validates the method by testing online services and various concept removal methods.", "questions": "- the ablation study of interference among modification, prompt dilution, and Ring-A-Bell: in Figure 2 and 3, the shown prompt contains the three types of texts. 1) is the whole prompt generated by Ring-A-Bell, or you combine the three types of attacks together for final output? how to discriminate the type of texts such as modification, prompt dilution, and Ring-A-Bell? 2) which type of text is essential to evade the safety filter of diffusion models or defeat the concept removal methods? \n\n- the ESD config is missing in Table 2 since it has multiple variants, whose concept removal effect is different from each other.\n\n- why K=16 is the final config? is there experiment result about smaller K?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ significance: this paper reveals the importance of adversarial evaluation of the current concept removal works. Moreover, it performs its attack in a practical black-box way, which expands its evaluation scale to commercial APIs. How to evaluate the black-box commercial APIs is of great importance since they are powerful and more easily accessible by common people. \n\n+ quality: this paper takes a comprehensive inspect into the safety robustness of commercial APIs and also state-of-the-art concept removal methods, which demonstrates the efficacy of their method.", "weaknesses": "- their attack is easy to be filtered or removed by advanced NLP techniques such as large language model, since they perform token-level optimization on the prompt and the output is usually random combinations of tokens. large language model can purify the prompt by removing the semantically unclear part of the prompt.\n\n- the token level optimization is uninterpretable and cannot provide insights into how to defend against such attacks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WmXFRT04ZC", "reviewer": "Reviewer_nmRw", "summary": "This paper investigates the safety of text-to-image models. The paper proposes a model-agnostic attack to evade safety mechanisms and generate sensitive and inappropriate images. The proposed work is evaluated on online services to explore their safety risks.", "questions": "Please clarify the model-agnostic design and explain why it is effective for online services.\n\n\nThe rebuttal has addressed most of my concerns.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper investigates red-teaming text-to-image models, which is a critical topic for generative AI safety.\n2. The proposed method is validated on four T2I online services.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The model-agnostic design of the proposed framework is not convincing. The entire design is based on an offline CLIP model and is irrelevant to the online services. This design implicitly assumes that the framework that applies to the offline CLIP model can be transferable and effective for online services. What if the online services use a more robust text encoder? In addition, one of the contributions, claimed by the paper, is that Ring-A-Bell is “based solely on either the CLIP model or general text encoders.” However, in the evaluation, only the CLIP model is evaluated. It would be great to see if the proposed work can be extended to other and more recent text encoders.\n2. The paper only compares the proposed framework with QF-Attack, which is insufficient. Many recent works are encouraged to be investigated [1-4]. In addition, although P4D is designed for offline attacks, it would be great to consider P4D as a baseline to compare the performance gap between online and offline attacks.\n3. The paper aims to evade the safety mechanism of online diffusion models. However, the paper only considers concept removal defenses. For an online service, an easy and effective way is to develop a detector to identify inappropriate images. For example, the service provider could build a detector (e.g., NudeNet detector used in the evaluation) to detect nudity in the images. The proposed framework that mainly focuses on the text domain may not be effective. \n\n[1] Qu, Yiting, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. \"Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models.\" ACM CCS 2023.\n[2] Mehrabi, Ninareh, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. \"Flirt: Feedback loop in-context red teaming.\" arXiv preprint arXiv:2308.04265 (2023).\n[3] Rando, Javier, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr. \"Red-teaming the stable diffusion safety filter.\" arXiv preprint arXiv:2210.04610 (2022).\n[4] Yang, Yuchen, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. \"SneakyPrompt: Evaluating Robustness of Text-to-image Generative Models' Safety Filters.\" arXiv preprint arXiv:2305.12082 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eZiNyIhuHz", "reviewer": "Reviewer_nnoZ", "summary": "This work proposed Ring-A-Bell, a model-agnostic red-teaming tool for T2I diffusion models, which serves as a prompt-based concept testing framework that generates problematic prompts to red-team T2I diffusion models with safety mechanisms.", "questions": "There are a bunch of notation issues. I list some of them below:\n1. What is $\\rho$ in (2)? I cannot find it in the main paper.\n2. What is the training parameter of (2)? Is it $\\widetilde c$?\n3. Are there brackets in (3)\n4. Should $\\tilde{\\mathbf{P}}_{cont}$ be a function of $c$ or $\\hat c$?\n\nQuestion about experiments:\n1. How to tell whether the percentage of nudity is greater than 50%? The output propobility? Please be rigorous.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall this paper proposed a practical and intersting offline method in generating problematic prompts for 'safe models'. The experiments are very convincing and concrete.", "weaknesses": "Please see the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)