Stable Diffusion is Unstable
Authors: Chengbin Du
Yanxi Li
Zhongwei Qiu
Chang Xu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Adversarial Attack
Generative Model
Diffusion Model
Latent Diffusion Model
Conditional Latent Diffusion Model

Abstract: Recently, text-to-image models have been thriving. Despite their powerful generative capacity, our research has uncovered a lack of robustness in this generation process. Specifically, the introduction of small perturbations to the text prompts can result in the blending of primary subjects with other categories or their complete disappearance in the generated images. In this paper, we propose **Auto-attack on Text-to-image Models (ATM)**, a gradient-based approach, to effectively and efficiently generate such perturbations. By learning a Gumbel Softmax distribution, we can make the discrete process of word replacement or extension continuous, thus ensuring the differentiability of the perturbation generation. Once the distribution is learned, ATM can sample multiple attack samples simultaneously. These attack samples can prevent the generative model from generating the desired subjects without tampering with the category keywords in the prompt. ATM has achieved a 91.1\% success rate in short-text attacks and an 81.2\% success rate in long-text attacks. Further empirical analysis revealed three attack patterns based on: 1) variability in generation speed, 2) similarity of coarse-grained characteristics, and 3) polysemy of words. The code is available at https://github.com/duchengbin8/Stable_Diffusion_is_Unstable

Stable Diffusion is Unstable
Chengbin Du, Yanxi Li, Zhongwei Qiu, Chang Xu School of Computer Science, Faculty of Engineering University of Sydney, Australia chdu5632@uni. http:ss2eeuis.ndneeeeeeaeeee sydney. eoddey.edu...e au, yali0722@uni syr.//auiiisdddey.......... sydney.dddye..... edu. au zhongwei.. qiu@sydney edu. au, c. xxxusyddne edu. au
Abstract
Recently, text-to-image models have been thriving. Despite their powerful gen- erative capacity, our research has uncovered lack of robustness in this gener- ation process. Specifically, the introduction of small perturbations to the text prompts can result in the blending of primary subjects with other categories or their complete disappearance in the generated images. In this paper, we propose Auto-attack on Text-to-image Models (ATM), a gradient- based approach, to effectively and efficiently generate such perturbations. By learning Gumbel Softmax distribution, we can make the discrete process of word replacement or extension continuous, thus ensuring the differentiability of the perturbation genera- tion. Once the distribution is learned, ATM can sample multiple attack samples simultancously. These attack samples can prevent the generative model from generating the desired subjects without tampering with the category keywords in the prompt. ATM has achieved 91.1% 1.% success rate in short-text attacks and an 31.2% success rate in long-text attacks. Further empirical analysis revealed three attack patterns based 1) variability in generation speed, 2) similarity of coarse-grained characteristics, and 3) polysemy of words. The code is available at https://gitai whw.g.tt.bb.... cww dwwhengginn//ttablllee SeclleDDfffusee Dwffusionnnfiiii Unstable
Introduction
In recent years, the field of rext-to-image generation has witnessed remarkable advancements, paving the way for groundbreaking applications in computer vision and creative arts. Notably, many significant developments have captured the attention of researchers and enthusiasts, such as Stable Diffusion [28, 32], DALL-E2 [26, 27, 23] and Midjourney [19]. These developments push text-to- image synthesis boundaries, fostering artistic expression and driving computer vision research.
Despite the remarkable progress in text-to-image models, it is important acknowledge their current limitations. One significant challenge lies in the instability and inconsistency of the generated outputs. In some cases, it can take multiple attempts to obtain the desired image that accurately represents the given textual input. An additional obstacle revealed by recent researches [33, 2, 8] that the quality of generated images can be influenced by specific characteristics inherent text prompts. Tang et al. [33] proposes DAAM, which performs a text- text-image attribution analysis on conditional text-to-image model and produces lixel-level attribution maps. Their research focuses on the phenomenon of feature entanglement and uncovers that the presence of cohyponyms may degrade quality of generated images and that descriptive adjectives can attend too broadly across image. Attend-and-Exen Excite [2] investigates the presence of catastrophic neglect in the Stable diffusion model, where the generative model fails to include one or more of the subjects specified in the input prompt. Additionally, they discover instances where the model fails accurately associate attributes such as colors with their respective subjects. Although those works have some progress, there is still work to be done to enhance the stability and reliability of text-to-image models, ensuring consistent and satisfactory results for a wide range of text prompts.
One prominent constraint observed in those works related to the stability of text-to-image models lies in their dependence on manually crafted prompts for the purpose of vulnerability identification. This approach presents several challenges. Firstly, it becomes difficult to quantify the success and
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

GM 3r c))
vocabulary
Diffusion Model
Vision
Logits
Margin
Sofmax
Classifier
Loss
X tokees
Gumbel
Softmax
e
Optimating
Frozon
GM
Forward
Constraint
Baokward
Figure The overall pipeline our attack method. The selection words is relaxed to be differentiable by using Gumbel Softmax with temperature When 0, the Gumbel Softmax outputs exhibit proximity to one-hot vectors while retaining differentiability. After an image is generated, CLIP [25] classifier and margin loss are employed to optimize w aiming to generate images that cannot be correctly classified by CLIP.
failure cases accurately, as the evaluation largely depends on subjective judgments and qualitative assessments. Additionally, the manual design of prompts can only uncover limited number of potential failure cases, leaving many unexplored scenarios. Without substantial number of cases, becomes challenging to identify the underlying reasons for failures and effectively address them. To overcome these limitations, there a growing demand for a learnable method that can automatically identify failure cases, enabling a more comprehensive and data-driven approach to improve text- to-image models. By leveraging such an approach, researchers can gain valuable insights into the shortcomings of current methods and develop more robust and reliable systems for generating images from textual descriptions.
In this paper, we propose Auto-attack on Text-to-image Models (ATM), to automatically and efficiently generate attack prompts with high similarity to given clean prompts (Fig. 1). We use Stable Diffusion [28, 32] as our target model. With the open-source implementation and model parameters, can generate attack prompts with a white-box attack strategy. Remarkably, those attack prompts can transfer to other generative models, enabling black-box attacks. Two methods to modify a text prompt are considered, including replacing an existing word extending with new ones. By incorporating Gumbel Softmax distribution into the word embedding, the discrete modifications can transformed into continuous ones, thereby ensuring differentiability. the similarity between clean and the attack prompts, binary mask that selectively preserves the noun representing the desired object is applied. Moreover, two constraints are imposed: a fluency constraint that ensures the attack prompt is fluent and easy read, and similarity constraint that regulates the extent semantic changes.
ensure
After the distribution is Iearned, ATM can sample multiple attack prompts at once. The attack prompts can prevent the diffusion model from generating desired subjects without modifying the nouns of desired subjects and maintain high degree of similarity with the original prompt. We have achieved 91.1%% success rate short-text attacks and a 81.2% success rate in long-text attacks. Moreover, drawing upon extensive experiments and empirical analyses employing ATM, we are able to disclose the existence of three distinct attack patterns, each which corresponds to a vulnerability in the generative model: 1) the variability in generation speed; 2) the similarity of coarse carsengr characteristics; 3) the polysemy of words. In the following, we will commence with an analysis of the discovered attack patterns in Section 4, followed by detailed exposition of our attack method in Section 5.
In addition, ATM can automatically and efficiently generate plenty of successful attack prompts, which serves as a valuable tool for investigating vulnerabilities text-to-image generation pipelines. This method enables the identification of a wider range of attack patterns, facilitating a comprehensive examination of the underlying causes. It will inspire the research community and garner increased attention toward exploring the vulnerabilities present in contemporary text-to-image models and will foster further research concerning both attack and defensive mechanisms, ultimately leading to enhanced security within the industry.

---

2 Related Work
2.1 Diffusion Model.
Recently, the diffusion probabilistic model [30] and its variants [12, 21, 31, 28, 29, 4, 15] have achieved great success in content generation [31, 13, 291, including image generation [12, 31], conditional image generation [28], video generation [13, 361, 3D scenes synthesis [18] and on. Specifically, DDPM [12] adds noises to images and learns to recover images from noises step by step. Then, DDIM [31] improves the generation speed of the diffusion model by skipping steps inference. Then, the conditional latent diffusion model [28] formulates the image generation in latent space guided by multiple conditions, such texts, images, and semantic maps, further improving the inference speed and boarding the application of the diffusion model. Stable diffusion [28], a latent text-to-image diffusion model capable of generating photo-realistic images given any text input, and its enhanced versions [37, 14, 20J. have been widely used in current Al-generated content products, such as Stability-AI [32], Midjourney [19]. DALL-E2 [23], and Runaway [7]. However, these methods and products cannot always generate satisfactory results from the given prompt. Therefore, in this work, we aim to analyze the robustmess of stable diffusion in the generation process.
2.2 Vulnerabilities in Text-to-image Models.
With the open-source of Stable Diffusion [28], text-to-image generation achieves great process and shows the unparalleled ability on generating diverse and creative images with the guidance of a text prompt. However, there are some vulnerabilities have been discovered in existing works [8, 2, 33]. Typically, StructureDiffusion [8] discovers that some attributes in the prompt are not assigned correctly in the generated images, thus they employ consistency trees or scene graphs to enhance the embedding learning of the prompt. In addition, Attend-and -xcciteeee [2] also introduces that the Stable Diffusion model fails to generate one or more of the subjects from the input prompt and fails to correctly bind attributes to their corresponding subjects. These pieces of evidence demonstrate the vulnerabilities of the current Stable Diffusion model. However, to the best of our knowledge, no work has systematically analyzed the vulnerabilities of the Stable Diffusion model, which is the goal of this work.
3 Preliminary
The architecture of the Stable Diffusion [28] comprises of an encoder & : X + Z and decoder D Z - X. where ā (xx)). Additionally, conditional denoising network eg and a condition encoder Tg are employed. In the text-to-image task, the condition encoder is text encoder that maps text prompts into latent space. The text prompt typically a sequence of word tokens c= (c....., CK J. where K is the sequence length. During the image generation process, a random latent representation žr is draw from a distribution such as a Gaussian distribution. Then, the reverse diffusion process is used to gradually recover noise-free latent representation z. Specifically, conditional denoising network (2,, t, Tg()) is trained to gradually denoise 2, each time step t= TT, .,, 1 to gradually reduce the noise level of 2t, where the condition is mapped in to a latent space using To (e) maps and the cross-attention between the condition and features is incorporated in eg to introduce the condition. Finally, the denoised latent representation z is decoded by decoder D to produce the final output ह. The aim of our study is to introduce slight perturbations to the text prompts, thereby inducing the intended object to be blended with other objects entirely omitted from the generated images. For the sake of simplification, in the forthcoming sections, we shall represent the image generation process using the GM as ã GM(zrlc).
Vulnerabilities of Stable Diffusion Model
By applying the attack method proposed in this paper, a variety of attack text prompts can be generated and analyzed. In this section, the identified attack patterns are discussed. Details of the attack method are introduced in Section 5. We have discovered three distinct patterns of impairment in Stable Diffusion model: 1) Variability in Generation Speed, where the model struggles to reconcile the differences in generation speed among various categories effectively. 2) Similarity of Coarse- grained Characteristics, which arises from the feature entanglement of global or partial coarse-grained characteristics, which possess a high degree of entanglement. 3) Polysemy of Words, which involves the addition of semantically complementary words to the original prompt, resulting in the creation of images that contain brand-new content and are not related to the original category.
3

---

1.0
A large mountain, natural elovation of the Earth's surface, churaderized steep stopes and pask."
Scores
0.8
"A photo mountain""
0.6
0.4
Concatonation of the embeddingg of gromge photo mountain." And the promut "A photo of pealow!"
0.2
40
20 Steps
"A large mountain, natural elevation of the Earth's surface, ofie choractrizod storp dopes pesfowl."
"A phuto poofows
Figure 3: The generation speeds of "mountain" and "peafowl".
Figure 2: case study on "mountain" and "peafowl".
1.0
0.9
0.8-
0.77
0.6
0.5
0.4
0.3
0.2
0.1
0.0
Steps
Figure 4: A violin plot illustrating the generation speeds of 1, 000 images of various classes. The horizontal axis represents the number of steps taken, ranging from 49 to while the vertical axis displays the SSIM scores. The width of each violin represents the number of samples that attained specific range SSIM scores at given step.
4.1 Variability Generation Speed
Observation 1.
When given text prompt contains a noun A representing the object to be generated, the introduction of another noun B into the prompt, either through addition replacement, leads to the disappearance of noun A from the generated image, replaced instead by noun B.
In Observation we identify phenomenon when replacing or adding a noun in the description in text prompt, the new noun will lead to the completely disappear of the desired subject. As shown in Fig. 2, if the noun "peak" replaced by "peafowl". the desired subject "mountain" disappears and the new subject "peafowl" is generated. To further investigate this phenomenon, we use two short prompts C₁ and C2 containing "mountain" and "peafowl", respectively. to exclude the influence of other words in the long prompts. To eliminate the additional impact of all possible extraneous factors, such as contextual relationships, they are embedded separately and then concatenated together: Concat(Ta (C2)). The result shows that almost no element of the mountain visible in the generated image (Fig. 2).
Further analysis reveals a nontrivial difference in the generation speeds of the two subjects. To define the generation speed, a metric to measure the distance from a generated image ä, at a given time step & T 1,.... ...... 0 to the output image Eo is desired (note, ØT is the initial noise). We use the structural similarity (SSIM) [34] as the distance metrics: s(t) := SSIM (W, ão). Therefore, the generation speed can be formally defined as the derivative of the SSIM regarding the time step: utt) := ds(t)/dt dt zs (s(t)- - s(t + 1))/At, At, where At 1. Thereby, we propose our Pattern 1.
Pattern (Variability in Generation Speed). Comparing the generation speeds (v) and U2) of two subjects ( (1 and S2), it can be observed that the outline of the object in the generated image will be taken by S1 if v1 > V2. And it can be inferred that S2 will not be visible in the resulting image.
further generates 1, 000 images of various classes with the same initial noise and visualize their generation speed in Fig. 4 as violin plot. The SSIM distance from the generated images at each

---

(a) photo silver salmon and a feather.
(b) photo tench and a buck.
(c) photo of a zebra and giraffe.
(d) photo howler monkey and a snake.
Figure 5: Images generated by the template "A photo of A and B".
49
42
35
28
feather"
"a magician"
allver
silver magician
Figure 6: The first row illustrates the generation process with the prompt "a photo of silver salmon". The second row, based on the forty-second step the first row, shows the generation process with the prompt "a photo of feather". The third row, also building upon the forty-second step of the first row, presents the generation procedure when the prompt is "a photo of magician", The fourth row depicts the generation process in the presence of feature entanglement. The fifth row demonstrates the generation process for two distinct categories without feature entanglement.
step to the final image is calculated. The horizontal axis represents 49 ~ O steps, while the vertical axis represents the SSIM scores. Each violin represents the distribution of the SSIM scores of the 1, 000 images in a step, with the width corresponds to the frequency of images reaches the score. In the early stages of generation, the median of the distribution is positioned closer the minimum value, indicating that majority of classes exhibit slow generation speeds. However, the presence of high maximum value suggests the existence of classes that generate relatively quickly. In the middle stages of generation, the median of the distribution gradually increases, positioning itself between the maximum and minimum values. In the later stages of generation, the median of the distribution is positioned closer to the maximum value, indicating that majority of classes are nearing completion. However, the persistence of low minimum value suggests the presence of classes that still exhibit slow generation speeds. This analysis highlights the variation in generation speeds across different classes throughout the entire generation process. This phenomenon can be interpreted as characteristic of the generation process, where different classes exhibit varying speeds throughout the stages. It possible that certain classes have inherent complexities or dependencies that cause them to generate more slowly. Conversely, other classes may have simpler structures or fewer dependencies, leading to faster generation.
4.2 Similarity of Coarse-grained Characteristics
Observation When a text prompt contains noun representing the object to generated, the introduction of another noun B, which describes object with similar coarse-grained characteristies to the object represented by noun A, into the prompt, either through addition or replacement, results in the generation of an image that contains object combining elements of both nouns A and B.
5

---

(b)
(a) tsne
(b) logits
(d)
(e)
(f)
Figure 8: a) t-SNE Visualization of 100 images each of "bat", "baseball bat", "bat and ball" and text "a photo of bat." b) The boxplot of cosine similarities between the text embedding of "a photo of bat" and 100 of image embeddings each of "bat", "baseball bat", and "bat and ball".
Figure 7: a) "A photo of a bat"; b) "A photo of bat and ball;" c) Heat map of the word "bat" in generated image; d) "A photo of a warthog", : e) "A photo of a warthog and traitor"; f) Heat map of the word "warthog" in generated image.
The second case that we observed in our attacks is when two nouns in the text prompt share similar coarse-gran coarse-grained characteristics, the generated image will contain subject that is a combination of these two nouns. As illustrated in Fig. 6, when given the text silver salmon and feather," the GMs generate an image of feather with the outline of salmon. This happens because these two nouns (i.e., salmon and feather) exhibit a certain degree of similarity in their coarse-grained attributes. In contrast, there is no feature entanglement between "salmon" and "magician" because their coarse-grained features are vastly different from each other.
To verify this assumption, we first obtain the generated latent variable for the prompt "a photo of silver salmon" " at the early sampling step (e.g., 8 steps). Using this latent variable, we replace the prompt with "a photo of feather" and continue generating images. The results confirm that feathers can continue be generated based on the coarse-grasd sourse-grained properties of silver salmon, and the final generated graph has high similarity with the generated graph of the prompt "a photo of silver salmon and feather". However, replacing "silver salmon" with "magician" " does not seem to generate any object similar to "magician" This observation indicates that there is no coarse-grained feature entanglement between these two subjects. We summarize this observation Pattern
Pattern 2 (Similarity of Coarse- grained Characteristics). Let X²A and X denote the latent variables generated by the GM for word tokens A and B, respectively. Suppose t is small and let d represent the metric that measures the outline similarity between two images. If the text prompt contains both A and B, and d(X XB) falls below the threshold then feature entanglement occurs in the generated image.
Based on the observed Pattern 2, The types of feature entanglement be further divided into, direct entanglement and indirect entanglement. As shown in Fig. 5, direct entanglement represents the direct entanglement triggered by two categories of coarse-grained attributes that have global local similarities. Indirect entanglement is shown in Fig. 5d, where the additional attribute trunk brought by the howler monkey has high similarity with the goarse-grained attribute of the snake, thus triggering the entanglement phenomenon.
4.3 Polysemy of Words
When text prompt contains a noun A representing the object to be generated, if the semantic scope of noun A encompasses multiple distinct objects, the generated image contains one of the objects described by noun A. If there exists another word B that, when combined with noun A. directs its semantics to a particular object, the introduction of word B into the prompt, either through addition or replacement, leads to the generation of an image that specifically contains that particular object.
Observation 3.
The third scenario we observed in the attack is that the content of the generated image is not directly related to either the desired image or the added word. However, this is again different from a category of disappearance, where the desired target has clear individual in the image. As illustrated in Figs. 7a, 7b and 7c, when the cleaning prompt "a photo of a bat" was modified to photo of a bat and a ball", the bat disappeared completely from the generated image, but the DDAM[33] heat map showed that the word "bat" is highly associated with the stick-like object the newly generated image.

---

Pattern 3 (Polysemy of Words), When interpreting polysemous words, language models must rely on contextual cues to distinguish meanings. However, in some cases, the available contextual information may be insusficient confuse the model by modifying specific words, resulting in moddel-generated image that deviates from the actual intention of the user.
To further investigate the phenomenon of word polysemy in the Stable diffusion model, we used the prompts "a photo of a bat", "a photo of baseball bat" and "a photo of a bat and ball" to generate 100 images each using the stable diffusion model, and transformed these images into embedding form by CLIP image encoder, and transformed "a photo of a bat" into embedding form by CLIP text encoder, then visualized these 301 embeddings by t-SNE. As illustrated in Fig. 8a, Considering the entire set of bat images, bats (the animal) are closer to the text "a photo of bat" than baseball bats are, as depicted in the t-SNE visualization. However, the distribution of the two categories also has relatively close proximity, indicating their underlying similarities. The category of "bat and ball" shows a more expansive distribution, almost enveloping the other two. This suggests that by modifying the original text from "a photo of a bat" to "a photo of bat and ball", the distribution of the clean text can be pulled towards another meaning in the polysemous nature of the word "bat". From the perspective of text-to-image model, this kind of modification can stimulate the polysemous property of the word, thereby achieving an attack effect.
In addition to this explicit polysemy, our algorithm further demonstrates its aptitude for detecting subtler instances of polysemous words. As depicted in Figure 7, the transformative capacity of our algorithm is evident when an image of warthog (Fig. 7d) transfigures into an image of military chariot (Fig. 7c) with the incorporation of the term "traitor".
5 Auto-attack on Text-to-image Model
We aim to design automatic attack method that targets the recent popular sext-to-image models. The objective of our method is to identify attack prompts c' based on a clean prompt c, which leads to vision model h X + y failing to predict the desired class y, i.. argmax, h(#); # y:
e' argmax Ely, (G,,((G(ee))) d'Eaa(e,E)
(1)
where GM(-) is text-to-image model, B(c, = (c' d(cc c') S {), .(: d(;))) is distance measure regularizing the similarity between e and c', and & is a maximum distance. To enable auto-attack, a differenniablee method that can be optimized using gradient descent is desired. We introduce a Gumbel-Softmax sampler to enable differentiable modifications on text prompts during the word embedding phase. To minimize the distance d(c, c'), we introduce two constraints, including a fluency constraint and similarity constraint.
In our experimental setup, the open-source Stable Diffusion model is employed as the targeted generative model GM(.). By generating white-box attack prompts for Stable Diffusion, we can subsequently transfer these prompts other generative models execute black-box attacks. To facilitate the classification task, we utilize a CLIP classifier as the vision model h(-), benefiting from its exceptional zero-shot classification accuracy. To establish the desired classes, we employ the 1,000 classes derived from M.. In the case of generating short prompts, fixed template of "A photo of [CLASS [NMME"" utilized generate the prompts. Conversely, for the generation of long prompts, we employ ChatGPT 4 [24] as a prompt generation model. Subsequently, human experts verify the correctness of the prompts and check that the prompts indeed contain the noun associated with the desired class.
5.1 Differentiable Text Prompt Modification
A text prompt is typically sequence of words e = (cI...., CK). J. Owing to the discrete nature of text prompts c, perturbations can be incorporated either by replacing an existing word e where 1 < k < K or augmenting with new ones {cx++|1 < i < K'}. However, the nin-iifferentiable nature of this procedure makes it unsuitable for optimization utilizing gradient- based techniques. Therefore, there is need for a mechanism that guarantees the differentiability of the word selection process. In this regard, we integrate a Gumbel Softmax sampler v)(:;)) into the word embedding phase. The Gumbel Softmax function has the ability to approximate one-hot distribution as the temperature T + O. Additionally, the Gumbel distribution has the ability to introduce further randomness, thereby enhancing the exploitability during the initial stages of perturbation search.
7

---

Differentiable Sampling. We employ a trainable matrix w E RKXV to learn the word selection distribution, where K is the length of the text prompt and V is the vocabulary size. the scenario of augmenting with new word, the sequence length K can be extended to K + K' to facilitate the addition of new words. The Gumbel Softmax can represented as follows:
exp((wa, + gk.))/T) GumbelSoftmxx(wee7) = E, exp(( +9,))/T)
where 9k.i Gumbel(O, 1) are i..d. samples from Gumbel distribution. The word embedding stage employs matrix E RVXD, where V is the vocabulary size and D is for the embedding dimensionality. The discrete process of word embedding is to select E; from E based on the index 1 S i V of a word in the vocabulary. To make this process differentiable, the dot product between GumbelSoftmax(un and can be calculate:
de ((wii) GumbelSoftmax(oat)) E RJ E,, s.t. argmax Wke
where c is the new word selected according to wi- As T - O, Eq. 3 can effectively emulate an argmax selection procedure.
Additionally, in order to ensure similarity, it is desired to preserve the noun representing the desired object in the new prompt. This is achieved by utilizing a binary mask M E {0, 1 1 where the position corresponding to the desired noun is set to while other positions are set to 1. By computing c' < (1 - M) c + M e', the desired noun can be retained in the prompt while other words can be modified.
Attack Objective. To generate images E that cannot be correctly classified by the classifier, a margin loss [1] can be used as the loss function .(.,)) in Eq. 1:
Emargin (0 = max 0, (,,(()), max h()); + +xx) ify
where in is a margin. Eq. reduces the classifier's confidence on the true class y and improve its confidence on the class with the largest confidence, excluding until margin x is reached.
5.2 Constraints on Fluency and Similarity
Given that we search for perturbations in RxV space to attack the text prompt, the attack prompts may be too diverse if the added perturbations are not properly constrained, making it easily detectable. Eq. includes a distance constraint such that d(c, c') S E, which ensures that the added perturbations are subtle and hard to notice. The measurement of distance between two pieces of text can be approached through various methods. We introduce two constraints to reduce this distance, namely fluency constraint and a semantic similarity constraint. The fluency constraint ensures that the generated sentence is smooth and readable, while the semantic similarity semantic constraint regularize the semantic changes introduced by the perturbations, making the attack prompt e' closely resemble the clean prompt e.
Fluency constraint. The fluency constraint can be achieved visa a Casual Language Model (CLM) with log-probability outputs [9]. The next token distribution we learn is compared with the next token distribution predicted by é. Given a sequence of perturbed text c', we use á to predict the token eę based on (c/.... d/1)). Therefore, we can have a log-likelihood of the possible next word:
log Pa
- b(c.......... .(...
(5)
The next token distribution we learn can be easily obtained by GumbelSoftmax(w 7). Subsequently, a cross-entropy loss function can be employed to optimize the learned distribution:
K D
CEe(w) = - GumbelSoftmax(wirr)) (atpa/offmmmoooo..oooooooot ((((W1))))...... ((Wi)i)))
(6)
Eq. serves as regularizer to encourage the next word selection distribution to resemble the prediction of the CLM ó, thereby ensuring fluency.
8

---

Algorithm 1 Auto-attack on Text-to-image Models (ATM)
Input: The maximum number of iterations T. The maximum number of attack candidates N. The clean prompt c. The desired class y. A binary mask M. A learning rate 7)-
Output: A set of attack prompts S.
1: Initialize
2: fort=1-2 do
The search stage
3: Sample an attack prompt c' {ib(ws 1 KK)
4: Apply the mask by c' < (1 - M d'd
5: Generate an image x' GM(zrle)
6: Get classification results s/ h((²)
7: Conduct a gradient descent step w t w 11 Vwl(w)
8: end for
9: Initialize
Ø
10: for = 1
N do
The attack stage
11:
Sample an attack prompt c' {u((w:)) 1 KK}
12:
Apply the mask by c' t (1 M) ct M .'
13: Generate an image n' = GM(zzlc)
14: if aargmax h(x') # y then
If attack success
15:
Save the success attack prompt S + Suic end if
17:
end for
Semantic similarity constraint. Rather than simply considering word similarity, we concern more about semantic similarity. One prominent metric used to evaluate semantic similarity the BERTScore [39]. The calculation of BERTScore requires contextualized word embeddings. The aforementioned CLMM é is used again to extract the embeddings gi(emb) (c), where ((emb) denotes the embedding network used ø. The BERTScore between the clean prompt and the attack prompt c' can be calcualted by
SBERT (c,
Σ
Wi
u. V.''', D'j, 1
where w; idf(c:)) ΣN idf(c,) is the normalized inverse document frequency (idf), N K, and M is either K or K + K' depending on whether existing words are being replaced or new words are being added. To improve the similarity, we use 1 SBERT (c, e') as the our loss term.
The constrained objective. Considering that the addition of constraints may limit the diversity of perturbation search, we introduce two hyper-parameters, À and y, control the strength of the constraints. Then, the overall objective function can be written as:
max w
(0. [Cmargin (GM(zrle'),
(8)
s.t. min A- A-C - (1 - -SBERT (c, c')).
(9)
5.3 Generation of Attack Prompts
The overall procedure of ATM as described in Algorithm 1. It consists of two stages: search stage, where the Gumbel Softmax distribution is learned, and an attack stage, where we generate attack prompts using the learned distribution. In the search stage, we use gradient descent to optimize the parameters w for each clean prompt over a period of T iterations. Once w is learned, we proceed to the attack stage. this stage, we sample N attack prompts from each learned w. An attack prompt e' is considered successful if the image a' generated from it cannot be correctly classified by the visual classifier
5.4 Attack Experiment Results
To corroborate the effectiveness of our proposed algorithm, we implemented a structured evaluation scheme based on the model template: "A photo of "ELASS_NAMEE". An extensive dataset compris- ing one thousand abbreviated textual descriptions, corresponding to all classes within ImageNet [3], was meticulously curated. Additionally, comprehensive long-form textual descriptions for these

---

(a)
(b)
(c)
(d)
(c)
(f)
(g)
(h)
Figure 9: (a) are generated by Midjourney, and the corresponding prompts are, "a photo of a zebra and a giraffe"; "a photo of a tree frog and a cyclist"; "a photo of a volleyball and a Nemo"; and "a photo of warthog and traitor", respectively. (g). are generated by DALL-E2, and the corresponding prompts are "a photo of a tench and lizard"; "a photo of a volleyball and a Nemo"; "A photo of a howler monkey and a snake"; and "a photo of a silver salmon and a feather", respectively.
same classes were generated utilizing the capabilities of ChatGPT4 [24]. Upon rigorous testing, our algorithm demonstrated noteworthy performance, recording attack success rates of 91.1% and 81.2% on the short-text and long-text datasets, respectively. Moreover, the calculated cosine similarity between the embeddings of the original and attack prompts yielded values of 0.7227 and 0.8364, respectively. Due to the length of this paper, detailed experiment results, including comparisons to baselines and ablation studies, are reported in the supplementary material.
To further investigate whether our generated attack prompts can be transferred to different text- to-image models, we randomly select several attack prompts to attack DALL- and Midjourney, respectively. The experimental results prove that our attack prompts can also be used for black-box attacks. More results of black-box attacks are reported in the supplementary material.
Conclusion
The realm of text-to-image generation has observed remarkable evolution recent years, while concurrently exposing several vulnerabilities that require further exploration. Despite the many advancements, there are key limitations, specifically concerning the stability and reliability of generative models, which remain to be addressed. This paper has introduced Auto-attack on Text- to-image Models (ATM), a novel approach that generates plethora of successful attack prompts. providing an efficient tool for probing vulnerabilities in text-to-image models. ATM not only identifies a broader array of attack patterns but also facilitates a comprehensive examination of the root causes. We believe that our proposed method will inspire the research community to shift their focus toward the vulnerabilities of present-day text-to-image models, stimulating further exploration of both attack and defensive strategies. This process will be pivotal in advancing the security mechanisms within the industry and contributing the development of more robust and reliable systems for generating images from textual descriptions.
7 Acknowledgments
This work was supported in part by the Australian Research Council under Projects DP210101859 and FT230100549.
References
[1] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium security and privacy (sp). pages 39-57. leee, 2017.
[2] H. Chefer, Y. Alaluf, Y. Vinker, I.. Wolf, and D. Cohen-Or. Attend-and-excite: Attention- hassd semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.1382. 2023.
[3] J. Deng, W. Dong. Socher, L.JJ. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[4] A.DD. Dinh, D. Liu, and C. Xu. PixcrAsParam: A gradient view on diffusion sampling with guidance. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors. Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 8120-8137. PMLR, 23-29 Jul 2023. URL httptt:ttppooeeeeeeeeeeoooo http///ppooeeeeeeeee/ee//eec mlr, wta..ppess/v/i v202/ dinh23a. h.ml..
[5] A.D.. Dinh, D. Liu, and C. Xu. Rethinking conditional diffusion sampling with progressive guidance. In sevent- Conference on Neural Information Processing Systems, 2023. URL https:/ //nuuris...... co/virtuse 2attuua//2023/pssee////6/
10

---

[6] A. Dosovitskiy, L. Beyer, A. Kolesnikov. D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale, arXiv preprint arXiv:2010.
2020.
[7] P. Esser, J. Chiu, P. Atighehchian, J. Granskog, and A. Germanidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011 2023.
[8] W. Feng, X. He, T.J. Fu, V. Jampani, A. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Y. Wang- Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05052. 2022.
[9] C. Guo, A. Sablayrolles, H. Jégou, and D. Kiela. Gradient-basce adversarial attacks against text transform- ers, arXiv preprint urXiv:2104, 13733, 2021.
[10] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 770-778, 2016.
[11] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[12] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[13] Ho. T. Salimans, A. Gritsenko, W. Chan, M. Norouzi. and D. J. Fleet. Video diffusion models. arXiv preprint arXf::2004033555 2022.
[14] L. Huang, D. Chen, Y. Liu, Y. Shen, D. Zhao, and J. Zhou. Composer: Creative and controllable image synthesis with composable conditions, arXiv preprint 29X7722302.09778 2023.
[15] T. Huang, Y. Zhang, M. Zheng, S. You, F. Wang, C. Qian, and C. Xu. Knowledge diffuusio for distillation. arXiv preprint arXiv:2305. 200.2335551.22... 2023.
[16] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[17] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, 2022.
[18] G. Metzer, E. Richardson, O. Patashnik, R. Giryes, and D. Cohen-Or. Latent-nerf for shape-guides generation of 3d shapes and textures. arXiv preprint arXiv:2211.076000 2022.
[19] Midjourney. Midjourney. https:/ //www midjjurneyyy...m com/ 2023. Accessed on 2023-05-17.
[20] C. Mou, X. Wang. L. Xie, J. Zhang. Z. Qi. Y. Shan, and X. Qie. 12i-adapter: Learning adapters dig more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.068: 084553 2023.
[21] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 10741, 2021.
[22] A. Obukhov, M. Seitzer, P.W. Wu, S. Zhydenko, J. Kyl. and E. Y.-. Lin. High-fidelity performance metrics for generative models in pytorch, 2020. htLitppss///habb.cc////aaaa//rrr fidelit.. Version: 0.3.0, DOI: 10.5281/zenoda 1222112222o0.4957733
[23] OpenAI. Dall-e 2. https: http://///pnnaa.cooooooocctttt/e
2023. Accessed on 2023-05-17.
[24] OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.007744 2023.
[25] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748 8763. PMLR, 2021.
[26] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pages 8821-8831, PMLR. 2021.
[27] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0611 06125, 2022.

---

[28] R. Rombach, A. Blattmann, D. Lorenz, P. Esser. and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[29] L. Ruan, Y. Ma, H. Yang, H. He. B. Liu, J. Fu, J. Yuan, Q. Jin, and B. Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. arXiv preprint arXiv:2212.0975. 2022.
[30] Dick-liicsttin, Weiss, Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256 2265. PMLR, 2015.
[31] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[32] Stability-AA
http::///stabilityaa/bbbbo stable- www.ddddfffinuuuiioeeeee wwt--ppppllllleeeeeeeee 2023. Accessed on 2023-05-17.
Stable diffuusion public release.
[33] R. Tang, A. Pandey, Z. Jiang, G. Yang, K. Kumar, J. Lin, and E. Ture. What the daam: Interpreting stable diffusion using cross attention. arXiv preprint arXiv:2210.04885, 2022.
[34] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Iransactions image processing, 13(4):600-612, 2004.
[35] X. Yang, D. Zhou, S. Liu, J. Ye, and X. Wang. Deep model reassembly. In Advances in Neural Information Processing Systems, 2022.
[36] S. Yin, C. Wu, H. Yang, J. Wang, X. Wang, M. Ni, Z. Yang, Li, S. Liu, E. Yang. et al. Nuwa-xl: Diffuusion over diffusson for extremely long video generation. arXiv preprint arXiv:2303. /2346, 2023.
[37] I.. Zhang and M. Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[38] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 386-595, 2018.
[39] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger. and Y. Artzi Bertscore: Evaluating text generation with bert. In International Conference Learning Representations, 2020.
12

---

Supplementary Material
In this supplementary material, we first delve into additional analyses concerning the vulnerabilities observed in the Stable Diffusion model (Section A). Subsequently, offer instances of long and short prompt attacks, accompanied by the corresponding generated images, as well as instances of black-box attacks (Section B). Lastly, we undertake comprehensive series of experiments to substantiate the effectiveness of our approach (Section C). These experiments include the evaluation of attacks targeting both long and short prompts. Additionally, ablation studies explore attacks employing different search steps, assess the influence of our fluency and semantic similarity constraints text similarity, and target diverse samplers (e.g., DDIM and DPM-Solver) in the attack process.
conducted to
A
Vulnerabilities of Stable Diffusion Model
A.1 Pattern 1: Variability in Generation Speed
Fig. 10 demonstrates the entire 50-step violin diagram which has been discussed before. To eliminate possible bias due to single metric, we further verified the difference in generation speed of one thousand images based on the LPIPS [38] metric, as shown in Fig. 11, The calculation of the LPIPS distance from the images generated at each stage to the ultimate image is performed. The horizontal axis signifies the range of steps from 49 down to O, whereas the vertical axis denotes the respective LPIPS scores. Each violin plot illustrates the distribution of the LPIPS scores associated with 1 1,000 images at a specific step. The width of the violin plot is proportional to the frequency at which images achieve certain score. During the initial stages of generation, the distribution's 's median is situated nearer to the maximum LPIPS value, suggesting a preponderance of classes demonstrates slower generation velocities. Nonetheless, the existence classes that generate at comparatively faster rates. As the generation transitions to the intermediate stages, the distribution's 's median progressively decreases, positioning itself between the maximum and minimum LPIPS values. In the concluding stages of generation, the distribution' 's median is found closer the minimum LPIPS value, implying that the majority of classes are nearing completion. However, the sustained high maximum value suggests that there are classes still exhibiting slower generation rates.
low minimum value indicates the presence of
A.2 Pattern 2: Similarity of Coarse-grainel Characteristics
To further verify that coarse-grained feature similarity is the root cause of feature entanglement, provide more cases in Fig. 12. From these cases, it is evident that for any two classes prone to feature entanglement, they can each proceed with image generation based on the other's 's coarse-grained information.
42-
Figure 10: A violin plot illustrating the generation speeds of 1, 000 images of various classes. The horizontal axis represents the number of steps taken, ranging from 49 to 0, while the vertical axis displays the SSIM scores. The width of each violin represents the number of samples that attained a specific range of SSIM scores at given step.
13

---

A.3 Pattern 3: Polysemy of Words
As shown in Fig. 13, when attack the prompt "A photo of warthog" to "A photo of warthog and traitor", the original animal warthog becomes an object similar to military vehicle or military aireraft, while the images generated by attack prompt is not directly related to the image of the animal warthog or traitor. From the t-SNE visualization (Fig. 14), we can see that the distance from the picture generated by the attack prompt the text "a photo animal warthog picture to the text, so we can see that by attacking the original category word that guided the original category word (animal warthog) into its alternative meaning.
warthog" has similar distance the
From the box plots (Fig. 15). it can be observed that the image of "warthog" exhibits the highest similarity with the prompt's embedding, while the image of "traitor" demonstrates the lowest similar- ity, as anticipated. Simultaneously. the similarity distribution between the images of "warthog" and "traitor" with the prompt text is relatively wide, indicating that some images have high similarity with "warthog," while others lack features associated with "warthog."
Figure 11: A violin plot illustrating the generation speeds of 1, 000 images of various classes. The horizontal axis represents the number of steps taken, ranging from 49 to 0, while the vertical axis displays the LPIPS scores. The width of each violin represents the number samples that attained specific range of LPIPS scores at given step.
A photo of and elephant
photo ell and rook
weasel from elephant
elephant from weasel
rook from
ell from
A photo of a rhino and armadillo
photo a sea lion and spaniel
thina from armadillo
armadillo from rhino
spaniel from sea lion
sea lion from spaniel
Figure 12: The image caption, "A photo of A and class represents the generated image when feature entanglement occurs; And "classA from classB" represents the final generated image of prompt "A photo of class.A" based on the eighth step of the prompt "A photo of class
14

---

Figure 13: The images in the first row are generated by the prompt "A photo warthog". The images in the second row are generated by the prompt "A photo of traitor". The images in the third row are generated by the prompt "A photo of a warthog and a traitor".
Text
30.0
traitor
27.5
warthog and trattor
25.0
22.5
20.0
17.5
15.0
-10
12.5
warthog
traitor warthos and traitor
The boxplot of cosine similarities between the text embedding of "a photo of warthog" and 100 of image embeddings each of "warthog", "traitor", and "warthog" and traitor".
Figure 15:
Figure 14: t-SNE Visualization of 100 images each of "warthog". "traitor", "warthog and traitor" and text "a photo of warthog."
(a)
(b)
(c)
Figure 16: a) "A type of footwear with thick, rigid sole, often made of wood, and upper made of leather or another material. Clogs can be open or closed, and are commonly associated with Dutch and Scandinavian cultures." b) "footwear" is replaced by "pistol". c) "Dutch" is replaced by "pistol".
(a) 0-1-2
(b) 0-2-1
(c) 1-0-2
(d) 1-2-0
(e) 2-0-1
(0) 2-1-0
Figure 17: template, "A photo of A. B and C", is used to generate prompts, where A, B, C {" cat", "pistol", "clogs"} }. For exmaple, "0-1-2" represents A "caa", B "pistol" and C " "logs", and so on.
15

---

A.4 Pattern 4: Positioning of Words
In addition to the three aforementioned observations and patterns outlined in the paper, there is fourth observations (Observation 4), which is related to positioning of words.
Observation 4. When a text prompt contains a noun A representing the object to be generated. there exists a preceding word B and succeeding word C around noun A. When replacing either word B or C with another noun D. for certain instances of noun A, replacing word B results the generation of an image containing noun D, while replacing word C still results in the generation of an image containing noun A. Conversely, for other instances of noun A, the opposite scenario occurs.
An example of Pattern is shown in Fig. 16. When "footwear" replaced by "pistol", the generated image contains a pistol instead of clogs, However, when "Ductch" is replaced by "pistol", the model still generates an image of clogs. addition to differences in the words being replaced, significant distinction between the two aforementioned examples of success and failure lies in the relative positioning of the word being replaced with respect to the target class word. We hypothesize that this phenomenon occurs due to the different order of the replaced words B or C' with respect to the noun A. To exclude the effects of complex contextual structures, a template for a short prompt, "A photo of A, B and C", is used, and the order of A, B, and C' are swapped (Fig. 17).
When these sentences with different sequences of category words are understood from human perspective, they all have basically the same semanties: both describe a picture containing cat, clogs, and pistol. However, the processing of language models (including CLIP), the order of words may affect their comprehension. Although positional encoding provides the model with the relative positions of words, the model may associate different orders with different semantics through learned patterns. Therefore, we propose our Pattern 4.
Pattern (Positioning of Words). Let V denote a set of vocabulary. Let N C V denote the subset of all nouns in the vocabulary. Consider a text prompt containing noun A E N representing the object to be generated. Furthermore, assume there exist preceding word E V and succeeding word C' E V surrounding noun A. There exists condition-dependen dependent behavior regarding the replacement of words B and C with another noun E N:
DA, 3B,CEV, P(B + D)
generate
D A P(C - D)
generate A;
BA, BB,CEV, P(B + D)
generate
A A P(C D)
generate
D.
B Cases of Short/Long-Promg Attacks and Black-box Attacks
B.1 Attack on Long Prompt
In Fig. 18, we demonstrate more cases of long text prompt attacks.
B.2 Attack on Short Prompt
In Fig. 19, we demonstrate more cases of long text prompt attacks.
B.3 Black-box Attack
In Fig. 20, and Fig. 21, we demonstrate black box attacks targeting Midjournyy and DALL-E2, respectively.
C Experiments
In our experiments, we conduct comprehensive analyses of both long and short prompts. Furthermore, we conduct ablation studies specifically on long prompts, focusing on three key aspects. Firstly, we evaluate our attack method with different numbers of search steps T. Secondly, we investigate the influence our constraints, including fluency and semantic similarity as measured by BERTScore. Lastly, we attack different samplers, including DDIM [31] and DPM-Solver [17].
C.1 Experimental Setting.
Attack hyperparamters. The number of search iterations T is set to 100. This value determines the number of iterations in the search stage, during which we aim find the most effective attack prompts. The number of attack candidates N is set to 100. This parameter specifies the number of
16

---

loose, luil-engih abaya worn by some Muslim women, primanty the Arablan Paninsuua, an outer covering ansure madesty.
enee, full adult abaya embroidery by some German women, marly the netharlands peninauta an outar covaring to ensure modosly.
An ancient abacus tool, cona ising of framm with raws beads other movoble for matherat cal operwtoon and counting
An optical abasus tool consiating of wheal with rows of ties and other movable symbols, used for whiel opanations and counting.
large, mounted cannon that fires neavy projecti es, histonically used wartare and for other canemonnall saluties.
A large, hand cornon that fires heavy projectles, historically used warfare and for other purposss such ceremonid selutes.
such
This emall brussels anition breed has
penguns appaarancs, wiy muscular body, good, tampenament, they
tppe foolwear with thick, niiid saie, often made wood, and upper
A Appe camouflage a long. figit beak, otten made wood, and
"A loose, sleevelless clcai that is warn over the shoulders and tastlaned the neck, often used for warnth decorative acoessory."
"A loose, car: cloaki that is warn over the shoulders and rastened the neck, othen uaved for wormihh ass decorativv accaisooy"
distinctive, coal,
uppar made
frlendly,
leather another materal. Glogs can be sommony associated with Dutch and Scandinaven cultures.
affectionalle
Clogs can be open
desed, are
open dosed, and are commonly associated with Dutch and Scandiravian cultures.
were orignally
originally hunding game
game
antimas Balgium
Belgium ligenca,,
inteligence, agllty
agllty, speed
amm
ccomputer
sirllar keyboard with mallower shase
input computor's typically leaturing buttons
device camputerss
sharks
instrumant similar
spinal strong,
but
oursor, typically
strong,
They
They
scroil wheel
ropical and are sholls and edible
tropical
selecting
selecting
their
known their
novigating on-soreon wlements.
navigating
large, designed g people, found communal spaces."
large upholstered weapons design geeegnedd sealing people, lving rooms
A rich, treamy, cairy- based equnoo tradiionally made of milks, cream, sugar, and aggs, ottan spieed with alcahol and served during the winter holidays.
A doll creamy, calry- based egonog tradiionnaly made of
round bosty,
body,
and
adapted swimming adiyped swimming and diileling for "hey come vorious solors and partems, with some species having bright, iidessent piumage."
milk, cream, supar, and eggs, often spiead with
cabblingg
communal spacis."
They come in vanious colors patterss, with some species heviig lrightl. indescent plumage."
aloshol and served during winter
hot
Electric ray flatlened, disc sha-ed fish with smooth skin and pair of kidney-shaped electric ongane on each side their bocy, used for hurning prey and self defense.
Electric rays flattened, diso shaped senses with smooth skin and par didhey-shapdd electric organs on each side of their body, used for hunding prey and setl- defe-eeee.e.
timeleeping hourglass consising of "wo glasa chambers cornected by namow passage, with sand flowing through the passage at constant rata to massure specific time intorval.
A mean leke hourglase ponsicts of two rontropanels separatad by central passaga, with sunlight flowing through the cave in constant rate enter specific time frame.
pig species
minosppcies
All
large, curved
facial features,
inckuding warts foows. Wirthigs
including warts
faces.
primanly grazers and live savarnas and gras graesbandss
primerly grazees
grasslands.
Figure 18: To the left of the arrow is the clean long text prompt (highlighted by green) and its corresponding image, to the right the arrow is the generated attack prompt (highlighted by red) and its corresponding image. (Section B.I Attack on Long Text Prompt)

---

photo of abacus
photo abaous and batsman
photo of an American black beor
photo American black bear and a pands
a photo an American lobster
photo an kmmerican looster and dachehund
photo
photoo and buzzard
photo
photo spider
photo spider
an
of
photo
photo of a sea ion and spaniel
photo of pea lon and spaniel
a photo of stingray
a photo of stngray and vulture
photo of weasel
photo of wansel and elephant
a photo of ox
photo of OK and knife
photo of pig
photo
pig and haka
a photo of rooster
photo of rouster and mition
Figure 19: To the left of the arrow is the clean short text prompt (highlighted by green) and its corresponding image, to the right the arrow is the generated attack prompt (highlighted by red) and its corresponding image. (Section B.2 Attack Short Text Prompt)
candidate attack prompts considered in the attack stage, allowing for a diverse range of potential attack prompts to be explored. The learning rate for the matrix is set to 0.3. The margin in the margin loss is set to 30.
Text prompts. Our experiments consider the 1, 000 classes from ImageNet- IK [3], which serves as the basis for generating images. To explore the impact of prompt length, we consider both short and long prompts. For clean short prompts, we employ a standardized template: "A photo of [CLASS_NAME]" Clean long prompts, on the other hand, are generated using ChatGPT [24], with prompt length restriction of 77 tokens to align with the upper limit of the CLIP [25] word embedder.
Evaluation metrics. To evaluate the effectiveness of our attack method, we generate attack prompts from the clean prompts. We focus on three key metrics: success rate, Fréchet inception distance [1]] (FID), Inception Score (IS), and text similarity (TS). Subsequently, 50, 000 images are generated using the attack prompts, ensuring representative sample of 50 images per class. The success rate is determined by dividing the number of successful attacks by the total of 1,000 classes. FID and are computed by comparing the generated images to the ImageNet- IK validation set with (orch-fidelity)[2] [22]. TS is calculated by embedding the attack prompts and clean prompts using the CLIP [25] word embedder, respectively. Subsequently, the cosine similarity between the embeddings computed to quantify the text similarity.
18

---

photo of barn spider and reven
photo of black swan a turben
photo of eel and rook flew
A photo of roester and
photo lion and spaniel
photo stingray and vulture
photo of and buzzard
photo of armadillo and a rhino
A photo of wolf spider and frog
wolf spider
and girafie
type camoullage with long, rigid beak, often made of wood, and an upper made of leather or another material. Clogs can open or closed, and commanly associated with Dutch Scandinavian cultures
species native to Africa, known for its large, curved tusks and distinctive facial fe ettures, includ ing warts their faces. Warthogs primarily grazers and live savannas grasslands.
A large, hand cannon that that heavy projectilss, historically used warfare and
cloth, circular yurt statue nomadic peoples in central Asia, has wooden frame and balance praying covering. include excel weg.
round body, flat bill, webbed feet, dapted for swimming and dabaling for food. They come in various colors and patterns, with species having bright, iridescent plumage
other purposes such
ceremonia lutes.
Figure 20: Black-box attack on Midjourney (Section B.3 Black-box Attack).
Table 1: Main results of short-prompt and long-prompt attacks.
Prompt
Method
Success (%)
FID (L)
IS (+)
TS (t)
Clean
18.51
101.33 1.80
1.00
Short
Random
79.2
29.21
66.71+1087
0.69
ATM (Ours)
91.1
30.09
65.98
0.72
Clean
17.95
103.59+
1.00
Long
Random
41.4
24.16
91.33++1.58
0.94
ATM (Ours)
81.2
29.65
66.09+1.88
0.84
19

---

photo spaniel
photo stingray vulture
photo spider and a frog
photo a brakes spear
photo an and whale
type camouflage with a long, rigid beak, often made wood, and an upper leather or another material. Clogs can be open closed, and are commonly associated with Dutch and Scandinavian cultures.
type cork- double air aircraft, also known chrigible, ized by its llongated shape, powered by engines, and steered by a rudder. hips typically consist o rigid plane covered with envelope filled with oxygen, such hellum hydrogen.
cloth, circular statue by nomadic peoples central has wooden frame and balance praying covering.
A mean like hourglass consists of two renfrepanels separated by central passage, with sunlight flowing through the cave constant rate enter specific time
large, artichoke crab with long, fleshy, leaf s, scales, often or steamed and eaten by removing the thin and wiping off the body surface with the teeth.
frame.
Figure 21: Black-box attack on DALL-E2 (Section B.3 Black-box Attack).
C.2
Main Results
Table reports our main results, including short-prompl and Long-prompt attacks. Compares to long text prompts, short text prompts comprise only a small number of tokens. This leads to a relatively fragile structure that is extremely vulnerable to slight disturbance. Therefore, random attacks can reach an impressive success rate of 79.2% targeting short prompts but a low success rate of 41.4% 11.4% targeting the long prompts. In the contrast, our algorithm demonstrates its true potential, reaching an impressive success rate 91.19 and 51.2% targeting short and long prompts, respectively.
As further evidence of the effectiveness of our algorithm, it's worth noting the text similarity (TS) metrics between the random attacks and our algorithm's 's outputs. For short-promph attack, the values stand at 0.69 and 0.72, respectively, illustrating that the semantic information of short texts, while easy to disrupt, can be manipulated by a well-designed algorithm with fluency and semantic similarity constraints. Our attacks preserve more similarity with the clean prompts. For the TS score of random attacks (0.94) higher compared to our attacks (0.84). One possible reason is that random attacks tend to make only minimal modifications as the length of the prompt increases. This limited modification can explain the significantly lower success rate random attacks on longer prompts.
attacks,
From the perspective of evaluating the quantitative metrics of generated images. For short and long texts, images generated from the clean text have the lowest FID (18.51 and 17.95) and the highest IS (101.33+1.80 and 103.59 1.68). As the attack success rate rises, FID shows an upward trend. Examining this situation from the perspective of FID, a metric that gauges the distance between the distribution of generated images and the original data As the attack becomes more successful, the image set generated by the attack prompt tends to deviate substantially from the distribution of the original data set. This divergence consequently escalates the FID score, indicating larger distance between the original and generated distributions. On the other hand, the attack objective of our algorithm, namely the margin loss, used reduce the classifier's confidence the true class y and improve its confidence in the class with the largest confidence excluding y. until a margin reached. Consequently, generating low-quality images is also an objective of our attack. These images contribute to the increase in the FID index and the decrease in the IS score.
20

---

Table Results of the ablation study on the number of steps attack prompt search.
#Steps
Success (%)
FID ())
IS (1)
TS (t)
50
68.7
34.00
93.94+11.84
0.97
100
81.2
29.65
6..09=1.83
0.84
150
67.2
45.23
98.51
0.82
Table 3: Results of the ablation study on the constraints
Fluency
BERTScore
Success (%)
FID (4)
IS (1)
TS (1)
x
91.3
39.14
47.21
0.37
/
x
81.7
29.37
64.93+11.57
0.79
x
89.8
39,93
46,94:10.99
0.51
81.2
29.65
66.09++1.83
0.84
C.3 Different Search Steps
Table 2 presents the results of using different numbers of steps T in the search stage. For the T = 50 step configuration, the success rate is 68.7%. The FID value is 34.00, with lower values suggesting better image quality. The IS is reported as 93.94=1.84, with higher values indicating diverse and high-quality images. The TS value is 0.97, representing high level of text similarity. Moving on to the 7 100 step configuration, the success rate increases to 81.2 31.2%, showing an improvement compared to the previous configuration. The FID value decreases 29.65, indicating better image quality. The IS is reported as 66.09 16.09-1..83 showing a slight decrease compared to the previous configuration. The TS value is 0.84, suggesting a slight decrease in text similarity. In the T 150 step configuration, the success rate decreases to 67. 67.2%, slightly lower than the initial configuration. The FID value increases to 45.23, suggesting decrease in image quality. The IS is reported as 58.51 79, indicating decrease in the diversity and quality of generated images. The TS value remains relatively stable at 0.82.
When using T = 50, the attack prompt fails to fit well and exhibits a higher text similarity with the clean prompt. Although the generated images at this stage still maintain good quality and closely resemble those generated by the clean prompt, the success rate of the attack is very low. On the other hand, when 7 150, overfitting occurs, resulting in decrease in text similarity and image quality due to the overfitted attack prompt. Consequently, the success rate of the attack also decreases. Overall, the configuration of T 100 proves to be appropriate.
C.4 The Impact of Constraints
Table 3 examines the impact of the fluency and semantic similarity (BERTScore) constraints. When no constraints are applied, the attack success rate is notably high at 91.3%. However, this absence of constraints results in lower text similarity (TS) score of 0.37, indicating a decreased resemblance to clean text and a decrease in image quality. By introducing fluency constraints alone, the attack success rate decreases to 81.7% but increases the text similarity to 0.79. Furthermore, incorporating semantic similarity constraints independently also leads to a slight reduction in success rate to 89.8%, but only marginally improves the text similarity 0.51. The introduction of constraints, particularly fluency constraints, leads to an increase in text similarity. The fluency constraint takes into account the preceding tokens of each token, enabling the integration of contextual information for better enhancement of text similarity. On the other hand, BERTScore considers a weighted sum, focusing more on the similarity between individual tokens without preserving the interrelation between context. In other words, the word order may undergo changes as result and leads to low text similarity. Certainly, this outcome was expected, as BERT itself prioritizes the semantic consistency between two prompts, while the order of context may not necessarily impact semantics. This further highlights the importance of employing both constraints simultaneously. When both constraints are utilized together, the text similarity is further enhanced to 0.84. Mcanwhile, the success rate of the
21

---

Table 4: Results of the ablation study on the samplers
Sampler
Success (%)
FID (4)
IS (1)
TS (1)
DDIM [31]
81.2
29.65
66.09 1.83
0.84
DPM-Solver [17]
76.5
27.23
81.31=2.09
0.88
Table 5: Classification Accuracy of Different Classifiers both clean short and long text prompts.
Classifiers
Short Text Prompts
Long Text Prompts
VIT-B/I6(CLIP)
82.0%
78.8%
ViT-B/32(CLIPP
80.8%
74. 1%
VIT-L/I4(CLIP))
84.0%
80.8%
ResNet50(CLIP))
77.2%
69.9%
ResNet50 [10]
10.7%
10.8%
Swin-B [T6]
79.8%
76.2%
ViT-B/16 [6]
82.0%
78.9%
attack (81.2%) is comparable that achieved when employing only the fluency constraint, while the text similarity surpasees that obtained through the independent usage of the two constraints,
C.5 Different Samplers
Table 4 illustrates the effectiveness of our attack method in successfully targeting both DDIM and the stronger DPM-Solver. For the DDIM sampler, our attack method achieves a success rate of 81.2%, indicating its ability to generate successful attack prompts. Similarly, our attack method demonstrates promising results when applied to the DPM-Solver sampler. With success rate of 76.5%, it effectively generates attack prompts. The TS scores of 0.84 and 0.88, respectively, indicate reasonable level text similarity between the attack prompts and clean prompts. These outcomes demonstrate the transferability of our attack method, showcasing its effectiveness against both DDIM and the more potent DPM-Solver sampler.
C.6 Ensemble of Multiple Classifiers
The amalgamation of multiple models, often termed as "ensemble or reassembly," can significantly enhance model performance [35]. The efficacy of the attack is potentially contingent upon the performance of the classifiers. Notably, even when the stable diffusion model generated the desired subjects with precision, there remains possibility that the classifier may erroneously classify it. To eliminate the impact of the performance of single classifier, we design voting mechanism by using an ensemble of multiple classifiers. To be specific, the attack is defined as a failure if at least one classifier in the ensemble still recognizes the original categories in the text prompt. When we took this mechanism to evaluate the generated image after our attack, the success rate of the attack on the long and short prompts remained the same, 81.8% and 91. 1% respectively. This further proves the effectiveness of our algorithm. Additionally, we test the voting mechanism with images generated from clean prompts covering the 1000 classes in ImageNet. As shown in the Table most of the classifiers have relatively high Top1 accuracy. We conducted experiments with three distinct classifiers: CLIP+ VIT-B/II Swin-B, B, and the vanilla VIT-B/16. The classification accuracies of these models model, the classification accuracies with long and short text prompts were found to be 78.80% and 82%, respectively. The Swin-B B model demonstrated classification accuracies of 76.20% with long text prompts and 79.80% with short ones. Lastly, the vanilla VIT-B/16 model achieved classification accuracies of 78.90% and 82% with long and short text prompts, respectively. The probability that least one of them successfully recognizes the current generated image is 90.40% and 92.30% for long and short text prompts, respectively, which much higher than using only one classifier. This proves that our mechanism is effective.
evaluated using both long and short text prompts. For the CLIP+VIT-B/I
22


Reviews:
[{"review_id": "AnHllDi6d6", "reviewer": "Reviewer_kKNr", "summary": "In this paper, the authors use Gumble softmax as well as the gradient based method to learn the distribution of an attack text prompt, defined as an attack text prompt that enables a text-to-image generation model to generate images that do not match the text description without changing the category keywords in the clean text prompt, for exploring the vulnerability of stable diffusion model. Based on the failure cases generated by the algorithm, the authors place the possible reasons for the failure of the generation into four main categories.", "questions": "See weakness", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This algorithm able to automatically finds the prompt that causes the text to image generation model to fail, offering the possibility to explore the vulnerability of the stable diffusion model systematically. \n\n2. Various patterns of generative failure have been identified based on a large number of generative failure cases, and a number of reasonable experiments have been designed to further support these conjectures.\n\n3. This algorithm able to learn multiple attack text prompts based on a clean prompt, further boosting the number of generated failure cases.", "weaknesses": "1. It seems that in pattern 1 only the speed of generation of different categories under the same random seed was explored, would the speed of generation of the same category with different random seeds also be different? \n\n2. In the section on quantitative experiments, what is the classification accuracy of CLIP for clean text prompts?\n\n3. How do you handle the gradient of 50 steps in the stable diffusion model?\n\n4. Most contents in this paper are clear and detailed, but the setting in the quantitative experiments section lacks some details, e.g. it does not explain how the random algorithm changed the text prompt, which Casual Language Model was used, and which version of Clip was used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewers for their diligent review of this paper and for providing constructive feedback.\n\n**RE Generation Speed with Different Random Seeds:** We prefer to focus on the relative generation speed difference of different categories under the same initial noise since the different categories in the same prompt share the same initial noise. As shown in the pdf, we selected \"tench\" as an example, and used 100 different initial noises to generate 100 images of tench, and the results show that even for the same category, there is a significant difference in the generation speed under different random seeds. Although this phenomenon exists, it does not affect our previous conclusion that there is a difference in generation speed between different classes with the same initial noise\n\n**RE Classification Accuracy:** In the process of learning attack prompt distribution and the sampling of Gumble-Softmax, we have used the pre-trained ViT-B/16 Clip as the classifier, which has a classification error of 18% in the short text, and when perturbed using a randomized algorithm, the classification error rate rises from 18% to 79.2%, and our proposed algorithm improves it from 18% to 91.1%. In the long text prompt, Clip's probability of classification error is 21.2%, and with the randomized algorithm, the classification error rate is only improved to 41.4%, while our algorithm improves it to 81.2%, and based on these results, the effectiveness of our algorithm is proved. \n\nAdditionally, we also add an experiment with an ensemble of various classifiers, which provides an improved classification performance. To be specific, the attack is defined as a failure if at least one of them still recognizes the original categories of this text prompt. This approach may eliminate the impact of the performance of a single classifier. When we took this mechanism to evaluate the generated image after our attack, the success rate of the attack on the long and short prompts remained the same, 81.8% and 91.1% respectively. thus further proving the effectiveness of our algorithm. To this end, we test the voting mechanism with images from the  1000 classes in ImageNet generated by the stable diffusion model. As shown in the table below, most of the classifiers have a relatively high Top1 accuracy.  We conducted experiments with three distinct classifiers: CLIP+ViT-B/16, Swin-B, and the vanilla ViT-B/16. The classification accuracies of these models were evaluated using both long and short text prompts. For the CLIP+ViT-B/16 model, the classification accuracies with long and short text prompts were found to be 78.80% and 82%, respectively. The Swin-B model demonstrated classification accuracies of 76.20% with long text prompts and 79.80% with short ones. Lastly, the vanilla ViT-B/16 model achieved classification accuracies of 78.90% and 82% with long and short text prompts, respectively. The probability that at least one of them successfully recognizes the current generated image is 90.40% and 92.30% for long and short text prompts, respectively, which is much higher than using only one classifier. This proves that our mechanism is effective.\n\n\n|  | Short Text Promot | Long Text Prompot |\n|-----|:-----:|:-----:|\n| ViT-B/16(CLIP) | 82.0% | 78.8% |\n| ViT-B/32(CLIP) | 80.8% | 74.1% | \n| ViT-L/14(CLIP) | 84.0% | 80.8% |\n|ResNet50(CLIP)| 77.2% | 69.9% |\n| ResNet50 | 10.7% | 10.8% |\n| Swin-B | 79.8% | 76.2% |\n| ViT-B/16 | 82.0% | 78.9% |\n\n**RE Gradient:** In order to save GPU computing cost and memory, in the 50-step sampling process, we turn off the gradient of image and text in the first 49 steps and only keep the gradient of the last step. According to our quantitative experimental results, our algorithm can effectively improve the classification error rate of the CLIP classifier for both short text and long text.\n\n**RE Lack of Details:**\n1. The random algorithm extends or replaces words randomly.  For the long text prompts, we replace one or two words at random positions (without category keywords) and extend one word at the end. For short text prompts, the prompt template (i.e., “A photo of a [NOUN]”) will affect the quality of the final generated image, if the template is modified, some poor-quality images may be generated and make the classifier unable to recognize, but the category of the generated image is still the original category. In addition, the category keywords in the prompt also cannot be changed, because we want to remain the noun in the prompt and make the DM generate other objectives. Directly replace the original category keywords to make the original category disappear in the generated image, which is not a reasonable attack method. Therefore, we only extend one or two words at the end of the clean prompt, these words are randomly selected from the vocabulary of CLIP.\n2. CLM is GPT2. It is trained from scratch with the wikitext-103 L dataset under Clip's tokenizer. Because in our optimization goal, we use GP2 as the reference model to constrain the language fluency of the generated attack text prompt. For the same text, different tokenizers will map different input ids. For the language model, That is, different semantics, and there is currently no open source GPT2 trained by a Clip tokenizer, so we need to retrain a GPT2 based on a Clip tokenizer.\n3. The version of  Clip is ViT-B/16.\n\n\n\n\n\n\n\n"}, {"review_id": "ECRRodZrAl", "reviewer": "Reviewer_adct", "summary": "This paper proposes an adversarial attack against text-to-image models that can generate adversarial prompts to prevent the stable diffusion models from generating the desired subjects. The attack is gradient-based by utilizing the Gumbel Softmax to make the work embedding differentiable. Then, the authors provide a comprehensive analysis of the vulnerabilities of the stable diffusion models.", "questions": "1. What are the computation consumption and requirement to conduct the white-box attack against the stable diffusion?\n\n2. The attack success rate could be affected by the performance of the network. For example, the network could misclassify a subject even if the subject is correctly generated by the diffusion model. Could you discuss how to mitigate this issue during the evaluation procedure?\n\n3. It seems the attack will make the FID score higher and IS score lower. Does it indicate a drawback of the attack? I am confused about whether we need to maintain the FID and IS score during the attack. Could the authors provide some explanations?\n", "limitations": "None.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "\n1. This paper is well-written and well-organized.\n\n2. The proposed automatic attack is effective and convenient for implementation. \n\n3. The authors provide interesting and inspiring analyses of the vulnerabilities of the generative models. They found the differences in the generation speed, the similarity of coarse-grained subjects in the prompt, and the polysemy are the important factors in the robustness of the generative models. This analysis is definitely beneficial to the future study of the robustness of generative models.\n\n4. The authors show that the proposed attack is transferable as well. \n", "weaknesses": "1. Although the constraint in the attack seems to be novel, the attack objective is similar to the previous proposed C&W attacks.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewers for their diligent review of this paper and for providing constructive feedback.\n\n**RE Objective and C&W:**\nOur objective differs from that of C&W from three perspectives:\n\n1. **The type of target models:** C&W attacks commonly target classification models, aiming to alter the model's classification results. On the other hand, in our optimization objective, a generative model is targeted, aiming to alter the content of the generated image to make it different from the target class. Although our pipeline also includes a classification model, its role is to measure the success of the attack, and this classification model is not our target of attack.\n2. **The continuity/discreteness and differentiability of perturbations:** In C&W attacks, perturbations are applied to image inputs, which are continuous and differentiable (within the range of 0.0 to 1.0). In our optimization objective, the inputs consist of words represented by a dictionary of embedding vectors, which are discrete and non-differentiable. Therefore, it is necessary to introduce a mechanism that renders them continuous and differentiable, thereby enabling gradient-based optimization. To be specific, Gumbel Softmax distributions are introduced to make the perturbations continuous and differentiable. Two types of modifications, namely replacing and extending are to automatically learn the replacement or expansion of words, and then pass a text-to-image transformation function (i.e., stable diffusion) to detect the effectiveness of the perturbation.\n3. **The complexity of constraints:** The constraints applied in C&W attacks include regularization terms to restrict the p-norm of perturbations and the clamping of pixel values within the range of 0.0 to 1.0 after the attack. These terms are used to make perturbations very small and thus difficult for humans to detect. However, in our method, p-norm and clamping cannot be used since the difference between the text and image. Therefore, we design constraints that are more suitable for the textual data,i.e., fluency constraints, and BERT similarity constraints. This constraint not only limits the size of the perturbation, but also limits how similar the perturbed text is to the original text, and how close the generated distribution corresponding to the perturbation is to the true distribution.\n\n**RE Computation Consumption:** Our algorithm consumes 0.1 GPU Hours to learn the attack distribution of a sample on a single RTX3090 with 100 steps of gradient search, and 0.075 GPU Hours to sample 100 attack prompts from this distribution, so for a single sample, it consumes a total of 0.175 GPU Hours\n\n**RE Attack Success Rate:** The classifier we use for the attack is CLIP ViT-B/16.\nFor eliminate the impact of the performance of a single classifier, we design a voting mechanism by using an ensemble of multiple classifiers. To be specific, the attack is defined as a failure if at least one classifier in the ensemble still recognizes the original categories in the text prompt. When we took this mechanism to evaluate the generated image after our attack, the success rate of the attack on the long and short prompts remained the same, 81.8% and 91.1% respectively. This further proves the effectiveness of our algorithm. Additionally, we test the voting mechanism with images generated from clean prompts covering the 1000 classes in ImageNet. As shown in the PDF, most of the classifiers have a relatively high Top1 accuracy. We conducted experiments with three distinct classifiers: CLIP+ViT-B/16, Swin-B, and the vanilla ViT-B/16. The classification accuracies of these models were evaluated using both long and short text prompts. For the CLIP+ViT-B/16 model, the classification accuracies with long and short text prompts were found to be 78.80% and 82%, respectively. The Swin-B model demonstrated classification accuracies of 76.20% with long text prompts and 79.80% with short ones. Lastly, the vanilla ViT-B/16 model achieved classification accuracies of 78.90% and 82% with long and short text prompts, respectively. The probability that at least one of them successfully recognizes the current generated image is 90.40% and 92.30% for long and short text prompts, respectively, which is much higher than using only one classifier. This proves that our mechanism is effective.\n\n**RE FID and IS Scores:** This is not a drawback, instead this is an echo of our algorithm design. The attack objective of our algorithm, namely the margin loss, is used to reduce the classifier’s confidence in the true class $y$ and improve its confidence in the class with the largest confidence excluding $y$, until a margin $\\kappa$ is reached. Thereby suppressing the categories in the clean text prompt from appearing in the generated images. Therefore, from the perspective of this loss function, whether it is to reduce the quality of the image or make the original category disappear from the generated image, both align with this expectation.\n\nFID is a metric that measures the distance between the distribution of generated images and the reference dataset. In our paper, the reasons for the FID metrics increased can be concluded as follows. Firstly, we use the ImageNet validation set as the reference dataset, the content of the image generated after the attack will deviate from the distribution of this reference dataset, as the nouns used for replacement or expansion might not exist within ImageNet's 1000 categories.\n\nThe second point is that after the attack, images generated by the stable diffusion model indeed contain low-quality images. This allows us to explore the types of prompts that can diminish the ability to generate images of stable diffusion, which is also one of our attack purposes. Those aforementioned images will not only increase the FID but also reduce the IS score.\n"}, {"review_id": "QL0czefytH", "reviewer": "Reviewer_czYq", "summary": "The paper introduces Auto-attack on Text-to-image Models (ATM), a method to efficiently generate attack prompts that closely resemble clean prompts. \nThe method modifies text prompts by replacing or extending words, using a Gumbel Softmax distribution for differentiability.\nIt further applies a binary mask to preserve the desired object noun and imposes fluency and similarity constraints to ensure similarity. \nThe authors identify four distinct attack patterns through this attack method.\nThe method utilizes Stable Diffusion as the target model, allowing for white-box attacks, and demonstrates the transferability of these attacks to other generative models (black-box attacks).", "questions": "Please address questions mentioned in \"Weaknesses\".", "limitations": "There are no potential negative societal impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors introduce a novel method to generate successful attack prompts in text-to-image generation pipelines. It can aid in vulnerability investigation. The method enables the identification of a broader range of attack patterns, prompting further research in both attack and defense mechanisms. Therefore, it can enhance security in the industry.\n\n2. Related to the previous point, the authors themselves employ their method to uncover four distinct attack patterns, which are highly enlightening. This convincingly demonstrates the significant importance of this approach in studying the vulnerability of generative models and enhancing their robustness.\n\n3. In their experiments, the proposed method demonstrates a high success rate in white-box attacks. It also exhibits excellent transferability to other generative models, e.g. DALL-E 2 and Midjourney. This proves that the method has a broad range of applications and can be used for various generative models, not just limited to Stable Diffusion.", "weaknesses": "1. Two types of modifications, namely replacing and extending, are considered. It can be confusing to determine when to use each type. How do they determine whether to replace or extend a word? \n\n2. Most of the analyses are solid. However, in Section 3.1, the authors mention embedding two prompts c_1 and c_2 separately to \"eliminate the additional impact of all possible extraneous factors\". It is unclear what specific impact they want to eliminate and how embedding prompts separately can achieve this.\n\n3. The authors mention the use of the method to design defensive strategies but do not discuss how to utilize it. Is it possible to provide an example?\n\n4. A minor issue: In Fig. 6, the steps should range from 49 to 0 instead of 1 to 50. This is because in the reverse diffusion process, T represents noise, and 0 represents the generated image (similar to the steps shown in Fig. 4).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewers for their diligent review of this paper and for providing constructive feedback.\n\n**RE Replacing and Extending:** The two types of modifications can be automatically selected by our sampling mechanism and can be optimized using gradients. In the process of prompt tokenization, we uniformly pad all the prompts into the maximum length that the Clip can accept, which is 77.  After learning the attack distribution, when sampling by Gumble-Softmax, if the maximum value falls in the corresponding position of the clean text, it is the replacing, otherwise extending. For short text prompts, the prompt template (i.e., “A photo of a [NOUN]”) will affect the quality of the final generated image if the template is modified, some poor-quality images may be generated and make the classifier unable to recognize, but the category of the generated image is still the original category. In addition, the category keywords in the prompt also cannot be changed, because we want to remain the noun in the prompt and make the DM generate other objectives. Directly replace the original category keywords to make the original category disappear in the generated image, which is not a reasonable attack method. so only the extension is adopted.\n\n**RE Additional Impact:** In section B.4 in the supplementary material, we mentioned that the different positions of the category keywords in the prompt will lead to different contents in the final generated image. The reason is that there is a difference between the language model and humans in the understanding of the text. Even if the relative order of the category words is changed, the semantics will not change for humans, but for Clip’s text encoder, there will be a certain gap in the extracted semantic information. Thus, we first split one prompt into two parts and then concatenate the two embeddings, which can cut off the positional embedding of the two keywords, therefore eliminating the impact of the positional information.\n\n**RE Defensive Strategies:** Our algorithm can generate a large number and variety of attack text prompts, which is difficult to achieve manually.\n\n**(1)** Researchers can discover more vulnerabilities in the stable diffusion model by observing and analyzing these attack samples and designing an effective algorithm to improve the stability of the stable diffusion model based on the observed failure cases. We design some possible defense strategies against the four patterns of vulnerability we found as follows:\n\n**Pattern 1. Variation in generation speed:** The speed at which the model generates images may depend on the relative generation speed in the text prompt, which may cause image features of one **noun** to overwrite those of another, or the sampling process enters into fine-grained generation information, the coarse-grained information of one of the **noun** has not been fully generated (the experiment that cannot generate categories without coarse-grained information can be seen in the third row of Figure 6 in the paper). defense mechanism may include:\n\n1. The development of new training strategies so that the model can generate image features of each **noun**  in a more balanced manner.\n2. Adopt or design a new mechanism to try to solve the coordination problem in the sampling process of each token in the prompt. Intuitively, it is to keep the generation speed of each token consistent, so as to solve the problem of content disappearing in the generated images.\n3. Add or design diversity losses or regularization terms for model training to encourage the model to generate more balanced and diverse images.\n\n**Pattern 2. The similarity of coarse-grained features:** When two **noun** have similar coarse-grained features, the model may generate images that mix the features of these two **noun**. Defense mechanisms may include: \n1. enhancing the feature discriminative ability of the model so that it can better handle **noun** that are different but have similar coarse-grained properties.\n2. Introduce adversarial training to make the model better learn the differences between different **nouns**, instead of only focusing on their shared characteristics.\n3. For coarse-grained similarity, we can preset multiple anchors during the generation process, forcing the category words in the prompt to be generated only in the corresponding anchors to solve the problem of attention map overlap.\n\n**Pattern 3. Polysemy of Words:** When **noun** is ambiguous, the model may produce images that do not match user expectations. Defense mechanisms may include a more comprehensive consideration of the semantic context of **noun** during model training, enabling models to more accurately understand the specific meaning of **noun**. A solution might be to introduce context-aware language models to enhance the model's understanding of lexical polysemy.\n\n**Pattern 4. Word position:** The position of a descriptor in a sentence may affect the image generated by the model. Defense mechanisms may include\n1. Train the model to better understand the location information of **noun** to reduce the impact of location changes on generated images.\n2. Design a module that can manually adjust the weight of each word to eliminate the influence of location information.\n\n\n**(2)** We can use this algorithm to generate a large number of offensive text prompts and use these attack samples to create a data set specifically for generation stability. This data set can be used to test the stability of the diffusion generation model, and can also be used for Adversarial training, to improve the stability of the model. \n\n**RE minor issue:** Thank you for pointing out this mistake, we will correct it immediately.\n"}, {"review_id": "tObUb84uex", "reviewer": "Reviewer_L6BS", "summary": "This paper finds some vulnerabilities of stable diffusion model, and proposes an auto-attack model to generate attack prompts.", "questions": "Are there some related work in this field? I would be better to mention some related work, and discuss or compare them in your experiments.\n\nWhen describing the Variability in Generation Speed, the definition of speed is little confusing. What does the speed value represent? How to understand it?", "limitations": "This paper proposes an attack model, but lack of discussion of defense model. It would be better to discuss them together.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well written and is easy to understand.\nThis paper discusses some vulnerabilities of the stable diffusion model, and does many experiments to verify.\nThe methodology illustration is clear and easy to follow.", "weaknesses": "The purpose and motivation of ATM is not well explained. Authors have find some vulnerabilities of the stable diffusion model, but the motivation of ATM is not correlated well with them. It would be better to connect the ATM and the aforementioned vulnerabilities.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewers for their diligent review of this paper and for providing constructive feedback.\n\n**RE Purpose and Motivation:** Stable diffusion along with other text-to-image generative models (e.g., Midjourney and DALL-E 2) have wide-ranging applications and implications in the field of AIGC. Firstly, in practical application scenarios, the model may receive a variety of text prompts. Therefore, we need to explore under what form of text prompts, the text-to-image model will not be able to complete the image generation task, so as to improve the stability of the stable diffusion model. Our ATM algorithm is able to retrieve multiple attacking prompts based on a clean text prompt that prevents the stable diffusion model from generating images that match human aesthetics. Secondly, the current stable diffusion model still has many defects, some may be more common, but some are more invisible defects. Based on our algorithm to generate a large number of failure cases, researchers can systematically analyze the logic behind the failure cases. Also, the four vulnerabilities mentioned in the paper are summarized by observing these cases.\n\n\n**RE Discussion of Defense Model:** Our method can inspire the design of defense strategies and can work as an effective and reliable metric to evaluate those strategies. We design some possible defense strategies against the four patterns of vulnerability we found as follows:\n\n**Pattern 1. Variation in generation speed:** The speed at which the model generates images may depend on the relative generation speed in the text prompt, which may cause image features of one **noun** to overwrite those of another, or the sampling process enters into fine-grained generation information, the coarse-grained information of one of the **noun** has not been fully generated (the experiment that cannot generate categories without coarse-grained information can be seen in the third row of Figure 6 in the paper). defense mechanism may include:\n\n1. The development of new training strategies, so that the model can generate image features of each **noun**  in a more balanced manner.\n2. Adopt or design a new mechanism to try to solve the coordination problem in the sampling process of each token in the prompt. Intuitively, it is to keep the generation speed of each token consistent, so as to solve the problem of content disappearing in the generated images.\n3. Add or design diversity losses or regularization terms for model training to encourage the model to generate more balanced and diverse images.\n\n**Pattern 2. The similarity of coarse-grained features:** When two **noun** have similar coarse-grained features, the model may generate images that mix the features of these two ** noun**. Defense mechanisms may include: \n1. enhancing the feature discriminative ability of the model so that it can better handle **noun** that are different but have similar coarse-grained properties.\n2. Introduce adversarial training to make the model better learn the differences between different **nouns**, instead of only focusing on their shared characteristics.\n3. For coarse-grained similarity, we can preset multiple anchors during the generation process, forcing the category words in the prompt to be generated only in the corresponding anchors to solve the problem of attention map overlap.\n\n**Pattern 3. Polysemy of Words:** When **noun** is polysemy, the model may produce images that do not match user expectations. Defense mechanisms may include a more comprehensive consideration of the semantic context of **noun** during model training, enabling models to more accurately understand the specific meaning of **noun**. A solution might be to introduce context-aware language models to enhance the model's understanding of lexical polysemy.\n\n**Pattern 4. Word position:** The position of a descriptor in a sentence may affect the image generated by the model. Defense mechanisms may include\n1. Train the model to better understand the location information of **noun** to reduce the impact of location changes on generated images.\n2. Design a module that can manually adjust the weight of each word to eliminate the influence of location information.\n\n\n**RE Related Work:** In the section of A in Supplementary Material we refer to previous related work in exploring and addressing the stability of stable diffusion model generation as follows. DAAM performs a text-image attribution analysis on conditional text-to-image model and produces pixel-level attribution maps. Their research focuses on the phenomenon of feature entanglement and uncovers that the presence of cohyponyms may degrade the quality of generated images and that descriptive adjectives can attend too broadly across the image. Attend-and-Excite investigates the presence of catastrophic neglect in the Stable diffusion model, where the generative model fails to include one or more of the subjects specified in the input prompt. Additionally, they discover instances where the model fails to accurately associate attributes such as colors with their respective subjects. Although those works have some progress, there is still work to be done to enhance the stability and reliability of text-to-image models, ensuring consistent and satisfactory results for a wide range of text prompts， StructureDiffusion discovers that some attributes in the prompt not assigned correctly in the generated images, thus they employ consistency trees or scene graphs to enhance the embedding learning of the prompt.\n\n**RE Definition of Speed:** Take the generation process of an image as an example, stable diffusion requires N steps of the denoising process. We take the last step image as the reference image and use SSIM and LPIPS to calculate the distance from each step of the image to the last step. The speed of this distance converging to 1 (for SSIM) or 0 (for LPIPS) is the generation speed.\n"}]

Author rebuttal:
We would like to thank the reviewers for their diligent review of this paper and for providing constructive feedback.

This PDF includes a Violin plot illustrating the generation speed of the same class with different initial noises. Additionally, there is a table that details the classification accuracy of various classifiers on both clean long and short text prompts.

Meta-review: This paper proposes an adversarial attack against text-to-image models that can generate adversarial prompts to prevent the stable diffusion models from generating the desired subjects. The attack is gradient-based by using Gumbel Softmax to make the work embedding differentiable. The authors provide a comprehensive analysis of the vulnerabilities of the stable diffusion models. The proposed method enables the identification of a broader range of attack patterns, prompting further research in both attack and defense mechanisms, which would be useful to enhance security in the industry.
Decision: Accept (spotlight)