SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chengbin Du, Yanxi Li, Zhongwei Qiu, Chang Xu
School of Computer Science, Faculty of Engineering
University of Sydney, Australia
chdu5632@uni.sydney.edu.au, yali0722@uni.sydney.edu.au
zhongwei.qiu@sydney.edu.au, c.xu@sydney.edu.au
Recently, text-to-image models have been thriving. Despite their powerful generative capacity, our research has uncovered a lack of robustness in this generation process. Specifically, the introduction of small perturbations to the text prompts can result in the blending of primary subjects with other categories or their complete disappearance in the generated images. In this paper, we propose Auto-attack on Text-to-image Models (ATM), a gradient-based approach, to effectively and efficiently generate such perturbations. By learning a Gumbel Softmax distribution, we can make the discrete process of word replacement or extension continuous, thus ensuring the differentiability of the perturbation generation. Once the distribution is learned, ATM can sample multiple attack samples simultaneously. These attack samples can prevent the generative model from generating the desired subjects without tampering with the category keywords in the prompt. ATM has achieved a  $91.1\%$  success rate in short-text attacks and an  $81.2\%$  success rate in long-text attacks. Further empirical analysis revealed three attack patterns based on: 1) variability in generation speed, 2) similarity of coarse-grained characteristics, and 3) polysemy of words. The code is available at https://github.com/duchengbin8/Stable_Diffusion_is_Unstable
In recent years, the field of text-to-image generation has witnessed remarkable advancements, paving the way for groundbreaking applications in computer vision and creative arts. Notably, many significant developments have captured the attention of researchers and enthusiasts, such as Stable Diffusion [28, 32], DALL-E2 [26, 27, 23] and Midjourney [19]. These developments push text-to-image synthesis boundaries, fostering artistic expression and driving computer vision research.
Despite the remarkable progress in text-to-image models, it is important to acknowledge their current limitations. One significant challenge lies in the instability and inconsistency of the generated outputs. In some cases, it can take multiple attempts to obtain the desired image that accurately represents the given textual input. An additional obstacle revealed by recent researches [33, 2, 8] is that the quality of generated images can be influenced by specific characteristics inherent to text prompts. Tang et al. [33] proposes DAAM, which performs a text-image attribution analysis on conditional text-to-image model and produces pixel-level attribution maps. Their research focuses on the phenomenon of feature entanglement and uncovers that the presence of cohyponyms may degrade the quality of generated images and that descriptive adjectives can attend too broadly across the image. Attend-and-Excite [2] investigates the presence of catastrophic neglect in the Stable diffusion model, where the generative model fails to include one or more of the subjects specified in the input prompt. Additionally, they discover instances where the model fails to accurately associate attributes such as colors with their respective subjects. Although those works have some progress, there is still work to be done to enhance the stability and reliability of text-to-image models, ensuring consistent and satisfactory results for a wide range of text prompts.
One prominent constraint observed in those works related to the stability of text-to-image models lies in their dependence on manually crafted prompts for the purpose of vulnerability identification. This approach presents several challenges. Firstly, it becomes difficult to quantify the success and
Figure 1: The overall pipeline of our attack method. The selection of words is relaxed to be differentiable by using a Gumbel Softmax with temperature  $\tau$ . When  $\tau \rightarrow 0$ , the Gumbel Softmax outputs exhibit proximity to one-hot vectors while retaining differentiability. After an image is generated, a CLIP [25] classifier and a margin loss are employed to optimize  $\omega$  aiming to generate images that cannot be correctly classified by CLIP.
failure cases accurately, as the evaluation largely depends on subjective judgments and qualitative assessments. Additionally, the manual design of prompts can only uncover a limited number of potential failure cases, leaving many unexplored scenarios. Without a substantial number of cases, it becomes challenging to identify the underlying reasons for failures and effectively address them. To overcome these limitations, there is a growing demand for a learnable method that can automatically identify failure cases, enabling a more comprehensive and data-driven approach to improve text-to-image models. By leveraging such an approach, researchers can gain valuable insights into the shortcomings of current methods and develop more robust and reliable systems for generating images from textual descriptions.
In this paper, we propose Auto-attack on Text-to-image Models (ATM), to automatically and efficiently generate attack prompts with high similarity to given clean prompts (Fig. 1). We use Stable Diffusion [28, 32] as our target model. With the open-source implementation and model parameters, we can generate attack prompts with a white-box attack strategy. Remarkably, those attack prompts can transfer to other generative models, enabling black-box attacks. Two methods to modify a text prompt are considered, including replacing an existing word or extending with new ones. By incorporating a Gumbel Softmax distribution into the word embedding, the discrete modifications can be transformed into continuous ones, thereby ensuring differentiability. To ensure the similarity between clean and the attack prompts, a binary mask that selectively preserves the noun representing the desired object is applied. Moreover, two constraints are imposed: a fluency constraint that ensures the attack prompt is fluent and easy to read, and a similarity constraint that regulates the extent of semantic changes.
After the distribution is learned, ATM can sample multiple attack prompts at once. The attack prompts can prevent the diffusion model from generating desired subjects without modifying the nouns of desired subjects and maintain a high degree of similarity with the original prompt. We have achieved a  $91.1\%$  success rate in short-text attacks and a  $81.2\%$  success rate in long-text attacks. Moreover, drawing upon extensive experiments and empirical analyses employing ATM, we are able to disclose the existence of three distinct attack patterns, each of which corresponds to a vulnerability in the generative model: 1) the variability in generation speed; 2) the similarity of coarse-grained characteristics; 3) the polysemy of words. In the following, we will commence with an analysis of the discovered attack patterns in Section 4, followed by a detailed exposition of our attack method in Section 5.
In addition, ATM can automatically and efficiently generate plenty of successful attack prompts, which serves as a valuable tool for investigating vulnerabilities in text-to-image generation pipelines. This method enables the identification of a wider range of attack patterns, facilitating a comprehensive examination of the underlying causes. It will inspire the research community and garner increased attention toward exploring the vulnerabilities present in contemporary text-to-image models and will foster further research concerning both attack and defensive mechanisms, ultimately leading to enhanced security within the industry.
2.1 Diffusion Model.
Recently, the diffusion probabilistic model [30] and its variants [12, 21, 31, 28, 29, 4, 5, 15] have achieved great success in content generation [31, 13, 29], including image generation [12, 31], conditional image generation [28], video generation [13, 36], 3D scenes synthesis [18] and so on. Specifically, DDPM [12] adds noises to images and learns to recover images from noises step by step. Then, DDIM [31] improves the generation speed of the diffusion model by skipping steps inference. Then, the conditional latent diffusion model [28] formulates the image generation in latent space guided by multiple conditions, such as texts, images, and semantic maps, further improving the inference speed and boarding the application of the diffusion model. Stable diffusion [28], a latent text-to-image diffusion model capable of generating photo-realistic images given any text input, and its enhanced versions [37, 14, 20], have been widely used in current AI-generated content products, such as Stability-AI [32], Midjourney [19], DALL-E2 [23], and Runaway [7]. However, these methods and products cannot always generate satisfactory results from the given prompt. Therefore, in this work, we aim to analyze the robustness of stable diffusion in the generation process.
2.2 Vulnerabilities in Text-to-image Models.
With the open-source of Stable Diffusion [28], text-to-image generation achieves great process and shows the unparalleled ability on generating diverse and creative images with the guidance of a text prompt. However, there are some vulnerabilities have been discovered in existing works [8, 2, 33]. Typically, StructureDiffusion [8] discovers that some attributes in the prompt are not assigned correctly in the generated images, thus they employ consistency trees or scene graphs to enhance the embedding learning of the prompt. In addition, Attend-and-Excite [2] also introduces that the Stable Diffusion model fails to generate one or more of the subjects from the input prompt and fails to correctly bind attributes to their corresponding subjects. These pieces of evidence demonstrate the vulnerabilities of the current Stable Diffusion model. However, to the best of our knowledge, no work has systematically analyzed the vulnerabilities of the Stable Diffusion model, which is the goal of this work.
The architecture of the Stable Diffusion [28] comprises of an encoder  $\mathcal{E}:\mathcal{X}\to \mathcal{Z}$  and a decoder  $\mathcal{D}:\mathcal{Z}\rightarrow \mathcal{X}$ , where  $\tilde{\boldsymbol{x}} = \mathcal{D}(\mathcal{E}(\boldsymbol {x}))$ . Additionally, a conditional denoising network  $\epsilon_{\theta}$  and a condition encoder  $\tau_{\theta}$  are employed. In the text-to-image task, the condition encoder is a text encoder that maps text prompts into a latent space. The text prompt is typically a sequence of word tokens  $c = \{c_1,\dots ,c_K\}$ , where  $K$  is the sequence length. During the image generation process, a random latent representation  $z_{T}$  is drawn from a distribution such as a Gaussian distribution. Then, the reverse diffusion process is used to gradually recover a noise-free latent representation  $z$ . Specifically, a conditional denoising network  $\epsilon_{\theta}(z_t,t,\tau_{\theta}(c))$  is trained to gradually denoise  $z_{t}$  at each time step  $t = T,\ldots ,1$  to gradually reduce the noise level of  $z_{t}$ , where the condition  $c$  is mapped in to a latent space using  $\tau_{\theta}(\pmb {c})$  maps and the cross-attention between the condition and features is incorporated in  $\epsilon_{\theta}$  to introduce the condition. Finally, the denoised latent representation  $\pmb{z}$  is decoded by a decoder  $\mathcal{D}$  to produce the final output  $\tilde{\boldsymbol{x}}$ . The aim of our study is to introduce slight perturbations to the text prompts, thereby inducing the intended object to be blended with other objects or entirely omitted from the generated images. For the sake of simplification, in the forthcoming sections, we shall represent the image generation process using the GM as  $\tilde{\boldsymbol{x}} = \mathrm{GM}(\boldsymbol {z}_T|\boldsymbol {c})$ .
4 Vulnerabilities of Stable Diffusion Model
By applying the attack method proposed in this paper, a variety of attack text prompts can be generated and analyzed. In this section, the identified attack patterns are discussed. Details of the attack method are introduced in Section 5. We have discovered three distinct patterns of impairment in Stable Diffusion model: 1) Variability in Generation Speed, where the model struggles to reconcile the differences in generation speed among various categories effectively. 2) Similarity of Coarse-grained Characteristics, which arises from the feature entanglement of global or partial coarse-grained characteristics, which possess a high degree of entanglement. 3) Polysemy of Words, which involves the addition of semantically complementary words to the original prompt, resulting in the creation of images that contain brand-new content and are not related to the original category.
Figure 2: A case study on "mountain" and "peafowl".
Figure 3: The generation speeds of "mountain" and "peafowl".
Figure 4: A violin plot illustrating the generation speeds of 1,000 images of various classes. The horizontal axis represents the number of steps taken, ranging from 49 to 0, while the vertical axis displays the SSIM scores. The width of each violin represents the number of samples that attained a specific range of SSIM scores at a given step.
4.1 Variability in Generation Speed
Observation 1. When a given text prompt contains a noun  $A$  representing the object to be generated, the introduction of another noun  $B$  into the prompt, either through addition or replacement, leads to the disappearance of noun  $A$  from the generated image, replaced instead by noun  $B$ .
In Observation 1, we identify a phenomenon when replacing or adding a noun in the description in a text prompt, the new noun will lead to the completely disappear of the desired subject. As shown in Fig. 2, if the noun "peak" is replaced by "peafowl", the desired subject "mountain" disappears and the new subject "peafowl" is generated. To further investigate this phenomenon, we use two short prompts  $c_{1}$  and  $c_{2}$  containing "mountain" and "peafowl", respectively, to exclude the influence of other words in the long prompts. To eliminate the additional impact of all possible extraneous factors, such as contextual relationships, they are embedded separately and then concatenated together:  $\text{Concat}(\tau_{\theta}(c_{1}), \tau_{\theta}(c_{2}))$ . The result shows that almost no element of the mountain is visible in the generated image (Fig. 2).
Further analysis reveals a nontrivial difference in the generation speeds of the two subjects. To define the generation speed, a metric to measure the distance from a generated image  $\tilde{\boldsymbol{x}}_t$  at a given time step  $t = T - 1,\dots ,0$  to the output image  $\tilde{\boldsymbol{x}}_0$  is desired (note,  $\tilde{\boldsymbol{x}}_T$  is the initial noise). We use the structural similarity (SSIM) [34] as the distance metrics:  $s(t)\coloneqq \mathrm{SSIM}(\tilde{x}_t,\tilde{x}_0)$ . Therefore, the generation speed can be formally defined as the derivative of the SSIM regarding the time step:  $v(t)\coloneqq ds(t) / dt\approx (s(t) - s(t + 1)) / \Delta t$ , where  $\Delta t = 1$ . Thereby, we propose our Pattern 1.
Pattern 1 (Variability in Generation Speed). Comparing the generation speeds  $(v_{1}$  and  $v_{2})$  of two subjects  $(S_{1}$  and  $S_{2})$ , it can be observed that the outline of the object in the generated image will be taken by  $S_{1}$  if  $v_{1} > v_{2}$ . And it can be inferred that  $S_{2}$  will not be visible in the resulting image.
We further generates 1,000 images of various classes with the same initial noise and visualize their generation speed in Fig. 4 as a violin plot. The SSIM distance from the generated images at each
(a) A photo of a silver salmon and a feather.
(b) A photo of a tench and a buck.
(c) A photo of a zebra and a giraffe.
(d) A photo of a howler monkey and a snake.
Figure 5: Images generated by the template "A photo of  $A$  and  $B$ ".
Figure 6: The first row illustrates the generation process with the prompt "a photo of a silver salmon". The second row, based on the forty-second step of the first row, shows the generation process with the prompt "a photo of a feather". The third row, also building upon the forty-second step of the first row, presents the generation procedure when the prompt is "a photo of a magician". The fourth row depicts the generation process in the presence of feature entanglement. The fifth row demonstrates the generation process for two distinct categories without feature entanglement.
step to the final image is calculated. The horizontal axis represents  $49 \sim 0$  steps, while the vertical axis represents the SSIM scores. Each violin represents the distribution of the SSIM scores of the 1,000 images in a step, with the width corresponds to the frequency of images reaches the score. In the early stages of generation, the median of the distribution is positioned closer to the minimum value, indicating that a majority of classes exhibit slow generation speeds. However, the presence of a high maximum value suggests the existence of classes that generate relatively quickly. In the middle stages of generation, the median of the distribution gradually increases, positioning itself between the maximum and minimum values. In the later stages of generation, the median of the distribution is positioned closer to the maximum value, indicating that a majority of classes are nearing completion. However, the persistence of a low minimum value suggests the presence of classes that still exhibit slow generation speeds. This analysis highlights the variation in generation speeds across different classes throughout the entire generation process. This phenomenon can be interpreted as a characteristic of the generation process, where different classes exhibit varying speeds throughout the stages. It is possible that certain classes have inherent complexities or dependencies that cause them to generate more slowly. Conversely, other classes may have simpler structures or fewer dependencies, leading to faster generation.
4.2 Similarity of Coarse-grained Characteristics
Observation 2. When a text prompt contains a noun  $A$  representing the object to be generated, the introduction of another noun  $B$ , which describes an object with similar coarse-grained characteristics to the object represented by noun  $A$ , into the prompt, either through addition or replacement, results in the generation of an image that contains an object combining elements of both nouns  $A$  and  $B$ .
(a)
(d)
Figure 7: a) "A photo of a bat"; b) "A photo of a bat and a ball;" c) Heat map of the word "bat" in generated image; d) "A photo of a warthog"; e) "A photo of a warthog and a traitor"; f) Heat map of the word "warthog" in generated image.
(c)
(f)
Figure 8: a) t-SNE Visualization of 100 images each of "bat", "baseball bat", "bat and ball" and text "a photo of a bat." b) The boxplot of cosine similarities between the text embedding of "a photo of a bat" and 100 of image embeddings each of "bat", "baseball bat", and "bat and ball".
(a) tsne
(b) logits
The second case that we observed in our attacks is when two nouns in the text prompt share similar coarse-grained characteristics, the generated image will contain a subject that is a combination of these two nouns. As illustrated in Fig. 6, when given the text "a silver salmon and a feather," the GMs generate an image of a feather with the outline of a salmon. This happens because these two nouns (i.e., salmon and feather) exhibit a certain degree of similarity in their coarse-grained attributes. In contrast, there is no feature entanglement between "salmon" and "magician" because their coarse-grained features are vastly different from each other.
To verify this assumption, we first obtain the generated latent variable for the prompt "a photo of a silver salmon" at the early sampling step (e.g., 8 steps). Using this latent variable, we replace the prompt with "a photo of a feather" and continue generating images. The results confirm that feathers can continue to be generated based on the coarse-grained properties of silver salmon, and the final generated graph has high similarity with the generated graph of the prompt "a photo of a silver salmon and a feather". However, replacing "silver salmon" with "magician" does not seem to generate any object similar to "magician". This observation indicates that there is no coarse-grained feature entanglement between these two subjects. We summarize this observation in Pattern 2.
Pattern 2 (Similarity of Coarse-grained Characteristics). Let  $X_{t}^{A}$  and  $X_{t}^{B}$  denote the latent variables generated by the GM for word tokens  $A$  and  $B$ , respectively. Suppose  $t$  is small and let  $d$  represent the metric that measures the outline similarity between two images. If the text prompt contains both  $A$  and  $B$ , and  $d(X_{t}^{A},X_{t}^{B})$  falls below the threshold  $\sigma$ , then feature entanglement occurs in the generated image.
Based on the observed Pattern 2, The types of feature entanglement can be further divided into, direct entanglement and indirect entanglement. As shown in Fig. 5, direct entanglement represents the direct entanglement triggered by two categories of coarse-grained attributes that have global or local similarities. Indirect entanglement is shown in Fig. 5d, where the additional attribute trunk brought by the howler monkey has a high similarity with the coarse-grained attribute of the snake, thus triggering the entanglement phenomenon.
4.3 Polysemy of Words
Observation 3. When a text prompt contains a noun  $A$  representing the object to be generated, if the semantic scope of noun  $A$  encompasses multiple distinct objects, the generated image contains one of the objects described by noun  $A$ . If there exists another word  $B$  that, when combined with noun  $A$ , directs its semantics to a particular object, the introduction of word  $B$  into the prompt, either through addition or replacement, leads to the generation of an image that specifically contains that particular object.
The third scenario we observed in the attack is that the content of the generated image is not directly related to either the desired image or the added word. However, this is again different from a category of disappearance, where the desired target has a clear individual in the image. As illustrated in Figs. 7a, 7b and 7c, when the cleaning prompt "a photo of a bat" was modified to "a photo of a bat and a ball", the bat disappeared completely from the generated image, but the DDAM[33] heat map showed that the word "bat" is highly associated with the stick-like object in the newly generated image.
Pattern 3 (Polysemy of Words). When interpreting polysemous words, language models must rely on contextual cues to distinguish meanings. However, in some cases, the available contextual information may be insufficient or confuse the model by modifying specific words, resulting in a model-generated image that deviates from the actual intention of the user.
To further investigate the phenomenon of word polysemy in the Stable diffusion model, we used the prompts "a photo of a bat", "a photo of a baseball bat" and "a photo of a bat and a ball" to generate 100 images each using the stable diffusion model, and transformed these images into embedding form by CLIP image encoder, and transformed "a photo of a bat" into embedding form by CLIP text encoder, then visualized these 301 embeddings by t-SNE. As illustrated in Fig. 8a, Considering the entire set of bat images, bats (the animal) are closer to the text "a photo of a bat" than baseball bats are, as depicted in the t-SNE visualization. However, the distribution of the two categories also has relatively close proximity, indicating their underlying similarities. The category of "bat and ball" shows a more expansive distribution, almost enveloping the other two. This suggests that by modifying the original text from "a photo of a bat" to "a photo of a bat and a ball", the distribution of the clean text can be pulled towards another meaning in the polysemous nature of the word "bat". From the perspective of text-to-image model, this kind of modification can stimulate the polysemous property of the word, thereby achieving an attack effect.