SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In addition to this explicit polysemy, our algorithm further demonstrates its aptitude for detecting subtler instances of polysemous words. As depicted in Figure 7, the transformative capacity of our algorithm is evident when an image of a warthog (Fig. 7d) transfigures into an image of a military chariot (Fig. 7e) with the incorporation of the term "traitor".
5 Auto-attack on Text-to-image Model
We aim to design an automatic attack method that targets the recent popular text-to-image models. The objective of our method is to identify attack prompts  $\pmb{c}^{\prime}$  based on a clean prompt  $\pmb{c}$ , which leads to a vision model  $h: \mathcal{X} \to \mathcal{Y}$  failing to predict the desired class  $y$ , i.e.  $\operatorname{argmax}_i h(\tilde{\pmb{x}})_i \neq y$ :
$$
\boldsymbol {c} ^ {\prime} = \underset {\boldsymbol {c} ^ {\prime} \in B _ {d} (\boldsymbol {c}, \xi)} {\operatorname {a r g m a x}} \ell (y, h (\mathrm {G M} (\boldsymbol {c} ^ {\prime}))) \tag {1}
$$
where  $\mathrm{GM}(\cdot)$  is a text-to-image model,  $B(\pmb {c},\xi) = \{\pmb {c}^{\prime}:d(\pmb {c},\pmb {c}^{\prime})\leq \xi \}$ ,  $d(\cdot ,\cdot)$  is a distance measure regularizing the similarity between  $\pmb{c}$  and  $\pmb{c}^{\prime}$ , and  $\xi$  is a maximum distance. To enable auto-attack, a differentiable method that can be optimized using gradient descent is desired. We introduce a Gumbel-Softmax sampler to enable differentiable modifications on text prompts during the word embedding phase. To minimize the distance  $d(c,c^{\prime})$ , we introduce two constraints, including a fluency constraint and a similarity constraint.
In our experimental setup, the open-source Stable Diffusion model is employed as the targeted generative model  $\mathrm{GM}(\cdot)$ . By generating white-box attack prompts for Stable Diffusion, we can subsequently transfer these prompts to other generative models to execute black-box attacks. To facilitate the classification task, we utilize a CLIP classifier as the vision model  $h(\cdot)$ , benefiting from its exceptional zero-shot classification accuracy. To establish the desired classes, we employ the 1,000 classes derived from ImageNet-1K. In the case of generating short prompts, a fixed template of "A photo of [CLASS_NAME]" is utilized to generate the prompts. Conversely, for the generation of long prompts, we employ ChatGPT 4 [24] as a prompt generation model. Subsequently, human experts verify the correctness of the prompts and check that the prompts indeed contain the noun associated with the desired class.
5.1 Differentiable Text Prompt Modification
A text prompt is typically a sequence of words  $c = \{c_1, \dots, c_K\}$ . Owing to the discrete nature of text prompts  $c$ , perturbations can be incorporated either by replacing an existing word  $c_k$  where  $1 \leq k \leq K$  or augmenting with new ones  $\{c_{K + i}|1 \leq i \leq K'\}$ . However, the non-differentiable nature of this procedure makes it unsuitable for optimization utilizing gradient-based techniques. Therefore, there is a need for a mechanism that guarantees the differentiability of the word selection process. In this regard, we integrate a Gumbel Softmax sampler  $\psi(\cdot; \tau)$  into the word embedding phase. The Gumbel Softmax function has the ability to approximate a one-hot distribution as the temperature  $\tau \rightarrow 0$ . Additionally, the Gumbel distribution has the ability to introduce further randomness, thereby enhancing the exploitability during the initial stages of perturbation search.
Differentiable Sampling. We employ a trainable matrix  $\omega \in \mathbb{R}^{K\times V}$  to learn the word selection distribution, where  $K$  is the length of the text prompt and  $V$  is the vocabulary size. In the scenario of augmenting with a new word, the sequence length  $K$  can be extended to  $K + K^{\prime}$  to facilitate the addition of new words. The Gumbel Softmax can be represented as follows:
$$
\text {G u m b e l S o f t m a x} \left(\omega_ {k}; \tau\right) := \frac {\exp \left(\left(\omega_ {k , i} + g _ {k , i}\right) / \tau\right)}{\sum_ {j} \exp \left(\left(\omega_ {k , j} + g _ {k , j}\right) / \tau\right)}, \tag {2}
$$
where  $g_{k,i} \sim \mathrm{Gumbel}(0,1)$  are i.i.d. samples from a Gumbel distribution. The word embedding stage employs a matrix  $E \in \mathbb{R}^{V \times D}$ , where  $V$  is the vocabulary size and  $D$  is for the embedding dimensionality. The discrete process of word embedding is to select  $E_{i}$  from  $E$  based on the index  $1 \leq i \leq V$  of a word in the vocabulary. To make this process differentiable, the dot product between GumbelSoftmax  $(\omega_{k})$  and  $E$  can be calculate:
$$
\boldsymbol {c} _ {k} ^ {\prime} = \psi (\boldsymbol {\omega} _ {k}; \tau) = \operatorname {G u m b e l S o f t m a x} (\boldsymbol {\omega} _ {k}; \tau) \cdot E \approx E _ {i}, \quad \text {s . t .} i = \underset {i} {\operatorname {a r g m a x}} \boldsymbol {\omega} _ {k, i}, \tag {3}
$$
where  $c_k'$  is the new word selected according to  $\omega_k$ . As  $\tau \to 0$ , Eq. 3 can effectively emulate an argmax selection procedure.
Additionally, in order to ensure similarity, it is desired to preserve the noun representing the desired object in the new prompt. This is achieved by utilizing a binary mask  $M \in \{0,1\}^K$ , where the position corresponding to the desired noun is set to 0 while other positions are set to 1. By computing  $c' \gets (1 - M) \cdot c + M \cdot c'$ , the desired noun can be retained in the prompt while other words can be modified.
Attack Objective. To generate images  $\tilde{\pmb{x}}$  that cannot be correctly classified by the classifier, a margin loss [1] can be used as the loss function  $\ell (\cdot ,\cdot)$  in Eq. 1:
$$
\ell_ {\text {m a r g i n}} (\tilde {\boldsymbol {x}}, y; h) = \max  \left(0, h (\tilde {\boldsymbol {x}}) _ {y} - \max  _ {i \neq y} h (\tilde {\boldsymbol {x}}) _ {i} + \kappa\right), \tag {4}
$$
where  $\kappa$  is a margin. Eq. 4 reduces the classifier's confidence on the true class  $y$  and improves its confidence on the class with the largest confidence, excluding  $y$  until a margin  $\kappa$  is reached.
5.2 Constraints on Fluency and Similarity
Given that we search for perturbations in a  $\mathbb{R}^{K\times V}$  space to attack the text prompt, the attack prompts may be too diverse if the added perturbations are not properly constrained, making it easily detectable. Eq. 1 includes a distance constraint such that  $d(c,c^{\prime})\leq \xi$  , which ensures that the added perturbations are subtle and hard to notice. The measurement of distance between two pieces of text can be approached through various methods. We introduce two constraints to reduce this distance, namely a fluency constraint and a semantic similarity constraint. The fluency constraint ensures that the generated sentence is smooth and readable, while the semantic similarity semantic constraint regularize the semantic changes introduced by the perturbations, making the attack prompt  $c^{\prime}$  closely resemble the clean prompt  $c$
Fluency constraint. The fluency constraint can be achieved via a Casual Language Model (CLM)  $\phi$  with log-probability outputs [9]. The next token distribution we learn is compared with the next token distribution predicted by  $\phi$ . Given a sequence of perturbed text  $c'$ , we use  $\phi$  to predict the a token  $c_i'$  based on  $\{c_1', \ldots, c_{i-1}'\}$ . Therefore, we can have a log-likelihood of the possible next word:
$$
\log p _ {\phi} \left(\boldsymbol {c} _ {i} ^ {\prime} \mid \boldsymbol {c} _ {1} ^ {\prime}, \dots , \boldsymbol {c} _ {i - 1} ^ {\prime}\right) = \phi \left(\boldsymbol {c} _ {1} ^ {\prime}, \dots \boldsymbol {c} _ {i - 1} ^ {\prime}\right). \tag {5}
$$
The next token distribution we learn can be easily obtained by GumbelSoftmax  $(\omega_{i};\tau)$ . Subsequently, a cross-entropy loss function can be employed to optimize the learned distribution:
$$
\mathrm {C E} _ {\phi} (\boldsymbol {\omega}) = - \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {D} \operatorname {G u m b e l S o f t m a x} \left(\boldsymbol {\omega} _ {i}; \tau\right) _ {j} \cdot \phi \left(\psi \left(\boldsymbol {\omega} _ {1}; \tau\right), \dots \psi \left(\boldsymbol {\omega} _ {i - 1}; \tau\right)\right) _ {j}. \tag {6}
$$
Eq. 6 serves as a regularizer to encourage the next word selection distribution to resemble the prediction of the CLM  $\phi$ , thereby ensuring fluency.
Algorithm 1 Auto-attack on Text-to-image Models (ATM)
Input: The maximum number of iterations  $T$ . The maximum number of attack candidates  $N$ . The clean prompt  $c$ . The desired class  $y$ . A binary mask  $M$ . A learning rate  $\eta$ .  
Output: A set of attack prompts  $S$ .  
1: Initialize  $\omega$   
2: for  $t = 1 \rightarrow T$  do  
3: Sample an attack prompt  $c' = \{\psi(\omega_k; \tau) | 1 \leq k \leq K\}$   
4: Apply the mask by  $c' \gets (1 - M) \cdot c + M \cdot c'$   
5: Generate an image  $\tilde{x}' = \mathrm{GM}(z_T | c')$   
6: Get classification results  $y' = h(\tilde{x}')$   
7: Conduct a gradient descent step  $\omega \gets \omega - \eta \cdot \nabla_{\omega} \mathcal{L}(\omega)$   
8: end for  
9: Initialize  $S = \emptyset$   
10: for  $n = 1 \rightarrow N$  do  
11: Sample an attack prompt  $c' = \{\psi(\omega_k; \tau) | 1 \leq k \leq K\}$   
12: Apply the mask by  $c' \gets (1 - M) \cdot c + M \cdot c'$   
13: Generate an image  $\tilde{x}' = \mathrm{GM}(z_T | c')$   
14: if  $\argmax h(\tilde{x}') \neq y$  then  
15: Save the success attack prompt  $S \gets S \cup \{c'\}$   
16: end if  
17: end for
Semantic similarity constraint. Rather than simply considering a word similarity, we concern more about semantic similarity. One prominent metric used to evaluate semantic similarity is the BERTScore [39]. The calculation of BERTScore requires contextualized word embeddings. The aforementioned CLM  $\phi$  is used again to extract the embeddings  $\pmb{v} = \phi^{(\mathrm{emb})}(\pmb{c})$ , where  $\phi^{(\mathrm{emb})}$  denotes the embedding network used in  $\phi$ . The BERTScore between the clean prompt  $c$  and the attack prompt  $c'$  can be calculated by
$$
S _ {\text {B E R T}} \left(\boldsymbol {c}, \boldsymbol {c} ^ {\prime}\right) = \sum_ {i = 1} ^ {N} w _ {i} \max  _ {j = 1, \dots , M} \boldsymbol {v} _ {i} ^ {\top} \boldsymbol {v} _ {j} ^ {\prime}, \tag {7}
$$
where  $w_{i}\coloneqq \mathrm{idf}(\pmb {c}_{i}) / \sum_{i = 1}^{N}\mathrm{idf}(\pmb {c}_{i})$  is the normalized inverse document frequency (idf),  $N = K$  and  $M$  is either  $K$  or  $K + K^{\prime}$  depending on whether existing words are being replaced or new words are being added. To improve the similarity, we use  $1 - S_{\mathrm{BERT}}(\pmb {c},\pmb{c}^{\prime})$  as the our loss term.
The constrained objective. Considering that the addition of constraints may limit the diversity of perturbation search, we introduce two hyper-parameters,  $\lambda$  and  $\gamma$ , to control the strength of the constraints. Then, the overall objective function can be written as:
$$
\max  _ {\boldsymbol {\omega}} \mathbb {E} _ {\boldsymbol {z} _ {T} \sim \mathcal {N} (0, 1), \boldsymbol {c} ^ {\prime} = \psi (\boldsymbol {\omega}; \tau)} \left[ \ell_ {\text {m a r g i n}} \left(\mathrm {G M} \left(\boldsymbol {z} _ {T} \mid \boldsymbol {c} ^ {\prime}\right), y; h\right) \right] \tag {8}
$$
$$
\mathbf {s}. \mathbf {t}. \quad \min  _ {\boldsymbol {\omega}} \quad \lambda \cdot \mathrm {C E} _ {\phi} (\boldsymbol {\omega}) + \gamma \cdot \left(1 - S _ {\text {B E R T}} \left(\boldsymbol {c}, \boldsymbol {c} ^ {\prime}\right)\right). \tag {9}
$$
5.3 Generation of Attack Prompts
The overall procedure of ATM is as described in Algorithm 1. It consists of two stages: a search stage, where the Gumbel Softmax distribution is learned, and an attack stage, where we generate attack prompts using the learned distribution. In the search stage, we use gradient descent to optimize the parameters  $\omega$  for each clean prompt  $c$  over a period of  $T$  iterations. Once  $\omega$  is learned, we proceed to the attack stage. In this stage, we sample  $N$  attack prompts from each learned  $\omega$ . An attack prompt  $c'$  is considered successful if the image  $\tilde{\boldsymbol{x}}'$  generated from it cannot be correctly classified by the visual classifier  $h$ .
5.4 Attack Experiment Results
To corroborate the effectiveness of our proposed algorithm, we implemented a structured evaluation scheme based on the model template: "A photo of [CLASS_NAME]". An extensive dataset comprising one thousand abbreviated textual descriptions, corresponding to all classes within ImageNet [3], was meticulously curated. Additionally, comprehensive long-form textual descriptions for these
same classes were generated utilizing the capabilities of ChatGPT4 [24]. Upon rigorous testing, our algorithm demonstrated noteworthy performance, recording attack success rates of  $91.1\%$  and  $81.2\%$  on the short-text and long-text datasets, respectively. Moreover, the calculated cosine similarity between the embeddings of the original and attack prompts yielded values of 0.7227 and 0.8364, respectively. Due to the length of this paper, detailed experiment results, including comparisons to baselines and ablation studies, are reported in the supplementary material.
To further investigate whether our generated attack prompts can be transferred to different text-to-image models, we randomly select several attack prompts to attack DALL-E2 and Midjourney, respectively. The experimental results prove that our attack prompts can also be used for black-box attacks. More results of black-box attacks are reported in the supplementary material.
The realm of text-to-image generation has observed a remarkable evolution over recent years, while concurrently exposing several vulnerabilities that require further exploration. Despite the many advancements, there are key limitations, specifically concerning the stability and reliability of generative models, which remain to be addressed. This paper has introduced Auto-attack on Text-to-image Models (ATM), a novel approach that generates a plethora of successful attack prompts, providing an efficient tool for probing vulnerabilities in text-to-image models. ATM not only identifies a broader array of attack patterns but also facilitates a comprehensive examination of the root causes. We believe that our proposed method will inspire the research community to shift their focus toward the vulnerabilities of present-day text-to-image models, stimulating further exploration of both attack and defensive strategies. This process will be pivotal in advancing the security mechanisms within the industry and contributing to the development of more robust and reliable systems for generating images from textual descriptions.
[1] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE symposium on security and privacy (sp), pages 39-57. IEEE, 2017.
[2] H. Chefer, Y. Alaluf, Y. Vinker, L. Wolf, and D. Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.13826, 2023.
[3] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
[4] A.-D. Dinh, D. Liu, and C. Xu. PixelAsParam: A gradient view on diffusion sampling with guidance. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 8120-8137. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/dinh23a.html.
[5] A.-D. Dinh, D. Liu, and C. Xu. Rethinking conditional diffusion sampling with progressive guidance. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://neurips.cc/virtual/2023/poster/70851.
[28] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[29] L. Ruan, Y. Ma, H. Yang, H. He, B. Liu, J. Fu, N. J. Yuan, Q. Jin, and B. Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. arXiv preprint arXiv:2212.09478, 2022.
[30] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[31] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[32] Stability-AI. Stable diffusion public release. https://stability.ai/blog/stable-diffusion-public-release, 2023. Accessed on 2023-05-17.
[33] R. Tang, A. Pandey, Z. Jiang, G. Yang, K. Kumar, J. Lin, and F. Ture. What the daam: Interpreting stable diffusion using cross attention. arXiv preprint arXiv:2210.04885, 2022.
[34] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
[35] X. Yang, D. Zhou, S. Liu, J. Ye, and X. Wang. Deep model reassembly. In Advances in Neural Information Processing Systems, 2022.
[36] S. Yin, C. Wu, H. Yang, J. Wang, X. Wang, M. Ni, Z. Yang, L. Li, S. Liu, F. Yang, et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation. arXiv preprint arXiv:2303.12346, 2023.
[37] L. Zhang and M. Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[38] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[39] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020.
In this supplementary material, we first delve into additional analyses concerning the vulnerabilities observed in the Stable Diffusion model (Section A). Subsequently, we offer instances of long and short prompt attacks, accompanied by the corresponding generated images, as well as instances of black-box attacks (Section B). Lastly, we undertake a comprehensive series of experiments to substantiate the effectiveness of our approach (Section C). These experiments include the evaluation of attacks targeting both long and short prompts. Additionally, ablation studies are conducted to explore attacks employing different search steps, assess the influence of our fluency and semantic similarity constraints on text similarity, and target diverse samplers (e.g., DDIM and DPM-Solver) in the attack process.
A Vulnerabilities of Stable Diffusion Model
A.1 Pattern 1: Variability in Generation Speed
Fig. 10 demonstrates the entire 50-step violin diagram which has been discussed before. To eliminate possible bias due to a single metric, we further verified the difference in generation speed of one thousand images based on the LPIPS [38] metric, as shown in Fig. 11. The calculation of the LPIPS distance from the images generated at each stage to the ultimate image is performed. The horizontal axis signifies the range of steps from 49 down to 0, whereas the vertical axis denotes the respective LPIPS scores. Each violin plot illustrates the distribution of the LPIPS scores associated with 1,000 images at a specific step. The width of the violin plot is proportional to the frequency at which images achieve a certain score. During the initial stages of generation, the distribution's median is situated nearer to the maximum LPIPS value, suggesting a preponderance of classes demonstrates slower generation velocities. Nonetheless, the existence of a low minimum value indicates the presence of classes that generate at comparatively faster rates. As the generation transitions to the intermediate stages, the distribution's median progressively decreases, positioning itself between the maximum and minimum LPIPS values. In the concluding stages of generation, the distribution's median is found closer to the minimum LPIPS value, implying that the majority of classes are nearing completion. However, the sustained high maximum value suggests that there are classes still exhibiting slower generation rates.
A.2 Pattern 2: Similarity of Coarse-grained Characteristics
To further verify that coarse-grained feature similarity is the root cause of feature entanglement, we provide more cases in Fig.12. From these cases, it is evident that for any two classes prone to feature entanglement, they can each proceed with image generation based on the other's coarse-grained information.
Figure 10: A violin plot illustrating the generation speeds of 1,000 images of various classes. The horizontal axis represents the number of steps taken, ranging from 49 to 0, while the vertical axis displays the SSIM scores. The width of each violin represents the number of samples that attained a specific range of SSIM scores at a given step.
A.3 Pattern 3: Polysemy of Words
As shown in Fig. 13, when we attack the prompt "A photo of a warthog" to "A photo of a warthog and a traitor", the original animal warthog becomes an object similar to a military vehicle or military aircraft, while the images generated by attack prompt is not directly related to the image of the animal warthog or traitor. From the t-SNE visualization (Fig. 14), we can see that the distance from the picture generated by the attack prompt to the text "a photo of a warthog" has a similar distance to the animal warthog picture to the text, so we can see that by attacking the original category word that guided the original category word (animal warthog) into its alternative meaning.
From the box plots (Fig. 15), it can be observed that the image of "warthog" exhibits the highest similarity with the prompt's embedding, while the image of "traitor" demonstrates the lowest similarity, as anticipated. Simultaneously, the similarity distribution between the images of "warthog" and "traitor" with the prompt text is relatively wide, indicating that some images have a high similarity with "warthog," while others lack features associated with "warthog."
Figure 11: A violin plot illustrating the generation speeds of 1,000 images of various classes. The horizontal axis represents the number of steps taken, ranging from 49 to 0, while the vertical axis displays the LPIPS scores. The width of each violin represents the number of samples that attained a specific range of LPIPS scores at a given step.
Figure 12: The image caption, "A photo of class  $A$  and class  $B$ " represents the generated image when feature entanglement occurs; And "class  $A$  from class  $B$ " represents the final generated image of prompt "A photo of class  $A$ " based on the eighth step of the prompt "A photo of class  $B$ ".
Figure 13: The images in the first row are generated by the prompt "A photo of a warthog". The images in the second row are generated by the prompt "A photo of a traitor". The images in the third row are generated by the prompt "A photo of a warthog and a traitor".
Figure 14: t-SNE Visualization of 100 images each of "warthog", "traitor", "warthog and traitor" and text "a photo of a warthog."
Figure 15: The boxplot of cosine similarities between the text embedding of "a photo of a warthog" and 100 of image embeddings each of "warthog", "traitor", and "warthog" and traitor".