SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 16: a) "A type of footwear with a thick, rigid sole, often made of wood, and an upper made of leather or another material. Clogs can be open or closed, and are commonly associated with Dutch and Scandinavian cultures." b) "footwear" is replaced by "pistol". c) "Dutch" is replaced by "pistol".
(a) 0-1-2
Figure 17: A template, "A photo of \( A \), \( B \) and \( C \)", is used to generate prompts, where \( A, B, C \in \{”cat”, ”pistol”，”clogs" \}. For example, "0-1-2" represents \( A = \) "cat", \( B = \) "pistol" and \( C = \) "clogs", and so on.
(b) 0-2-1
(c) 1-0-2
(d) 1-2-0
(e) 2-0-1
(f) 2-1-0
A.4 Pattern 4: Positioning of Words
In addition to the three aforementioned observations and patterns outlined in the paper, there is a fourth observations (Observation 4), which is related to positioning of words.
Observation 4. When a text prompt contains a noun  $A$  representing the object to be generated, there exists a preceding word  $B$  and a succeeding word  $C$  around noun  $A$ . When replacing either word  $B$  or  $C$  with another noun  $D$ , for certain instances of noun  $A$ , replacing word  $B$  results in the generation of an image containing noun  $D$ , while replacing word  $C$  still results in the generation of an image containing noun  $A$ . Conversely, for other instances of noun  $A$ , the opposite scenario occurs.
An example of Pattern 4 is shown in Fig. 16. When "footwear" is replaced by "pistol", the generated image contains a pistol instead of clogs. However, when "Ductch" is replaced by "pistol", the model still generates an image of clogs. In addition to differences in the words being replaced, a significant distinction between the two aforementioned examples of success and failure lies in the relative positioning of the word being replaced with respect to the target class word. We hypothesize that this phenomenon occurs due to the different order of the replaced words  $B$  or  $C$  with respect to the noun  $A$ . To exclude the effects of complex contextual structures, a template for a short prompt, "A photo of  $A$ ,  $B$  and  $C$ ", is used, and the order of  $A$ ,  $B$ , and  $C$  are swapped (Fig. 17).
When these sentences with different sequences of category words are understood from a human perspective, they all have basically the same semantics: both describe a picture containing a cat, clogs, and a pistol. However, in the processing of language models (including CLIP), the order of words may affect their comprehension. Although positional encoding provides the model with the relative positions of words, the model may associate different orders with different semantics through learned patterns. Therefore, we propose our Pattern 4.
Pattern 4 (Positioning of Words). Let  $\mathcal{V}$  denote a set of vocabulary. Let  $\mathcal{N} \subset \mathcal{V}$  denote the subset of all nouns in the vocabulary. Consider a text prompt containing noun  $A \in \mathcal{N}$  representing the object to be generated. Furthermore, assume there exist preceding word  $B \in \mathcal{V}$  and succeeding word  $C \in \mathcal{V}$  surrounding noun  $A$ . There exists a condition-dependent behavior regarding the replacement of words  $B$  and  $C$  with another noun  $D \in \mathcal{N}$ :
$$
\left\{ \begin{array}{l l} \exists A, D \in \mathcal {N}, \quad \exists B, C \in \mathcal {V}, \quad P (B \to D) \xrightarrow {\text {g e n e r a t e}} D \wedge P (C \to D) \xrightarrow {\text {g e n e r a t e}} A; \\ \exists A, D \in \mathcal {N}, \quad \exists B, C \in \mathcal {V}, \quad P (B \to D) \xrightarrow {\text {g e n e r a t e}} A \wedge P (C \to D) \xrightarrow {\text {g e n e r a t e}} D. \end{array} \right.
$$
B Cases of Short/Long-Prompt Attacks and Black-box Attacks
B.1 Attack on Long Prompt
In Fig. 18, we demonstrate more cases of long text prompt attacks.
B.2 Attack on Short Prompt
In Fig. 19, we demonstrate more cases of long text prompt attacks.
B.3 Black-box Attack
In Fig. 20, and Fig. 21, we demonstrate black box attacks targeting Midjourney and DALL-E2, respectively.
In our experiments, we conduct comprehensive analyses of both long and short prompts. Furthermore, we conduct ablation studies specifically on long prompts, focusing on three key aspects. Firstly, we evaluate our attack method with different numbers of search steps  $T$ . Secondly, we investigate the influence of our constraints, including fluency and semantic similarity as measured by BERTScore. Lastly, we attack different samplers, including DDIM [31] and DPM-Solver [17].
C.1 Experimental Setting.
Attack hyperparameters. The number of search iterations  $T$  is set to 100. This value determines the number of iterations in the search stage, during which we aim to find the most effective attack prompts. The number of attack candidates  $N$  is set to 100. This parameter specifies the number of
A loose, full-length abaya worn by some Muslim women, primarily in the Arabian Peninsula, as an outer covering to ensure modesty.
a knee, full adult or ababa embroidery by some German women, primarily in the netherlands peninsula as an outer covering to ensure modesty.
A large, mounted cannon that fires heavy projectiles, historically used in warfare and for other purposes such as ceremonial salutes."
A large, hand cannon that fires heavy projectiles, historically used in warfare and for other purposes such as ceremonial salutes."
This small brussels griffon breed has a distinctive, wiry coat, a compact, muscular body, and a friendly, affectionate temperament. They were originally bred for hunting small game and as companion animals in Belgium and are known for their intelligence, agility, and charm
This ancient brussels griffon penguins has a distinctive, wiry appearance, a strong, muscular body, and a good, affectionate temperament. they were originally bred for hunting small game and for pack animals in Belgium and are known for their intelligence, agility, and speed.
An ancient abacus tool, consisting of a frame with rows of beads or other movable counters, used for mathematical operations and counting.
A type of footwear with a thick, rigid sole, often made of wood, and an upper made of leather or another material. Clogs can be open or closed, and are commonly associated with Dutch and Scandinavian cultures.
An optical abacus tool, consisting of a wheel with rows of tiles and other movable symbols, used for wheel operations and counting.
A type of camouflage with a long, rigid beak, often made of wood, and an upper made of leather or another material. Clogs can be open or closed, and are commonly associated with Dutch and Scandinavian cultures.
"A loose, sleeveless cloak that is worn over the shoulders and fastened at the neck, often used for warmth or as a decorative accessory."
"A loose, cat cloak that is worn over the shoulders and fastened at the neck, often used for warmth or as a decorative accessory."
A cornel musical instrument similar to a trumpet but with a more compact shape and mellower tone.
A cornet keyboard is similar to a trumpet but with a more compact shape and mellower tone.
A computer mouse input device used to control a computer's cursor, typically featuring buttons and a scroll wheel for selecting and navigating on-screen elements.
"A large, upholstered couch of furniture designed for seating multiple people, often found in living rooms or other communal spaces."
A computer mouse puzzle device used to control a computer's cursor, typically featuring buttons and a scroll wheel for selecting and navigating on-screen elements
"A large, upholstered couch of weapons designed for seating multiple people, often found in living rooms or other communal spaces."
These large, conch snails have a distinctive, spiral shell and a strong, muscular foot. They are found in tropical waters and are known for their ornate shells and edible flesh."
"These duck have a round body, a flat bill, and webbed feet, adapted for swimming and dabbling for food. They come in various colors and patterns, with some species having bright, iridescent plumage."
These large, conch sharks have a distinctive, spiral shell and a strong, muscular foot. They are found in tropical waters and are known for their ornate shells and edible flesh."
These duck shark a round body, a flat bill, and webbed feet, adapted for swimming and dabbling for food. They come in various colors and patterns, with some species having bright, iridescent plumage."
A rich, creamy, dairy-based eggnog traditionally made of milk, cream, sugar, and eggs, often spiked with alcohol and served during the winter holidays.
A doll, creamy, dairy-based eggnog traditionally made of milk, cream, sugar, and eggs, often spiked with alcohol and served during the winter holidays.
Electric ray is flattened, disc-shaped fish with smooth skin and a pair of kidney-shaped electric organs on each side of their body, used for hunting prey and self-defense.
Figure 18: To the left of the arrow is the clean long text prompt (highlighted by green) and its corresponding image, to the right of the arrow is the generated attack prompt (highlighted by red) and its corresponding image. (Section B.1 Attack on Long Text Prompt)
Electric rays flattened, disc-shaped senses with smooth skin and a pair of kidney-shaped electric organs on each side of their body, used for hunting prey and self-defense."
A timekeeping hourglass consisting of two glass chambers connected by a narrow passage, with sand flowing through the passage at a constant rate to measure a specific time interval.
A mean like hourglass consists of two renfrepanels separated by a central passage, with sunlight flowing through the cave in a constant rate to enter a specific time frame."
A wild pig species native to Africa, known for its large, curved tusks and distinctive facial features, including warts on their faces. Warthogs are primarily grazers and live in savannas and grasslands.
A wild rhinospecies native to Africa, known for its large, curved tusks and distinctive facial features, including warts on their faces. Warthogs are primarily grazers and live in savannas and grasslands.
Figure 19: To the left of the arrow is the clean short text prompt (highlighted by green) and its corresponding image, to the right of the arrow is the generated attack prompt (highlighted by red) and its corresponding image. (Section B.2 Attack on Short Text Prompt)
candidate attack prompts considered in the attack stage, allowing for a diverse range of potential attack prompts to be explored. The learning rate  $\eta$  for the matrix  $\omega$  is set to 0.3. The margin  $\kappa$  in the margin loss is set to 30.
Text prompts. Our experiments consider the 1,000 classes from ImageNet-1K [3], which serves as the basis for generating images. To explore the impact of prompt length, we consider both short and long prompts. For clean short prompts, we employ a standardized template: "A photo of [CLASS_NAME]". Clean long prompts, on the other hand, are generated using ChatGPT 4 [24], with a prompt length restriction of 77 tokens to align with the upper limit of the CLIP [25] word embedder.
Evaluation metrics. To evaluate the effectiveness of our attack method, we generate attack prompts from the clean prompts. We focus on three key metrics: success rate, Fréchet inception distance [11] (FID), Inception Score (IS), and text similarity (TS). Subsequently, 50,000 images are generated using the attack prompts, ensuring a representative sample of 50 images per class. The success rate is determined by dividing the number of successful attacks by the total of 1,000 classes. FID and IS are computed by comparing the generated images to the ImageNet-1K validation set with (torch-fidelity)[22]. TS is calculated by embedding the attack prompts and clean prompts using the CLIP [25] word embedder, respectively. Subsequently, the cosine similarity between the embeddings is computed to quantify the text similarity.
A photo of a barn spider and a raven
A photo of a black swan and a turban
A photo of a rooster and a minion
A photo of a sea lion and a spaniel
A photo of a stingray and a vulture
A photo of a eel and a rook flew
A photo of an armadillo and a rhino
A photo of a wolf spider and a frog
A type of camouflage with a long, rigid beak, often made of wood, and an upper made of leather or another material. Clogs can be open or closed, and are commonly associated with Dutch and Scandinavian cultures
Figure 20: Black-box attack on Midjourney (Section B.3 Black-box Attack).
A photo of an ant and a buzzard
These duck shark a round body, a flat bill, and webbed feet, adapted for swimming and dabbling for food. They come in various colors and patterns, with some species having bright, iridescent plumage
A wild rhino species native to Africa, known for its large, curved tusks and distinctive facial features, including warts on their faces. Warthogs are primarily grazers and live in savannas and grasslands.
A large, hand cannon that fires heavy projectiles, historically used in warfare and for other purposes such as ceremonial salutes.
A photo of a wolf spider and a giraffe
A cloth, circular yurt statue by nomadic peoples in central Asia, has a wooden frame and a balance or praying covering. include excel weg.
Table 1: Main results of short-prompt and long-prompt attacks.
Prompt	Method	Success (%)	FID (↓)	IS (↑)	TS (↑)
Short	Clean	-	18.51	101.33±1.80	1.00
Random	79.2	29.21	66.71±0.87	0.69
ATM (Ours)	91.1	30.09	65.98±1.10	0.72
Long	Clean	-	17.95	103.59±1.68	1.00
Random	41.4	24.16	91.33±1.58	0.94
ATM (Ours)	81.2	29.65	66.09±1.83	0.84
A photo of a balloon and a brakes spear
A type of camouflage with a long, rigid beak, often made of wood, and an upper made of leather or another material. Clogs can be open or closed, and are commonly associated with Dutch and Scandinavian cultures.
Figure 21: Black-box attack on DALL·E2 (Section B.3 Black-box Attack).
A photo of a wolf spider and a frog
A type of cork- and double air aircraft, also known as a chrigible, (ized by its elongated shape, powered by engines, and steered by a rudder. airships typically consist of a rigid plane covered with an envelope filled with oxygen, such as helium or hydrogen.
A cloth, circular yurt statue by nomadic peoples in central Asia, has a wooden frame and a balance or praying covering.
A mean like hourglass consists of two renfrepanels separated by a central passage, with sunlight flowing through the cave in a constant rate to enter a specific time frame.
A photo of an armadillo and a whale
A large, thin artichoke crab with long, fleshy, leaf-, scales, often oil or steamed and eaten by removing the thin and wiping off the body surface with the teeth.
C.2 Main Results
Table 1 reports our main results, including short-prompt and long-prompt attacks. Compares to long text prompts, short text prompts comprise only a small number of tokens. This leads to a relatively fragile structure that is extremely vulnerable to slight disturbance. Therefore, random attacks can reach an impressive success rate of  $79.2\%$  targeting short prompts but a low success rate of  $41.4\%$  targeting the long prompts. In the contrast, our algorithm demonstrates its true potential, reaching an impressive success rate of  $91.1\%$  and  $81.2\%$  targeting short and long prompts, respectively.
As a further evidence of the effectiveness of our algorithm, it's worth noting the text similarity (TS) metrics between the random attacks and our algorithm's outputs. For short-prompt attack, the values stand at 0.69 and 0.72, respectively, illustrating that the semantic information of short texts, while easy to disrupt, can be manipulated by a well-designed algorithm with fluency and semantic similarity constraints. Our attacks preserve more similarity with the clean prompts. For long-prompt attacks, the TS score of random attacks (0.94) is higher compared to our attacks (0.84). One possible reason is that random attacks tend to make only minimal modifications as the length of the prompt increases. This limited modification can explain the significantly lower success rate of random attacks on longer prompts.
From the perspective of evaluating the quantitative metrics of generated images. For short and long texts, images generated from the clean text have the lowest FID (18.51 and 17.95) and the highest IS  $(101.33\pm 1.80$  and  $103.59\pm 1.68)$ . As the attack success rate rises, FID shows an upward trend. Examining this situation from the perspective of FID, a metric that gauges the distance between the distribution of generated images and the original data set. As the attack becomes more successful, the image set generated by the attack prompt tends to deviate substantially from the distribution of the original data set. This divergence consequently escalates the FID score, indicating a larger distance between the original and generated distributions. On the other hand, the attack objective of our algorithm, namely the margin loss, is used to reduce the classifier's confidence in the true class  $y$  and improve its confidence in the class with the largest confidence excluding  $y$ , until a margin  $\kappa$  is reached. Consequently, generating low-quality images is also an objective of our attack. These images contribute to the increase in the FID index and the decrease in the IS score.
Table 2: Results of the ablation study on the number of steps in attack prompt search.
#Steps	Success (%)	FID (↓)	IS (↑)	TS (↑)
50	68.7	34.00	93.94±1.84	0.97
100	81.2	29.65	66.09±1.83	0.84
150	67.2	45.23	58.51±0.79	0.82
Table 3: Results of the ablation study on the constraints
Fluency	BERTScore	Success (%)	FID (↓)	IS (↑)	TS (↑)
X	X	91.3	39.14	47.21±1.25	0.37
✓	X	81.7	29.37	64.93±1.57	0.79
X	✓	89.8	39.93	46.94±0.99	0.51
✓	✓	81.2	29.65	66.09±1.83	0.84
C.3 Different Search Steps
Table 2 presents the results of using different numbers of steps  $T$  in the search stage. For the  $T = 50$  step configuration, the success rate is  $68.7\%$ . The FID value is 34.00, with lower values suggesting better image quality. The IS is reported as  $93.94 \pm 1.84$ , with higher values indicating diverse and high-quality images. The TS value is 0.97, representing a high level of text similarity. Moving on to the  $T = 100$  step configuration, the success rate increases to  $81.2\%$ , showing an improvement compared to the previous configuration. The FID value decreases to 29.65, indicating better image quality. The IS is reported as  $66.09 \pm 1.83$ , showing a slight decrease compared to the previous configuration. The TS value is 0.84, suggesting a slight decrease in text similarity. In the  $T = 150$  step configuration, the success rate decreases to  $67.2\%$ , slightly lower than the initial configuration. The FID value increases to 45.23, suggesting a decrease in image quality. The IS is reported as  $58.51 \pm 0.79$ , indicating a decrease in the diversity and quality of generated images. The TS value remains relatively stable at 0.82.
When using  $T = 50$ , the attack prompt fails to fit well and exhibits a higher text similarity with the clean prompt. Although the generated images at this stage still maintain good quality and closely resemble those generated by the clean prompt, the success rate of the attack is very low. On the other hand, when  $T = 150$ , overfitting occurs, resulting in a decrease in text similarity and image quality due to the overfitted attack prompt. Consequently, the success rate of the attack also decreases. Overall, the configuration of  $T = 100$  proves to be appropriate.
C.4 The Impact of Constraints
Table 3 examines the impact of the fluency and semantic similarity (BERTScore) constraints. When no constraints are applied, the attack success rate is notably high at  $91.3\%$ . However, this absence of constraints results in a lower text similarity (TS) score of 0.37, indicating a decreased resemblance to clean text and a decrease in image quality. By introducing fluency constraints alone, the attack success rate decreases to  $81.7\%$  but increases the text similarity to 0.79. Furthermore, incorporating semantic similarity constraints independently also leads to a slight reduction in success rate to  $89.8\%$ , but only marginally improves the text similarity to 0.51. The introduction of constraints, particularly fluency constraints, leads to an increase in text similarity. The fluency constraint takes into account the preceding tokens of each token, enabling the integration of contextual information for better enhancement of text similarity. On the other hand, BERTScore considers a weighted sum, focusing more on the similarity between individual tokens without preserving the interrelation between context. In other words, the word order may undergo changes as a result and leads to a low text similarity. Certainly, this outcome was expected, as BERTScore itself prioritizes the semantic consistency between two prompts, while the order of context may not necessarily impact semantics. This further highlights the importance of employing both constraints simultaneously. When both constraints are utilized together, the text similarity is further enhanced to 0.84. Meanwhile, the success rate of the
Table 4: Results of the ablation study on the samplers
Sampler	Success (%)	FID (↓)	IS (↑)	TS (↑)
DDIM [31]	81.2	29.65	66.09±1.83	0.84
DPM-Solver [17]	76.5	27.23	81.31±2.09	0.88
Table 5: Classification Accuracy of Different Classifiers on both clean short and long text prompts.
Classifiers	Short Text Prompts	Long Text Prompts
ViT-B/16(CLIP)	82.0%	78.8%
ViT-B/32(CLIP)	80.8%	74.1%
ViT-L/14(CLIP)	84.0%	80.8%
ResNet50(CLIP)	77.2%	69.9%
ResNet50 [10]	10.7%	10.8%
Swin-B [16]	79.8%	76.2%
ViT-B/16 [6]	82.0%	78.9%
attack  $(81.2\%)$  is comparable to that achieved when employing only the fluency constraint, while the text similarity surpasses that obtained through the independent usage of the two constraints.
C.5 Different Samplers
Table 4 illustrates the effectiveness of our attack method in successfully targeting both DDIM and the stronger DPM-Solver. For the DDIM sampler, our attack method achieves a success rate of  $81.2\%$ , indicating its ability to generate successful attack prompts. Similarly, our attack method demonstrates promising results when applied to the DPM-Solver sampler. With a success rate of  $76.5\%$ , it effectively generates attack prompts. The TS scores of 0.84 and 0.88, respectively, indicate a reasonable level of text similarity between the attack prompts and clean prompts. These outcomes demonstrate the transferability of our attack method, showcasing its effectiveness against both DDIM and the more potent DPM-Solver sampler.
C.6 Ensemble of Multiple Classifiers