Asymmetric Bias in Text-to-Image Generation with Adversarial Attacks (2024)

Abstract: The widespread use of Text-to-Image (T2I) models in content generation requires careful examination of their safety, including their robustness to adversarial attacks. Despite extensive research on adversarial attacks, the reasons for their effectiveness remain underexplored. This paper presents an empirical study on adversarial attacks against T2I models, focusing on analyzing factors associated with attack success rates (ASR). We introduce a new attack objective entity swapping using adversarial suffixes and two gradient-based attack algorithms. Human and automatic evaluations reveal the asymmetric nature of ASRs on entity swap: for example, it is easier to replace “human” with “robot” in the prompt “a human dancing in the rain.” with an adversarial suffix, but the reverse replacement is significantly harder. We further propose probing metrics to establish indicative signals from the model’s beliefs to the adversarial ASR. We identify conditions that result in a success probability of 60% for adversarial attacks and others where this likelihood drops below 5%. 1

text: Findings of the Association for Computational Linguistics ACL 2024, pages 5779–5796 August 11-16, 2024 ©2024 Association for Computational Linguistics
“robot” in the prompt “a human dancing in the rain.” with an adversarial suffix, but the reverse replacement is significantly harder. We further propose probing metrics to establish indicative signals from the model’s beliefs to the adversarial ASR. We identify conditions that result in a success probability of 60% for adversarial attacks and others where this likelihood drops below 5%. 1

heading: 1 Introduction
text: The capabilities of Text-to-Image (T2I) generation models, such as DALL-E 2 (Ramesh et al., 2022), DALL-E 3 (Betker et al., 2023), Imagen (Saharia et al., 2022) and Stable Diffusion (Rombach et al., 2022), have improved drastically and reached commercial viability. As with any consumer-facing AI solution, the safety and robustness of these models remain pressing concerns that require scrutiny.
The majority of research related to T2I safety is associated with the generation of Not-Safe-ForWork (NSFW) images with violence or nudity (Qu et al., 2023; Rando et al., 2022; Tsai et al., 2023). To counter this, pre-filters that check for NSFW texts and post-filters that check for NSFW images
1The code and data are available at https://github. com/Patchwork53/AsymmetricAttack
are used (Safety-checker, 2022). However, these filters are not infallible (Rando et al., 2022), and research into bypassing them, termed ‘jailbreaking’ is advancing (Yang et al., 2023b,a; Noever and Noever, 2021; Fort, 2023; Galindo and Faria; Maus et al., 2023; Zhuang et al., 2023). These attacks typically view the creation of NSFW-triggering adversarial prompts as a singular challenge, without sufficiently investigating the reasons behind these attacks’ effectiveness.
On the other hand, explainability studies have examined the capabilities and shortcomings of textto-image (T2I) models. They show that T2I models often generate content without understanding the composition (Kong et al., 2023; West et al., 2023), and reveal compositional distractors (Hsieh et al., 2023). We identified a specific bias of T2I models linked to adversarial attack success rates, bridging the gap between attack and explainability research. We demonstrate the asymmetric bias of the T2I models by conducting adversarial attacks in a novel entity-swapping scenario, in contrast to the existing setup of removing objects (Zhuang et al., 2023) or inducing NSFW content (Yang et al., 2023b,a). This setup enables us to investigate the attack success rate in a cyclical setting.
To study the underlying reasons for the success of adversarial attacks, the attack must be powerful and have a high success rate. This would allow us to ensure that cases with low success rates arise due to the model’s internal biases, not simply as a result of the algorithm’s shortcomings. We propose two optimizations of existing gradient-based attacks (Shin et al., 2020; Zou et al., 2023) using efficient search algorithms to find adversarial suffix tokens against Stable Diffusion. This approach is based on the observation that existing algorithms for LLM attacks are unnecessarily conservative in generating adversarial perturbations and struggle to efficiently navigate the larger vocabulary size of the T2I text encoder.
1 5779
Our novel setup and efficient adversarial attack have allowed us to observe an asymmetric attack success rate associated with entity swap. Initially, we hypothesized that long-tail prompts with high perplexity would be more vulnerable to attacks. Surprisingly, we found no strong correlation between the Attack Success Rate (ASR) and the perplexity of the prompt. However, with our proposed measure that evaluates the internal beliefs of CLIP models, we detected indicative signals for ASR, which help identify examples or prompts that are more susceptible to being attacked. Our contributions can be summarized as follows.
1. We introduce a new attack objective: replacing entities of the prompt using an adversarial suffix. This allows us to study the relation between adversarial attacks and the underlying biases of the model (Figure 1a).
2. We apply an existing gradient-based attack algorithm to execute entity-swap attacks and propose improvements that take advantage of the bag-of-words nature of T2I models. This powerful attack method reveals a clear distinction in the ASR when two entities are swapped in opposite directions, indicating an asymmetry in adversarial attacks (Figure 1b).
3. We propose a new metric tied to the asymmetric bias of T2I models. This helps us identify vulnerable preconditions and estimate ASR without performing an attack (Figure 1c).

heading: 2 Related Works
text: Adversarial Attacks Adversarial attacks, which perturb inputs to cause models to behave unpredictably, have been a long-studied area in the field
of adversarial robustness (Szegedy et al., 2013; Shafahi et al., 2018; Shayegani et al., 2023). Previous studies on adversarial attacks focused on discriminative models involving convolutional neural networks (Athalye et al., 2018; Hendrycks and Dietterich, 2018), while recent work has shifted towards examining generative models such as large language models (LLMs) (Shin et al., 2020; Zou et al., 2023; Liu et al., 2023c; Mo et al., 2023; Cao et al., 2023), Vision Language models (VLMs) (Dong et al., 2023; Khare et al., 2023; Shayegani et al., 2024), and Text-to-Image (T2I) models.
Attacks on T2I Models Zhuang et al. (2023) were among the first to demonstrate that a mere five-character perturbation could significantly alter the generated images. Tsai et al. (2023) and SneakyPrompt (Yang et al., 2023b) proposed adversarial attacks using genetic algorithms and reinforcement learning algorithms to perturb safe prompts to generate NSFW content. VLAttack (Yin et al., 2023), MMA-Diffusion (Yang et al., 2023a), and INSTRUCTTA (Wang et al., 2023) demonstrated that cross-modality attacks can achieve higher success rates than text-only attacks. For defense, Zhang et al. (2023) proposed Adversarial Prompt Tuning to enhance the adversarial robustness of VLMs. However, to the best of our knowledge, no comparable defense against nonNSFW attacks exists for T2I models.
Vulnerability Analysis Previous studies (Ilyas et al., 2019; Shafahi et al., 2018; Brown et al., 2017) have explored the reasons for the vulnerability of neural networks to adversarial attacks, especially in image classification. Ilyas et al. (2019) suggested that adversarial examples stem from non-robust fea-
2
tures in models’ representations, which are highly predictive yet imperceptible to humans. Subhash et al. (2023) suggested that adversarial attacks on LLMs may act like optimized embedding vectors, targeting semantic regions that encode undesirable behaviors during the generation process.
Distinct from previous research, our study analyzes factors in the model’s beliefs linked to attack success rates. Unlike prior work focusing on untargeted attacks to trigger NSFW image generations, we introduce a unique entity-swapping attack setup and develop a discrete token-searching algorithm for targeted attacks, identifying asymmetric biases in success rates due to the model’s internal bias. Our experiments emphasize the relationship between prompt distributions, model biases, and attack success rates.

heading: 3 Entity Swapping Attack
text: This section describes the proposed setup of the entity-swapping attack and the corresponding evaluation metric. Designing a new attack scenario may be straightforward, but developing a suitable measure is not trivial. Towards this end, we propose two efficient discrete token search algorithms for the attack, resulting in improved success rates in entity-swapping attacks.

heading: 3.1 Stable Diffusion
text: We study entity-swapping attacks using Stable Diffusion (Rombach et al., 2022), an open-source 2 T2I model based on a denoising diffusion probabilistic model with a U-Net architecture. It uses cross-attention and CLIP (Radford et al., 2021) for text-image alignment and a variational autoencoder (Kingma and Welling, 2013) for latent space encoding. The model’s dependence on CLIP text embeddings increases its vulnerability to adversarial attacks. See Appendix E for more details.

heading: 3.2 Entity Swapping Dataset
text: We first constructed datasets with the following key properties to study model bias through entityswapping attacks.
1. Each data point should be a pair of sentences - input and target - and T2I models should be able to generate both reliably.
2. The input and target sentences should differ by exactly one noun (i.e., an entity).
2Licensed under CreativeML Open RAIL++-M License for intended for research purposes only.
3. The input and target sentences must be visually distinct.
As an example, the pair (“a person in a park.”, “a man in a park.”) satisfies requirements 1 and 2 but not 3. As our setup for entity-swapping attacks is targeted, namely adversarial attacks need to swap the entities in the images without affecting other parts compared to other attacks that aim to either generate NSFW images or remove objects, we created two datasets to study the effects of adversarial attacks. We manually constructed a small high-quality dataset HQ-Pairs and a larger-scale set derived from an existing dataset MS-COCO.
HQ-Pairs For the first dataset, we manually crafted 100 pairs for entity-swapping that satisfy all the requirements. We refer to this first dataset as HQ-Pairs (High Quality).
COCO-Pairs To ensure that our results were not due to selective data selection, we generated a second dataset of 1,000 pairs deterministically from the test split captions of MS-COCO (Lin et al., 2014). We refer to this dataset as COCO-Pairs 3. Since COCO-Pairs is automatically generated, we attempted to ensure that each data pair satisfies all three requirements. However, generating sentence pairs through stable diffusion and verifying them as visually distinct automatically is not always reliable. We observed some visually non-distinct pairs, such as (“Herd of zebras ...”, “Images of zebras ...”) within COCO-Pairs despite automatic checks and filtering. See Appendix A for details.

heading: 3.3 Proposed Attack
text: We examine how the underlying data distribution of prompts influences the success rate of entityswapping attacks on T2I models. Our approach is straightforward: rather than manipulating T2I to produce NSFW images or completely removing an object, we aim to replace an object in the image with another targeted one. This approach also allows us to explore the feasibility of reverse attacks by inserting adversarial tokens. Examples of our attack setup can be found in Figure 2.
The CLIP text-encoder transforms prompt tokens x1:n into n hidden states with dimension D. Let the operationH represent the combined process of encoding tokens x1:n and reshaping the hidden states into a vector of length n×D.
3The code to reproduce COCO-Pairs is provided in our codebase.
3
H(x1:n) = Flatten(CLIP(x1:n)) (1) Our attack targets the CLIP embedding space and aims to maximize a score function that measures the shift from the input token embeddings H(xT1:n) towards the target token embeddings H(xS1:n) using cosine similarity:
S(x1:n) = wt × cos(H(xT1:n),H(x1:n))− ws × cos(H(xS1:n),H(x1:n)) (2)
Optimizing S is challenging due to the discrete token set and the exponential search space (k|V | for k suffix tokens), making simple greedy search intractable. Current solutions based on HotFlip (Ebrahimi et al., 2017) and concurrent work applied to Stable Diffusion (Yang et al., 2023a), take gradients w.r.t. one-hot token vectors and replace tokens for all positions in the suffix simultaneously. The linearized approximation of replacing the ith token, xi, is computed by evaluating the following gradients:
∇exiL(x1:n) ∈ R |V |, L(x1:n) = −S(x1:n)
(3) where exi denotes the one-hot vector representing the current value of the ith token.

heading: 3.4 Proposed Optimization Algorithms
text: Based on existing gradient-based methods (Zou et al., 2023; Shin et al., 2020), we propose two efficient algorithms to find adversarial suffix tokens against Stable Diffusion.
Single Token Perturbation This is a straightforward modification of the Greedy Coordinate Gradient algorithm (Zou et al., 2023) using our loss function defined in Eqn. 3. At each optimization step, our algorithm selects k tokens with the highest negative loss as replacement candidates, χi, for each adversarial suffix position i. It then creates B new prompts by randomly replacing one token from the candidates. Each prompt in B differs from the initial prompt by only one token. The element of B with the highest S is then assigned to x1:n. We repeat this process T times.
Multiple Token Perturbation Unlike the LLMs targeted by Zou et al. (2023), CLIP models operate more like bag-of-words (Yuksekgonul et al., 2022) without capturing semantic and syntactical relations between words. Furthermore, Genetic Algorithms (Sivanandam et al., 2008) have proved effective on Stable Diffusion (Zhuang et al., 2023; Yang et al., 2023b) for generating adversarial attacks. Inspired by this apparent weakness in CLIP models, we hypothesized that replacing multiple
4
tokens simultaneously could improve the convergence speed.
In detail, the algorithm selects k tokens and creates B new prompts by randomly replacing multiple token positions. Drawing inspiration from the classic exploration versus exploitation strategy in reinforcement learning (Sutton and Barto, 2018), we initially replace all tokens and then gradually decrease the replacement rate to 25%. Figure 2 illustrates some adversarial suffixes generated using this algorithm. Details of both algorithms are provided in the Appendix B.
Token Restrictions For finer control over token search, we can limit the adversarial suffix to a set of tokens A. By setting the gradients of the V −A tokens to infinity before the Top-k operation, we ensure only A tokens are chosen. This method allows us to mimic QFAttack (Zhuang et al., 2023), as shown in Figure 3, or generate undetectable attacks by excluding target synonyms in the attack suffix.

heading: 3.5 Proposed Attack Evaluation
text: To assess the success of a targeted entity-swapping attack, we use a classifier to verify if the generated image matches the input or target prompt. Given a tuple (input text, target text, generated image), we define a classifier C as follows:
C(input text, target text, generated image)
=    +1 if image matches target text −1 if image matches input text 0 otherwise. (4)
When trying to change “A backpack in a forest” to “A cabin in a forest”, we noticed that some of the generated images depicted “People in a forest”
or “A cabin and a backpack in a forest” instead. We define such cases as class 0. Class +1 alone indicates a successful attack, but this three-class framework enables a more comprehensive comparison between human judgments and our proposed classifiers.
Attack Success Rate (ASR) We define an adversarial suffix as successful if the target text is a suitable caption for the majority of images generated by an attack prompt using a T2I model. For example, if we generate 5 images with an appended adversarial suffix prompt “A backpack in a forest. titanic tycoon cottages caleb dojo ”, we will consider the adversarial suffix successful if 3 or more images match the target prompt “A cabin in a forest”.
Human Evaluations/Labels We gather evaluations from three human evaluators 4 for 200 random samples by presenting them a WebUI (Appendix H) with the generated image and two checkboxes for input text and target text. They are instructed to select texts that match the image and can select one, both, or neither, i.e. into three classes as established in Eqn. 4. The Gwet-AC1 metric (Gwet, 2014) of the three evaluators is 0.765 and the pairwise Cohen’s Kappa κ metrics (Cohen, 1960) are 0.659, 0.736, and 0.779, indicating a high degree of agreement. We consider the majority vote among evaluators as ground truth.
Choice of the Classifier We generate multiple attack suffixes for each input-target pair to determine attack success rates. Due to the large volume of images, we employ human evaluators for a subset and VLM-based classifiers for the full set evaluation. We test InstructBLIP (Liu et al., 2023a), LLaVA1.5 (Liu et al., 2023b) and CLIP (Radford et al., 2021), and compare their performance with human labels.
For InstructBLIP and LLaVA-1.5, we use the prompt ‘Does the image match the caption [PROMPT]? Yes or No?’. For CLIP models, an image is classified as +1 if its target text similarity is above 1− γ and its input text similarity is below γ and −1 for the reverse case. All other cases are classified as 0. Table 1 shows the agreement of different automatic classifiers with ground truths from
4Our evaluations were conducted by three non-author, native English-speaking volunteers who generously offered their time without compensation. We sincerely thank them for their commitment and good faith effort in labeling.
5
our human evaluators. We use the optimal threshold γ (γCLIP = 0.0034 and γCLIP−336 = 0.0341) that maximizes the F1 score. Since InstructBLIP shows the best alignment with human evaluation, we use InstructBLIP as our sole classifier in subsequent sections.

heading: 4 Experiments and Results
text: This section presents the experimental details and results of adversarial attacks for entity-swapping, involving the insertion of adversarial suffixes.

heading: 4.1 Experimental Setups
text: We evaluate Stable Diffusion v2-1-base on the HQPairs dataset of 100 input-target pairs to compare the effectiveness of Single and Multiple Token Perturbation. We run each algorithm 10 times per pair with T = 100 steps with k = 5 and B = 512, which yields 10 adversarial attacks per pair, and we generate 5 images per attack. The two algorithms are evaluated against each other on
100 × 10 × 5 = 5000 generated images. We set wt = ws = 1 in Eqn. 2 for the experiments. Afterward, we evaluate COCO-Pairs (1000 pairs) using the Multiple Token Perturbation algorithm to establish the asymmetric bias phenomenon with the same hyperparameters. We used a single Nvidia RTX 4090 GPU for all experiments, including attack, image generation, and automated evaluation, totaling around 500 GPU hours.

heading: 4.2 Overall Attack Results
text: Using the same hyperparameters and compute budget, our Multiple Token Perturbation algorithm outperforms the Single Token Perturbation ( ASR 26.4% vs. 24.4% for 1000 attacks). Zou et al. (2023) showed that Single Token Perturbation was an effective adversarial suffix-finding strategy for LLMs. However, the CLIP text is relatively lightweight compared to LLMs and behaves more like a bag-of-words model (Yuksekgonul et al., 2022). CLIP also has a larger vocabulary compared to LLMs ( 50K vs. 32K) which leads to a larger unrestricted search space (∼ 1024 vs. ∼ 1023 for 5 token suffixes). We find that updating multiple tokens at each time step leads to faster convergence, likely because CLIP demonstrates a reduced emphasis on the semantic and syntactical relationships between tokens. Our findings corroborate the effectiveness of the Genetic Algorithm in Zhuang et al. (2023), which resembles multiple token perturbations but operates in an untargeted setting without a gradient-based algorithm. We employ Multiple Token Perturbation for all subsequent experiments.

heading: 4.3 Forward and Backward Attack Results
text: One of our key findings is the strong asymmetry of adversarial attack success rate, as illustrated in Fig-
6
ure 4. For instance, attacks from ‘A swan swimming in a lake.’ to ‘A horse swimming in a lake.’ failed in all ten attempts, whereas the reverse direction achieved an ASR of 0.9. In other cases, the forward and backward ASRs aren’t inversely proportional. For example, both directions between ‘A man reading a book in a library.’ and ‘A woman reading a book in a library.’ have moderate ASRs of 0.7 and 0.5, respectively, while pairs like (‘A dragon and a treasure chest.’, ‘A knight and a treasure chest.’) fail in both directions. Inspired by these asymmetric observations, we conducted further experiments to analyze the relationship between prompt distribution and attack success rate.

heading: 5 Asymmetric ASR Analysis
text: This section discusses our experiments to analyze the asymmetric ASR observed in Section 4.3. We aim to investigate the model’s internal beliefs that may lead to these distinct attack success rate (ASR) differences from opposite directions. We propose three potential factors for this asymmetry: the difficulty of generating the target text (BSR, Eqn. 5), the naturalness of the target text relative to the input text (∆1, Eqn. 6), and the difference in distance from the target text to the baseline compared to that from the input text (∆2, Eqn. 7).

heading: 5.1 Probe Metrics
text: We initially speculated that ASR might be related to the difficulty in generating the target prompt, leading us to evaluate the Base Success Rate (BSR) of target generation.
BSR = Successful Generations Generation Attempts
(5)
BSR assesses the T2I model’s ability to generate an image that matches the input prompt without any adversarial suffixes. Stable Diffusion is often unable to generate novel compositions not present in its training data (West et al., 2023) and struggles with generating co-hyponym entities in the same scene (Tang et al., 2022). We find that even simple scenes such as “A dragon guarding a treasure.” are inconsistently produced (See Appendix F for examples). Therefore, if the T2I models struggle with the target alone, adversarial attacks aimed at generating them are likely to be even more challenging.
We also speculated that the difference in Perplexity ∆1, measuring how natural or plausible
a prompt is, might be associated with asymmetric ASR. For example, “A swan swimming in a lake” is a more natural scene than “A horse swimming in a lake”. Using text-davinci-003 by OpenAI (Brown et al., 2020), we calculate the perplexity difference
∆1(x T 1:n, x S 1:n) = PPL(x T 1:n)− PPL(xS1:n). (6)
where PPL(x1:n) = e− 1 n ∑n i=1 logP (xi|x1:i−1) is the perplexity for the sequence x1:n.
Furthermore, we introduce a new metric termed Baseline Distance Difference, denoted as ∆2. Figure 5 shows that T2I models have inherent biases towards certain objects. We denote this phenomenon as the baseline - answering what would Stable Diffusion generate if prompted with “A swimming in a lake”. Intuitively, targets closer to this baseline should be easier to generate.
∆2(x T 1:n, x S 1:n) = cos(H(xT1:n),H(xB1:n)) −cos(H(xS1:n),H(xB1:n)). (7)

heading: 5.2 Results
text: We generated 64 images for each sentence in HQPairs and COCO-Pairs. We counted the number of successful generations to determine the BSR as defined in Eqn. 5.
On the HQ-Pairs dataset, we find that Perplexity Difference ∆1 has a negligible correlation with ASR (Pearson r = 0.05 and Spearman ρ = −0.06). This is counterintuitive because we expected that a target with lower perplexity compared to the input text would be easier to generate through an adversarial attack. We also observed that ASR
7
has a weak positive correlation with BSR (Pearson r = 0.28 and Spearman ρ = 0.38) and a moderate correlation with ∆2 (Pearson r = −0.39 and Spearman ρ = −0.46. See Figure 6a). In particular, Figure 6b shows that the mean ASR is 0.40 when ∆2 is negative, while it drops to just 0.12 when ∆2 is positive. Thus, ∆2 allows us to estimate, to some extent, the probability of a successful adversarial attack. We present more correlation plots of ASR with Perplexity Difference and BSR in Appendix F.

heading: 5.3 Predictor for Successful Attack
text: Considering the observed correlations of BSR (of the target text) and ∆2 with attack success rates, this section explores whether the combination of these two indicators can predict the probability of a successful entity-swapping attack.
Table 2 shows that our probe metric acts as a reliable predictor of attack success: when BSR (of the target text) is high and ∆2 is negative for a
given input-target text pair, adversarial attacks have a 60% chance of success on the HQ-Pairs dataset, compared to only 5% when BSR is low and ∆2 is positive. Thus, considering both BSR and ∆2 together enhances the prediction accuracy of an attack’s success likelihood. We further validate our findings on the much larger COCO-Pairs dataset. Although the differences are not as pronounced as those in the HQ-Pairs, due to limitations explained in Section 3.2, we still observe that high BSR and negative ∆2 remain indicative of a higher likelihood of successful adversarial attacks. We also identified factors akin to existing research on general elements associated with attack success rates, like the length of the adversarial suffix. These factors, together with our experimental results, are detailed in Appendix G.

heading: 6 Conclusion
text: This paper presents an empirical study on adversarial attacks targeting text-to-image (T2I) generation models, with a specific focus on Stable Diffusion. We define a new attack objective: entity-swapping, and introduce two gradient-based algorithms to implement the attack. Our research has identified key factors for successful attacks, revealing the asymmetric nature of attack success rates for forward and backward attacks in entity-swapping. Furthermore, we propose probing metrics to associate the asymmetric attack success rate with the asymmetric bias within the T2I model’s internal beliefs, thus establishing a link between a model’s bias and its robustness against adversarial attacks.
8

heading: 7 Limitations
text: Our analysis establishes the asymmetric bias phenomenon for Stable Diffusion but whether all T2I models have such bias is an open question. Closedsource T2I models with different architectures such as Imagen and DALL·E may be immune to the asymmetric bias phenomenon or their creators may have mitigated biases through careful data curation.
One of our key findings is that asymmetric bias is not intuitive. Although humans might consider “fish” to be a more natural option (and likely more abundant in the training data) for “A in an aquarium”, we find that Stable Diffusion is strongly biased towards “turtle” instead. We leave exploring the underlying reason for this non-intuitive bias as future work.
We observed that gradient-based algorithms tend to include the target word in the adversarial suffix. Concurrent works that aim to generate undetectable NSFW attacks use a dictionary to prevent this. Since we target benign words and have different targets for every attack, we could not use a similar approach. We explore explicitly forbidding tokens corresponding to the target word, but the algorithm still finds synonyms or different tokenizations of the target word. Forbidding the target word proved to be a nontrivial and ultimately, we did not consider generating true adversarial attacks to be a central focus of our investigation of model bias. Another technical challenge is the need to compute BSR which involves generating a statistically significant number of images (64 in our experiments) for the same prompt. Finding ways to approximate the BSR is an area for future research.

heading: A Generating COCO-Pairs
text: Starting from 5000 captions, we filter out long captions and use a Named-Entity-Recognition model (Nadeau and Sekine, 2007) to identify the first noun in the sentence and use a Fill-Mask model (Devlin et al., 2018) to replace it with another noun. We use the NLTK (Loper and Bird, 2002) library and several heuristics to prevent synonyms, hyponym-hypernym, and nonvisualizable nouns from being selected. We are left with 2093 (base caption, synthetic caption) pairs, from which we sample 500. This yields 1000 sentence pairs in total by considering both directions.

heading: B Algorithms
text: Algorithm 1 Single Token Perturbation Require: Initial prompt x1:n, modifiable subset I , iterations T , loss L, score S, batch size B, k
1: for t ∈ T do 2: for i ∈ I do 3: χi ← Top-k(−∇xiL(x1:n)) {Compute top-k promising token substitutions} 4: end for 5: for b = 1, . . . , B do 6: x
(b) 1:n ← x1:n {Initialize element of batch}
7: x (b) i ← Uniform(χi), where i← Uniform(I) {Select random replacement token} 8: end for 9: x1:n ← x(b ∗) 1:n , where b
∗ = argmaxb S(x(b)1:n) {Compute best replacement} 10: end for Ensure: Optimized prompt x1:n
Algorithm 2 Multiple Token Perturbation Require: Input: Initial prompt x1:n, modifiable subset I , iterations T , loss L, score S, batch size B, k,
ϵf , ϵs 1: ϵ← ϵs 2: for t ∈ T do 3: for i ∈ I do 4: χi ← Top-k(−∇xiL(x1:n)) {Compute top-k promising token substitutions} 5: end for 6: for b = 1, . . . , B do 7: x
(b) 1:n ← x1:n {Initialize element of batch}
8: for i ∈ I do 9: if P(ϵ) then
10: x (b) i ← Uniform(χi) {Select random replacement token} 11: end if 12: end for 13: end for 14: x1:n ← x(b ∗) 1:n , where b
∗ = argmaxb S(x(b)1:n) {Compute best replacement} 15: ϵ← max(ϵf , ϵs − tT ) {Reduce the replacement probability} 16: end for Ensure: Output: Optimized prompt x1:n
12

heading: C Additional Examples of Asymmetric Bias
text: 13

heading: D Changing the Number of Adversarial Tokens

heading: E T2I Model Basics
text: Stable Diffusion (Rombach et al., 2022) is built on a denoising diffusion probabilistic model (DDPM) (Ho et al., 2020) framework, utilizing a U-Net architecture for its core operations. Key to its text-to-image capabilities is the cross-attention mechanism, which aligns textual inputs with relevant visual features. Specifically, the U-Net attends to image-aligned text embeddings produced by a CLIP (Radford et al., 2021) model. Stable Diffusion also incorporates a Variational Autoencoder (Kingma and Welling, 2013) to efficiently encode images into a latent space, significantly reducing computational requirements while maintaining image quality. Since text embedding generation using a CLIP model is the first stage of the
14
Stable Diffusion pipeline, it is particularly susceptible to adversarial attacks (Galindo and Faria; Zhuang et al., 2023). If an adversary can perturb the text embeddings, later stages in the Stable Diffusion pipeline will reflect the perturbed embeddings.
E.1 Exploiting CLIP’s Embedding Space The CLIP text-encoder maps the textual prompt tokens x1:n, with xiϵ{1, ..., V } where V denotes the vocabulary size, namely, the number of tokens to x1:n, where hi is the hidden state corresponding to the token xi. The U-Net component in Stable Diffusion attends to all h1:n embeddings using cross-attention. x1:n can be flattened into Φ, a one-dimensional vector of shape n × D, where D is the embedding dimension (typically 768 for CLIP and its variants). For simplicity, we refer to Φ as the text embedding of x1:n from here on. LetH represent the combined operation for encoding tokens x1:n and reshaping the hidden output states.
Φ = H(x1:n) = Flatten(CLIP(x1:n)) (8) Since input text and target text can vary in the number of tokens and to allow for an arbitrary number of adversarial tokens, we pad all input and targets to 77 tokens each, the maximum number of tokens supported by CLIP.
E.2 Score Function The cosine similarity metric approximates the effectiveness of appending adversarial tokens at some intermediate optimization step t. Moving away from the input tokens’ embedding and gradually towards the target tokens’ embeddings through finding better adversarial tokens can be thought of as maximizing the following score function, similar to the metric in (Zhuang et al., 2023).
S(x1:n) = wt × cos(H(xT1:n),H(x1:n))− ws × cos(H(xS1:n),H(x1:n))
(9)
Here, wt and ws are weighing scalars and cos denotes the standard cosine similarity metric between two one-dimensional text embeddings. For simplicity, we set wt = ws = 1 for all experiments.
E.3 Optimization over Discrete Tokens The main challenge in optimizing S is that we have to optimize over a discrete set of tokens. Furthermore, since the search space is exponential (k|V | for k suffix tokens), a simple greedy search is intractable. However, we can leverage gradients with respect to the one-hot tokens to find a set of promising candidates for replacement at each token position. We use the negated Score Function as the loss function L(x1:n) = −S(x1:n). Maximizing the score is equivalent to minimizing the loss. Since losses are used for top K token selection, the absolute value of the loss does not matter. We can compute the linearized approximation of replacing the ith token i, xi by evaluating the gradient
∇exiL(x1:n) ∈ R |V | (10)
Here exi denotes the one-hot vector that represents the current value of the i th token. Taking gradient with respect to one-hot vectors was pioneered by HotFlip (Ebrahimi et al., 2017) and applied on Stable Diffusion by a concurrent work (Yang et al., 2023a). Based on this heuristic, we presented two algorithms for finding adversarial suffix tokens against Stable Diffusion.
15

heading: F Primary Determinants of Attack Success
text: 16

heading: G Additional Determinants of Attack Success
text: These sections discuss factors beyond the asymmetric properties that are related to the success rate of the attack. We have found factors like whether target token synonyms are allowed, attack suffix length and attack POS types are factors indicating the attack’s success. We also found that, unlike LLM attacks, adversarial suffixes do not transfer across T2I, indicating that these models might be harder to attack than single-modality models.
G.1 Restricted Token Selection
Emulating QFAttack We can restrict certain tokens to emulate QFAttack (Zhuang et al., 2023) or prevent the exact target word from being selected. We find that QFAttack can be consistently emulated by restricting token selection to tokens corresponding to ASCII characters. We find that such adversarial suffixes can remove concepts (e.g. “a young man” from “a snake and a young man.” or “on a flower” from “a bee sitting on a flower.”) but fail to perform targeted attacks (e.g. changing “a bee sitting on a flower.” to “a bee sitting on a leaf.”). We suspect that this is mainly because ASCII tokens can perturb CLIP’s embedding but are unable to add additional information to it.
Blocking Selection of Target Tokens Another potential use case is preventing the selection of the exact target word. However, we find that the algorithm simply finds a synonym or subword tokenization for the target word when the exact target word (token) is restricted. For example, when attempting to attack the input text “a backpack on a mountain.” to “a castle on a mountain.”, restricting the token corresponding to “castle” leads to the algorithm including synonyms like “palace”, “chateau”, “fort” or subword tokenization like “cast le” or “ca st le” in the adversarial suffix. We find that the effectiveness of the algorithm isn’t affected when the exact target token is restricted and it still finds successful adversarial suffixes using synonyms (when preconditions are met).
Changing the Number of Adversarial Tokens k We set the number of adversarial tokens to k = 5 for all experiments. However, we observe that not all input text-target text pairs require k = 5. “a red panda/car in a forest.” can be attacked with a few as k = 2, i.e. “a red panda/car in a forest.” while “a guitar/piano in a music store.” required all k = 5 (see Appendix D). We leave a comprehensive study on the effect of changing the number of tokens for future work.
G.2 Certain Adjectives Resist Adversarial Attacks
We observed that adversarial attacks targeting certain adjectives, such as color, had a very low ASR. For example, swapping out “red” with “blue” in the prompt “a red car on a city road.” failed in all instances. Further challenging examples include “a red/purple backpack on a mountain.” and “a white/black swan on a lake.”. However, other adjectives like “a sapling/towering tree in a forest” or “a roaring/sleeping lion in the Savannah.” had high ASR in at least one direction. We leave further analysis of this phenomenon for future work.
G.3 Adversarial Suffixes Do Not Transfer across T2I Models
Table 4 shows that different variants of Stable Diffusion were susceptible to entity-swapping attacks and exhibited similar levels of asymmetric bias on prompt pairs.
17
However, the adversarial suffixes generated using SD 2.1-base did not work on SD 1.4 and vice versa. SD 2.1-base uses OpenCLIP-ViT/H (Cherti et al., 2023) as the text encoder while SD 1.4 uses CLIP ViTL/14 (Radford et al., 2021). Although OpenCLIP-ViT/H and CLIP ViT-L/14 have the same architecture and parameter count, the lack of transferability indicates that training data likely plays the main role in determining adversarial attack success.
Similarly, the attack suffixes generated by SD 1.4 or SD 2.1-base did not work on DALL·E 3 (Betker et al., 2023) which likely has a different architecture and training data.

heading: H Human Evaluation WebUI
text: 18

heading: Future Work
text: We leave a comprehensive study on the effect of changing the number of tokens for future work. G. 2 Certain Adjectives Resist Adversarial Attacks
We observed that adversarial attacks targeting certain adjectives, such as color, had a very low ASR. For example, swapping out “red” with “blue” in the prompt “a red car on a city road. ” failed in all instances. Further challenging examples include “a red/purple backpack on a mountain. ” and “a white/black swan on a lake. ”. However, other adjectives like “a sapling/towering tree in a forest” or “a roaring/sleeping lion in the Savannah. ” had high ASR in at least one direction. We leave further analysis of this phenomenon for future work. G. 3 Adversarial Suffixes Do Not Transfer across T2I Models
Table 4 shows that different variants of Stable Diffusion were susceptible to entity-swapping attacks and exhibited similar levels of asymmetric bias on prompt pairs. 17
However, the adversarial suffixes generated using SD 2. 1-base did not work on SD 1. 4 and vice versa. SD 2. 1-base uses OpenCLIP-ViT/H (Cherti et al. , 2023) as the text encoder while SD 1. 4 uses CLIP ViTL/14 (Radford et al. , 2021). Although OpenCLIP-ViT/H and CLIP ViT-L/14 have the same architecture and parameter count, the lack of transferability indicates that training data likely plays the main role in determining adversarial attack success. Similarly, the attack suffixes generated by SD 1. 4 or SD 2. 1-base did not work on DALL·E 3 (Betker et al. , 2023) which likely has a different architecture and training data