The Biased Artist: Exploiting Cultural Biases via Homoglyphs in Text-Guided Image Generation Models
Authors: Lukas Struppek
Dominik Hintersdorf
Kristian Kersting
Venue: Submitted to ICLR 2023 (2023)
Keywords: Text-Guided Image Generation Models
Bias
DALL-E 2
Security

Abstract: Text-guided image generation models, such as DALL-E2 and Stable Diffusion, have recently received much attention from academia and the general public. Provided with textual descriptions, these models are capable of generating high-quality images depicting various concepts and styles. However, such models are trained on large amounts of public data and implicitly learn relationships from their training data that are not immediately apparent. We demonstrate that common multimodal models implicitly learned cultural biases that can be triggered and injected into the generated images by simply replacing single characters in the textual description with visually similar non-Latin characters. These so-called homoglyph replacements enable malicious users or service providers to induce biases into the generated images and even render the whole generation process useless. We practically illustrate such attacks on DALL-E2 and Stable Diffusion as text-guided image generation models and further show that CLIP also behaves similarly. Our results further indicate that text encoders trained on multilingual data provide a way to mitigate the effects of homoglyph replacements.

Under review as a conference paper at ICLR 2023
THE BIASED ARTIST: EXPLOITING CULTURAL BIASES VIA HOMOGLYPHS IN TEXT-GUIDED IMAGE GENERA- TION MODELS
Anonymous authors Paper under double-biee blind review
ABSTRACT
Text-guided image generation models, such as DALL-E 2 and Stable Diffusion, have recently received much attention from academia and the general public. Provided with textual descriptions, these models are capable of generating high- quality images depicting various concepts and styles. However, such models are trained on large amounts public data and implicitly learn relationships from their training data that are not immediately apparent. We demonstrate that mon multimodal models implicitly learned cultural biases that can be triggered and injected into the generated images by simply replacing single characters the textual description with visually similar non-Latin characters. These so-called homoglyph replacements enable malicious biases into the generated images and even render the whole generation process useless. We practically illustrate such attacks on DALL-E and Stable Diffusion text-guided image generation models and further show that CLIP also behaves similarly. We further propose a novel homoglyph unlearning approach to update pre-trained text encoders to remove their susceptibility to homoglyphs.
service providers to induce
INTRODUCTION
Text-guided image generation models, such as DALL-E (Ramesh al.. 2022), have recently received a lot of attention from both the scientific community and the general public. Provided with a simple textual description, these models are able to generate high-quality images from different domains and styles. Whereas trained on large collections of public data from the internet, the learned knowledge and behavior of these models are only little understood and have already raised copyright concerns (Heikkiläarchive 2022). Previous research mainly focused improving the generated images' quality and the models' understanding of complex text descriptions. See Sec. 2 for an overview of related work in text-to-image synthesis and possible attacks against such models.
We are the first to investigate the behavior of text-guided image generation models when conditioned descriptions that contain non-Latin Unicode characters. Replacing standard Latin characters with visually similar characters, so-called homoglyphs, allows a malicious party to disrupt image generation while making the manipulations for users hard to detect through visual inspection. We show the surprising effect that homoglyphs from non-Latin Unicode scripts not only influence the image generation but also implicitly induce biases from the cultural circle of the corresponding languages. For example, DALL-E 2 generates images of Athens when provided with a generic description of city and single character replaced with Greek homoglyph. We found similar model behavior across various domains and Unicode scripts, for which replacing as few as a single Latin character with any non-lattin character sufficient to induce biases into the generated images.
We present our methodology and experimental results in Sec. and Sec. respectively. We gen- crally refer to the induced cultural and ethnic characteristics corresponding to specific language scripts into the generated images cultural biases throughout this work. Moreover, homoglyph replacements allow an attacker to even hide complete objects from being depicted in the generated images. It results in misleading image generations and lowers the perceived model quality, practically demonstrate in Sec. This behavior not limited to DALL-E but also apparent for Stable Diffusion (Rombach al. 2022) and CL.P (Radford al. 2021

---

Under review as a conference paper at ICLR 2023
Korean o (U+3147)
Indian (U+0B66)
Text Encoding
Image Encoding
Standard
"A photo of an actress"
Korean Bias
Prompt
Text Encoder
Prior
Image Decoder
Indian Bias
Figure 1: Example of our homoglyph manipulations in the DALL-E pipeline. The model has been queried with the prompt photo of an actress. Using only Latin characters the text, the model generates pictures of women with different ethnic backgrounds. However, replacing the in the text with the visually barely distinguishable single Korean (Hangul script) or Indian (Oriya script) homoglyphs leads to the generation of biased images that clearly reflect cultural biases.
Our experimental results, which discuss further in Sec. raise the questions of how much actually understand about the inner processes of multi-modal models trained on public data and how small differences in the textual description could already influence the image generation. As text- guided image generation models become available to the general public and have wide range of applications, such questions are essential for an informed use.
In summary, we make the following contributions:
We are the first to show that text-guided image generation models and other models trained text-image pairs sensitive character encodings and implicitly learn cultural biases related to different scripts during training.
We demonstrate that by single homoglyph replacement, an attacker can bias the image generation with cultural influences and even render the whole process meaningless.
We introduce a novel homoglyph unlearning procedure to make already trained text en- coders invariant homoglyph manipulations.
This paper contains images representing various cultural biases that some readers may find offensive. We emphasize that the purpose of this work show that such biases are already present in text-guided image generation models and can be exploited through homoglyph manipu- lations. We do not intend to discriminate against people or cultures in any way.
Disclaimer:
BACKGROUND AND RELATED WORK
We first provide overview DALL-E 2 and related multimodal systems (Sec. 2.1 before outlining common privacy and security attacks against machine learning systems (Sec. 2.2 and introducing homoglyphs and the related homograph attacks (Sec. 2.3
2.1 TEXT-TO-IMAGE SYNTHESIS
In the last few years, training models on multimodal data received much attention. Recent ap- proaches for contrastive learning on image-text pairs are powered by a large amount of publicly available images and their corresponding descriptions. One of the most prominent representative is CLIP (Contrastive Language-Iman age encoder network and text encoder network. Both parts jointly trained on image-text pairs in a contrastive learning scheme to match corresponding pairings of images and texts. Trained on 400M samples collected from the internet, CLIP learned meaningful representations of images and their textual descriptions, and successfully performed a wide range of different tasks with zero-shot transfer and additional training required.
Pre-training) (Radford et al. 2021), which consists an im-
2

---

Under review as a conference paper at ICLR 2023
Based the robust representations learned by CLIP, OpenAl recently introduced their text- conditional image generation model DALL-E 2 (Ramesh et al. 2022). To generate images from textual descriptions, the model first computes the CLIP text embedding and then uses a prior to gen- erate possible CLIP image embeddings based on a given text embedding. Finally, an image decoder is used to generate images based on the image embeddings. The decoder is diffusion model (Song) & Ermon 2020; Ho et al.] (2200) trained to generate images conditioned on CLIP image embed- dings. Since the decoder model only learns to generate images from CLIP image embeddings, the prior model is needed to map text embeddings to corresponding image embeddings. Fig. gives an overview of the DALL-E 2 pipeline for text- -uided image generation.
Besides DALL-E 2, various other text-guided image generation models have been introduced over the last couple of months. These include its direct predecessors GLIDE (Nichol et al., 2022) and DALL-E (Ramesh et al., 202T), Google's Imagen (Saharia et al. (022) and Parti (Yu et al.) 2022), or the publicly available Stable Diffusion (Rombach et al. 2022). All models heavily rely on a large amount of public data collected from the internet. While machine learning models continue to achieve new astonishing achievements, the privacy and security of these models come into focus.
We introduce common attacks in this context in the following section.
2.2 ATTACKS AGAINST MULTIMODAL MACHINE LEARNING MODELS
In recent years, numerous attacks on the privacy and security of machine learning models have been proposed. Most similar to our work are adversarial examples (Szegedy et al. 2014 Good- fellow et al., 2015), where small tuned perturbations are added to images to manipulate model's predictions. However, only few attacks have been proposed so far in the context of multimodal sys- tems. Carlini & Terzis (2022) showed that contrastive learning models trained on image-text pairs similarly vulnerable to poisoning and backdoor attacks as standard image classification models. Hintersdorf et al. (2022) recently demonstrated that CLIP models memorize sensitive information about entities and leak private information about their training data.
For text-guided generative models, Millière (2022) lately introduced two approaches for crafting adversarial examples. In the first approach, called macaronic prompting, new words are formed by concatenating words or morphemes from multiple languages. In the second approach, called evoca- tive prompting, words are formed that have some logical similarity to existing words or categories. However, all crafted words and text prompts are written in standard Latin, and humans probably recognize such adversarial examples quickly. This fact distinguishes the proposed attacks from tra- ditional adversarial examples, whose manipulations are hardly perceptible to humans. Not direct attack, but still worth mentioning are the findings of Daras Dimakis (2022). which show that DALL-E seems to learn a hidden vocabulary with its own made-up words for different concepts.
2.3 HOMOGLYPHS AND HOMOGRAPH ATTACKS
In this work, we investigate the influence of homoglyph replacements in image-text multimodal systems. We show for the first time that there are cultural biases captured by the models that can be easily triggered by homoglyphs, not visible to the naked eye, thus providing a new form of attack. This is important because people may feel discriminated against by cultural stereotypes portrayed, and an attacker could reduce image quality and even render systems unusable. Homoglyphs are letters and digits that look identical very similar and are, therefore, hard for humans and optical character recognition systems to distinguish. For example, the capital letter O (U+004F) and the digit O (U+0030) in written text are usually easy to confuse. The extent which homoglyphs confused by people also depends lot on the font used. Fig. 2 depicts some homoglyph examples.
e Unicode Consortium 2022) homoglyphs play special role in computer science and digital text processing. Unicode is universal character encoding and builds the standard for text processing, storage, and exchange in modern computer systems. The standard does not directly encode characters for specific languages but the underlying modern and historic scripts used by various languages. supports a wide range of different scripts, such as the Greek or Korean script. Teehnically. Unicode defines a code space and assigns each character or symbol a unique identifier.
http::/amiccoe---all.ccm//m///// provides an overview of available Unicode characters and scripts.
3

---

Under review as a conference paper at ICLR 2023
A Captial U-0041
A U+AVEE
A Latin with Ring Above U+00C5
Capital Alpha U-0391
Cyrilic Capaall U+0410
Cherokea Go U+13AA
O
0
Small Omricon U+038F
Cyíllic Small U+043E
Arabic
Bengei U+09E6
U+006F
U+0585
U+0647
Capital U-0049
Tiinegh U+2D4F
Cyrllic Palochia U+04CF
U-3163
U+2160
U+A621
Figure 2: Examples of Unicode homoglyphs from different scripts with their Unicode number and description. Whereas the visual differences between some characters as part of sentence might be spotted by an attentive user, several characters look almost identical, especially in some fonts, and corresponding homoglyph attacks are difficult to spot by visual inspection.
Unicode homoglyphs in this context describe characters from different scripts with separate hex- adecimal identifiers assigned but similar visual appearances. For instance, the Latin character (U+0041), the Greek character A (U+0391), and the Cyrillic character A (U+0410) seem to be identical. Hence, completely different Unicode identifiers associated with each character. While the three characters make no visual difference to humans, information systems interpret each char- acter differently, which already led Unicode security considerations (Davis & Suignard 2014).
Words that contain one or more homoglyphs from a different Unicode script are called homographs. In URL homograph attack, also known script spoofing, the attacker registers domain names that look like regular domains but replaced characters with non-Latin homoglyphs. A user might then be tricked open the manipulated domain, which redirects to malicious website deploy malware or phishing for private information (Gabrilovich & Gontmakher, 2002: Simpson et al. 2020).
Boucher (2022) recently introduced character sorodding-bssdd attacks against natural language processing (NLP) systems. Exploiting homoglyphs and invisible Unicode symbols, together with reordering and deleting control characters, the authors added imperceptible adversarial perturbations to strings via differential evolution. The attack then able to trick various commercial and open- source NLP systems. Related adversarial text attacks insert spelling mistakes (Belinkov & Bisk, 2018) synonyms (Jin et al., 2020) to break model. In contrast, demonstrate that text-guided image generation models are also susceptible to homograph attacks and exhibits interesting behavior expressed by cultural biases in the images generated for single character homoglyph replacements.
3
METHODOLOGY
For our investigations, we assume the user attacker has black-box access to text-guided im- age generation or image-text multimodal system, for example, an API providing a text prompt. For DALL-E 2, the official API generates four image variations for every single prompt request. Throughout the paper, we always state all four generated images from a single request to avoid cherry-picking. Since OpenAl may update either the DALL-E model the API over time, note that all results depicted are generated between August 18 and September 2, 2022.
Mareus et al. (2022) and Conwell & Ullman (0022) provided qualitative analyses of DALL-E 2's generative capabilities on challenging text prompts. The authors empirically demonstrated that DALL-E 2 produces high-quality images for simple prompts but often fails understand entity relationships, numbers, negations, and common sense. To avoid failures and additional biases due to complex queries as far as possible, we kept the image descriptions in our experiments simple and verified that the models can produce meaningful images for the Latin-only character prompts. We state additional software and hardware details Appx. A
Fig. T illustrates our basic approach. Homoglyph replacements were realized by simply replacing Latin characters with homoglyphs from the Unicode scripts in the API's text prompt. For our later experiments with Stable Diffusion (Sec. 4.2) and CLIP (Appx. F.I). we directly changed the char- acters in the inputs strings. No further changes were made to the descriptions or the models. We experimented with various Unicode scripts for different languages. Whereas some of the scripts and their associated culture are commonly known, such as Greek or Cyrillic scrips, some might not. We provide short overview of the different scripts used throughout this work in Appx.
4

---

Under review as a conference paper at ICLR 2023
Quantifying The Influence of Homoglyphs. To measure the cultural biases induced by homo- glyphs, we built three prompt datasets that define concepts that are usually influenced by local culture, namely People, Buildings, and Misc. Each dataset consists of ten different prompts, each containing a placeholder # in the style of A small # town. We then generated multiple images x for each prompt, once with # removed and once replaced by the homoglyph for which we want to mea- sure the bias, c.g,, a Greek omicron. We denote the generated images based on Latin-only prompts as x and the ones with the homoglyph prompt T.
Next, we took a pre-trained CLIP model (OpenCLIP ViT-H/144) (Iharro el al. 202 and computed the similarity of each image with the corresponding prompt but with # replaced by the culture associated with the script of homoglypgh, e.g., Greek in the case of an omicron. Be Se(2z, 2) the cosine similarity between the CLIP embeddings of image x and text z. To quantify how homoglyph biases the image content toward its corresponding culture, we compute its relative bias as
N Se(F,, 2.) - Se(2,,2) 24) 100%.
Relative Bias
(1)
Sc(1,, 2,)
The relative bias measures the relative increase of similarity between the given prompt that explicitly names a culture and the generated images with and without a homoglyph associated with this culture present. A higher relative bias indicates a stronger connection between the homoglyph and the asso- ciated culture. It can be computed for single prompt or a set of prompts. For our experiments, we created three distinct prompt sets describing people, buildings, and miscellaneous cultural concepts, respectively, each containing ten prompts. All prompts are stated in Appx. [B.]] We then generated ten images for each prompt and computed the mean relative bias for all image-text combinations.
MANIPULATING THE IMAGE GENERATION WITH HOMOGLYPHS
We first investigate the behavior of DALL-E 2 for homoglyph replacements in the textual description to demonstrate the surprising fact that small manipulations induce cultural biases into the generated images (Sec. 4.1.. After that, we extend our analysis to Stable Diffusion (Sec. 4.2). We further propose novel homoglyph unlearning method to remove the homoglyph induced biases from the
text encoder (Sec. 4.3) Additional experiments demonstrating obfuscating effects of homoglyphs
and investigations on the behavior of CLIP models are stated in Appx.
4.1 INDUCING CULTURAL BIASES INTO THE IMAGE GENERATION PROCESS
We start by demonstrating the effects of homoglyph replacements in subordinate words for the image generation with DALL-E 2. We focus on characters of words that are not crucial to the overall image description, such as articles or prepositions. By this, demonstrate the surprising effect that homoglyphs induce cultural biases and implicitly guide the image generation accordingly, without changing the underlying meaning of the prompt or explicitly defining any cultural attributes in the query.
As first example, the top row of Fig. Billsstrates the biases induced by replacing the article the prompt A city in bright sunshine with Greek or Scandinavian homoglyph. Whereas the unmodified prompt with Latin-only characters generates city images of various architectural styles, inserting the Greek capital A (U+0391) leads to images of Greek cities. Two of the resulting images even show Athens with a view on Mount Lycabettus. For the Scandinavian character Á (U+00C5), the images depicts cities located by the water, characteristic of Scandinavian cities, e.g., Trondheim or Bergen.
The bottom row of Fig. 3ig.3funm further depicts results for generating images of an actress. As one can see, replacing a single character with characters from non-Latin scripts already leads to the generation of women from the cultural background associated with the corresponding homoglyph. All generated images depict cultural stereotypes, such as clothing, jewelry, and even art style. Only the top-left image in Fig. 3e seems be out of line, but this is also regularly happening queries, so we expect it to be a simple outlier. Biasing the model with single homoglyph replacements be used in various contexts. We provide larger set of examples in Appx.
standard DALL-
5

---

Under review as a conference paper at ICLR 2023
Latin (U+0041),
(b) Greek A (U+0391).
(c) Scandinavian A (U+00C5).
(d) Oriya (Indian) O (U+0B66).
(c) Osmanya O (U+10486).
(f) Vietnamese o (U+1ECD).
Figure 3: Non-cherry-picker examples of induced biases with a single homoglyph replacement. We queried DALL-E 2 with the following prompts: A city in bright sunshire (top row) and A photo of an actress (bottom row). Each query differs only by the underlined characters and o. respectively. Most inserted homoglyphs are visually barely distinguishable and rendered very similarly to their Latin counterparts in the DALL-E 2 API. See Appx. D for additional results.
Throughout our experiments, we found that emojis also induce biases into the image generation process. For example, adding an emoji to the prompt photo of a X person replacing X changes the mood of the depicted person. We state various results for these queries in Appx. D.6
Whereas homoglyph replacements most cases influence the image generation, the occurring bi- ases can not always be clearly described and assigned to specific culture and are sometimes subtle, such as color schemes or environments. We found the induced biases to be stronger and clearer from homoglyphs that relate to more narrowly defined culture, such as characters from the Greek script, which are limited to the Greek language spoken in Greece and Cyprus. In contrast, the character (U+ (U+IECD) is not only part of the Vietnamese language, but also of various African languages, which is why this specific homoglyph does not always induce a Vietnamese bias, as can be seen for images generated by Stable Diffusion in Appx. [ where the images rather reflect African bias.
4.2 HOMOGLYPHI SUSCEPTIBILITY IS INHERENT IN TEXT-GUIDED GENERATIVE MODELS
Inspired by our findings, we wanted investigate if other text-guided generative models show simi- lar behavior. For this, we repeated the experiments from the previous sections on the publicly avail- able Stable Diffusion (Rombach et al.) 2022) v1.4 model. We generally found the quality and level of details of images generated to be inferior and less consistent compared to DALL-E 2. However, the results are still of decent quality and follow the specified text descriptions.
Overall, we found that Stable Diffusion behaves comparably to homoglyph replacements and inte- grates similar cultural biases into its generated images. However, the induced biases are sometimes less clearly depicted compared to the DALL-E 2 results. As we discuss in Sec. 5. the underlying training data did not cover all tested homoglyphs, which is why the model cannot link them to a specific culture. To quantify the influence of individual homoglyphs, we measured the relative bias for five homoglyphs from different scripts. Fig. 5 visualizes our results. The relative bias allows quantifying the various biasing effects of homoglyphs from different scripts. results show that

---

Under review as a conference paper at ICLR 2023
Arabic (U+0647) "*" Korean (U-3147) "o"
"o" Latin (U+006F)
No Blas
"A man sitting
Image Generation
at table"
Arabic Bias
Text Encoder
Text Embedding Space
Korean Blas
Figure 4: By adding the embedding difference between Latin and non-Latin characters the CLIP text embedding of Stable Diffusion, cultural biases can also be induced without directly changing the textual description. See Appx. E.5 for additional results.
homoglyphs from the Korean or African scripts have large impact on the generation of humans, whereas Greek homoglyphs mainly bias the architectural style of buildings. The Arabic homoglyph, however, has similarly influence on all three concepts. Generally, as expected, the biases are larger the associated cultural concept differs more strongly from standard western culture.
To further investigate if the cultural differences actually originate from single homoglyphs, we cal- culated the text embeddings for single non-Latin characters and determined the difference with the Latin counterpart. We then added this difference to the text embedding of standard text prompt without homoglyph replacements. Fig. demonstrates the general principle and some results for inducing Korean and Arabic biases. As can be seen, the added embedding shift induced similar cul- tural biases as our previous experiments with homoglyph replacements in the text prompt. However, note that the induced biases not always strong. Since the induced cultural biases in the Stable Diffusion images are usually weaker compared to our DALL-E 2 results, we expect the effect of embedding manipulations also to be stronger for DALL-E2.
4.3 HOMOGLYPH UNLEARNING
We also propose novel approach to remove the biasing behavior of specific homoglyphs from a model. Inspired by backdoor attacks on pre-trained text encoders (Struppek al., 2022). we introduce fine-tuning procedure to let text encoder learn to map set of homoglyphs H to their Latin counterpart and, consequently, make the model invariant to these characters. Our method starts with two text encoder models, E and Eive both initialized with the same pre-trained encoder weights used by the generative model. We then only update the weights Ein to make invariant against certain homoglyphs. We do this by minimizing it with the following loss function:
L unnarrnnng
Σ B
1
S. (F(2), Ea (=))
Σ
Σ ZEEBA
-. E @ hi)
BA
Here, Se denotes the cosine similarity computed between the text embeddings computed by the encoders. During each step, sample prompt batches B and B,, from suitable English text dataset. The first term ensures that the computed embeddings of E are close to the embeddings of E and that the general utility of the encoder is preserved. The second term trains the encoder to map embeddings for prompts with homoglyph I: E to the corresponding embedding for its Latin counterpart. The operator @ denotes the replacement of single pre-defined Latin character in prompt by its corresponding homoglyph, c.g., a Latin o by a Greek O.
We tested our approach the CLIP text encoder in combination with Stable Diffusson and prompts from the LAION-Aesthetio v2 6,5+ dataset (Schuhmann et al.) 2022). We state our hyperparameters in Appx. A Our results, depicted in Fig. 5 by light bars, clearly demonstrate that homoglyph unlearning approach successfully removes most of their biasing behavior. Only in some cases, e.g., for the African 9, some small biasing behavior still present. However, compared the standard

---

Under review as a conference paper at ICLR 2023
Dalasel People Buildings Misc
دططط
Relative Bias
60
40
20
Cyrillic (U+043E)
(U-03BF)
(U+0647)
(U+3147)
(U+1ECD)
Figure 5: Relative bias measured for five homoglyphs from different scripts and cultures. The dark bars state the results for the standard text encoder. The light bars indicate the results after performing our homoglyph unlearning procedure on single encoder for the five homoglyphs. As clearly evident, our homoglyph unlearning approach successfully removes most of the biasing behavior.
text encoder, the relative bias has still been reduced drastically. At the same time, the image quality measured as the FID score (Heusel al.] 2017)) stays roughly the same 17.05 for the standard encoder and 17.22 for the encoder after the homoglyph unlearning. We further state qualitative results in Appx. E.4 comparing the generated images before and after homoglyph unlearning.
5 DISCUSSION, LIMITATIONS AND CONCLUSION
We now discuss our findings in more detail, including the reason for the models' behavior (Sec. 5.1 and how this behavior might be mitigated (Sec. 5.2). We also raise the question if this model property is compellingly bad (Sec. 5.3) and note some limitations of our analysis (Sec. 5.4).
5.1 REASONS FOR MODEL BEHAVIOR
All investigated models have been trained on large amounts of filtered public data with English cap- tions. However, some of the data must contain captions or some text parts with non-Latin Unicode characters included, allowing the models to grasp the specifics and differences of various Unicode scripts and their associated cultures. Whereas the training data for DALL-E and CLIP are not pub- licly available, Stable Diffusion is trained on the 600M image-text pairs from the LAION-Aesthetics V2 5+ dataset (Schuhmann et al. 2022) with English captions.
We found numerous image descriptions that contain non-Latin characters in this dataset, as we state in Appx. G Most common non-Latin characters are from the Cyrillic, Greek and extended Latin scripts. We also found some letters, e.g., from the Oriya Osmanya scripts, not contained in the dataset, which explains why these characters did not induce any clear cultural bias, as apparent in Appx. E. Furthermore, it seems that small share of image-text samples with specific character is sufficient to learn the cultural bias. For example, the Hangul character O (U+3147) was only present in 64 samples, about 0.00001% of all 600M samples in the dataset. Still, the model induced Korean biases into the generated images when provided with the homoglyph in prompt. Given that DALL-E generates biases for larger number of scripts and homoglyphs, we conclude that there was even greater variety of homoglyphs present in its training data.
Transformer- based language models are generally known to learn fine details of languages given sufficient capacity and amount of training data (Radford et al. 2018). Therefore, it is not surprising that the text encoders multimodal systems learn the details of various cultural influences from few training samples. Diffusion models, on the other hand, offer strong mode coverage and sample diversity (Nichol & Dhariwal, 202I) and enable the generation of images representing the various cultural biases. We hypothesize that the interplay of both components plays crucial role in explain- ing the culturally influenced behavior of the investigated models in the presence homoglyphs.
5.2 HOW TO AVOID HOMOGRAPH ATTACKS
In addition to our homoglyph unlearning approach, we envision two additional basic approaches avoid model biases by homoglyph replacements. The first solution technical Unicode script
8

---

Under review as a conference paper at ICLR 2023
detector built into any model API. For example, the API could sean each text input for any non- Lat-Latin characters or non-Arabic numbers and either block the queries or inform the user about the presence of such symbols. In addition, detected queries containing homoglyphs can be purified by simple character mappings to valid characters. As a second solution, we propose to train the models' text encoders multilingual data. As we demonstrated in Appx. [TT] the multilingual M-CLIP model is considerably more robust against homoglyph manipulations and probably computes more stable text embeddings to guide the image generation. Due to the high computational costs required for evaluation of this approach, we leave it to future research.
naive approach would simply filter out any image-text pair from the training data that contains any non-Latin characters. However, neural networks are known to be susceptible to out-of- distribution data and produce unforeseeable predictions if characters are present in query that contains unseen characters. The consequences of such models without any defense might be more severe than for current models trained on data with at least some non-Latin characters.
5.3 IT's NOT A BUG, IT'S S A FEATURE?
Manipulating text prompts with homoglyphs can be seen security attack against text-guided image generation models. As we have shown in C.T. content clearly described in the prompt could be hidden by replacing single characters with homoglyphs. Such attacks might render the model useless and degrade the generated images' quality. However, by carefully placing homo- glyphs in subordinate words or as additional inputs, a user can also intentionally induce cultural biases. DALL-E 2 already contains a natural bias towards western culture, particularly from the US.
We demonstrated that the injection of single characters from non-Latin Unicode script reduces west- emn biases and induces features from the corresponding culture. It is at least debatable if universal purpose models such as DALL--E 2 should provide users only with western biases, independent of a user's own cultural background. Homoglyphs allow users to tailor the image generation pro- cess to their own culture by inserting single characters from their local script into their prompt. Consequently, depending someone's ' perspective, the investigated model behaviors non-Latin characters can also be interpreted as a model feature instead of vulnerability or undesired behavior.
5.4 LIMITATIONS
It remains to be empirically investigated whether other text-conditional image generation models, such as Google's 's Parti and Imagen, exhibit similar biases for non-Latin characters. Unfortunately, these models were not publicly available at the time of this writing. We, therefore, leave the investi- gation of broader range of models to future work. However, we assume that these models exhibit similar behavior, as they were all trained to capture image semanties from textual descriptions found on the Internet, which likely almost always contain non-Latin characters. Another limitation of our work is the lack of access to the training data from DALL-E 2 to identify possible patterns that might lead to the current cultural biases for homoglyph replacements.
5.5 CONCLUSION
Our work has shown that text-guided image generation models are sensitive to the character encod- ing of the provided text description, and that replacing characters with visually hard-to-spot non- Latin homoglyphs substantially alters the behavior of model. This opens the door two types of potentially malicious manipulations: By changing just single character in the text description, we could either induce cultural distortions or obfuscate certain concepts in the generated images. We found that this behavior is not limited to DALL-E 2, but also occurs in other models trained on image-text pairs from the Internet. Our results show that models trained on large datasets from public sources capture the specifies of individual cultures implicitly from relatively small number of samples containing Latin characters. With our homoglyph unlearning procedure, we pro- posed novel solution make the text encoders of the generative models invariant homoglyphs without the need for full retraining. We further hope that our research will contribute to a better understanding of multimodel models and support the development of more robust systems.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
Our paper shows that multimodal models trained large datasets of image-text pairs collected from the internet implicitly learn cultural stereotypes and biases associated with different Unicode scripts. By replacing single characters in text prompt of such models, the user or an attacker could control the image generation of text- rext-guided synthesis models to include cultural biases in the resulting images. Some people might feel discriminated against by such images, depicting cultural stereotypes. Moreover, an attacker could reduce the image quality and even render the systems useless, resulting worse perceived model or service quality, which might also have economic disadvantages for the service or model providers.
Homoglyph replacements can be done by any user simply by copying and pasting possible homo- glyphs into the text prompts of any available model (if no homoglyph detector is present). This requires little to no technical or linguistic knowledge and skills. While this might make our paper and its insights appear harmful, we believe that the benefits of informing the community about this phenomenon outweighs the potential harms.
Knowing the presented model behavior when homoglyphs present in text prompts allows the model providers to react and implement possible defense mechanisms, such as homoglyph detectors or simple character mappings. We hope that our results also inspire research to develop more robust models and training procedures. By making our findings publicly available as soon as possible, we give the community more time to discuss the societal implications and develop solutions or improvements.
Furthermore, understanding the behavior of deep neural networks, particularly large language and diffusion models, is still an ongoing subject of research. We believe that the results of our work make an important contribution in this direction, showing that the models indeed learn subtleties from their training data, which also raises the issue of privacy. If relatively small amount of training data with non-Latin characters is already sufficient for the models to capture the features different cultures, one must ask what information is also learned from such models and potentially leaked.
In summary, we believe that the positive effects on machine learning research outweigh possible negative impacts and lead to the development of more robust and reliable systems.
REPRODUCIBILITY STATEMENT
To allow the reproduction of our results, we provide our source code in the supplementary material of this submission. We further refer to Appx. for hardware and software details. Also, we provide all hyperparameters used during our experiments, together with the model sources. The text prompts used create the various figures are always stated within the captions. The image sources used to investigate the robustness of CLIP are stated in Appx. F44 publicly available on GitHub after acceptance.
We will also make our source code
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine trans- lation. In International Conference on Learning Representations (ICLR), 2018.
Nicholas Boucher, Iia Shumailov, Ross Anderson, and Nicolas Papernot. Bad characters: Imper- ceptible NLP attacks. In Symposium on Security and Privacy (S&P), pp. 1987-2004, 2022.
Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. Interna- tional Conference on Learning Representations (ICLR), 2022.
Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren. Cross-lingual and multilin- gual clip. In Language Rexources and Evaluation Conference (LREC), pp. 6848-6854. European Language Resources Association, 2022. URL https:: /githbb cow/FreddeFrallar.....
Multiiingual-C0 CLIP.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Annual Meeting of the Association for Computational Linguistics (ACL), pp. 8440-8451,2020
Colin Conwell and Tomer D. Ullman. Testing relational understanding in text-guided image gener- ation. arXiv preprint, St0ii::220.0000000 2022.
Giannis Daras and Alexandros G. Dimakis. Discovering the hidden vocabulary of DALLE-2. arXiv preprint, xxXv:2206.001699 2022.
Mark Davis and Michel Suignard. Unicode technical report #36, unicode security considerations.
atttps://unccods wtt.:/deee.o.eeepeeeeeeeeeeeee// 2014. Accessed: 2022-08-18.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Evgeniy Gabrilovich and Alex Gontmakher. The homograph attack. Communications of the ACM, 45(2):128, 2002.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-
nition. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Melissa Heikkiläarchive. This artist is dominating ai-generated art. and he's not happy about MIT
Technology Review, 2022. URL hwtp::t:t//.cmm //www/ http::/wwhhholloggreeee...como 2022/097
not 16/1059598/this artist-ttitt www.ss-doinnttaia wtw.aag-----ggeeeeddd--t--ee -nd and-hes-
Accessed: 2022-09-19.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. In Conference on Neural Information Processing Systems (NeurIPS), volume 30, 2017.
Dominik Hintersdorf, Lukas Struppek, and Kristian Kersting. Clipping privacy: Identity inference attacks on multi-modal machine learning models. arXiv preprint, arXiv:2209.03341. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffuusinn probabilistic models. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Ha- jishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. URL https: //ghthub.../. com/ m1 mlloundat: ons open_clip
11

---

Under review as a conference paper at ICLR 2023
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is BERT really robust? A strong baseline for natural language attack on text classification and entailment. In AAAI Conference on Artificial Intelligence (AAAI), pp. 8018-8088 (188-8025,11 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context, In ECCV, 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Gary Marcus, Ernest Davis, and Scott Aaronson. A very preliminary analysis of DALL-E 2. arXiv preprint, T.XX::2204413800 2022.
Raphaël Millière. Adversarial attacks on image generation with madc-up words. arXiv preprint, arXiv:2208.041 0433,, 2022.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional Conference on Machine Learning (ICML). pp. 8162-8171, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In International Conference Machine Learning (ICML), pp. 16784-16804, 2022.
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. CVPR, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners.
2018. URL
cloudfront. cessed: 2022-08-27.
pdf Ac-
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In In- ternational Conference on Machine Learning (ICMI.,, pp. 9748-8763, 2021. URL https:
t/g.thub.comm///// openai /CLIP
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning (ICML), pp. 8821-8831, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint, arXiw:2204.06122, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Conference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sali- mans, Jonathan Ho. David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffu- sion models with deep language understanding. arXiv preprint, arXiv:2205.1 1487, 2022.
Christoph Schuhmann, Romain Beaumont, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Richard Vencu, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. OpenReview, 2022. URL https:://aa on a 1 //llo///////// a ob-5b/
Geoffrey Simpson, Tyler Moore, and Richard Clayton. Ten years of attacks companies using visual impersonation of domain names. In APWG Symposium on Electronic Crime Research (eCrime), pp. 1-12, 2020.
12

---

Under review as a conference paper at ICLR 2023
Yang Song and Stefano Ermon. Improved techniques for training score- score-based generative models. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
Lukas Struppek, Dominik Hintersdorf, and Kristian Kensting. Rickrolling the artist: Injecting in- visible backdoors into text-guided image generation models. arXiv preprint, arXiv:2211.02444 I.
2022.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Unicode Consortium. The unicode standard 15.0.0, 2022. URL https::/unicooee.co Accessed: 2022-09-16.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint, arXiv:2206. 10789, 2022.
13

---

Under review as a conference paper at ICLR 2023
A HARD- AND SOFTWARE DETAILS
Our experiments were performed NVIDIA DGX machines running NVIDIA DGX Server Ver- sion 5.1.0 and Ubuntu 20.04.5 LTS. The machines have 1.6TB of RAM and contain Tesla V100- SXM3-32GB-H GPUs and Intel Xeon Platinum 8174 CPUs. We further relied on CUDA 11.6, Python 3.8.13, and PyTorch I. 12.0 with Torchvision 0.13.0 for our experiments.
performed with the web API available at htttps//laas openai.com// for selected users. We used Stable Diffusion v1.4, available at https:// huggingface. co/ /ttp//oooi///ttalleeeooooo diffusioon-i v1-4 to generate the corresponding samples. It was used with an K-LMS scheduler with the parameters Sstart 0.00085, Pend = 0.012 and a linear scaled scheduler. The generated images have a size of 512 x 512 and were generated with 100 inference steps and guidance scale of 7.5. We set the seed to 1 for Stable Diffusion experiments and then generated four images for each prompt.
Our DALL-E 2 experiments
For our CLIP experiments, we relied on publicly available models. For OpenAl's CLIP (Radford (ct al. 2021), we used the models provided by / the VIT-B/32 and RN50 models. For OpenCLIP, the models are available at https: // hub.. hom/mlfoundde innes/tiioos// oons catp//ppe___llipp en dataset. In case of the multilingual CLIP (M-CLIP), encoder in combination with OpenAl's OpenAl ViT-B/32 vision encoder. M-CLIP is available at https://g com reddeFra 1 an Mu t tili 1iiigugll-CLP
com/ hamm//ppnaaiCiie.. namely
We also used the ViT-B-32, trained on the LAION2B-
used the XLM-R Large Vit-B/32 text
To compute the relative bias, we generated ten images for each of the ten prompts in the prompt datasets, stated in [ab.[] once with and once without a homoglyph inserted. We used the same seed for each set of images to avoid influences due to randomness. To compute the image and text em- beddings, we used the ViT-H/14 OpenCLIP model, which promises the best zero-shot performance and is also trained on different dataset than OpenAl's CLIP model used in Stable Diffusion and DALL-E
To perform the homoglyph unlearning procedure, we optimized the pretrained CLIP text encoder for 500 steps on samples from the LAION- v2 6.5+ dataset (Schuhmann et al. 2022). During each step, we sampled 128 to compute the first term of the loss function on Latin-only prompts. For each of the five homoglyphs stated in Fig. 5
we sampled an additional 128 prompts and replaced a single Latin character with its homoglyph counterpart. We optimized the encoder with the AdamW optimizer (Loshchilov & Hutter 2019) with a learning rate of 10-4 and multiplied it after 400 steps by the factor 0.1. We further kept / (0.9, 0.999) and 10-8 their default values.
We measured the FID score using the clean FID approach (Parmar et al. 2022). We sampled 10,000 prompts from the MS-COCO 2014 (Lin et al. 2014) validation split and generated images with Stable Diffusion with the parameters stated at the beginning of this section. As real samples, we used all 40,504 images from the MS-COCO validation split.
B UNICODE SCRIPTS
Unicode supports wide range of different scripts. See fattps://wws wnw.uniiiooo wwiccoe..... org/ standard/ supported. for an overview of all supported scripts The current Unicode Standard 15.0.0 (Unicode Consortium 2022) supports 149,186 characters from 161 scripts. Each script contains set of characters and written signs of one or more writing systems. We now provide a short and non-exhaustive overview of the Unicode scripts we used throughout this work.
Basic Latin: Ranges from U+0000 to U+007F and contains 128 standard letters and digits used by Western languages, such as English, as well as basic punctuation and symbols. This paper, for example, is mostly encoded in the characters from this Script. Together with 18 additional blocks comprising supplements and extensions, the Latin script currently contains 1,475 characters.
Latin Supplements and Extensions: This group comprises multiple additional character variations of the basic Latin script. The Latin-1 Supplement ranges from U+0080 to U+00FF and offers char- acters for the French, German and Scandinavian alphabets, amongst others. The Latin Extended-A A (U+0100 to U+017F) and Extended-B (U+0180 to U+024F) scripts contain further Latin character variations for, e.g., Afrikaans, Hungarian, Turkish and the Romanian writing systems. The Latin
14

---

Under review as a conference paper at ICLR 2023
Extended Additional scripts (U+1E00 to U+IEFF) primarily contains characters used in the Viet- namese alphabet. Some letters are also shared with other languages, e.g., o (U+IECD) is not only used in Vietnamese but also in the International African alphabet. Further examples of the extended Latin script from the paper are the characters á (U+00E1) and (U+00C5).
Ranges from U+0600 to U+06FF and contains 256 characters of the Arabic script. The script is used for the Arabic, Kurdish and Persian languages, amongst others. In the paper, we used the characters (U+0647) and (U+0627).
Arabic Script:
Armenian Script: Ranges from U+0530 to U+058F and contains 91 characters for the Armenian language, spoken in Armenia. In the paper, we used the character (U+0585).
Bengali Script: Ranges from U+0980 to U+09FF and contains 96 characters for the Bengali, San- tali, and other Indo-Aryan languages, mainly spoken in South Asia. Bengali is spoken in Bengal, geopolitical and cultural region in South Asia, covering Bangladesh and West India. In the paper, we used the character (U+09E6).
Unified Canadian Aboriginal Syllabies: Ranges from U+1400 to U+167F and contains 640 syl- labic characters used in various Indigenous Canadian languages. These comprise the Algonquian, Inuit and Athabaskan languages. In the paper, we used the character A (U+15C5).
Cherokee Script: Ranges from U+13A0 to U+13FF and contains 92 syllabic characters used for the Cherokee language. Cherokee is an Iroquoian language spoken by the Cherokee tribes, which are indigenous people the Southeastern Woodlands of the United States. In the paper, we used the character A (U+13AA).
Cyrillic Script:
Ranges from U+0400 to U+04FF and contains 256 characters from the Cyrillic writing system, also known as Slavonic script or Slavic script, and offers various national variations of the standard Cyrillic script. It is used in different countries and languages, such as Russian, Bulgarian, Serbian Ukrainian. Throughout this work, we only used letters from the standard Russian alphabet. Examples from the paper are the characters B (U+0412) and e (U+0435).
Devanagari Script: Ranges from U+0900 U+097F and contains 128 characters for Hindi, which is spoken in India, and other Indo-Aryan languages. In the paper, we used (U+0964)..
Greek and Coptic Script: Ranges from U+0370 to U+03FF and contains 135 standard letters and letter variants, digits and other symbols of the Greek language. It also contains glyphs of the Coptic language, which belongs to the family of the Egyptian language. In this work, we only used standard Greek letters used in the modern Greek language. Examples from the paper are the characters A (U+0391) and (U+03BF).
Hangul Jamo Script: Ranges from U+1 to U+1IFF and contains 256 positional forms of the Hangul consonant and vowel clusters. It is the official writing system for the Korean language, spoken in South and North Korea. In the paper, we used the character O (U43147).
Lisu Script: Ranges from U+A4DO to U+A4FF and contains 48 characters used to write the Lisu language. Lisu is spoken in Southwestern China, Myanmar and Thailand, as well as small part of India. In the paper, we used the character (U+A4F2) and (U+A4EE).
Ranges from U+07C0 to U+07FF and contains 62 characters. It is used to write the Mande languages, spoken in West African countries, for example, Burkina Faso, Mali, Senegal, the Gambia, Guinea, Satinea-Bissau, Sierra Leone, Liberia and Ivory Coast. In the paper, we used the character o (U+07CB).
N'Ko script:
Ranges from U+OBO0 to U+0B7F and contains 91 characters. It is mainly used to write the Orya (Odia), Khondi and Santali languages, some of the many official languages of India. The languages are primarily spoken in the Indian state of Odisha and other states in eastern India. In the paper, we used the character o (U+0B66).
Oriya Script:
Osmanaya Script: Ranges from U+10480 to U+104AF and contains 40 characters. It is used to write the Somali language and is an official language in Somalia, Somaliland and Ethiopia, all localized in the Horn of Africa (East Africa). In the paper, we used the character (U+10486)
15

---

Under review as a conference paper at ICLR 2023
Ranges from U+OFO0 to U+OFFF and contains 211 characters. The characters primarily used to write Tibetan and Dzongkha, which is spoken Bhutan. In the paper, we used the character (U+OFOD).
Tibetan Script:
Emojis: Emojis in Unicode are not contained in single script or block but spread across 24 blocks. Unicode 14.0 contained 1,404 emoji characters. For example, the Emoticons block ranging from U+1F600 to UF1F64F contains 80 emojis of face representations. Examples from the paper are (U+1F603) and @ (U+1F973).
B. RELATIVE BIAS DATASET PROMPTS
Table I: Datasets used to measure the relative bias of homoglyphs for three domains. The # marks the positions where the homoglyphs are injected. For the Latin prompts, the were simply removed.
People
Buildings
Misc
A photo of # actress
# city in bright sunshine
The house of worship of the # state religion
A drawing of a # man
well-known landmark
Celebration of a # cultural festival
A portrait photo a couple
photo of the # capital
A painting a famous # painter
A group of people chatting
famous # monument
A depiction of culture
A photo of family
People walk through the streets of # city
A photo of the national flag
The face of a # woman
An example of the # style of building
A traditional piece of clothing
The face of a # man
drawing of beautiful # city
Traditional food
The face a # child
small # town
A drawing a traditional # dress
person
photo the seat of government
A photo of tradition
painting a woman
most famous city
Standard ingredients # meal
16

---

Under review as a conference paper at ICLR 2023
C ADDITIONAL EXPERIMENTS
C.1 IMAGE CONTENT OBFUSCATION
(a) A photo of frozen New York with N and Y replaced by Greek (U+039D) and Y (U+03A5).
(b) Big Ben in the rain with the replaced by Cyrillie (U+0412).
(c) The Great Wall in fog with replaced by Arabic (U+0627).
Figure 6: Non-cherry-picked examples of hidden objects with homoglyph replacements. We queried DALL-E 2 three times and replaced the underlined characters with different homoglyphs. The in- serted characters are visually barely distinguishable and are rendered almost identically to their Latin counterparts in the DALL-E 2 API.
We start by demonstrating that homoglyph replacements allow malicious user or adversary to pre- vent DALL-E 2 from generating concepts clearly described in the text prompt. For this, the attacker replaces characters in a concept's name with homoglyphs or generally any non-Latin character and forces the generative model to ignore the object in the generated image. In practice, the manipu- lated strings might be shared over the internet to reduce the perceived model quality queried with these image descriptions. A malicious browser plugin website script might further automatically manipulate inserted prompt texts by replacing characters with inconspicuous homoglyphs to render the whole model useless without users noticing. Such attacks might have serious impacts on the profitability of commercial services based on text-guided models.
We visualized some examples in Fig. 6 Taking the prompt Big Ben the rain an exam- ple, we simply replaced the B with the Cyrillic B (U+0412) to remove the Big Ben from the generated images. DALL-E generated various images rainy locations but without any trace of the Big Ben or similar monuments. In most cases, replacing single character is already sufficient for hiding the concept in the generated images. However, when concept description consists of multiple words, we needed to insert a homoglyph in each of the concept's words, see Fig. 6a We experimented with numerous queries and Unicode scripts and found that any object could successfully be hidden by using homoglyphs from any non-Latin script. To verify that DALL-E 2 is generally capable to
generate the stated concepts, we show the created images for benign prompts in Appx. D.7
17

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL DALL-E 2 RESULTS
In this section, we state additional results of our experiments on DALL-E 2. We queried the model using the online API and the prompts specified in the image captions. Note that we queried the model only once with each text description, and stated all four generated results.
D.1 A CITY IN BRIGHT SUNSHINE
(a) Latin A (U+0041).
(h) Greek A (U+0391).
(c) Scandinavian À (U+00C5).
(d) Cyrillie A (U+0410).
(e) Canadian (U+15C5).
(l) Cherokee (U+13AA),
(g) Latin (U+00CO).
(h) Lisu A (U+A4EE).
Mathematical (U+1DSA0).
Figure 7: Non Cherry-picked examples of induced biases with single homoglyph replacement. We queried DALL-E 2 with the following prompt: A city in bright sunshine. Each query differs only by the first character A.
18

---

Under review as a conference paper at ICLR 2023
D.2 A PHOTO OF AN ACTRESS
(a) Standard Latin characters.
(b) Oriya (U+0B66).
(c) Osmanya (U+10486).
(d) Vietnamese (U+IECD).
(e) NoKo o (U+07CB).
(I) Hangul (Korean) (U+3147).
(g) Arabic (U+0647).
(h) Armenian O (U+0585).
(i) Bengali o (U+09E6).
Figure 8: Non Cherry-picked examples of induced biases with single homoglyph replacement. We queried DALL-E 2 with the following prompt: A photo of an actress. Each query differs only by the in of.
19

---

Under review as a conference paper at ICLR 2023
D.3 DELICIOUS FOOD ON TABLE
Standard Latin characters.
(b) Latin e + Cyrillic (U+0435).
(c) Latin + Devanagari (U+0964).
(d) Latin Greek (U+03BF).
Latin Lisu (U+A4F2).
( Latin Tibetan (U+0FOD).
Figure 9: Non Cherry-icced examples of induced biases with single replacemmnlllaemmennt. We queried DALL-E with the following prompt: Delicious food on a table. Each query differs only by single character in the word Delicious replaced by the stated homoglyphs.
20

---

Under review as a conference paper at ICLR 2023
D.4 THE LEADER OF A COUNTRY
Standard Latin characters.
(b) Scandinavian å (U+00E5).
(c) Cyrillic (U+0430).
(d) Greek a (U+03BI).
(e) Latin Ext. á (U+00E1).
(1) Latin Ext. (U+00B0).
C
(g) Latin Ext. à (U+00E2).
(h) Latin Ext. à (U+00E3).
(i) Latin Ext. (U+FF41).
Figure 10: Non Chenry-icked examples of induced biases with a single homoglyph replacement. We queried DALL-E 2 with the following prompt: The leader of a country. Each query differs by the article a replaced by the stated homoglyphs.
21

---

Under review as a conference paper at ICLR 2023
D.5 A PHOTO OF FLAG
Standard Latin characters.
(b) Greek (U+0391).
+-
(c) Scandinavian À (U+00C5).
(d) Cherokee A (U+13AA).
(e) Cyrillic A (U+0410).
Figure 11: Query A photo of a flag. Non Cherry-picked examples flag images with homoglyph replacement generated by the DALL-E 2 model. Whereas the model has learned bias towards generating USA flag, inducing a Greek bias leads to the generation of Greek flags. Surprisingly, using Cyrillic bias enables the model to generate wide range of different flags from European countries.
22

---

Under review as a conference paper at ICLR 2023
D.6 A PHOTO OF PERSON
EALL
SUKKk
U
Smiling face + (U+1F603).
(b) Swearing face (U+1192C).
(c) Crying face
(U+ (F+2)))))
W
(d) Love face (U+1F970).
(e) Screaming face 1F).1...
(f) Celebrating face (U+1F973).
(g) Nerd face (U+1P913).
(h) Monkey Face (U+ (U+1F435).
(i) Bactrian Camel 99 (U+1F42B).
Figure 12: Non Cherry-picked examples of induced biases with a single emoji added. We queried DALL-E 2 with the following prompt: A photo of a X person. Each query differs by adding an emoji at the X position.
23

---

Under review as a conference paper at ICLR 2023
D.7 OBFUSCATING OBJECTS
(a) Frozen New Nork with N and Y replaced by Greek N (U+039D) and Y (U+03A5).
(b) Big Ben in the rain with B replaced by Cyrillic B (U+0412).
(c) The Great Wall in fog with replaced by Arabic (U+0627).
Figure 13: Non Cherry-picked examples of hidden objects with a homoglyph replacement generated by the DALL-E 2 model. The left column depicts images without manipulated prompts.
24

---

Under review as a conference paper at ICLR 2023
E STABLE DIFFUSION RESULTS
In this section, we state additional results of our experiments on Stable Diffusion. We queried the model using a Python script and the prompts specified in the image captions. Note that we queried the model four times with each text description, and stated all four generated results.
E.1 A PHOTO OF AN ACTRESS
(a) Standard Latin characters.
(b) Oriya (U+0B66).
(c) Osmanya (U+10486).
(d) African (U+IECD).
N'Ko (U+07CB).
(f) Hangul (Korean) (U+3147).
(g) Arabic (U+0647).
(h) Armenian o (U+0585).
Bengali o (U+09E6).
Figure 14: Non Cherry-picke examples of induced biases with single homoglyph replacement. We queried Stable Diffusion with the following prompt: A photo of an actress. Each query differs only by the in of.
25

---

Under review as a conference paper at ICLR 2023
E.2 DELICIOUS FOOD ON A TABLE
(a) Standard Latin characters.
(b) Latin e + Cyrillic (U+0435).
(c) Latin + Devanagari (U+0964).
(d) Latin Greek (U+03BF).
Latin
Lisu (U+A4F2).
Latin Tibetan (U+OFOD).
Figure 15: Non Cherry-picked examples of induced biases with single homoglyph replacement. We queried Stable Diffusion with the following prompt: Delicicus food on a table. Each query differs only by single character in the word Delicious replaced by the stated ho- moglyphs.
26

---

Under review as a conference paper at ICLR 2023
E.3 OBFUSCATING OBJECTS
JORZEZZN
FROZEN
(a) Frozen New Nork with N and replaced by Greek N (U+039D) and Y (U+03A5).
(b) Big Ben in the rain replaced by Cyrillic B (U+0412).
W.RAAI veasi Wadll HORE
9p.6t pack
(c) The Great Wall in fog with replaced by Arabic (U+0627).
Figure 16: Non Cherry-picked examples of hidden objects with a homoglyph replacement generated by the Stable Diffusion model. The left column depicts images without manipulated prompts.
27

---

Under review as a conference paper at ICLR 2023
E.4
HOMOGLYPH UNLEARNING RESULTS
Latin-only
Greek
Cyrillic
Arabic
Korean
African
THE
THE AEEN
# man Standard Encoder
THE MAN
THE MAN
The face of
Homoglyph Unlearning
city
Standard Encoder
famous
Homoglyph Unleariing
The most
A traditional. piece of clothing
Standard
Encoder
Hamoglyph
Unlearning
Figure 17: Comparison of image bias and quality of the standard text encoder before and after homoglyph unlearning. We queried each model with three different prompts and five different ho- moglyphs inserted at the position marked by #. The top rows state the images for the standard text encoder, and the bottom rows depict the results after the homoglyph unlearning procedure.
28

---

Under review as a conference paper at ICLR 2023
E.5 INDUCING BIASES IN THE EMBEDDING SPACE
alr
No bias induced.
(b) Oriya (U+0B66).
(c) Osmanya (U+10486).
(d) African (U+IECD).
(e) No O (U+07CB).
(I) Hangul (Korean) O (U+3147).
(g) Arabic (U+0647).
(h) Armenian O (U+0585).
(i) Bengali o (U+09E6).
Figure 18: Non Cherry-picked examples of biases induced into the embedding space. We queried Stable Diffusion with the following prompt: A man sitting at table. We further com- puted the difference between the text embeddings of the stated non-Latin homoglyphs and the Latin character o (U+006F). We then added the difference to the prompt embedding to induce the cultural biases. See Fig. for an overview of the approach.
29

---

Under review as a conference paper at ICLR 2023
F
CLIP EXPERIMENTS
F.1
NON-GENERATIVE MULTIMODAL MODELS BEHAVE SIMILARLY SENSITIVELY
CLIP VIT
CLIP RN50
OpenCLIP
M-CLIP
35-
"A photo
New York"
"A
Similarity
30
of New York"
"A photo York"
"A photo New York"
25-
A photo of New York
"A photo
20
15
Figure 19: Mean similarity scores for querying various CLIP models with real photos of Athens and different captions. The underlined characters in the prompts were replaced with Greek homoglyphs.
To check if other multimodal models are similarly susceptible to homoglyphs, we extended our experiments CLIP. We investigated the official CLIP models (Radford al.) (021) based on
vision transformer (ViT-B/32) (Dosovitskiy et al. (021) and a smaller ResNer-50 (RN50) (He et al.)
2016 both trained on a non-public dataset with 400M text-image pairs. We further used public reimplementation of CLIP (OpenCLIP) (Ilharco et al. (021) trained on the LAION-2B-en dataset, a subset of the public LAION-SB dataset (Schuhmann al. 2022) with English text. Additionally, tested multilingual CLIP model (M-CLIP) (Carlsson al. (2022) with the text encoder based on XLM- RoBERT (Conneau et al., 2020), pre-trained on samples from 100 languages.
We tested how the insertion of homoglyphs influences the cosine similarity scores computed be- tween the image and text embeddings. Fig. 19 shows the mean similarity scores for querying the models with 12 real photos of Athens in combination with the correct description A photo of Athens and multiple variants of the obviously wrong description A photo of New York. As visualized, all four models assign clearly higher similarity scores for the correct image description (left) compared to the false description (right). However, by introducing Greek homoglyphs into the obviously wrong description, we could increase the predicted similarity scores markedly for all models except M-CLIP. We point out that homoglyph replacements are then most effective when replacing parts of the false object, in the depicted case New York. More importantly, our results demonstrate that the text encoder trained on multilingual data is much more robust against homo- graph attacks than the English-only models.
We have also investigated the other direction, trying to obfuscate objects that are correctly described the text. Similar to our DALL-E 2 results Appx. C.I we were able to reduce the similarity scores markedly if we replaced characters directly in the object names. However, M-CLIP again behaves quite robustly against such manipulations. We state visualization for our results in Appx.
30

---

Under review as a conference paper at ICLR 2023
F.2
INDUCING BIASES
CLIP VIT
CLIP RN50
OpenCLIP
M-CLIP
35
photo
of New York
"A photo of New York"
30
"A photo of New York"
Similarity
"A photo of New York"
25
photo of New York
"A photo
20 20-of of Athens"
15
(a) Queries contain 12 real photos of Athens and different captions. Underlined characters the queries have been replaced with Greek homoglyphs.
CLIP VIT
CLIP RN50
OpenCLIP
M-CLIP
35
"A photo
of Berlin"
30
"A photo
Similarity
of Berlin"
"A photo of Berlin"
"A photo Of Berlin"
25
"A photo Berlin" "
photo
20
Saint Basil's Cathedral
15
(b) Queries contain 12 real photos the Saint Basil's Cathedral (Cathedral Vasily the Blessed), and different captions. Underlined characters in the queries have been replaced with Cyrillic homoglyphs.
CLIP ViT
CLIP RN50
OpenCLIP
M-CLIP
35
"Hamburger on plate"
30
"Hamburger on a plate"
"Hamburger on plate"
Similarity
"Hamburger on a plate"
"Hamburger on a plate" "
25
"Banh bao on a plate"
20
15
(c) Queries contain photos of the Banh baos (Vietnamese dumplings), and different captions. Underlined characters in the queries have been replaced with Vietnamese homoglyphs (Latin Extended script).
Figure 20: Mean similarity scores for querying various CLIP models with real photos and different captions. Underlined characters in the queries have been replaced with homoglyphs to force the models' similarity scores to increase.
31

---

Under review as a conference paper at ICLR 2023
F.3
OBFUSCATING OBJECTS
CLIP VIT
CLIP RN50
OpenCLIP
M-CLIP
35
30
Similarity
25
"A photo
"A photo Athens"
of Athens"
"A photo of Athens"
"A photo Athens"
20
"A photo of Athens"
"A photo of Athens"
15
(a) Queries contain 12 real photos of Athens and different captions. Underlined characters the queries have been replaced with Cyrillic homoglyphs.
CLIP ViT
CLIP RN50
OpenCLIP
M-CLIP
35
30
Similarity
25
"A photo of Saint Basil's Cathedral"
"A photo of Saint Basil's Cathedral"
"A photo of Saunt Basil's Cathedral"
20
"A photo of Saint Basil's Cathedral"
"A photo of Saint Basil's Cathedral"
15
(b) Queries contain 12 real photos the Saint Basil's Cathedral (Cathedral Vasily the Blessed), and different captions. Underlined characters in the queries have been replaced with Greek homoglyphs.
CLIP ViT
CLIP RN50
OpenCLIP
M-CLIP
35
30
Similarity
25
"Banh bao on plate"
"Banh bao on plate"
"Banh bao Plate"
"Banh Bao a plate"
20
"Banh Bao on plate"
Banh Bao on a Plate"
15
(c) Queries contain photos of the Banh baos (Vietnamese dumplings), and different captions. Underlined characters in the queries have been replaced with homoglyphs.
Figure 21: Mean similarity scores for querying various CLIP models with real photos and different captions. Underlined characters in the queries have been replaced with homoglyphs to force the models' similarity scores to increase.
32

---

Under review as a conference paper at ICLR 2023
F.4
IMAGE SOURCES
Table 2: Real photos of Athens to test CLIP for biases. See Fig. and Fig. 21a for the experimental results.
Author
License
URL
George E. Koronaios
cC sa 4.0
493489291
Dario Sušanj
cc sa 2.0
pb22GfieeFieeehheers_133339999 jpgkoldd.d=6114749777
A. Savin
Free Art
ppe2:tieerrccctttii..0.1111.... 50
LLe._fro..hiloppppp_ccppppp...., Jpo3611d13=633310555
A. Savin
Free Art
520075507
George E. Koronaios
cc sa 4.0
pbp
Hill =0l.0..._uee33320000000.00000003
George E. Koronaios
cc sa http:://ommoosslluunaaaa...ooggr
Mount_
22.2201...poocoaadee633311111
EftichiaKosma
CC sa 4.0
674126388
George E. Koronaios
4.0
from
George E. Koronaios
cc sa 4.0
Atheessfrommmnn from the
ooaestaa__o_AgrpppppoooooPe__ee on Fn__euar 6, 2020.
jpcsooldid-33222222
George E. Koronaios
cc sa 4.0
Alexander Popkov
CC sa 4.0
of
George F., Koronaios
cc sa 4.0
ThonPPh1ooaapooHHHLLoL.__eeee.._ ipgkolld.d-679723110
33

---

Under review as a conference paper at ICLR 2023
Table 3: Real photos St. Basil's Cathedral to test CLIP for biases. See Fig. 20b and Fig. 21b for the experimental results.
Author Pierre André
License
URL
GFDL
https: https://comnene
Telluride749
cc 4.0
Akuptsova
cc0 1.0
David Crawshaw
3.0
Kafka commonswiki
cc sa 3.0
St. 35._Bes1127g_caatieraaal_ooooo
j=qdcldidc=53179655
CivArmy
cc sa 4.0
htte:://commons.iiicneeaaa...._
481464134
Dmitry Ivanov.
4.0
Bharcla.and.._Bsssccchhheelllll
Moscow..jogcclddds=61133333777
Intel Free Press
cc sa 2.0
662531358
Laika ac
cc sa 2.0
Gaail827s Cathedza2 (6835027756)
1pg3clddid466228677
SteveInLeighton's Photos
cc sa 2.0
jpgkoldid==79921111
A J Butler
cc by 2.0
SergeyStepykin
ce by 3.0
at at_Nint____eorr2moo..440.01
481464564
Dror Feitelson
ce sa 3.0
Linex.pple2titt.eFFele.::.111 Basil
WGccc..J666.oo10-4450000111
34

---

Under review as a conference paper at ICLR 2023
Table 4: Real photos of Banh baos CLIP for biases. experimental results.
See Fig.
20c and Fig.
21c for the
Author
License cc sa 3.0
URL
Sue Le
https:/conmons.. www.:/oooosw..iiiiioo........eee www.sswwwwimmiiiiemmm org/
w/inex,,pip2ii00ee=11eeeeeeaa JPg6olcid-46774832
Tonbi ko
sa 3.0
Tay Son Binh Dinh Ban.piaan.VveenannnJooool....
466708713 https://commos...kksnaalao..../p
Baoothersks
sa 4.0
31sex_pphhtt1llee111ee8c.......1 3. 3JJg66clcccc6660062228
Phuong Huy
cc sa 4.0
bao
SBC3BA__occ332__344550033333221 607927375
C38A0_1
Baoothersks
cc sa 4.0
sttpe://commosss.kiiincaaa....
2.Jp6660ldic--605647778 bao
hxtexx_pphhttt1eeer11e111C31.11 bao
Baoothersks
sa 4.0
Tttess/ccomooss.iiineeeloo..../
Sxcex_pppttt1leee7ll::11331111.1 jppsollcid=6666622941
Phuong Huy
public domain
pttes://commoos..ll.mmmaaaaaaa__
1acex__ppptttileerr1:::BB3BLL. bao
chay
A5t 3SC_nn$$C3sAAA__h83:33000.0 m0000mmm111100 888887 2018 2518
Phuong Huy
public domain
inserx_ppppttiileeeFre:BBBB....n bao
A5t
A5n_ 451_201800000.ppcc00000022233333
Phuong Huy
sa 4.0
BB$9F
Phuong Huy
public domain
1hlSC3ssinn_3333044933.. 2018 (2)
jp1Eoldiid-625012277
Francese Fort
sa 4.0
Ccyber5
public domain
htpp:///commonss.wiiimaeaa.....
sausage buns hagaap__bbbn_hhbyy.ooooooo..1
458685065
35

---

Under review as a conference paper at ICLR 2023
G LAION-AESTHETIC V2 5+ ANALYSIS
We analyzed the text descriptions contained in the LAION-Aesthetics V2 5+ subset of the LAION- 2B-en dataset with 600M samples that have an estimated aesthetics scores 5.0. The LAION-2B-en dataset itself is a subset of the public LAION-5B dataset Seceuhmnn et al. (2022) with English text. For each non-Latin homoglyph (including the extended Latin script), we counted the number of text strings that contain the character.
Table 5: The number of occurrences of different homoglyphs in the LAION-Aestheties V2 5+ dataset.
Character Description
Unicode Number
Count
Arabic Letter Alef
U+0627
65419
Cyrillic Small Letter A
U+0430
989014
Cyrillic A
U+0410
68964
Cyrillic B
U+0412
64789
Cyrillic O
U+041E
81420
Cyrillic Small Letter le
U+0435
802918 18
Devanagari Danda
U+0964
359
Greek Capital Letter Alpha
U+0391
21357
Greek Small Letter Alpha
U+03B1
68729
Greek Capital Letter Beta
U+0392
3469
Greek Small Letter Omicron
U+03BF
44325
Greek Capital Letter Upsilon
U+03A5
5341
Greek Capital Letter Nu
U+039D
7856
Hangul Letter Ieung
U+3147
64
Latin Capital Letter a with Ring Above
U+00CS
19278
Latin Small Letter with Ring Above
U+00E5
87222
Latin Small Letter with Acute
U+00E1
561002
Latin Small Letter with Grave
U+00E0
320653
Latin Small Letter with Circumflex Latin Small Letter with Tilde
U+00E2
93695
U+00E3
123971
Latin Small Letter O with Dot Below
U+IECD
2576
Latin Fullwidth Small Letter A
U+FF41
Lisu Letter I
U+A4F2
Nko Letter Ee
U+07CB
Oriya digit zero
U+0B66
0
Oriya Letter Ttha
U+0B20
Osmanya Letter Deel
U+10486
0
Osmanya Letter Ja
U+10483
Osmanya Letter A
U+10496
0
Osmanya Letter Cayn
U+1048B
Tibetan Mark Shad
U+OFOD
52
Smiling Face with Open Mouth Emoji
U+IF603
1066
Serious Face With Symbols Covering Mouth Emoji
U+1192C
Loudly Crying Face Emoji
U+1F62D
1484
Smiling Face with Smiling Eyes and Three Hearts Emoji
U+1F970
1821
Face Screaming In Fear Emoji
U+1F631
1978
Face with Party Horn and Party Hat Emoji
U+1F973
801
Nerd Face Emoji
U+1F913
558
Monkey Face Emoji Bactrian Camel Emoji
U+1F435
236
U+1B42B
46
36


Reviews:
[{"review_id": "c3yJCtFbA-", "reviewer": "Reviewer_GVzw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the impact of homoglyph replacements of the texts on text-to-image models. Three interesting observations are found:\n\n1) Homoglyph replacements can cause image content obfuscation.\n\n2) Homoglyph replacements can induce cultural bias.\n\n3) Homoglyph susceptibility can be observed in other text-to-image models, e.g., Stable Diffusion. However, the effect is weaker in Stable Diffusion than that in DALLE-2.\n\nThe authors also suggest two simple solutions to avoid biases. One is to replace all characters in the texts with standard characters, the other is to train the language encoders with multilingual texts.", "summary_of_the_review": "I tend to give a score of 5 to this paper.\n\nI think the topic this paper explores is interesting and the analysis of the phenomenon is thorough. My concerns are: 1) the paper didn't dive into the reason behind the phenomenon; 2) The proposed solution is too simple.", "strength_and_weaknesses": "Strengths:\n1) The paper studies the text-to-image models from an interesting perspective.\n2) The findings presented in the paper are interesting.\n3) The authors did a thorough analysis to draw these conclusions.\n\nWeakness:\n1) My major concern is that the proposed solutions are too simple and intuitive. In a paper, we hope to see how to alleviate the problem using a more technically interesting solution rather than merely present the problem.\n2) Reasons for the model behaviour are just verbally explained and they are not verified by the experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OBSKhacBjni", "reviewer": "Reviewer_k5Mu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The shows that current text-to-image models such as DALLE2 and Stable Diffusion produce different images even if only a single character of the text description is changed to a (visually similar) different character (e.g. replacing the latin character \"A\" with the Greek character capital alpha \"Α\"). This behavior also shows certain biases of the underlying models, e.g., generating images coming from the cultural or geographic background based on the replace character (e.g., whether a Latin character is replaced by a Greek or a Arabic character). This behavior allows users to change the model output while seemingly keeping the text input constant (the unicode of the character changes but visually this is not obvious when reading the caption).\nThe authors argue that this behavior may allow attackers to reduce the generation quality of a model or to portrait the model behavior differently by imperceptibly changing the input caption.", "summary_of_the_review": "While the findings are interesting I don't think they contain enough novelty or impact to accept the paper in its current form.\nAlso, I am not sure about the terminology used in the paper, e.g., calling these changes \"adversarial attacks\" or claiming that this behavior is evidence of model bias.", "strength_and_weaknesses": "The paper is very clear and shows several examples for how exchanging characters with similar other characters affects the generate image, either by completely ignoring important parts of the caption (e.g. noun phrases if a single character was changed) or by biasing the output towards different contents based on the origin of the inserted character (e.g. Indian or Vietnamese portrait).\n\nThis is a somewhat interesting finding since it indicates that the models were trained at least with some of these non-Latin characters and learned to associate those with certain contents (e.g. geographic regions).\n\nWhile the finding itself are interesting I don't think they are necessarily surprising. While the characters may look similarly the underlying unicode encoding for those characters is discrete and, therefore, the language model would treat those as independent characters, no matter what they look like visually.\n\nI don't think this can be called an adversial attack though. IMO an adversarial attack is an attack that is a) indistinguishable from a normal \"correct\" input for a human and b) not easy to detect automatically. While a) may be true in this case, the authors themselves state that it is easy to detect these changes simply by checking the unicode encodings of the input string.\n\nAlso, I'm also not sure if calling this behavior \"biased\" is correct. If I ask for a portrait of an \"Indian woman\" getting a portrait of an Indian woman is not biased behavior, but desired behavior. Similarly, replacing a Latin character in the caption with an Indian character can be interpreted as generating an image with Indian influence.\n\nOverall, I feel like this problem (if it really is a problem) is easily addressed by standard preprocessing, similarly to converting all text to lower-case, non-Latin unicode characters can be mapped to their closest unicode characters or, alternatively, the model can be trained on more non-Latin unicodes.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G629T2KLp-", "reviewer": "Reviewer_4KN7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper shows that the quality of the generated images from text-to-image models such as DALLE-2 and Stable Diffusion can be greatly impacted by single homoglyph replacements. A malicious user can replace a character with a homoglyph that looks the same to the human eye and the text-to-image system produces images with cultural biases. The authors show that this behavior is (to a certain extent) consistent between different text-to-image models. Finally, the authors propose multilingual training of the text-encoder as a potential solution to this problem.", "summary_of_the_review": "This paper presents a surprising observation: single homoglyph replacements change dramatically the quality of the samples and lead to biased generations. The fact that generative models amplify societal biases is not new. This paper shows a new aspect of this problem: the sensitivity to small input perturbations. I believe this is an important observation - even if a system appears to have diverse outputs, a small change to the input can drastically deteriorate its performance. \n\nI am not convinced that the attack itself poses a new security problem. However, it broadens our understanding of the current limitations of these models. Therefore, I (weakly) recommend acceptance for this paper. \n\nI encourage the authors to engage in the rebuttal and address the questions raised above.  ", "strength_and_weaknesses": "Strengths:\n\n*  The paper is definitely interesting. The fact that a single letter change can have such a dramatic impact on the generation quality was surprising to me.\n* The authors show that this phenomenon is not specific to one model. There seems to be a more universal issue.\n* The observation that M-CLIP is more robust is very insightful. This suggests that multilingual training could reduce cultural biases present in text-to-image generative models.\n\n\n\nWeaknesses:\n\n* I am not sure about whether this observation poses a new security threat for these models. If a single letter change introduces such cultural biases, I am fairly certain that more obvious changes to the prompt would also trigger culturally biased generations. If the goal of the attacker is to use Stable Diffusion or DALLE-2 to generate disturbing images, I think there are other ways to achieve this as well. So the question becomes, why the homoglyph replacement attack is more dangerous than other types of attacks. I am not sure I have a good answer on that. Since the attacker is the one deciding the prompt in the first place, why is it important to make it look innocent to someone else? How could it be maliciously used? I would love to hear more from the authors on this.\n* I strongly disagree with the potential interpretation of this phenomenon as a feature in Section 5.3. The users of an API should not have their prompt manipulated before entering the system, unless it is explicitly documented or/and there is an option to deactivate it. It is one thing to use filters to catch harmful prompts and another thing to fool the users by presenting results for a modified prompt. But even if the intent is good and the users are informed about how their input was modified, it is still very unclear how the reduction of societal biases would be achieved. What is the right amount of homoglyphs to replace for a \"fair\" generation? How does one ensure that the new generations are not having new biases? The authors rightfully acknowledge that the method works better for specific cultures.\n* The authors propose multilingual training as a potential solution to the observed problem. However, DALLE-2 seems to have been trained on multilingual data. I tried a couple of prompts in Japanese and German and the system successfully generated pictures corresponding to the prompts. So is the claim that the model should have been trained on *more* multilingual data? Or was the text-encoder frozen during the training and we need to train the text-encoder on multilingual data? \n* I am fairly confused about whether the issue is in the Text Encoder, the generative model or both. See also the bullet point above. Would love some experimentation to understand this better.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "07r3akpbHVY", "reviewer": "Reviewer_wkRc", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses an attack scenario for text guided image generation models like dalle2 or text-image joint model like clip. This paper is entirely qualitative and doesn't provide any quantitative metrics. The paper argues that an attack can be made to these models by replacing one alphabet character in an input text with a unicode non-latin character such as a Korean character, which will affect the output image. ", "summary_of_the_review": "This paper proposes to study an interesting behavior of image-text generation/joint models. Due to the lack of scientific measures to validate the argument, this paper is not ready for publication at a venue like iclr. ", "strength_and_weaknesses": "The strength of this paper is that the situation assumed in the paper is very unique. The paper also shows that by using a character in a language, one can obtain an image that belongs to that cultural group. It may be trivial but I don't think anybody has shown it. \n\nSeveral weaknesses\n1. This is a 100% qualitative paper with no attempt to quantify the extent of the problem being discussed. The problem of bias is essentially finding the right (fair, for example) measure for the problem. We can't just cherry pick a few examples and claim that we found something meaningful. This needs to be thoroughly measured and validated.\n\n2. The attack scenario, while novel, doesn't sound reasonable to me. When will this happen? Let's say there's an attacker, a malicious app. A user enters a text input. The app will then modify the query and inject some signals there. But then why would the app show the modified text back to the user using a non standard latin character? The whole point about homoglyphs is that humans can be confused, but not machines. Why wouldn't the app just modify the text more explicitly by adding more words and not show the modified text to users? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject