ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation (2024)

Abstract: Recent studies have shown that Text-toImage (T2I) model generations can reflect social stereotypes present in the real world. However, existing approaches for evaluating stereotypes have a noticeable lack of coverage of global identity groups and their associated stereotypes. To address this gap, we introduce the ViSAGe (Visual Stereotypes Around the Globe) dataset to enable evaluation of known nationality-based stereotypes in T2I models, across 135 nationalities. We enrich an existing textual stereotype resource by distinguishing between stereotypical associations that are more likely to have visual depictions, such as ‘sombrero’, from those that are less visually concrete, such as ‘attractive’. We demonstrate ViSAGe’s utility through a multi-faceted evaluation of T2I generations. First, we show that stereotypical attributes in ViSAGe are thrice as likely to be present in generated images of corresponding identities as compared to other attributes, and that the offensiveness of these depictions is especially higher for identities from Africa, South America, and South East Asia. Second, we assess the stereotypical pull of visual depictions of identity groups, which reveals how the ‘default’ representations of all identity groups in ViSAGe have a pull towards stereotypical depictions, and that this pull is even more prominent for identity groups from the Global South. CONTENT WARNING: Some examples contain offensive stereotypes.

heading: 1 Introduction
text: Text-to-Image (T2I) models are increasingly being used to generate visual content from textual descriptions (Rombach et al., 2022; Ramesh et al., 2021), enabling downstream tasks such as creative design, advertising, marketing, and education. More often than not, these T2I models have been trained on large amounts of web-scale data with minimal
∗ Work done while at Google Research
curation, if any. As a result, these models often reflect and propagate stereotypes about identities and cultures present in the data (Bianchi et al., 2023; Ghosh and Caliskan, 2023; Luccioni et al., 2023).
While these studies provide crucial evidence for fairness failures of T2I generations, they have a noticeable lack of coverage of global identity groups and their associated known stereotypes. Some studies offer qualitative insights across different identity groups but focus on only a limited set of identities and known stereotypes (Bianchi et al., 2023; Qadri et al., 2023). On the other hand, existing larger scale studies focus primarily on stereotypes in the US or Western society (Luccioni et al., 2023), and lack global identities and contexts. Given the widespread adoption of these models in global contexts, it is imperative to build extensive, stratified evaluations that cover broader identity groups to prevent further under-representation of already marginalized groups.
We address the above challenges by presenting a systematic, large-scale, cross-cultural evaluation of regional stereotypes present in the generated images from T2I models. Specifically, we ground our evaluations in an existing geo-culturally broadcoverage, textual resource, SeeGULL (Jha et al., 2023) that incorporates societal stereotypes for 175 nationality based identity groups. This grounding in existing social stereotype resources aids the critical distinction between spurious correlations in models and stereotypical tendencies which are necessary for model safety interventions (Blodgett et al., 2021; Selvam et al., 2023). We enrich this data with human annotations to incorporate the notion of which stereotypes are more likely to be visually depicted versus those that cannot be. Building on this societal grounding, we conduct a study to investigate the extent of stereotypical and offensive depictions of different nationalities in a scalable manner. We also study how the default representation of a nationality relates to their stereotypical as
12333
well as non-stereotypical representations. Figure 1 provides a thematic representation of our approach. Our main contributions are as follows.
• We evaluate Text-to-Image generations for cross-cultural regional stereotypes at scale and with global coverage by leveraging existing resources from textual modality. • We publicly release the dataset ViSAGe1 : Visual Stereotypes Around the Globe where we make a critical distinction between ‘visual’ and ‘non-visual’ stereotypes in images. We identify a list of 385 visual attributes, and also introduce a broad-coverage image dataset with annotations of visual markers of stereotypes present in 40,057 image-attribute pairs, representing different identity groups. • We demonstrate the offensiveness of the generated images and investigate the feasibility of using automated methods employing captioning models to identify visual stereotypes in images at scale. • We analyze the default representations of identity groups and demonstrate how T2I models disproportionately lean towards their stereotypical representations, even when explicitly
1https://github.com/google-research-datasets/ visage
prompted otherwise. The above contributions collectively offer a systematic approach for critically examining the stereotypes in images generated from T2I models.

heading: 2 Related Work
text: Stereotypes in Text-to-Image Models Cho et al. (2023) show that T2I models reflect specific gender/skin tone biases from training data. Fraser and Kiritchenko (2024) examine gender and racial bias in vision language models. Zhang et al. (2023) study gender presentation in T2I models. Ungless et al. (2023) demonstrate that images of noncisgender identities are more stereotyped and more sexualised. Bianchi et al. (2023) highlight the presence of stereotypes by using prompts containing attributes and associating the generated visual features with demographic groups. Stable Bias (Luccioni et al., 2023) prompts T2I models with a combination of ethnicity/gender and profession and evaluates profession-based stereotypes. Basu et al. (2023) use location-based prompts to quantify geographical representativeness of the generated images. Qadri et al. (2023) identify harmful stereotypes and an ‘outsiders gaze’ in image generation in the South Asian context. We prompt T2I models with ‘identity groups’ around the globe to study
the prevalent regional stereotypes in their default visual representation. We identify ‘visual’ stereotypes, related to the concept of ‘imageable’ synsets (Yang et al., 2020), to conduct this analysis at scale.
Several efforts have also been made to quantify the harms associated with generative models. Hao et al. (2023) propose a theoretical framework for content moderation and its empirical measurement. Naik and Nushi (2023) and Wang et al. (2023) quantify social biases in T2I generated images. Garcia et al. (2023) and Gustafson et al. (2023) highlight demographic biases in image captioning and classification tasks. We investigate the feasibility of automatically identifying visual stereotypes in image generations, and demonstrate their offensiveness.
Stereotype Benchmarks in Textual Modality StereoSet (Nadeem et al., 2021) and CrowS-Pairs (Nangia et al., 2020) are datasets used for detecting stereotypes in NLP prediction and classificationbased tasks. SeeGULL (Jha et al., 2023) is a stereotype repository with a broad coverage of global stereotypes – containing close to 7000 stereotypes for identity groups spanning 178 countries across 8 different geo-political regions across 6 continents. It comprises of (identity, attribute) pairs, where ‘identity’ denotes global identity groups, and ‘attribute’ denotes the associated descriptive stereotypical adjective/adjective phrase, or a noun/noun phrase, such as (Mexicans, sombrero), (Germans, practical), and (Japanese, polite). In this work, we leverage resources from textual modalities, such as SeeGULL, to ground our evaluation of prevalent stereotypes in the generated images.

heading: 3 Our Approach
text: Some stereotypes can be recognized in terms of clearly visual attributes (e.g.: ‘Mexicans, sombreros’) whereas other stereotypes are defined through non-visual characteristics such as social constructs, and adjectives (e.g.: ‘Chinese, intelligent’). While the latter may have some visual markers, they are difficult to represent accurately in images. Therefore, we follow a nuanced two-step approach. As a first step, we answer the fundamental question of which stereotypes can be objectively represented in an image. We identify inherently ‘visual’ attributes which we subsequently use in the second step to detect stereotypes in the generated images by (i) conducting a large-scale annotation study, and (ii) using automated methods.

heading: 3.1 Identifying Visual Stereotypes
text: To mitigate subjectivity in visual representation and ensure a more reliable evaluation of the generated images from T2I Models for stereotypes, it is crucial to make a distinction between clearly identifiable ‘visual’ stereotypes and difficult to represent ‘non-visual’ stereotypes. We undertake a systematic annotation task to differentiate the two. We use an existing stereotype resource in textual modality, SeeGULL (Jha et al., 2023), as a reference to ground our evaluations. The dataset has 1994 unique attributes – both ‘visual’ and ‘nonvisual. We deduce visual attributes and then map it back to the identity terms in SeeGULL to obtain visual stereotypes.
Annotating Visual Attributes The annotators are presented with an attribute and a statement of the form of ‘The attribute [attr] can be visually depicted in an image.’ The attribute presented in the sentence is selected from a list of all unique attributes (noun/noun phrases, and adjective/adjective phrases) from SeeGULL. The annotators are required to assign a Likert scale rating to each attribute indicating the extent to which they agree or disagree with the above statement w.r.t. to the given attribute term. We note that the visual nature of attributes lie on a spectrum. While there may be clearly defined attributes on either extremes, the ‘visual’ and ‘non-visual’ attributes in the middle are more subjective. For our subsequent analysis, we exclude all attributes where any annotator expressed uncertainty or disagreement regarding the visual nature. Consequently, we deem terms where all annotators at least agreed about their visual nature, as ‘visual’ resulting in a selection of 385 out of the original 1994 attributes. Please refer to the Appendix 7.1.1 for details. We recruited annotators based out of the United States, and proficient in English reading and writing. For each attribute, we get annotations from 3 annotators that identify with different geographical origin identities - Asian, European, and North American. Annotators were professional data labelers and were compensated at rates above the prevalent market rates.
Mapping to Visual Stereotypes Using the identified visual attributes (Appendix 7.1.1), we filter out the associated ‘visual stereotypes’ in SeeGULL. Figure 2 shows the global distribution of visual attributes. It also contains examples of visual
stereotypes associated with different countries. We observe that around 30 identities, including Omani, Ukranian, Swiss, Canadian, Mongolian, etc., just have a single visual attribute associated with them. Australians (63) have the highest number of visual attributes associated with them followed by Mexicans (46), Indians (34), New Zealand (31), Ethiopians (27), and Japanese (20). The numbers in bracket indicate the number of visual attributes associated with the identity group in SeeGULL. We use these visual stereotypes for identifying their prevalence in images in the next step.

heading: 3.2 Detecting Visual Stereotypes in Text-to-Image Generation
text: Evaluating generated images for the presence of stereotypes is essential to not amplify or reinforce existing biases in the society. We conduct a largescale annotation task to detect prevalent visual stereotypes in the generated images of 135 identity groups across the globe. The scale of our evaluation is significantly larger than the prior work – in terms of the global coverage of identity groups, and their associated stereotypes.
We use Stable Diffusion-v1.4 (Rombach et al., 2022) as the base Text-to-Image model, denoted as M , and generate images representative of various identity groups by defining three distinct prompts as follows: (i) P1(id) = ‘a photo of id person’, (ii) P2(id) = ‘a portrait of id person’, and (iii) P3(id) = ‘an id person’, where id ∈ X such that X is the set encompassing identity terms extracted from the SeeGULL dataset. For each prompt Pi(i = 1, 2, 3) and each identity term id, we generate 5 images, resulting in an output set of images I = {Ii,1, Ii,2, . . . , Ii,5} for a combination of prompt and identity term. The image generation process can be denoted as: Ii,j =M(Pi(id)), where j = (1, 2, . . . , 5) is the generated images for a given prompt Pi and identity term id.
Detecting Stereotypes through Human Annotations The annotation task aims to examine whether well-known visual stereotypes about identity groups are reflected in the generated images. We generate 15 images per identity group and present it to the annotators. Each image is accompanied by two sets of attributes: (i) a set of visual attributes stereotypically associated with the identity group (Figure 2), and (ii) an equal number of randomly selected visual non-stereotypical attributes, i.e., attributes that are not stereotypically
associated with the identity group. We combine the two sets and randomly show 5 attributes in succession alongside each image, until every attribute has been covered. An additional option of ‘None of the above’ is also displayed with each set. The annotators are asked to select all attribute(s) that they believe are visually depicted in the image. Additionally, they are also asked to draw bounding boxes to highlight specific regions, objects, or other indicators that support their selection of the visual attribute within the image. If annotators believe that no attributes are visually represented, they can choose ‘None of the above.’ Overall, we get annotations for 2,025 identity-image pairs and 40,057 image-attribute pairs. Similar to 3.1, we recruited annotators proficient in English reading and writing, residing in United States but identifying with different nationalities. We get annotations for each attribute-image pair from 3 annotators.
Detecting Stereotypes through Automated Methods Annotating stereotypes in images at scale can often times be resource-intensive and time-consuming. Therefore, we also investigate the feasibility of using automated techniques for stereotype detection in images. We use the already existing CLIP (Contrastive Language-Image Pre-training) embeddings combined with the BART model to generate image captions. An image is mapped into the CLIP embedding space, and the embeddings are then scored against a cache of n-grams. The top-k embeddings are then sent to BART to generate candidate captions. We get top 50 captions for each image for the same set of the 15 images per identity group previously generated. Building on the already identified visual stereotypes, we identify the stereotypes in captions by performing a string match. To further understand how uniquely a stereotypical attribute attrs is present in the caption of the images of an identity group, we compute a salience score of the attributes w.r.t. the identity group S(attrs, id). We use a modified tf-idf metric as follows: S(attrs, id) = tf(attrs, id) · idf(attrs, C). The function tf(attrs, id) represents the smoothed relative frequency of attribute attrs for the identity group id; and the function idf(attrs, C) denotes the inverse document frequency of the attribute term attrs in all the captions ‘C’, reflecting the importance of the attribute across all the captions. A higher salience score of an attribute, indicates a more unique association of the attribute with
the identity group. For each identity group, we then extract the most salient associated visual stereotypes as present in the captions.

heading: 4 Study 1: Stereotypical Depictions

heading: 4.1 Stereotypes Identified through Human Annotations
text: Do generated images reflect known stereotypes? We use the identified visual stereotypes for evaluating whether already known stereotypical attributes are visually represented in the images of different identity groups. We only consider identity groups that have more than one associated visual stereotypes for our analysis. We find the likelihood L(attrs, id) of a stereotypical attribute attrs being present in any image of the identity group id as follows. If a stereotypical attribute attrs was shown n times per identity group across all 15 images and selected k times by the annotators, then L(attrs, id) = kn . Figure 3 presents examples of the most likely stereotypes associated with certain identity groups as annotated by different annotators. We observe that the attributes with the highest likelihood for an identity group, also align with the known stereotypes present in SeeGULL. For example, ‘dark’, ‘thin’, ‘skinny’, and ‘underweight’, the most likely attributes depicting Sudanese individuals, are also known stereotypes in SeeGULL. The attribute ‘poor’ which has a likelihood measure of 0.5 for Bangladeshi, i.e., approximately 50% of images representing Bangladeshis contained a representation of ’poor’, is also an annotated stereotype in SeeGULL. Similarly, attributes ‘beaner’, ‘brown’, ‘festive, ‘dark skin’, and ‘sombrero’ are stereotypes in SeeGULL for Mexicans, and are also present in the images representing a Mexican per-
son. An Indian person is often represented as ‘dark, ‘brown’, and ‘religious’ which are its associated visual stereotypes in SeeGULL. We note that the terms ‘brown’, ‘black’ etc., in this work, as identified in images do not imply race, and instead are about appearance, or appearance-based observed race (Hanna et al., 2020; Schumann et al., 2023), or even the colors themselves.
Are some identity groups depicted more stereotypically than others? It is crucial to understand if representations of some identity groups tend to be more stereotypical than others. To analyze this better, we compute the ‘stereotypical tendency’ θid for each identity group. We measure the mean likelihood of a stereotype being present in any image representing an identity group L(stereo, id) and compare it with the likelihood of a randomly selected non-stereotypical attribute being depicted visually in the same set of images L(random, id).
Let L(attrs, id) denote the likelihood of a stereotypical attribute being present in an image, and L(attrr, id) denote the likelihood of a randomly selected non-stereotypical attribute being present in the image corresponding to an identity group id. We select an equal number, k, of stereotypical and random attributes for any given identity group for a fair comparison. The likelihood of an image being stereotypical for a given identity group is, then, denoted as L(stereo, id) = 1k ∑k i=1 L(attris, id), where i denotes the i-th stereotypical attribute associated with the identity group. Similarly, the likelihood of a random attribute being present in the image of the identity group is given by L(random, id) = 1k ∑k j=1 L(attr j r, id), where j denotes j-th attribute selected randomly with the
identity group. Computing the ratio between the two for an identity group
θid = L(stereo, id) L(random, id)
gives the ‘stereotypical tendency’ of the generated images. Higher the ratio, greater is the likelihood of its stereotypical representation.
We observe that on average, the visual representation of any identity group is thrice as likely to be stereotypical than non-stereotypical, i.e., the visual stereotypical attributes associated with an identity group are thrice as likely to appear in their visual representation when compared to randomly selected visual non-stereotypical attributes. We also compute θid for all 135 identity groups. We observe that images representing Togolese, Zimbabwean, Swedes, Danish, etc., only contained stereotypical attributes when compared to random attributes, i.e., θid is infinite or N/A; whereas images representing Nigerians were 27 times more likely to contain stereotypical attributes than randomly selected non-stereotypes. (Please refer to the Tables 1 and 2 in the Appendix for all scores).
How offensive are the depictions of different identity groups? Visual representation of some
stereotypes can be more offensive than others. For example, the stereotypical depiction of an identity group as ‘poor’ can be considered more offensive when compared to the visual depiction of a stereotype ‘rich’. We investigate the offensiveness of images and whether certain identity groups have a more offensive representation compared to others.
Using the offensiveness rating of stereotypical attributes in SeeGULL, we infer an overall offensiveness score O(id) for representation of identity groups. Let O(attrs, id) denote the offensiveness score of a stereotypical attribute attrs associated with an identity group id in SeeGULL. The mean offensiveness score for each identity group is then
O(id) = 1
n L(stereotype, id) ·
n∑
i=1
O(attris, id)
where i denotes the i-th stereotypical attribute attrs identified as being present in the image representing an identity group by the annotators. Figure 8 visualizes the normalized offensiveness score for different identity groups. We observe that the representations of people from countries in Africa, South America, and South East Asia, are comparatively more offensive. Jordanians, Uruguayans, Gabonese, Laotian, and Albanians
have the most offensive representation; whereas Australians, Swedes, Danish, Norwegians, and Nepalese have the least offensive representation.

heading: 4.2 Stereotypes Identified through Automated Methods
text: We check the feasibility of using automated methods employing captioning models for stereotype detection, and compare the results with and without using visual stereotypes as a reference. Without visual stereotypes to ground the evaluations, the automated techniques detect non-visual attributes like ‘attractive’, ‘smart’, etc., for identity groups. However, using visual attributes as a reference, our approach uncovers more objectively visual stereotypes for identity groups. These stereotypes also have a high likelihoodL(attrs, id) of being present in the images as marked by the annotators 44.69% of the time. Figure 3 shows the most salient visual stereotypes associated with the identity group. Attributes like ‘sombrero’, ‘dark’, and ‘brown’ were the most salient visual stereotypes for Mexicans; ‘poor’ was highly salient with Bangladeshi, ‘dark’ and ‘thin’ for Sudanese, and ‘elegant’ for French. These attributes were also marked as being present by the annotators. This approach also identified stereotypical attributes which were not necessarily depicted in the images, e.g., attributes like ‘cow’, ‘elephant’ for Indians. This could be a limitation in our automated approach or existing errors/biases in the generated captions themselves. Further analysis
is required to tease out error and biases propagated by different components of such a system.

heading: 5 Study 2: Stereotypical Pull
text: Prior research has shown that Text-to-Image models can have a ‘very homogenized lens’ when representing certain identities (Bianchi et al., 2023; Qadri et al., 2023). We conduct a deeper analysis to better understand this across the generated images of different identity groups. We define ‘stereotypical pull’ for any identity group as a Text-to-Image model’s inclination to generate images aligning with the stereotypical representations of an identity group when presented with (i) neutral prompts, and (ii) explicit non-stereotypical prompts. This points to the model’s tendency to revert to stereotypical depictions, reflecting its inherent biases.
We use the below sets of prompts, to generate 15 images per prompt for 135 identity groups and demonstrate the prevalence of stereotypical pull.
• Default Representation (d): ‘A/An id person; where id denotes the identity group’. • Stereotypical Representation (s): (i)‘A/An id person described as attrs’, (ii) ‘A photo/potrait of a/an id attrs person’; where attrs is the visual stereotypical attribute associated with the identity group id in SeeGULL. • Non-Stereotypical Representation (ns): (i) ‘A/An id person described as attrns’, (ii) ‘A photo/potrait of a/an id attrns person’; where attrns is a visual attribute not associated with
id in SeeGULL.
Figure 4 visually demonstrates the ‘stereotypical pull’ for the generated images of the identity groups ‘Bangladeshi’, ‘Swiss’, ‘Ethiopian’, and ‘Australian’. For demonstrative purposes, we select opposing stereotypical and non-stereotypical attributes. The default representation of ‘a Bangladeshi person’, looks very similar to the sets of images of ‘a poor Bangladeshi person’ (stereotype), as well as ‘a rich Bangladeshi person’ (nonstereotype). However, the visual representation of ‘a Swiss person’ is quite distinct from the visual representation of ‘a poor Swiss person’ (nonstereotype) and ‘a rich Swiss person’ (stereotype). We observe this pattern even for physical characteristics, where images of ‘an Ethiopian person’, ‘a skinny Ethiopian person (stereotype)’, and ‘a rich Ethiopian person’ (non-stereotype) are all similar looking when compared to that of ‘an Australian person’, ‘a skinny Australian person’ (nonstereotype), and ‘an overweight Australian person’ (stereotype) which are visually quite distinct. The generative models have a tendency to ‘pull’ the image generation towards stereotypes for certain identities when prompted with neutral prompts, and non-stereotypical prompts.
We also compare the stereotypical pull across identity groups from 8 different regions (Jha et al., 2023) and demonstrate this phenomenon quantitatively in Figure 5. We compute the mean pairwise cosine similarity S(·) between the CLIP embeddings of default (d) representation of the identity group with (i) stereotypical (s) representation S(d, s), and (ii) non-stereotypical (ns) images
S(d, ns). The X-axis in Figure 5 represents the difference of stereotyped and non-stereotypes image sets from the default representations (S(d, s)− S(d, ns)). We also compute the similarity between the stereotyped (s) and the non-stereotyped (ns) images S(s, ns), represented by the Y-axis. For 121 out of 135 identity groups, the default representation of an identity group has a higher similarity score with the ‘stereotyped’ images compared to the ‘non-stereotyped’ images indicating an overall ‘pull’ towards generating stereotypical looking images. Moreover, for identity groups from global south, the similarity between stereotypes and nonstereotyped image sets S(s, ns) is also very high, indicating an overall lack of diversity in the sets of generated images.

heading: 6 Conclusion
text: In this work, we present a comprehensive framework for evaluating Text-to-Image models with a focus on ‘regional stereotypes.’ Leveraging existing stereotype benchmarks in textual resources, we perform a three-fold evaluation of generated images. Firstly, we distinguish between inherently visual stereotypes, which can be represented in an image, and ‘non-visual’ stereotypes. Subsequently, we identify visual stereotypes prevalent in the generated images from T2I models across 135 identity groups by conducting a large-scale annotation task. We will publicly release our dataset ‘ViSAGe’ which contains visual stereotypes and the annotated images featuring highlighted visual markers of these stereotypes. Additionally, we quantify the offensiveness of the generated images across differ-
ent identity groups, and investigate the feasibility of automatically detecting stereotypes in images. Through an extensive case study, we also demonstrate that generated images of almost all identity groups exhibit a more visually ‘stereotypical’ appearance, even when Text-to-Image models are explicitly prompted with neutral or non-stereotypical attributes. This phenomenon is more prominent for identity groups from the global south. We hope that the presented approach and the dataset will provide valuable insights for understanding and addressing regional stereotypes in visual content generation.

heading: Acknowledgements
text: We thank Kathy Meier-Hellstern, and Utsav Prabhu for their detailed feedback and Susan Hao for useful discussions. We also thank the anonymous reviewers for the constructive feedback during the review process.
Limitations
Intended Usage Not all attributes are undesirable in a T2I generation context. Determining whether a T2I model should reflect certain attributes associated with specific identity groups is a non-trivial question and can be most accurately determined by the respective developer and the downstream use case, taking into account the type of harm (Shelby et al., 2023) experienced by the end user. Our objective in this work is not to advocate for a stereotype-neutral generation but to equip practitioners with resources and evaluation methods to assesss, on a scalable and effective basis, the extent to which global stereotypes are reproduced in model generations. We also aim to facilitate an understanding of the nature of stereotypes, including their potential offensiveness, through this work. We hope this enables model builders and platforms to better prioritize and appropriately intervene in their model generations based on their use case.
Limitations Although we cover a wide range of visual stereotypes, annotation about the visual nature of attributes is potentially subjective. While we attempt to account for this subjectivity by (1) using a Likert scale as opposed to binary labels, and (2) capture diversity in our annotator pool w.r.t. gender and geographical region, we recognize that our annotations may still fail to capture other dimensions of subjectivity. Moreover, our evaluation of prevalence of stereotypes in the generated images
is limited by the stereotypes present in textual resources like SeeGULL. More regional stereotypes from different textual resources can be included to further expand the overall coverage. We evaluate the images generated using Stable Diffusion V1-4 and believe the results would hold even on other Text-to-Image models, but that needs to be verified in the future work where our analysis may be applied to other generative image models. We chose Stable Diffusion due to its ease of access for academic research, and since our core objective is to demonstrate gaps in existing stereotype bias evaluation approaches, rather than demonstrate biases in any particular or all image generation models. Our work focuses only on regional stereotypes and their presence in the generated images. The proposed framework could be used to experiment with other axes like gender, race, ethnicity, etc., as well. We believe that our approach is extensible to other types of stereotypes, provided such data repositories exist. Future iterations of such data collection and evaluation should take more participatory approach and involve communities with lived experiences on the harms of bias in society.
Ethical Considerations
We evaluate stereotypes in the generated visual depictions of people, based on their identity associated with a geographical location. We acknowledge that this is a complex notion of identity and overlaps with other facets of a person’s identity, such as race, which is not a focus of this work. We also emphasize that this work does not cover all possible visual depictions of stereotypes and thus, it’s methods or dataset should not be used as a standard to deem model generations free of stereotypes. This is an initial step towards a systematic, global-scale evaluations of T2I models for stereotypical depictions, and we hope for future work to build on this to attain more coverage of potential stereotypical and harmful depictions in generations.

heading: 7 Appendix

heading: 7.1 Annotations
text: All annotations were procured through a partner vendor who handled the recruitment, obtained informed consent, and provided clean, anonymous ratings within each task. Annotators were paid above prevalent market rates, respecting minimum wage laws. They were recruited such that every data point was annotated by at least one non-male identifying person. Annotators were also diverse in region of residence.

heading: 7.1.1 Annotating Visual Attributes
text: The Likert Scale labels for the annotation task were as follows:
• Strongly Agree: When the attribute can be explicitly identified within an image, like objects, colors, or similar visual elements (e.g., ‘hat,’ ‘sombrero,’ ‘short’).
• Agree: When the attribute can be deduced from visual cues, albeit not explicitly depicted in the image (e.g., ‘fashionable,’ ‘poor,’ ‘impoverished’). • Disagree: When the attribute is challenging to detect visually but may be inferred from visual cues in specific contexts. • Strongly Disagree: When the attributes cannot be inferred visually, either explicitly or through visual cues, such as ‘kind,’ ‘talkative,’ ‘warmhearted,’ and the like. • Unsure: When annotators are uncertain about the attribute’s visual nature.
We assessed the visual nature of 1994 attributes present in the SeeGULL dataset. Recognizing the potential limitations associated with majority voting (Prabhakaran et al., 2021), we explored different thresholds to determine the degree of visual nature of an attribute. Figure 6 presents one such consensus plot for the ‘visual nature’ of attributes. X-axis demonstrates the consensus labels, and Yaxis indicates the percentage of attributes for which at least 2 out of 3 annotators reached a consensus.
We observe that 20.41% of the attributes had a ‘strongly agree’ consensus rating from at least 2 annotators, suggesting that these attributes were perceived as extremely visual. Some examples of strongly visual attributes include ‘blonde haired’, ‘champagne’, ‘dark skin’, ‘elephant’, and ‘fat’. For 22.16% of the attributes, annotators ‘agreed’ that they were somewhat visual. Attributes like ‘rich, ‘poor’, ‘snake charmer’, and ‘swarthy’ were included in this category. A roughly equal portion, comprising of 20.26% of the attributes were considered non-visual where majority of the annotators ‘disgareed’ about their visual nature. Attributes like ‘professional’, ‘unemployable’, ‘carefree’, ’abusive’, and ‘calm’ were present in this category. Approximately 7.52% of the attributes were deemed extremely non-visual as indicated by the ’strongly disagree’ label. Some examples include ‘good sense of humor’, ‘just’, and ‘unintelligent’; and annotators were unsure regarding the visual nature of 0.2% of the attributes.
For our subsequent analysis, we exclude all attributes where any annotator expressed uncertainty, disagreement, or strong disagreement regarding the visual nature. Consequently, our ‘visual attributes’ dataset consists solely of attributes for which all annotators either agreed or strongly agreed upon their visual nature, resulting in a selection of 385
out of the original 1994 attributes. We categorize these select attributes as ‘visual’ for our study.

heading: 7.1.2 Detecting Visual Stereotypes
text: Figure 7 presents an example of an annotated data point. We release the image, the identity group, the annotated attribute, and the coordinates of the attribute in the image along with a unique annotator ID identifying the annotator of the given imageattribute pair. We also release the data card 2 with more details.

heading: 7.2 Additional Results for Stereotypical Tendency of Identity Groups
text: We compute θid for all 135 identity groups. TableS 1 and 2 presents the θid and the likelihood of images being stereotypical over non-stereotypical for all identity groups.
2https://github.com/google-research-datasets/visage

heading: 7.3 Offensiveness of Visual Stereotypes
text: Figure 8 visualizes the normalized offensiveness score for different identity groups.

heading: 7.4 Additional Results for Stereotype Pull
text: • S(d, s) : Similarity between the default (d)
representation of the identity group with the stereotypical (s) representation. • S(d, ns): Similarity between the default (d) representation of the images with the nonstereotypical (ns) images. • S(s, ns): Similarity between the stereotyped (s) and the non-stereotyped (ns) images.
Tables 3, 4 and 5 present the mean cosine similarity between the sets of images for all identity groups across all stereotyped and non-stereotyped attributes. For 121 out of 135 identity groups, the default representation of an identity group has a higher similarity score with the ‘stereotyped’ images compared to the ‘non-stereotyped’ images indicating an overall ‘pull’ towards generating stereotypical looking images. Moreover, for identity groups from global south the mean similarity score across all three measures is constantly higher indicating an overall lack of diversity in their representations across the three sets.
Identity Group S(d, s) S(d, ns) S(s, ns) Mean Sim
Identity Group S(d, s) S(d, ns) S(s, ns) Mean Sim
Identity Group S(d, s) S(d, ns) S(s, ns) Mean Sim
12347

heading: Future Work
text: We evaluate the images generated using Stable Diffusion V1-4 and believe the results would hold even on other Text-to-Image models, but that needs to be verified in the future work where our analysis may be applied to other generative image models. We chose Stable Diffusion due to its ease of access for academic research, and since our core objective is to demonstrate gaps in existing stereotype bias evaluation approaches, rather than demonstrate biases in any particular or all image generation models. Our work focuses only on regional stereotypes and their presence in the generated images. The proposed framework could be used to experiment with other axes like gender, race, ethnicity, etc. , as well. We believe that our approach is extensible to other types of stereotypes, provided such data repositories exist. Future iterations of such data collection and evaluation should take more participatory approach and involve communities with lived experiences on the harms of bias in society. Ethical Considerations
We evaluate stereotypes in the generated visual depictions of people, based on their identity associated with a geographical location. We acknowledge that this is a complex notion of identity and overlaps with other facets of a person’s identity, such as race, which is not a focus of this work. We also emphasize that this work does not cover all possible visual depictions of stereotypes and thus, it’s methods or dataset should not be used as a standard to deem model generations free of stereotypes. This is an initial step towards a systematic, global-scale evaluations of T2I models for stereotypical depictions, and we hope for future work to build on this to attain more coverage of potential stereotypical and harmful depictions in generations