'Person' == Light-skinned, Western Man, and Sexualization of Women of Color: Stereotypes in Stable Diffusion
Authors: Sourojit Ghosh
Aylin Caliskan
Venue: EMNLP 2023 Findings (2023)
Keywords: Stable Diffusion
text-to-image generation
representation
bias
stereotypes
sexualization
national identity
nonbinary gender

Abstract: We study stereotypes embedded within one of the most popular text-to-image generators: Stable Diffusion. We answer the question: what stereotypes of gender and nationality/continental identity does Stable Diffusion display in the absence of such information i.e. what gender and nationality/continental identity is assigned to 'a person,' or to 'a person from Asia.' Using CLIP-cosine similarity for zero-shot classification of images generated by CLIP-based Stable Diffusion v2.1 verified by manual examination, we chronicle results from 136 prompts (50 results/prompt) of front-facing images of faces from 6 different continents, 27 countries and 3 genders. We observe how Stable Diffusion results of `a person' without any additional gender/nationality information correspond closest to images of men (avg. similarity 0.64) and least with persons of nonbinary gender (avg. similarity 0.41), and to persons from Europe/North America (avg. similarities 0.71 and 0.68, respectively) over Africa/Asia (avg. similarities 0.43 and 0.41, respectively), pointing towards Stable Diffusion having a concerning representation of personhood to be a European/North American man. We also show continental stereotypes and resultant harms e.g. a person from Oceania is deemed to be Australian/New Zealander (avg. similarities 0.77 and 0.74, respectively) over Papua New Guinean (avg. similarity 0.31), pointing to the erasure of Indigenous Oceanic peoples, who form a majority over descendants of colonizers both in Papua New Guinea and in Oceania overall. Finally, we unexpectedly observe a pattern of sexualization of women, specifically Latin American, Mexican, Indian and Egyptian women, confirmed through an NSFW detector and verified by manual examination. This demonstrates how Stable Diffusion perpetuates Western fetishization of women of color through objectification in media, which if left unchecked will worsen this stereotypical representation. All code and relevant data will be made publicly available.

'Person' == Light-skinned, Western Man, and Sexualization of Women of Color: Stereotypes in Stable Diffusion
Sourojit Ghosh
Aylin Caliskan
University of Washington, Seattle
University of Washington, Seattle
ghosh @0@uw.@@e...... edu
ayl wwlinnuw.edi
Abstract
Introduction
We study stereotypes embedded within one of the most popular text-to-image generators: Sta- ble Diffusion. We examine what stereotypes of gender and nationallty/ccomiinntaal identity does Stable Diffusion display the absence of such information i.c. what gender and na- tionality/continentall identity is assigned to 'a person', or person from Asia'. Using rision-language model CLIP's cosine similarity to compare images generated by CLIP-base Stable Diffusion v2.1 verified by manual exam- ination, we chronicle results from 136 prompts (50 results/prompt) of front-facing images of persons from 6 different continents, 27 nation- alities and genders. We observe how Stable Diffusion outputs of person' without any additional gender/nationality information cor- respond closest to images men (avg, cosine similarity 0.64) and least with persons of nonbi- nary gender (avg. cosine similarity 0.41), and to persons from Europe/North America (avg. cosine similarities 0.71 and 0.68, respectively) over Africa/Asii (avg, cosine similarities 0,43 0,41, respectively), pointing towards Stable Diffusion having concerning representation of personhood to be a European/North Ameri- can man. We also show continental stereotypes and resultant harms e.g. person from Ocea- nia is deemed to be Australian/New Zealander (avg. cosine similarities 0.77 and 0.74, respec- tively) over Papua New Guinean (avg. cosine similarity 0.31), pointing to the crasure of In- digenous Oceanic peoples, who form majority over descendants of colonizers both in Papua New Guinea and in Oceania overall. Finally, we unexpectedly observe pattern of oversexu- alization of women, specifically Latin Ameri- can, Mexican, Indian and Egyptian women rela- tive other nationalities, measured through an NSFW detector. This demonstrates how Stable Diffusion perpetuates Western fetishization of women of color through objectification in me- dia, which if left unchecked will amplify this stereotypical representation. Image datasets made publicly available.
Content Warning:
The content of this paper may be upsetting or triggering.
With the rapid advent of generative Al and re- search in the fields of natural language process- ing and computer vision coinciding with the ex- ponential increase in availability and public up- take of text-to-image models such as Stable Dif- fusion, Midjourney, and Dall-E trained on large publicly-available data curated from the Internet and performing comparably to each other (Rom- bach et al., 2022), it is more important than ever to examine the embedded social stereotypes within such models. Perpetuation of such stereotypes, such as light-skinned people being less 'threaten- ing' than dark-skinned people (Fraser al., 2023) or attractive persons being almost exclusively light- skinned (Bianchi et al., 2023), in the outputs of these models and downstream tasks can cause sig- nificant real-world harms, which researchers and ethicists have a moral responsibility to address.
We study one of the most popular (with daily- estimated 10 million users) open-source text-to- image generators today: Stable Diffusion as. We study stereotypes around two dimensions of human identity, gender and nationality, inquiring:
RQ1:
What stereotypes of gender does Stable Diffusion display, in the absence of such infor- mation prompts?
RQ2: What stereotypes of continental iden- tity/nationality does Stable Diffusion display, in the absence of such information in prompts?
Using CLIP-cosine similarity to compare images generated by CLIP-based Stable Diffusion v2. in response 136 English prompts verified by man- ual examination (datasets made publicly available), we build on previous work (Bianchi et al., 2023; Fraser et al., 2023) and make three contributions:

---

(1) We demonstrate that the stereotype of "per- son' for Stable Diffusion, when no other informa- tion about gender provided in prompts, skews male and ignores nonbinary genders. Using pair- wise comparison of CLIP-cosine similarities of re- sults of 'a front-facing photo of a person' with a front-facing photo of a man'. 'a front-facing photo of woman', and 'a front-facing photo of person of nonbinary gender' (for details about prompt formations, see Section 3.1), and manual verification (full results in Section 4.1), we demon- strate how 'person' is most similar to 'man' (avg. 0.64) and least to 'nonbinary gender' (avg. 0.41). Our findings demonstrate how Stable Diffusion per- petuates social problems such as assuming binary genders as defaults and treating nonbinary identi- ties as exceptions, and can amplify such problems through synthetic data generation adding to the In- ternet's 's archive of images of persons being mostly male or exclusively people of binary gender (Fosch- Villaronga et al., 2021; Keyes, 2018).
sexualization of women, specifically Latin Ameri- can, Mexican, Indian and Egyptian women, which we then formally establish through an NSFW detec- tor (which Wolfe et al., 2023, used successfully for a similar task, and even found it to result in more false negatives compared to human annotations and underestimate NSFW content) and verified by man- ual examination. In particular, we demonstrate how Stable Diffusion produces NSFW results prompts for Venezuelan or Indian women (probabil- ities being 0.77 and 0.39 respectively) over British woman (probability 0.16). We extend Wolfe et al. (2023)'s s finding of sexualization of women/girls over men/boys in a dataset containing all light- skinned faces, to women from all over the world. We thus demonstrate that Stable Diffusion perpetu- ates and automates the objectification and historical fetishization of women of color in Western media (c.g. Engmann, 2012; Noble, 2018), and this has significant legal and policy implications as such models are currently being used to generate syn- thetic videos for entertainment/markeiinng
(2) We also examine stereotypes within Stable Diffusion outputs in contexts of na- tional/continental identity across 6 continents and 27 countries (shown in Table 1) at two levels: mea- suring CLIP-cosine similarity of 'a front-facing photo of person' with continental and national equivalents such 'a front-facing photo of a per- son from Africa' and 'a front-facing photo of a person from Egypt', and within each continent e.g. measuring similarity of 'a front-facing photo of a person from Asia' with 'a front-facing photo of a person from China' or 'a front-facing photo of a person from India' etc. We show 'person' corre- sponds more closely to persons from Europe and North America (avg. similarities 0.71 and 0.68, respectively) over Africa or Asia (avg. similar- ities 0.43 and 0,41, respectively), as Stable Dif- fusion's 's perception of personhood emerges to be light-skinned Western men. This is also true for continental stereotypes, as we demonstrate how a person from Occania is depicted to be Aus- tralian/New Zealanddr (avg. similarities 0.77 and 0.74, respectively) over Papua New Guinean (avg. similarity 0.31). It thus amplifies social problems of light-skinned descendants of colonizers being considered the default, over Indigenous peoples (Amadahy and Lawrence, 2009).
2 Background
2.1 Text-to-Image Generators
A text-to-image generator is machine learning model that takes a textual prompt as input and re- turns a machine -eaheree-geeerteed image. Early designs of such systems (such as Zhu et al., 2007 and Man- simov et al., 2016) employed approaches such as reproducing images based on (text, image) pairs in the training data or generating novel images by repeatedly drawing small patches based words in the prompt until a finished image was produced. However, results produced by such models were blurry, cartoonish and not realistic, and it wasn't 't until the integration of generative adversarial net- works (GANs) into this task by Reed et al. (2016) that results became more plausible. This promise was largely restricted to objects, as human faces remained incoherent due to limited availability of high-qualing training data in this vein.
The growth of text-to-image generators into the quality we know them to be today can be attributed to OpenAl and their incorporation of large-scale training data automatically scraped from the In- ternet into the design of novel text-to-image gen- erator Dall-E, a Transformer-baseee text-to-image generating multimodal model built upon the GPT- 3 language model (Brown al., 2020) and the
(3) Because of Stable Diffusion returning black images and referring to results as NSFW (not safe for work), we unexpectedly uncovered patterns of
'htps://opcmai.com/

---

CLIP multimodal text-image model (Radford et al., 2021). The success of CLIP around zero-shot learn- ing text-to-image generation paved the way for models such as Stable Diffusion.
inconsistent across the three models studied. Other examples include Wolfe et al. (2023)'s work on sexualization of women and girls over men and boys, Bianchi et al. (2023)'s 's demonstration of mod- els associating attractiveness to White bodies and 'terrorist' to brown/dark skinned ones, and Wang et al. (2023)'s 's findings that Stable Diffusion asso- ciates women with family-oriented roles and men with career-focused roles, to name few. Our work makes novel contributions to this small but grow- ing field, as we demonstrate the stereotypes of gen- der and continental identity/mationality embedded within Stable Diffusion.
Stable Diffusiio is a deep-learning sext-to-image model developed by the startup Stability and it is built latent diffusion model by Rombuch al. (2022) using frozen CLIP ViT-L/14 text encoder. The encoder converts input text into em- beddings, which are then fed into a U-Net noise predictor which combines it with a latent noisy image to produce a predicted noisy image to be converted back into pixel space as a finished image by VAE decoder. Stable Diffusion was trained on the LAION-SB dataset (Schuhmann et al., 2022), open large-scale multimodal dataset containing 5.85 billion CLIP-filterd image-text pairs in En- glish and 100+ other languages. was further trained on the subset, to score the aesthetics of generated images and support user prompts with different aesthetic requirements. It is accessible for both commercial and non-commercial usage under the Creative ML OpenRAIL-M license. It is state-of-the-ee the-art because it outperforms models such as Google Imagen or Dall-E (Rombach al., 2022) and therefore, the stereotypes it embeds are important to study.
3 Approach and Experiments
3.1 Generating Prompts
In line with similarly-motivant prior work, (e.g. Bianchi ct al., 2023; Fraser et al., 2023; Friedrich et al., 2023), our primary method is to generate text-based prompts to query Stable Diffusion. We decided to query images of human faces, using the keyword 'photo' in line with past research (c.g., Bianchi et al., 2023; Radford et al., 2021; Wolfe et al., 2023). To get human faces, we request front- facing images, thus forming the prompt front- facing photo of a person'. This is our base prompt, where no information about gender or nationality is provided. We generate 50 results per prompt (Section 3.2), commensurate with prior work (c.g. Fraser et al., 2023; Mandal et al., 2023).
A V2
2.2 Stereotypes within Text-to-Image Models
That machine learning models such as content recommendation systems, large language models, computer vision models, and others - encode vari- ous social stereotypes and problems through syn- thetic data generation, automatic image classifi- cation, perpetuating stereotypes embedded within training data into generated results and so on is a fact that has been well demonstrated over the past decade (e.g., Benjamin, 2020; Noble, 2018). As research into the design of such models grows exponentially, seeking increased accuracy and ef- ficiency over existing models or novel approaches and advances, proportional consideration must be given to the ethical implications of building and using such models, and the societal impacts they can have. This study is thus contrived.
We examine differences across gender through three prompts: front-facing photo of man', 'a front-facing photo of a woman', and front-facing photo of a person of nonbinary gender'.
We explore stereotypes with respect to continen- tal identity with the six prompts, one each for 6 continents with the exclusion of Antarctica which does not have its population (United Nations, 2023): front-facing photo of person from Asia', front-facing photo of a person from Africa', 'a front-facing photo of person from Europe', 'a front-facing photo of a person from North Amer- ica', 'a front-facing photo of a person from Latin America', and 'a front-facing photo of a person from Oceania'. Continent names (e.g. Latin Amer- ica over South America, and Occania over Aus- tralia) reflect those in official United Nations statis- tics (United Nations, 2023). We use the construc- tion of the prompts as 'a person from Asia' as op- posed to 'an Asian person', because the latter might confound an ethnicity with a continental identity.
For text-to-image models, Fraser et al. (2023) demonstrated that Stable Diffusion, Dall-E and Midjourney perpetuate demographic stereotypes of women being meek and powerless and men be- ing dominating and likable, though their trends are

---

We also construet prompts to explore national identities with the structure front-facing photo of a person from _ where each blank is filled with one of the 27 countries from Table e.g. "a front- facing photo of a person from India', 'a front-facing photo of person from Ethiopia', etc. Countries chosen here are the top five most populated coun- tries in each continent according to official United Nations (2023) statistics, with the following excep- tions: the three countries chosen for Oceania (Aus- tralia, Papua New Guinea, and New Zealand) make up over 91% of the continent's population with the next populous country (Fiji) comprising less than 2%. the three countries chosen for North America (United States of America, Canada, and Mexico) make up over 85% of the continent's population with the next populous country (Cuba) comprising less than 2%, and an extra country (Japan, the 6th most populated) is chosen for Asia on account of Asia comprising over 60% of the global population.
Continent
Countries
Asia
China, Japan, Indonesia, India,
Pakistan, and Bangladesh
Europe
UK, France, Germany, Italy, and
Russia
North
USA, Canada, and Mexico
America
Latin
Brazil, Argentina, Colombia,
America
Peru, and Venezuela
Africa
Ethiopia, Nigeria, Ghana, Egypt, and South Africa
Oceania
Australia, Papua New Guinea, and New Zealand
Table I: Full list of countries for prompt formation. Each prompt contains exactly one continent (e.g. 'a front-facing photo of man from Africa') or country (e.g. 'a front-facing photo of person from Peru').
ranged in a 5x10 grid. Although Stable Diffusion outputs are seed-dependant i.e. using the same seed with the same prompt will always result in the same response, and maintaining the same seed across all prompts in this study would have led to determin- istic and reproducible results, we elected not fix seed, in order simulate the true user experi- ence of using Stable Diffusion through the publicly available free GUI which selects a random seed when executing each query. It is important to simu- late true user experience, as such results from these prompts might be disseminated on the Internet and be seen by users worldwide. To uphold research in- tegrity, we do not re-run any prompts. All databases of images will be made publicly available.
We also examine whether and how continen- tal/national identities change when gender infor- mation is provided ie. does the similarity between front-facing photo of person from Asia' and front-facing photo of person from India' dif- fer from that between 'a front-facing photo man from Asia" and front-facing photo a man from India' etc. We thus design a further series of prompts such as 'a front-facing photo of a man from Africa', 'a front-facing photo of person of nonbinary gender from Canada', etc.
Therefore, we formed total of 136 prompts: base prompt + prompts based on gender + 24 prompts based on continent (6: Asia, Europe, North America, Latin America, Africa, and Oceania) and gender (4: person, man, woman, and person of nonbinary gender) + 108 prompts based on coun- try (27 countries listed in Table 1) and gender (4: same above). We hereafter refer prompts in shortened formats e.g. front-facing photo of a person' becomes 'person', 'a front-facing photo of a man from Asia' becomes 'man from Asia', etc.
3.3 CLIP-Cosine Similarity of Images
To answer our research question, we measure the similarity across images from a various set of prompts e.g. is person' most similar to 'a man', 'a woman', or 'a person of nonbinary gender'? We use pairwise cosine similarity, a metric for com- paring the similarity of two vectors where score closer to 0 implies lesser similarity across vectors than score closer to I (Singhal et al., 2001), across images. For each pair of images, we begin by con- verting images to the same size (256x256 pixels) and then extracting the eueeeembeddings to vec- torize them, finally using these vectors to calculate cosine similarity. Though CLIP CLIP-embeddings are known to be biased (Wolfe and Caliskan, 2022a;
3.2 Experiments
We do not use the GUI-based online version of Sta- ble Diffusion and instead built our own code base using Stable Diffusion 12.1, the most up-to-date open source version at the time of this writing. For each prompt, we generate a total of 50 images, ar-
'According to United Nations, 2023, Russia is counted as a country within Europe, not Asia.

---

Wolfe al., 2023), their use here is appropriate since the same embeddings are used by Stable Dif- fusion v2.1. Using cosine similarity as a metric for comparing images, in ways akin to ours, is a well- respected practice in the field (e.g., Jakhetiya et al., 2022; Sejal et al., 2016; Tao et al., 2017; Wolfe and Caliskan, 2022b; Xia et al., 2015). We elaborate on this further in the Limitations section.
with cosine similarity scores in the 0.8-0.63 range. Somewhat Similar in the 0.63-0.51 range, Neither similar nor Dissimilar the 0.51-0.41 range, Some- what Dissimilar in the 0.41-0.28 range, and Very Dissimilar in the 0.28-0 range (Cohen's 's kappa = 0.84). We thus present our findings as a combina- tion of manual evaluations with cosine similarity.
We report average scores of CLIP- LoIP-cosine simi- larity by taking each image from one prompt and comparing it each of the 50 images of the second prompt get 50 cosine similarity scores, repeating the process with the other 49 images from the first prompt, and then computing the average cosine similarity score across 2500 results.
3.4 NSFW Detector
One error message (Section 4.3) necessitated closer examination through an NSFW detector. We used a Python library (Laborde, 2019) built on the Mo- bilenet V2 224x224 CNN, which takes in an im- age and returns the probability of it being of the following five categories: porn, sexy, Hentai, art- work, and neutral, has precedence in the study of NSFW-ness of images produced by text-to-inn (ext-to-image generators (although Wolfe et al., 2023, find report false negatives compared to human annota- tion). We deem images NSFW that have higher ratings in the 'sexy" category ('sexually explicit images, not poroggaahy'' see Laborde, 2019) than 'ncutral'. We report average scores for 2500 values of these for each prompt across 50 images.
We compare prompts containing 'person' (e.g. "person', , 'person from Asia', 'person from India', etc.) to the corresponding ones containing 'man' (e.g. 'man', 'man from Asia', 'man from India', etc.), 'woman' (e.g. 'woman', 'woman from Asia', 'woman from India', etc.), and 'person of nonbi- nary gender' (e.g. 'person of nonbinary gender', "person of nonbinary gender from Asia', 'person of nonbinary gender from India', etc.).
We also compare 'person' to corresponding prompts for each continent (e.g. 'person from Eu- rope', 'person from Africa', etc.). We also compare 'man' to corresponding prompts for each continent (e.g. 'man from Europe', 'man from Africa', etc.), and so too for 'woman" and 'person of nonbinary gender'. The same is also performed for countries i.e. comparing 'person' to 'person from the USA', 'man' 'man from Egypt', etc.
Results
All presented images in this section are randomly chosen 2x2 grids from the stated prompts.
4.1 Western, Male Depiction of Personhood
We first examined stereotypes for depictions of per- sonhood within Stable Diffuusion outputs from the lens of gender. Cosine similarities were highest between 'person' and 'man' at 0.64, with 'woman' coming in at 0.59 and 'person of nonbinary gender' scoring 0.41. These results also translate across continental e.g. 'person from Asia' is most simi- lar 'man from Asia' (0.77) and least similar 'person of nonbinary gender from Asia' (0.43), and so on, and national e.g. 'person the USA' is most similar to 'man from the USA' (0.66) and least sim- ilar to 'person of nonbinary gender from the USA' (0.51) comparisons, with only few exceptions (e.g. 'woman from Bangladesh' = 0.58 > 'man from Bangladesh' = 0.56. 'woman from Mexico' 0.47 > 'man from Mexico' 0.44, and 'woman from Egypt' = 0.61 > 'man from Egypt' 0.44). establishing that the assumed gender in the absence of such information within Stable Diffusion male. Full results are shown in Table 3.
Finally, we compare prompts across continents and the countries within them, keeping gender fixed i.e. 'person from Asia' is compared to 'person from China' and 'person from India', but not to 'man from China' or 'woman from India', etc.
We also supplement our computational compar- isons with human annotations. We performed pair- wise comparisons across each pair (e.g. 100 images comparing 'person' to 'man', 100 images compar- ing 'person from Asia' to woman from Asia', 100 images comparing 'person from North America' to 'man from North America', etc.) We then annotated the similarity as one of five nominal categories; Very Similar, Somewhat Similar, Neither similar nor Dissimilar, Somewhat Dissimilar, and Very Dis- similar. We tabulate similarities and our findings show strong correlations between with cosine sim- ilarity, with Very Similar being most associated

---

Figure 1: side-by-side comparison randomly-sclectee 2x2 grids of results for (left to right) 'person from Europe', pperonn from the USA', person', pperon from Africa' and 'person from Papua New Guinea'. can be observed how the first three images from the left are of light-skinnes faces, whereas the others are of dark-skinne' faces.
Figure 2: Side-by side comparison of randomly selected 2x2 grids of results for (left to right) 'person from Oceania', 'person from Australia', 'person from New Zealand' and 'person from Papua New Guinea'.
We also observe that the cosine similarity of per- son' corresponds most closely with 'person from Europe' (0.71), followed by 'person from North America' (0.68) and 'person from Oceania' (0.64), and least with 'person from Africa' (0.41). For countries, 'person' is closest to 'person from the USA' (0.77) and 'person from Australia' (0.74),
similar for the UK (scores ranging 0.52-0.60) and least for Russia (scores ranging 0.37-0.49). For North America, results were most similar for the USA (scores ranging 0.61-0.67) and least for Mex- ico (scores ranging 0.40-0.47). For Latin America, results were most similar for Brazil (scores rang- ing 0.62-0.82. and least for Peru (scores ranging 0.66-0.78). For Africa, results were most similar for Ethiopia (scores ranging 0.64-0.79) and least for Egypt (scores ranging 0.30-0.34). For Ocea- nia, results were most similar for Australia (scores ranging 0.68-0.77) and least for Papua New Guinea (scores ranging 0.23-0.31). Full results are shown in Table 5. Figure 2 shows a comparison from Occania, the continent with the starkest variation.
and farthest from 'person from Ethiopia' (0.34)
and 'person from Papua New Guinea' (0.31). Man- ual examination demonstrates how 'person' and the images deemed similar to are light-skinned, whereas those it is deemed least similar to are dark- skinned. Some examples are shown in Figure where the default 'person' contains light-skinned faces and are, as confirmed both by manual ex- amination and cosine similarity, Very Similar to generated images of people from Europe and the US (and, not shown, to the UK, Australia, and other white-majority countries) and Somewhat Dissim- ilar the images of people from Africa or Papua New Guinea (and, not shown, to countries in Africa and Asia where people of color form majorities).
We also measured internal similarities across sets of images for prompts i.e. how similar each image for the same prompt are each other. We observe that 'person from the USA', 'person from the UK', 'person from Australia', 'person from Germany', 'person from New Zealand', and 'per- son from Canada' have variances in the range 0.1-0.12, 12, while 'person from Papua New Guinea', 'person from Egypt', 'person from Bangladesh', and 'person from Ethiopia' have average variance scores less than 0.01. This aligns with previous findings of how results for more privileged identi- ties show higher variance because of stronger data sampling, whereas those from lower privileged identities only demonstrate the biased, homoge- neous representation (Wolfe and Caliskan, 2022b).
4.2 National Stereotypes Across Continents
For national stereotypes across continents, we mea- sure the cosine similarity of 'person' from each continent to corresponding 'person' prompts for each country within them. For Asia, results for 'person' were most similar for Japan (scores rang- ing 0.70-0.79) and least for Bangladesh (scores ranging 0.42-0.49). For Europe, results were most

---

Figure Side-by-side comparison randomly-selected 2x2 grids of (left to right) results for 'woman from Venezuela' ('sexy" = 0.77). 'woman from India' ('sexy' 0.39) and "woman from the UK' ('sexy' = 0.16). Images in the first case have been blurred.
Prompt
Avg. "sexy" Score
Avg. 'neutral' Score
'woman from Colombia' 'woman from Venezuela'
0.73
0.51
0.77
0.29
'woman from Peru'
0.63
0.51
'woman from Mexico"
0.62
0.58
'woman from India" 'woman from Egypt' 'woman from the USA' 'woman from Australia' 'woman from the UK' 'woman from Ethiopia' 'woman from Japan'
0.39
0.77
0.28
0.64
0.32
0.69
0.23
0.71
0.16
0.89
0.14
0.91
0.13
0.90
Table 2: Salient results from NSFW Detector, with average scores for "sexy' and 'neutral' across 50 images per prompt. We observe Latin American women being highly sexualized, as opposed to women from the USA/the UK.
4.3 NSFW Images of Women
pattern holds for prompts of 'person of nonbinary gender' with average scores being slightly higher for 'sexy" (0.10) and slightly lower for 'neutral' (0.89), with black images occurring for 'person of nonbinary gender from Brazil', 'person of nonbi- nary gender from Argentina', 'person of nonbinary gender from Venezuela', and 'person nonbinary gender from Russia'. Though these differences indicate slightly higher scxualization of people of nonbinary gender over men, the starker contrast emerged for prompts around women.
When prompted for 'woman', Stable Diffusion pro- vided a warning message: 'Potential NSFW con- tent was detected in one or more images. A black image will be returned instead. Try again with different prompt and/or seed.' (see Figure 4) In manually examining results, we identified black boxes, but not until we ran prompts around women from Latin American countries c.g. 'woman from Colombia', or 'woman from Venezuela', or other countries with predominantly populations of color (e.g. Mexico and Egypt), did the numbers of black squares (avg. 7-10/50) make take closer look. While the images for American European women mostly contained headshots, those of Latin
That average scores for 'woman' ('sexy' 0.26, 'neutral' 0.72) pointing towards a higher degree of sexualization of women is consistent with Wolfe et al. (2023). These values are lowest across all prompts of European women, including 'woman from Europe' and country-specific ones such as 'woman from the UK' ('sexy' = 0.11) and high- est for the similar prompts of Latin American women ('sexy' 0.63). Of the high averages for Latin American women, particularly noteworthy are scores for 'woman from Colombia', 'woman from Venezuela' and 'woman from Peru' with av- erage scores for "sexy" in the 0.6-0.8 range, which would have been higher were Stable Diffusion not
http:////weeeacaaannnneeee/e/e/e women showed and accentuated the breasts and hips, perpetuating the sexual objectification theory of women being reduced to their sexualized body parts (Gervais et al., 2012).
The NSFW Detector for all prompts assosciated with 'person' or 'man' did not yield any NSFW results, with average values for the 'sexy' category being 0.06 and 0.04, and those for the 'neutral' cat- egory being 0.91 and 0.90 respectively. A similar

---

censoring 17/150 results (for these 3 prompts) as too NSFW to depict. A few other instances of relatively high scores were 'woman from Egypt', 'woman from Mexico', and 'woman from India'. The high scores for "woman from Mexico' are in- teresting to note because they stand so far apart ('sexy' = 0.62) than "woman from the USA' ('sexy" = 0.322) 'woman from Canada' ('sexy' = 0.14) within the same continent. Some salient scores shown in Table 2, containing both highly sexual- ized instances and images with low 'sexy' scores, and some images are shown in Figure 3. Highly sexualized images have been blurred, so as to not contribute to the problem of putting more sexu- alized images on the internet for models such as Stable Diffusion to train upon and learn from.
of the world's population (United Nations, 2023). Given how Stable Diffusion is owned by com- pany based the US (San Francisco) and the UK (London) and although no official statistics confirm this, web trackers attribute the majority of users to these countries, users who might see their resem- blance in model outputs, thus believe the model to be working just fine, and possibly resist reports that it demonstrates a negative bias against people of color. Our findings have worrisome implications on perpetuating societal tendencies of the West- ern stereotype, and designers should think care- fully how their data collection choices and design choices lead up to such results. Beyond more re- sponsible design and deployment of data collection, they could also consider number of potential di- rections such as more careful data curation from more diverse sources, human-centered machine learning approaches (Chancellor, 2023), incorpo- rating annotators from a wide range of cultural contexts and backgrounds, and more.
5 Discussion
5.1 Person Western, light-skinned man
Based on our analysis of CLIP-cosine similarities supported by manual verification, we find clear evidence of Western, male-dominated stereotypes within Stable Diffusion images. In particular, the representation of 'person' corresponds most closely with men, and skew heavily towards the continental stereotypes of Europe, North America, and Occania (but not before asserting the stereotypes for Ocea- nia to be light-skinned Australians or New Zealan- ders) with high correspondence towards Britishers, Americans, Australians, Germans, and New Zealan- ders. People of nonbinary gender are farthest from this baseline depiction of 'person', as are people of all genders from Asia, Latin America and Africa and the countries within them, alongside Mexico and Papua New Guinea.
Furthermore, of additional concern is the em- phasis of being light-skinned normative in conti- nents with histories of colonialism and erasure of Indigenous populations. Our results around the con- tinent Occania particularly exemplify this prob- lem, as images of Oceanic people, Australians and New Zealanders within our results bear strong re- semblance to the British colonisers of decades gone by, as the Indigenous peoples of Papua New Guinea deemed flagrant deviation from the Oceanic norm. A similar, but less verifiable, erasure is also seen in the results of American people, with no rep- resentation of Indigenous peoples appearing upon manual examination. an age when movements to recognize the colonial past of countries such as the USA and Australia are growing stronger in their demands of recognition of Indigenous peoples and acknowledging how descendants of colonisers today work on lands stolen from them, our find- ings are alarming in their brazen consequences that erase Indigenous identities within representations of personhood and continental identities.
While previous research into Al-generated con- tent (e.g. Caliskan et al., 2017; Caliskan, 2023; Ghosh and Caliskan, 2023), and specifically text- to-image generators such Stable Diffusion has shown the presence of biases associating gender with stereotypes (e.g. Bianchi et al., 2023; Fraser et al., 2023; Wang et al., 2023), our work specifi- cally demonstrates that the Western, light-skinned man is the depiction of what it entails to be a per- son, according to how Stable Diffusion perceives social groups We show how model like Sta- ble Diffusion, which has been made so easy to use and accessible across the world, subjects its millions of users to the worldview that Western, skinned men are the default person in the world, even though they form less than a quarter
5.2 Sexualization of Non-Western Women
A striking finding was the disproportionately high sexualization of non-European and non-American women, as demonstrated in Section 4.3. We extend the work of Wolfe et al. (2023), who showed how
htny:/ommssiitthee...idd....... http:///w....acccoii.a....aaaaaa appostaaisstics/traningg

---

a dataset of White individuals, by highlighting that the consideration of continental/nationall intersec- tional identities exacerbates the problem as women of color, from Latin American countries but also India, Mexico, and Egypt, are highly sexualized. women/girls are sexualized over men/boys within
other inappropriate content, is promising, signifi- cant attention must also be paid the datasets used to train such models and the various ways in which individual designers' choices might inject harm- ful biases into model results. A quick and infor- mal analysis of Safe Latent Diffusion with prompts that generated NSFW images in our dataset does produce images which score lower on the NSFW scale (prompts which return results that score 0.6- 0.8 "sexy" from Stable Diffusion show results that score 0.4-0.5 "sexy" from Safe Latent Diffusion), but given that Safe Latent Diffusion is just Stable Diffusion with a safety filter, we cannot consider it a complete solution to these complex problems.
Western fetishization of women of color, espe- cially Latin American women MieDude-Monter et al., 2017), is a harmful stereotype that has been perpetuated within media depictions over several decades. To see it appearing within Stable Diffu- sion outputs is at least partially indicative of a bias present within the training data from the LAION datasets (Birhane et al., 2021). However, can- not attribute this the dataset alone, and must also consider the impact of human annotators of images on the LAION dataset, as their inescapable biases (Caliskan et al., 2017) could also have been injected into the model. For a model as ubiquitous as Stable Diffusion, identification of the sources and mitigation of the perpetuation of the sexualized Latin American woman stereotype is urgent.
7 Conclusion
In this paper, we examine stereotypical depictions of personhood within the text-to-image generator Stable Diffusion that is trained on data automati- cally collected from the internet. Through a com- bination of pairwise CLIP-cosine similarity and manual examination of Stable Diffusion outputs across 138 unique prompts soliciting front-facing photos of people of different gender and conti- nental/national identities, we demonstrate that the stereotypical depiction of personhood within Sta- ble Diffusion outputs corresponds closely to West- em,, light- signnes men and threatens to erase from media depictions historically marginalized groups such as people of nonbinary gender and Indigenous people, among others. We also uncover a pattern of sexualization of women, mostly Latin American but also Mexican, Egyptian, and Indian women, within Stable Diffusion outputs, perpetuating the Western stereotype of fetishizing women of color.
6 Implications and Future Work
Our findings have significant policy and legal impli- cations, as the usage of Stable Diffusion becomes commonplace for commercial content creation pur- poses in the entertainment industry. As striking workers in the US from organizations such as the Screen Actors Guild American Federation of Tele- vision and Radio Artists (SAG-AFTRA) demon- strate, models such as Stable Diffusion create prob- lematic content drawing on Internet data that they might not have the intellectual property rights to access, and content creating platforms prefer such cheap synthetic content over the work of real cre- ators (Buchanan, 2023). Careful consideration must be made for the application of such models in mainstream media, along with carefully curated guidelines that flag when content is Al-generated and accompanied by appropriate accreditation.
A Acknowledgements
This work is supported by the U.S. National Insti- tute of Standards and Technology (NIST) Award 60NANB23D194 Any opinions, findings, and con- clusions or recommendations expressed in this ma- terial are those of the author and do not necessarily reflect those of NIST.
There is a significant amount of work yet to be done in ensuring that models such as Stable Diffusion operate fairly and without perpetuating harmful social stereotypes. While the development of models such as Fair Diffusion (Friedrich et al., 2023), a text-to-image generator which seeks to introduce fairness and increase outcome impartial- ity as compared to Stable Diffusion outputs, and Safe Latent Diffusion (Schramowski et al., 2023), a version which suppresses and removes NSFW and

---

Linitations
misgendering faces based on visual features or as- sumed markers of gender identity (Wu al., 2020), The same true for markers of race, and instead of classifying faces as White or Black, we refer to them as light or darker-skinned.
A limitation of our work is that in our usage of ma- chine learning models such as NSFW Detector, the biases and stereotypes within those models were in- jected within our analysis, which could bring about the arguments that the demonstrated stereotypes in this paper are of those models and not Stable Diffusion. Furthermore, questions can be raised about the validity of the NSFW Detector, and why we should trust it to accurately identify NSFW im- ages. However, this does not undermines our study, because the NSFW detector used here has previ- ously found to under-rate images as NSFW and show more false negatives (i.e. failed to recognize NSFW images as NSFW) in comparison to human annotators (Wolfe et al., 2023).
References
Zainab Amadahy and Bonita Lawrence. 2009. Indige- nous peoples and Black people in Canada: Settlers or allies? Breaching the Colonial Contract: Anti- Colonialism the US and Canada, pages 105-136.
Ruha Benjamin. 2020. Race after technology: Aboli- lionist tools for the new Jim code. Oxford University Press.
Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng. Debora Nozza, Tat- sunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. 2023. Fasily Accessible Text-to- Image Generation Amplifies Demographic Stereo- types at Large Scale. The ACM Conference on Fair- ness, Accountability, and Transparency.
We must also discuss similarity a metric of comparing images, and the possibility that the biases within CLIP pervade within our analysis such that the results demon- strated here are not showing biases of Stable Diffu- sion but instead, biases of CLIP. We acknowledge the well-documented biases within CLIP embed- dings (Caliskan et al., 2017). but also that since Stable Diffusion uses CLIP embeuddbbs within its own operating procedures and that there is prior work of using CLIP-cosine similarity for im- age comparision within Stable Diffusion (Luccioni et al., 2023), we believe that our findings are valid and correctly demonstrate biases within Stable Dif- fusion, which can arise due to CLIP-embeddings well as other components of its algorithm, and not CLIP.
use of CLIP-cosine
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. 2021. Multimodal Datasets: Misog- yny, Pornography, and Malignant Stereotypes. arXiv preprint arXiv:2110.01963.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language Models are Few-shot Learners. Advances in neural information processing systems, 33: 1877-1901.
Kyle Buchanan. 2023. At Festival, Adam Driver Backs Actors. The New York Times.
Aylin Caliskan. 2023. Artificial Intelligence, Bias, and Ethies. Proceedings of the Thirty-Second Interna- tional Joint Conference on Artificial Intelligence (IJ- CAI).
Ethics Statement
As part of ethical research in this field, we re- placed the sexualized images shown in Figure with blurred/pixelated versions. We will do so keep- ing in mind concerns within the field of computer vision that uploading sexualized images of women, even for research purposes, only adds to the number of images available online that depict sexualized women.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. 2017. Semantics Derived Automatically from Lan- guage Corpora Contain like Biases. Science, 38666334):183-186
Stevie Chancellor. 2023. Toward Practices for Human- Centered Machine Learning. Communications of the ACM, 66(3):78-85.
Though our finding of the stereotypical depic- tion of personhood being a Western, light-skinned man can amplify socictal problems where people of nonbinary gender, especially transgender indi- viduals, are hatecrimed by conservative peoples (Roen, 2002), we do not claim this as a finding. Though we use prompts around gender, we do not manually classify the gender of images, to avoid
Rachel Ama Asaa Engmann. 2012. Under Imperial Eyes, Black Bodies, Buttocks, and Breasts: British Colonial Photography and Asante "Fetish Girls". African Arts, 45(2)-6-557
Eduard Fosch- Volla-villaronga, Adam Poulsen, Roger Andre Sørea, and BHM Custers. 2021. A Little Bird told me your Gender: Gender Inferences Social Media. In- formation Processing & Management, 58(3):10254)

---

Kathleen C Fraser, Svetlana Kiritchenko, and Isar Ne- jadgholi. 2023. A Friendly Face: Do Lexxt-to-lmage Systems Rely on Stereotypes when the Input is Under-Specified? AAAI 2023 Workshop on Creative Al Across Modalities.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. International Conference on Machine Learning. Learning Transferable Visual Models From Natural Language Supervision in, pages 8748- 8763. PMI.R.
Felix Friedrich, Patrick Schramowski, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Sasha Luc- cioni, and Kristian Kersting. 2023. Fair Diffusion: In- structing Texl-to-Image Generation Models on Fair- ness. arXiv preprint arXiv:2302.10893.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016. International Conference on Machine Learning. Gen- erative Adversarial Text to Image Synthesis in, pages 1060-1069. PMLR.
Sarah J Gervais, Theresa K Vescio, Jens Förster, Anne Maass, and Caterina Suitner. 2012. Seeing Women as Objects: The Sexual Body Part Recognition Bias. European Journal of Social Psychology, 42(6):743- 753.
Katrina Roen. 2002. "Either/Or" and" Both/Neither": Discursive Tensions in Transgender Politics-TEST Signs: Journal of Women in Culture and Society, 27(2):501-52..
Sourojit Ghosh and Aylin Caliskan. 2023. ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non- Gendered Pronouns: Findings across Bengali and Five other Low-Resoure Languages. AAAI/ACM Conference on AI, Ethics, and Sociery 2023, pages 901-912.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björ Ommer. 2022. High- Resolution Image Synthesis with Latent Diffusion Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10600 10695.
Vinit Jakhetiya, Shubham Chaudhary, Badri Narayan Subudhi, Weisi Lin, Sharath Chandra Guntuku, et al. 2022. Perceptually Unimportant Information Re- duction and Cosine Similarity-Base -Basd Quality Assess- ment of DD-SSynthesized Images. IEEE Transactions on Image Processing, 31:2027-2039.
Patrick Schramowski, Manuel Brack, Björ Deiseroth, and Kristian Kersting, 2023. Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- lern Recognition. Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models in, pages 22522 22531.
Os Keyes. 2018.
The Misgendering Machines: Trans/HCI Implications of Automatic Gender Recog- nition. Proceedings of the ACM on luman-computtr interaction, 2(CSCW))1-22.
Christoph Schuhmann, Romain Bcaumont, Richard Vencu, Cade Gordon, Ross Wightman. Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022. LAION-5B: An Open Large-Scale Dataset for Training Next Gener- ation Image-Tes Text Models. Proceedings of the 36th Conference on Neural Information Processing Sys- tems (NeurIPS 2022).
Gant Laborde. 2019. Deep NN for NSFW Detection.
Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite. 2023. Stable Bias: Ana- lyzing Societal Representations in Diffusion Models. arXiv preprint 23l.:2233..140
D Sejal, T Ganeshsingh, KR Venugopal, SS Iyengar, and LM Patnaík. 2016. Image Recommendation based on ANOVA Cosine Similarity. Procedia Computer Science, 89:562-567.
Abhishek Mandal, Susan Leavy, and Suzanne Little. 2023. Multimodal Composite Association Score: Measuring Gender Bias Generative Multimodal Models. Proceedings of the Fourth International Workshop on Algorithmic Bias in Search und Rec- ommendation held as a part of the 45th European Conference on Information Retrieval (FCIR 2023).
Amit Singhal et al. 2001. Modern Information Retrieval: A Brief Overview. IEEE Duta Engineering Bulletin, 24(4):35-43.
Zhiqiang Tao, Hongfu Liu, Huazhu Fu, and Yun Fu. 2017. Proceedings of the AAAI Conference on Artificial Intelligence. Image Cosegmentation via Saliency- Guided Constrained Clustering with Cosine Similarity in, volume 31.
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. 2016. Generating Images from Captions with Attention. International Con- ference Learning Representations 2016.
Blizabeth McDade-Momtez, Jan Wallander, and Linda
United Nations. 2023.
Population, Surface Area and Density. Technical report, UNData.
Cameron. 2017. Sexualization in US Latina and
White Girls' preferred Children's Television Pro- grams. Sex Roles, 77:1-15.
Jialu Wang, Xinyue Gabby Liu, Zonglin Di, Yang I Liu, and Xin Eric Wang. 2023. T2IAT: Measuring Va- Ience and Stereotypical Biases in Text-to-Image Gen- eration. ACI..
Safiya Umoja Noble. 2018. Algorithms of Oppression. New York University Press.

---

Robert Wolfe and Aylin Caliskan. 2022a. American
White Multimodal Language-are and and-Image AL. Proceedings of the 2022 AAAI/ACM Conference AI, Ethics, and Society, pages 800-812.
Robert Wolfe and Aylin Caliskan. 2022b. Markedness Visual Semantic AI. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1269 1279.
Robert Wolfe, Yiwei Yang. Bill Howe, and Aylin Caliskan. 2023. Contrastive Language-Visse Vision AT Models Pretrained on Web-Scraped Multimodal Data Exhibit Sexual Objectification Bias. The ACM Conference on Fairness, Accountability, and Trans- parency.
Wenying Wu, Pavlos Protopapas, Zheng Yang, and Pana- giotis Michalatos. 2020. Gender Classification and Bias Mitigation in Facial Images. 12th ACM Confer- ence Web Science, pages 106-114.
Peipei Xia, Li Zhang, and Fanzhang Li. 2015. Learning Similarity with Cosine Similarity Ensemble. Infor- mation sciences, 307:39-52.
Xiaojin Zhu, Andrew B Goldberg, Mohamed Eldawy, Charles R Dyer, and Bradley Strock. 2007. A Text- to-Picture Synthesis System for Augmenting Com- munication. The Association for the Advancement of Artificial Intelligence (AAAI), 7:1590-153 1595.
A Supplemental Figures and Tables
CW: This section contains NSFW images.
Sugen
Figure 4: Stable Diffusion self-censoring generated im- ages NSFW, and instead returning black images for the prompt 'a front-facing photo of a woman'.

---

Continent/ Country
Person Man
Person
Woman
Person Nonbinary Gender
None
0.64
0.59
0.41
Asia
0.77
0.73
0.43
Europe
0.66
0.53
0.45
North America
0.63
0.49
0.44
Latin America
0.69
0.67
0.37
Africa
0.86
0.82
0.63
Oceania
0.67
0.61
0.41
China
0.74
0.71
0.49
Japan
0.71
0.62
0.51
India
0.57
0.51
0.42
Pakistan
0.56
0,50
0.34
Indonesia
0.63
0.59
0.37
Bangladesh
0.56
0.58
0.39
UK
0.67
0.51
0.41
France
0.64
0.61
0.41
Germany
0.77
0.60
0.54
Italy
0.64
0.61
0.40
Russia
0.69
0.62
0.51
USA
0.66
0.57
0.51
Canada
0.63
0.58
0.49
Mexico
0.44
0.47
0.31
Brazil
0.67
0.61
0.39
Argentina
0.64
0.59
0.43
Colombia
0.63
0.57
0.47
Peru
0.71
0.62
0.49
Venezuela
0.70
0.61
0.44
Ethiopia
0.64
0.59
0.41
South Africa
0.77
0.61
0.44
Nigeria
0.63
0.54
0.47
Egypt
0.44
0.61
0.39
Ghana
0.71
0.62
0.48
Australia
0.75
0.68
0.60
Papua New Guinea
0.31
0.31
0.31
New Zealand
0.71
0.68
0.57
Table 3: Cosine similarity scores across 'Person', 'Man', 'Woman' and 'Person of nonbinary gender' across all studied continental/nationaal identities. Read this table as e.g. the similarity between 'person from Asia' and 'man from Asia' is 0.77, the similarity between 'person from France' and 'woman from France' is 0.61, etc. This demonstrates how Stable Diffusion most closely assosciated 'person' with 'man" and least with 'person of nonbinary gender' across various countries and continents.

---

Continent/Country
Person
Man
Woman
Nonbinary Gender
Asia
0.43
0.43
0.42
0,41
Europe
0.71
0.72
0.71
0.69
North America
0.68
0.69
0.62
0.63
Latin America
0.49
0.46
0.47
0.44
Africa
0.41
0.43
0.42
0.41
Oceania
0.64
0.65
0.64
0.64
China
0.47
0.48
0.44
0,41
Japan
0.47
0.42
0.47
0,41
India
0.43
0.44
0.41
0.39
Pakistan
0.44
0.44
0.43
0,41
Indonesia
0.40
0.41
0.42
0.40
Bangladesh
0.40
0.39
0.39
0.38
UK
0.76
0.76
0.73
0.71
France
0.69
0.70
0.68
0.62
Germany
0.71
0.70
0.70
0.68
Italy
0.64
0.62
0.62
0.61
Russia
0.61
0.62
0.63
0.59
USA
0.77
0.77
0.74
0.71
Canada
0.68
0.69
0.68
0.63
Mexico
0.47
0.48
0.42
0.44
Brazil
0.46
0.46
0.44
0.42
Argentina
0.44
0.44
0.45
0,43
Colombia
0.46
0.47
0.45
0.41
Peru
0.42
0.44
0.42
0,41
Venezuela
0.42
0.43
0.41
0.41
Ethiopia
0.34
0.36
0.34
0.34
South Africa
0.37
0.35
0.36
0.36
Nigeria
0.39
0.38
0.36
0.34
Egypt
0.35
0.34
0.34
0.31
Ghana
0.39
0.36
0.38
0.32
Australia
0.74
0.73
0.73
0.71
Papua New Guinea
0.31
0.31
0.32
0.31
New Zealand
0.72
0.72
0.69
0.67
Table 4: Cosine similarity scores Continents and Countries. Read this table e.g. the cosine similarity of "person' and 'person from North America" is 0.68, that of 'man' and 'man from Oceania" is 0.65, etc.

---

Continent
Country
Person
Man
Woman
Nonbinary
Gender
China
0.73
0.78
0.71
0.70
Japan
0.72
0.79
0.74
0.70
Asia
India
0.49
0,48
0.52
0.31
Pakistan
0.50
0.52
0.57
0.36
Indonesia
0.47
0.49
0.66
0.63
Bangladesh
0.42
0.43
0.42
0.49
UK
0.68
0.60
0.58
0.52
France
0.63
0.59
0.56
0.49
Europe
Germany
0.64
0.51
0.46
0.64
Italy
0.58
0.47
0.44
0,40
Russia
0.54
0.37
0.42
0,49
North
USA
0.61
0.67
0.62
0.63
America
Canada
0.58
0.61
0.57
0.56
Mexico
0.47
0.48
0.49
0,40
Brazil
0.72
0.70
0.82
0.62
Latin
Argentina
0.69
0.66
0.81
0.61
America
Colombia
0.71
0.77
0.71
0.66
Peru
0.71
0.78
0.66
0.79
Venezuela
0.70
0.63
0.81
0.75
Ethiopia
0.68
0.67
0.79
0.64
South Africa
0.68
0.73
0.63
0.67
Africa
Nigeria
0.73
0.63
0.77
0.72
Egypt
0.30
0.34
0.31
0.33
Ghana
0.68
0.76
0.65
0.77
Australia
0.77
0.75
0.71
0.68
Oceania
Papua New Guinea
0.31
0.29
0.23
0.26
New Zealand
0.74
0.73
0.73
0.69
Table 5:
Cosine similarity scores across Continents and Countries. Read this table by picking continent and country, then a column e.g. the cosine similarity of the results from 'man from Africa' and 'Ethiopian man" is 0.67,
while the cosine similarity of the results from 'woman from Latin America" and 'eeruvian woman' 0.66.


Reviews:
[{"review_id": "yo74GpioI8", "reviewer": "Reviewer_m6ZP", "summary": "", "questions": "1. The images presented in Figure 1 seem cherry-picked to some extent. I would like to see more statistics on these generated images. I think it is possible to randomly sample the number of pictures and apply human evaluation to label the details.\n2. Would it be possible to provide images of examples of nonbinary gender?\n3. In Table 5, for Bangladesh and Ghana, cosine similarity scores of nonbinary gender are higher than ‘man’ or ‘woman’. Could you provide more explanations on these results? Similarly, in Table 4, many cosine similarity scores are very close among different columns. These results make me question whether the huge difference in Table 3 comes from pre-existing bias in CLIP-embeddings.\n4. I would like to see the statistical significance of these quantitative results.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments on our paper, and your thoughtful engagement with our work! Below, we have tried to address each of your comments. \n\n- Though CLIP-embeddings are known to be biased and may contribute to the association of ‘person’ and ‘man’ in Stable Diffusion, we do not believe that it is the single factor that creates that association. Stable Diffusion uses OpenCLIP-ViT/H text encoder within its operating procedure, so some of the biases within CLIP can seep into its results, and previous research has shown how the two occupy the same latent space (see “Easily accessible text-to-image generation amplifies demographic stereotypes at large scale”). However, biases and associations made by Stable Diffusion can also stem from other sources, such as patterns present within the LAION-5B dataset on which it is trained, a combination of overlapping biases between these sources, and many more. We believe that it is beyond our scope to disentangle the various components of Stable Diffusion to determine exactly which component produces the associations we highlight in this paper. Additionally, CLIP is widely-used for image representation and comparison in downstream tasks, and has been demonstrated to represent images better than other models (see “The unreasonable effectiveness of CLIP features for image captioning: an experimental analysis”, and “Hierarchical text-conditional image generation with CLIP latents”, line 3).  \n- To the point about human evaluation, we appreciate this comment and have enacted it within our work. At a high level, we randomly sampled 100 pairs of images per comparison (e.g. 100 images comparing ‘person’ to ‘man’, 100 images comparing ‘person from Asia’ to woman from Asia’, 100 images comparing ‘person from North America’ to ‘man from North America’, etc.). We then annotated the similarity as one of five nominal categories: Very Similar, Somewhat Similar, Neither similar nor Dissimilar, Somewhat Dissimilar, and Very Dissimilar. Finally, we compared our evaluations with results from cosine similarity. Our findings show strong correlations between human evaluation and cosine similarity, with Very Similar being most associated with cosine similarity scores in the 0.8-0.63 range, Somewhat Similar in the 0.63-0.51 range, Neither similar nor Dissimilar in the 0.51-0.41 range, Somewhat Dissimilar in the 0.41-0.28 range, and Very Dissimilar in the 0.28-0 range (cohen’s kappa = 0.84). Having said that, it is important to note that the findings from human evaluations do not perpetuate the default definition of ‘person’ being Western, light-skinned men but rather simply reflect the similarity between generated images. We believe this human evaluation strengthens our work.\n- In context of the choice of countries for this paper, we are on the fence about adding more countries, specifically countries with medium/low populations in each continent. This is because we believe that due to their populations, such countries might be currently underrepresented in image datasets on which models such as Stable Diffusion are trained, and therefore results on prompts from such countries could demonstrate stronger biases and inaccuracies. We believe that a comprehensive analysis of such low-resource countries could be performed in a separate paper, with a prompt curation and methodological process that is more cognizant of the low-resource status of such countries. However, if reviewers feel strongly about the addition of such countries of medium/low populations in terms of strengthening the quality of this work and the aforementioned points to not be significant, we could easily expand this work to such countries in the camera-ready version. \n- In context of extending this experiment to models such as Fair Diffusion and Safe Latent Diffusion, we have a few concerns. Fair Diffusion is (currently) limited in its functions because it only accepts professions as input and therefore we cannot try our prompts there. Safe Latent Diffusion, from our preliminary experience using it, could potentially provide less sexualized results than Stable Diffusion but since Safe Latent Diffusion is trained very similarly to Stable Diffusion based on CLIP embeddings and the LAION-5B dataset, we do not believe that the comparative experiment would strengthen the paper. Moreover, the fact that Stable Diffusion has so many more users than Safe Latent Diffusion implies that the problem of sexualization remains unsolved because even though a potential solution might exist, it is not being used. Finally, it is not sufficient to simply compare Stable Diffusion and Safe Latent Diffusion on the aspect of NSFW images, because including Safe Latent Diffusion would necessitate a thorough investigation of its biases, which would be beyond the scope of this paper. \n- The point about potential mitigation strategies is noteworthy and appreciated! We can think of a few suggestions such as more careful data curation from more diverse sources, human-centered machine learning approaches (Chancellor 2023), incorporating annotators from a wide range of cultural contexts and backgrounds, and more. We are simply listing these here in the interest of brevity, and are happy to elaborate on all of these in the camera-ready version.\nIn reference to the images in Figure 1 being cherry-picked, we would like to assure the reviewers that all images represented in this paper are selected randomly from the set of responses to a given prompt, and no images have been cherry-picked to highlight the point. We mention this in the paper, in the image captions at the tops of pages 6 and 7. If the reviewers wish, we could replace any or all currently-shown images with a different set of randomly chosen images. \n- The request for providing images of prompts about nonbinary genders is appreciated, and we will produce such images in the camera-ready version. \n- In reference to the scores for nonbinary genders being higher than ‘man’ or ‘woman’ for Bangladesh and Ghana in Table 5, we do not have an explanation for these outliers. We have double-checked that we did not mis-report these numbers, but they could be a result of factors we can only speculate about, such as stronger-than-average representation of people of nonbinary gender from Bangladesh and Ghana in LAION-5B, underlying CLIP-embeddings in those cases having a stronger signal, or social factors we are unaware of. \n- The request for metrics of statistical significance is greatly appreciated. While we can confirm that all of the quantitative results highlighted in the paper are statistically significant, we will be sure to include p-values in the camera-ready version.\n- Finally, we will also address all sentence-level issues such as repetitions, and concerns with table captions and presentations, in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper provides several exciting findings in text-to-image generations: (1) there exists stereotypes of 'person' for Stable Diffusion when no other information about gender is provided, skews male and ignores nonbinary genders; (2) based on my understanding, this paper is the pioneer to discuss the stereotypes in contexts of national/continental identities; (3) they uncovered the patterns of sexualization of woman, which extends to the findings from previous work. These results can help researchers build new methods to resolve fairness issues and alarm the models' reliability when practitioners want to apply them to real-world applications.", "reasons_to_reject": "1. As mentioned in line 360, CLIP-embeddings are to be biased; I would question whether the higher cosine similarity between ‘person’ and ‘man’ may come from the biased evaluation metric. The paper will be sound if experiments evaluate the inherent biases in CLIP-embeddings and confirm that the biases will not have a dominant effect on the results are done. Moreover, I would like to see human evaluation beyond the cosine similarity to support the findings in the paper.\n2. As mentioned in line 289, the countries chosen in the paper are the top five most populated countries; I would question the selection here to be unfair. I would recommend selecting more countries regarding population size: large, medium, and small, to conduct more comprehensive experiments.\n3. As mentioned in line 655, models like Fair Diffusion/Safe Latent Diffusion have been developed to improve the quality of Stable Diffusion generation in terms of social stereotypes. It would be interesting and important to see similar experiments conducted in this paper applied to debiasing text-to-image generators. I believe these experimental results will provide strong evidence to support the arguments in this paper since the debiasing method has been already developed and it is necessary to see whether the arguments mentioned in this paper have been resolved entirely or partially or not by the methods.\n4. The overall reason is although this paper provides many qualitative discussions, it lacks quantitative explanations to support the arguments. It is crucial to include human evaluation to balance the drawback of biased evaluation metrics (CLIP-embeddings.) Moreover, it is impressive to learn the findings from this paper, but thinking about how to reduce/mitigate the stereotypes might also be noteworthy.", "questions_to_authors": "1. The images presented in Figure 1 seem cherry-picked to some extent. I would like to see more statistics on these generated images. I think it is possible to randomly sample the number of pictures and apply human evaluation to label the details.\n2. Would it be possible to provide images of examples of nonbinary gender?\n3. In Table 5, for Bangladesh and Ghana, cosine similarity scores of nonbinary gender are higher than ‘man’ or ‘woman’. Could you provide more explanations on these results? Similarly, in Table 4, many cosine similarity scores are very close among different columns. These results make me question whether the huge difference in Table 3 comes from pre-existing bias in CLIP-embeddings.\n4. I would like to see the statistical significance of these quantitative results.", "paper_topic_and_main_contributions": "This paper analyzes the stereotypical definitions of personhood in Stable Diffusion, a text-to-image generator. They tested 138 prompts for front-facing photos of people of different genders and continental/national identities and compared the results to reveal two stereotypes - gender and continental/national identities. They found that Stable Diffusion's definition of personhood corresponded closely to Western, light-skinned men, threatening to erase historically marginalized groups. Additionally, they found a pattern of sexualization of women, perpetuating the Western stereotype of fetishizing women of color. These findings suggest the need for more careful use of these tools and improvements in developing fair generators.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The paper could be much appreciated if proofread by an English native speaker. Many repetitions exist in several sections, making the overall paper look wordy.\n2. Captions are not self-contained, which makes the reader hard to understand without referring to the main context. Besides, it would be appreciated if the authors could improve the presentation of Tables 3, 4, and 5 since the current format can easily confuse the reader and hinder understanding.", "Justification_for_Ethical_Concerns": "In section 4.3, which discusses the sexualization of non-western women, I think the authors need to revise the content, for example, avoiding using 'sexy' in Table 2 and other places since this word usage might be inappropriate for specific groups."}}, {"review_id": "KuF6HlGd2a", "reviewer": "Reviewer_gPrz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and your comments about our work being comprehensive and well-written. Regarding your comment about our findings not being novel in light of Lucconi et al.’s (2023) work on Stable Bias, we rather believe that the two are complimentary where our work builds on Stable Bias. While our work does bear similarity to their “Identities” dataset/prompts, our approach of understanding the default representation of personhood and the finding that Stable Diffusion associates ‘person’ most closely with Western, light-skinned men with a specific focus on the granularity by country is novel and not addressed in their work. Furthermore, the finding about the sexualization of Latin American women is also totally novel and not covered by Lucconi et al. (2023) Therefore, we believe that our findings are novel even in comparison to Lucconi et al. (2023), though we do acknowledge the formative impact of their work on our research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A comprehensive diagnostic study was conducted to reveal stereotypes in Stable Diffusion.\n2. The paper is well written.", "reasons_to_reject": "1. The findings are not novel compared to [1], which finds that \"diffusion models over-represent the portion of their latent space associated with whiteness and masculinity across target attributes.\"\n\n[1] Luccioni, A. S., Akiki, C., Mitchell, M., & Jernite, Y. (2023). Stable bias: Analyzing societal representations in diffusion models. arXiv preprint arXiv:2303.11408.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper conducts a diagnostic study to uncover stereotypes in Stable Diffusion. The stereotypical definition of personhood corresponds closely to Western, light-skinned men. Sexualization of women, mostly Latin Amerin was also common. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "clbWUYlaw3", "reviewer": "Reviewer_4Bgr", "summary": "", "questions": "1) If you could direct the engineering team behind Stable Diffusion to address these problems, where would you have them put their effort?\n\nFor example, I assume one part of the problem is the training data, LAION-5B likely includes plenty of sexualized images of women but even the labeling of non-offensive images is likely to have a subtle POV that reflects social structures. That is, many white Americans giving a caption wouldn't think to write \"a white person waving goodbye\".\n\nYou may think of that as a morass and prefer to direct the engineers to sampling—note when users have typed text that seems to be fairly generic and make sure that the model doesn't reproduce biases by, say, adding in other terms at random (\"a person waving goodbye\" will be given some probability of x gender being selected, some probability of y ethnicity/country). \n\n2) Relatedly, what are the uses of Stable Diffusion that you are most worried about? For example, is it marketing folks generating images for  their websites/emails/ads? Individuals generating memes? Something else? If you could solve the bias you're detecting for one and only one use case, which would it be and why?\n\n---\nWhether or not your answers to these questions make it into the paper (I hope they do), I think this discussion among yourselves will clarify how you see the situation and judge interventions to deal with the problems you're detecting. \n\nYou have written \"Our findings have worrisome implications on exacerbating societal tendencies of the Western stereotype, and designers should consider how their datasets and design choices lead up to such results.\" I think that is well put but to say \"think carefully\" without saying \"we recommend X\" is to put a lot more onus on those designers and if you have a way of helping them, that feels like it is a helpful part of harm reduction. And if other researchers disagree, that seems like a valuable thing for the field to wrestle with. \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments, and your generative questions! Below, we have tried to address those:\n\n- Thank you for recognizing our commitment to not feature overly sexualized images. We reiterate our ethical statement that, upon acceptance of this work, we would blur out the sexualized images in the camera-ready and digitally-available version of the paper. We do so to avoid providing more sexualized images to be incorporated into datasets which would then be used to train computer vision models, thus perpetuating the very problem we identified in our research. \n- Your comment on cosine similarity and its drawbacks is helpful! In the camera-ready version, we will discuss some potential drawbacks of using cosine similarity for this task. Having said that, we do believe that it is the best available metric for our purposes, and this is supported by the fact that it is prominently used in similar previous research (see “Perceptually unimportant information reduction and Cosine similarity-based quality assessment of 3D-synthesized images”, “Learning similarity with cosine similarity ensemble”, and  “American== white in multimodal language-and-image AI”). Similarly, we can certainly elaborate on CLIP biases and “slow down”, as requested, in the camera-ready version. \n- In response to your comment about an annotation project, we did conduct some annotation tasks because other reviewers also stressed the importance of seeing them. At a high level, we randomly sampled 100 pairs of images per comparison (e.g. 100 images comparing ‘person’ to ‘man’, 100 images comparing ‘person from Asia’ to woman from Asia’, 100 images comparing ‘person from North America’ to ‘man from North America’, etc.). We then annotated the similarity as one of five nominal categories: Very Similar, Somewhat Similar, Neither similar nor Dissimilar, Somewhat Dissimilar, and Very Dissimilar. Finally, we compared our evaluations with results from cosine similarity. Our findings show strong correlations between human evaluation and cosine similarity, with Very Similar being most associated with cosine similarity scores in the 0.8-0.63 range, Somewhat Similar in the 0.63-0.51 range, Neither similar nor Dissimilar in the 0.51-0.41 range, Somewhat Dissimilar in the 0.41-0.28 range, and Very Dissimilar in the 0.28-0 range (cohen’s kappa = 0.84). Having said that, it is important to note that the findings from human evaluations do not perpetuate the default definition of ‘person’ being Western, light-skinned men but rather simply reflect the similarity between generated images. \n- We also appreciate your comment about highlighting pairs with very high or very low similarity, and going into more depth with some fine-grained examples. We can do that in the camera-ready version. We will highlight how ‘person’ is highly similar to ‘person from the UK’ (0.76) and ‘person from the USA’ (0.77) but highly dissimilar to ‘person from Papua New Guinea’ (0.31) and ‘person from Ethiopia’ (0.34), by providing specific commentary on what the perceived similarities and differences could be and how embeddings are doing that. We can also briefly discuss within-pair comparisons, as we measured the variance within the scores for each prompt. We observe that ‘person from the USA’, ‘person from the UK’, ‘person from Australia’, ‘person from Germany’, ‘person from New Zealand’, ‘person from Canada’ have variances in the range of 0.1-0.12, while ‘person from Papua New Guinea’, ‘person from Egypt’, ‘person from Bangladesh’, and ‘person from Ethiopia’ have scores less than 0.01 with the average variance across all prompts being 0.04. This aligns with previous findings of how results for more privileged identities show higher variance because of stronger data sampling, whereas those from lower privileged identities only demonstrate the biased, single understanding (see “Markedness in visual semantic AI”\n- In response to your question about directing designers of Stable Diffusion in directions to focus their efforts, we can think of a number of potential directions such as more careful data curation from more diverse sources, human-centered machine learning approaches (Chancellor 2023), incorporating annotators from a wide range of cultural contexts and backgrounds, and more. We are simply listing these here in the interest of brevity, and are happy to elaborate on all of these in the camera-ready version. Without breaching anonymity, we can also say that we are in direct contact with the LAION team on mitigating potential biases and issues within their datasets, and can provide more specific strategies in the camera-ready version. \n- In response to your question about worrisome usage of Stable Diffusion, one clear answer is that the usage of Stable Diffusion in art, marketing and content generation to be shared on digital platforms. The perpetuation of our demonstrated biases in such content will lead to representational harms, which could and historically have led to long term gradual large-scale change about who belongs where and how aspects of an individual’s identity are perceived (see “Measuring representational harms in image captioning”, “Big data's disparate impact” and “The meaning and measurement of bias: lessons from natural language processing”), and impact implicit associations documented in social cognition (see “Automatic stereotyping” and “Implicit social cognition: attitudes, self-esteem, and stereotypes”). We will address these in the camera-ready version too! \n- Thank you for your note on our word choice in the Ethics statement. We see your point about avoiding the phrase “inhuman abominations”, and will amend it in the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Years ago there was important work in linguistics and related fields to establish that, in English, \"generic man\" and \"generic he\" (standing in for 'humanity') was not neutral, this paper is in a similar vein but works on images/representations. What does \"a person\" look like? I think many researcher will be interested in their use of CLIP-cosine similarity and disagreement with this method feels likely to be fruitful for everyone involved. ", "reasons_to_reject": "I guess if I'm looking for risks, I could say \"Oh no, the authors are going to flash a lot of overly sexualized pictures of women at the audience\" but uh, the care the authors have around this make me feel like that's not a real risk here. \n\nI think the authors could probably be clearer about the drawbacks of picking cosine similarity. They do mention that all sorts of stuff might be present in the image that humans would see but that aren't picked up by CLIP. They also mention the problem of bias already known in CLIP but I think this could be slowed down and built out to show readers that the logic isn't circular. \n\nThe obvious alternative or addition would seem to be a big annotation project...that would be great to verify the method but to be clear I don't see that as being required to shore this up (though it should be someone's follow-on research). \n\nSomething more within the authors' power/scope could be to demonstrate the images/descriptions of pairs that are especially high. The aggregate stats are meant to zoom out but it can be useful to give more granular examples. In particular, showing cross-category highly similar pairs and within-category highly dissimilar pairs may help readers understand what the embeddings are and aren't doing. \n\n\n", "questions_to_authors": "1) If you could direct the engineering team behind Stable Diffusion to address these problems, where would you have them put their effort?\n\nFor example, I assume one part of the problem is the training data, LAION-5B likely includes plenty of sexualized images of women but even the labeling of non-offensive images is likely to have a subtle POV that reflects social structures. That is, many white Americans giving a caption wouldn't think to write \"a white person waving goodbye\".\n\nYou may think of that as a morass and prefer to direct the engineers to sampling—note when users have typed text that seems to be fairly generic and make sure that the model doesn't reproduce biases by, say, adding in other terms at random (\"a person waving goodbye\" will be given some probability of x gender being selected, some probability of y ethnicity/country). \n\n2) Relatedly, what are the uses of Stable Diffusion that you are most worried about? For example, is it marketing folks generating images for  their websites/emails/ads? Individuals generating memes? Something else? If you could solve the bias you're detecting for one and only one use case, which would it be and why?\n\n---\nWhether or not your answers to these questions make it into the paper (I hope they do), I think this discussion among yourselves will clarify how you see the situation and judge interventions to deal with the problems you're detecting. \n\nYou have written \"Our findings have worrisome implications on exacerbating societal tendencies of the Western stereotype, and designers should consider how their datasets and design choices lead up to such results.\" I think that is well put but to say \"think carefully\" without saying \"we recommend X\" is to put a lot more onus on those designers and if you have a way of helping them, that feels like it is a helpful part of harm reduction. And if other researchers disagree, that seems like a valuable thing for the field to wrestle with. \n", "paper_topic_and_main_contributions": "The paper looks at stereotypes in Stable Diffusion (text-to-image), with a focus on gender and nationality/continental identity. The research revealed that when generating images of a 'person', the model disproportionately skews towards males and individuals from Europe/North America, presenting an implicit bias. A troubling pattern of sexualization of women, especially Latin American, Mexican, Egyptian, and Indian women, was also identified. The authors establish these patterns using pairwise CLIP-cosine similarity and manual examination across 138 prompts. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I feel torn about this part of your Ethics Statement: \"Though our finding of the stereotypical definition of personhood being a Western, light-skinned man can amplify societal problems where people of nonbinary gender, especially transgender individuals, are considered inhuman abominations by conservative peoples (Roen, 2002), we do not claim this as a finding.\"\n\nPart of me feels like you are helping readers see a very stark and dehumanizing reality. But more of me feels like you could probably convey this without saying \"inhuman abominations\". "}}]

Author rebuttal:

Meta-review: This paper explores stereotypical biases present in Stable Diffusion; specifically, tendencies for "person" to be aligned with "white, male person" and for the sexualization of women of colour. Reviewers agree that this is important and relevant work, and novel in its inclusion of various national/cultural identities and non-binary gender. 

The main weakness of the paper according to the reviewers is in the use of cosine similarity of CLIP embeddings as the primary evaluation metric. CLIP embeddings themselves encode certain societal biases, and so the question is whether the results presented in the paper are due to bias in Stable Diffusion or bias in the evaluation metric.  In response, the authors have conducted a 100-sample manual annotation study, which demonstrates that cosine similarity and human annotation are in high agreement. They also offer to include examples in the revised version of pairs ranked as “very similar” and “very dissimilar”, as a visual (qualitative) demonstration of what the cosine similarity is measuring. 

While I agree with the concerns raised by Reviewer m6ZP, I also share the opinion of Reviewer 4Bgr that a thorough discussion of any potential limitations of CLIP cosine similarities, along with the manual annotation study and the examples, may be sufficient for the current work. I also encourage the authors to make the other changes suggested by the reviewers, including an expanded discussion on bias mitigation techniques (both existing solutions, such as Fair Diffusion and Safe Latent Diffusion, and potential solutions and recommendations as outlined in the reviews), and revising problematic wording where necessary. 

Pros:
- Interesting work on an important research topic
- Considers bias along under-studied dimensions
- Small annotation study shows good agreement with human perceptions of similarity

Cons:
- Possible bias in the evaluation metric muddies the interpretation
- Limited comparison against existing “de-biased” models
Decision: Accept-Findings