SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Very few of the generated image captions mention gender-neutral terms such as 'person' (an average of less than  $1\%$  of captions for any of the systems, distributed equally across professions), and none use the 'non-binary' gender marker. Also, less than  $0.5\%$  of captions and  $2\%$  of VQA generations explicitly mention the profession in the prompt, but it is interesting to note that a single profession, police officer, had explicit mentions of the profession name in  $80.95\%$  of captions. In the case of VQA, several others also had a significant number of explicit mentions, including doctor  $(70.48\%)$  of VQA, firefighter  $(45.71\%)$  and pilot  $(29.84\%)$  of VQA. We believe this to be due to the markedness [3] of these professions and of the high proportion of gendered references to individuals (as opposed to using gender-neutral terms such as 'person') in the data used for training both TTI systems.
4.2 Gender and Ethnicity Distribution in the Image Space
Section 4.1 leveraged image-to-text systems to help surface a first category of biases in the output of the three TTI systems of interest. In order to further quantify the systems' diversity in terms of both GENDER and ETHNICITY without having to solve a poorly-specified and poorly-motivated identity label assignment problem, we now turn to our proposed cluster-based method to identify relevant trends in the visual features produced by these systems.
4.2.1 Characterizing Identity Regions in the Image Space
We apply the method described in Section 3.2 to identify significant regions in the TTI systems' output space that help summarize the variation of visual features corresponding to different combinations of gender and ethnicity phrases in the generation prompts. Specifically, we cluster images corresponding to 68 combinations of 4 phrases for gender and 17 for ethnicity into 24 regions. Thus, by quantifying the distribution of a system's generations over these regions, we can identify trends in their visual features that are correlated with the identity characteristics used to identify the regions.
Table 2 showcases this by providing summaries of 10 of the most represented regions. While the regions themselves do not correspond to specific genders or ethnicities, we can get sense of the visual features they represent by looking at the top gender and ethnicity phrases that were featured in prompts assigned to that region in the "Identities" dataset. For example, we see that region 4, which accounts for over  $40\%$  of the "Professions" images, tends to contain images that were generated for prompts describing White men. We can further see that regions which mostly features prompts with the word woman make up a significantly smaller part of the "Professions" dataset overall (regions 15, 13, 1, and 10 together add up to  $25.5\%$ ).
3The BLS only provides statistics for two gender categories on their website.
Markedness is a linguistic concept that refers to the distinctiveness of a word or concept compared to others – in the case of professions, the fact that firefighters and pilots have distinctive uniforms that allow them to be easily distinguished from other professions
Figure 2: Cluster assignments of "Laboratory technician" images.
Figure 3: Examples of Profession images for "laboratory technician" assigned to clusters 4, 15, 1, and 3. There are no Dall-E 2 images assigned to 1.
Region #	Share	Top gender phrases	Top ethnicity phrases
4	40.1%	unspecified	man	Caucasian	White	unspecified	Latinx
15	12.6%	woman	non-binary	White	Caucasian	unspecified	First Nation
21	9.2%	man	unspecified	unspecified	White
18	8.8%	man	unspecified	unspecified	White	Caucasian
13	7.5%	woman	unspecified	Latinx	Hispanic	Latino	unspecified
22	3.1%	unspecified	man	SE Asian	Black	Indigenous	Multiracial
1	2.7%	woman	non-binary	Black	Afr. American	Multiracial	unspecified
10	2.7%	non-binary	woman	Latinx	White	Indigenous	unspecified
3	1.9%	man	unspecified	Black	Afr. American	Multiracial	Pacific Isl.
17	1.3%	non-binary	White	unspecified	Caucasian	Hispanic
Table 2: Identity clusters can be represented by the top gender and ethnicity phrases in their generation prompts. We show 10 of the main clusters above and outline those that feature the woman and Black gender and ethnicity phrases.
Figures 2 and 3 help us better understand how this cluster assignment functions in practice by focusing on the 630 images (210 for each system) generated for the "Photo portrait of a laboratory technician." prompt. Figure 2 shows the distribution over the regions separately for each of the systems, outlining a significant difference between the Dall-E 2 and Stable Diffusion outputs: with region 4 twice as prominent for the former. In order to verify our intuition about what cluster assignments mean, we visualize generations for the prompt assigned to selected regions in Figure 3, and find that the representations are coherent with the region summaries from Table 2 - Dall-E 2 is missing images for Cluster 1 because none of its generations are assigned to that cluster.
4.2.2 Gender and Ethnicity Representation across Systems
The previous paragraph shows how we can use the identified regions to better understand the diversity of generations by a given system for any profession of interest. Given a prompt, cluster assignments tell us whether the model tends to generate images for this prompt that are more similar to the ones generated for a given GENDER or ETHNICITY. While this level of detail enables fine-grained analysis of a model's specific biases, more general bias trends across multiple models can be harder to apprehend at a glance.
In the following paragraphs, we propose an aggregation scheme across professions and clusters to better support such comparisons. We group professions based on the US Bureau of Labor Statistics. Rather than looking at 146 distributions for each of the professions in the list, we rank jobs based on the gender and ethnicity distributions reported by workers in the US and group them into 5 bins of 29 to 30 (quintiles). We rank and group professions by the proportion of workers who self-report as women when looking at regions of the image space that are correlated with GENDER, and by the proportion of workers who self-report as Black when examining ETHNICITY.
For regions, we create groups based on whether the corresponding gender and ethnicity phrases are prominently features in their "Identities" prompts (see Table 2), with woman and Black in the top-2 and top-4 gender and ethnicity phrases respectively. We then compare the proportion of images assigned to the grouped region per corresponding quintile to the average BLS value for this quintile. This allows us to assess not just whether a group is under- or over-represented by the models, but also whether a model attenuates, reproduces or exacerbates social biases.
Table 3 provides the results for these three analyses for both Stable Diffusion versions and for Dall-E 2. The quintile analysis makes the systems easier to compare while still retaining an important level of specificity:
Quintiles by rank	BLS Woman	Clusters featuring "woman" phrase %	BLS Black	Clusters featuring "Black" phrase %
SD 1.4	SD 2	Dall-E 2	SD 1.4	SD 2	Dall-E 2
Low 20%	7.5	8.5 (±0.36)	5.3 (±0.28)	1.4 (±0.17)	4.7	6.6 (±0.60)	4.8 (±0.56)	3.9 (±0.48)
20 to 40	26.5	15.2 (±0.46)	14.9 (±0.46)	2.8 (±0.20)	7.1	8.2 (±0.55)	4.9 (±0.54)	4.3 (±0.32)
40 to 60	47.1	32.2 (±0.57)	23.1 (±0.53)	6.0 (±0.28)	10.5	7.3 (±0.61)	8.1 (±0.63)	3.4 (±0.53)
60 to 80	68.4	54.3 (±0.60)	46.3 (±0.63)	21.6 (±0.57)	14.4	11.2 (±0.64)	11.4 (±0.64)	5.3 (±0.54)
Top 20%	86.8	83.0 (±0.45)	78.3 (±0.49)	56.3 (±0.62)	22.1	16.8 (±0.66)	16.1 (±0.57)	4.5 (±0.46)
Table 3: Comparing cluster assignments to Bureau of Labor Statistics (BLS). Each line corresponds to one fifth of the professions grouped by BLS-reported percentage of women (left), and Black workers (middle).  $95\%$  confidence intervals per a bootstrap estimator are provided.
for example, if a system under-represents women in its outputs, this lets us know whether it is reproducing or exacerbating societal biases. Taking the example of Stable Diffusion v1.4, we can see that while the system seems to match the US distribution for the least diverse professions (whether the workforce is mostly identified as men or women), regions that feature woman are under-represented across the more balanced professions. We can also identify broad trends across systems: while all under-represented regions of the Space associated with the woman and Black phrases, the phenomenon is least pronounced for Stable Diffusion v1.4, and most pronounced for Dall-E 2.
Figure 4: Comparing 11 TTI systems' representations of the regions associated with the woman and Black phrases. The format of results is the same as in Table 3 above, presented as line plots here to more easily contrast multiple models. On the "rank" x-axis, each tick corresponds to a quintile of professions grouped by the BLS-reported proportions of women (left), and Black workers (right) in those professions. The "percentage" y-axis corresponds to the percentage of image generations for those professions assigned to the clusters selected as described in Table 2 and in the text.
Evaluating additional systems While we initially focused our evaluation on three TTI systems with the highest visibility at the start of this work, our method is easily applicable to any TTI system that can generate an image given a prompt - we release the necessary code for this alongside our paper. We further benchmarked an additional 11 systems selected from the most downloaded text-to-image models on the Hugging Face Hub at the time of writing. We summarize results for the same gender and ethnicity quintiles as above in Figure 4. We see that even though these systems all share Stable Diffusion models as their pre-trained initialization, the diversity of their outputs across both dimensions does seem to depend on the specific finetuning and adaptation process, sometimes independently from each other. These results should provide a starting point for further investigation into these systems' specific datasets and help guide specific use cases.
4.3 Interactive Tools for Interactive Exploration
As part of our own analysis of images generated by different TTI systems, we have created tools to help us delve into the images in more detail and identify relevant patterns to guide our analyses. We present the three of the tools that we have created, and the observations that they have allowed us to make, below:
Diffusion Bias Explorer One of the first tools that we created in the scope of this project was the Diffusion Bias Explorer (See Fig. 5 (a)), which enabled users to compare what the same set of prompts – based on the list of professions described in Section 3 – resulted in when fed through the 3 TTI systems. It allowed us to uncover initial patterns – including the homogeneity of certain professions (such as CEO) or the differences between versions of Stable Diffusion as well as Dall-E 2. In fact, we used the Diffusion
Diffusion Bias Explorer
Choose from the prompts below to explore how the text-to-image models like Stable Diffusion v1.4, Stable Diffusion v.2 and DALLE-2 represent different professions and adjectives
(a) Diffusion Bias Explorer
(b) Average Face Comparison Tool
Text-to-Image Diffusion Model Average Faces
We ran 150 professions and 20 adjectives through 3 text-to-image diffusion models to examine what they generate.
Choose one of the professions and adjectives from the dropdown menus and see the average face generated by each model.
(c) k-NN Explorer
Figure 5: The 3 interactive tools created as part of our analysis: Diffusion Bias Explorer, Average Faces Comparison, and the k-NN Explorer.
Bias Explorer all throughout the project in order to visually verify tendencies discovered using our analyses, since it allowed us to quickly view images for any given profession+TTI system combination.
Average Face Comparison Tool Another tool that we created was the Average Face Comparison Tool (see Fig. 5 (b)), which leverages the Facer Python package to carry out face detection and alignment based on facial features and averaging across professions. This helped us further see more high-level patterns in terms of the visual aspects of the images generated by the different TTI systems while avoiding facial recognition and classification techniques that would prescribe gender or ethnicity. Also, the blurriness of the average images gave us signal about how homogeneous and heterogeneous certain professions were.
Nearest Neighbors Explorers: BoVW and Colorfulness To enable a more structured exploration of the generated images we also developed two nearest-neighbor lookup tools. Users can choose a specific image as a starting point—for example, a photo of a Black woman generated by a specific TTI system—and explore that photo's neighborhood either by colorfulness (as defined by [27]), or by a bag-of-visual-words [61] TF-IDF index. To build this index, we used a bag-of-visual-words model to obtain image embeddings that do not depend on an external pre-training dataset. We then extracted each image's SIFT features [36] and used k-means [34] to quantize them into a codebook representing a visual vocabulary and computed TF-IDF sparse representations and indexed the images using a graph-based approach [23]. These search tools enable a structured traversal of the dataset and thereby facilitate a qualitative exploration of the images it contains, either in terms of color or structural similarity. This is especially useful in detecting stereotypical content, such as the abundance of stereotypical Native American headdresses or professions that have a predominance of given colors (like firefighters in red suits and nurses in blue scrubs).
We provide code and detailed instructions about how to duplicate and edit these tools to work with other sets of prompts and images, to facilitate their dissemination and customization in the community.
Despite our best efforts, our research presents several limitations that we are cognizant of. First, the models that we used for generating captions and VQA answers both have their own biases (many of which we describe in Section 2.1), which we are unable to control for in our analyses. We aimed to compensate for these by leveraging multiple models and comparing their outputs, as well as by using less symbolic models such as BoVW. Second, while the open nature of the Stable Diffusion models makes us reasonably certain that we did not miss any major confounding factors in our comparative analysis, the same is not true of Dall-E 2. Given that it is only available via an API, we were unable to control for any kind of prompt injection or filtering  $^{5}$ , or indeed whether we were at all prompting the same model on different days. We were therefore only able to compare the output of all three models on the assumption that the images they generate correspond to the model outputs based on the input prompts. Third, our analyses are limited to a given
<sup>5</sup>A blog post from the creators of Dall-E 2in July 2022 stated that they added a technique to improve the diversity of its representations [48], with many speculating that it is based on prompt injection [59, 47].
set of social attributes in terms of gender and ethnicity, which, as we describe in Section 3, are attributes that are inherently fluid, multidimensional and non-discretizable. Finally, we recognize that none of the authors of this paper have primary academic backgrounds in scientific disciplines relevant to the social science dimensions of gender and ethnicity, and we do not have first-hand experiences for many of the identity characteristics that we refer to. We also recognized our bias towards discrimination axes that are relevant to the Western world, and the importance of extending this analysis to different cultures and different contexts.
We consider our work to be a first step in exploring societal biases in text-to-image models, with much follow up work needed to make this work more complete and nuanced. An important part of this work would be to keep exploring different dimensions and aspects of social bias, including age and visual markers of religion, as well as other target attributes that are tied to stereotypes and power dynamics. We hope that future work will continue to carefully consider the complex, interconnected nature of many types of biases and the fact that many attributes cannot be inferred visually from generated images. Finally, we believe that there is much potential work to be done in further developing interactive tools such as those we created to support qualitative analysis and storytelling around the model biases, as well as empowering stakeholders and communities with less technical expertise to engage with and probe TTI systems and other ML artifacts.
[65] WARD JR, J. H. Hierarchical grouping to optimize an objective function. Journal of the American statistical association 58, 301 (1963), 236-244.
[66] WENG, L. What are diffusion models? Lilian Weng GitHub (Jul 2021).
[67] WOLFE, R., AND CALISKAN, A. American== white in multimodal language-and-image ai. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society (2022), pp. 800-812.
[68] WOLFE, R., AND CALISKAN, A. Markedness in visual semantic AI. In 2022 ACM Conference on Fairness, Accountability, and Transparency (2022), pp. 1269-1279.
[69] ZHAO, D., WANG, A., AND RUSSAKOVSKY, O. Understanding and evaluating racial biases in image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 14830-14840.
[70] ZHAO, J., WANG, T., YATSKAR, M., ORDONEZ, V., AND CHANG, K.-W. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (Copenhagen, Denmark, Sept. 2017), Association for Computational Linguistics, pp. 2979-2989.