SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hugging Face, Canada sasha.luccioni@huggingface.co
Margaret Mitchell,
Hugging Face, USA
Christopher Akiki
Leipzig University and ScaDS.AI  
christopher.akiki@uni-leipzig.de
Yacine Jernite,
As machine learning-enabled Text-to-Image (TTI) systems are becoming increasingly prevalent and seeing growing adoption as commercial services, characterizing the social biases they exhibit is a necessary first step to lowering their risk of discriminatory outcomes. This evaluation, however, is made more difficult by the synthetic nature of these systems' outputs: common definitions of diversity are grounded in social categories of people living in the world, whereas the artificial depictions of fictive humans created by these systems have no inherent gender or ethnicity. To address this need, we propose a new method for exploring the social biases in TTI systems. Our approach relies on characterizing the variation in generated images triggered by enumerating gender and ethnicity markers in the prompts, and comparing it to the variation engendered by spanning different professions. This allows us to (1) identify specific bias trends, (2) provide targeted scores to directly compare models in terms of diversity and representation, and (3) jointly model interdependent social variables to support a multidimensional analysis. We leverage this method to analyze images generated by 3 popular TTI systems (Dall-E 2, Stable Diffusion v 1.4 and 2) and find that while all of their outputs show correlations with US labor demographics, they also consistently under-represent marginalized identities to different extents. We also release the datasets and low-code interactive bias exploration platforms developed for this work, as well as the necessary tools to similarly evaluate additional TTI systems.
Diffusion-based approaches are one of the most recent Machine Learning (ML) techniques in prompted image generation, with models such as Stable Diffusion [52], Make-a-Scene [24], Imagen [53] and Dall-E 2 [50] gaining considerable popularity in a matter of months. These generative approaches are inspired by the principles of non-equilibrium thermodynamics [66] and trained to reverse the gradual addition of noise that is layered onto an image. One key difference that has led to the widespread adoption of diffusion models is that they are simpler to train than previous generations of generative models owing to their likelihood-based loss function, whose mode-covering characteristic [38] is also one of the main reasons why model outputs are so diverse compared to other classes of generative models. However, this iterative approach also makes it particularly difficult to directly access the latent space of the model, making it difficult to directly access and analyze the latent space of diffusion models.
Both the training and inference functions of diffusion models also rely on additional phases of text and image processing (e.g. safety filters, embeddings, etc.), which is why we refer to them as Text-to-Image (TTI) Systems as opposed to models: they are an assemblage of components and modules that all contribute to generating the final image, and it is hard to disentangle which module is responsible for downstream
Figure 1: Our approach to evaluating bias in TTI systems
behavior, and what kind of biases are baked in due to models such as CLIP [29] being used to guide training. Many of these systems are also increasingly finding their way into applications ranging from generating stock imagery [35] to aiding graphic design [43] and used to generate realistic and diverse images based on user prompts. And as with any ML artifact—or, indeed, technology in general—TTI systems exhibit biases, and widely deploying them in different sectors makes them particularly prone to amplifying and perpetuating existing societal inequities. However, these biases remain sparsely documented and hard to audit, often only described in broad terms in model cards [20, 41] and in papers describing new models [53].
In the present article, we introduce a set of tools and approaches to support the auditing of the social biases embedded in TTI systems, by comparing model outputs for a targeted set of prompts (see Fig 1). We illustrate our approach by comparing images generated by Stable Diffusion v.1.4, v.2, and Dall-E 2in terms of the social biases they encode through the lens of different professions and explicit mentions of words related to gender and ethnicity. We also share tools that enable users to explore other TTI systems and topics - contributing towards lowering the barrier to entry for exploring these systems - as well as the datasets of profession generations for all of the TTI systems analyzed. We conclude with a discussion of the implications of our work and propose promising future research directions for continuing work in this space.
Bias in ML is a complex concept, encompassing disparate model performance across different categories, social stereotypes propagated by models, and (mis-)representations present in datasets that can then be encoded by models and misunderstood by users. There has been extensive and insightful work analyzing the biases of ML models, both from the ML community and from many fields at the intersection of technology and society, such as social science, cognitive science, law, and policy (e.g. [54, 26, 21, 18, 58], among many others). Our project builds upon this work in ML – we endeavor to briefly describe the most relevant findings in the current section, and refer readers to further readings for more details about these topics.
2.1 Bias Evaluation in Multimodal ML
While it has been well-established that text models and image models encode problematic biases independently (see [13, 12, 15, 2], among many others), the rapid increase in multimodal models (e.g., text-to-image, image-to-text, text-to-audio, etc.) has outpaced the development of approaches for understanding their specific biases. It has not yet been established whether biases in each modality amplify or compound one another, a gap that our work seeks to address. For example, if a multimodal model represents the visual concepts of "people" and "White people" in a very different space than "Black people", yet represents the linguistic concept of "people" with higher positive sentiment terms than "Black people", then a compounded outcome could be model output where Black people are depicted more negatively than other identity groups.
Indeed, there is support for this concern: past work on multimodal vision and language models found that societal biases regarding race, gender and appearance propagate to downstream outputs in tasks such as image captioning (e.g., [28, 9]) and image search (e.g., [70, 42]), and has found that such biases are learned
by state-of-the-art multimodal models [67, 68]. For instance, in the case of image captioning, models are much more likely to generate content that stereotypes the people depicted, such as by associating women with shopping and men with snowboarding [70, 28], or producing lower-quality captions for images of people with darker skin [69]. Similarly, research on biases in image search has found that algorithmic curation of images in search and digital media are likely to reinforce biases, which can have a negative impact on the sense of belonging of individuals from these groups [30, 60, 40, 64]. Given the exponential popularity and widespread deployment of image generation systems, this can translate into negative impacts on the minority groups who feel under- or mis-represented by these technologies [47].
2.2 Text-to-Image Systems and their Biases
The recent increased popularity of TTI systems has also been accompanied by work that aims to better understand their inner workings, such as the functioning of their safety filters [51], the structure of their semantic space [14], the extent to which they generate stereotypical representations of given demographics [10, 19, 44] and memorize and regenerate specific training examples [16, 62]. Work in the bias and fairness community has also examined how models such as CLIP [49], which are used to guide the diffusion process, encode societal biases between race and cultural identity [1, 67, 68]. Some initial research has attempted to propose approaches for debiasing text and image models [8, 4, 63], as well as reducing the bias of images generated by Stable Diffusion models by exploring the latent space of the model and guiding image generation along different axes to make generations more representative [14, 55], although the generalizability of these approaches is still unclear.
One of the reasons for this is that there are many sources from which biases can originate in TTI systems; for instance, the training data used for training diffusion such as Stable Diffusion is scraped from the Web and has been shown to contain harmful and pornographic content [11, 37] as well as mistrabeled and corrupted examples [59]. These datasets then undergo filters to isolate subsets that are considered, for example, 'aesthetic' or 'safe for work', which are created based on the output of classification models trained or fine-tuned on other datasets [56], which can itself result in further unintended consequences. For instance, the creators of Dall-E 2 observed that attempting to filter out "explicit" content from their training data actually contributed to bias amplification, necessitating additional post-hoc bias mitigation measures [45]. Further biases can be introduced during the training process, given that many TTI systems use the CLIP model [49] to guide the training and generative process despite its biases (see above), the impact of which on image generation is still unclear. Finally, the biases of safety filters and prompt enhancement techniques used in TTI systems are poorly understood and largely undocumented [47]. For instance, keyword-based approaches have been shown to have a disparate impact on already marginalized groups [22], so using them for safety filtering at prompt level can have similar consequences.
3 Methodology: Auditing Social Biases in TTI Systems
In this work, motivated by the necessity to better understand and audit social dynamics in multimodal ML, we propose a new approach for quantifying the biases of TTI systems. Our approach stems from the following intuition: images generated by TTI systems may lack inherent social attributes, but they do showcase features in their depiction of human figures that viewers interpret as social markers. We cannot define these markers a priori due to the social nature of identity characteristics such as race or gender; they are multidimensional notions spanning a spectrum along which people choose to associate themselves [17] rather than discrete external quantities, and cannot be determined by a person's appearance. Indeed, while previous work in ML has adopted binary gender and fixed prior ethnicity categorizations for the sake of convenience, the downstream impact of these choices that are propagated from (labeled) datasets to trained models can contribute to perpetuating algorithmic harms and unfairness [7, 5, 31, 15]. We therefore endeavor to use more flexible proxy representations of the visual features in TTI systems' generated images to identify regions of the representation space that viewers may associate with social variation, and use those to audit the diversity of TTI systems in an application setting.
Our approach, shown in Figure 1, is the following: first, we define a set of identity characteristics for which we want to evaluate diversity and representativity – in the rest of this paper, we will consider both gender and ethnicity. Second, we generate a set of images with TTI systems by controlling the variation of markers of these social attributes in the systems' input prompts – i.e. by spanning different ways of referring to
<sup>1</sup>The criteria used to establish which images are esthetically-pleasing and safe for work are also unclear and merit further investigation.
a person's gender or ethnicity in the prompt. Third, we generate sets of images for prompts with different social attributes that we want to audit – here, we focus on the profession attribute and generate images for a set of 146 professions. Finally, given the multimodal nature of TTI systems, we carry out two sets of analyses: a text-based analysis that leverages Visual Question Answering (VQA) models in the text modality, and a clustering-based evaluation to characterize correlations between social attributes and identity characteristics directly in the image modality. We explain each step in more detail in the sections below.
3.1 Generating a Dataset of Identity Characteristics and Social Attributes
In order to generate a diverse set of prompts to evaluate the system outputs' variation across dimensions of interest, we use the pattern "Photo portrait of a  $[X][Y]$ ",<sup>2</sup> where  $X$  and  $Y$  can span the values of the identity characteristics — ethnicity and gender — and of the professional attribute that we focus our analysis on — the name of the profession. For the professional names, we rely on a list of 146 occupations taken from the U.S. Bureau of Labor Statistics (BLS), which also provides us with additional information such as the demographic characteristics and salaries of each profession that we can leverage in our analysis. For the identity characteristics, we add a suffix to make the pattern "Photo portrait of a [ethnicity] [gender] at work" – since adding the "at work" suffix makes the images more directly comparable to those generated for professions, given that these are often set in workplace settings. For gender, we use three values in this study: "man", "woman", and "non-binary person"; and as an additional option we also use the unspecified "person" to use the same pattern without specifying gender. This is still far from a complete exploration of gender variation, and in particular misses gender terms relevant to trans* experiences and gender experiences outside of the US context. For ethnicity, we are similarly grounded in the North American context, as we started with a list of ethnicities in the US census which we then expanded with several synonyms per group (the full list is available in Appendix A). Enumerating all values of the gender and ethnicity markers we defined led to a total of 68 prompts. Examples of generations for both the social attributes and target attributes sets are shown in Figure 1.
We use these prompts to generate two supporting datasets for our evaluation of three TTI systems to compare the social biases they encode: Stable Diffusion v.1.4, v.2, and Dall-E 2. We choose these three systems because they represent popular TTI systems that were state-of-the-art at the start of this project (early 2023) and allow us both to compare two versions of the same system (Stable Diffusion v.1.4, and v.2) as well as open versus closed-source systems (Stable Diffusion vs Dall-E 2). We generate an "Identities" dataset to ground our analysis by generating a set of images per model and combination of gender and ethnicity phrases in the prompts, for a total of 68 prompts and 2040 images. We then generate a "Professions" dataset to evaluate for each model, by generating a set of images for each system for each of the 146 professions in our set. In the following section, we describe our two methods for evaluating the social diversity showcased in the "Professions" datasets without gender or ethnicity label assignment, including a novel non-parametric method that compares its images to those in the "Identities" dataset.
3.2 Different Approaches for Analyzing Generated Images
Given the multimodal nature of TTI systems we wish to audit for social biases, we propose two methods to analyze their outputs; one method relying on text representations of the images, and one that focuses on direct comparisons in the image space; we explain these in the sections below.
Text Features Analysis: Image-to-Text Systems One category of image representations that we can leverage more easily to find evidence of bias is that of text-based representations; specifically, textual representations of the figures depicted in an image. We use two ML-based systems to automatically obtain such descriptions — one designed for image captioning (the ViT GPT-2 Model [46] trained on the MS-COCO dataset [33]) and one designed for Visual Question Answering, or VQA (the BLIP VQA base model [32] which was pre-trained on a set of 124M image-text pairs including MS-COCO [33] and a subset of LAION-400M [57], then fine-tuned on VQA 2.0 [25]). While the image captioning model generates multi-word captions for each of the images, the VQA system outputs a single word or a short phrase that answers the question "What word best describes this person's appearance?" Both models are open-ended, which means that we do not know a priori whether the outputs will feature words directly related to the social attributes we are studying: our goal in using them is to analyze aggregate statistics of these words across the generated images, without constraining either model to labels that reflect our own biases. We also
2Before converging on this prefix, we experimented with several others, including "Photo of", "Photograph of", "Portrait of", "Close up of", but found that "Photo portrait of" gave the most realistic results.
recognize that both of the model contain confounding factors and biases that will be reflected in the text that they output, which is why we do not interpret the captions as a ground truth, but a feature among others.
Visual Features Analysis: Clustering-Based Approach While textual descriptions of the images are much more tractable than their raw pixel representations, they also carry more limited information, especially in the case of the short VQA answers. A middle ground between the two levels of faithfulness to the information contained in an image can be achieved by leveraging dense embedding techniques that project the images into a multidimensional vector space. To that end, we leverage the same BLIP VQA system as above to also obtain image embedding; namely, the normalized average of the question token embeddings produced by the VQA encoder conditioned on the image. We chose this approach because it allowed us to focus the embedding model on the person depicted in an image - which produced an embedding structure better suited to our goal than alternatives including the CLIP image encoder [49] (see the Appendix for comparisons).
In order to make use of those embedding systems to evaluate social biases in a model's output, we need to be able to identify regions of the embedding space that correspond to visual features associated with a viewer's perception of gender or ethnicity. We do this by leveraging the "Identities" dataset described above. Specifically, we cluster the embeddings of the 2040 data points into 24 sets of images with a Ward linkage criterion for the dot product [65]. We can then identify trends in a TTI systems' generated images, including those in the "Professions" dataset, by quantifying which of these 24 regions they tend to over- or under-represent. Assigning an example to one of these regions is different from assigning it a predefined identity characteristic. Indeed, each of the region corresponds to identity prompts that span different gender and identity phrases. Additionally, delineating 24 regions in the space would not be enough to cover the 68 combinations of identity phrases showcased in the prompts. However, since the clustering was run on a space whose main source of variation came from those identity phrases, we can expect that they do encode visual features that are meaningfully associated with those - and jointly varying gender and ethnicity phrases in the prompts allows the regions to encode phenomena that are specific to their intersection. We will validate this intuition and propose specific analyses based on a TTI system outputs' cluster assignments in Section 4.2.
Interactive Exploration In parallel to the approaches described in previous sub-sections, we also introduce a series of interactive tools to support story-based examination of biases in images generated by the TTI systems (which we will describe in more detail in Section 4.3). The primary goal of these tools is not to produce quantitative insights into the images, but to allow more ad-hoc in-depth explorations of them, since there are many aspects of generated images that are hard to analyze automatically (such as the presence of certain specific characteristics and visual elements in images), but which can be observed during interactive explorations, based on different angles defined by the user.
The goal of our analysis is to develop ways of analyzing and comparing the biases of text-to-image systems that would allow users of these systems to shed light on these otherwise impenetrable systems. Section 4.1 compares the PROFESSION target attribute in the "Professions" dataset to the GENDER identity characteristic through the use of discrete textual representations of the images obtained with a captioning and a VQA system; Section 4.2 then explores the system outputs' joint variation across dimensions of GENDER and ETHNICITY following the methodology introduced in Section 3.2, and provides a detailed analysis of our 3 systems of focus as well as an overview of results for a larger set of open TTI systems. Finally, Section 4.3 describes new interactive tools that we developed based on our approach and showcases how the tools can support a qualitative analysis and storytelling around TTI system biases.
4.1 Gender Bias Analysis through Text Markers
As described in Section 3, we used captions and open-ended VQA to obtain textual representations of the generated images, whose discrete nature makes identifying trends related to social attribute variation more tractable. More specifically, we base our evaluation on the likelihood that a caption or VQA answer for a given profession contains gender-marked words such as 'man' and 'woman' or gender-unspecified descriptors such as 'person' or the profession name — we present these results in Table 1.
In total,  $97.66\%$  of the captions generated contained gender-marked terms, versus  $45.56\%$  of VQA appearance predictions: this is consistent given the fact that VQA mostly consists of single word predictions, whereas captions are full sentences. To put the percentage of predictions that contain
captions	VQA	Labor Bureau
% woman	% man	% gender markers	% woman	% man	% gender markers	% woman	% man
SD v.1.4	38.04%	61.96%	97.24 %	37.77 %	62.23%	47.92%	47.03%	52.97%
SD v.2	33.45%	66.55 %	96.66%	31.10 %	68.90%	44.50%
Dall-E 2	19.96%	80.04%	99.09%	21.95 %	78.05%	44.25%
Average	30.48%	69.52%	97.66%	30.06%	69.67%	45.56%
Table 1: The average percentage of mentions of 'woman', 'man', 'person' in the captions generated by a Vision Transformer Model, the BLIP VQA model and the difference between these percentages and those provided by the U.S. Bureau of Labor Statistics. N.B. these percentages are based on the number of captions/VQA appearance words containing gender markers, not the total number of data points.
gender-marked terms such as 'man' and 'woman' into perspective, we compare them with the percentages of men and women in these professions provided by the BLS  ${}^{3}$  . We find that Dall-E 2 has the largest discrepancy compared to the BLS-provided numbers, with its captions mentioning women on average 27% less, and its VQA mentioning them 25% less, and Stable Diffusion v.1.4 having the least (approximately 9% for both captions and VQA). The professions with the biggest discrepancy between the BLS and both captions and VQA across all systems are: clerk (57 and 55% less), data entry keyer (55/53% less) and real estate broker (52/54% less), whereas those that have more captions that mention women are: singer (29/36% more), cleaner (20/16% more) and dispatcher (19/16% more).