SlowGuess/NeurIPS_2023_OCR huggingface dataset:
VISOGENDER: A dataset for benchmarking gender bias in image-text pronoun resolution
Siobhan Mackenzie Hall
Fernanda Gonçalves Abrantes
Hanwen Zhu
Grace Sodunke
Aleksandar Shtedritski
Hannah Rose Kirk
Oxford Artificial Intelligence Society, University of Oxford
We introduce VISOGENDER, a novel dataset for benchmarking gender bias in vision-language models. We focus on occupation-related biases within a hegemonic system of binary gender, inspired by Winograd and Winogender schemas, where each image is associated with a caption containing a pronoun relationship of subjects and objects in the scene. VISOGENDER is balanced by gender representation in professional roles, supporting bias evaluation in two ways: i) resolution bias, where we evaluate the difference between pronoun resolution accuracies for image subjects with gender presentations perceived as masculine versus feminine by human annotators and ii) retrieval bias, where we compare ratios of professionals perceived to have masculine and feminine gender presentations retrieved for a gender-neutral search query. We benchmark several state-of-the-art vision-language models and find that they demonstrate bias in resolving binary gender in complex scenes. While the direction and magnitude of gender bias depends on the task and the model being evaluated, captioning models are generally less biased than Vision-Language Encoders. Dataset and code are available at https://github.com/oxai/visogender.
Vision-language models (VLMs) are advancing rapidly and reaching ever-wider audience across numerous applications, such as classification and captioning, as well as text-to-image retrieval and generation. However, these models are pre-trained from uncurated image-text pairs scraped from the internet [1, 2] and so, their outputs can perpetuate or amplify social biases [3, 4, 5, 6]. How the VLM is used determines the mechanisms of how biases transfer from pre-training to downstream representational and/or allocational harms [7]. For example, a VLM used for image retrieval may skew towards returning more images of male doctors, thus reinforcing stereotypical associations between gender and career success; or a VLM used for captioning may more frequently misgender women or non-binary image subjects, aligning with a capability (un)fairness [8].
Despite the growing body of work on evaluating and mitigating the bias of VLMs [9, 10, 11, 12, 13], there is a dearth of specifically-designed benchmark datasets to evaluate the presence of social biases across downstream tasks (such as captioning or image retrieval): most prior work has measured biases using pre-existing image datasets such as FairFace [14] or COCO [15, 16], despite their limited real-world transferability and spurious correlations [12, 17]. In this paper, we introduce the VISOGENDER benchmark for evaluating bias in VLMs. The design of VISOGENDER is inspired by two prior bodies of works. Firstly, we apply stress-testing of vision-linguistic reasoning capabilities of VLMs as in the Winoground benchmark [18] but introduce the dimension of social biases. Secondly, we adopt
Figure 1: Resolution of gender pronouns and retrieval with a neutral query. We resolve gender by (i) using zero-shot classification with Cross Models Encoders, such as CLIP, and (ii) next-token prediction with captioning models, such as BLIP. We have an additional simpler task to resolve the gender of a single person, e.g., with a template "The doctor and her / his stethoscope".
the templated structure to test gender bias in occupational pronoun resolution from NLP research, but apply it to the vision-language domain—specifically the WinoGender [19] and WinoBias [20] frameworks, in turn inspired by Winograd Schema [21]. To our knowledge, VISOGENDER is the first dataset to combine both of these contributions by stress-testing gender bias in visual-linguistic reasoning and coreference resolution capabilities.
VisoGENDER contains images of a person depicted in an occupational role ("the doctor"), combined with either an object ("the stethoscope") or a participant ("the patient"). Each image is labelled by human annotators for the perceived gender presentation of the occupation and/or the participant, and the dataset is balanced across different genders occupying these roles. For each image, we construct natural language captions that use the perceived gender labels of the image subject(s) to derive a possessive binary pronoun relationship ("the doctor and his/her patient"). We test bias in two tasks: pronoun resolution and image retrieval (see Fig. 1 for summary). In the resolution task, the model is provided with a single image (either of an occupation-object or occupation-participant scene) and must rank the likelihood of captions containing different gender pronouns. There are varying levels of difficulty in the resolution task—from a single person resolution in the occupation-object case; to two person resolution in the occupation-participant case, where either both subjects are perceived to have the same gender presentation (easier), or different gender presentations (harder). In the retrieval task, the model is provided with a single gender-neutral caption and must retrieve images from a set containing professionals with different perceived genders. We measure resolution bias using the gender accuracy gap in correct pronoun resolution (corresponding to capability fairness) and retrieval bias using commonly-applied metrics such as Bias@K, MaxSkew and NDKL (corresponding to representational fairness).
We present preliminary results for six state-of-the-art Vision-Language Encoders (VLEs) models [1, 22, 2, 23, 24, 25] and two state-of-the-art captioning models [26, 27]. We find that models still struggle to resolve pronoun relationships, especially when there are two people in the image of different perceived gender presentations (where performance is close to random). Our benchmark also recovers that (i) models display substantial accuracy gaps in resolving pronouns of masculine versus feminine-presenting subjects, indicating the presence of resolution bias; and (ii) when provided with a neutral pronoun query ("the doctor and their patient"), models predominately rank images of masculine-presenting subjects higher than those with a perceived feminine presentation, indicating a retrieval bias. We compare these results to U.S. Labor Force statistics (as in [28, 20, 19]) and find some correlations between model bias and societal occupational skew in the US labour market. Our findings demonstrate there is still substantial progress to be made in improving scene disambiguation for visual-linguistic reasoning, as well as reducing the gender gap in resolution performance and retrieval outcomes. Some caveats to our findings are needed: first, while we do find capability unfairness evidenced in differential performance across gender identity groups, this paper works within a hegemonic system of binary and stereotypical gender presentation that remains prevalent in Western constructions and perceptions of gender, where datasets typically originate [29, 30]. Models are likely to perform even more poorly in downstream tasks where there is additional complexity introduced by the inclusion of greater cultural diversity, as well as transgender, non-binary and gender-diverse individuals who are underrepresented here and in other vision-language datasets.
Second, our benchmark is designed to measure poor performance on the two tasks, thus identifying potential downstream harms from occupational gender bias in VLMs. However, being good at the task requires not only advanced visual-linguistic reasoning but also accurate gender prediction, a capability that can be concerning if misused for surveillance purposes. We discuss both of these limitations and concerns in Sec. 5. The pace at which VLMs are being developed is only set to grow in coming years—VISOGENDER provides a much-needed benchmark to evaluate their potential downstream harms before large-scale deployment.
Bias in coreferences in NLP Coreference resolution aims to identify which mentions in a natural language document link to the same real-world entity [31]. In the past decade, significant progress has been made moving from rule-based systems and expert knowledge [32], to statistical models [33, 34] and deep neural networks [35, 36, 37, 38, 39]. Pronoun resolution involves linking a noun such as "doctor" to a pronoun in the sentence. Biases have been identified, with respect to machine translation [40], non-binary pronouns [41], and favouring masculine entities when resolving gender ambiguous cases [42]. Our work is most similar to gender pronoun resolution tasks based on Winograd schemas [21], like Winogender [19] and WinoBias [20] which investigate occupational-related biases. Both of these works demarcate "hard" and "easy" cases based on (anti-)stereotypical gender-occupation associations as measured relative to U.S. Labor Force statistics. We extend this work to the vision-language domain. In our resolution task, we modify the typical Winograd scheme because the correct resolution is unambiguous, i.e., there is a correct caption (and pronoun) for a corresponding image. However, our retrieval task is a closer vision-language analogy to [19, 20] because there is no groundtruth for a "correct" ranking of images given a gender-neutral search query.
Evaluating visual reasoning There is an emerging body of work on visual reasoning tasks [43], such as VQA [44, 45, 46], visual word sense disambiguation [47], compositionality understanding [48, 49, 50], comprehension [51] or visual entity linking [52]. Most similar to our work, Winoground [18] evaluates visio-linguistic compositional reasoning by tasking a model to match two images with two captions containing the same set of words, only in a different order—such as “there is a mug in some grass” vs. “there is some grass in a mug”. The task is challenging, with state-of-the-art VLMs rarely performing better than chance, though [53] demonstrate some of these failures may be due to atypical images in the dataset. Our vision-linguistic stress-tests are inspired by adapting Winoground to social biases, but a key difference is that our caption-image pairs do not contain the exact same set of words—for example, matching “the doctor and her patient” versus “the doctor and his patient”.
Measuring bias in vision-language models Measuring the social bias of VLMs is a growing area of research. While early works measure misclassification rates into harmful categories [9, 54], more recent methods investigate face-to-text retrieval [11, 10, 55, 56], or captioning [57]. However, these approaches rely on off-the-shelf datasets, such as COCO [16], which have been shown to contain spurious correlations [17] and thus are not suitable for evaluating model bias [12]. Similar to [12], we balance our dataset by gender across different occupational settings, but instead using naturally-occurring images rather than synthetic edits.
3 The VISOGENDER Benchmark
The VISOGENDER dataset contains 690 images of people in various occupational settings, where each image is annotated for the perceived gender presentation of the subject(s) in the image. We use these annotations to construct a templated caption of an inferred pronoun relationship. The dataset covers 23 unique occupations in a hierarchical taxonomy. Each occupation appears in the dataset with two template forms—either as a single person in the image with a possessive pronoun to an object ("the doctor and his/her stethoscope"), or as one of two people in the image with a possessive pronoun to a participant ("the doctor and his/her patient") (see Sec. 3.2). A summary of the dataset is presented in Tab. 1. In the following subsections, we introduce the terminology used throughout our paper (Sec. 3.1); describe the dataset (Sec. 3.3); and provide detail of the templates (Sec. 3.2). We then summarise the two types of VLMs which are compatible with VISOGENDER (Sec. 3.4); and finally, define the two tasks through which we measure model bias (Sec. 3.5).
3.1 Terminology
- Perceived gender presentation: We do not have direct communication with image subjects, so we cannot know the pronoun with which the subjects identify. Instead of referring directly to gender identity, we use perceived gender presentation to indicate that an inference is made by an external human annotator. We do not make any claims of assigning a person's gender identification, which we recognise as an individual's personal experience of gender [58], and acknowledge that our labels may differ from the subject's personal identity.
- Masculine and feminine presentation: We opt to use terms for gendered characteristics (masculine and feminine) over terms for biological sex (male and female). These perceived gender labels (denoted as "M" for masculine and "F" for feminine unless otherwise stated), are assigned by an external human annotator based purely on their perception of the person's visual presentation such as facial features, stereotypical clothing, hair or other signals. We recognise that gendered characteristics from masculine to feminine lie on a spectrum and assigning one dominant presentation over another is subject to annotator internal biases, which are in turn endogenous to lived experience. Please see our positionality statement for more information (Sec. 5.3).
3.2 Templates
Each templated caption contains three components, adapted from Winogender [19]:
- Occupation: a person referred to by an occupational noun and definite article, "the doctor"
- Pronoun: a pronoun corresponding to the perceived gender presentation of the occupation in the image, e.g., "her" or "his"
- either Object: a noun corresponding to typical professional items, e.g., "the stethoscope"
- or Participant: A second person in a typical professional relationship with the occupation, e.g., "the patient"
For occupations, we use the list from [19], but remove (i) occupations without a clear possessive pronoun relationship between the occupation and participant, e.g., "the plumber and their houseowner" is not semantically correct; and (ii) occupations without sufficient open-domain images across genders (for both men and women occupying the occupation and participant roles). We classify the remaining occupations into a hierarchical taxonomy to permit aggregate group-wise analysis: Sector describes the general field, and includes education, medical, office, retail and service; Specialisation describes subcategories within the sector, where, for example, services includes food services, fashion, animal or household; and finally, Occupations are nested within specialisations, where, for example, food services contains waiter, bartender, and baker. Similar to [19, 20, 28], we match U.S. Labor Force statistics on the percentage of men working in each occupation to compare model biases to occupational skew in the real-world US labour market. When comparing our results to these statistics, we will use the perceived masculine gender presentation counts as men ("M"), and the remaining percentages to be for women ("W"). The full taxonomy and list of occupations is presented in the Supplementary Materials. We also source the list of participants from [19] but replace any references to children as participants and in some cases, make modifications for a more natural possessive pronoun, e.g., "the lawyer and the witness" becomes "the lawyer and their client". For objects, we manually define a typical professional item for each occupation. Using these components, we construct three templates (subtasks) of increasing difficulty for coreference resolution:
- Single subject: The template of captions is "The {occupation} and {his/her} {object}", e.g., "the doctor and her stethoscope". For each occupation, we collect 10 occupation-object images, 5 for each gender. Here, models only need to resolve the pronoun of one subject in the image, thus testing simple visual-linguistic reasoning.
- Two subjects of the same perceived gender presentation: The template of the captions is "The {occupation} and {his/her} {participant}" e.g. "the doctor and her patient". In this case, the perceived gender presentation of the occupation and the participant are the same (both masculine or both feminine). Per occupation, we collect 5 images for each of these two cases (M-M, F-F). Here, the model must resolve the inferred pronouns of two subjects but assigning which subject is the occupation and which is the participant does not affect the prediction.
Table 1: VISOGENDER dataset summary, showing the hierarchy of included Sectors, Specialisations, and Occupations; the Perceived Presentation of Gender (P.P.Gender) pairs per template type, and the counts of images within each split of the dataset.
Categories	Number of images
Sect.	Spec.	Occ.	P.P.Gender Pairs	Images per Occ.	Images per P.P.Gender Pair	Images per P.P.Gender Pair and Occ.	Overall
Single person(occupation-object)	5	13	23	[M, F]	10	115	5	230
Two-person(occupation-participant)	5	13	23	[MM, FF, MF, FM]	20	115	5	460
Total	690
- Two subjects of different perceived gender presentations: Finally, we use the same occupation-participant template but now the participant and the occupation are of opposite perceived gender presentations (one masculine and one feminine). Per occupation, we collect 5 images for each of case (M-F, F-M). Here, the model must resolve the perceived gender of the subject, and infer from image context which is the occupation and which is the participant to infer the pronoun.
3.3 Dataset Collection
The VISOGENDER dataset comprises image URLs with annotations for the occupation noun, the participant or object noun, and the perceived gender presentations of the occupation and participant. These annotations can be used to reconstruct the templated captions. Data collection, which includes data labelling, was carried out by the authors of the paper from March to May 2023 on a variety of image databases and search providers, such as Pexels and Google Image Search. We followed a set guidelines to specify exclusion and inclusion criteria, detailed in the Supplementary Materials. $^{1}$
We ensure that there are no duplicate images (no overlaps between occupations) and no invalid URLs across the dataset. In the early stages of data collection, we used the entire list of occupations from [19]. However, we only include those with at least 20 viable URLs (5 per gender pair) for occupation-participants and 10 viable URLs for occupation-object (5 per gender). The image curation process (and availability of viable URLs) is dependent on the retrieval of different gendered roles across occupational search queries and so therefore compromised by inherent representational biases in these search engines. We mitigate effects of imbalance across genders by only including occupations with a full set of images (equal images across all gender pairs) but this may introduce a sample selection bias to the included occupations. Furthermore, inferring gender from an image depends on ingrained biases of the dataset curators. We discuss limitations and biases of data collection in Sec. 5.3, and suggest possible expansions in the future with further resources e.g., partnering with a stock photo company. The dataset is accompanied by Data Clause (which details the Licence and Terms of Use) as well as a Datasheet for Datasets [59] in the Supplementary Materials.
3.4 Two Supported Types of Vision-Language Models
VISOGENDER is designed to accommodate two types of VLMs. Here we discuss their properties, and how bias can be measured in common use cases.
Vision-Language Encoders (VLEs) VLEs, such as CLIP[1], have separate vision and language encoders and are trained to jointly match images and text. Given an image  $i \in \mathcal{R}^{3 \times H \times W}$  and text  $t$ , a VLE outputs a score  $s(i, t)$  that expresses the degree of compatibility between the image and text. The first common use case of VLEs is zero-shot classification of images [1, 60, 61]. This is done by providing a query image  $i_q$  and text prompts  $t_n, n \in \{1, \dots, N\}$ . For example, if we wish to zero-shot classify the perceived gender of a doctor in an image using pronoun resolution, we can provide text prompts "This is an image of a doctor and {his, her} notebook", and select the pronoun with the highest compatibility score to the image. Such a classifier can be considered biased if, for example, it more accurately infers the pronoun of one perceived gender presentation in some occupations. The
For example, images were required to have Creative Commons and/or royalty free licences, and were to be photo-realistic, have maximum two subjects, contain no children nor NSFW content.
second common use case of VLEs is for text-to-image retrieval [1]. Given a text query  $t_q$ , images  $i_n, n \in 1, \dots, N$  and a query size  $K$ , we select the  $K$  images with the highest compatibility score to the text prompt. In this setting, the model can be biased if, for example, when searching for a given occupation, people from a given demographic are over or under-represented in the top  $K$  retrieval results.
Captioning models Captioning models are most commonly trained to autoregressively predict a text caption given an image. For an image  $i$  and, optionally, a partially completed caption with N tokens  $c = [t_1, \ldots, t_N]$ , the model outputs the probability for the next token  $t_{N+1}$  as  $p_{\mathrm{cap}}(t_{N+1}|i, t_1, \ldots, t_N)$ . Similar to VLEs, we can apply the captioning model to infer the perceived gender of a subject in an image via pronoun resolution. We first supply a query image  $i_q$  (say, an image of a doctor) and a caption  $c_q$  like "An image of a doctor and". We then inspect the probability distribution for the next token  $t_n$ , denoted by  $p_{\mathrm{cap}}(t_n) = p_{\mathrm{cap}}(t_m|i_q, c_q)$ . We can now compare the probabilities  $p_{\mathrm{cap}}(t_n) = \text{"her"}$  and  $p_{\mathrm{cap}}(t_n) = \text{"his"}$ , choosing the one with the higher score as the model's selection. It has been demonstrated that comparing token probabilities is a more reliable measure of a generative language model's performance compared to free generation [62], and such templates have been successfully used to evaluate bias in LLMs [28].
3.5 Two Angles of Model Bias
The VISOGENDER setup has the flexibility to measure model bias in two ways:
Resolution task The resolution task considers a single image with perceived gender label and matches it to multiple candidate captions containing different gender pronouns. For example, we start with an image containing a doctor perceived as feminine, and specify the set of candidate captions as "the doctor and her/his patient". We define resolution accuracy,  $RA$ , as the percentage of correctly resolved pronouns. This can be calculated over all occupations, across main occupation categories, or per occupation. For a given occupation  $o \in O$  and a perceived gender presentation  $g$  (either masculine  $m$  or feminine  $f$ ), we have:
$$
R A _ {g} (o) = \frac {\text {N u m b e r o f c o r r e c t l y r e s o l v e d p r o n o u n s o f g e n d e r g i n o c c u p a t i o n o}}{\text {T o t a l n u m b e r o f p r o n o u n s o f g e n d e r g i n o c c u p a t i o n o}}
$$
An unbiased outcome is one where the model resolves both perceived gender pronouns equally, i.e.,  $RA_{m}(o) = RA_{f}(o), \forall o \in O$ . We now define resolution bias as the gender resolution accuracy gap
$$
\Delta (o) = R A _ {m} (o) - R A _ {f} (o), \tag {1}
$$
where a positive value of  $\Delta$  shows a model more accurately resolves masculine-presenting subjects, and vice versa. Our definition of resolution bias measures a form of capability fairness, i.e., whether a system performs equally well across subgroups [8]. This task is applicable to both types of VLMs.