SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Retrieval task The retrieval task considers a single gender-neutral caption for a given occupation and matches it to multiple images containing subjects with different perceived gender presentations from the same occupation. For example, we start with the caption "the doctor and their patient" and define the set of candidate images as containing  $50\%$  images of doctors who are perceived as masculine and  $50\%$  who are perceived as feminine. Given there is no groundtruth for a "correct" ranking of images for a gender-neutral caption, we cannot define a retrieval accuracy metric. For defining retrieval bias, we use 3 commonly used metrics—Bias@K [63], Skew@K [64, 11] and NDKL [65, 64]. Bias@K measures the overrepresentation of men in the top K retrieval results. Skew@K measures the difference between the desired proportion of image attributes and the observed one, and MaxSkew@K is the maximum Skew among all attributes, or the "largest unfair advantage" [64] belonging to images of any perceived gender presentation. NDKL is a ranking measure that measures the distance from a fair distribution. For further definitions and discussions of these, please refer to the Supplementary Materials. Our definition of retrieval bias measures a form of representational fairness, i.e., with a gender-balanced set of images and a gender-neutral caption, whether each perceived gender group have equal chances of being retrieved. The retrieval task is only applicable to VLEs.
For the resolution task, we evaluate six VLEs—CLIP [1], OpenCLIP [22] (trained on LAION 2B and 400M [2]), SLIP [23], DeCLIP [24], FILIP [25] (last 3 trained on YFCC-15M [66]); and two state-of
Table 2: Resolution bias. We present resolution accuracy averaged for masculine and feminine gender presentations, as well as the resolution accuracy gap  $\Delta$ , as defined in eq. (1). "Same perceived presentation of Gender (P.P.Gender)" and "Different P.P.Gender" are images with two people from the same or different P.P.Gender, respectively. A positive gap  $\Delta$  denotes better resolution accuracy for masculine-presenting subjects. We also present reported zero-shot classification accuracy on ImageNet [67].
Model	Overall	Single Person	Two people	ZS Imagenet
RAavg	Δ	Overall	Same P.P.Gender	Different P.P.Gender
RAavg	Δ	RAavg	Δ	RAavg	Δ
CLIP [1]	0.75	0.92	-0.14	0.57	-0.27	0.79	-0.18	0.36	-0.35	63.2
OpenCLIP2B[22]	0.78	0.96	-0.07	0.60	-0.37	0.77	-0.42	0.44	-0.32	66.2
OpenCLIP400M[22]	0.74	0.84	-0.27	0.64	-0.29	0.80	-0.26	0.46	-0.33	62.9
SLIP [23]	0.60	0.77	0.14	0.43	0.14	0.51	0.12	0.34	0.17	34.3
DeCLIP [24]	0.70	0.87	0.06	0.52	-0.17	0.74	-0.14	0.29	-0.19	43.2
FILIP [25]	0.45	0.41	0.06	0.49	0.36	0.49	0.36	0.50	0.37	39.5
BLIP-2 [26]	0.84	0.92	-0.09	0.76	0.07	0.93	0.06	0.60	0.09	—
GIT [27]	0.84	0.96	-0.07	0.72	-0.27	0.97	-0.07	0.48	-0.47	—
the-art captioning models—BLIP-2 [26] and GIT [27]. For two candidate models (CLIP and BLIP-2, which are among the most downloaded models in the respective model family on Huggingface), we go into more detail by investigating their resolution capabilities and resolution biases, which are also compared to U.S. Labor Force Statistics (Sec. 4.1). We ablate the VISOGENDER setup by changing the order of templates and including a neutral caption. For the retrieval task, we benchmark the same six VLEs on the resolution task. Captioning models are not compatible with the retrieval task. We also compare retrieval bias metrics with U.S. Labor Force Statistics. For all VLEs, we use ViT-B/32 encoders, and for GIT we use the GIT-Large model. We show more detailed analysis for retrieval bias for CLIP and BLIP-2. We present ablation studies and error bars robustness analysis for both tasks in the Supplementary Materials.
4.1 Resolution Task
We present results for the resolution task in Tab. 2, disaggregated by different levels of difficulty. We report the mean resolution accuracy  $RA_{\mathrm{avg}}$  for each difficulty level, together with the resolution bias or accuracy gap  $\Delta$ . We highlight the difference between model capabilities and model bias—here we evaluate both, where the latter is the gap between model capabilities for perceived genders.
Evaluating resolution capabilities As expected, the resolution accuracy is highest when there is one person in the image, and lowest when there are two people of different perceived gender in the image. The accuracies for the latter are consistently worse than random chance, pointing at the models' inability to reason about scenes with multiple people and attributes associated with each of them. This confirms the findings of prior works that conclude that VLMs are not capable of complex visio-linguistic [18] or spatial [68] reasoning. Captioning models are better than, or on par with, VLEs for all levels of difficulty. In Fig. 2, we see that BLIP-2 outperforms CLIP on all perceived presentation of gender splits in the dataset. From Tab. 2 we also see that models with better zero-shot classification accuracy on ImageNet [67] tend to have a better overall resolution accuracy.
Evaluating resolution bias From Tab. 2, we see that models tend to exhibit a larger resolution accuracy gap with more difficult subtasks, such as two people with different genders, where there is higher variation and almost random predictions across models. In Fig. 2, we compare the resolution bias, or accuracy gap, for CLIP and BLIP-2. We see that (i) CLIP shows a larger accuracy gap, and (ii) CLIP is more biased towards correctly resolving pronouns for feminine-presenting subjects, whereas BLIP-2 correctly resolves pronouns for masculine-presenting subjects more often. For further analysis and per-occupation results, see the Supplementary Materials.
To interpret the results in a real-world context, we compare U.S. Labor Force Statistics on on proportions of different genders in occupations with resolution bias in Fig. 3. These statistics only account for binary gender, and we have adjusted our perceived masculine and feminine gender presentation counts to be for "men" and "women" respectively. We measure the correlation in the absolute values (with Pearson's R) and correlation in ranked values (with Kendall-Tau), i.e., testing for the monotonicity of relationship between model bias and societal occupation skew [28]. While we see no pattern for the bias of CLIP, the accuracy resolution gap of BLIP-2 correlates with the U.S.
Figure 2: Resolution accuracy (top) and resolution bias (bottom). Top: We compare the resolution accuracy for different perceived presentations of gender pair combinations. The lowest RA scores occur in cases with two people with different perceived gender presentations. Bottom: We present the resolution bias (Gap) for either a single person or two people. A bigger Gap score indicates a bias towards one perceived gender presentation. A positive Gap score shows a bias towards masculine-presenting subjects
U.S. Labor Force Statistics (Male Proportion)
Figure 3: Resolution bias relative to U.S. Labor Force Statistics. We compare the gender accuracy Gap per occupation to U.S. Labor Force statistics data. While there are no patterns for CLIP, BLIP-2 shows a bias similar to the real-world data.
proportions—for occupations with fewer women, such as "engineer", the model correctly resolves men more often than women, and vice versa.
4.2 Retrieval Bias
We evaluate VLMs on retrieval bias in Tab. 3. We see that all models have positive Bias@5 and Bias@10 values, which suggests that images of masculine-presenting subjects in professional settings are retrieved more often than images of feminine-presenting subjects, despite the candidate images always being gender-balanced.
5.1 Key Findings
Models struggle to resolve pronouns in the presence of both perceived presentations of genders We found that all VLEs show close to random performance on gender resolution when there are people
Table 3: Retrieval bias. We present mean and standard deviation across all occupations. Positive Bias@K shows more images of men were retrieved.
Model	Bias@5	Bias@10	MaxSkew@5	MaxSkew@10	NDKL
Mean	σ	Mean	σ	Mean	σ	Mean	σ	Mean	σ
CLIP [1]	0.11	0.38	0.16	0.22	0.27	0.15	0.18	0.13	0.19	0.07
OpenCLIP2B [22]	0.10	0.44	0.08	0.23	0.29	0.17	0.18	0.11	0.18	0.07
OpenCLIP400M [22]	0.17	0.47	0.11	0.22	0.33	0.18	0.16	0.13	0.19	0.07
SLIP [23]	0.06	0.52	0.00	0.24	0.32	0.21	0.17	0.12	0.19	0.09
DeCLIP [24]	0.11	0.40	0.15	0.26	0.28	0.16	0.20	0.14	0.17	0.07
FILIP [25]	0.01	0.43	0.03	0.26	0.29	0.16	0.17	0.13	0.18	0.07
of different perceived gender presentation in the scene. This hints at insufficient visuo-linguistic capabilities for handling complex scenes in current VLMs.
Captioning models have a higher accuracy and smaller accuracy gap between genders We find that captioning models outperform VLEs on all subtasks. We attribute this to the way resolution is done in captioning models—the pronoun of the subject is extracted using the start of the template and next token prediction. Meanwhile, VLEs need to rely on a global c1s text feature, which seems to not capture the nuanced difference between entities in the sentence.
Resolution and retrieval bias are not in the same direction Across models, there is not a consistent pattern of bias direction—VLEs are more accurate at resolving “her” pronouns, while BLIP models are more accurate for “his” pronouns. In contrast, we find that all VLEs are predominately biased towards retrieving images of masculine-presenting subjects. This highlights a risk of representative harm in deploying VLEs in image search systems.
The harms of performing poorly and exceptionally well on VISOGENDER In developing this benchmark, we recognise that performing at either end of the spectrum can have harmful side effects. Performing poorly on VISOGENDER can lead to increased gender bias through the use of VLMs when considering the representation of stereotypically feminine binary gender, which is already heavily discriminated in many historically male-dominated industries. For example, if an automatic captioning VLM in a downstream application repeatedly misgenders doctors as “he”, this weakens the representation of “women” as doctors too. The risk of erasure and misgendering via pronouns in caption assignment also harms the LGBTQIA+ community. An evaluative tool such as VISOGENDER is designed to flag a model’s bias prior to deployment. However, we also recognise that in order to do well on VISOGENDER, a model must perform well not only at scene disambiguation (based on the image subjects’ occupational roles) but also have capabilities for recognising (binary) gender. If the VISOGENDER dataset is misused, and does not comply with the terms of use prohibiting its use for training (see Supplementary Materials), there is a potential for increased gender recognition capabilities that can contribute to the development of automatic gender recognition technology. We recognise that the development of this technology denies people—especially transgender, non-binary, and gender-diverse individuals—their dignity, respect, and sometimes safety to exist in public and private spaces. We absolutely condemn any use of VLMs for automatic gender recognition in surveillance use cases.
Dataset collection with regards to privacy and consent All images in the VISOGENDER benchmark dataset are collected and used within the scope of their Creative Commons and royalty-free associated licences. However, we respect that these images depict real-life people, that may be misgendered in the development of this benchmark. Importantly, we do not make any claims to correctly assign a person's gender identification and we have included mechanisms for all data subjects to amend the labels and/or request removal of their images.
Subjective assessment when creating datasets The authors note this work is necessarily influenced by their respective identities. In a random order with pseudo-anonymised identifiers, we present our positionality: A1 is a White Bulgarian cisgendered man; A2 is a White British cisgendered woman;
A3 is a Black British cisgendered woman; A4 is a White South African cisgendered woman; A5 is a Chinese cisgendered man; and A6 is a White Brazilian cisgendered woman. Four authors are pursuing postgraduate degrees and two are pursuing undergraduate degrees, all at the University of Oxford. We draw on our experience in measuring and mitigating social biases in VLMs. Further, when designing, conducting and writing up this research, we consulted closely with domain experts in Gender Studies and members of the LGBTQIA+ community. We acknowledge that visual markers of gender presentation may not reflect a subject's self-identified gender as gender presentation does not necessarily align or reflect in a binary manner with one's sex, pronouns or identity. As such, we acknowledge that gender and gender identity is fluid and exists on a spectrum that is generally misrepresented by binary distinctions. However, irrespective of the societal model of gender, bias exists and leads to representational and allocational harms [8]. This work is not able to address the diversity of gender and other intersectional characteristics that come into the societal image of a typical person doing a certain job as well. However, we attempt at creating a benchmark that could help detect a single level of bias, which is the difference between typical Western presentation of someone perceived as masculine- or feminine-presenting in an occupational role. We advocate that this work is extended to include more genders and avoid erasure of non-binary individuals represented across occupations [69]. The codebase is designed to be flexible to include neopronouns in the future, in order to ensure these systems do not perform poorly when faced with data related to underrepresented communities [70].
Stacking biases from the internet We source images from a variety of search platforms (such as Google Image Search) and image hosting sites. While we balance included occupations across perceived presentation of gender search terms, those that we leave out are not "missing at random" due to biases that already exist in images on the internet. We could not find enough images for some occupations, e.g., there were not 5 images of a female-presenting plumber and a male-presenting client that met our criteria for data accessibility and format.
Dataset representation This dataset is only intended for evaluation purposes and, as such, requires fewer images than if it were used for training. However, the dataset is still relatively small, and excludes some "non-random" occupations omitted due to existing biases in images on the internet. This exclusion can, in turn, introduce bias into the gender and role depictions. It was beyond our means to partner with a StockImage provider, such as Getty Images [71], but this could be an avenue in future work to expand dataset size and include self-identified pronouns in order to counteract some of the aforementioned image availability biases. Future work could also augment the dataset with synthetic data from generative VLMs [12, 72].
We introduced VISOGENDER, a novel dataset for benchmarking social biases in VLMs for both pronoun resolution and retrieval settings. On some parts of the benchmark, we demonstrated that current state-of-the-art models perform no better than random chance, and that they do not perform equally well for resolving for both masculine and feminine gender presentations, nor give equal retrieval likelihood to images of masculine- or feminine-presenting professionals. There is significant headroom for improvement both in the reasoning abilities of VLMs, and in the gender gap of their abilities, when it comes to complex scenes with multiple humans. We hope this work encourages the benchmarking of future VLMs, so the risk of downstream harms and negative biases can be measured, compared and mitigated.
The authors would like to thank the following people for their feedback and insight during the development of VISOGENDER: Max Bain, Hugo Berg, Seb Wilkes, Juliana Mota, Daniel Kochin, Carolyn Dickson, Avishkar Bhoopchand and Rosemary Duffy. This work has been supported by the Oxford Artificial Intelligence student society, the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines & Systems [EP/S024050/1] (A.S.); the Economic and Social Research Council Grant for Digital Social Science [ES/P000649/1] (H.R.K.); and the Clarendon Fund in partnership with the St Cross College Scholarship, Oxford (F.G.A). For computing resources, the authors are grateful for support from Jonathan Caton and the Google Cloud and the CURE Programme under Google Brain Research.
[75] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144, 2022.
1 Introduction 1
2 Related Works 3
3 The VISOGENDER Benchmark 3
4 Results 6
5 Discussion 8
6 Conclusion 10
A Retrieval Bias Metrics 17
B Fine-Grained Resolution Bias Results 18
C Fine-grained Retrieval Bias Results 21
D Ablations 21
E Results for Additional VLMs 22
F Results Compared to U.S. Labor Force Statistics 23
G Error analysis 27
H VISOGENDER Dataset Criteria 28
I VISOGENDER Occupational Taxonomy 28
J How To Guide for Benchmarking a VLM 31
K VISOGENDER Data Clause 32
L VISOGENDER Datasheet 32
A Retrieval Bias Metrics
Here we define the retrieval bias metrics reported in the paper - Bias@K, MaxSkew@K, and NDKL.
Bias@K [63] measures the proportions of individuals with perceived masculine gender presentations and individuals with perceived feminine gender presentations images in the retrievals of a search result with a given text query. For an image  $I$ , we define a function  $g(I) \in \{0,1\}$  that denotes the perceived gender label of the image:
$$
g (I) = \left\{ \begin{array}{l l} 1 & \text {i f t h e g e n d e r o f t h e s u b j e c t i n I i s p e r c e i v e d a s m a s c u l i n e} \\ - 1 & \text {i f t h e g e n d e r o f t h e s u b j e c t i n I i s p e r c e i v e d a s f e m i n i n e .} \end{array} \right.
$$
Given a set of  $K$  retrieved images  $\mathcal{R}_K(q)$  for a query  $q$  of a certain occupation, we define the gender bias metric as:
$$
\operatorname {B i a s} @ \mathrm {K} (q) = \frac {1}{K} \sum_ {I \in \mathcal {R} _ {K} (q)} g (I),
$$
and let  $\mathrm{Bias}@\mathrm{K}$  be the average of  $\mathrm{Bias}@\mathrm{K}(q)$  over all occupations  $q \in Q$ .  $\mathrm{Bias}@\mathrm{K}$  is positive if the retrieval model skews toward professionals perceived with masculine gender presentations in retrieved images, and negative if retrieved for those with perceived feminine gender presentations.
Note that Bias@K approximates a form of representational fairness, and is only applicable to CLIP-like models.
$\mathbf{Skew}@\mathbf{K}$  [64, 11] measures the difference between the desired proportion of image attributes in  $\mathcal{R}_k(q)$  for the query  $q$  and the actual proportion. Let the desired proportion of images containing a professional of gender  $A$  in the set of retrieved images be  $p_{d,q,A} \in [0,1]$  and the actual proportion be  $p_{\mathcal{R}(q),q,A} \in [0,1]$ . The resulting  $\mathbf{Skew}@\mathbf{K}$  of  $\mathcal{R}(q)$  for a gender  $A \in \mathcal{A}$  is:
$$
\operatorname {S k e w} _ {A} @ \mathrm {K} (q) = \ln \frac {p _ {\mathcal {R} _ {K} (q) , q , A}}{p _ {d , q , A}},
$$
where  $p_{d,q,A}$  is the proportion of gender  $A$  of occupation  $q$  in our dataset, which always equals 0.5 (gender balanced).
A disadvantage of Skew@K is that it only measures bias with respect to a single gender at a time and must be aggregated to give a holistic view of the bias over all attributes. Following [11], we take the maximum value of Skew@K among all attribute labels  $A$  of the retrieved images for the text query  $q$ :
$$
\operatorname {M a x S k e w} @ \mathrm {K} (q) = \max  _ {A \in \mathcal {A}} \operatorname {S k e w} _ {A} @ \mathrm {K} (\mathcal {R} (q)),
$$
which gives us the "largest unfair advantage" [64] belonging to images within a given perceived gender presentation. Here, a MaxSkew@K of 0 for the attribute gender and a given text query  $q$  implies that masculine- and feminine-presenting subjects are equally represented in the retrieved set of  $K$  images  $\mathcal{R}_K(q)$ . MaxSkew@K is the average of MaxSkew@K(q) over all occupations  $q \in Q$ .
NDKL [65, 64] (normalised discounted cumulative KL-divergence) measures the distance of the retrieval model from a fair distribution, in a weighted average over  $K$ . Let  $D_{\mathcal{R}_K(q)}$  and  $D$  denote the binary distribution of occupational gender over the top  $K$  retrieved images and the desired distribution, respectively. Then the NDKL for occupation  $q$  is defined as:
$$
\mathrm {N D K L} (q) = \frac {1}{Z} \sum_ {i = 1} ^ {K} \frac {1}{\log_ {2} (i + 1)} D _ {\mathrm {K L}} (D _ {\mathcal {R} _ {K} (q)} \| D),
$$
where  $Z$  is a normalising factor  $Z = \sum_{i=1}^{K} \frac{1}{\log_2(i+1)}$ .
B Fine-Grained Resolution Bias Results
In this section, we present fine-grained resolution bias results by sector (Fig. 4), by specialisation (Fig. 5) and by occupation (Fig. 6).
Bias per sector Fig. 4 shows that CLIP is better at resolving perceived feminine gender presentations for most sectors, except for the medical sector, where CLIP performs better for perceived masculine gender presentations in some specialisations. BLIP-2, on the other hand, is better at resolving perceived masculine gender presentations in office environments.
Bias per specialisation Fig. 5 shows that the specialisations within a sector show similar trends for CLIP, but we observe inter-sector variations for BLIP-2. For example, within the service sector, food and household services are biased towards perceived masculine gender presentations, and fashion and animal services are biased towards perceived feminine gender presentations. Similarly, for office jobs, only legal are biased towards feminine gender presentations, and financial, structure, and general office jobs are biased towards perceived masculine gender presentations.
Bias per occupation Fig. 6 shows several occupations with zero accuracy resolution gap – auditor for CLIP, and doctor and lawyer for BLIP-2. For most occupations, CLIP is biased toward correctly resolving perceived feminine gender presentations, and the largest gaps we observe are for accountant, architect, engineer, and painter. BLIP-2 is most biased towards correctly resolving perceived masculine gender presentations in the auditor, broker and waiter occupations, and perceived masculine gender presentations in the counselor, paralegal and veterinarian occupations.
Figure 4: Resolution accuracy. Resolution bias (Gap) per Sector
Figure 5: Resolution accuracy. Resolution bias (Gap) per Specialisation
Figure 6: Resolution accuracy. Resolution bias (Gap) per Occupation
Figure 7: Retrieval bias by occupation. We show the bias per occupation of CLIP. Occupations are ranked by the Bias@5 values. Positive Bias@K shows more images of masculine-presenting subjects were retrieved for the given occupation.
Figure 8: Resolution accuracy for template flipping. The resolution accuracy is calculated for all gender pair combinations. The subject of the template refers to either the occupation or the participant.
C Fine-grained Retrieval Bias Results
Here we present retrieval bias results per occupation. In Fig. 7, we see that most occupations (15 out of 23) are skewed towards masculine-presenting subjects in both Bias@5 and Bias@10.
D Ablations
D.1 Ablations on Resolution Bias Results
Template flipping We change the subject of the prompt sentences for images with two people by reordering the {participant} and {occupation}, e.g., "The doctor and his patient" becomes "The patient and her doctor". We compare both templates in Fig. 8. While we observe similar trends in the two settings, the resolution accuracy of CLIP is worse when the pronoun refers to the participant. Neutral pronoun resolution Here we attempt to move away from binary gender classification and introduce a third, neutral pronoun - "their", which is always grammatically correct. In Tab. 4 we see