SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Neutral pronoun resolution. We measure resolution rates into neutral pronouns. We also show  $\Delta_N$ , which is the difference between the ratio of images with both masculine and feminine perceived presentations of gender (P.P.Gender) that were resolved with a neutral pronoun.
Model	Overall	Single Person	Two people
Rneutral	ΔN	Overall	Same P.P.Gender	Different P.P.Gender
Rneutral	ΔN	Rneutral	ΔN	Rneutral	ΔN
CLIP [1]	0.17	0.07	0.12	0.26	0.09	0.20	0.17	0.31	0.02
BLIP-2 [26]	0.01	0.01	-0.02	0.00	0.00	0.00	0.00	0.00	0.01
Table 5: Retrieval bias with template flipping. We present mean and standard deviation across all occupations, when the pronoun refers to the participant in the occupation-participant pair. Positive Bias@K shows more images of masculine-presenting subjects were retrieved.
Model	Bias@5	Bias@10	MaxSkew@5	MaxSkew@10	NDKL
Mean	σ	Mean	σ	Mean	σ	Mean	σ	Mean	σ
CLIP [1]	0.13	0.41	0.10	0.23	0.29	0.16	0.17	0.14	0.19	0.08
OpenCLIP2B[22]	0.04	0.36	0.06	0.24	0.26	0.13	0.16	0.13	0.16	0.05
OpenCLIP400M[22]	0.23	0.40	0.10	0.25	0.30	0.18	0.16	0.16	0.18	0.08
SLIP [23]	-0.18	0.49	-0.06	0.26	0.35	0.18	0.17	0.14	0.21	0.09
DeCLIP [24]	0.15	0.49	0.13	0.27	0.35	0.16	0.20	0.14	0.19	0.08
FILIP [25]	-0.01	0.38	0.03	0.29	0.27	0.14	0.19	0.15	0.17	0.06
that while BLIP-2 almost never chooses the neutral pronoun, it is selected by CLIP in  $17\%$  of all images and  $31\%$  of images containing two people with different perceived gender presentations. We also see that for the more difficult settings, the neutral pronoun is selected more frequently, with  $31\%$  in the "two people, different gender" setting, which corresponds to almost random chance  $(33\%)$ . Finally, we see that images, where the perceived gender presentation is masculine, tend to be resolved as neutral more often.
D.2 Ablations on Retrieval Bias Results
We evaluate retrieval bias when reversing the order in which {occupation} and {participant} occur. We present results in Tab. 5. While most bias measures are of similar magnitude, Bias@5 and Bias@10 for SLIP are negative – meaning more images of individuals perceived with feminine gender presentations are retrieved. Meanwhile, as reported in the main paper, using the original template, these bias measures were positive.
E Results for Additional VLMs
We repeat the analysis in the main paper on additional VLMs – ALIGN [73], FLAVA [74] and GroupViT [75]. These models, similar to the other VLEs evaluated, are biased towards resolving images of individuals with perceived feminine gender presentations more accurately than masculine-presenting subjects, i.e., have a negative gender accuracy gap (see Tab. 6a). The exception is ALIGN in the two-person case, with a slightly positive in the gender accuracy gap. The performance is
Model	Overall	Single Person	Two people	ZS Imagenet
RAavg	Δ	Overall	Same P.P.Gender	Diff. P.P.Gender
RAavg	Δ	RAavg	Δ	RAavg	Δ
ALIGN [73]	0.52	0.78	-0.33	0.26	-0.02	0.42	-0.01	0.10	0.03	85.5
FLAVA [74]	0.73	0.91	-0.06	0.54	-0.29	0.75	-0.22	0.34	-0.35	-
GroupViT [75]	0.64	0.79	-0.16	0.50	-0.09	0.64	-0.11	0.34	-0.07	-
(a) Resolution bias. We present resolution accuracy averaged for masculine and feminine perceived presentations of gender (P.P.Gender); resolution accuracy gap  $\Delta$  as defined in the main paper and zero-shot classification accuracy on Imagenet [67].
Model	Bias@5	Bias@10	MaxSkew@5	MaxSkew@10	NDKL
Mean	σ	Mean	σ	Mean	σ	Mean	σ	Mean	σ
ALIGN [73]	0.10	0.42	0.06	0.30	0.29	0.16	0.22	0.14	0.18	0.06
FLAVA [74]	0.10	0.47	0.11	0.20	0.31	0.18	0.16	0.11	0.20	0.07
GROUPVIT [75]	0.18	0.37	0.16	0.25	0.28	0.16	0.18	0.17	0.18	0.07
(b) Retrieval bias. We present mean and standard deviation across all occupations for two-person images.
Table 6: Additional bias results.
Table 7: Pearson's correlation coefficiants and Kendall  $\tau$  when the resolution accuracy results compared to the U.S. Labor Force Statistics (Male Proportion)
Pearson's r	Kendall τ
Occupation | Participant	CLIP	BLIP-2	CLIP	BLIP-2
Same pair	M | M	-0.45	0.40	-0.31	0.36
W | W	-0.23	0.12	-0.24	0.05
Diff. pair	M | W	-0.28	0.42	-0.28	0.22
W | M	-0.05	-0.53	-0.06	-0.42
significantly worse in the two-person case, with worst performance for resolution accuracy coming from the setting of different gender subjects. FLAVA has highest overall resolution accuracy, along with the biggest gender gap scores relative to the other two models. Similar to the other VLEs evaluated, we find that the models have a bias toward retrieving images of masculine-presenting subjects in occupational settings (see Tab. 6b).
F Results Compared to U.S. Labor Force Statistics
The U.S. Labor Force Statistics publishes an annual report of population demographics across occupations. We record the male/female proportion in the U.S. Labor Force statistics by matching each VISOGENDER occupation to either a single U.S. occupation category or by taking a mean across any relevant categories. For VISOGENDER jobs that cover multiple U.S. categories, we also report the standard deviation. The statistics used are predominantly from 2022, with the exception of medical - specialist occupation which is taken from 2020. This mapping and accompanying metadata is available in our GitHub repository: https://github.com/oxai/visogender.
F.1 Evaluating Resolution Bias Relative to U.S. Skew
In Fig. 9 and Fig. 10, we compare the resolution accuracy of CLIP and BLIP-2 per occupation to the proportions of U.S. men working in that occupation. The correlation coefficients for each occupation-participant pair is summarised in Tab. 7. There is no correlation between the resolution accuracy scores and the proportions of the men in the U.S. Labor Force occupation statistics, with the exception of CLIP same pair (M-M) and BLIP-2 different pair (W-M) showing some correlation, that is not significant.
F.2 Evaluating Retrieval Bias Relative to U.S. Skew
In Fig. 11, we compare the retrieval bias of CLIP to the U.S. male proportions for each occupation. We find that for CLIP, OpenCLIP-400M, and OpenCLIP-2B, the Bias@10 metric (skew toward men) is slightly positively correlated with the proportion of male in each occupation in the U.S., in both Pearson's  $r$  and Kendall's  $\tau$ . Although this correlation is not significant, it reflects potential bias in the methods or dataset of CLIP and OpenCLIP that warrants future investigation.
Figure 9: Resolution accuracy. CLIP: Mapping of Resolution Accuracy for the same and different gender pairs to the U.S. Labor Force Statistics (Male Proportion)
Figure 10: Resolution accuracy. BLIP-2: Mapping of Resolution Accuracy for the same and different gender pairs to the U.S. Labor Force Statistics (Male Proportion)
Figure 11: Retrieval bias. Comparison of the Bias@10 scores of CLIP to the to the U.S. Labor Force Statistics (Male Proportion).
G Error analysis
These analyses are included to ensure the presented results are due to gender bias, and not necessarily from variance in the how many and which images were sampled for inclusion in the dataset.
G.1 Resolution Bias Error Analysis
In this analysis, we demonstrate that sub-sampling our dataset does not skew the results by much on average. We ran multiple experiments with random sub-samples of the dataset and measured the variance in resulting  $\mathrm{RA}_{\mathrm{avg}}$  and accuracy gap  $\Delta$ . In particular, we randomly keep  $k$  out of the 5 images per each perceived presentation of gender pair and each occupation, and repeat this trial for 500 times for each  $k$  (see Tab. 8). We find that all standard deviations are small ( $< 0.02$  in every metric) compared to their mean even if  $k = 1$ . This means even if we had a dataset with  $20\%$  the original size, we would likely have observed the same result (e.g., see Fig. 12). For instance, the difference between models (CLIP, OpenCLIP, etc.) for the same metric is often on a order much larger than 0.02, showing choice of models having a much larger impact on the metrics than the choice or number of images. This shows that despite our dataset being small, it is large enough such that results are significant.
Figure 12: Random analysis showing the variance in overall accuracy (mean  $\approx 0.75$ ) over 500 trials as the number of images kept varies.
G.2 Retrieval Bias Error Analysis
We compared our existing retrieval bias Bias@10 to those from experiments where: of the 20 images for each occupation, 10 images are randomly chosen and relabeled as "his" and the other 10 as "her" (in other words the top 10 retrieved images are chosen randomly from the 20 images). This is repeated 3000 times, with results presented in Tab. 9. In summary, the average Bias@10 is 0.0003 (which should approach 0), with a standard deviation of 0.0475. Our actual Bias@10 for CLIP is 0.1565,
Table 8: Resolution bias random sub-samples. We show mean and standard deviation of each metric, over 500 runs on subsets of our dataset where  $k = 1$  image is kept per each group of 5.
Overall	Single Person	Two people
RAavg	Δ	Overall	Same gender	Different gender
RAavg	Δ	RAavg	Δ	RAavg	Δ
Mean	0.7490	0.9216	-0.1392	0.5763	-0.2652	0.7871	-0.1825	0.3655	-0.3479
σ	0.0038	0.0051	0.0101	0.0053	0.0113	0.0068	0.0140	0.0082	0.0176
Table 9: Retrieval bias random splits. We present the mean and standard deviation of each metric (in columns), following 3000 runs of random split experiments.
Bias@5	Bias@10	MaxSkew@5	MaxSkew@10	NDKL
Mean	σ	Mean	σ	Mean	σ	Mean	σ	Mean	σ
Mean	0.0014	0.3937	0.0003	0.2271	0.2769	0.1467	0.1504	0.1261	0.1673	0.0609
σ	0.0821	0.0563	0.0475	0.0335	0.0307	0.0223	0.0260	0.0164	0.0129	0.0110
which is more than 3 standard deviations away from the expectation. This is repeated for the all metrics (Bias@5, Bias@10, MaxSkew@5, MaxSkew@10, NDKL) and all models (CLIP, OpenCLIP, SLIP, DeCLIP, FILIP), and most of them show the models being more biased than random retrievals (mostly at least 1 standard deviation away). This shows that our models are more biased when we split by perceived gender presentation, as opposed to randomly splitting.
H VISOGENDER Dataset Criteria
Data collection was carried out by the authors of the paper from March to May 2023 on a variety of image databases and search providers, such as Pexels and Google Image Search. We followed a set of guidelines to specify exclusion and inclusion criteria:
1. There is either only one or two people in the image, depending on the task,
2. The image does not contain any children,
3. The images are safe-for-work,
4. The image is a photograph (in colour or black and white) but not e.g., a cartoon,
5. The image does not contain stock photo watermarks,
6. The image can be accessed by a URL,
7. The image is under a Creative Commons licence, and does not fall under a specific clause disallowing its use for machine learning purposes.
I VISOGENDER Occupational Taxonomy
The taxonomy, as indicated in Fig. 13, is characterised according to a three tier system: Sector (the general field), Specialisation (subcategories within the Sector) and Occupation (job categories nested within the Specialisation). Tab. 10 outlines the number of images per Sector and Specialisation category.
Table 10: Dataset statistics. Number of images per sector and specialisation.
Single-person
{Occupation}-{Object}	Two-person
{Occupation}-{Participant}
Sector	Education	20	40
Medical	40	80
Office	90	180
Retail	20	40
Service	60	120
Specialisation	General courses	10	20
Institutional	10	20
Mental heath	10	20
Hospital worker	30	60
Legal	20	40
Financial	40	80
General office	10	20
Structure	20	40
Customer facing	20	40
Food service	30	60
Fashion	10	20
Animal	10	20
Household	10	20
Figure 13: Taxonomy of the VISOGENDER dataset based on Winogender occupations [19]. All participant roles indicated by a * have been adapted from Winogender.
J How To Guide for Benchmarking a VLM
A VLM can be evaluated on our benchmark using the provided code (https://github.com/oxai/ visogender). The benchmark scores are saved in JSON files, as depicted below.
To perform well on VISOGENDER, the scores should be optimised as follows:
- Resolution bias:
- resolution accuracies should be as close to  $100\%$  as possible indicating a high capability in performing gender coreference resolution
- the gender gap score should be as close to zero as possible to ensure the model is not biased towards either gender
- Retrieval bias: all scores should be as close to zero as possible to demonstrate a model is not biased towards either gender.
{
"visogender_blip2_results": {
"metadata": {"experiment_desc": "CAPTIONING", "model_name": "blipv2"},
"resolution.bias": {
"All_images": {"overall.accuracy": 0.84},
"single_person_images": {"RA_avg": 0.92, "gender_gap": -0.09},
"two_person_images": {"RA_avg": 0.76, "gender_gap": 0.07},
"two_person_images同樣_gender": {"RA_avg": 0.93, "gender_gap": 0.06},
"two_person_images_diff_gender": {"RA_avg": 0.60, "gender_gap": 0.08}
}
}
{
"visogenderclip_results": {
"metadata": {"experiment_desc": "CLIP", "model_name": "clip"},
"resolution.bias": {
"All_images": {"overall.accuracy": 0.75},
"single_person_images": {"RA_avg": 0.92, "gender_gap": -0.14},
"two_person_images": {"RA_avg": 0.57, "gender_gap": -0.27},
"two_person_images_same_gender": {"RA_avg": 0.79, "gender_gap": -0.18},
"two_person_images_diff_gender": {"RA_avg": 0.36, "gender_gap": -0.35}
},
"retrieval.bias": {
bias@5: {"mean": 0.13, "sigma": 0.35},
bias@10: {"mean": 0.16, "sigma": 0.22},
maxskew@5: {"mean": 0.25, "sigma": 0.15},
maxskew@10: {"mean": 0.18, "sigma": 0.13},
NDKL: {"mean": 0.18, "sigma": 0.07}
}
}
K VISOGENDER Data Clause
K.1 Terms of Use
This dataset is solely intended for use as a benchmark for evaluating vision-language models under the constraints of the licence. This dataset is strictly not to be used for training under any circumstances.
K.2 Licence
The VISOGENDER dataset only contains URLs that reference images that (at the time of curation) are under Creative Commons and/or royalty free licences that allow for their use and distribution. No images are stored directly. The VISOGENDER dataset is bound under a CC-BY 4.0 licence and is used as such by the authors. However, it is important to note that individual images in the model may have licences that do not allow commercial use, and users of this dataset will assume liability if they use the dataset beyond the terms of use as indicated by the benchmark. The authors do not take responsibility for any licences that change with time.
The authors confirm that, to the best of their knowledge, they are using all intellectual property in accordance with their licences, and the use of the data as stipulated in this terms of use and the accompanying manuscript and GitHub repository does not violate any rights. The GDPR allows the processing of personal data for research purposes and only includes the URLs so there is no personal data shared.
K.3 Dataset Maintenance
The URLs are curated manually, and at the time of collection in April - May 2023, they did not point to any images containing harmful or disturbing imagery, nor do any images depict children. The associated metadata is provided by manual labelling, and is based on Google and Pexels image search query tags.
The authors undertake to do the following:
- The authors will proactively investigate the dataset for broken links, with randomised checks of the images themselves to ensure URLs are not redirecting every 6 months
- We have uploaded code and instructions to GitHub for easy command line running of a script which checks for URL integrity and that the images can be utilised by models. This will be run by the authors every 6 months
We also welcome feedback and scrutiny from the community making use of the benchmark. In order to facilitate this process, we put forward the following:
- We have included instructions for running the code to check the dataset.
- Further, we have included a Google Form which can be used to identify broken and/or inappropriate links
K.4 Reporting and/or Addressing Issues with the Dataset
In the event that there are any issues with the dataset, or any specific links to images, or associated images, please contact the authors by filling out this Google Form and any offending information will be removed immediately. These issues can include, but are not limited to issues with deprecated links, links that have redirected to disturbing, inappropriate content, or you would like images related to yourself, personally removed.
L VISOGENDER Datasheet
We present a Datasheet for the VISOGENDER dataset [59] which is available on GitHub: https://github.com/oxai/visogender/. The information in the datasheet is up to date as of June 2023. Any amendments to the datasheet subsequent to this version will be made on GitHub.
L.1 Motivation
For what purpose was the dataset created? The dataset was created as a benchmark to evaluate vision-language models (VLM). Specifically, it is designed to measure a model's abilities and biases for gender pronoun coreference resolution and image retrieval in an occupational setting.
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organisation)? The authors of the paper, as a part of the Oxford Artificial Intelligence Society Research Labs.
Who funded the creation of the dataset? N/A
Any other comments? None.
L.2 Composition
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? The dataset comprises images of people in a professional setting. There is at least one person in the image which represents the professional person as referred to by their occupation.
How many instances are there in total (of each type, if appropriate)? Single person: 230 | Two people in the image: 460 | Total number of images: 690. See Tab. 1 for more information.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? This images were curated from Google image search (Creative Commons Licence) and Pexels images which are royalty free. In most cases, we took the first 5 images per gender-pair and occupation meeting the criteria and guidelines. Our restrictive licensing criteria and resource constraints (i.e., limiting partnerships with StockPhoto providers) meant that the set of total images to chose from was limited. In some cases, there were insufficient images that met the criteria. However, compared to all images on the internet of individuals in professional settings, this is a non-random subsample.
What data does each instance consist of? Each image contains one person, as referred to by their occupation, or, in the case of two people: one person referred to by their occupation and another referred to by a participation noun, relative to the interaction with the professional. In the case of one person in the image, they are accompanied by an object.
Is there a label or target associated with each instance? Yes, there is a perceived presentation of gender label and the labels refer to the professional person and their participant (if applicable). There is an object label as well, in the case of a single person being present. There are also labels related to the occupation, based on the taxonomy.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? N/A
Are there recommended data splits (e.g., training, development/Validation, testing)? This is purely an evaluation set. It is not intended for training purposes.
Are there any errors, sources of noise, or redundancies in the dataset? At the time of initial release, there are no errors, redundancies or sources of noises to the best of the authors' knowledge, based on internal review.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? The dataset contains URLs that reference images that (at the time of curation) are under Creative Commons and/or royalty free licences that allow for their use and distribution.
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? The data collected (URLs) does contain identifiable information. The metadata doesn't contain any personal identifiable information such as names, emails etc.. However, the images are only pseudonymised as the faces are recognisable. However, they are publicly available under Creative Commons and royalty-free licenses.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? The URLs are curated manually, and at the time of collection, they did not point to any images containing harmful or disturbing imagery, nor any images containing children. Any URL endpoints that change to become problematic or are determined to infringe on privacy will be removed immediately.
Does the dataset identify any subpopulations (e.g., by age, gender)? The images are of people with identifiable features that can infer such information. However, no identifiable or personal information is shared with the dataset. However, we do infer a perceived gender presentation that is necessarily influenced by our internal biases and we acknowledge we risk misgendering a person depicted in the image. We have included opt-out mechanisms and the option to report any adjustments to our labels. Please see this Google Form. The authors are notified immediately when an entry is submitted to the form.
Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? The images depict real people who can be identified but there is no text data linked to their physical image. These images are publicly available under Creative Commons and/or royalty free licences, and our terms of use, and use of VISOGENDER in research is not in violation of these licences. We rely on the informed consent processes to which we do not have access.
Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? Only race can potentially be inferred from the images. To the best of the authors' knowledge, these public images were not collected alongside this sensitive information.
Any other comments? We have included opt-out options and the option to request amendments to the labelling process. This is available in our GitHub repository: https://github.com/oxai/ visogender.
L.3 Collection Process
How was the data associated with each instance acquired? Search queries such as "a photo of a {gendered adjective} {occupation} and a {gendered adjective} {participant}" or "a photo of a {gendered adjective} {occupation} and a {object}" were used on Google image search (with the Creative Commons licence filter) and Pexels image search.
What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? Image search queries, and manual labelling using a shared Google Sheets document.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? N/A
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? The authors of the paper were responsible for the data collection, and no compensation was provided. No external parties were involved.
Over what timeframe was the data collected? Two months in 2023 (March to May).
Were any ethical review processes conducted (e.g., by an institutional review board)? No, as the images are under open licences, and no human subjects were recruited for the purpose of the study.