Title: VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution
Benchmark: VisoGender
Summary: This paper examines occupation-related gender bias in vision-language models. The authors introduce VisoGender, a new benchmark of images associated with captions containing pronoun relationships of the depicted subjects and objects. They use VisoGender to probe resolution bias (difference between pronoun resolution accuracies for image subjects with masculine versus feminine gender presentations) and retrieval bias (bias in ratios of professionals with masculine and feminine gender presentations retrieved for a gender-neutral search query). Experiments on VisoGender provide evidence for bias in state-of-the-art vision-language models.
Phenomenon type: Specific form of bias
Target phenomenon: The benchmark measures occupation-related gender bias in vision-language models.
Phenomenon defined by authors: Yes
Phenomenon definition: "stress-testing gender bias in visual-linguistic reasoning and coreference resolution capabilities" (p. 2)
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The authors define two tasks. In the resolution task, the model is provided with a single image (either of an occupation-object or occupation-participant scene) and must rank the likelihood of captions containing different gender pronouns. In the retrieval task, the model is provided with a single gender-neutral caption and must retrieve images from a set containing professionals with different perceived genders.
Task item definition: Resolution task: image and candidate captions. Retrieval task: caption and candidate images.
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: Authors specify templates and search for images matching those templates among existing image databases as well as search providers.
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Resolution bias: constructed. Retrieval bias: representative.
Dataset size notes: 690
Dataset metadata available: Yes
Dataset metadata detail: Difficulty (single subject, two subjects of the same perceived gender presentation, two subjects of different perceived gender presentations)
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Ranking of images
Response format detail: Resolution task: multiple choice. Retrieval task: ranking of images.
Metric definition: Resolution task: accuracy gap. Retrieval bias: Bias@K, Skew@K, NDKL.
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Based on difficulty: single subject, two subjects of the same perceived gender presentation, two subjects of different perceived gender presentations
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Alignment / Bias
New taxonomy: General Capability / Bias