Title: From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models
Benchmark: GlobalRG
Summary: Current benchmarks often overlook a crucial aspect of cultural diversity: how universal concepts are represented across cultures. GlobalRG addresses this gap with two tasks inspired by popular vision-and-language benchmarks, image-text retrieval and visual grounding. Extensive evaluations reveal notable cross-cultural discrepancies. In particular, they show that even when models retrieve or ground images that appear culturally diverse, those images frequently share underlying Western-centric elements.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: multicultural visual understanding
Phenomenon defined by authors: Yes
Phenomenon definition: Models' pre-training on large-scale datasets tends to predominantly contain images from Western cultures. The underrepresentation of certain cultures in the data translates into performance disparities across cultures.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: - 1st task: Cultural visual grounding, which evaluates models' ability to ground culture-specific concepts within an image.

- 2nd task: Retrieval across universals: They also introduce the novel task of Retrieval across
Universals, aimed at retrieving culturally diverse images for a given universal concept.  Formally, let Q be a set of textual queries representing universal concepts, and I the set of images from different cultures. Given a query, the goal is to retrieve a ranked list of images R that maximizes both relevance and cultural diversity.
Task item definition: a cultural concept, the image, the region/country
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 3,000
Dataset metadata available: Yes
Dataset metadata detail: region/country
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall), Diversity@k: that measures the cultural diversity among the retrieved images, helping to identify models’ bias towards specific countries or regions.
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Country, region, culture
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Knowledge / Cultural
New taxonomy: General Application / Cultural Knowledge