Title: CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark
Benchmark: CVQA
Summary: The paper introduces CVQA2, a culturally diverse multilingual Visual Question Answering benchmark that includes 10,000 questions across 31 languages and 30 countries, incorporating input from native speakers and cultural experts.  The benchmark measures cultural understanding and multilingual visual question answering.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: cultural understanding and multilingual visual question answering
Phenomenon defined by authors: Yes
Phenomenon definition: The phenomenon is defined as a model’s ability to understand local common-ground knowledge.

Culture is hard to define, and our CVQA ultimately serves only as a proxy to benchmark the model’s understanding of culture through local common knowledge. ...we follow Adilazuarda et al. by using common-ground knowledge (e.g., information surrounding local dishes, history, places, etc. that is generally shared by the people within the region) as a proxy of culture.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: multiple-choice visual question answering
Task item definition: an image, question, answer options, the correct answer
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 10k
Dataset metadata available: Yes
Dataset metadata detail: Country-Language pair, language and script used, question category (10 types, e.g., Food, Pop Culture, Geography), image source (self-made vs. web-sourced), question type (e.g., “what,” “how,” “where”) via string-matching heuristics
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Language: English vs. local language performance 
Question categories: e.g., Food, Pop Culture, People & Everyday Life
Image source: self-made vs. web images
Prompt type: location-aware vs. location-agnostic
Input format: multiple-choice vs. open-ended answers
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: No
Taxonomy: Knowledge / Cultural
New taxonomy: General Application / Cultural Knowledge