Title: WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models
Benchmark: WikiDO
Summary: The authors argue that current VLM benchmarks are insufficient to assess the OOD generalization capability of models due to high visual and linguistic similarity between the evaluation and finetuning datasets. The propose WIKIDO which consists of image-text data derived from Wikipedia Diversity Observatory, a diverse source of Wikipedia articles spanning several diversity axes including geography, gender, ethnicity and domains/topics.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Generalization / OOD performance
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: The proposed dataset can be used for both image-to-text, i.e. retrieve the most relevant textual description(s) from a set, and text-to-image retrieval, i.e. retrieve the most relevant image(s) from a dataset.
Task item definition: A single row in the dataset will have the path of the image, the Wiki ID of the image, the reference text from Wikipedia, the title of the wikipedia article, the topic label from Wikipedia Diversity Observatory and the generated caption of the image
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: train: 384K pairs, 2 test sets (ID and OOD) of size 3K each.
Dataset size notes: train: 384K pairs, 2 test sets (ID and OOD) of size 3K each.
Dataset metadata available: Yes
Dataset metadata detail: topic
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train
Response format: Retrieval 
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: In-distribution vs Out-of-distribution
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors show that across various settings, nearly all models perform better on in-distribution (ID) data than on out-of-distribution (OOD) data, except for CLIP, which performs equally well in both settings.
Taxonomy: Retrieval / 
New taxonomy: General Application / Document Retrieval