Title: Image Retrieval from Contextual Descriptions
Benchmark: ImageCoDe
Summary: The paper proposes a new multimodal challenge, Image Retrieval from Contextual Descriptions (ImageCoDe), to assess vision-and-language models’ ability to integrate context cues into interpretation of linguistic utterances. Models such as ViLBERT and CLIP are evaluated and found to lag significantly behind human performance.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: retrieve images based on textual descriptions
Phenomenon defined by authors: Yes
Phenomenon definition: "we present a new challenge that requires multimodal models to leverage context to retrieve images from text. In particular, given a contextual description and a set of minimally contrastive candidate images, i.e. differing only in some details, the model has to retrieve the target image."
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Retrieving the correct image from a set of minimally contrastive candidates based on a contextual description.
Task item definition: Each sample consists of a brief textual description, ten candidate images, and the index of the target response.
Task source: Real task examples (e.g. GitHub issues), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 2306
Dataset size notes: Train: 16594; Validation: 2302
Dataset metadata available: Yes
Dataset metadata detail: average length, average # sentences, number of word types
Sampling method: Random sample (creators defined a task space and sampled from it), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric subscores detail: video frames, static pictures
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Retrieval / 
New taxonomy: General Application / Document Retrieval