Title: VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
Benchmark: VALSE
Summary: This paper proposes VALSE (Vision And Language Structured Evaluation), a novel benchmark designed for testing general-purpose pretrained vision and language (V&L) models for their visio-linguistic grounding capabilities on specific linguistic phenomena. The authors cover a broad spectrum of basic linguistic phenomena affecting the linguistic and visual modalities.  The overall weak performance of these models indicates that there is a need for a reliable foiling dataset targeting the visual grounding capabilities of V&L models through the lens of linguistic constructs.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Visio-linguistic grounding
Phenomenon defined by authors: Yes
Phenomenon definition: The phenomenon is defined as the ability of VLMs to ground linguistic phenomena, from morphosyntax to semantics, in the visual modality. For example, recent evidence suggests that models are insensitive to linguistic distinctions of verb-argument structure and word order. 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: VALSE is composed of 6 pieces, each corresponding to a specific linguistic phenomenon: existence, plurality, counting, relations, actions and coreference. For all pieces, given a visual input, a model is asked to distinguish real captions from foils, where a foil is constructed from a caption by altering a word or phrase that realizes a specific linguistic phenomenon, e.g., semantic number of nouns, verb argument structure, or coreference. 
Task definition: Two tasks: 
- Binary classification: predict whether an image-sentence pair is foiled
- Predict image-sentence matching score between the image and the caption vs the image and the foil caption
Task item definition: piece category (sub-phenomenon), image, caption, foil
Task source: Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 5668
Dataset metadata available: Yes
Dataset metadata detail: piece category (sub-phenomenon), difficulty
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean, Weighted Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: difficulty, sub-phenomenon
Metric metascoring: pairwise ranking accuracy
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: VQA / 
New taxonomy: General Capability / Grounding