Title: ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models
Benchmark: ViLMA
Summary: This paper introduces VILMA, a zero-shot benchmark for evaluating VidLMs, designed to require strong temporal understanding. The authros adopt the following methodology: (i) they harvest high-quality examples from existing video-language datasets; (ii) they create counterfactual examples or ‘foils’, so that a test requires distinguishing correct from counterfactual video+text pairs; (iii) they create a proficiency test to gauge if a model learns the capabilities we deem necessary to solve the main test; (iv) they apply automatic and manual validation of the examples and their counterfactuals to control for biases and to ensure a high-quality evaluation benchmark.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: temporal grounding in video-language models
Phenomenon defined by authors: Yes
Phenomenon definition: In principle, VidLMs can visually ground linguistic phenomena which are beyond the reach of
image-language models (ILMs), since videos include dynamically evolving phenomena (e.g., events, actions, physical processes). This temporal dimension makes learning more complex.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: - The Action Counting task probes the ability of models to accurately count the occurrences of actions within a given video input stream. 
- The Situation Awareness task shows how effectively VidLMs grasp the interaction between visual clues and verbal context by testing whether they recognise actors, actions, and their relationships.
- The Change of State task examines the ability of VidLMs (i) to recognise and distinguish different sub-phases of actions, especially those that induce a change of state (CoS) of objects or entities involved in it; and (ii) to align the beginning and ending phases of these actions across modalities.
- The Rare Actions task probes how well VidLMs identify novel compositions and recognise unusual interactions between human beings and objects. 
- The Spatial Relations task focuses on the ability of models to distinguish different spatial and spatio-temporal relations related to the actions carried out in a video (e.g. moving an object ‘over’, or ‘towards’ another object). 
Task item definition: Video frames, ground truth video caption, foil video caption
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 5,934
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: results are stratified based on the temporal dimension/temporal task
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: They manually checked every video-caption-foil sample, retaining only those in which the foil was unambiguously false with respect to the input video. This resulted in the removal of 1278 (15.11%) of samples in the proficiency tests. The main tests were validated independently, in a study conducted on AMTurk.
Taxonomy: Grounding / 
New taxonomy: General Capability / Grounding