Title: WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models
Benchmark: WinoGAViL
Summary: WinoGAViL introduces a gamified benchmark where humans create vision-language association tasks that are easy for humans but challenging for AI models. Inspired by Codenames, it evaluates models’ abilities to reason about commonsense associations between textual cues and visual candidates.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multimodal commonsense reasoning via visual-textual association
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to reason about abstract associations between a textual cue and a set of images, incorporating commonsense knowledge, abstraction, and general world understanding.
Definition integrity: Single cohesive phenomenon
Phenomenon contested: Widely-agreed
Task definition: Given a textual cue and a set of candidate images, select the images most closely associated with the cue.
Task item definition: Each instance includes a single-word textual cue and 5–12 images; the task is to select k images that best match the cue.
Task definition detail: Associations are generated adversarially against AI models and validated by multiple human players to ensure human-solvability.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 3,568
Dataset metadata available: Yes
Dataset metadata detail: Metadata includes cue text, selected images, number of candidates, human agreement scores, model performance, reasoning type annotations (e.g., visual similarity, general knowledge).
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Jaccard Index between model predictions and human-labeled associations
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Scores stratified by number of candidates (5–6 vs 10–12) and by reasoning type (visual, general knowledge, abstraction, etc.)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Validation includes new human solvers, human-machine agreement measures, category-wise error analysis, and Jaccard agreement distribution.
Taxonomy: Reasoning / Commonsense
New taxonomy: General Capability / Commonsense Reasoning