Title: Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models
Benchmark: Visual Riddles
Summary: This paper introduces Visual Riddles, a benchmark designed to evaluate LLMs on complex visual reasoning tasks that require commonsense and world knowledge. The dataset includes 400 carefully crafted riddles, each combining images, questions, and textual hints, revealing significant performance gaps between current models and human reasoning abilities.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: knowledge, visual understanding
Phenomenon defined by authors: Yes
Phenomenon definition: While humans easily recognize such contextual nuances, existing image-understanding models struggle to integrate visual cues with world knowledge stemming from cultural aspects, life-experiences, and physical or social knowledge
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: (1) Main Task: Solve open-ended questions. 
(2) Utilizing Hints: Use textual aids to identify key visual clues in riddles. 
(3) Employing Attributions: Apply web-sourced attributions to improve world-knowledge. 
(4) Multiple Choice: Select the correct answer to the riddle from five options. 
(5) Automatic Evaluation: Evaluate open-ended answers in two scenarios— Reference-Free, assessing the correctness of a candidate answer (CA) based only on the visual riddle, and Reference-Based, comparing CAs to the ground truth answer (GTA).
Task definition detail: it covers five different subtasks
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 500
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: They use human rating as primary evaluation method, while also compare the performance between LLM-as-a-judge and human rating in ablation study
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Open-ended VQA, Multiple-choice VQA, Open-ended VQA Automatic Evaluation
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: VQA / 
New taxonomy: General Capability / Commonsense Reasoning