Title: JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
Benchmark: JourneyBench
Summary: JourneyBench is a diverse vision-language understanding benchmark using AI-generated images from the Midjourney platform. These images are paired with text prompts for QA tasks through various extensive human annotation and human-machine-in-the-loop filtering systems. The resulting benchmark is substantially harder than multimodal benchmarks that use common images from COCO or Flickr.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: "Vision Language Understanding"
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: JourneyBench has five tasks: "MCOT (multimodal chain-of-thought), multi-image MCOT (MMCOT), fine-grained cross-modal retrieval (CR), open-ended visual question answering (VQA) with hallucination triggers, and imaginary image captioning" (2).
Task item definition: Too many highly bespoke tasks to describe in two sentences. E.g., "Strictly Complementary MCOT" consists of GSM8K questions where quantities are replaced with visual processing subtasks, e.g. "Brianna bakes as many cookies as there are Stormtroopers in this image."
Task source: Human exam questions (e.g. GRE questions), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Extreme variability in task format
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: ~13.5K
Dataset metadata available: Yes
Dataset metadata detail: visual reasoning category, task subcategory
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: subtask, including with and without distractors
Metric metascoring: recall@k
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: "existing Visual Language Understanding (VLU) benchmarks... tend to emphasize commonly occurring subjects, predicates, and objects, over unusual or abstract scenes. This enables models to excel by leveraging previously acquired common-world knowledge without necessarily understanding the actual content of the images. While this bias might inflate scores on academic benchmarks, it can lead to significant challenges when transitioning to real-world applications" (2)
Taxonomy: VQA / 
New taxonomy: General Capability / Visual Understanding
Procedural notes: 2200 total human annotation hours!