Title: CODIS: Benchmarking Context-dependent Visual Comprehension for Multimodal Large Language Models
Benchmark: CODIS
Summary: This paper introduces the CODIS benchmark for evaluating MLLMs on their ability to incorporate free-form textual context to improve image understanding. The authors show that current MLLMs underperform compared to humans and struggle to effectively extract and utilize contextual information to improve their understanding of images.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: context-dependent visual comprehension
Phenomenon defined by authors: Yes
Phenomenon definition: understand the extent to which MLLMs can leverage context to enhance their visual comprehension
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Take an image and question, and provide a short-form answer
Task item definition: <image>
Question: When was this photo probably taken, the first or second half of the year?
Context: I took this photo when I was in Australia.
Ground-truth Answer: First half
Task source: Expert-crafted task examples (e.g. hand-written examples)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 706
Dataset metadata available: Yes
Dataset metadata detail: Taxonomy of Context
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: They report accuracy as the main evaluation metric, assessed by humans for the primary results. They also evaluate LLM-as-judge for accuracy measurement in ablation study
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Five annotators participated in the data collection process. To ensure the quality of our dataset, each submission by an annotator was cross-checked by the other four.
Taxonomy: VQA / 
New taxonomy: General Capability / Visual Understanding