Title: MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Benchmark: MMT-Bench
Summary: This paper introduces the MMT-Bench benchmark for evaluating LVLMs across different complex, expert-level multimodal tasks
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: multimodal reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: whether multimodal foundation models can perform a wide range of tasks across different
domains with human-like proficiency
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Take an image and multiple-choice question, provide an answer
Task item definition: <image> Q: How to make a cup of flat white? <4 choices>
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 31325
Dataset metadata available: Yes
Dataset metadata detail: meta task, image type
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: meta task, image type
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Authors discuss validity: No
Taxonomy: Reasoning / Multimodal
New taxonomy: General Capability / Multimodal Reasoning