Title: M^3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought
Benchmark: M3CoT
Summary: This paper introduces the M3CoT benchmark to address current benchmarks' limitations such as lack of visual reasoning, single-step reasoning, and limited domain diversity. It focuses on the evaluation of multi-domain, multi-step, and multi-modal reasoning.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: multimodal reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: VLLM performance on more challenging multimodal CoT reasoning
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: take an image and multiple-choice question and provide an answer
Task item definition: <image> Q: What can you infer about the airplane? <4 options>
Task source: Human exam questions (e.g. GRE questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 2358
Dataset size notes: train: 7863, validation: 1108
Dataset metadata available: Yes
Dataset metadata detail: topic, category
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: topic, category
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: They apply human verification for collected data
Taxonomy: Reasoning / Multimodal
New taxonomy: General Capability / Multimodal Reasoning