Title: MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations
Benchmark: MMScan
Summary: The paper develops MMScan, a benchmark of 3D scenes that tests spatial and attribute understanding via visual grounding and QA tasks. 3D scene data from an existing dataset is annotated with a human-machine-in-the-loop setup, and human annotators create questions from these annotations.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multi-modal 3D perception
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The model is given a 3D scene and either (i) a question to answer about an object in the scene, or (ii) a textual description of an object it must locate with a bounding box.
Task item definition: QA task instances are nontrivial, open-ended questions, like "Where can I get a comfortable seat in this room?" Visual grounding questions ask to identify objects like "the wooden guitar leaning against the white wall."
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 504,790
Dataset size detail: Sizes combine the 3D visual grounding benchmark and the 3D QA benchmark both reported separately in the paper.
Dataset size notes: 2,016,833 train; 514;016 validation
Dataset metadata available: Yes
Dataset metadata detail: annotation type (object / region, space / attribute)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM post-processing (extracting answers, reformatting for automated scoring), 3D IoU-based Average Precision
Metric definition detail: ChatGPT categorises free responses into "Correct," "Ambiguous," and "Error." It's unclear how this maps to accuracy but accuracy is reported.
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: annotation type (object / relation, space / attribute)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: No
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: Yes
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: "For constructing a multi-modal 3D dataset, we prioritize selecting a foundational 3D scene dataset with extensive, real-scanned sensor data to minimize the sim-to-real gap" (3)
Taxonomy: Grounding / 
New taxonomy: General Capability / Grounding