Title: Multi-modal Situated Reasoning in 3D Scenes
Benchmark: Multi-modal Situated Question Answering (MSQA),  Multi-modal Situated Next-step Navigation (MSNN)
Summary: Introduces MSQA, a large-scale dataset (251K pairs) for multi-modal situated reasoning in 3D scenes, and two corresponding benchmarks: Multi-modal Situated Question Answering (MSQA) and Multi-modal Situated Next-step Navigation (MSNN). The MSQA dataset was collected scalably using 3D scene graphs and vision-language models, while the benchmarks use a novel interleaved input setting (text, image, point cloud) to improve situation awareness and resolve ambiguity present in single-modality approaches.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Situated Reasoning or Situation Awareness within 3D scenes.
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The primary tasks require a model to either answer diverse, multi-modal situated questions about a 3D scene (MSQA) or predict the immediate next navigation action towards a goal based on the current situation (MSNN), using interleaved text, image, and point cloud context.
Task item definition: A single data instance includes the 3D scene point cloud, a specific situation (location, orientation, multi-modal description), an interleaved multi-modal question (for MSQA) or goal description (for MSNN), and the ground truth answer (for MSQA) or the correct next-step navigation action (for MSNN).
Task definition detail: A key feature is the use of interleaved multi-modal inputs (text, images embedded within text, point clouds) for both defining the situation and the question/goal, aimed at resolving ambiguity found in single-modality descriptions. Additionally, the MSNN task deliberately focuses only on the immediate next navigation step to isolate situated understanding from complex, long-horizon planning.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: The task generation is a multi-stage process: Situations (location/orientation) are sampled procedurally within real-world 3D scene datasets (ScanNet, 3RScan, ARKitScenes). Situated scene graphs are created, which are then used with author-designed prompts to generate question-answer pairs (for MSQA) or navigation goals (for MSNN) via LLMs (GPT-3.5/GPT-4V). Finally, author-led refinement and balancing steps were applied to the generated data.
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Dataset size: 1413 (This is the total test set size for the MSQA benchmark, calculated by summing the test set items reported for ScanNet (832), 3RScan (315), and ARKitScenes (266) in Appendix Table 12. The specific test set size for the MSNN task (total size 34K) is not explicitly stated in the reviewed sections/tables.)
Dataset size notes: MSQA Train: 248,328; MSQA Validation: 2,147 (Justification: Calculated by summing the respective splits reported for ScanNet, 3RScan, and ARKitScenes in Appendix Table 12. Train/Val split sizes for the separate MSNN dataset are not explicitly stated.)
Dataset metadata available: Yes
Dataset metadata detail: Question type, Situation location, Situation orientation, Situation multi-modal description components, Source scene ID, Referenced object attributes, Goal description (for MSNN task).
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Response format detail: For MSQA, the expected output is open-ended text, ranging from short answers (like "yes", "no", counts) to brief descriptive sentences (e.g., explaining spatial relationships or object attributes). For MSNN, the output is a short textual command representing the immediate next navigation action (e.g., "Turn right", "Move forward").
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: For the open-ended MSQA task, the authors employ a "GPT-score," an LLM-as-a-judge approach following OpenEQA, to evaluate response correctness on a 1-5 scale, as they argue standard metrics like Exact Match are unsuitable. For the MSNN next-step prediction task, standard Accuracy is used.
Metric aggregation: Simple Mean
Metric statistics: Simple mean/average scores (MSQA Correctness Score C, MSNN Accuracy) are used to aggregate results. Different models or settings are compared directly based on these mean scores presented in tables.
Metric subscores: Yes
Metric subscores detail: Scores are provided broken down by: question category (for MSQA, e.g., Counting, Spatial, Navigation), source domain (ScanNet, 3RScan, ARKitScenes), presence/location of images in the input (situation vs. question), and specific question properties (e.g., ground truth count value for counting questions, questions involving directional answers).
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors provide evidence for validity by: 1) Justifying the need based on limitations of prior benchmarks (scope, scale, ambiguity). 2) Arguing their interleaved multi-modal task design resolves ambiguity and is more versatile. 3) Conducting a human study showing the quality (clarity, correctness) of their generated data is comparable to human-annotated data. 4) Demonstrating benchmark utility and internal consistency through model performance analysis (e.g., showing tasks are challenging, situation modeling matters, MSQA pre-training benefits MSNN).
Taxonomy: Reasoning / 
New taxonomy: General Capability / Multimodal Reasoning
Procedural notes: It utilises three existing real-world 3D scan datasets (ScanNet, 3RScan, ARKitScenes) as base environments. The data generation and evaluation processes significantly use specific LLMs (GPT-3.5, GPT-4V).
Other notes: A key contribution highlighted is the novel interleaved multi-modal input format (text, images, point clouds) designed to resolve ambiguity inherent in situated tasks. The paper also emphasises the large scale of the generated MSQA dataset (251K pairs) and includes a human study specifically assessing the quality of this LLM-generated data compared to human annotations.