Hypo3D: Exploring Hypothetical Reasoning in 3D
Authors: Ye Mao
Weixun Luo
Junpeng Jing
Anlan Qiu
Krystian Mikolajczyk
Venue: ICML 2025 poster (2025)
Keywords: 3D Computer Vision
Vision-Language Model
Hypothetical Reasoning
Primary area: deep_learning->large_language_models
TL;DR: Exploring the Hypothetical 3D Reasoning Capabilities of Foundation Models.

Abstract: The rise of vision-language foundation models marks an advancement in bridging the gap between human and machine capabilities in 3D scene reasoning. Existing 3D reasoning benchmarks assume real-time scene accessibility, which is impractical due to the high cost of frequent scene updates. To this end, we introduce *Hypothetical 3D Reasoning*, namely Hypo3D, a benchmark designed to evaluate models' ability to reason without access to real-time scene data. Models need to imagine the scene state based on a provided change description before reasoning. Hypo3D is formulated as a 3D Visual Question Answering (VQA) benchmark, comprising 7,727 context changes across 700 indoor scenes, resulting in 14,885 question-answer pairs. An anchor-based world frame is established for all scenes, ensuring consistent reference to a global frame for directional terms in context changes and QAs. Extensive experiments show that state-of-the-art foundation models struggle to reason effectively in hypothetically changed scenes. This reveals a substantial performance gap compared to humans, particularly in scenarios involving movement changes and directional reasoning. Even when the change is irrelevant to the question, models often incorrectly adjust their answers. The code and dataset are publicly available at: https://matchlab-imperial.github.io/Hypo3D.

Hypo3D: Exploring Hypothetical Reasoning in 3D
Ye Mao Weixun Luo'
Junpeng Jing'
Anlan Qiu Krystian Mikolajczyk
Abstract
full spectrum of human cognitive abilities (Goertzel, 2014; Rayhan al., 2023). Reasoning,
The rise of vision-language foundation models marks advancement in bridging the gap be- tween human and machine capabilities in 3D scene reasoning. Existing 3D reasoning bench- marks assume real-time scene accessibility, which is impractical due the high cost of frequent scene updates. To this end, we introduce Ily- pothetical 3D Reasoning, namely Hypo3D, benchmark designed to evaluate models' abil- ity to reason without access to real-time scene data. Models need to imagine the scene state based on a provided change description before reasoning. Hypo3D is formulated as a 3D Visual Question Answering (VQA) benchmark, com- prising 7,727 context changes across 700 in- door scenes, resulting in 14,885 question-answer pairs. An anchor- based world frame is established for all scenes, ensuring consistent reference to a global frame for directional terms context changes and QAs. Extensive experiments show that state-of-le the-art foundation models struggle reason effectively in hypothetically changed scenes. This reveals a substantial performance gap compared to humans, particularly in scenar- ios involving movement changes and directional reasoning. Even when the change is irrelevant to the question, models often incorrectly adjust their answers. The code and dataset are publicly avail- able at: https: / /matchhall cemb--mmprrii iill github. io Hoooo///........
cognitive ability, is major research focus. The emergence of vision-language foundation models (Hong et al., 2023; Liu et al., 2024; Wang et al., 2024; Fu et al., 2024; Mao et al., 2024) has marked a significant step toward narrowing the gap between human and machine reasoning, enabling progress from text summarization (Lewis, 2019; Zhang et al., 2020) to complex scene understanding (Huang al., 2023; Driess al., 2023).
These models are typically evaluated under the assumption that all knowledge about perception is immediately acces- sible during reasoning. For instance, in the evaluation of model performance in 3D scene reasoning, existing bench- marks (Ye et al., 2021; Azuma et al., 2022: Ma et al., 2022; Linghu al., 2024; Zhang et al., 2024) presuppose real- time availability of scene data (e.g., point clouds). Yet, this assumption does not always hold, as real-world scenes are dynamic, and maintaining up-to-date 3D scenes is chal- lenging. Unlike 2D image capture, 3D scene collection demands specialized equipment, extended scanning times, and a complex reconstruction process for accurate geomet- ric representation (Nießner al., 2013; Daneshmand et al., 2018). Thus, immediate perceptual knowledge for reason- ing is unavailable in many real-world cases. Imagination, rooted in prior knowledge, allows humans to overcome such limitations by deducing missing details and approximat- ing reality. Even without direct visual input, humans can mentally simulate changes in scene and reason about the changed scene in their minds. This ability, known as mental imagery (Pylyshyn, 2002; Moulton & Kosslyn, 2009), is crucial to human intelligence and raises critical question: Can current foundation models employ imagination to fill perceptual knowledge gaps and enhance reasoning?
1. Introduction
In response to this question, we propose a new reasoning concept: hypothetical reasoning. It evaluates models' ability to reason without immediate perceptual knowledge, requir- ing models to formulate reasonable hypotheses to bridge knowledge gaps. As an initial attempt, this study focuses specifically on hypothetical reasoning in 3D scenes, referred to as Hypo3ID. As shown in Figures and 2, the Hypo3l) task follows a 3D visual question-ansse answering format but can- not be solved using the given scene alone. Instead, models need to rely on an accessible context change description to imagine the current scene state after the change and adjust
"Imagination is more important than knowledge." "
ALBERT EINSTEIN
Artificial General Intelligence (AGI) aims to replicate the
'Department of Electrical and Electronic Engineering, Imperial College London, United Kingdom. Correspondence to: Junpeng Jing <j.jing3 amperial.c..uk..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Hypothetical Reasoning in 3D
GPT40 Text
Linma-322
Addition Change A laplop has been added the wooden desk.
LLaVA-OV-72B
LLaVA-3D
Movement Change
40
The guitar has been moved onto the top of the blue bed.
Attribute Change
Qwen-VL (non scth)
LEO
The green curtain by TV has been fully opened.
Replacement Change
The trash cans by the door hus been replaced with floor lamp.
GPT4e (oen-sem.)
GP 14o
Removal Change
(scm.)
The brown pillow has bocn removed from the hed.
LLAYA-V--18
Owen2-VL-728
(sem.)
Sonartic
Homan:: 92.1%%
Haman
Buunan 92.4%
Is the laptop closer to the couch than the backpack now? Proximity
What the largest item by couch now? w............................... comeeeeenn
Which object is encountered first along the direct path from the lamp to the bed?
Scale
Size-based Recognition
Path Reasoning
Direction
What is the position of the guilar relative to the TV after relocation?
What object is now the right of the laptop? www.owet........................
From the chair, which direction should you: take to reach the new location of the guitar?
Relative Position
Direction-baa based Recognition
Navigation
Semantic
What furniture now can still hold items like the removed coffee table?
How many lamps are in the room now? Commooonntt.....................
Are there any curtains in the room that remain completely drawn? co
Functionality
Counting
Attribute
Figure 1. Overview of the Hypo3D benchmark. ! Examples of five context change types. 2 Sample questions, including scale-based and direction- based questions requiring spatial reasoning, as well as semantic questions, all of which have open-ended answers. 3 The radar chart highlights a notable performance gap between models and humans, especially in direction-based questions.
their answers accordingly. Based on the Hypo3D task, we constructed dataset comprising 7,727 context changes and 14,885 question-answw answer pairs across 700 indoor scenes. As shown in Figure 1, these context changes span five cate- gories: (1) Movement Change, involving geometric trans- formations like translation or rotation; (2) Removal Change, taking away objects; (3) Attribute Change, modifying object properties such as color and state; (4) Addition Change, introducing new objects; and (5) Replacement Change, sub- stituting existing objects with new ones.
rectional relationships and introduces inconsistencies when multiple observers are involved. Towards this, Hypo3D establishes world frame anchored to reference objects in each scene, ensuring that all directional terms are defined relative to a consistent global reference frame.
Extensive experiments on ten foundation models reveal a substantial performance gap between models and humans on the Hypo3D task, particularly in movement changes and directional reasoning. Surprisingly, closed-source models (c.g., GPT-4o (OpenAI, 2024)) do not outperform open- source alternatives. Furthermore, the models frequently hallucinate context changes, modifying their answers even when those changes are irrelevant to the posed question. Notably, their performance consistently degrades when re- quired to imagine a scene after changes before reasoning, as compared to directly reasoning with the provided scene. These findings highlight key limitation of current mod- els: the inability to leverage imagination to infer missing perceptual knowledge and reason hypothetically.
The questions in the dataset range from simple proximity and relative position queries to intricate path reasoning and navigation tasks, broadly categorized into three types: (1) scale-based, focusing on proximity and size relationships; (2) srrectionbbssed. requiring reasoning about directional terms; and (3) semantic, highlighting scene semantics with minimal spatial reasoning. In constructing direction-breg based questions, we observe that existing 3D reasoning datasets reveal ambiguities in defining directional terms in 3D. Early datasets (Ye et al., 2021; Azuma et al., 2022) employ object- centric definitions, which lead to confusion when dealing with symmetrical or amorphous objects. such as round ta- bles or cushions. Recent datasets (Ma et al., 2022; Linghu al., 2024; Zhang al., 2024) define directions relative to the observer. This strategy struggles to describe global di-
2. Related Work
2.1. 3D Scene Update
The evolving nature of 3D environments requires continu- ous scene updates for effective understanding, particularly
2

---

Hypothetical Reasoning in 3D
Past
Point Cloud, RGB Video
Scone Orientation: The coffee table was located
Foundation Models
3D Scene
Top-View View Map
Scene Caption
at the back of the scene.
3D VLM (Point Cloud, RGB-D
Context Change: The white table has been moved to the left of the coffee table.
2D VLM (Top-View Map)
Question: W hich item is currently positioned to the left of the white table?
LLM (Scene Caption)
Guidance:
(1) A Algg the scene with the given orientation.
(2) Imagine the scene afrer change and answer
the question based on the updated scene.
Answer: Couch.
Figure2 2. Example of hypothetical reasoning in a 3D scene. Given a 3D scene and an anchor-hased frame description (Scene Orientation), models first align the scene the specified frame. Then, hased context change description and question, models hypothetically modify the aligned scene and answer questions about the changed seene. Various models (c.g., LLMs, 2D VLMs, 3D VLMs) can tackle this task using corresponding scene representations, including scene captions, top-view maps, point clouds, and egocentric RGB-D videos.
in autonomous driving (Yurtsever et al., 2020) and robotic navigation (Wong & Spetsakis, 2000). A fundamental ap- proach to scene updating involves reconstructing the entire scene from scratch. Advances in 3D reconstruction, such as multi-view stereo (Seitz et al., 2006; Jing et al., 2025), depth sensor-based (Zollhöfer et al., 201 4), and volumetric methods (Newcombe et al., 201 1), have improved recon- struction fidelity. But current methods struggle to balance accuracy, efficiency, and scalability (Nießner et al., 2013), making reconstruction primarily suitable for situations in- volving major changes. Recently, iilaance-bbsed 3D scene editing approaches, such as NeRF and 3D Gaussian Splat- ting (3DGS), have emerged as more efficient alternatives for incremental scene updates. NeRF-based methods (Liu et al., 2021; Kania et al., 2022) were limited to basic object edits and struggled with complex, cluttered scenes (Ye et al., 2025). 3DGS-based methods enable finer control over scene content, including geometry (Huang al., 2024; Waczyńska al., 2024; Ye et al., 2025), texture (Chen et al., 2024), and lighting (Gao et al., 2025). However, they struggle to disen- tangle these components and require costly re-optimization, limiting editability and efficiency (Wu et al., 2024).
clouds. As models advance, there is growing interest in developing more comprehensive benchmarks to better as- sess their 3D reasoning capabilities. Initial 3DQA dataset (Ye et al., 2021), derived from ScanNet (Dai al., 2017), introduced 6K manually annotated QA pairs for scene-level reasoning. ScanQA (Azuma et al., 2022) expanded this with 41K QA pairs using automated question generation and human refinement. Qian et al. (Qian et al., 2024) fur- ther extended 3D VQA to autonomous driving scenarios, offering domain-specific challenges. SQA3D (Ma al., 2022) introduced "situated reasoning," requiring models contextualize answers based agent's 's position and ori- entation. MRSA (Linghu et al., 2024) and SPARTUN3D (Zhang et al., 2024) further scaled SQA3D, adding multi- modal inputs such as images for richer situational context.
Unlike previous benchmarks, where answers are derived fully from the given scene, Hypo3D hypothetically applies change to the scene and derives answers from the changed scene, increasing the hallucination risk.
3. Hypo3D Benchmark
Therefore, accurate and efficient 3D scene updates remain challenge, making real-time scene acquisition not always feasible. This highlights the need for hypothetical reasoning.
This section first defines the Hypo3D task and explains how humans, assisted by LLMs, generate high-quality context changes and QAs. It then presents the dataset statistics.
2.2. 3D Visual Question Answering
3.1. Task Definition
3D Visual Question Answering (3D VQA), a key task for evaluating 3D reasoning, has advanced with the rise of Vision-Language Models (VLMs) (Hong al., 2023; Liu et al., 2024; Anthropic, 2024). By integrating vision en- coders with Large Language Models (LLMs) (Peng et al., 2023; Bai et al., 2023), VLMS enable multimodal percep- tion 3D scenes using inputs like tiew maps and point
A task instance in Hypo3D formulated as a tuple (S, q), where S denotes the scene representation; F specifies the world frame defined by anchor objects in S that standardizes all directional terms in the task; describes context change to be applied to the scene; and q denotes a question. The task is to first rotate S into Š so that the anchor object is located in the orientation described in F, followed by computing an
3

---

Hypothetical Reasoning in 3D
Scene Curation
Frame Annotation
Context Change Preparation
QA Preparation
Post- processing
5
Top-View
50h
Scene Caption
ObjertListt 0
700
900
100h
1200
423
C
Il
Point Cloud
Alignel Scene
Raw Changes
Refined Changes
RaN Questions Q.
Refined QA Pairs Q' A
RGB-D Video
10h
20h
S
"The hed is located loft side of the scar""
3
ScanNee + 3RScan
Q'
Figure 3. Dataset Generation Pipeline. The Hypo3D collection pipeline consists of five stages: Stage ! curates scenes (50 hours per person), Stage 2 defines world frames (10 h/p), Stages ! and 4 collect context changes and QA descriptions from human annotators (thousands of hours) and LLM, and Stage @ conducts grammar checks and filters data based on semantic similarity. (25 h/p).
answer to after applying to S. Figure 2 provides example of hypothetical reasoning in 3D scene.
evaluation of hypothetical reasoning performance. Scene captions are textual descriptions detailing the attributes of objects and their relationships within the scenes.
3.2. Dataset Generation Pipeline
Anchor-Based World Frame Annotation. The same seene in 3D can be captured from different viewpoints in any ori- entation. Thus, it does not have clearly defined, fixed reference frame. Here, we defined the world coordinate frame textually based the locations of anchor objects in each scene, such as: "The desk is located to the left of the scene". When selecting anchor object candidates, visual prominence prioritized over the extent to which they cover multiple sides of the scene. This is because, conventionally. defining single primary orientation (e.g.. "front") is suf- ficient infer the remaining orientations ("left", "right", "back") using the right-hand rule (Hamilton, 2008), ensuring consistent and unambiguous spatial frame
As depicted in Figure 3, the Hypo3D benchmark was con- structed through rigorously designed five-stage pipeline. The initial Stage ① involves a comprehensive 50-hour man- ual curation process to collect and standardize diverse 3D scene representations from established datasets. This is followed by Stage 2 meticulous 10-hour manual amno- tation phase where each scene is assigned anchor-based world frame to ensure spatial consistency. The core data generation occurs in Stages 3 and ④ where we imple- mented hybrid approach combining human expertise with LLMs to generate context changes and question-ansee -nnwwer pairs. These critical stages demanded substantial human oversight, with over 1,000 hours dedicated to human refine- ment to guarantee exceptional data quality and contextual coherence. Finally, the pipeline concludes with Stage O 25-hour post-proet post-processing phase that employs quality con- trol measures to eliminate redundancies while maintaining grammatical precision.
Context Change Preparation. The Hypo3D benchmark defines five distinct types object-level context changes, illustrated in Figure For scene S with an object list O {01, 02. ..., ). each change type defined as follows:
(1) Movement changes relocate objects Om S O to new positions or orientations within the scene
Scene Curation. Each 3D scan S in the Hypo3D bench- mark can be represented through multiple modalities to facilitate reasoning, including point cloud, RGB-D video, top-view map, and scene caption. The point clouds and RGB-D videos are directly curated from the original scene datasets. Two variants of top-view maps are generated: non- semantic and semantic. Non-semantic maps are generated by positioning a simulated orthographic camera above the scene to produce photorealistic top-down renderings. Se- mantic maps augment non-semantic ones by incorporating hoverable semantic labels, with each label precisely posi- tioned at the centroid of the bounding box corresponding to its object. An example semantic map can be seen in Figure 2. Unlike their non-semantic counterparts, semantic maps reduce object recognition errors in models, prioritizing the
Removal changes eliminate objects O, S from S, resulting an updated object list Ô 0 O,,
(3) Attribute changes modify properties of subset of objects Outi C O, such as color, material, state.
(4) Addition changes introduce new objects O2 into S, updating the object list to Ô = OUO..
(5) Replacement changes replace objects Orp S with new objects Ou, resulting in an updated object list Ô = (0 Orp) U.
As illustrated in Figure 3 human annotators and GPT-4o initially generate raw context changes uniformly to ensure data diversity. Human data is collected via crowdsourcing

---

Hypothetical Reasoning in 3D
on the CleurResearch platform. For each scene aligned to the world frame & with an associated object list O, anno- tators create a set of distinct descriptions C' for a specified change category, following the rules below:
model's ability to interpret object attributes with minimal spatial reasoning. Notably, questions can belong to multiple types if they require diverse reasoning.
Pust-processioe To ensure data diversity, SBERT (Reimers, 2019) 9) is used to remove semantically similar deseriptions. It encodes the refined context changes C" and questions Q', producing embeddings Ec and Eqr, respectively. Con- text changes and questions are filtered by excluding pairs with cosine similarity Sim(e;, ej) > 0.8 for ei, ej ECc, and Sim(ex; ex) 0.8 for ex, E Eqy. The remaining context changes, questions, and corresponding answers are further refined for grammatical accuracy using GPT4-Turoo, resulting in descriptions for the final dataset C, Q, and A.
(1) Each object o E O referenced in change oa E C must have a uniquely specified location if it appears multiple times in the scene S.
(2) Each context change C must be spatially feasible within the layout of S.
(3) Each q must be independent, meaning it is derived solely from the original scene S and does not rely on any version of 5 modified by ej. where i * j.
For GPT-4o, context changes are generated using the se- mantic top-view map of S and textual prompt that follows the same criteria as those provided to human annotators. The detailed prompts and human guidelines for raw change generation are provided in Appendix A.
3.3. Statistics
The Hypo3D benchmark comprises 700 unique scenes, with 500 sourced from the ScanNet (Dai et al., 2017) dataset and 200 from the 3RSean (Wald et al., 2019) dataset, randomly sampled from their respective sources. The dataset includes 7,727 context changes and 14,885 question-answer pairs. On average, context change description contains 13.62 words, a question description contains 13.69 words, and an answer contains 1.28 words. The word cloud in Figure 4 ! highlights that the most frequent words in context change descriptions are verbs representing change actions, such as "moved," " "removed," and "replaced". Figure 2 illustrates the distribution of context change types, with movement changes being the most frequent, as they often result in more pronounced scene layout rearrangements. The bar charts in Figure 4 3 show that scale-based and direction- based questions, which require more spatial reasoning, constitute a larger portion of the dataset compared to semantic questions.
Finally, each raw change e generated by humans and GPT- 4o edited by an independent group of human reviewers, producing a refined version d'
Question-Answer Preparation. Raw questions are initially generated by GPT-4o, as illustrated in Figure 3 a Seven prompt templates, each corresponding to specific question type, are designed for each type of context change. Each template includes the object list O in the scene S, the context change c, example questions, and question type description (e.g., ask for the position of the changed object relative to other objects in the scene). In total, eleven unique question types are evenly distributed across the context change types (see Appendix A.2). For each context change 21 raw questions are generated, denoted as Qc-
Human reviewers then refine Qe to produce a new question set Q- During this process, only a small portion of raw questions are retained. Specifically, 91% of the questions are filtered out, and the remaining questions undergo addi- tional editing to strictly ensure Q' satisfies the following criteria: (1) each q E Q. can only be answered by com- bining S and c, as neither is sufficient on its own; (2) the answer to each q must be potentially impacted by the change ct (3) each q has unique and unambiguous answer; and (4) answers cannot be inferred from commonsense knowledge (e.g.. bed is larger than pillow). Finally, all questions in Q' are annotated with concise human- human-provided answers A' and reclassified into general types to evaluate models based on their reasoning capabilities: scale-based, direction- based, and semantic questions, as shown in Figure Scale-based questions assess spatial reasoning related to proximity and size perception, direction-basee based questions focus on orien- tation understanding, and semantic questions evaluate the
4. Experiments
4.1. Evaluation Protocol
Exact Match (EM) and Partial Match (PM) are the met- rics used for evaluation. EM measures the percentage of model-predicted answers that exactly match the ground- truth answers. PM quantifies the percentage of overlapping words between the predicted answers and the ground truth. For the same model, PM is typically higher than EM, as accounts for partial correctness. When computing EM and PM, both the predicted and ground-truth answers are nor- malized by lowercasing, removing punctuation, and aligning semantically similar terms (e.g., "left" and "west").
4.2. Baselines
A total of ten foundation models were evaluated in a zero- shot setting on the Hypo3D benchmark, grouped into: (1) LLMs, (2) 2D VLMs, and (3) 3D VLMs. The evaluated
'Cluudeeeearch, Connect Platform, https:/ //www.
el wwureeserrh.commm
5

---

Hypothetical Reasoning in 3D
Quedionn Types
bed
aabaal
bable e
13.6%
15.3%
Numberal Quaaiee
2500
Dracctin
Senuane
Scalc
top
move
eed
cour
chair capinnene
7787 Changes
reprace
added
23.2%
29.5%
Temoved
Mevomeett Renovl
window aced desk
Atmbrre
18.4%
Replacement
Movenerr
Ramoval
Change
Anrruuz
Addininn
Aephoorarr
Figure 4. Dataset Statistics. ! Word cloud representing context change descriptions. @ Frequency distribution of context change types across 7,727 instances. 3 Distribution of question types across change categories, with question frequency consistently highest for scale-based, then direction-based and finally semantic.
LLMs, Llama3.2 (3B) (Dubey et al., 2024) and GPT-4o (OpenAI, 2024), utilize textual scene captions from the ScanRefer (Chen et al., 2020) and Scene Verse (Jia et al., 2025) datasets to represent 3D scenes. The 2D VLMs con- sist of open-source models Qwen2-VL (7B, 72B) (Wang et al., 2024) and LLaVA-OV (7B. 72B) (Li et al., 2024a), alongside closed-souree models GPT-4o (OpenAI, 2024) and Claude 3.5 Sonnet(Anthropic, 2024)², both employing semantic and non-semantic top-view maps. The 3D VLMs assessed include LEO (7B) (Huang et al., 2023), which encodes 3D scenes using egocentric 2D images with 3D point clouds, and LLaVA-3D (7B) (Zhu et al., 2024), which represents 3D scenes through multi-view images. All 3D scene point clouds have been explicitly aligned to top- view perspective with the floor on the XY-plane and vertical structures along the Z-axis.
matches, and inherent noise in 3D scenes. Human per- formance is slightly lower for addition and replacement changes, as these introduce new objects, causing confusion with existing ones.
In contrast, foundation models perform significantly worse, with the top-perforning model, GPT-4o, achieving under 50% overall in both EM and PM metrics, even with se- mantic top-view map input. Notably, it lags 45.5% behind human EM performance. Furthermore, models show sig- nificant performance bias across different question types, as highlighted by the distinct non-overlapping regions in the radar chart in Figure 5. In comparison, humans achieve over 90% accuracy across all question types, demonstrating consistently strong performance.
LLMs vs. 2D VLMs vs. 3D VLMs. Table shows that 2D VLMs outperform other foundation models across all change types. Their performance improves significantly with semantic top-view maps that provide explicit object labels compared to non-semantic maps. When using non- semantic maps, most 2D VLMs perform worse than the text-only version of GPT-4o, highlighting the impact of image recognition errors on 2D VLM performance.
In addition to scene inputs, all models receive the anchor- based world frame, hypothetical context changes applied to the scenes, the question, and task guidance as part of the textual prompt. Specific prompt templates for each model are provided in Appendix B.2. As shown in Figure 2, The task guidance generally consists of three steps: (1) Rotate the scene align with the provided world frame, (2)
Imagine the scene after the specified context change, and
For open-source VLMs like Qwen2-VL and LLaVA-OV, larger model sizes yield better results. Closed-source source models (GPT-4o and Claude 3.5 Sonnet), despite their reputation for superior performance, do not maintain this advantage on the Hypo3D task. They excel with semantic maps but struggle on non-semantic maps, where the open-source VLV 72B delivers the best performance. This pattern aligns with findings by Li al. (2024b), where closed-source models particularly struggled on unlabeled top-view maps.
(3) Answer the question based on the modified scene.
4.3. Results and Discussions
Human-Model Performance Gap. To manage the high costs of human evaluation, we sampled 50 scenes and 250 context changes with 50 questions per change type for as- sessment. To avoid contamination, human evaluators were excluded from benchmark annotation and provided only 10 QA pairs for task familiarization. As shown in Table 1, human performance exceeds 85% in EM across all change types, though it doesn't it reach 100% due to the open-ended nature of Hypo3D questions. Most errors from human eval- uators were due to typos, vague phrasing, formatting mis- We use GPT-40-08-11 16 and Claude 3.5 Sonnet-10--2..
Interestingly, 3D VLMs, despite receiving the richest geo- metric information, do not demonstrate a clear advantage over 2D VI.Ms or I.LMs. Notably, the LEO model strug- gles with the instruction following. often failing to interpret task guidance and achieving the lowest EM score (14.83%). LLa LAaVA-3D. although outperforming all other 7B 2D VLMs when using non-semantic map, lacks larger model size
6

---

Hypothetical Reasoning in 3D
Table EM and PM accuracy of foundation models and human evaluators on Hypo3D. The highest model performance for each type of context change is in bold, while the best- best-performing model within each family is underlined.
Movement
Removal
Attribute
Addition
Replacement
Overall
Model
EM
PM
EM
PM
EM
PM
EM
PM
EM
PM
EM
PM
LLM (Scene Caption)
Llama-3.2 3B
25.31
28.37
29.85
33.65
24.95
29.59
26.78
30.78
23.75
27.68
26.08
29.91
GPT-4o API (Text)
35.76
38.66
36.88
41.71
34.05
39.58
74
43.28
31.33
35.24
35.54
39.65
VLM (Non-Semantic Vop-View Map)
Qwen2-VL 7B
29.23
35.08
30.71
34.69
29.04
33.94
31.48
35.17
28.41
33.10
29.68
34.47
Qwen2-VL. 72B
33.02
37.38
33.88
37.57
33.48
37.62
35.95
40.29
30,66
34.64
33.39
37.51
LLaVA-OV 7B
30.34
34.17
29.81
33.24
31,37
36.13
33. 12
35.64
28.41
31.81
30.62
34.34
LLaVA-OV 72B
36,46
39.83
36,45
40.22
35.70
40.46
39.64
42.25
33.83
37.85
36.38
40.13
Claude Sonnet API
17.49
30.24
19.90
27.34
33.47
22.90
31.61
20.35
27.70
30.29
GPT-4o API
34.49
37.69
32.85
36.53
31.23
35.38
38.09
40.70
30.04
33.22
33.58
36.75
2D VLM (Semantic Top- View Map)
Qwen2-VL 7B
31,26
36.41
38.09
41.90
34.83
39.41
37.64
41.41
31.86
36.62
34.40
38.91
Qwen2-VL. 72B
38.42
42.56
47.36
51.05
46.76
51.10
47.63
50.87
44.43
48.7 78
44.25
48.25
LLVVA-OV 7B
33.32
36.80
34.34
37.84
39.50
38.96
41.98
33.93
38.33
34.81
38.60
VL-OV 72B
39.39
42.99
43,44
46.87
44.57
49.37
46.12
49.06
44.10
48.15 18.18
43.01
46.83
Claude 3,5 Sonnet API
30.92
42.98
40.26
48.54
42.29
52.72
43.16
$1.59
43.28
50.73
38. 86.86
48.65
GPT-4o API
40.77
43.79
47.36
50.40
47.42
51.39
50,59
53.77
44.24
47.68
45.50
48.82
VIM (RGB-D Video, Point Cloud)
LEO 7B
14.40
22.96
18.54
22.82
14.35
21.56
14.64
24.83
11.76
19.50
14.83
22.40
LLaVA-3
31.63
35.11
30.60
33.91
31,60
36.16
33.67
36.70
30.42
34,16
31.56
35.23
Human
95.00
96.00
93.00
95.00
93.00
94.83
89.00
90.67
85.00
86.00
91.00
92.50
variant fully showcase its potential in this task. See Ap- pendix B.4 for additional qualitative results illustrating the performance gap between models.
inter-object spatial relationships. Another finding is that replacement changes perform worse than both removal and addition changes, likely because they involve both object removal and addition simultaneously. making them more challenging than handling either change individually. For example, the replacement change "a cup replaced with a phone" can be interpreted as first removing the cup and then adding the phone in its place.
4.4. Analyses and Insights
Our key insights difficulties in reasoning about hypothetically changed scenes compared to static ones, particularly with movement and replacement changes. Direction-basee questions that require scene orientation also pose challenges, with performance deteriorating in the absence of defined world frame. Most models exhibit severe hallucinations, often altering their answers even when the context changes irrelevant to the questions. All 2D VLM results presented in Tables 2, 3, and 4 are derived from the semantic top-view map.
follows: Models face significant
Insight 2: Models struggle with direction- -based questions.
The radar chart in Figure 5 shows EM results across different question types, revealing that most models, except LEO, per- form better on semantic questions than spatial ones. A more distinct pattern can be observed in Figure 14. When using semantic map where object labels are provided, leading models like Qwen2-VL. 72B and GPT-4o achieve EM, indicating that semantic questions are less challeng- ing when models correctly identify objects. Within spatial questions, models struggle more with direction- tions compared to scale-based ones. Even with scene input aligned to the world frame, performance on direction- questions remains low (see Appendix B.3.2), suggesting models struggle more with orientation understanding than with size and proximity reasoning.
60%
Insight 1: Models struggle with hypothetical movement and replacement changes.
ques-
The bolded values Table indicate the highest EM and PM accuracy achieved by models across different context changes. Movement changes show the lowest performance, scoring 9.82% lower in EM and 9.98% lower in PM com- pared to the best-perfoo best-performing addition changes. This out- come highlights the models' difficulty handling changes that heavily reconfigure the scene's 's spatial layout and alter
Insight 3: inchor-based frame definition improves orienta-
7

---

Hypothetical Reasoning in 3D
LLaYA-OV 7B
Table 3. Comparison of model performance when using and not using context change, where the changes affect the answer.
LLaVA-OV 72B
(nom-sem.)
Qwen2-VL 72B
(mon-sem.)
60
(non-sem..
GPT40 (non-sem.)
QwenZ-VL.
w/o change EM
change
(ROn-Sem.)
Model
40
PM
FM
PM
L.aMA-3.2 3B
19.00
23.25
20.50 (+1.50)
24.50 (+1.25)
Claude
(sem.)
20
(non-sem.)
Qwen2-VLL VL 72B
37.00
41.50
31.50 (-5.50)
36,00 (-5.50)
GPT-4o API
38.00
40.25
33.00 (-5.00)
36.00 (-4.25)
Claude 3.5 Sonnet API
33.00
39.75
29.00 (-4.00)
35.50 (-4.25)
Qwen2-V1, 7B (sem.)
LEO
LLaVA-3D 7B
27.00
31.00
20.50 (-6.50)
24.00 (-7,00)
answering questions under both conditions. Most models, except Llama-3.2 3B, exhibit a consistent performance drop in EM and PM accuracy when reasoning in changed scenes compared to the unchanged scene. This finding addresses the core research question discussed in Sec. 1, showing that the imagination capability required for hypothetical reasoning is lacking in current foundation models.
Qwen2-VL 72B (sem.)
LLaVA-3D
LLaVA-OV 7B (sem.)
Llama-3.2
LL 72B (scm.)
GPT4 (sem.)
GPT4o Text
Direction Human: 92.00%
Scale
Semantic Human: 92.6%
Human: 91.1%
Table 4. Comparison of model performance when using and not using context change, where the changes do not affect the answer.
Figure 5. Model and human EM performance across question types. Humans consistently achieve strong performance, whereas models struggle, particularly with direction based questions.
w/o change EM
w. change
Model
PM
EM
PM
LLaMAA3.2 3B
27.50
31.42
29.00 (+1.50)
33.25 (+1.83)
Table Comparison of model performance on directional ques- tions using no frame, camera view frame, and anchor object frame. Red is lower, and green is higher, compared to w/o frame.
Qwen2-VL 72B
56.50
60.17
51.50
55.17 (-5.00)
GPT-4o API
57.00
60.00
52.50 (-4.50)
56.92 (-3.08)
Claude 3.5 Sonnet API
52.50
59.00
49.00 (-3.50)
53.25 (-5.75)
LLaVA-3D 7B
37.50
40.17
37.00 (-0.50)
40.17 (0.00)
Model
w/o frame
w. camera
w. anchor (ours)
EM
PM
EM
PM
EM
PM
Insight 5: Models hallucinate when changes are irrelevant.
Ilama 3.2. 3B
9.7
18.55
13,36
21.52
Qwen2-VL 72B
20.54
30.61
19.97
30.14
22.03
33.63
Previous results indicate that models struggle to understand how context changes influence answers. Here, we show that they also struggle to ignore irrelevant context. To test this, we constructed 250 new context change-QA triplets, separate from the benchmark. In these triplets, the context changes have no impact on the answers to the questions. For example, the context change can be "The object is moved from A to B", while the question asks, "What is the color of the object?" Models were evaluated both with and without the context change description. Ideally, they should provide consistent answers, as object movement does not affect its color. However, shown in Table 4, all 2D and 3D VL.Ms. exhibit performance degradation when context changes are introduced. Notably, while 2D VLMs achieve the highest performance on Hypo3D tasks in previous evaluations, they also exhibit more severe hallucinations.
GPT-40 API
18.45
28.36
18.20
28.04
19.37
29.48
LLaVA-3D 7B
15.57
25.57
15.31
25.78
tion understanding.
Table 2 presents model results on 3,495 pure direction-baid -baedd questions from Hypo3D under three conditions: without a world frame (w/o frame), using the camera view as the frame (w. camera), and using the anchor-based frame (w. anchor). The camera view is an image that captures the X side of the room from its center, where X E {left, right, front, back). The results indicate that not all frame definition methods can be effectively interpreted by current models. Only our anchor-based definition method consistently improves the performance of 2D and 3D VLMs, whereas using the camera view as a frame reduces the performance. See Appendix B.2 for details on inference templates across different settings.
Reasoning in hypothetically changed scenes is more challenging than in unchanged scenes.
Insight 4:
5. Conclusion
In this paper, introduce the Hypo3D task to investigate the hypothetical reasoning ability of foundation models in 3D. This task challenges models to imagine scene changes before 3D reasoning, demanding robust reasoning without real-time scene access. To standardize directional term def- initions in 3D, Hypo3D employs an anchor- based world
250 context changes and corresponding question pairs were sampled from Hypo3D to validate this insight. Each pair was annotated with two answers: one based on the un- changed scene and the other based on the hypothetically changed scene. Table presents model performance for
8

---

Hypothetical Reasoning in 3D
frame for each scene. Extensive experiments on foun- dation models, including L.Mss, 2D VLMMs, and 3D VLMs, reveal that all models struggle with the Hypo3D task, es- pecially when handling movement changes and directional reasoning questions. These models exhibit severe halluci- nations, altering answers even when the context changes do not affect the questions. These findings confirm that current models struggle simulate scene changes without direct ob- servation. We hope this study inspires further research into strengthening foundation models' hypothetical reasoning to narrow the gap with human cognitive abilities.
Anthropic. Claude 3.5 sonnet, 2024. URL https://p///// htpps://wwww /www. anthropic. wwt.roppc....ooooooooooooooooooo www.m//ww///eee wwww///awaeeeee cww.ceaaaddeeeeeee 3-rele------ooo----
Azuma, D., Miyanishi, T., Kurita, S., and Kawanabe, M. Scanqu: 3d question answering for spatial scene under- standing. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19129- 19139, 2022.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309.
2023.
Chen, D. Z., Chang, A. X., and Nicßner, M. Scanrefer: 3d object localization in rgb-d seans using natural language. In European conference on computer vision, pp. 202-221. Springer, 2020.
Acknowledgments
We would like to thank Endong Sun, Aiden Deng, Wenqiang Lai, Yicheng Zhan, and Zihan Jiang for their assistance in pre-testing the website for human crowdsourcing. We are also grateful to Eva Yin, Yan Lin, Vasandani Gavin, and all annotators from the CloudResearch platform for their contributions in providing high-quality context changes and answers for the Hypo3D dataset. We thank Chengzu Li for his valuable feedback on early drafts of the paper, and Jiangnan Ye for his help in designing prompt templates for collecting context changes using L.MMs. This research was supported by the Imperial College President's PhD Scholarship.
Chen, Y., Chen, Z., Zhang, C., Wang, F., Yang, X., Wang, Y., Cai, Z., Yang, L.,, Liu, H., and Lin, G. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Putlern Recognition, pp. 21476-21485, 2024.
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., and Nicßner, M. Scannet: Richly-annotall structions of indoor scenes. In Proceedings of the IEFE conference on computer vision and pattern recognition. pp. 5828-5839) 2017.
3d recon-
Impact Statement
Daneshmand, M., Helmi, A., Avots, E., Noroozi, F., Alisi- nanoglu, F., Arslan, H. S., Gorbova, J., Haamer, R. E., Ozcinar, C., and Anbarjafari, G. 3d scanning: A compre- hensive survey. arXiv preprint arXiv: 1xXiv:880100880 2018.
The impact of our study multifaceted. First, while hypo- thetical reasoning is motivated by challenges in handling frequent updates in 3D scenes, it is crucial for foundation models across modalities, including 3D, 2D, and text. It enables models to simulate perceptual outcomes internally without requiring physical action. This capability is espe- cially valuable in scenarios where obtaining sensory output technologically constrained or costly. Second, Hypo3D, the first 3D VQA benchmark, standardizes directional rela- tionships using a fixed world frame, enabling unified rep- resentations. Finally, we will release prompt templates, WebUI source code for data collection, 3D scene repre- sentations, and reasoning scripts for model evaluation, em- powering researchers to develop customized hypothetical reasoning datasets and ensuring result reproducibility.
Driess, D., Xia, F. Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q. Yu, T., et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv: 2:XXv:2407.22. 000007227733 2024.
Fu, R., Liu, J., Chen, X., Nie, Y., and Xiong, W. Scene-Ilm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403.11440 2024.
References
Gao, J., Gu, C., Lin, Y., Li, Z., Zhu, H., Cao, X., Zhang, I.., and Yao, Y. Relightable 3d gaussians: Realistic point cloud relighting with brdf decomposition and ray tracing. In European Conference on Computer Vision, pp. 73-89, Springer, 2025.
Achlioptas, P., Abdelreheem, A., Xia, F., Elhoseiny, M., and Guibas, L. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Computer Mitonn-ECV2200 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pp. 422- 440. Springer, 2020.
Goertzel, B. Artificial general intelligence: concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5(1):1, 2014.

---

Hypothetical Reasoning in 3D
Hamilton, W. R. Lectures on quaternions. (No Tide), 2008.
Liu, H., Li, C., Wu, Q., and Lee, Y.J. Visual instruction
ing. Advances neural information processing systems, 36, 2024.
Hong, Y., Zhen, H., Chen, P., Zheng, S., Du, Y., Chen, Z., and Gan, C. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, (6:20482-20494, 2023.
Liu, S., Zhang, X., Zhang, Z., Zhang, R., Zhu, J.-Y., and Russell, B. Editing conditional radiance fields. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp- 5773-5777775 2021.
Huang, J., Yong, S., Ma, X., Linghu, X., Li, P., Wang. Y., Li,
Q., Zhu, S.-C., Jia, B., and Huang, S. An embodied gener- alist agent in 3d world. arXiv preprint arXiv:231..12871 2023.
Lyu, R., Lin, J., Wang, T. Mao, X., Chen, Y., Xu, R., Huang, H., Zhu, C., Lin, D., and Pang. J. Mmscan: multi-modal 3d scene dataset with hierarchical grounded language annotations. Advances in Neural Information Processing Systems, 17:50898-50995 2024.
Huang, Y.-H., Sun, Y.-T., Yang, Z., Lyu, X., Cao, Y.-P., and Qi, X. Sc-gs: Sparse-controlled gaussian splatting for ed- itable dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4220 4230, 2024.
Ma, X., Yong, S., Zheng, Z., Li, Q. Liang, Y., Zhu, S.-C., and Huang. S. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:22 2022.
Jia, B., Chen, Y., Yu, H., Wang. Y., Niu, X., Liu, T., Li, Q.. and Huang, S. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. In European Conference on Computer Vision, pp. 289-310. Springer, 2025.
Mao, Y., Jing, J., and Mikolajezyk, K. Opendlign: Open- world point cloud understanding with depth-aligned im- ages. arXiv preprint arXiv:2404.1633 2024.
Moulton, S. T. and Kosslyn, S. M. Imagining predictions: mental imagery as mental emulation. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1521):1273-2222 1280, 2009.
Jing. J., Mao, Y., and Mikolajczyk, K. batch-stereo-videed Bidirectional alignment for consistent dynamic stereo matching. In European Conference Computer Vision, pp. 432. Springer, 2025.
Newcombe, R. A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., Hodges, S., and Fitzgibbon. A. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE international symposium on mixed and augmented reality, pp. 127-136. leee, 2011.
Kania, K., Yi, K. M., Kowalski, M., Trzciński, T., and Tagliasacchi, A. Conerf: Controllable neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18623- 18632, 2022.
Nießner, M., Zollhöfer, M., Izadi, S., and Stamminger, M. Real-time 3d reconstruction scale using voxel hashing. ACM Transactions on Graphics (ToG), 32(6):1-11, 2013.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
OpenAL. Hello gpt-4o, 2024. URL https: attpps://ppnnae /openai. com
Lewis, M. Bart: Denoising sequence-tor to-eeee----euuencee pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 1910. 13461, 2019.
Peng. B., Li, C., He, P., Galley, M., and Gao, J. Instruc- tion tuning with gpt-4. arXiv preprint arXiv:2304.03227, 2023.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., et al. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024a.
Pylyshyn, Z. W. Mental imagery: In search of theory. Behavioral and brain sciences, 25(2):157-182, 2002.
Qian, T., Chen, J., Zhuo L., Jiao, Y., and Jiang, Y.-G. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4542 4550, 2024.
Li, C.. Zhang, C., Zhou, H., Collier, N., Korhonen, A., and Vulié, I. Topviewrs: Vision-language models top- view spatial reasoners. arXiv preprint arXiv:2406.0255. 020.37 2024b.
Linghu, X., Huang, J., Niu, X., Ma, X., Jia, B., and Huang, S. multi-modal situated reasoning in 3d scenes. arXiv preprint ar Xiv.2449.00288,, 2024.
Rayhan, A., Rayhan, R., and Rayhan, S. Artificial general intelligence: Roadmap to achieving human-leved capabil- ities, 2023.
10

---

Hypothetical Reasoning in 3D
Reimers, N. Sentence-bert: Sentence embeddings using siamese hert-networks. arXiv preprint arXiv: 10084, 2019.
Zhu, C., Wang, T., Zhang, W., Pang, J., and Liu, X. Llava- 3d: A simple yet effective pathway empowering Imms with 3d-awareness. arXiv preprint arXiv:2409.1811 2024.
Seitz, S. M., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. A comparison and evaluation of multi-view stereo reconstruction algorithms. In 2006 IEEE com- puter society conference on computer vision and pattern recognition (CVPR'06) 06), volume 1, pp. 519-528. IEEE, 2006.
Zhu, Z., Ma, X., Chen, Y., Deng, Z., Huang, S., and Li, Q 3d-vista: Pre-trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2911 19122,9,,, 2023.
Zollhöfer, M., NieBner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., et al. Real-time non-rigid reconstruction using an rgb-d camera. ACM Transactions on Graphics (TòG), 33 (4): 2014.
Waczyńska, J., Borycki, P., Tadeja, S., Tabor, J., and Spurek, P. Games: Mesh-based adapting and modification of gaus- sian splatting. arXiv preprint arXiv:2402.01455 2024.
Wald, J., Avetisyan, A., Navab, N., Tombari, F,, and Nießner, M. Rio: 3d object instance re-localization in changing indoor environments. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 7658- 7667, 2019.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., al. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint arXiv:2409.12191 2024.
Wong, B. and Spetsakis, M. Scene reconstruction and robot navigation using dynamic fields. Autonomous Robots, 8: 71-86, 2000.
Wu, T., Yuan, Y.J., Zhang, L.-X., Yang, J., Cao, Y.P.,, Yan, L..Q.. and Gao, I.. Recent advances in 3d gaussian splatting. Computational Visual Media, 10(4):613-642, 2024.
Ye, M., Danelljan, M., Yu, F., and Ke, L. Gaussian grouping: Segment and edit anything in 3d scenes. In European Conference on Computer Vision, pp. 162-179. Springer, 2025.
Ye, S., Chen, D., Han, S., and Liao, J. 3d question answering. arXiv preprint arXiv: /tt::://22.8820 /2.08399, 2021.
Yurtsever, E., Lambert, J., Carballo, A., and Takeda, K. A survey of autonomous driving: Common practices and emerging technologies. IEEE access, 8:58443-58469, 2020.
Zhang, J., Zhao, Y., Saleh, M., and Liu, Pegasus: Pre- training with extracted gap-sentences for abstractive sum- marization. In International conference machine learn- ing, pp. 1 11328-112 11339. PMLR, 2020.
Zhang, Y., Xu, Z., Shen, Y., Kordjamshidi, P., and Huang, L. Spartun3d: Situated spatial understanding of 3d world in large language models. arXiv preprint arXiv:2410.03878, 2024.

---

Hypothetical Reasoning in 3D
Appendix
In the appendix, we will present more details about Hypo3D benchmark, more experimental results, and limitations and future work.
A
Benchmark Details
12
A.1
Human Collection via Crowdsourcing
12
A.2
LLM Collection
12
A.3
Comparison with Existing 3D Vision-Language Datasets
17
A.4
Dataset Statisties
17
B
Experiments
17
B.1
Model Hyperparameter Settings
B.2
Reasoning Prompts
18
B.3
More Quantitative Results
19
B.3.1
More Complete Main Results
19
B.3.2
More Results on Directional Questions
21
B.3.3
Effect of Chain-of-Thought
22
B.3.4
Effect of In-Context Learning
22
B.3.5
Effect of Number of Views on 2D VLMs
23
B.3.6
Effect of Caption Detail Level LLMs
23
B.4
More Qualitative Results
24
Limitations and Future Work
A. Benchmark Details
A.1. Human Collection via Crowdsourcing
Half of the raw context changes were collected using the crowdsourcing platform CloudResearch, following multi-stage procedure facilitated by a dedicated WebUI designed for collecting specific types of changes. Annotators were recruited from English-speaking countries, including the United States, Australia, Canada, Ireland, New Zealand, and the United Kingdom, to ensure linguistic consistency. Annotators were compensated at a rate of $1.00 for every five change descriptions derived from 3D scenes. Figure 6 outlines the annotation protocol, specifying requirements for accurate descriptions of moved object locations and ensuring that changed objects repeated across submissions.
A.2. LLM Collection
GPT-4o integral to both the context change and question collection processes in Hypo3D. For context change collection, each type of change is generated using dedicated prompt templates. Figure 7 highlights the templates specifically designed for capturing addition changes.
For the question collection, 11 unique question types are designed, encompassing categories such as proximity, size-based recognition, and path reasoning, as outlined in Table 5. Figure and 9 illustrate the prompt templates for generating questions related to proximity and relative positions, respectively. Each context change associated with 7 specific question types, and the distribution question types across different changes is detailed Table While some context changes share the same question types, they are provided with different prompt templates and example questions for diverse generations. Once collected, the questions categorized into broader, coarse coarse-grained categories based on the primary capability required answer them, such scale-based firection-based and semantic questions.
12

---

Hypothetical Reasoning in 3D
Data Collection Guidelines Pleess Read
Welcome!
Explore the given 3D scene visualization and describe five different ways to move objects within it.
Consider the example scene below, possible movements can include:
The brown pillow, originally on the bed, has been moved to the gray couch.
The desk, which was next to the white cabinet, is now positioned between the refrigerator and the two trash cans.
Instructions:
Movements must be spatially feasible within the scene's layout.
Each movement description should clearly specify the object(s) being moved, its original location, and its new location a unique way. Ambiguous or wrong descriptions will be rejected.
Good Description: The red apple on the table has been moved to the sink near to the refrigerator. Bad Description: The apple has been moved the sink. (Which apple? Which sink?)
Each description should move different objects in the scene.
Each description should be more than 10 words.
All movements must occur within the same scene and be independent of one another.
Once you have finished the task, click the Submit button to receive your Completion Code.
Please your imagination and creativity to come up with unique and interesting movements!
Figure 6, Guidelines for movement change collection in crowdsourcing.
13

---

Hypothetical Reasoning in 3D
Table Comparison of model performance in direction-hand based questions non-aligned versus aligned top-view maps. The results show no significant improvement, and in some cases, decline in performance with aligned maps
Proximity
Which direct distance is shorter: from the nightstamd to the window or to the box?
Direction-based Recognition
What item directly positioned below the shelf now?
Size-based Recognition
What is the largest item remaining on bed now?
Functionality
What item in the room now provides storage functionality similar to the removed shelf?
Counting
What is the current count of chairs next the table?
Navigation
Which direction should you move from plant reach new chair?
Placement Height
Is the box positioned higher or lower than the keyboards?
Relative Position
What is the relative position of the armchair the footstool now?
Attribute
How many different colors of the table are in the room now?
Path Reasoning
Is the direct path from the trash can't 's new location to the fire alarm obstructed by any objects?
Situational Reasoning
Are you closer to the repositioned trash can or the copier when 're next to the door?
Table Distribution question types across different context changes.
Movement
Removal
Attribute
Addition
Replacement
Proximity
Situational Reasoning
Proximity
Proximity
Proximity
Direction-based Recognition
Direction-basee Recognition
Direction- Recognition
Direction-based Recognition
Direction- Recognition
Size-based Recognition
Size-based Recognition
Size-based Recognition
Size-based Revognition
Size-based Recognition
Relative Position
Size Fitness
Relative Position
Relative Position
Relative Position
Path Reasoning
Functionality
Functionality
Functionality
Functionality
Placement Height
Navigation
Navigation
Placement Height
Attribute
Counting
Counting
Counting
Counting
Counting
Addition Change Generation
Generate four unique object additions within the given 3D scene. Each change should clearly specify which object(s) are being added and provide detailed descriptions. Two changes should add more existing items, while two should introduce new items not present in the scene.
Requirements:
1. The added object's location and appearance should be described clearly.
2. Changes should be deterministic, detailed, realistic, and contextually appropriate for the scene.
3. The number of objects added in each change should be varied.
List the changes in the order to 4, with each change starting new line.
Figure 7. Prompt template for addition change generation.
14

---

Hypothetical Reasoning in 3D
Proximity Question Generation
Input and Output
The input consists of 3D scene description, a context change description, and question related to the change.
Your Role
Your task is to assist in generating questions for 3D Visual Question Answering dataset, The dataset aims to evaluate the model's reasoning ability based on changes in a 3D scene.
Guidelines for Generating Questions:
Focus each question the proximity (close/far) between the added object and the other objects in the scene.
The 'other objects' should not be the object mentioned in the change description.
Use various starters such as "what," "where," " "which" etc., to ensure diverse question structures. Make sure each question has unique sentence structure and is phrased totally differently from the others.
Avoid asking questions that can be inferred from the change. For example, if the change is, "A mug is added next to the laptop," " avoid questions like, "What object is placed beside the laptop?". Each question should have a clear, definitive answer that eliminates ambiguity.
Example:
Object List:
This scene contains kitchen counter, 1 shower, 1 desk, I sink, 1 scale, tv, 1 pillow, I clock, backpack, couch, refrigerator, 1 coffee table, I toilet, bed, trash can.
Context Change: A new coffee machine has been added on the desk in the scene.
Example questions:
(1) Of the couch, refrigerator, and clock, which one is situated nearest to the added coffee machine? Between the couch and the clock, which item is farther away from the new coffee machine?
Object List for Question Generation Context Changes for Question Generation
Now, generate three unique and diverse questions for each scene change above. The response should be in JSON format, with each change as a key and its questions as an array of values.
Figure 8. Prompt template for generating raw proximity questions based on context changes and the object list of the 3D scene.
15

---

Hypothetical Reasoning in 3D
Relative Position Question Generation
Input and Output
The input consists of 3D scene description, a context change description, and question related to the change.
Your Role
Your task is to assist in generating questions for 3D Visual Question Answering dataset. The dataset aims to evaluate the model's reasoning ability based on changes in a 3D scene.
Guidelines for Generating Questions:
Focus on asking the relative position of other objects in the scene respect the changed object.
Potential answers could be (e.g., in front of, back, left, right).
The 'other objects' should not be the object mentioned in the change description.
Use various starters such as "what," "where," "which" etc., to ensure diverse question structures.
Make sure each question has a unique sentence structure and is phrased totally differently from the others.
Avoid asking questions that can be inferred from the change. For example, if the change is, "The brown pillow is moved the front of the couch," avoid questions like, "What is behind the pillow?".
Each question should have a clear, definitive answer that eliminates ambiguity.
Example:
Object List: This scene contains kitchen counter, 1 shower, I desk, 1 sink, I scale, tv, I pillow, clock, backpack, couch, refrigerator, coffee table, 1 toilet, bed, trash can.
Context Change: The brown pillow, originally the bed, has been moved to the gray couch.
Example questions:
(1) What is the current position of the coffee table compared to the pillow?
(2) How is the sink positioned relative to the relocated pillow now?
Object List for Question Generation Context Changes for Question Generation
Now, generate three unique and diverse questions for each scene change above. The response should be in JSON format, with each change key and its questions array of values.
Figure 9. Prompt template for generating raw relative position questions based context changes and the object list of the 3D scene.
16

---

Hypothetical Reasoning in 3D
Table comparison between Hypo3D and existing 3D vision-language datasets. "Hypothetical" indicates whether the dataset includes context changes. "Question Type" denotes whether questions are categorized into predefined types. "VG" and "QA" refer to visual grounding and question guestion-answeringg respectively. "World Frame" denotes the scene's orientation in space is explicidy defined.
Dataset
Task
Question Type?
Hypothetical?
World Frame?
#Scams
#Language
Text Collection
ScanRefer (Chen et ul., 2020)
N/A
0.7k
11k
Human
Sr3D (Achlioptas al., 2020)
VG
NIA
X
0.7k
115k
Template
ScanQA (Azuma et al., 2022)
QA
X
0.8k
41k
Template
SQA3D (Ma ul., 2022)
QA
0.65k
33.4k
Hummn
ScanSeribe (Zhu et al., 2023)
Captioning
N/A
x
X
1.2k
278k
LLM
MMScan yu et al., 2024)
Captioning QA
6.9M
L.M Temp- + Human
Hypo3D (Ours)
QA
0.7k
15k
LLM Human
A.3. Comparison with Existing 3D Vision-Language Datasets
Comparisons with relevant 3D scene understanding tasks and benchmarks summarized in Table 7.
COlOE
the
arotther
the
there
itmm
What
Is
Which
object
1f
tho
How
many
thter
hoighh
Clossal
Figure 10. Question distribution in Hypo3D.
A.4. Dataset Statistics
Figure 10 demonstrates that the questions in our benchmark begin with various start words, such as "what", "which", "is", "how", and "when". None of these words dominate the dataset, highlighting the balanced and diverse nature of our dataset.
B. Experiments
B.I. Model Hyperparameter Settings
Our experiments primarily used the default inference hyperparameters for zero-shot models, as detailed in Table For Claude 3.5 Sonnet, the maximum new token parameter was set to 40, reduced from its default value due to the model's
17

---

Hypothetical Reasoning in 3D
tendency generate lengthy responses, even when instructed to be concise.
Table 8, Inference hyperparameter settings for the baseline models.
Model
Max New Tokens
Temperature
Llama3.2 3B
32
0.6
Qwen2-VL. 7B & 72B
128
0.01
LLaVA-OV 7B & 72B
128
0.7
GPT-4o API
1024
1.0
Claude 3.5 Sonnet API
40
1.0
LLaVA-3D 7B
512
0.2
LEO 7B
256
1.0
Hypothetical Reasoning Prompt
3D VLM: Given a 3D scene, mentally rotate the scene to align with the specified orientation.
2D VLM: Given top-view of 3D scene, mentally rotate the image to align with the orientation.
LLM: Given a 3D scene description, mentally match it with the specified orientation.
Scene Orientation: 0
Now, given a context change, imagine how the scene would appear after the change has been applied. Then, answer question based on the modified scene.
Context Change: 0
Question: Q
The answer should be a single word or a short phrase.
The answer is:
Figure 11. Prompt template for the main hypothetical reasoning experiments, with differences between prompts for 3D VLM, 2D VLM, and LLM underlined.
B.2. Reasoning Prompts
The prompt template used for the main results in Table 1 is shown in Figure 11. The only difference between prompts for LLM, 2D VLM, and 3D VLM how the 3D scene is introduced, tailored their specific scene representation formats, while all other parts remain consistent ensure fair comparison.
For the experiment in Table 2, which assesses the effectiveness of the world frame, the prompt templates in Figure 12 were used to evaluate model performance without a frame description.
Experiments in Tables and evaluate model performance in scenarios where the context change description is not given, with the exact prompt template provided Figure 13.
18

---

Hypothetical Reasoning in 3D
Reason without World Frame
Given a top-view of a 3D scene and a context change, imagine how the scene would appear after the change has been applied. Then, answer a question based on the modified scene.
Context Change: 0
Question: {
The answer should be single word short phrase.
The answer is:
Figure 12. Prompt template evaluating model performance without using an anchor-hased world frame.
Reason without Context Change
Given a top-view of a 3D scene, mentally rotate the image to align with the specified orientation.
Scene Orientation: ?
Then, answer a question based on the aligned scene.
Question: 0
The answer should be single word short phrase.
The answer is:
Figure 13. Prompt template for evaluating models in static without context changes.
B.3. More Quantitative Results
B. MORE COMPLETE MAIN RESULTS.
We present the complete main results for each context change type and question type in Table Additionally, we evaluate open-ended responses the Hypo3D dataset using GPT-based scoring approach, following the MSQA (Linghu et al., 2024) framework. Each GPT score C is computed as:
8 1 X 100
C=
where N is the number of questions, and E [1,5] is the discrete score assigned by GPT-4o-mini based on the question, ground truth, and model response (higher is better). The results are shown in Table 10.
19

---

Hypothetical Reasoning in 3D
Table 9. Complete EM and PM results of foundation models and human evaluators on Hypo3D.
Input
Modid
Metric
Movemeett Din:
Renoval Dis:
Attribute Dir:
Addition Dinc
Replacement
Overall
Scale:
Sem.
Scali:
Scm
Scal:
Sen.
Seale.
Sem
Scal:
Dir.
Sun
Liau 3.2
BM
29.62
15.89
30.94
35.29
14.14
30.54
25.50
13,49
32.25
30.16
12.09
32.12
26.30
10.56
35
26,08
PM
20.36
31.18
32.23
31,00
20.97
34.15
32.54
23.90
17.34
34.54
29,91
Captions
CI/T-40 API (Tzx))
BM
45.36
21.88
57.37
64.01
20.49
$7.00
35.03
21.52
65.85
45.31
19.05
50.00
36.41
15.23
42.54
35,54
PM
47.20
26,01
37.37
40.32
39.63
30.74
48.65
50.54
20.35
44.05
QwenZ V1.76
EM
34.82
19.96
35.49
53.75
17.27
35.76
29.14
15:25
40.53
53.50
14.49
37.99
30.29
13.00
41.54
29.85
PM
39.75
25.75
35.49
37.00
24.16
37.55
39.39
33,90
45.15
34,47
Qwas2/11 72B
BM
19.25
21:33
55.35
37.34
18:41
40.00
35.47
19.90
47.18
39.41
18:02
46.13
33
15.05
42.29
33.99
42.18
38.13
41.09
27.14
47.79
43.11
25.97
44.41
34.19
22.29
43.66
37.51
LLVA-OV 7B
BM
16.33
20.34
30.95
32.61
16.13
38.25
35.24
20.29
19.53
58.75
13:31
40.50
31.70
14.06
19.80
30,02
Somantic
PM
18.90
34.99
56.05
33.51
13.29
40.53
37.50
28.07
40.05
40.02
17.84
41.48
$3.90
19.69
42.04
$4.34
Top View
LLAAA-VV 72B
BM
43.53
24.30
41.01
00,90
20.08
41.77
38.01
25.70
63.69
43.84
19.07
48.32
36.24
17.26
47.76
36.38
PM
.4421
29:58
41.01
42.07
79.15
42.88
42.43
52.18
44.34
45.10
25.01
48.32
57.78
24.45
49.13
40.13
Chusle 3.5 Summes
BM
21.28
8:00
20.02
18.64
11.48
29.27
26.17
12.90
20.51
25.76
7.54
53.24
24.33
7.37
26.87
20,42
PM
15
21.70
20.00
23.86
12:30
35.23
33.41
27.50
16.99
32.77
18.60
58.59
23.60
17.90
12:21
90.79
EM
21.16
34.53
37.66
37.82
18.01
45.90
35.83
34.29
40.30
33.58
API-4o-API
PM
45.08
25.34
54.55
58.71
04.08
35.62
35.77
25,02
40.97
47.52
21.97
99.39
35.60
18.66
42.04
96.75
Qwar2-V1.78
BM
37.25
23.00
57.37
35.78
20.11
53.01
35.36
19.77
08.07
40.89
19:31
52.23
33.47
17.05
47.51
34,40
PM
40.8
25.04
50.17
41.58
18.35
53.80
16.74
23.76
50.11
42.71
25.80
$4.47
35.92
24.31
$2.96
58.91
Qwaa2/11 72B
BM
45.81
25.00 00.0
40.52
50.30
27.13
03.13
47.29
29.74
08:27
51.81
24.85
64.80
40.70
20.96
63:43
44.25
PM
48.01
90.63
46.52
51.85
15.56
63.61
50.05
57.45
68.94
5338
30.51
64.80
43.75
54.79
64.6%
68.25
LLaVA OV 7B
EM
22.70
37.97
20.49
41.61
22.05
19.01
49.72
37.71
18.72
44.53
34.81
Non-Smmantic Top-Viow
PM
41.04
21.44
50.51
19.09
18.27
43.04
41.20
90.11
45.05
44.85
25.67
$1.96
$5.57
26.32
47.51
%..00
EM
23.11
47.00
43.69
51.42
20.97
51.22
25.65
27.06
56.22
43.01
LLaVA DV 72B
PM
48.07
31.95
40.00
49.34
15.09
$3.06
$1.65
58.85
$6.89
52.90
31.90
$7.68
90.73
54.11
58.31
46.83
EM
19:87
42.02
30.02
25.10
57.81
47.68
21.05
64.80
55.72
Clande 3.5 Sounet
PM
46.39
32,67
50.60
47.64
34.49
65.11
51.01
39.29
71.23
52.64
34.25
69.27
54.22
35.97
61.82
48.65
EM
25.53
51.32
52.18
26.47
61.71
20.15
58.52
27.44
$0.20
25,41
60.70
45.50
GPT 4o API
PM
50.62
30.21
$1.32
53.05
33.35
62.18
52.63
35.94
65.70
60.15
33.33
62.15
$1.14
31.59
61.82
48,82
Muli-Nies View
LLMV 3D 7B
EM
38.03
21.63
35,49
33.60
17.36
32.91
17.77
50.34
17.35
37.99
33.55
15.52
43.28
31.56
RGB-D D
PM
39.38
26.86
35.49
36.74
25.04
33.57
38.72
26.22
38.18
40.44
24.03
38.13
34,88
22.34
44.15
35.25
Poiu
EM
21.13
7.79
27.62
10.44
3.46
20.66
2.00
LBO 7B
RCB
PM
10.55
14.54
10.07
$1.60
5.30
17.01
24.91
7.60
36.99
30.12
10.65
13.8
21.
6.69
15.67
22.40
Table 10. GPT Score of 10 foundation models and human evaluators on Hypo3D (Overall only). The best-performing model within each family is underlined.
Model
GPT
1LM (Scene Caption)
Llama3.2 3B
28.13
GPT-4o API (Text)
37,89
20 VLM Semanammatic Top- View Maps
Qwen2-VL 7B
32.01
Qwen2-VL 72B
35,58
LLaVA-OV 7B
32.29
LLaVA-OV 72B
38.20
Claude 3.5 Sonnet API
25,27
GPT-lo API
35.49
VLM (Semantic Top- View Map)
Qwen2-VV. 7B
36.74
QwenZ-VL 72B
45.90
LLaVA-OV 7B
36.91
LLaVA-OV 72B
45.11
Claude 3.5 Sonnet API
42.76
GPT-4o API
46.55
(Mutri
Cloud)
LEO 7B
17.47
V.L 7B
33,80
20

---

Hypothetical Reasoning in 3D
LLaVA-OV 7B
(non-semn.)
Qwen2-VL
(hon-sem.)
(non-sem.)
GPT4o (nou-sem.)
Owen2-VL (non-sem.))
Claude non-Sem.)
(sem.)
30
Qwen2-VL.7B (sem.)
Qwen2-VL 72B (sem.)
LLaVA-3D
LLaVA-OV 7B (sem.)
Llama-3.2
(sem.)
GPT4e Text
(sem.)
Direction Human: 95.3%
Scale
Semantic
Human:
Figure 14. Model and human SBERT scores across question types. Models struggle the most with direction-basse questions, followed by scale-based and semantic questions.
B.3.2. MORE RESULTS ON DIRECTIONAL QUESTIONS
The radar chart in Figure 14 shows model performance across different question types using the SBERT metric, which measures cosine similarity between the text embeddings of predicted and ground-truth answers. These embeddings are generated using the SBERT model (Reimers, 2019). The results clearly show that most models struggle the most with directional questions, perform better on scale-based questions, and achieve their best performance on semantic questions.
Table 11 further highlights that, even when the top-view map physically aligned with the world frame (i.e., no mental alignment required), model performance direction- questions shows significant improvement, particularly for non-semantic maps. This indicates that current foundation models struggle with direction- hypothetical reasoning, regardless of frame alignment.
Table 1. Comparison of model performance direction- questions non-aligned aligned top-view maps. The results show no significant improvement, and in some cases, decline in performance with aligned maps.
Non-aligned
Aligned
Model
EM
PM
EM
PM
2D VLM -Non-Semanic Top-View View Map)
Qwen2-VL 7B
16.70
23.93
16.60
23.40
Qwen2-VL 72B
19.21
25.95
19.99
27.08
LLaVA-OV 7B
17.91
24.08
18.86
25.29
LLaVA-OV 72B
21.97
28.81
21.69
28.25
GPT-4o API
17.95
23.64
18.64
24.69
2D VLM (Semantic
Top-View Map)
Qwen2-VL 7B
20.22
27.18
21.94
29.17
Qwen2-VL VL 72B
26.77
33.65
33.95
43.28
LLaVA-OV 7B
21.28
27.82
23.77
31.64
LLaVA-OV 72B
27.60
34.74
31.93
40.77
GPT-4o API
26.57
32.67
32,38
42.53
21

---

Hypothetical Reasoning in 3D
Table 12, Performance comparison of models with and without Chain-of-Thought (CoT) prompting.
Model
w/o CoT
w/ Co'T
LIama3.2 3B
23.91
26.08
LLaVA-OV 72B
42.78
43.01
Qwen2-VL 72B
44.90
44.25
LLaVA-3D
29.30
31.56
B.3.3. EFFECT OF CHAIN-OF-THOUGHT
Figure 11 illustrates our use of the Chain-of-Thoughh (CoT) strategy, which explicitly decomposes the task into two steps: (1) imagining how the context change affects the scene, and (2) answering the question based on the altered scene.
To further examine the impact of CoT, we evaluated models using a simplified prompt structure:
Scene orientation: 0
Context Change: f}
Question:
0
Answer:
The results in Table 12 show that removing CoT prompting leads to decreased performance in most models, except for Qwen2-VL. 72B. This suggests that step-by-sted reasoning supports hypothetical understanding to some extent. Nonetheless, model performance still falls short of human-level reasoning.
Table 13. Performance comparison of models with and without in-context learning (ICL).
Model
w/o ICL
w/ ICL
Llama3.2 3B
29.30
23.88
LLaVA-OV 72B
40.26
33.53
Qwen2-VL 72B
41.94
36.52
Table 14. Comparison of EM and PM scores for different models across Top and Multi-view settings.
Model
View
EM
PM
Top
34.81
38.60
LLaVA-OV 7B
Multi
34.24
38.19
Top
43.01
46.83
LLaVA-OV 72B
Multi
42.52
47.06
Top
34.40
38.91
Qwen2-VL 7B
Multi
35.99
41.19
Top
44.25
48.25
Qwen2-VL 72B
Multi
43.04
47.50
B.3.4. EFFECT OF IN-CONTEXT LEARNING
We also investigated whether in-context learning (ICL) can enhance model performance in hypothetical reasoning tasks. Specifically, we applied three-shot ICI. to both 2D VLMs and LLMs. The results Table 13 show that ICL. generally leads
22

---

Hypothetical Reasoning in 3D
Table 15. Effect of caption quantity EM and PM scores Llama3.2-3B.
#Captions
EM
PM
30
23.95
28.62
50
23.88
28.34
100
24.34
28.91
200
22.91
28.01
to decrease EM performance. This can be due to the limited number of examples failing to capture the diversity of context changes and question types in our dataset. Moreover, we observed that models often copied answers directly from the in-context examples rather than learning from them.
B.3.5. EFFECT OF NUMBER OF VIEWS ON 2D VLMs
We evaluated 2D VLMs (semantic maps) using multi-view inputs (top, front, back, left, and right) compared to using top-view only. The results on 50 randomly sampled scenes in Table 14 show that performance remains comparable to using only the top view. This suggests that while multi-view inputs offer richer visual information, integrating visual features from different views presents another challenge for the models.
B.3.6. EFFECT OF CAPTION DETAIL LEVEL ON LLMS
To assess how caption detail affects LLM performance on hypothetical 3D reasoning, tested Llama3.2 3B with varying numbers of sampled captions. As shown in Table 15, more detailed inputs do not consistently improve performance, possibly due to the increased challenge of long-text reasoning. Following the SQA3D protocol, we use 30 randomly sampled object captions for the final scene description.
Movement
Removal
Attribute
Addition
Replacement
Context Change
Context Change
Context Change
Context Change
Context Change
plate, was stand, has bees removed from the scone.
is partially drawn with water pooling on the floor outaide the tub.
neae the foot the bed for wamthh and comfort.
The TV coploced by decorative vosc.
above the monitor to above the tv.
Question and Answer
Question and Answer
Question and Answer
Question and Answer
Question and Answer
Q: Which shocter: from the clock to the TV or to the plan?
Q: What is on the TV stand now after the plate's removal?
Q: Which dircetion should you go from the minor the curtai??
Q: What is the tallest object near the added rig the hed foot? Shelff
Q: What object is placed closest in front ofe deconative vase?
TV and lamp.
left.
A.TV.
Model Output
Model Output
Model Outpat
Model Output
Model Output
LLaa--3.2 2 3B: Shonter
1Lamm-3.2 3R- TV:
[Lama-3 3B: Right
L.aam-33.2 3B: Commode
1aamn33.2 38: Couch.
LV 720: TV.
LLVVA-VV 73B TV.
LLAVA-OV 73R: Left
LLaVA-OV 72R: Shalf.
ILVVA-VV 728 TV.
GPT46-08- TVE.TV
GPT46-08-16
GPT4o-08- API:
APE Nightstnd.
GPT46-08-16 Cabinet.
LLVA--3D TV.
H.VA--D 7B:
LLAVA-1D Right
L.aVVA-3 7B: Cabinot.
ILVVA-3 78: Table
Figure 15. Qualitative Results. The changed object described in the context change highlighted with a red bounding box. Model outputs are shown in green for correct and red for incorrect predictions. Results indicate that while models struggle with most examples, 2D VLMs more likely to provide partially correct answers.
23

---

Hypothetical Reasoning in 3D
B.4. More Qualitative Results
The qualitative results of model performance across various context change types are shown in Figure 15, Although models answer most questions incorrectly, 2D VLMs (LLaVA-OV 72B, GPT-4o) are more likely to provide partially correct answers, suggesting relatively better capability for hypothetical reasoning.
C. Limitations and Future Work
One limitation is relying solely on text to describe context changes, which may lack precision for complex scenarios. Future work will incorporate multimodal approaches, such as images or egocentric videos, for more accurate and complementary representation. Additionally, dataset focuses on hypothetical reasoning in indoor scenes, with plans extend outdoor environments. Lastly, Hypo3D addresses object-level changes (e.g., modifying specific objects); future work will explore scene-level changes involving significant layout rearrangements.
24


Reviews:
[{"review_id": "LtmmLhMNqa", "reviewer": "Reviewer_kxrB", "summary": "This paper introduces a novel 3D-VQA benchmark called Hypo3D. Given a 3D scene representation (e.g., point clouds, BEV images) and a description of how the scene has changed, the model must infer the updated scene and answer a question based on it. The authors benchmark a range of open-source and closed-source 2D and 3D VLMs, identifying their limitations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback. We’re glad to hear that you found our work to be novel and interesting.\n\n**Q1: Scene Orientation** \n\nFor 3D VLMs using point clouds (e.g., LEO), inputs have been explicitly aligned to a top-view perspective with the floor on the XY-plane and vertical structures along the Z-axis. We acknowledge that the 3D VLMs taking ego-view images (e.g., LLaVA-3D) are expected to align  3D scenes internally for orientation understanding. Yet, they are provided with comprehensive information (i.e, multi-view RGB, depth, camera poses, and axis-alignment matrices) for implicit scene alignment. Again, while orientation may be more challenging to interpret for 3D VLMs compared to 2D VLMs, all models are provided with sufficient information to support effective orientation comprehension.\n\n**Q2: GPT-4 as Judger** \n\nBeyond EM and PM, we used SBERT (Fig. 14) for text similarity scoring to reduce evaluation bias in phrasing. Here, we explored GPT-based evaluation for open-ended responses following MSQA [1]. Each GPT score $\\( C \\)$ is formulated as:\n\n\n$$C = \\frac{1}{N} \\sum_{i=1}^{N} \\frac{s_i - 1}{4} \\times 100\\%$$\n\nwhere $N$ is the number of questions,  $s_i \\in [1, 5] $ (higher is better) is the discrete score assigned by GPT-4o-mini taking the question, ground truth, and model response as input. The scores for all models are shown below:\n\n| Model                    | GPT Score     |\n|-|-|\n| llama-3.2                | 28.13         |\n| GPT-4o (text-only)       | 37.89         |\n| Qwen2-VL-7B (non-sem.)   | 32.01         |\n| Qwen2-VL-72B (non-sem.)  | 35.58         |\n| llava-ov-7B (non-sem.)   | 32.29         |\n| llava-ov-72B (non-sem.)  | 38.20         |\n| Claude 3.5 Sonnet (non-sem.)        | 25.27         |\n| GPT-4o (non-sem.)        | 35.49         |\n| Qwen2-VL-7B (sem.)       | 36.74         |\n| Qwen2-VL-72B (sem.)      | 45.90         |\n| llava-ov-7B (sem.)       | 36.91         |\n| llava-ov-72B (sem.)      | 45.11         |\n| Claude  3.5 Sonnet (sem.)            | 42.76         |\n| GPT-4o (sem.)            | 46.55         |\n| LEO                      | 17.47         |\n| LLaVA-3D                 | 33.80         |\n\nRankings of GPT scores closely align with PM rankings in Table 1, validating the reliability of PM.\n\n**Q3: Human Performance**\n\nHuman performance rarely reaches 100% in open-ended VQA datasets. For example, the best EM@1 on ScanQA is 51.6%, and SQA3D reports 85–95% accuracy. Our dataset achieves even higher human scores, suggesting fewer ambiguities. Most errors are due to typos, vague phrasing, formatting mismatches, and inherent noise in 3D scenes. E.g.,\n\n| Question | Human Answer | GT |\n|-|-|-|\n| Where in the room would you stand to be the furthest from the liquid spill?                                 | Door                 | Next to the door       |\n| How does the placement position of the added Nescafé espresso machine compare with the paper towel?         | Above the paper towel| Higher                 |\n| Which is closer to the newly added rubber gloves, the trash can or the paper towel roll?                    | Paper tower roll     | Paper towel roll       |\n\n**Q4: InternVL-2.5** \n\nIn response, we have evaluated InternVL-2.5 [2] (8B and 34B) on the Hypo3D benchmark using both semantic and non-semantic top-view maps. EM results show that InternVL-2.5 (8B version) consistently underperforms LLaVa-OV 7B and Qwen2-VL 7B across all settings on Hypo3D task.\n\n| Model                      | Movement | Removal | Attribute | Replacement | Addition | Overall |\n|---------------------------|----------|---------|-----------|-------------|----------|---------|\n| InternVL-2.5 8B (non-sem.)     | 20.07    | 19.97   | 19.22     | 16.12       | 20.57    | 19.38   |\n| InternVL-2.5 38B (non-sem.)    | 26.56    | 27.67   | 27.02     | 24.47       | 29.47    | 27.01   |\n| InternVL-2.5 8B (sem.)         | 21.96    | 24.82   | 27.41     | 16.92       | 27.74    | 23.83   |\n| InternVL-2.5 38B (sem.)        | 30.66    | 36.77   | 37.42     | 35.36       | 38.37    | 35.18   |\n\n**Q5: Instruction-tuning** \n\nWe agree that exploring instruction-tuned 2D and 3D VLMs on our task would be valuable. Given the limited availability of computing resources, we intend to carry out a systematic study as part of our future work. We currently focus on zero-shot evaluation to ensure a fair comparison of all models. Notably, LLaVa-3D (7B), the SoTA 3D VLM, performs comparably to similar-sized 2D VLMs, suggesting that 3D VLMs' relatively weaker performance may stem from smaller model sizes.\n\n[1] msqa: multi-modal situated reasoning in 3d scenes, NeurIPS, 2024.\n\n[2] Expanding performance boundaries of open-source multimodal models with model, arxiv, 2024.", "claims_and_evidence": "Most of the claims are supported.", "methods_and_evaluation_criteria": "Strengths\n1. This paper introduces a novel and compelling benchmark, tackling a crucial problem in 3D scene understanding—namely, how to handle changing scenes.\n2. The authors provide a clear and detailed data collection process that involves human annotators, helping to ensure quality.\n\nWeaknesses\n1. According to Figure 2, the scene orientation descriptions rely on a top-view perspective (using terms such as “at the back of the scene” or “left side of the scene”). However, when assessing 3D-based VLMs, the inputs are point clouds and ego-view images rather than top-view maps, which may lead to ambiguity in interpreting orientation.\n2. The benchmark primarily uses Exact Match (EM) and Partial Match (PM) for evaluation. While somewhat reasonable for single-word or short-phrase answers, these metrics may not reliably assess correctness if models produce longer responses. A potential improvement would be to employ GPT-4 to reformat the model outputs (e.g., summarizing them into a single word or short phrase) or to have GPT-4 judge correctness based on both the model’s response and the ground-truth answers. This approach could mitigate formatting-related biases and focus more accurately on the model’s scene-understanding capabilities.\n3. Lines 315–318 reveal that human annotators do not achieve 100% accuracy on the benchmark, implying the presence of ambiguous samples. The authors may wish to analyze and remove such ambiguous cases before releasing the dataset, as this would be pivotal for the broader research community.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Strengths\n1. This work offers valuable insights into the performance of various models, which is much appreciated.\n\nWeaknesses\n1. For open-source 2D VLMs, the InternVL series is also widely recognized. It would be beneficial to include these models in the benchmark results.\n2. Current 3D VLMs underperform, likely due to insufficient instruction-tuning for such tasks. To help the field better understand their limitations, it would be useful if the authors fine-tuned these models with a small amount of relevant data before conducting the benchmark.", "relation_to_broader_scientific_literature": "Overall I think the proposed benchmark is novel and interesting.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "88hODBrPcH", "reviewer": "Reviewer_RWSo", "summary": "The paper introduces a 3D reasoning benchmark called Hypothetical 3D Reasoning (Hypo3D). Specifically, the components of the benchmark can be summarized as follows. Consider a 3D scene representation (S) and a world frame from the scene (F) that contains an anchor object for specifying the direction to the model, e.g. “the table is to the left” will signal the model which direction is to the left. A set of context changes (C) are designed that are capture possible modifications to the scene S to obtain modified scenes S* (note that S* is not obtained). And finally, a set of question (Q) and answers (A) on the modified scene S*. The idea is to assess the “imagination” of existing foundation models to answer questions (Q) given the scene (S) and the context changes (C) by imagining what S* would look like. The authors consider three broad categories of questions: (1) scale-based, (2) direction-based, (3) semantic and include five types of context changes: (1) movement, (2) removal, (3) attribute, (4) addition, and (5) replacement. Experimental results for a variety of LLMs, 2D VLMs, and 3D VLMs are shown on this benchmark highlighting that the existing models fail in hypothetical reasoning.\n\n## Update after rebuttal\n\nI thank the authors for addressing all of my concerns and questions. I will keep my weak accept rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful feedback. We’re pleased to hear that you found our dataset to be a great contribution. All 2D VLM results reported below use the semantic map by default.\n\n**Q1: Answer Annotation**\n\nSimilar to the answer types in Fig. 12 of SQA3D, our annotations include object names, attributes (e.g., shape, color, size, functionality, state, etc.), directions, numbers, and so on. The complete answer distribution will be included in the final version.\n\n**Q2: World Frame**\n\nWe randomly rotated the top-view images before inputting them to VLMs, ensuring the world frame definition aligns with the person's orientation in 25% of scenes.\n\n**Q3: Anchor**\n\n4.41% of context changes involve changes to the anchor object. E.g.,\n\n*Orientation: The shower is on the front side of the scene.*\n\n*Change: The shower has been moved to the left of the toilet.*\n\n**Q4: PM**\n\nFollowing TopviewRS, we used PM to measure word-level overlap, particularly for answers like \"front\" and \"front left\". We also employed SBERT scores to reduce evaluation bias in phrasing (Fig. 14). We further computed GPT-based score following MSQA, with results detailed in our response to **kxrB**. Both metrics show consistent ranking with PM.\n\n**Q5: Baseline**\n\nWe conducted a baseline experiment with GPT-4o, which showed significantly lower EM accuracy than models in Table 1. This confirms that LLMs/VLMs are not merely recognizing objects or guessing, but possess limited capacity for hypothetical reasoning. Also, using semantic maps improved performance over non-semantic inputs, as VLMs benefit from explicit hover-text labels for more accurate recognition.\n\n|Model|Baseline|Reasoning|\n|-|-|-|\n|GPT-4o (non-sem.) |14.86|33.58\n|GPT-4o (sem.)|17.15|45.50\n\n**Q6: In-context learning (ICL)**\n\nWe applied three-shot ICL to 2D VLMs and LLMs. ICL generally reduced EM performance, potentially because the limited example set cannot adequately represent the diversity in our context changes and questions. Moreover, we observed models directly copying answers from examples, indicating their inability to process long context effectively.\n\n|Model|w/o ICL|w/ ICL|\n|-|-|-|\n|LLaMA-3.2|29.30|23.88|\n|LLaVA-OV 72B|40.26|33.53|\n|Qwen2-VL 72B|41.94|36.52|\n\n**Q7: Chain-of-Thought (CoT)**\n\nWe have utilized CoT to explicitly decompose the task into: (1) imagine how the change affects the scene, and (2) answer the question based on the changed scene.\n\nTo further investigate CoT, we tested models with a simplified prompt:\n\n*Scene orientation: {}*\n\n*Context Change: {}*\n\n*Question: {}*\n\n*Answer:*\n\nRemoving CoT prompting reduces performance in most models except Qwen2-VL 72B, suggesting that step-by-step reasoning aids hypothetical reasoning to some extent. Still, results lag behind human levels, and we plan to explore more advanced CoT methods in the future.\n\n|Model|w/o CoT|w/ CoT|\n|-|-|-|\n|LLaMA-3.2|23.91|26.08|\n|LLaVA-OV 72B|42.78|43.01|\n|Qwen2-VL 72B|44.90|44.25|\n|LLaVA-3D|29.30|31.56|\n\n**Q8: Hallucination**\n\nThe hallucination in line 359 refers to the model incorrectly adjusting its answer in response to an irrelevant change, as noted in Insight 5. More examples of such hallucinations are provided in the response to Q10.\n\n**Q9: Anchor Object as Answer**\n\nThe table displays the probability at which models predict anchor objects as answers. LLaVA-3D exhibits a much higher rate compared to both other models and ground truth frequencies, suggesting it tends to copy anchor objects as answers rather than engaging in reasoning.\n\n||GT|GPT4o (Text)|GPT4o (sem.)|LLaVA-3D|\n|-|-|-|-|-|\n|Rate (%)|3.4|3.2|2.8|5.3|\n\n**Q10: No Overlap**\n\nWe considered questions where the queried and changed object do not overlap. E.g.,\n\n*C: The lamp has been put onto the bath cabinet.*\n\n*Q: Where is the toilet paper relative to the trash can?*\n\n**Q11: World Frame Definition**\n\nThe main difference between the w. camera and w/o frame settings is the inclusion of an additional camera view image. While models may not accurately interpret scene orientation from it, the image still provides extra visual context, so a significant accuracy drop is not expected. \n \n**Q12: Non-semantic vs. Text-only**\n\nGPT4o (non-sem.) underperforms due to difficulty recognizing objects in noisy scenes. In contrast, GPT (text-only) directly receives explicit object names and attributes from the caption.\n\n**Q13: Human Performance**\n\nHuman performance falls short of 100% due to typos, formatting mismatches, and occasional misinterpretation of noisy scenes. Notably, less-than-perfect human performance is common in previous open-ended VQA benchmarks such as ScanQA and SQA3D. Below are human-failed examples:\n\nMore examples can be found in our response to **kxrB**.\n|Question|Human|GT|\n|-|-|-|\n|How does the added Nescafe espresso machine's position compare to the paper towel?|Above the paper towel|Higher|\n|Which is closer to the added rubber gloves: the trash can or the paper towel roll?|Paper tower roll|Paper towel roll|", "claims_and_evidence": "Most of the claims are clear. The major claim in the paper is that given the benchmark, existing LLMs and VLMs show unsatisfactory performance which is supported by the evaluation and experiments in the paper. However, I have a bunch of questions on the evaluation that are listed in the following sections.", "methods_and_evaluation_criteria": "The paper proposes a benchmark by itself and uses that to study LLMs and VLMs. The design of the benchmark explores an interesting problem of “hypothetical reasoning” and is a good contribution to the community and makes sense to assess LLMs and VLMs.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "**Benchmark design:**\n* More insights in the groundtruth answers of the questions: in Line-240 (right column) it is mentioned that an answer contains 1.28 words on average. What do the answers look like? Are they always name of the objects, directions, numbers, etc.?\n* How diverse are the directions defined using the world frame? What I mean by this is - from a given top-view image of a 3D scene that a person is looking at, let’s consider the person’s left as the “person-left” direction for the scene. But depending on the “left” that the world frame defines, that may or may not match with “person-left” -- if it matches how often it is? Or how often does it not match?\n* Are there examples in which the anchor object is the object under consideration for the context change instruction? E.g. the same object that is used as anchor object is modified in the context change instruction.\n\n\n**Evaluation:**\n* While the use of exact match (EM) as a metric makes sense, I am not satisfied with the use of partial match (PM) as a metric that computes the percentage of overlapping words. The reason is because the answers on average are 1.28 words long, so given such short-length answers, PM doesn’t make sense. Also, is the overlap computed over words or tokens?\n* Missing baselines: as the answers mostly would contain the name of objects, attribute of objects, and directions, it is very likely for the LLM/VLM to spit out random words by simply recognizing objects in the scene. So, one important baseline would be to ask the LLM/VLM to recognize the objects, their attributes, etc. in the scene (this will result in a set of words for a scene) and make a random prediction out of that set and then report the accuracy obtained.\n* In-context learning: Was in-context learning tried? It would be great to see the performance of these models when in-context examples are provided.\n* Chain-of-thought: Did the authors try CoT on Hypo3D? It would be also great to see the performance of these models using chain-of-thought prompting on this benchmark.\n* Lines-359: “... most models exhibit severe hallucinations …” - any analysis on that? What do the model hallucinations look like? Are there any correlations with the question or context change type?\n* How often does the model predict the anchor object as the answer?\n* In”Insight 5”: do the evaluation consider questions in which there is no overlap of the object on which the question is asked and the object that is considered for context change. E.g. “what is the color of the coffee table” is the question and “move the couch to the left of the TV” is the context change.\n* In Table 2, for the “w. Camera” results, the drop is accuracy wrt “w/o frame‘ is not much. Any further analysis on this?\n* Any insights on why in Table 1: GPT4o (with non-semantic top-view map) performs worse than GPT4o (text only)?\n* Lines-317-318 (left column): “... though it is not 100% due to the open-ended nature of Hypo3D questions … ” -- what does this mean exactly? Any examples?", "relation_to_broader_scientific_literature": "I think the major contribution of the paper is the benchmark that can be broadly used to assess LLMs and VLMs to make them spatially aware and enable spatial reasoning. However, I think the evaluation process and metrics used in the paper need to be made more rigorous (as discussed above).", "essential_references_not_discussed": "I think the references are sufficient.", "other_strengths_and_weaknesses": "Most of my concerns are listed above. Although the benchmark is a great contribution, I feel the assessment of the LLMs and VLMs in the paper could have been more rigorous. E.g. evaluation using in-context learning, chain-of-thought, etc.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I have included all of my concerns in the previous sections. I would like responses focussed on answering those concerns in the rebuttal.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HT7Mq6Dwbl", "reviewer": "Reviewer_2kmQ", "summary": "The paper introduces Hypo3D, a benchmark task evaluating foundation models' ability to use hypothetical reasoning to \"imagine\" missing perceptual information in dynamic 3D scenes. It provides a dataset with various context changes and questions, showing that current models significantly underperform humans, frequently hallucinate irrelevant details, and struggle especially with movement changes and directional reasoning. The findings highlight current MLLMs' limitations in imagination-based reasoning, crucial for achieving human-like cognitive abilities.\n\n## update after rebuttal\nThe rebuttal has addressed most of my concerns. The remained concern is that the key factors for the bottlenecks of 2D VLM and LLM in the hypothetical reasoning task need to be further discussed. Overall, the motivation for benchmarking MLLM's hypothetical reasoning ability is interesting. I will keep my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have added further explanations and additional experimental results to address your concern regarding the input data type issue.\n\n**Q1: 2D VLM Input**\n\nWe adopted top-view images for 2D VLMs to maintain consistency with established 3D reasoning benchmarks like SQA3D. While top-view images cannot fully capture 3D geometry, they provide the most comprehensive spatial information in a single image.\n\nBesides, modern 2D VLMs have been able to extract depth cues from single images and reason about 3D positioning. This has been validated by our results, where 2D VLMs perform adequately on questions involving vertical reasoning (e.g., height). Also, Human evaluators also have reported that top-view images were generally sufficient for them to answer questions in our dataset.\n\nAdditionally, we evaluated 2D VLMs (semantic maps) using multi-view inputs (top, front, back, left, and right) compared to using top-view only. The results on 50 randomly sampled scenes below shows that performance remains comparable to using only the top view. This suggests that while multi-view inputs offer richer visual information, integrating visual features from different views presents another challenge for the models.\n\n| Model         | View  | EM    | PM    |\n|---------------|-------|-------|-------|\n| LLaVA-OV 7B   | Top   | 34.81 | 38.60 |\n|               | Multi | 34.24 | 38.19 |\n| LLaVA-OV 72B  | Top   | 43.01 | 46.83 |\n|               | Multi | 42.52 | 47.06 |\n| Qwen2-VL 7B   | Top   | 34.40 | 38.91 |\n|               | Multi | 35.99 | 41.19 |\n| Qwen2-VL 72B  | Top   | 44.25 | 48.25 |\n|               | Multi | 43.04 | 47.50 |\n\n**Q2: LLM Input**\n\nTo assess how caption detail affects LLM performance, we tested LLaMA-3.2 with varying numbers of sampled captions. As shown in the table below, more detailed inputs do not consistently improve performance—possibly due to the increased challenge of long-text reasoning. Following the SQA3D protocol, we use 30 randomly sampled object captions for the final scene description.\n\n| #Captions | EM    | PM    |\n|----------|-------|-------|\n| 30       | 23.95 | 28.62 |\n| 50       | 23.88 | 28.34 |\n| 100      | 24.34 | 28.91 |\n| 200      | 22.91 | 28.01 |\n\n\n**Q3: Dataset Comparison**\n\nThe table below compares Hypo3D and existing 3D visual grounding (VG) and captioning datasets. Hypo3D is the first to annotate all question types and world frames, requiring models to understand hypothetical scenes.\n\n| Dataset             | Task                           | Question Type Annotation? | Hypothetical? | World Frame? | #Scans | #Language | Annotation         |\n|---------------------|--------------------------------|----------------------------|----------------|--------------|--------|-----------|---------------------|\n| ScanRefer           | VG                             | N/A                          | ✗              | ✗            | 0.7k   | 11k       | Human              |\n| Sr3D                | VG                             | N/A                          | ✗              | ✗            | 0.7k   | 115k      | Template           |\n| ScanQA              | QA                             | ✗                       | ✗              | ✗            | 0.8k   | 41k       | Template           |\n| SQA3D               | QA                             | ✗                        | ✗              | ✗            | 0.65k  | 33.4k     | Human              |\n| ScanScribe          | Captioning                     | N/A                        | ✗              | ✗            | 1.2k   | 278k      | GPT                |\n| MMScan              | VG + Captioning + QA           | ✗                          | ✗              | ✗            | 5.2k   | 6.9M      | GPT + Temp. + Human|\n| Hypo3D (Ours)       | QA                             | ✓                          | ✓              | ✓            | 0.7k   | 15k       | GPT + Human        |\n\n**Q4: Impact of Recognition on Reasoning**\n\nTo mitigate the impact of object recognition on hypothetical reasoning, most experiments in the paper utilized semantic top-view maps with explicit text labels. We also evaluated VLMs on unchanged scenes in Table 3, where the performance was significantly higher. This confirms that the main challenge primarily lies in reasoning about hypothetical changes. All results above will be included in the final version of the paper.", "claims_and_evidence": "The experiments on Hypo3D show the deficiency of current VLMs, supporting part of the motivation for proposing such a benchmark. \n\nOne concern is whether the input data types are appropriate for verifying the hypothetical reasoning ability of VLMs.\n\n- For 2D VLM, the input data is a top-view image, which compresses the 3D scene in a plane. There exists information loss. Besides, the movement is a 3D transformation, which could not be imagined in the top view. The reason is more related to the 3D understanding ability of 2D VLM. Therefore, this type of data may not be appropriate for testing the hypothetical reasoning ability of 2D VLM. \n\n- For LLM, the ability is related to the level of detail provided in the caption.", "methods_and_evaluation_criteria": "The main concern is about the input data type problem mentioned before, which may not support the motivation of testing the hypothetical reasoning ability but the 3D imaginary ability.", "theoretical_claims": "There is no proof for theoretical claims.", "experimental_designs_or_analyses": "The experiments show current VLMs' ability of hypothetical reasoning on Hypo3D.", "relation_to_broader_scientific_literature": "This paper proposes a benchmark to measure the ability of current VLMs' hypothetical reasoning in indoor scenes, which includes top-view images, scene caption, point cloud, and RGB-D video based on the ScanNet dataset. The benchmark demonstrates the deficiency of imagination.", "essential_references_not_discussed": "It is recommended that authors add a comparison between Hypo3D and current 3D grounding and captioning datasets, e.g. ScanRefer, MMScan, and a series of datasets.", "other_strengths_and_weaknesses": "The paper is well-written and organized.", "other_comments_or_suggestions": "Nil.", "questions_for_authors": "In fact, current 2D VLMs themselves may not be proficient at handling scale direction questions. The bad performance on Hypo3D comes from both hypothetical reasoning and fundamental ability. A good comparison experiment is testing the VLMs without adding changes in the scene, which can better help to find the key reason.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sIg45M7XZ6", "reviewer": "Reviewer_aKvW", "summary": "This paper presents the construction of a new dataset, Hypo3D, for evaluating the hypothetical reasoning performance on 3D scene. Then, the authors conduct evaluation of several benchmark methods on this dataset and show huge performance gaps between human's performance and model's performance.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Interesting.\n\n**Q1: About Novelty.**\n\nOur Hypo3D benchmark is a novel, methodological contribution in itself, as noted by other reviewers.  It focuses on structured evaluation and it is the first 3D reasoning benchmark with explicit question-type annotations, enabling fine-grained evaluation. Many influential multimodal reasoning benchmarks [1–3] focus on evaluation rather than proposing new models, yet remain highly impactful.\n\nCurrent vision models primarily handle basic spatial and semantic relationships but lack robust, holistic scene understanding. At the same time, reasoning in dynamic 3D environments remains a long-standing real-world challenge. We argue this challenge should not be deferred until all vision problems are solved. Instead, hypothetical reasoning offers a parallel research path, enabling progress on complex 3D reasoning alongside ongoing advances in foundational vision tasks. Furthermore, hypothetical reasoning is central to human cognition. As humans, we often reason with incomplete perceptual information, updating our mental models as new data emerges. This iterative process—forming, testing, and refining hypotheses—is fundamental to intelligent behavior. Integrating this capability into VLM is a step toward more human-like reasoning and a key milestone on the path to AGI.\n\nOur problem fits into the prediction-and-reasoning domain only when robust, real-time 3D scene updates are possible. However, current limitations in 3D reconstruction and editing, as noted in the paper, make accurate future scene prediction challenging. As a result, the task naturally shifts to an imagination (inference)-and-reasoning problem.\n\nEven widely used benchmarks like SQA3D could conceptually be constructed by posing situational descriptions and revising answers. Both our work and SQA3D deliberately avoided this. SQA3D emphasizes situational diversity, whereas Hypo3D focuses on diverse context changes. Our pipeline first collects a wide range of valid context changes and then generates corresponding questions and answers. Starting with fixed questions would have greatly limited feasible changes, especially given the strict constraints Hypo3D enforces on both changes and question design. \n\nText descriptions may not always be available, but they remain a cost-effective and widely accessible alternative to reconstructing the full 3D scene after each change. Our future work will explore multimodal representation of changes  (e.g., images, depth) for more comprehensive hypothetical reasoning. Notably, in other reasoning tasks, early works like SQA3D relied on text to define the observer's situation, while later studies such as MSQA expanded to multimodal representations. This progression highlights the importance of text as a foundational step in developing new reasoning tasks.\n\n**Q2: Evaluation Criteria.**\n\nMost methods we evaluate are vision-language models (VLMs), not standalone LLMs. While VLMs include an LLM backbone, it is instruction-tuned with large-scale visual data (e.g., image captioning, VQA), endowing it with visual perceptual abilities beyond standard language understanding. \n\nWe conducted additional experiments to examine the impact of visual encoders on VLMs' hypothetical reasoning. Specifically, we compared Cambrian-1 [4](using SigLIP, CLIP, DINOv2, and ConvNeXt) and LLaVA-Next (using only CLIP) [5], both sharing the same LLM backbone. Partial match results on semantic top-view maps below show that while leveraging more visual features can improve reasoning performance, a significant gap remains compared to human performance.\n\n| Model              | LLM Backbone     | PM     |\n|-|-|-|\n| Cambrian-1 13B     | Vicuna1.5-13B       | 41.80  |\n| LLaVA Next 13B     | Vicuna1.5-13B       | 38.87  |\n| Cambrian-1 34B     | Hermes2-Yi-34B       | 44.42  |\n| LLaVA Next 34B     | Hermes2-Yi-34B       | 41.47  |\n| Human     | -       | 92.50  |\n\n**Q3: Difference to Existing Problem Formulation.**\n\nUnlike traditional VQA tasks (2D or 3D), where answers can be directly inferred from the visual input, our hypothetical reasoning task introduces a key distinction: the visual input provides only prior context and is insufficient on its own.  Hypo3D shifts the focus from “see and answer” to the more complex cognitive process of “see, imagine, and answer”.\n\n[1] TopViewRS: Vision-Language Models as Top-View Spatial Reasoners, EMNLP, 2024.\n\n[2] SQA3D: Situated Question Answering in 3D Scenes, ICLR, 2023.\n\n[3] HourVideo: 1-Hour Video-Language Understanding, NeurIPS, 2024.\n\n[4] cambrian-1: a fully open, vision-centric exploration of multimodal llms, NeurIPS, 2024.\n\n[5] LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, Arxiv, 2024", "claims_and_evidence": "The authors claimed that existing models struggle to reason effectively in hypothetically changed scenes. Extensive experimental evaluation indeed validate this. But this claim does not involve any novel method. The only contribution is related to the dataset.", "methods_and_evaluation_criteria": "1) There is no proposed method. \n2) Evaluation criteria may be problematic. Most of the evaluation methods are based on large language models. To my understanding, in order to reason well, the perception module needs to be powerful enough to perceive as much useful information as possible, before performing any meaningful reasoning. The authors put too much emphasis on large language models while overlooking the capability of visual perception modules. As a result, the results are very poor compared to human performance.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "1) There is no proposed method, and the authors mainly evaluated existing approaches.\n2) As discussed above, the authors put too much emphasis on LLMs, and overlook the importance of visual perception modules, yielding models' performance much lower than human's.", "relation_to_broader_scientific_literature": "The key contribution of this paper is the dataset. This dataset has limited value to the society.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "1) The novelty in the dataset is limited. Conceptually, the dataset can be constructed by modifying existing 3D datasets through posing context changes, and revising the solutions of existing datasets in response to the context changes. (As we have full controls about the objects in the scene, this is convenient and easy.) As such, the value of constructing the Hypo3D dataset is greatly reduced.\n2) The significance of this research is limited. No new method is proposed in this paper. \n3) The so-called hypothetical reasoning is questionable. Before fully understanding the scene visually, hypothetical reasoning does not make too much sense. In addition, this problem in theory is similar to prediction-and-reasoning problem, i.e., given an existing scene, and the tendency of changing, we would like to reason over the future scene. Also, the context changing is provided in the form of text decription. This is difficult to achieve for real-world application where we have only a change in the scene while there is no such text description of context changes.", "other_comments_or_suggestions": "No.", "questions_for_authors": "In summary, I would like to see more justifications on:\n1) The novelty of this paper.\n2) The difference and significance of propose dataset in comparison to existing datasets.\n3) More justifications on hypothetical reasoning over existing problem formulation.\n4) More experimental results that utilize visual models to extract visual features for the scene, integrat with linguistic features, and leverage LLMs to better answer the questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by four experts in the field, and received three positive reviews and one negative (4, 3, 3, 1). The positive reviews commended the utility of the proposed dataset (with one reviewer calling it novel, compelling, crucial), and raised some minor clarification questions, which the authors diligently addressed in a well-crafted rebuttal. The negative review complains that the paper does not propose a new method, but ICML guidelines allow novel datasets to be contributed too. The negative review further states that the dataset/benchmark is easy to create, but the authors dispute this convincingly. The AC's recommendation is to accept.
Decision: Accept (poster)