SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenes
Authors: Xiongkun Linghu
Jiangyong Huang
Ziyu Zhu
Baoxiong Jia
Siyuan Huang
Venue: ICLR 2026 Poster (2026)
Keywords: 3D scene reasoning
chain-of-thought reasoning
multimodal LLM
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A step-by-step reasoning framework for 3D scene understanding

Abstract: Existing research of 3D LLMs still struggles to achieve efficient and explainable reasoning, primarily due to the under-exploration of the mechanism of human-like scene-object grounded reasoning. This paper bridges the gap by presenting a novel framework. We first introduce a Chain-of-Thought reasoning framework in 3D scenes (SceneCOT), decoupling a complex reasoning task into simpler and manageable problems, and building corresponding visual clues based on multimodal expert modules. To enable such a framework, we build the first large-scale 3D scene Chain-of-Thought reasoning dataset, SceneCOT, including more than 190k high-quality data instances. Extensive experiments across various complex 3D scene reasoning benchmarks demonstrate that our new framework achieves state-of-the-art with clear interpretability. To our knowledge, this is the first attempt to successfully implement the COT technique for achieving human-like step-by-step reasoning for 3D scene understanding, where we show great potential in extending it to a wider range of 3D scene understanding scenarios.

Published as a conference paper at ICLR 2026
SCENECOT: ELICITING GROUNDED REASONING IN 3D SCENES
Xiongkun Linghu', Jiangyong Huang 2 Ziyu Zhu Baoxiong Jia Styuan Huang!:
State Key Laboratory of General Artificial Intelligence, BIGAI "Peking University, TTsinghua University
ABSTRACT
Existing research 3D Large Language Models (LLMs) still struggles to achieve grounded question-answeringg primarily due to the under-exploration of the mech- anism of human-like scene-object grounded reasoning. This paper bridges the gap by presenting a novel framework. We first introduce grounded Chain-of- Thought reasoning method in 3D) scenes (SCENECOT), decoupling a complex reasoning task into simpler and manageable problems, and building corresponding visual clues based on multimodal expert modules. To enable such method, we develop SCENECOT-ISS 1-O0-18552 185,, the first large-scale grounded CoT reasoning dataset, consisting of 185K high-quality instances. Extensive experiments across various complex 3D scene reasoning benchmarks demonstrate that our new framework achieves strong performance with high grounding- QA coherence. To the best of our knowledge, this is the first successful application of CoT reasoning to 3D scene understanding, enabling step-by-step human-like reasoning and showing potential for extension to broader 3D scene understanding scenarios.
INTRODUCTION
Understanding 3D scenes is fundamental capability for building human-level embodied agents (Chen et al., 2019; Yang et al., 2025b; Chen et al., 2024a; Gong et al., 2023; Song et al., 2025). Despite growing interest and progress in this area (Chen al., 2024c; Jia et al., 2024; Zhu et al., 2024b; Huang et al., 2024b; Chen et al., 2023b; Zhou et al., 2024), reasoning in complex 3D environments remains highly challenging, especially in embodied and situated settings (Ma et al., 2023; Linghu et al., 2024). Reasoning in 3D scenes requires navigating large spaces, interpreting intricate spatial relations, and coping with partial observability. Addressing these challenges calls for principled task decomposition and step-by-step reasoning, yet existing research has largely overlooked this aspect.
Recent benchmarks highlight the consequences of this gap. In particular, Beacon3D (Huang et al., 2025a) reveals that current 3D vision-language models often produce plausible answers without grounding them in the scene, leading to poor grounding QA coherence. This underlines fundamental limitation: while models may generate fluent responses, they fail to connect intermediate grounding steps with the final reasoning outcome. We argue that achieving human-like 3D reasoning requires answers that emerge from transparent, grounded, and stepwise reasoning processes.
In the language domain, Chain-of- Thought (CoT) reasoning (Wei al., 2022) has demonstrated the power of step-by-step problem decomposition, enabling large language models to achieve superhuman performance in math, science, and logic tasks (Wei et al., 2022; Guo et al., 2025; Jaech et al., 2024). CoT mirrors human cognition by breaking complex problems into manageable subproblems, paradigm that naturally aligns with the multi-hop reasoning needed in 3D scenes. Yet, directly transferring CoT to 3D settings is nontrivial due to the difficulty of aligning language- reasoning with multimodal 3D scene representations. To date, CoT has been explored in 2D vision-language reasoning (Shao et al., 2024a; Liu et al., 2025), leaving its potential in 3D reasoning largely untapped.
ofain-of-Thought
"Corresponding author.

---

Published as a conference paper at ICLR 2026
I am facing desk. There is brown chair on my What the my 2 o'olock.
Thinking...
<think, typesThis queston attribute need to ground the objects the scene, 4think,
_gro_rge rgn-
<tinkk rgre-The objects on my of clolock<tthink, rgnn
<think, grdhGround the object: The bike on my 2 o'deock, inin grd>
TOBJ
retrieve image the target object <tthink, tasko
<highiggt_ 08jo-
<mng start>
<img_eedd> Lend>
<hink sumstlow answer the question based on the object image. <think, sumo
canserrsieercdnssee
Silver
Figure I: Reasoning chain visualization of SCENECOT, Example of how SCENECOT decomposes question into step-by-step reasoning: from identifying the task type, localizing relevant objects, and grounding the target entity, to retrieving the visual clue and generating the final answer.
In this work, we present SCENECOT, a novel framework for step-by-step grounded reasoning in 3D scenes. SCENECOT explicitly decomposes complex 3D reasoning tasks into four stages: (1) task recognition and analysis, (2) task-relevant region localization, (3) entity and attribute grounding with multimodal expert modules, and (4) grounded reasoning that integrates intermediate results into coherent final answers. This hierarchical workflow ensures that each answer is supported by explicit grounding steps. thereby enhancing grounding- -AA coherence. To enable such reasoning, we construct CCENECOT--18EL the first large-scale grounded CoT dataset for 3D reasoning, containing over 185K high-quality reasoning traces across diverse 3D QA benchmarks. Each trace captures the full stepwise reasoning trajectory, including task-oriented region selection, object grounding, and final answer generation. Extensive experiments on situated reasoning benchmark MSQA (Linghu et al., 2024), as well grounding-OU coherence evaluation on Beacon3D (Huang et al., 2025a), show that SCENECOT achieves strong performance while producing reasoning that both interpretable and faithfully grounded.
In summary, our contributions are as follows:
We propose SCENECOT, novel Chain-of- Thhoughf reasoning framework that decomposes complex 3D scene reasoning tasks into manageable steps, enabling human-like like, grounded, and interpretable reasoning.
We construct SCENECOT-185K. the first large-scale dataset with stepwise grounded reasoning traces in 3D scenes, containing over 185K high-quality instances.
We demonstrate that SCENECOT achieves strong performance on challenging 3D reasoning benchmarks, and importantly, improves grounding- coherence, as revealed by in-depth analyses on Beacon3D.
2 RELATED WORK
LLMs for 3D Scene Understanding Understanding 3D scenes is essential for developing human- like intelligence, and recent studies have increasingly explored the use of LLMs for 3D vision- language understanding. Early efforts such as 3D-LLM (Hong al., 2023) lift multi-view 2D features into 3D space and align them with text embeddings, while PointLLM (Xu et al., 2023) leverages point cloud encoders for object-level geometry reasoning. LEO (Huang et al., 2024b) aligns object-centric 3D representations with LLMs for 3D vision-language (3D-VL) tasks. Subsequent works have expanded this line of research (Chen et al., 2024b; Yang al., 2025a; Fu al., 2025; Chu et al., 2024; Zhang et al., 2024c; Qi et al., 2023). For instance. Grounded 3D-LLM (Chen et al., 2024c) enhances point-level semantic alignment with language, Chat-Scene (Huang et al., 2024a) builds per-object 3D features via mask proposals, LLa VA-3D> (Zhu et al., 2024a) integrates 3D positional embeddings into LA (Liu et al., 2023), Video-3D LLM (Zheng et al., 2024) leverages pretrained video MLLMs for temporal-spatial context, and SplatTalk (Thai et al., 2025) applies 3D Gaussian Splatting (Kerbl et al., 2023) to generate language-alige aligned 3D tokens. LEO-VE VL (Huang et al., 2025b) further improves the efficiency of scene representation for 3D MLLMs. Despite these advances, most 3D-LLMs rely on sparse supervision in an end-to end-to-end training paradigm, with limited exploration of intermediate reasoning processes. As shown in Beacon3D (Huang al., 2025a), these models
2

---

Published as a conference paper at ICLR 2026
often produce answers that appear correct but lack explicit links to scene grounding. leading poor grounding QA coherence. This reveals a fundamental gap: while 3D-LLMs can encode multimodal representations, they rarely incorporate structured. step-by-step reasoning mechanisms that are essential for robust 3D scene understanding. We argue that remains largely untapped in 3D, but is necessary for overcoming overfitting and building more interpretable, human-like 3D-VL models.
reveals
(CoT) reasoning
Reasoning Capability of LLMs and MLLMS LLMs have demonstrated remarkable reasoning abilities across diverse domains such as mathematics, programming. and scientific QA (Guo et al., 2023: Ou al., 2024; Chen et al., 2024d; Huang et al., 2022; Shao al., 2024b; Jaech et al., 2024). These capabilities are significantly enhanced by Chain-of-Thouggee (CoT) prompting (Wei et al., 2022), which decomposes complex tasks into step-by-stepy subproblems. Inspired by this, researchers have sought to extend reasoning into multimodal settings. Flamingo (Alayrac et al., 2022) bridges frozen vision and language backbones with aross-attention, Shikra (Chen et al., 2023a) and KOSMOS- 2 (Peng et al., 2023) emphasize grounded reasoning over visual inputs, while OMG-LLaVA (Zhang et al., 2024a) unifies pixel-, object-, and region-level understanding. More recent works attempt to bring step-by-step reasoning into MLLMs. V* (Wu & Xie, 2024) incorporates sequential visual search for fine-ge fine-graing gre-grrined recognition, Video-of- Video-of-Thought (Fei et al., 2024) introduces a perception-tor recognition workflow for video reasoning, and Visual COT (Shao al., 2024a) explicitly integrates bounding boxes and patch-level grounding as intermediate "thoughts." * Commercial models such as GPT-o3 and GPT-o3 mini (OpenAI, 2025) showcase impressive visual reasoning abilities by combining multiple visual experts. However, these advances remain confined to 2D inputs. They lack explicit grounding in 3D space, limiting their ability to reason about spatially complex, embodied scenarios. In summary, while CoT reasoning has transformed both text-only and 2D multimodal models, it has not yet been fully explored in 3D scene understanding. Bridging this gap requires designing frameworks that integrate step-by-ster grounded reasoning with explicit grounding-QL coherence, which is the focus of our work.
3 SCENECOT: STEP-BY-STEP REASONING IN 3D SCENES
In this section, we present the design of SCENECOT, beginning with the formalization of step-by- step reasoning traces in common 3D reasoning tasks (Sec. 3.1). This structure reflects the human problem-solving process for complex understanding, as discussed in Sec. 1. We then describe the detailed learning and inference pipeline of SCENECOT, illustrating how it integrates and leverages 3D-CoTs to enhance reasoning capabilities in 3D scenes (Sec. 3.2).
3.1 CUAIN-OF-THOUGIITY
SCENECOT
Given 3D scene, an agent's 's situation, and a question to be answered, we define the reasoning trace for answering the question as a concatenation of the following step-by-step descriptions:
1. Task Recognition and Analysis: The reasoning trace begins with the identification of the underlying task required to answer the question (e.g., counting, navigation) along with initial analysis for solving it (e.g., ground objects for counting). This information is critical for the subsequent reasoning process. It may also determine which specialized models (e.g., detection, segmentation) to invoke in subsequent steps. We encapsulate this task-level guidance using the special token <think_typee within the reasoning trace.
Task-relevant Region Localization: Based on the task hints in the question and the agent's situation context, we significantly reduce the reasoning space by first providing region-level grounding for localizing the task-relevant subregion of the scene. Specifically, depending on the question, we discretize the surrounding space using either directional clues (e.g., left, right, front, back) or blokkbbaee reference frame (e.g., 1-12 clock). The sentence enclosed by <think_rgm indicates the candidate region for the symbolic engine to parse the corresponding objects.
3. Entity Grounding: This step grounds the target objects relevant to answering the question. We first generate detailed grounding instructions that encode object semantics, attributes, and relational context. These instructions enclosed by <think _riii_ggdi>,,___ and followed by a special [OBJ] token, which serves as a trigger for invoking specialized grounding modules to localize the referenced object(s).
3

---

Published as a conference paper at ICLR 2026
What bike dghh??
Optional
Image tokens
attibuuee
What on right?
This question of spalial relationship.
prot: 0.7 cabinet 11,1,.4,0 4,0.8.
bike: (1.1, 1,0 0:3G:
0.2,0.3,1.20 0.8
How many bises are on my right?
This is question counting.
Symbollc Engine
bike: 0.7 sofe: 0.2 pillow: 0.1
0.1 cabinel:
One
Grounding Module
How get the tike?
This is question navigation.
bikk: (-40. 1.2) probx
At your Zoolock.
Provide diockoise direction.
sofa: (-80. 1.4)
prob: 0.1 (...)
"the objects on my right
"the bike right
Symbalic Engine
Mask3D
Answer
Task Recognition
Spetial Region Becognition
Entity
Grounded
Roasoning
Figure 2: SCENECOT framework. The model decomposes 3D scene reasoning into four steps: task recognition, spatial region recognition, entity grounding, and grounded reasoning. Each stage introduces explicit grounding signals (e.g., objects, attrihutes, spatial positions), ensuring step-by-step reasoning and improved grounding-Q2 coherence.
4. Grounded Reasoning: Given the candidate object(s), generate task-specific instructions, enclosed by for downstream reasoning. These instructions guide the grounding model to retrieve compute relevant information based on task requirements, such as retrieving 2D images for attribute recognition and exporting object class probabilities for object existence verification. We consider the following types of grounded object information, each annotated with distinct tags:
that specify what information regarding these objects is necessary
Object probability (<ob j_prr>>>:: For tasks such counting and existence verification, we record the class probabilities of grounded objects, indicating the presence of objects.
3D Object Location with Probability (<obj loc prob>, <obj loc_ _oc__prr_________p______________ plr prob>): For tasks that require spatial reasoning, we record object class probabilities along with their positions 3D space(<ob j_1 oc cc__rrbb))... For navigation- tasks, we represent object locations in a 2D polar coordinate (<abj_loc_plr_rre
frame to facilitate direction- based reasoning.
Object Image Tokens (<highl ight_obj>, <img_stari grained attribute description tasks, we use <highl iigh_obba to trigger image retrieval, inserting object-level image patches as visual tokens tagged by <img> to support appearance- based reasoning.
<img_end>> For fine-
After obtaining the grounded object information necessary for solving the task, we include a summa- rization hint marked with <think sum> to guide the reasoning process, followed by the generation of the final answer, tagged with <answer>, An illustrative walkthrough of a sample reasoning trace for situated question answering task from the MSQA dataset is shown in Fig. with further details on each sub-process of the reasoning trace discussed in Sec. A.
3.2 SCENECOT LEARNING AND INFERENCE
We provide illustrative explanation of how SCENECOT learns and performs inference with our defined 3D-CoTs in Fig. 2. At its core, SCENECOT is built upon a powerful Multi-modal LLM (MLLM), which serves as the primary reasoning engine. To support the step-wise reasoning structure introduced in Sec. 3.1, 1 we incorporate two types of modular components:
Specialized 3D-VI. and 2D-VL models are employed for entity grounding and image reasoning. These models are initialized with pre- ree-trained weights and jointly updated during the training of SCENECOT. In particular, 3D visual grounding model and a 2D vision-language model serve as the backbone for the grounded reasoning process.
Symbolic engines, including off-the-shelf parsers and pre-trained models, used to extract object-specific grounding information (e.g., location, coordinates, image patches) to support region recognition and grounded reasoning, described in Sec. 3.1. These models remain fixed and are not updated during SCENECOT training. The contextual inputs are constructed through a predefined programming procedure, with implementation details provided in Sec. A.
4

---

Published as a conference paper at ICLR 2026
To train SCENECOT, dataset of annotated reasoning traces described in Sec. 3.1, jointly optimizing the reasoning engine and grounding modules under the following objective:
C = LCST + Lans + Lground:
where Lctr and Lans are causal language modeling losses for predicting the reasoning trace and final answer, respectively. feround is a cross-entropy loss applied only the specialized grounding module for accurate object grounding. We train the MLLM model using LoRA.
In the inference stage, SCENECOT follows the reasoning steps specified by the predicted 3D-CoTS, invoking the appropriate modules generate the final answer. Specifically, we use Mask3D (Schult et al., 2022) to an initial set of object proposals for the specialized grounding model select. For special tokens that invoke function calls, the corresponding modules executed externally, and their outputs are concatenated with prior predictions and fed back into SCENECOT for new inference pass to complete the reasoning process. Additional details model inference provided in the Sec. A.4.
4 THE SCENECOT-185K DATASET
To enable the learning of SCENECOT, develop large-scale 3D-CoT's dataset, SCENECOT- 185K, containing 185K data instances to support step-by-step reasoning in 3D scenes. The dataset comprises two representative tasks in 3D scene reasoning: (1) Situated Reasoning and (2) Object- Centric Reasoning. It follows the standard 3D-Co'Tss structure as defined in Sec. 3. We construct the dataset through a two-step process: metadata collection and reasoning trace generation.
4.1 METADATA COLLECTION
For Situated Reasoning, we use MSQA as the data source. Following the definition Sec. 3.1, we collect the following components: (1) object instances within the corresponding sub-region, (2) the question type, and (3) the grounding text. For question types, we adopt the primary categorization defined in MSQA and construct the corresponding COT data using the official metadata. As Beacon3D emphasizes grounded reasoning, we treat it as part of the AtrributeDescriipion sub-tasks within our unified task space.
Region-relevant and Question-relevan Objects Extraction First, we design a rule-based proce- dure to extract target objects based on the agent's location and orientation within the 3D scene. This begins by parsing the question and extracting directional cues using regular expression matching. In MSQA, directional information typically falls into two categories: cardinal directions (left, right, front, back) and clock-based directions (e.g.. "at the 1-12 o' clock"). For instance, given the question "How many tables are on my right?", we extract all objects located to the right of the agent. This rule-based method ensures that answers can be accurately inferred from the object list within the correspouding sub-region. Secondly, for the target object entities, we design rule-based method for Existence/Counting to ensure the correctness. For the remaining sub-tasks, we inherit the official annotations the target objects the released data.
Data Generation of GQA3D For Beacon3D, cannot directly construct the COT data due the absence of metadata. To obtain high-quality training set, we leverage Nr3D as our metadata source and construct new grounded question answering dataset, GQA3D. Nr3D is 3D Visual Grounding benchmark that primarily provides grounding texts and the IDs of target objects. We use GPT-4o to generate QA pairs based on the corresponding object images. All generated QA pairs are categorized as Attribute type in SCENECOT- 185K. Implementation details are provided in Sec. B.2.
4.2 REASONING TRACE GENERATION
After collecting the metadata, we construct the full reasoning chain following the steps outlined in Sec. 3.
1) For the sub-tasks Counting, Existence, Refer, Room Type, and Affordance, generate ground-truth thoughts using semantic labels and pseudo probabilities. Specifically, we randomly assign values between 0.5 and 1.0 represent target objects, while assigning values between 0 and 0.5 non-target
5

---

Published as a conference paper at ICLR 2026
Agent location, orientation
to get object for sleeping? Provide clockwise direction
Question type: Navigation
Position Calculation
Object locations
<think, typesThis navigation question, so need to ground the corresponding objects answer R.44think, type>
<grd rgn>
<think, ignnNow ail the objects
<tthink, rgno <think, grd>Ground the object The pillos on my right the miodle distance. You should find all possible
Sttick, grd>
[oa]]
taskaloow
potentil objects the probability-/think
<ist obj, prob>
<obi_ probabed (-120, 1.2): prob: 0.9 desk 1-12,4, 2.1); 0. 114bbb_prreb>
sumzNow answer question
object probabilkies. <think sums
<answer>At your 4 o'doockllnswerr>
Figure Illustration of reasoning trace generation, An example of navigation question and its corresponding reasoning trace. The process involves identifying the question type, grounding relevant objects, computing relative positions based on agent location and orientation, and generating the final answer through step-by-step reasoning.
objects. To control the token length of the input prompts, we cap the number of objects per training instance.
2) For the Spatial Relationship sub-task, we compute relative coordinates using the agent's location, orientation, and the positions of objects under the 3D rectilinear coordinate system.
MAQA
ROAS
3) For the Navigation sub-task, object locations are represented in a 2D polar coordinate system. To support the Attribule and Description sub-tasks, which require image retrieval, we construct an object image library. Object images are extracted from RGB frames in ScanNet using object positions and camera poses. We illustrate a representative reasoning trace generation pipeline of Navigation in Fig. 3. We finally get 145.6K and 40K data instances for Situated Reasoning and Object- Centric Reasoning, respectively. We also conducted a manual check of data quality. The statistical results can be found in Sec. B.3.
78.4%
Figure 4: Data distribution of SCENECOT-188K 185K.
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Tasks and Data. We evaluate SCENECOT two representative 3D scene reasoning tasks: Situated Reasoning (Situated Reasoning) and Object-Centric Reasoning (Object-Centric Reasoning). For Situated Reasoning,
adopt the MSQA benchmark built ScanNet, following the evaluation protocol of (Thai al., 2025). To ensure fair and consistent comparison, we use the refined Version-2.1 of MSQA (Linghu et al., 2024), and re-implement MSR3D and GPT-4o this version. For MSR3D, we follow the official setup, which includes the merged training data and object mask proposals from Mask3D, For Object-Centric Reasoning, we evaluate on the Beacon3D benchmark (Huang al., 2025a), which explicitly measures grounding-Q2 coherence. Following the official settings, we use ground-truth object masks for all methods isolate reasoning capability from detection performance.
Evaluation Protocols and Baselines. On both MSQA and Beacon3D, we use GPT-score as the primary evaluation metric. For MSQA, models jointly predict object masks and textual answers. while for Beacon3D, evaluation is performed using ground-truth masks. Beacon3D further reports two complementary metries: (1) GPT-Score (by case), which averages scores across individual QA pairs, and (2) GPT-Score (by object), which requires all QA pairs associated with object to be correct, directly reflecting grounding- QA coherence. In addition, we analyze the grounding-QA coherence metrics provided in Beacon3D to highlight the improvements brought by our method. We compare SCENECOT against a broad set of 3D scene reasoning baselines. On Beacon3D, we focus object-centric baselines that explicitly incorporate grounding, including GPT-4o (OpenAI, 2023), MSR3D (Linghu et al., 2024), PQ3D (Zhu et al., 2024b), Chat-Scene (Huang et al., 2024a), and VcrneVerse (Jia al., 2024). On MSQA, we additionally include SplatTalk (Thai et al., 2025), which

---

Published as a conference paper at ICLR 2026
Table 1: Experimental Results MSQA and Beacon3D. *: GPT-4o's input contains ground-truth object labels, locations. attributes. $: The result of MSQA is not based on Version-2.1 data. MSR3D, Chat-Scene, and LEO are trained on SCENECOT- 185K----K-------- 885--AAnn grounded COT). f: The models are trained on our dataset. The best and second-best performances are highlighted across the entire table. In the third column, 'Grounded' indicates whether the reasoning results can be explicitly linked to specific entities.
MSQA
Beacon3D
Methods
Grounded!
Count.
Exist.
Attr.
Spatial
Navi.
Others
Overall
Case
Obj.
GPT-4o"
X
32.3
79.3
79.0
37.0
31.7
91.6
52.3
57.1
20.2
LEO
X
32.5
88.5
58.7
44.2
39.6
81.4
54.8
43.2
7.8
MSR3D
X
32.3
93.1
50.0
46.5
54.1
75.6
54.2
Chat-Scene
X
45.8
7.8
SplařTalk
19.6
60.3
44.0
35.8
35.5
61.8
41.8
LEO!
X
29.3
87.5
55.1
46.3
45.6
81.4
52.9
52.5
12.7
MSR3D#
32.7
87.5
53.7
51.5
72.3
52.8
51.4
11.9
Chat-Scene?
37.4
92.0
49.0
47.0
58.3
83.7
56.6
53.6
14.0
SCENECOT
47.9
82.1
49.6
47.2
51.6
80.3
55.6
58,9
23.2
Table Grounding- QA Coherence comparison across methods. Main metrics: GC: good coherence (both grounding and QA correct); QA (Obj.): per-object QA performance. Additional reference metrics: Type 1: grounding correct but QA wrong: Type 2: QA correct but grounding wrong; DF: double failure (both wrong); R1 Typel/ / (Typel + DF); R2 Type2 (Type2 GC).
Method
Grounded?
GC
QA (Obj.) 1
DF
Type
Type 2
R1
R2 1
LEO
X
1.6
7.8
2.2
55.2
40.9
3.7
96.2
PQ3D
X
16.5
3.5
40.6
31.9
10.8
56.6
39.7
Scrse
XXX
20.4
6.6
31.6
28.3
19.5
50.6
48.5
Chat-Scene
19.5
7.8
24.7
29.8
25.8
44.4
56.9
SceneCOT
34.7
23.2
16.8
24.1
16.8
58.9
41.0
reports results on this benchmark. To ensure faimesss, we retrain several baselines on our training set and report the updated results.
Implementation Details. We build SCENECOT based on LLaVA- an open-source multimodal LLM framework built upon Vicuna-7B. For the Attribute and Description sub-tasks, the selected object image is passed through 2D vision encoder. The resulting image feature is then projected into the language embedding space via learnable projection layers. During training, we freeze these projection layers and apply LoRA to fine-tune the parameters of the L.M.. For 3D visual grounding, adopt fine-tuned version of PQ3D as expert model. This model has been trained on a subset of the domains provided in Scene Verse (Jia et al., 2024). Additionally, we design a lightweight object mask predictor estimate object logits. During training, we fine-tune both the parameters of PQ3D and the object mask predictor. Our model is trained for 5 epochs using 4 A100 GPUs. We provide more implementation details in Sec. A.5.
5.2 MAIN RESULTS ON 3D QA AND GROUNDING-QA COHERENCE
In the main results, we evaluate both 3D QA performance and grounding- coherence, and compare SCENECOT with baseline methods.
SCENECOT demonstrates strong QA performance on Situated Reasoning and Object-Centric Reasoning. It achieves notable gains on the challenging Counting sub-task by explicitly enumerating objects the relevant sub-region based on semantic similarity, providing clear visual grounding for reasoning. SCENECOT also performs well on Spatial tasks, including Refer and Spatial Relationship. Its performance is lower on Existence and Attribute compared to LEO and MSR3D, mainly due to grounding errors (see Sec. 5.3). While slightly below Chat-Scene overall, SCENECOT substantially outperforms all baselines on Beacon3D thanks to its scene-object grounded reasoning. contrast, Chat-Scene directly predicts answers from question and object-centric tokens without explicit ground- ing, limiting its ability on complex tasks like Counting, where object-level reasoning is critical. Additional baseline comparisons in Sec. C.1 further confirm the competitiveness of SCENBCOT.

---

Published as a conference paper at ICLR 2026
SceneCoT
exlstenne
novipation
Spatial refttaeriiip
60
region recogrition ition
groundingg
EM-R
40
counting
existence
refer
navigation
spatial
attribute
Figure 5: Ablation study. We ablate three key factors: (1) question type recognition, (2) region recognition, and (3) grounding loss. Results show that removing any of these components degrades performance, highlighting their importance for robust step-by-step reasoning.
SCENECOT achieves strong grounding- -A coherence. On Beacon3D, SCENECOT achieves the highest Good Coherence (34.7), substantially ahead of all baselines. This demonstrates that framework most reliably combines correct grounding with correct QA, rather than succeeding one dimension while failing on the other. By contrast, LEO reaches QA (Obj.) score of 7.8 but suffers from an extremely low GC of 1.6, highlighting a large mismatch between answering correctly and grounding correctly. Similarly, methods like Scene Verse (GC: 20.4, QA (Obj.): 6.6) and Chat-Scens (GC: 19.5, QA ): 7.8) show moderate QA performance but struggle to align it with consistent grounding. addition to its advantage GC, SCENECOT also delivers the highest QA (Obj.) score (23.2), large margin over all other methods. This reflects not only stronger per-object QA ability but also a tighter alignment between answers and the objects they refer to. Together, these results confirm that explicitly enforcing grounding before reasoning enables SCENECOT to achieve both accurate QA and high grounding - coherence, setting it apart from prior baselines.
5.3 ABLATION STUDY AND ANALYSES
Question type recognition is essential. As illustrated Fig. 2, SCENECOT first identifies the question type and builds the reasoning chain accordingly. For example, the symbolic engine outputs polar coordinates for Navigation and 3D bounding boxes for Spatial Relationship. To assess its role, we conduct ablation where all questions are forced into the same type. As shown in Fig. 5, treating all questions "as existence" restricts the model object probabilities only, leading to clear performance degradation. Likewise, forcing all questions to Spatial Relationship significantly reduces performance on Navigation, where polar coordinates are more suitable. These results demonstrate that recognizing the correct question type is crucial for constructing grounded reasoning chains.
Region recognition crucial, particularly for object-centric reasoning. To enhance grounding in Situated Reasoning, SCENECOT explicitly filters out irrelevant objects, narrowing the scope of candidates to task-relevant regions. Removing this component and providing all scene objects results in substantial performance drop, especially on Counting, Refer, and Attribute (see Fig. 5). This confirms that region recognition not only reduces noise but also strengthens grounding -Aa coherence by aligning reasoning with localized evidence.
Grounding loss plays a key role. In SCENECOT, the grounding module is trained with an additional loss based PQ3D, encouraging accurate object grounding during learning. Ablating this grounding loss (see Eq. (1)) yields noticeable performance decline on MSQA, particularly for Counting, Refer, and Navigation (see Fig. 5). Interestingly, performance on Existence less affected, likely because semantic labels alone suffice for this type of question. Overall, these results highlight the grounding loss as an important factor for maintaining step-by-step grounding consistency and improving reasoning accuracy.
Exploring the QA upper bound with better grounding information. In MSQA, predictions are limited by noisy semantic labels, masks, and object probabilities. To measure their impact, we test oracle settings that remove: Semantic errors (SE): inaccurate labels and masks; Grounding errors (GE): incorrect object probabilities. Removing SE improves performance, especially Count- ing. Removing both SE and GE pushes results close to the oracle bound, with Counting/Existeen near 100 and Navigation/Refer reaching 87, $7.2/84.9. These gains confirm that accurate grounding and
8

---

Published as a conference paper at ICLR 2026
Table Experimental results oracle data. In main results, utilize Mask3D to provide object masks and semantic labels. In this table, we explore the upper boundary in two aspects: 1) perfect object masks and semantic labels. but still based on the predicted object probabilities. 2) Oracle ground truth text-based thought. In the table, "SE" indicates semantic error, "GE" indicates grounding erro.
Error sources
Count.
Exist.
Atrri
Spatial.
Refer
Navi.
Others
Overall
SE + GE GE
47.9
82.1
49.6
49.8
31.9
51.6
80.3
55.6
73.3
86,5
63.3
49.9
67.2
55.4
81.8
64.9
98.8
100.0
60.1
55.7
84.9
87.2
86.8
78.1
How many doors at your o'dlokk
Where the object for displaying output from computer? Provide clockwise direction.
How You reach the plano?
Think type: This is courting question, so need to
Think type This is a navigation question, so reed to gmamne the cornesaanding objects first. Think rgn: List al the stjects in the soene.
Think type: This is nevigation queetion, ieed
ground the correponndnng objects to answar
ground the consspondingg objecta first
Think rgn: List all the objects at 2 clcioo.
Think List the objects
Think ard Ground the object: The doors at my 2
Think ard Ground the object: The ubject used for displaying outpur from computer.
Think_grt Ground the otject: The
o'cockk You should find all the poss ble ectects.
Think task: Now need
locations
Think task: Now rreed to list the patential
Think task: Now need to ist all the locations and probabilles iile the potential objects.
probabi lies the potential ebeets.
objects and the protebility
Obj
planos 1588.1,
Obj prob: doors 0.75 doors 0.60 stack of chairs
Ohj loc. pir prob: monitor -18.9, 3.3; prob: 0.50
lamp: 63.4, prob: 0.14
0.31
monior 131.. prob: 0.51
Obj beg:
prob:
Ob) prob (GT): door: 0.50 door. 0.55 chair: 0.33 C.01%_.....
Obj loc ____boooo]]: monior--11 -118 3:4: prob:
prots (..)
0.57 chair. 175.8, 2.1: prob: 0.49
Think sum Now answer queston based on
Think sum: Now answer the quastton based on the
Think Now
the question based ooject prosobilties.
object probabil tios.
object probabiliies
Answer: tum to your right and short distanoe.
Answer: two Answer(GT)::w
dlclack
AnswerrGGT tur wound and walk the middle distance.
o'dlock
Figure 6: Visualization of qualitative examples of SCENECOT. We select two indicating sub-tasks Counting and Navigation to illustrate the reasoning traces of SCENECOT. SCEN:CT correctly constructs the visual clue and reasons the correct answer. Middle: SCENECOT correctly answers the question based on the accurate relative location. Right: Even though the visual clue exactly matches the correct entity, the model summarizes the wrong answer owing to the limited reasoning capability.
semantics directly translate into better answers, demonstrating strong grounding-Q2 coherence in our framework. Spatial Relationship remains natural descriptions. Overall, the analysis shows that improving label, mask, and grounding quality is key to advancing 3D reasoning.
the hardest task, as requires mapping coordinates to
5.4 ADDITIONAL EVALUATION OF GROUNDING PERFORMANCE
To more thoroughly evaluate the grounded aspect of our 3D-CoTs reasoning, consider both traditional grounding benchmarks and QA-driven grounding benchmarks. As shown in Tab. 4, we incorporate multiple benchmarks to comprehensively evaluate performance in both grounding-only settings (Nr3D,, Beacon3I) and QA-driven grounding tasks (MSQA, SQA3D, ScanQA). For the latter, the evaluation measures the model's ability to localize the specific object instances required to answer a question (e.g., grounding only the specific chairs to the agent's 's right). We provide the details of the benchmarks in Sec. C.3.
The results show that SCENECOT achieves strong and balanced grounding perfor- mance across all benchmarks. It performs competitively on Nr3D and Beacon3D, and it significantly surpasses prior methods on MSQA, SQA3D, and QA grounding. Notably, our model is not fine-tuned on SQA3D or ScanQA; thus, the strong zero- shot grounding performance highlights the effectiveness and generalizability of our approach. These high-quality grounding outputs provide clear and reliable visual cues that support the downstream reasoning chains.
Table 4: Evaluation of grounding tasks,
"B3D" denotes
"Beacon3D" in the table.
Nr3D
B3D
MSQA
SQA3D
ScanQA F1
Method
Top-1
Top-1
F1@50
F1
Chat-Scene PQ3D
39
62.7
15.9
25.9
35.9
66
76
10.6
10.8
19.2
SceneCOT 57.7
67.8
52.1
51.6
40.8

---

Published as a conference paper at ICLR 2026
In contrast, while baseline models like PQ3D demonstrate strong results traditional benchmarks like Nr3D, they perform poorly on QA-driven grounding tasks. This reveals limitation in their ability to follow complex instructions in reasoning scenarios. Chat-Seene shows relatively better grounding on SQA3D and ScanQA but still lags behind SCENECOT across the full spectrum of tasks. Overall, this evaluation demonstrates that framework is built upon robust grounding capabilities for both spatial localization and reasoning-oriented tasks.
5.5 REASONING CHAIN VISUALIZATION
Finally, we present a case study in Fig. illustrate the strengths and limitations of our framework. In the first example, the model grounds the doors at the agent's 2 o c'lock and constructs the corre- sponding visual clue jbj_prob. Although the predicted labels differ slightly from the ground-trute "door," the accurate probabilities and semantic similarity allow the model to generate the correct answer. the second example, the model identifies all monitors and uses the symbolic engine to compute their relative positions with 2D polar coordinates. By integrating spatial information with object probabilities, the model infers the answer "4 o' c'llok."' These two cases demonstrate interpretable reasoning chains for challenging tasks, making error sources easier diagnose. In the third example, however, the model fails in the final reasoning step despite an accurate visual clue, echoing the upper-bound analysis in Tab. where Navigation performance peaks 87.2. This highlights a gap due to limited reasoning capability in the base model, which could be improved as foundation models like multimodal LLM advance. Overall, these visualizations showcase both the progress and remaining challenges of SCENECOT, pointing toward more robust and interpretable 3D reasoning frameworks.
CONCLUSION
presented SCENECOT, a framework step-by-step grounded reasoning 3D scenes. Unlike prior approaches that treat 3D question answering single-step task, SCENECOT decomposes reasoning into task recognition, region localization, entity grounding, and grounded reasoning, closely reflecting the human problem-solving process. To enable this, we built SCENECOT-1854 the first large-scale dataset of grounded Mhaii--ffTThuuee reasoning traces with rich stepwise annotations linking language and 3D context. Experiments show that SCENECOT achieves strong performance on both situated reasoning benchmarks and the Beacon3D benchmark, where it significantly improves grounding-OU coherence compared to object-centric baselines. Beyond accuracy, our method generates interpretable reasoning traces that make the decision process transparent. These results demonstrate that step-by-step grounded reasoning both effective and necessary for robust 3D scene understanding, laying foundation for ad advancing multimodal LLMs toward human- reasoning in real-world 3D environments.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
In this paper, we propose a new reasoning framework together with large-scale dataset. We present the implementation details of the model design and training in both the main paper and the appendix. In Sec. 3.1, we define the different types of thoughts based question categories. In Sec. 3.2, we describe how the modular components are trained and the pre-trained models we adopt. Hyper- parameters used for training and evaluation are provided in Sec. A.5. Additional details include the definition of the symbolic engine Sec. A.2, the visual clue construction algorithm in Sec. A.3, and the design of the 3D grounding module in Sec. A.3.
For data generation, we outline the full pipeline, including metadata collection in Sec. 4.1 and reasoning trace generation in Sec. 4.2. To facilitate reproducibility, we also provide the detailed system prompt used in Sec. B.2.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. Our study focuses on improving 3D scene reasoning through step-by-step grounded reasoning. We use only publicly available 3D vasion-language datasets, none of which contain personal or sensitive information. No human subjects were involved this research.
Our method inherits biases and limitations from pre-trained models and datasets, such as poten- tial semantic or linguistic biases in object labels and answers. While framework improves interpretability and grounding-Q2 -A coherence, critical domains (e.g., autonomous navigation surveillance) without further safeguards. Code and processed data will be released to support transparency and reproducibility.
not intended for direct deployment in safety-
REFERENCES
Jean- nan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances Neural Information Processing Systems (NeurIPS), 2022.
Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities. Conference on Computer Vision and Pattern Recognition (CVPR), 2024a. 1
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint arXiv:2306. /5795 2023a. 3
Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d 3d understanding, reasoning, and planning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2024b.
Yilun Chen, Shuai Yang, Haifeng Huang, Tai Wang, Runsen Xu, Ruiyuan Lyu, Dahua Lin, and Jiangmiao Pang. Grounded 3d-llm with referent tokens. arXiv preprint arXiv:2405.10370 2024c.
1,2
Yixin Chen, Siyuan Huang, Tao Yuan, Siyuan Qi, Yixin Zhu, and Song-Chun Zhu. Holistic++ scene understanding: Single-view 3d holistic scene parsing and human pose estimation with human- object interaction and physical commonsense. In International Conference on Computer Vision (ICCV), 2019.
Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Zhiqing Sun, Dan Gutfreund, and Chuang Gan. Visual chain-of-though prompting for knowledge-hasse visual reasoning. In AAAI Conference on Artificial Intelligence (AAAI), 2024d. 3
https://iclr. http://rr....ppppooo//o/oooooooe

---

Published as a conference paper at ICLR 2026
Zhenyu Chen, Ronghang Hu, Xinlei Chen, Matthias Nießner, and Angel X Chang. Unit3d: A unified transformer for 3d dense captioning and visual grounding. In International Conference on Computer Vision (ICCV), 2023b. 1
Tao Chu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Qiong Liu, and Jiaqi Wang. Unified scene representation and reconstruction for 3d large language models. arXiv preprint arXiv:2404.13044 2024. 2
Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition. arXiv preprint 00Xi0:2501..32300, 2024. 3
Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. In Proceedings of Winter Conference on Applications of Computer Vision (WACV), 2025.
Ran Gong, Jiangyong Huang, Yizhou Zhao, Haoran Geng, Xiaofeng Gao, Qingyang Wu, Wensi Ai, Ziheng Zhou, Demetri Terzopoulos, Song-Chun Zhu, et al. Arnold: A benchmark for language- grounded task learning with continuous states in realistic 3d scenes. In International Conference on Computer Vision (ICCV), 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501. 12944 2025. 1,
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven CH Hoi. From images to textual prompts: Zero-shot vqa with frozen large language models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lIm: Injecting the 3d world into large language models. Advances Neural Information Processing Systems (NeurIPS), 2023. 2
Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. Advances Neural Information Processing Systems, 37:1 13991- 114017, 2024a. 6
Jiangyong Huang, William Yicheng Zhu, Baoxiong Jia, Zan Wang, Xiaojian Ma, Qing Li, and Siyuan Huang. Perceive, ground, reason, and act: A benchmark for arXiv preprint arXiv:2211.15402 2022. 3
visual representation.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In International Conference on Machine Learning (ICML), 2024b. 1,2
Jiangyong Huang, Baoxiong Jia, Yan Wang, Ziyu Zhu, Xiongkun Linghu, Qing Li, Song-Chun Zhu, and Siyuan Huang. Unveiling the mist over 3d vision-language understanding: Object-centric evaluation with chain-of-analysis. In Conference on Computer Vision and Pattern Recognition (CVPR), 2025a. 2, 6
Jiangyong Huang, Xiaojian Ma. Xiongkun Linghu, Yue Fan, Junchao He, Wenxin Tan, Qing Li, Song-Chun Zhu, Yixin Chen, Baoxiong Jia, et al. Leo-vl: Efficient scene representation for scalable 3d vision-language learning. arXiv preprint arXiv:2506.09935 2025b. 2
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, al. Openai ol system card. arXiv preprint arXiv:24/2.16773 16720, 2024. 1,3
Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision- language learning for grounded scene understanding. arXiv preprint arXiv:2401.099 009340, 2024. 1, 6, 7,
12

---

Published as a conference paper at ICLR 2026
Bernhard Kerbl, Georgios Kopanas, Thomas Leimküihler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 2023.
Xiongkun Linghu, Jiangyong Huang, Xuesong Niu, Xiaojian Shawn Ma, Baoxiong Jia, and Siyuan Huang. Multi-modal situated reasoning in 3d scenes. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 1, 2, 6
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint
arXiv:2412.19437, 2024. 19
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS), 2023. 2
Yuecheng Liu, Dafeng Chi, Shiguang Wu, Zhanguang Zhang, Yaochen Hu, Lingfeng Zhang, Yingxue Zhang, Shuang Wu, Tongtong Cao, Guowei Huang, al. Spatialcot: Advancing spatial reasoning through coordinate alignment and chain-of faiin-of-ther of-thhuuht for embodied task planning. arXiv preprint arXiv:2501. 10074, 2025. 1
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. In International Conference on Learning Representations (ICLR), 2023. 1
OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2003.08774, 2023. 6
OpenAL. Introducing openai 03 and 04-mini. https: / opppoop com/iiex/// index/ introducing-on and- 04 -inni/ April 16 2025. 3, 26
Jianjiu Ou,
Jianlong Zhou, Yifei Dong, and Fang Chen. Chain of thought prompting in vision-language model for vision reasoning tasks. In Australasian Joint Conference on Artificial Intelligence, 2024.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023. 3
Zhangyang Qi. Ye Fang, Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang, Dahua Lin, and Hengshuang Zhao. Gpt4point: A unified framework for point-language understanding and generation. arXiv preprint arXiv:2312.02990 2023. 2
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe.
Mask3d for 3d semantic instance segmentation. arXiv preprint arXiv:2210.03105 2022. 5
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of-thouuht reasoning in multi-modal language 2024a. 1,
models. arXiv e-prints, pp.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03.85 2024b. 3
Chan Hee Song, Valts Blukis, Jonathan Tremblay, Stephen Tyree, Yu Su, and Stan Birchfield. Robospatial: Teaching spatial understanding 2d and 3d vision- language models for robotics. In Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 1
Anh Thai, Songyou Peng, Kyle Genova, Leonidas Guibas, and Thomas Funkhouser. Splattalk: 3d vqa with gaussian splatting. arXiv preprint arXiv:2503.0627 06271, 2025, 6
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-448311 2022. 1,3
Penghao Wu and Saining Xie. V?: Guided visual search In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13084-13099 2024. 3
mechanism in multimodal llms.
13

---

Published as a conference paper at ICLR 2026
Runsen Xu, Xiaolong Wang, Tai Wang. Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. arXiv preprint arXiv:2308. 16911, 2023. 2
Jianing Yang, Xuweiyi Chen, Nikhil Madaan, Madhavan Iyengar, Shengyi Qian, David F Fouhey, and Joyce Chai. 3d-grand: A million-scale dataset for 3d-llms with better grounding and less hallucination. In Conference on Computer Vision and Pattern Recognition (CVPR), 2025a.
Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Conference on Computer Vision and Pattern Recognition (CVPR), 2025b.
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-Ilava: Bridging image-level, object-level, pixel-level reasoning and understanding. Advances in Neural Information Processing Systems, 37:71737-71767, 2024a. 3
Zhuofan Zhang, Ziyu Zhu, Pengxiang Li, Tengyu Liu, Xiaojian Ma. Yixin Chen, Baoxiong Jia, Siyuan Huang, and Qing Li. Task-oriented sequential grounding in 3d scenes. arXiv preprint arXiv:2408. 04034, 2024b. 25
Zhuofan Zhang, Ziyu Zhu, Pengxiang Li, Tengyu Liu, Xiaojian Ma, Yixin Chen, Baoxiong Jia, Siyuan Huang, and Qing Li. Task-oriented sequential grounding in 3d scenes. arXiv preprint arXiv:2408. 04034, 2024c.
Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d Ilm: Learning position-aware video represen- tation for 3d scene understanding. arXiv preprint arXiv:2412.00493, 2024. 2
Junsheng Zhou, Jinsheng Wang, Baorui Ma. Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. International Conference on Learning Representations (ICIR), 2024.
Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang. and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering imms with ad-aweressss. arXiv preprint arXiv:2409.18125 2024a. 2
Ziyu Zhu, Zhuofan Zhang, Xiaojian Ma, Xuesong Niu, Yixin Chen, Baoxiong Jia, Zhidong Deng, Siyuan Huang, and Qing Li. Unifying 3d vision-languag understanding via promptable queries. In European Conference on Computer Vision, pp. 188 2882006.22 Springer. 2024b. 1,
14

---

Published as a conference paper at ICLR 2026
A SCENECOT FRAMEWORK
A.1 DATA EXAMPLES OF REASONING TRACES
We provide typical examples of COT templates of some representative sub-tasks, including the integral special tokens and textual context.
Question: You are climbing the shelf. There a black and white monitor and white and black monitor your 5 o' clock. There is brown door at your 2 clook. How many pillows are on your right in the middle distance?
Response:
<think_type>Thine is counting question, so need to ground the corresponding objects to answer itt://hink_kyt ttpe>
<grd_rgma
<think_rgn>>Non I need list all the objects on my right in the middle distance</think rggn>
<think grd>Ground
<think_grbGGround the object: The pillows on my right in the middle distance. You should find all the possible objects..<thiine </think grd>
[OBJ]
<think Shhink_task>Now need to list all the potential objects and the probability.</hhink..ee
<list <list_obj bis__oj__rrbb>
<obj_ bobj_prob>pillow: 0.74 pillow: 0.78 cabinet: 0.31 legs: 0.01 cabinet: 0.29 pillow: 0.14
pillow: 0.40 file cabinet: 0.31 wall: 0.04 whiteboard: 0.45</obj_pruu prob>>
suink__smm>Nw answer the question based on the object probabilities.</hinkk sum> <answerttw<<answee>
Figure 7: COT example of Counting.
Question: You are painting the door. There is a brown trash can in your 9 cloock direction. There is turquoise table with smooth texture and four-legged structure in your 6 o' clock direction. there a sink in the room?
Response:
<think_type>Thnn Type>This is an existence question, so need to ground the corresponding objects to answer it.</think_type>
<grd_rgna
<think_rghh jrgnkkow_gNNNww need to list all the objects the s/hnkk</think rgn>
<think_grdee grai:kunddGrround the object: The sinks in the room.</think_rrds [OBJ]
<think task>NNww I need to list all the potential objects and the probability.</think task
<list_objj_poot
<obj_prob>sink: 0.95 traffic cone: 0.24 folded table: 0.06 ladder: 0.20 trash can: 0.45
vacuum cleaner: 0.35 ladder: 0.19 ledge: 0.14 table: 0.43 floor: 0.23 recycling bin: 0.13
recycling bin: 0.05 rack: 0.11 folded table: 0.08 boxes: 0.46 trash 0.40 trash 0.36
folded table: 0.16 wall: 0.28 door: 0.48 doorframe: 0.37 wall: 0.28 sink: 0.04 table: 0.15
wall: 0.18 wall: 0.38 trash can: 0.26 wall: 0.32 wall: 0.18 wall: 0.04</obi_prob>
<think suw>Now answer the question based on the object probabilities. </think sum> <unsweryyss<annswr>>
Figure 8: A COT example of Existence.
15

---

Published as a conference paper at ICLR 2026
Question: There is window in your 7 o'clock direction. There is black monitor with smooth texture in your 3 o' clock direction. What object is brown and used for seating at your 0''clock'
Response:
<think_ Biiiink_yyp>Thhiss is refer question, so I need to ground the corresponding objects to answer it.</think_tyy _ttpe
<grd_rgn>
<think_rgn>Now need to list all the objects my 6 o' c'cllockk//tiin b/bbckk./think_rn>>>
<think grd>Ground the object: The brown object used for seating at your 6
o' o'clock.</hhink _gd>>
[OBJ]
townkkowtskNowwww/ need to list all the potential objects and the probability.</think task <list_obj_prob>
cobj_prob>chair: 0.98 wall: 0.30 floor: 0.33 chair: 0.48 chair: 0.44 table: 0.10 window: 0.26 backpack: 0,40</obj_prrob
<think sum>Now answer the question based on the object probabilities.</hhink sum> <answer>the chair</answer>
Figure A COT example of Refer.
Question: You are replacing the trash bag in the trash can. front you, there's a beige wooden door. To your left, at middle distance, there's dark rectangular suitcase with hard shell. What is the state the door on your right?
Response:
<think_typee LypettyyThhtt This is an attribute question, so need to ground the corresponding objects to answer ht///hhin_ttpeee
<grd_rgh _gn_rrgn>___
<think rgn>>Nw I need to list all the objects in the sub-region related the ques-
tion.< boon.</hinkkrgnn
the object: The door on my ggght.</thin_ggrd
[OBJ]
<think task>Now <think_task<Now need to retrieve the image the target object.</think task> <highlight obj>
answer the question based on the object image. foaggee//thikssuum sum> <anwerrclossek<ansserr>
Figure COT example Attrihute.
A.2 DEFINITION OF THE SYMBOLIC ENGINE
Our symbolic engine serves two primary functions:
1. Spatial Region Recognition: It partitions the environment into sub-regions based on the agent's location and orientation.
2. Visual Clue Construction: It generates textual visual clues by integrating object probabili- ties, semantic labels, and spatial locations.
A.2.1 SPATIAL REGION RECOGNITION
In MSQA, there are two types of directional reference policies: 1) Cardinal-Relative Directions, such as left, right, front, and behind. 2) Clock-hased directions, such as to 12 o' clock positions. Here we provide some code fragments our implementation.
1. Calculate the cardinal-relative sub-regions. This function is used to calculate the object lists of four directions. Since MSQA also requires the distance information in some scenarios, we also develop the function to support this feature. Calculate the clock- based sub-regions. This function is used to calculate the clockwise information for each object.
16

---

Published as a conference paper at ICLR 2026
You sitting the desk. Behind you, there's burgundy backpack. From where You are, how do You get to the door? Please provide direction (bchind, in front of, left, right) and the distance information (far/middleenneat).
Question:
Response:
<think _ainnkkttyeTTTiss is navigation question, so need ground the corresponding objects
first.<think_We
<grd_rgnz
I need to list all the objects in the scene.</think_rg
<think grd>Ground the object: The door.</think. grd> <think_gro>Grrundd [OBJ]
<think_tasksln task>Now I need to list all the locations(polar coordinate: angle, distance(/m)) and probabilities of the potential objects.</thhee </think task>
1, 1.9; prob: 0.79 wall: -117.6, 2.7: prob: 0.01 window: 154.2, 2.0; prob: 0.25 fan: 152.1, 1.6; prob: 0.37 clothes: 96.4, 3.4; prob: 0.41<//bjjlo< le____o___rrp_pbb>> <think sum>Now answer the question based on the object probabilities.<<thin sum>>>> <answer>turn right and walk to the middle </answer//
Figure 11: A COT example of Navigation.
Question: You are hanging the picture. There is blue tissue box at your 10 clock direction, with a smooth cardboard texture, dispensing tissues. A cluttered dark table is at your 7 o' d'clock direction. Where the book located in relation to the desk?
Response:
a spatial relationship question, so I need to ground the corresponding objects first.</hink_tyype type>
tthink_type>1
<grd_rgn>
<think_ Brannk_wgg>Nwww/// I need list all the objects in the </think/thikk... rgm>
<think
<think_grd>Ground grd>Ground the object: The book and the </thikk -grbinn_-dd>>>> grd>
[OBJ]
Shiin_task><Now need to list all the locations and probabilities of the potential ob-
jects.</think_tast
<list_ Best_oob_ooc_poobee
sobj_loc_plr wttpplrroobbooo:::: -14.000.111.....00000000300001 prob: 0.89 desk: -09,5.0,005511777..00000
prob: 0.59 yoga mat: 2.5.44...077006660222.022
prob: 0.46 shelf: 112.113..1.00660000222111
prob: 0.05 sink: 3.0,2.6.08,,0..400555, prob: 0.06 bag: 1.5.2.88,011,06,,03330000 prob: 0.14
book: 11664.88..8,0.330033.3 refrigerator: 088.4,6,09.110000..4... prob: 0.11 lamp base: lale::-l.ee -14.2222,000002220011111 prob:
prob: 0.17 kitchen island: 2.7.2.6,0.4,1.2211..88 prob: 0.36
0.33</obj_le loc Lhr_pplppp_______P prob>
<think sum>Now answer the question based on the object probabilities.</hhin scchink_sum>>>>> <answer>the book is placed within the area of the </ans</a
Figure 12: A COT example of Spatial Relationship.
We parse the text between <think_ and //think_rgns to obtain the user's directional instruction, then set query_type by matching it known phrases like "on my left" "at my 10 o' "'oock".
A.2.2 VISUAL CLUE CONSTRUCTION
In the Grounded Reasoning step, the model generates the final answer by integrating both semantic information and a visual clue derived from object probabilities. We outline the algorithm for con- structing this visual clue during the reasoning process. Given prefix sequence P. (i.e., the input_ids _ds of the text prompt), the algorithm returns a modified sequence that incorporates relevant visual clues based object probabilities, spatial locations, and semantic labels. This enhanced sequence is then fed back into the LLM to generate the final response. The component functions -uuldduid obj__oroj_ build_ob sabj_loc_pon and build_ohe _oj__oooo______o Loc_ppr__rooteee are illustrated in Fig. 16 and Fig. 3. Additionally,
17

---

Published as a conference paper at ICLR 2026
Question: What color the printer top of the nightstand?
Response:
<think_ taannd__tpeTThsss is appearance (grounded qa) question, so need ground the corre- sponding objects to answer it. it..//tinn_tyy/ __tpp>> <grd_rgn>
<think_rgee Brannk_rgnNNow//// I need to list all the objects in the scene. cereee//thinhggggg rgn>
athink_grd>Groum the object: The printer on top of the gightstand.</thinng grd> [OBJ]
<think task>Now I need to retrieve the image of the target object.</think, task>
<highlight _bj>>
sum>Now answer the question based on the object image.</think_sum>
<answer>Blak<<ansswer>
Figure COT example GQA3D.
calculate the sub-graphs for Cardinal-Relatinn Directions
face_pt arrarray [np. (directien), (p.sin(directien))
stand_pt ar.aara (sttta_____ad_) stand_on_ioc([]]] pads - np.array (pcds) pod_2c pods[:, :2]
10
pod 2d pcd_
stand pt
11
pod _d__no___ norm = np. ap. wiw.lg.nonommmmmmm...om......... pod_2á_norm (ecpormppp____d_emee2_ee
axis=l)
== 0]
norrm np. expand dims (ped norm, axis=1)
pcd_2d 2dd_ 2dd_2d
15
16
### cal the angle between face_pt and 2dd_2d sum_all np. dpt (pcd_2dd))) face_pt)
sum all Isum_all >= 1| =
sum alll [all|sum_all
-11
20
angle np.arccos tsum al1)
21
angle angle / np.pi 180
IH cal the cross value between face_pt and pod_2d cross np crosr (face_pt,, 2dd_2d)
24
25
front_mas
(angle < 30)
back_mask (angle 150)
28
left_mask (cross
& (angle 30) & (angle 150)
29
right_mask (cross < 0) & (angle 30) & (angle 150)
list met_inst_iddinst__aate back_lis 1ist get_inst_iddinst__n
front_mask) back_mask)
left_list get get_inst_iddinst__
1eft_mask)
34
right list get inst id label right right_mask)
Fig. 14 provides visual comparison between the 2D polar coordinate system and the 3D spatial coordinate system used our framework.
A.3 IMPLEMENTATION DETAILS OF THE 3D VISUAL GROUNDING MODULE
We illustrate the implementation of 3D Visual Grounding Module in Fig. 15. In the framework. the object feature is extracted by PQ3D, while the text embedding is extracted by the embedding tokenizer based on the grounding text. We train PQ3D based on the grounding data Jia al. (2024), including the scenes in ScanNet, 3RSean, and MultiScan.
18

---

Published as a conference paper at ICLR 2026
def cal cwwcklooccooooooo wise direction (stand on pt, face_ pt, inst inst_ped) inst_ped = inst_ped[;;;2]]
inst_lec (inst_pod. max (0) inst_pod min (0))
inst_lcc inst_pod.mean (0)
obj_dir inst_loc stand on pt
if np.abs (obi_dir) sum () 1e-6:
return None, None
obj obj_dir np. np. np.linalg. norm face_pt pace_pt
no..inalg.nerm//e (face_pt)
11
12
+4 cal the angle between obj_dir and face_dir
angle dp. (obj_dir, face_pt)
angle op.clip(angle, -l, 1)
15
angle ap.arccos (angle)
16
direct np.cross (face_ obi_dir)
if direct 0: angle 2 np.pi angle
angle angle np.pi 180
20
21
clockvise_direction = round (angle / 30) $ 12 clockwise dinretin = 12 if backwise_directiaee - 0 else elockwise_direatioo
23
clockwise_directice int (elockwise
24
25
return blockwise_directeee angle
query:
the object 1ist of: cardinal di direction-bar and clock-based query type: four direction clockwise or whole_scene
if query_type 'cardiaal'
query_type
'clockwise':
parse_words
I'left', 'right', 'front', 'back', 'behind'
quezy_ == 'cardinal' else [f" o' for i in range (12)
direction arrre_direttioonddittan (question, parse_words)
direction direction if direction ! 'bhiind' else 'back'
distance parse_direction_ddstte '1)
(question, [' ['far', 'middle', 'near
if direction:
obj_list, label_ids = get_cbi_ wist (query, query_type, direction, distance)
count_dict Counter (obi_list)
11
else:
12
no direction found, cellect objects in all directions obj_list, label_ids [], [] distance
rencicidsttteettttttttee (qu_disttncequuestto ['far', 'middle',
near'l)
15
for direction in query.keys
obj
label_ _em_ get get_obi list (query,
query_type, direction, distance)
obi_list obj_list_temp
18
label_ids +- label_ids_temt
count_dict Counter [obt_riobb]isst
else:
obj_list, label_ids get_obj_list (query, query_type, " (1) count_dict Counter (obj_list)
22
A.4 INFERENCE
Recently, several state-of-the-art industrial LLMs such as DeepSeek- V3 Liu et al. (2024) have adopted the Mfxture-of-Expee -xperts (MoE) technique key gerformance-enhaccin strategy. Inspired by this insight, we implement a minimal routing mechanism during the inference stage. Empirically, observe that the model's performance several sub-tasks such Existence and Auribute sub-
19

---

Published as a conference paper at ICLR 2026
Algorithm 1: Visual Clue Construction for Grounded Reasoning
Require
:object probabilities P E RNx1 prefix sequence S RN*x object locations and sizes Og RNx object semantic labels OSL (N 1), object images O x3x H xW .maximmm object number K
TOreo__TOENN____ IDX in S then
content +
Or., OSt. K);
index - LISTOOO_ PROB_TOKEN, new www_sequenceeeee cntS::indee get_ggettt__teee text text_teeeeee______eee__eeeeeeeee imbeeddiggx(cbi_proo.... else if LIST_ ______0___PRO__L_00_____________ {dt_______________o__.0__.__1_11 IDX in S then
QBJ__________________ PRBB__RRBB___ TOKEN IDX);
jaj_prob_lloc_conteett== tbe prob(P, Or. OSL. K); index - i www.sppueneee-cuuttiiiieeeee=]
if oet.iiii.asaaoooo.........P....1 IDX in S then
bhuid prohpp___rr_pooh(P_.. Oz, Osz. K);
index + ingese_e_haaaiiissssc..aaaapppp_ new -
else if IECCTIGGGCCTTTCCC_TTTTTEEEE IDX in S then
top_ Top((P)
imq - 1
ain__okens=-i - Peoossssssssoooooooooonooooooonn
index +- HICG
new_sequences + int(s]:
get_text_cent http_//bbbdiings1n__SSAaa0 -MAAIIARIIIISSTAAIIAAI-A TOKEN IDX), img_tokens,
get_text_ Ltt.aaaccmcdiingX___B___DLOLO
return new ses_ueceeeeeeeeeeeeeeeeee
desk (0.6, 1.1, 0.5, 0.6,0.8,
mirror (45, 1.2)
y1
chair (0.5 0.6, 0.3, 0.3,0. 4, 0.4)
x1
21
Agent
Agent
2D polar coordinate
3D rectangle coordinate
Figure 14: Coordinate system comparison. We provide diagram of the two types of coordinate systems.
tasks tends degrade over time, while performance other tasks improves. Motivated by this training dynamic, we introduce a simple two-expert selection strategy.
Specifically, the model with the highest overall validation performance is designated as Expert-1, while the model with the best validation performance on partial sub-tasks is designated as Expert-2. During inference, if the predicted question type comes from the ones that have gradual decreasing trend in validation performance, the input prefix sequence is routed to Expert-2; otherwise, it is routed to Expert-1. This strategy feasibly utilizes the result of task recognition in the first reasoning step. This strategy is only applied for MSQA in evaluation. Balancing the training dynamics for different sub-tasks should be an important direction for future work.
As apply LoRA to the LLM during training, this inference strategy introduces only additional 330M parameters relative to a single base model, representing reasonable trade-off between performance and deployment cost.
20

---

Published as a conference paper at ICLR 2026
Concatenate
Linear projection
Sigmoid
Grounding head
Text head
Object feature
Text embedding
Object probabilities
Figure 15: 3D Visual Grounding Module Design.
Table 5: Hyperparameters for training.
Hyperparameter
Value
Optimizer
AdamW
Weight Decay
0.05
betas
[0.9, 0.999]
Learning Rate
3 10-5
Warmup Steps
400
Type of GPUs
NVIDIA A100
Number of GPUs
4
Accumulate Gradient Batches
5
Batch Size/GPU (total)
(80)
gradient norm epochs
5.0
Table 6: Hyperparameters for inference.
Hyperparameter
Value
Number of beams
5
maximum output length
256
repetition penalty
length penalty
1.0
A.5 HYPERPARAMETERS FOR MODEL TRAINING AND INFERENCE
We train the base LLM of SCENECOT in single stage, initializing from the pretrained weights of VA- The detailed hyperparameter settings are provided Tab. and Tab. 6.
B SCENECOT-185K DATASET
B. DATA GENERATION PIPELINE OF SCENECOT-185K
Our data generation pipeline shared between Object- Reasoning and Situated Reasoning, with Object-Centric Reasoning formulated Reasoning framework. The overall process is illustrated in Fig. 16.
Attriute sub-task under the broader Situated
B.2 DATA GENERATION DETAILS AND EXAMPLES OF GQA3D
To enable the training of Object-Centric Reasoning non-situated scenarios, we incorporate GQA3D as key component of the overall dataset. The primary distinction from Situated Reasoning lies in the nature of the grounding text, which describes the target object from global perspective rather than an egocentric view. Additionally, this task does not involve multi-object grounding reasoning. We use GPT-4o generate diverse question-answer pairs based on given object image and its corresponding grounding text. We provide the system prompt in Fig. 17.
B.3 DATA QUALITY
We conducted manual quality check to verify the data quality. First, manually checked the object IDs in MSQA. Note that all object IDs outside the scene were filtered out in our dataset. Since MSQA is based situated scene graphs, all objects in QA pairs correspond specific sub-regions,
21

---

Published as a conference paper at ICLR 2026
Direction parsing
table-1. table-5
How many brown tables are in my right?
right
Region object
table-1. choir-3, table -5, carpet-9
Question type: Counting
selection
<think_ bypnkk-s is counting question, so need to ground the corresponding objects to answer it </think__ <<think_btppe>>> type> <grd rgn>
<think_ <think_rgn=-Ww rgn=Now need to listall the objects on my right the middle distance=think, rgn>
Shiink_grdsGnnon the object The pillows on my rightin the middle distance. You should find all the possible objects.</think, </think_grdds [QB]]
<think task>Now
to list potential objects and the probability </think tasks
<ist obj_ prob>
<obi_ probstable 0.68 table chair 0.42 carpet0.23 poob_probb>
<think sum=Now answer the question based on the object probabilites </thnkk sum <nnser>kwo-<answer>
Figure 16: Data Generation Procedure of Counting.
You are expert visual question-answer pair generator.
You will receive: An image showing part of a scene. An object list that names all the objects supposed to be present in the scene. Several grounding texts of the target object, which refers to specific object that appears in the image. Your task is to generate multiple high-quality QA Qquestion-Answryy pairs based on the image and the object list, focusing on the target object. The QA pairs should be diverse and cover five categories: existence, appearance, geometry, spatial relationship, and class. Follow these detailed guidelines:
Existence Questions Generate both yes and no questions. For questions with the answer "Yes": refer to objects that are adjacent to the target object in the image. For questions with the answer "No": refer to objects listed in the object list but absent in the image.
2. Spatial Relationship Questions Ask about the spatial relations involving the target object (e.g.. "next to," "above," "below," "in front of," "close to," "adjacent to" etc. Please do not use the directional words like 'left' or "right'). Questions should describe how the target object is positioned relative to nearby objects.
3. Class Questions Ask about the category or type of objects adjacent to the target object. Examples: "What object is next to the target object?" or "What is under the target object?" Geometry Questions Ask about the shape and size comparison of the target object. Examples: "What is the shape of the target object?" or "Which is taller, the target object or another nearby object?"
5. Appearance Questions Ask about visible attributes like color, texture, or material of the target object.
Important Notes:
Base all questions and answers on the visual content of the image and the object list. For questions with the answer Yes', make sure they visually supported by the image. Ensure that every QA pair is grounded and accurate to the given visual context.
Always specify the question type (existence, appearance, geometry, spatial relationship, class) when you output a QA pair. Use the target object description naturally inside the questions to avoid ambiguity. Do not ask the questions that have been mentioned in the grounding text. You can choose the objects in the object list. While all the answers of the questions are based on the given image.
Output format: Q: [question] A: [answer] type: [type]
Figure 17: System Prompt for QA Pairs Generation in GQA3D.
as ensured by the MSQA paper. We randomly sampled 50 examples each from counting, spatial relationship, refer, attribute, and navigation tasks, totaling 250 examples. Annotators were given the question, answer, and object IDs to verify if the IDs align with the QA pairs. For example, if the question is "How many chairs are on my left?" with the answer "three," " but the target object IDs are ["chair-1", "chair-3"J. then the annotation is incorrect. The results are shown in Tab. 8. The results demonstrate the reliability of our annotation process. Next, verify the quality of the generated QA pairs in GQA3D. GQA3D uses Nr3D its data source to create QA pairs with reasoning traces. The grounding text is mostly suman-annotater ensuring data quality. We generated new QA pairs based object images using GPT-4V To ensure
22

---

Published as a conference paper at ICLR 2026
Table 7: Metadata Examples of GQA3D. GQA3D constructs QA pairs using the grounding text from Nr3D.
The QA pairs are generated from the image of target object.
Object Image
Grounding Text
QA pairs
Q: What material is 'the tallest white
box' made of?
A: Cardboard
type: appearance
Q: Is 'the tall box the left of the
the tallest white box, the tall box to the left of the 2 boxes sitting top of cach other, the tall white box that not stacked
boxes sitting on top each other'
the same color the door?
A: No
type: existence
Q: Is there fan next to 'the tall white box that is not stacked'?
A: No
type: existence
Table 8: Data quality of MSQA. We check the accuracy of target object IDs.
counting
spatial
refer
attribute
navigation
sampled instances
50
50
50
50
50 100%
accuracy
92%
100%
98%
100%
Table Data quality of GQA3D. sample some data instances and check the answer correctness.
# sampled instances # sampled objects
accuracy
100
11
90%
quality, we randomly sampled 100 QA pairs and manually verified their correctness, providing both the object images and QA pairs for review. The results are shown in Tab. 9. The results confirm the reliability of our data. The strong performance of Chat-Scene and SceneCOT further highlights the value of our dataset.
C ADDITIONAL EXPERIMENTS AND ANALYSES
C.I ADDITIONAL. BASELINE COMPARISON
In our main results, we focus on object-centric methods such as Chat-Scene, LEO, and MSR3D to clearly analyze both QA performance and grounding- -A coherence. To broaden the comparison, also include LLa VA-3D as a strong roxel-based bascline. As shown in Tab. 10, SCENECOT achieves both Situated Reasoning and Object-Centric Reasoning. An interesting
competitive performance direction for future work is integrating SCENECOT with voxel-based approaches.
C.2 ADDITIONAL. EVALUATION AND BASELINE COMPARISON
We assess our model on SQA3D and ScanQA under zero-shot settings and compare it with Chat- Scene and LEO. Both Chat-Scene and LEO fine-tuned SceneCOT-QA: specifically, LEO is fine-tuned on the SceneCOT-QA dataset, while Chat-Scene is fine-tuned on the SceneCOT-grounded QA variant, where the model must first predict the target object IDs before generating the answer. The results are summarized in Table 11.
We evaluate both QA and grounding performance. The results indicate that our QA performance is comparable Chat-Scene, while our grounding performance is substantially stronger. This reinforces our claim that although SceneCOT does not rely on scene tokens, it explicitly grounds objects before answering. Since answers must be inferred solely from grounded results, the model demonstrates strong grounding-QU coherence, whereas ChaatSScene shows weak coherence between its QA performance and grounding performance.
C.3 DETAILS OF QA-DRIVEN GROUNDING BENCHMARKS
23

---

Published as a conference paper at ICLR 2026
Table 10: Additional baseline on MSQA and Beacon3D.
MSQA
Beacon3D
Methods
Data
Grounded?
Count.
Exist.
Attr:
Spatial
Navi.
Others
Overall
Case
Obj.
LEO
ours
X
29.3
87.5
46.3
45.6
81.4
52.9
52.5
12.7
MSR3D
ours
X
32.7
87.5
53.7
44.3
51.5
72.3
52.8
51.4
11.9
Chat-Scene
ours
X
37.4
92.0
49.0
47.0
58.3
83.7
56.6
53.6
14.0
LLaVA-3D
X
38.5
89.5
57.0
49.8
38.5
84,4
54.9
59.1
21.0
SCENECOT
ours
47.7
82.1
49.6
47.2
51.6
80.3
55.6
58.9
23.2
Table 11: Zero-shot QA and grounding performance on SQA3D and ScanQA.
SQA3D
SQA3D-G
ScanQA (EM-R)
ScanQA-G (FI@50)
Method
(EM-R)
@F))
Chat-Scene
36.0
3.4"
24.5
4.1"
LEO
36.7
20.04
SceneCOT 39.7
51.6
21.0
40.8
Note on Chat-Scene: Despite following official instructions and using 8-GPU reproduction attempts, grounding performance remained low. Reports (c.g., GitHub Issue #38) suggest the model is highly sensitive to training seltings.
In Sec. C.2, we evaluate the grounding performance of the models on ScanQA and SQA3D. For ScanQA, we utilize the object instance IDs provided by the official annotations. For SQA3D, we built an agent workflow and chose the rules defined in Sec. A.2 to extract the question-answer related object IDs. We denote the corresponding data as "ScanQA-G" and "SQA3D-G". To guarantee the data quality, conducted manual quality verification. The results is shown in Tab. 12. The result reveals the reliability of SQA3D-G.
Table 12: SQA3D.
Data quality of
# Instance
Accuracy
33
93.3%
C.4 ADDITIONAL REASONING CHAIN VISUALIZATION RESULTS
You repairing the There hanging your o'tock, black with your atlock What object saritelion your a'ckoce)
To your right there's a clean white ceramic sink. Behind you. there's a white plastic switch... Where the tollet located in relation to the picture?
You are knocking on the door. in the back. at middle disiancee there's white plasic object for dispers paper lowel and metali silver tall for support. is there backpack in the room?
type: spottal mebitonsh ship question,
so need ground the corresonnding objects frsi.
question, need corresonndngg objects Think List the
Think This
Think type: This refer questinn, so need to ground the corrsponninn objects answer it.
Think rgn: Now meed to al the objecis the scene.
Think, Now need to all the objects
Think grd: Ground the object The toilet and the pictur.
Think Ground the ooject
the the potential
olbbck direction.
Think grd: Ground obeet The object for
Thank task: Now noed to attallln the locations and
Think tasi:: need
cantation o'dlock.
probobi the potental objects.
the probabilty.
Think
list potental
Objj loc, prob: pictures: 0.3.0.4 0,5.0.0...0...
Obj, prob: paper lowel dispenser 0.9 soep dispenser scap cispenser tovlet paper roll can
probabiliyy.
prob: 0.95 tollet 0.4.0.0......000 8.0.0.8,00333 5.0.8 prob: 0.94
0.17 window Obj_prob: pillos barkpack 0.00
wille -0. 4,0.4.4 A.C.B.B....... 8,0.8 prob: 0.94 tolle:-
cap bar
bockpack0.00
Daj loc. probiGTI: bilet 0.
0.1.0.4,..4.... A.S.......0.. 2.8. prac: 0.54
Obi, probiGTi, GT) tackpack: mil: 05 tolket popen 0.31 soap object wall: paper disperisan 0.43 Think sum Now answer the question besed on object probabillis..
tolket paper dispenser 8.00
Obj_proh (GT):
0.45 tolle: 0.36
0.87 picture: 0.3. 4.1. 0...2.0. prob:
backpack: 0.29 wall:
bed: 0828 object piloow sllow.
0.70 tollet paper: 0.3. 1,0.5.0.111.
0.2.0. pmmb 0.27
dispensen rad:
3.09 0.40
sinic. -0.3.0 6,0...60..........33
prob: 0.32
04 minon.
sum: Now answor the queston based the object inob
Think, Now
the question based the object probisiitees
Answer: tollet
Answer tollet below the picture.
Answer: no.
Answer(GT): the tolle
AnswerlGT]: the below the picture.
Answeris Tig yes
Figure 18: Additional Reasoning Chain Visualization. We provide more visualization results on Refer(l
Spatial Relationship(Middle) and Existence(Right).
We present additional visualizations the reasoning chains across multiple sub-tasks to provide deeper understanding of our reasoning mechanism. In the first example, the model correctly
24

---

Published as a conference paper at ICLR 2026
identifies the target object based on the semantic grounding text ("The object used for sanitation") and subsequently arrives at the correct answer by leveraging accurate visual cues. In the second example, the grounding module successfully locates the target objects ("The toilet and the picture"), which enables the model to reason effectively using object coordinates. However, in another case, the grounding module fails identify the target object ("backpack"), resulting in incorrect answer. We also include a video demonstration to intuitively showcase the entire workflow.
D FURTHER ANALYSIS AND EFFICIENCY
D.1 INFERENCE LATENCY AND TRADE-OFFS
Our framework primarily focuses on enabling a reliable and transparent reasoning process, which requires incorporating all necessary contextual information during inference. This approach inevitably incurs higher inference time due to multiple module calls and the construction of detailed reasoning contexts. We provide a qualitative comparison of latency and token length in Table 13.
Table 13: Comparison of reasoning-token length and inference latency.
Method
Object instance information?
Reasoning token length
Inference latency
LEO
No
4.8s
LLaVA-3D
No
0
0.2s
Chat-Scene
No
0
0.5s
SceneCOT
Yes
350 1500
10.4s
The majority of the latency in SceneCOT (10.4s total) stems from LLM sequence generation, specifically Stage 1-3 generation (4.2s) and Stage generation (3.8s), while visual clue construction remains relatively moderate 2.4s. We believe this represents reasonable trade-off between transparent, interpretable reasoning and inference speed.
D.2 ACCURACY OF TASK AND REGION RECOGNITION
In our framework, errors from Task Recognition and Region Localization can be largely ignored due to their high accuracy. As shown in Table 14, both sub-tasks achieve extremely high performance, which we attribute to the strong commonsense priors inherent in large language models.
Table 14: Accuracy of internal recognition modules.
Module
Accuracy (%)
Question type recognition
99.4
Region recognition
100.0
For Region Localization, we employ a rule-based filtering mechanism. For instance, given an instruction like "Now need to ground the objects on my right," " the system applies predefined rules to filter objects within the corresponding sub-region, ensuring accurate object selection aligned with the model's recognition results.
E LIMITATIONS
Though we propose first step-by-step reasoning framework and have demonstrated its advantages on typical 3D scene reasoning tasks. There also several limitations in our work. First, our framework focuses on the tasks pre-defined in MSQA, which is limited in more complex scenarios such as embodied AI. For example, we do not consider the long-horizon tasks such as embodied task planning. Recently, SG3D Zhang et al. (2024b) proposes new benchmark to evaluate 3D-VL models' capabilities of grounded task planning in embodied scenarios. We will consider extending our framework to this task in the future.
Second, SCENECOT- 185K is built upon MSQA-ScanNet and Nr3D, which only contains the 3D scenes in ScanNet. Extending the dataset to more diverse real-world scenes is important direction to unlock the real-world applications in the future.
Third, our thought design is still not perfect in partial sub-tasks. Even the have demonstrated promising upper boundaries in some challenging sub-tasks, such as Counting and Navigation, our thought design still struggles with solving problems like Spatial Relationship. How to design better
25

---

Published as a conference paper at ICLR 2026
3D-CoTs is another important direction to further increase the upper boundaries of the reasoning framework in 3D scenes. Besides, based on the recent practice in advanced reasoning LLMs Guo et al. (2025); OpenAI (2025), exploring more learning algorithms such Reinforcement Learning may also lead to more surprising capabilities for complex 3D scene reasoning.
F BROADER IMPACT
Understanding and reasoning in 3D scenes is cornerstone capability for building intelligent embodied agents that can operate safely, reliably, and effectively in the physical world. Our proposed approach. SCENECOT, introduces a structured, interpretable framework for 3D scene reasoning by incorporating Chain-of-Thonn ohhoht (CoT) paradigm into the 3D lisoon-aagguag (3DVL) domain. By explicitly modeling intermediate reasoning steps such as task recognition, region localization, entity grounding. and grounded reasoning. SCENECOT offers significant leap toward human-level understanding in complex 3D environments.
The potential societal benefits of this work are substantial. enables advancements in a wide range of real-world applications, including domestic robotics, assistive technologies for individuals with disabilities, autonomous navigation systems, and intelligent agents for virtual and augmented reality. The stepwise, interpretable nature of our method also enhances transparency and safety, which are essential for deploying AI systems in numan-centered environments.
At the same time, we acknowledge that powerful embodied AI systems can be misused in ways that may compromise privacy or safety, particularly in surveillance or military contexts. To mitigate such risks, we encourage responsible research practices, including dataset transparency, open evaluation protocols, and active engagement with the broader community on the ethical deployment of such systems. Our work serves as foundation for building grounded and generalizable reasoning agents, and we hope it will inspire future research that aligns technological advancements with human values.
G LLM USAGE STATEMENT
We LLM in two ways: (1) aid with polishing the writing, including improving grammar and coherence of the manuscript; and (2) to generate a portion of training data for our model. The LLM does not contribute to any significant part of this work.
26


Reviews:
[{"review_id": "lluy8qz5tt", "reviewer": "Reviewer_rALF", "summary": "1. The paper proposes SCENECOT, a grounded Chain-of-Thought (CoT) framework for interpretable 3D scene reasoning. It decomposes complex reasoning into four explicit steps—task recognition, region localization, entity grounding, and grounded reasoning—each supported by symbolic and multimodal expert modules.\n\n2. It introduces SCENECOT-185K, a dataset of 185K stepwise reasoning traces covering Situated (MSQA) and Object-Centric (Beacon3D, GQA3D) tasks.\n\n3. Experiments on MSQA and Beacon3D demonstrate improved grounding–QA coherence (34.7% vs. 19.5%) and validated gains from question-type recognition, region filtering, and grounding loss.", "questions": "1. In the Grounded Reasoning, how is the \"Object Image Tokens\" generated? Are they derived from cropped RGB patches, projected 3D features, or another visual encoding process?\n\n2. What exactly is the symbolic engine mentioned in the architecture?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Novel application of CoT in 3D reasoning: Introduces an interpretable step-by-step framework that explicitly grounds each reasoning stage in scene elements. Visualization of reasoning chains (Fig. 6) makes the model’s decision process transparent and easier to diagnose.\n\n2. Large-scale dataset: SCENECOT-185K is the first dataset pairing CoT traces with 3D scene data, supporting both situated and object-centric reasoning.\n\n3. Clear empirical validation: Comprehensive experiments (MSQA + Beacon3D) and ablations demonstrate consistent improvement in 3D VQA task.", "weaknesses": "1. Lack of Human Verification. All QA pairs in the SCENECOT-185K dataset are generated by GPT-4o, which can introduce factual or logical errors. The paper does not mention any human validation or quality control process to ensure the correctness of the generated reasoning traces.\n\n2. Limited Baseline Coverage. Although GPT-4o is included as a comparison model, the experiments omit stronger recent multimodal baselines such as Gemini 2.5 Pro or Claude Opus, which would provide a more comprehensive evaluation of reasoning capability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9BBSZ0wR1n", "reviewer": "Reviewer_WaKT", "summary": "This paper addresses the problem of poor grounding in 3D vision-language models, where models often generate plausible-sounding answers that are not factually connected to the 3D scene. The authors propose SCENECOT, a novel framework that introduces step-by-step, Chain-of-Thought (CoT) reasoning to 3D question answering. The method explicitly decomposes a complex 3D reasoning task into four manageable stages: task recognition, task-relevant region localization, entity/attribute grounding using expert modules, and final grounded reasoning. To train this framework, the authors also developed SCENECOT-185K, the first large-scale dataset containing 185,000 grounded CoT reasoning traces for 3D scenes. Experiments demonstrate that SCENECOT achieves competitive performance on the general MSQA benchmark and, most notably, significantly outperforms all baselines on the Beacon3D benchmark, which is specifically designed to measure grounding-QA coherence.", "questions": "1. The generation of the SCENECOT-185K dataset relies on rule-based methods and LLM (GPT-4O) generation. What was the extent of manual verification to ensure the quality and correctness of the intermediate reasoning \"thoughts\"?\n2. In the 4-stage pipeline, how does the model handle error propagation? For instance, if the initial \"Task Recognition\" step fails, or the \"Region Localization\" identifies the wrong area, does this inevitably lead to a final failure, or are there mechanisms for recovery in the later stages?\n3. The multi-step inference process, which involves calls to symbolic engines and expert grounding modules, seems computationally more intensive than an end-to-end model. What is the comparative inference latency of SCENECOT versus baselines like Chat-Scene or LLaVA-3D? Is the framework practical for real-time applications?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work directly targets the critical and well-documented problem of poor grounding-QA coherence in 3D-VL models. Instead of just aiming for better QA accuracy, it focuses on ensuring the answers are correctly derived from the scene's visual context.\n2. The framework's multi-stage design (task recognition, region localization, grounding, reasoning) is intuitive and inherently interpretable. This transparency makes it easier to diagnose failure cases, as shown in the qualitative examples.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The SCENECOT framework is a complex, multi-stage pipeline rather than a simple end-to-end model. Its performance is heavily reliant on a cascade of specialized, pre-trained modules (e.g., Mask3D for object proposals, PQ3D for grounding). This introduces multiple potential points of failure, and the overall performance is strongly coupled to the quality of these \"expert\" modules.\n2. The dataset, while large, is constructed from existing benchmarks (MSQA, Nr3D) that are primarily based on the ScanNet dataset. This limits the diversity of scenes, objects, and tasks. The paper acknowledges that the framework does not yet extend to more complex, long-horizon embodied tasks.\n3. The paper introduces a \"grounded CoT\" framework, but it lacks experiments on standard 3D visual grounding benchmarks (e.g., Nr3D, Sr3D, or ScanRefer). While Beacon3D measures coherence (grounding + QA), evaluating the grounding module's performance in isolation on these tasks seems essential to fully validate the \"grounded\" aspect of the CoT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MaAezHKmmb", "reviewer": "Reviewer_tikH", "summary": "This paper presents SceneCoT, a 3D scene understanding framework that employs step-by-step chain-of-thought (CoT) reasoning to enhance spatial reasoning performance on benchmarks such as MSQA and Beacon3D. To facilitate 3D CoT reasoning, the authors construct a large-scale dataset, SceneCoT-185K. Experimental results further demonstrate that SceneCoT effectively improves grounding–QA coherence.", "questions": "I have concluded all my questions in the weakness sections.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. SceneCoT demonstrates strong performance in spatial reasoning, particularly on counting and grounding questions. Moreover, its step-wise grounded reasoning provides a transparent and interpretable rationale.\n\n2. The construction of the CoT steps is reasonable and aligns well with how humans approach spatial question answering.\n\n3. The paper is well-written, clearly organized, and easy to follow.", "weaknesses": "1. SceneCOT is designed around specific reasoning tasks—namely Situated Reasoning and Object-Centric Reasoning—and limited question types such as counting, attribute, and spatial relationship queries. This task-specific design may restrict the model’s ability to generalize to unseen question types encountered in more complex 3D world.\n2. In Table 1, the overall performance of LEO and MSR3D on MSQA drops noticeably after fine-tuning on the SceneCoT-185K dataset. Could the authors provide analysis on the possible reasons behind this degradation?\n3. Since SceneCOT is designed based on LLaVA-1.5, it would be helpful to show the performance comparison of these two models.\n4. It would be helpful to evaluate SceneCoT on out-of-domain datasets (e.g., SQA3D [1], Hypo3D [2], VSIBench [3]) to verify whether CoT fine-tuning improves spatial reasoning beyond the training domain.\n\n[1] Ma, Xiaojian, et al. \"Sqa3d: Situated question answering in 3d scenes.\"\n\n[2] Mao, Ye, et al. \"Hypo3D: Exploring Hypothetical Reasoning in 3D.\"\n\n[3] Yang, Jihan, et al. \"Thinking in space: How multimodal large language models see, remember, and recall spaces.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TI9M1gnHNc", "reviewer": "Reviewer_rnc2", "summary": "This paper introduces grounded Chain-of-Thought reasoning into 3D large language models, decoupling complex reasoning tasks into simpler and more manageable problems, and constructing corresponding visual cues through multimodal expert modules. To achieve this, the authors focus on dataset curation. By training on the proposed dataset, the model attains state-of-the-art performance on several 3D understanding benchmarks.", "questions": "1. Can grounded large-scale datasets like **3D-GRAND** be integrated into your pipeline?\n2. What is the purpose of introducing probability during reasoning? Are there any ablation studies to justify its effect?\n3. The **LEO** model adopts a similar object-centric input as the proposed method. Why does the performance on MSQA decrease after training on your dataset? The effectiveness of the proposed reasoning mechanism remains questionable.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The first to introduce CoT reasoning into 3D understanding.\n2. Proposes a large-scale dataset to support the study.\n3. The model achieves leading performance across multiple benchmarks.", "weaknesses": "1. **Limited evaluation:** The authors primarily evaluate on MSQA and Beacon3D, while several widely used benchmarks such as ScanQA and SQA-3D are not considered.\n2. **Limited data sources:** The annotations mainly originate from Nr3D and MSQA. Incorporating larger and more diverse datasets and scenes, such as MMScan[3], could enhance generalization.\n3. **Object-centric input:** The method mainly relies on object-centric input, which requires an additional segmentation model during inference and thus limits broader applicability. The ablation results show that performance is highly dependent on segmentation labels. Since object-centric input already provides a strong spatial prior for grounded reasoning, it remains unclear whether the proposed method can generalize to video-based 3D LLMs such as Video-3D LLM[1] or LLaVA-3D[2].\n\n[1] https://arxiv.org/abs/2412.00493 (CVPR 25)\n[2] https://arxiv.org/abs/2409.18125 (ICCV 25)\n[3] https://arxiv.org/abs/2406.09401 (NIPS 24)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)