SCoT: Teaching 3D-LLMs to Think Spatially with Million-scale CoT Annotations
Authors: Jinpeng Li
Haiping Wang
Jiabin chen
Yuan Liu
Zhen Dong
Bisheng Yang
Venue: ICLR 2026 Poster (2026)
Keywords: 3D Large Language Model
Chain-of-Thought
Spatial Perception
Spatial Analysis
Spatial Planning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present a million-scale 3D visual-language dataset with CoT annotations that unifies perception, analysis, and planning tasks to advance interpretable 3D intelligence.

Abstract: Recent advances in 3D Large Language Models (3D-LLMs) show strong potential in understanding and interacting with 3D environments, yet their training data typically lack explicit reasoning processes, limiting complex spatial reasoning and task planning.
To address this, we annotate SCoT, a million-scale Chain-of-Thought dataset spanning three levels: a) Spatial Perception (what is there), recognizing object properties, relations, and scene attributes; b) Spatial Analysis (what does it mean), inferring rationality, functionalities, and physical implications; c) Spatial Planning (what should I do), integrating perception and reasoning for actionable strategies. Unlike prior datasets supervising only answers, SCoT annotates intermediate reasoning grounded in scene cues, specifically for analysis and planning tasks. Results show that CoT supervision greatly benefits complex analysis and planning but induces hallucinations and accuracy drops in simple perception. These findings highlight both the necessity and the nuanced challenges of scene-grounded reasoning for advancing 3D intelligence.

Published as a conference paper at ICLR 2026
SCOT: TEACHING 3D-LLMS TO THINK SPATIALLY WITH MILLION-SCALE COT ANNOTATIONS
Jinpeng Li Haiping Wang Jiabin Chen' Yuan Liu Zhen Donght Bisheng Yangh LIESMARS, Wuhan University Hong Kong University of Science and Technology
Level 1: Spatial Perception " What is there? "
Object Perception
Relationship Perception
Scene Perception
Visual Grounding
What color is the wooden door? Answer
Where dues coffee question photie
Please describe detail.
Please ID of obyect that fis the description "The TV setting stand"
question wring
<Annwers
berge
slagie word phrase.
with
<Ansner> to fiott
for dork lowp, and a fromn oetiongulrr unsdee door </Anrner"
<Annwers This object <OR3092> </Antner's
<Answers Brown </Aurwer>
</Kerwer>
Level 2: Spatial Analysis
What does it mean?"
Object Analysis
Scene Analysis
Whit are the lay viman fenturey of furmture prace located near round tabii
SCOT
Seved un Jormiure planonon und ur urrongmment what key acthiitis cue this room nppoor?
does
qpatialty to the space??
"Co'> <S The noo has carved couch
swcooading
contrutly positioned with coffee table and addiiinnl chairs creating seaning ana for
Cool> e Siz The forniurr piece us choir
casuul comversaioon and tomporary relatation.
characterized by rounded bachrest and woodon
<Sp> The coffee table can be used for placteg
acnture <S1- located nest to round sabie
There vinille sining sheff derk, indicaiig dedicated seap mannly workspace duing <oCur>
driskz muiia
seggasing part souting and indleating
seating darigg
relexing patheringe
supgoots
<Anywer
This object a wsodon chair with
<Anuw'' The now arangement supports
Rackmet It ovar the roudd labir and small mig raggssing it is Intended for relaving gathorings </inswrr
comersatiin relavation. with
compased chatrs
drinking smallsuek workspare dining apparent <ijurner
Relationship Analysis
Implicit Detection
What is the spanial relatlonshp Bemowas the carved cucch at P.4.
Whick tue-sudored haw sofe placiny contral?
188 1.0 anterz and bemn tabie ot [3.3,3. 0.3] meters?
<CCl''s Calenlaee distanee roing comniinatoo: apport 3-2 4 + 13 13.0 (0.24 oy 2) The and difforences show the couch forward
SCad's
sola
(s (-088809))
entera <St- posiion
aligus
reguirement moble
be
Esciideun
place TV remute comins mutch
brown tubie
funcmonal deserppton, confiruing targer object CoCos.
apprormmntly 1.7 the couch alightly in frum the table. They are lightt) attcched each other. </Answees
SAnsww"
Level 3: Spatial Planning
"What should do?"
Un-situated Planning
Situated Planning
Horo udjjutt room sentigs lake drief rolex ofier vaturing room?
Standing near wide, carved scating grea, wan to fil down for relecatios, and mht set ap portable foiding tuble beside me. it forsible both
contutre visee couch coiling lamp nour the couch, and ratliatoo along the wall First nure on the eviling lamp for lighting, nn that you van adfuct the
<<ooT> -S-- The
Occupled Ares
erposamic ruppori ritting dorm Humsses nearbyy occupied
"CoTs <Sts
The wide roating area proviles ample
Canck
radiater control for proper temperanire. Then, dim velting kamp and almn couch offere comflnt space.
place poctable folding Thorsioe, stning dows
can
plucing partable
folding table consrrmtee limird ores SoOT
"Anooorr Segin by borning on the cxiling inmp odhicr
penoddrs Soing support Houriar placing portable foading table way be difficatt due cocuptod arca and furmihie placemen. <iturners
is foanble
Redlater
Donn
coding komp
Figure I: SCoT provides 1.1M cases to supervise 3D-LLMs in learning spatial perception, analysis, and planning. Beyond answer pairs, each analysis and planning case explicitly includes Chain-of-Thhn Thought (CoT) annotation grounded in the scene context.
'The first two authors contribute equally.
1 CCorespondiig Authors.

---

Published as a conference paper at ICLR 2026
ABSTRACT
Recent advances in 3D Large Language Models (3D-LLMs) show strong potential in understanding and interacting with 3D environments, yet their training data typically lack explicit reasoning processes, limiting complex spatial reasoning and task planning. To address this, we annotate SCoT, a million- scale Chain- of-Thought dataset spanning three levels: a) Spatial Perception (what there), recognizing object properties, relations, and scene attributes; b) Spatial Analysis (what does mean), inferring rationality, functionalities, and physical implications: <) Spatial Planning (what should I do), integrating perception and reasoning for actionable strategies. Unlike prior datasets supervising only answers, SCoT annotates intermediate reasoning grounded in scene cues, specifically for analysis and planning tasks. Results show that CoT supervision greatly benefits complex analysis and planning but induces hallucinations and accuracy drops in simple perception. These findings highlight both the necessity and the nuanced challenges of scene-gre geneeggoonded reasoning for advancing 3D intelligence. The dataset and code are available at https:: / comm/WUC USI3DV/SCO
INTRODUCTION
The burgeoning field of 3D Large Language Model (3D-LLM) learning aims to equip LLMs with human-like understanding of and ability to interact with the three-dimensional world. While LLMs have revolutionized text and 2D image understanding, their extension into 3D realms remains superficial. Current 3D-LLMs are primarily trained datasets that provide only question-answer (Q-A) pairs (Table D. effectively teaching models to recall and associate but not to reason. This paradigm reduces powerful models to black boxes that output answers without transparency into their Seaision-making processes, severely limiting their reliability and applicability in real-world scenarios, such robotics and embodied AI, where trust and verifiability are paramount.
Chain-of- Chain-of-Thought (CoT) reasoning is a powerful method for enhancing model transparency, achieving remarkable success textual and 2D visual domains. However, its direct transfer to 3D understanding is non-trivial. Unsupervised CoT frequently produces rationales that are linguistically plausible but scenically ungrounded, relying on linguistic priors rather than actual 3D evidence. This results in new type of black box: one that appears interpretable yet fails to remain faithful, ultimately undermining trust (Huang et al. 2025c).
Thus, some existing works annotate CoT chains to enforce scene-grounded COT. Yet, these methods suffer from two key limita- tions. First, they do not differentiate be- tween simple perceptual tasks and com- plex reasoning tasks. However, we em- pirically show that indiscriminately apply- ing CoT to simple perceptual where direct observation suffices-can add un- necessary complexity and induce hallucina- tions (Huang al. 2025b) Liu al. [2025a] 2024b). yielding 4.9% accuracy drop as shown in Table and Fig. Second, their limited scale in Table prevents models from learning diverse and robust reasoning patterns.
the water founiains Hon
Question:
Water Foumiam
by Direct Perception
Answer: Twoo (Gronnd Trath: Tow)
Answer with Formal CoT
Water Fountain
CoT: The question ucks love many water foantoter are to the left when facing too
ostomuns in frost. Ole water fousiait austiiune the left ride Thereforv. there one mater fountiis the left. (Ground
Figure 2: Overuse of CoT in simple perceptual tasks can introduce hallucinations, overlooks, and errors.
To address these issues, we first introduce principled three-tier taxonomy that reflects natural progression in task complexity and explicitly dictates when CoT should be employed -nrrencinn reasoning where needed while avoiding overcomplication where unnecessary:
Spatial Perception ("What is there?"): This layer focuses on foundational scene understanding, including object properties (e.g., color, shape), object-to-object relations (e.g., left of, supporting), and global scene attributes (e.g.. furniture composition). Since these tasks require few inference beyond direct observation, we supervise the model using answer-only training to strengthen its grounding in visual evidence, avoiding unnecessary CoT that could introduce hallucinations.
2

---

Published as a conference paper at ICLR 2026
Spatial Analysis ("What does it mean?"): At this level, the model must interpret perceived elements by integrating them with world knowledge. Tasks include inferring functionality, causality, physical stability, and spatial logic (e.g., "This is dining room because the table is surrounded by chairs"). Here, we introduce CoT to explicitly guide the model through structured reasoning steps.
Spatial Planning ("What should do?"): The highest layer requires synthesizing perception and reasoning into actionable plans. The model must generate sequential strategies for embodied agents (e.g.. "To clean this room, first pick up the clothes, then... "). For these tasks, CoT is essential to articulate the logical flow from observation to action.
Guided by this taxonomy, we construct SCoT (Spatial Chain-of-Thoughh designed to teach 3D-LLMs how to think spatially, not just what to answer. For analysis and planning, SCoT asks CoT reasoning grounded on real scene information. To build it, given a scene, we first construct a scene context. Specifically, the scene is decomposed into objects, represented as a scene graph, and described with rich textual descriptions. Together with the BEV scene image, this scene context is used to prompt an VLM to generate "Question-CoT-Answe demonstrate how CoT reasoning leverages scene information, we require the CoT to be grounded in the scene context and introduce a dedicated <SI> token. Whenever scene information (e.g., object properties, spatial relationships, structural attributes) is utilized, it must be cited with an <SI> tag. In this way, the CoT transforms vague assertions into verifiable, grounded claims. For instance, instead of stating "The table is too small", the model is expected to reason as follows: "The table <SI>(with width of 0.8m))/SD>> is too small to fit <SI>all six chairs (each 0.5m wide)</SI> around it."
million-scale dataset
samples. To explicitly
We train the baselines and a self-designed model on this SCoT dataset. Experiments demonstrate that SCoT, evolving from answer-supervised to grounded-CoT training, achieves significant perfor- mance gains across benchmarks. More importantly, it yields models whose reasoning processes are transparent, faithful to the scene, and fundamentally more explainable and trustworthy.
2 RELATED WORK
3D Large Language Models. LLMs show strong reasoning and dialogue abilities, and extending them to 3D perception has become a key paradigm for natural language interaction in 3D environments. Early point-cloud LLMs (Xu et al.] 2024] Guo et al.) 2023; Wang et al. 2023) align point clouds with text to support object-level queries, but struggle with scene-level reasoning. Hybrid 3D Vision- Language models (Zhu et al.] [024 Deng et al.) 2025) combine multi-view image features from 2D backbones (e.g.. LA (Liu et al. 2023)) with 3D position-aware point cloud features, while Video ILMSs (Huang et al. 2024] Zheng et al., 2025) incorporate 3D spatial cues into video representations to capture semantics and relations, enabling more complex spatial reasoning.
Trainsets for 3D-LLM. Advances in 3D-LLM have driven the creation of 3D-Language datasets linking 3D perception with natural language. Early works introduced task-specific benchmarks for captioning (Chen et al., 202T] Jia et al. 2024), question answering (Azuma et al. 2022, Ma
et al. 2022 Zhao et al. 2022 Linghu et al. 2024). and grounding (Chen al. [2200] Zhang et al.
2023 Achlioptas et al. 2020). However, these resources restrict cross-task. transfer in 3D LLMs.
Recent efforts have shifted toward large-scale multi-task datasets (Li al. 2023 Wang al. 2024
Lyu et al., 2024 Yang et al., 2025a Huang et af.) 2025a] Zhang et al. 2025] Gong et al. 2025). supporting dense captioning, multi-turn dialogue, and embodied planning. 3D-CoT (Chen al. 2025) further emphasizes reasoning by adding CoT annotations for object-level tasks, yet remains
limited object-centric understanding. Importantly, most datasets still lack explicit reasoning traces or grounding in 3D evidence, leaving scene-level CoT reasoning largely unexplored.
Shain-of-Thoughee Reasoning, CoT prompting (Kojima et al. 2022: Wei et al. 2022) has proven effective in cliciting step-by-step reasoning for text tasks. Reinforcement learning further strengthens
CoT reliability, as shown in OpenAl ol (Jechh et al.] (024) and DeepSeek-R1 (Guo al.
2025).
Inspired by these successes, recent work extends CoT to
models (Yang et al. 2025b)
Huang et al. 2025d] Shen et al. 2025 Peng et al. 2025 Liu et al, 2025b). In 3D-LLM, CoT reasoning remains nascent but promising: approaches such as 3D-RI (Huang et al. (025c), SpaceR (Ouyang) et al. )025), and Spatial-MLLM (Wu et al. 2025) augment datasets with CoT annotation or apply reinforcement learning with ssecificciicc rewards, achieving chain-based outputs. Yet, current studies are constrained by dataset scale, task diversity, and transparency, highlighting the need for broader, high-quality 3D CoT resources covering perception, reasoning, and planning.
3

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of SCoT dataset and the existing 3D large language model train sets. "M.3D." means the Matterport3D dataset, "3RS." means the 3RScan dataset, "M.S." means the MultiScan dataset, "S.3D." means the Structured3D dataset, "P.T." means the ProcTHOR dataset, "Obj." means the Objaverse dataset, "ARK." means the ARKitScenes dataset, "3D.F." means the 3D-Front dataset. "Reasoning" refers to the reasoning process besides the final answer.
Dataset
Source
Scene
Size
Reasoning
Tasks
ScanRefer (ECCV 2020)
ScanNet
800
32k
Grounding
Scan2Cap (CVPR 2021)
ScunNet
800
32k
Caption
ScanQA (CVPR 2022)
ScanNet
800
27k
QA
SQA3D (ICLR 2023)
ScunNet
800
33k
QA
3D-LLLM (NeurlPS 2023)
ScanNet
1513
659k
Multiple Tasks
Embodieedcaan (CVPR 2024)
ScanNet, M.3ID.
5185
1M
Multiple Tusks
M3Dbench (ECCV 2024)
ScanNet, M.3D.
700
327k
Multiple Tusks
Scene (ECCV 2024)
ScanNer, 3RS. M.S., S.3D. P.T.
97000
2.4M
Multiple Tasks
LEO (ICMI. 2024)
ScanNet, 3RS., M.S., Obj.
3000
513k
Multiple Tasks
MSR3D (NeurIPS 2024)
ScanNet, 3RS., ARK.
1700
251k
Multiple Tasks
3D-GRRAND (CVPR 2025)
ScanNet, S.I.. 3D.F.
40000
6.2M
Multiple Tasks
3D-CoT (Arxiv)
1.6M
Object QA
3D-R1 (Arxiv)
ScanNet, 3RS., M.S., S.3.., P.T.
1513
30k
Multiple Tasks
SpaceR- (Arxiv)
ScanNet, General Image and Video ScanNet, ARK. ScanNet
151k
Multiple Tasks
Ppatial-MLLM 1200 (NeurIPS 2025)
120k
Multiple Tasks
SCoT (ours)
800
1.IM
Multiple Tasks
3 SCOT CONSTRUCTION
With approximately 1.1 million samples, SCoT provides grounded factual descriptions and reasoning chains spanning 3D intelligent tasks. We first introduce our three-tier taxonomy of 3D task classifica- tion to analyze their core characteristics and suitability for CoT reasoning. Based on this taxonomy, we construct task-specific "Query-CoT-Answer" data and introduce our quality control procedures.
3.1 TASK TAXONOMY
Spatial Perception (240K samples) tells "What is there". It only requires basic scene observation. We reveal that excessive use of CoT introduces hallucinations in perception tasks, as shown in Sec. 5.2. Therefore, by default, we discard CoT and adopt answer-only supervision for perception. We provide CoT-annotated perception data to validate this observation empirically. The perception tasks include: (a) Object Perception: Identifying explicit properties of individual objects (color, shape, size, spatial location); (b) Relationship Perception: Recognizing spatial and semantic relationships between objects (adjacency, containment, support); (c) Scene Perception: Understanding holistic scene attributes (room type, furniture composition, global arrangements): (d) Visual Grounding: Localizing objects based on text descriptions explicitly mentioning the object.
Spatial Analysis (460K samples) tells "What does it mean". It requires inference beyond direct observation, incorporating logical and quantitative reasoning chains grounded in 3D contexts, includ- ing: (a) Object Analysis: Conducting in-depth analysis of specific objects, including their attributes beyond visualization and potential functions; (b) Relationship Analysis: Quantitatively assessing relationships between object pairs by calculation (c.g., relative distances, orientations, topological dependencies); (c) Scene Analysis: Human- like reasoning at the scene level, such as layout evaluation, capacity analysis, space assessment, or improvement suggestions; (d) Implicit Detection: Localizing objects based on implicit text descriptions without explicit name or semantic mentions.
Spatial Planning (390K samples) tells "What should I do". It involves action sequencing and task execution planning within realistic environmental constraints, including: (a) Un-situated Planning: Planning and decision-making under additional requirements (e.g., power outages, blockages, water supply interruptions) without giving the situated conditions, requiring to decompose complex objec- tives into manageable sequential steps; (b) Situated Planning: Assessing the fcasibility and procedure for exccuting a specified task, given the agent's 's current state (including its position, orientation, and ongoing activities), allowing the use of spatial terms and interaction-basee analysis.
This structured taxonomy enables comprehensive coverage of 3D reasoning capabilities, from basic perception to complex planning. Most tasks in existing 3D vision-language benchmarks can be mapped to one or more of these categories. By providing a unified framework for task definition and evaluation, the SCoT dataset facilitates systematic and interpretable implementation of 3D LLMss.
4

---

Published as a conference paper at ICLR 2026
Buildding Scene Context
Building QA Pairs with CoT
Quality Control
Point Cloud Object Proposols
Scone Context
Existing Pairs
< Check
LLM Cross Check
Visual Bvidence
Qveatbe: book sta
Structured Tests
BFV Mop
Scone Graph
VLM-hased Dato Generation
Statistical Information
(Chama))
ChatGPT 4.1
Pronpe -Questive Stwellnggr
lagil adles doniderin to sdiemmo etriuat sich
Spatial Perception (240K)
Newer snayy win out Jehermn abuuyy fir red buener some Viteaaien (nl and srower
Arsept classering with Colbe
Bosk
Sportsl Analysh (AGOK)
22.2%
Sportar Placning (P805)
Multi-modarity Scene Context
CljectAnayyl
Scans Analysa
Stuoted Pomning
Quertion: ohet orr wixd Suttore of new abaet dhr delk, Aow
MaDioe Percurtes Am
Deage Ban
Soveciane Tants
Cotl abjert i ructaiedd ha ballond, connrnd Jahre
Greet Peceppion
Oupet diee
soff meterisis inedy ansiyeed mecinn warking
Atswe: Ths obred e callorz srar prown cols: amagned for small
Figure 3: Annotation and data statistics of SCoT. SCoT first constructs a detailed scene context for each 3D scene as input to the VLM, then prompts the VLM to generate Query- Query-CoT-Answer cases grounded in the scene. In total, we built 1.1M samples spanning perception, analysis, and planning.
3.2 DATA GENERATION
As discussed above, for Spatial Perception tasks, collect and adapt existing Q-A pairs from publicly available 3D-VL benchmarks, specifically ScanRefer, ScanQA, SQA3D and Scan2Cap. For Spatial Reasoning and Planning tasks, we leverage advanced VLMs to generate Q-A pairs along with CoT reasoning chains. The CoT generation process begins by providing the VLM with rich descriptions of 3D scene context, including object location, spatial relationships, and global layout information. Using this context, the VLM is guided to generate "Query Snsery-CoT-iee CoT-Answer" samples that incorporate multi-step reasoning and in-depth scene analysis.
Building Scene Context. Given a 3D scene, we first segmented it into object proposals. Each object assigned bounding box and a semantic label. These objects are organized into a scene graph. where nodes represent objects and edges encode spatial relations such as adjacency, support, and relative direction. These attributes are serialized into textual templates (e.g.. "Object-3 [chair, at (0.2, 0.4, 0.5)] is 0.8m to the left of Object-1 [desk, at (1.0, 0.4, 1.1)]"). In parallel, the scene is projected into a 2D Bird's-Eye View (BEV) map, where each object is highlighted with a unique identifier and bounding box. Local object-centric crops and global scene images further supply visual evidence. Finally, all elements above are consolidated and expressed as structured texts paired with visual evidence, serving our multi-modality scene context with rich scene information.
Building QA Pairs with COT. Existing datasets typically provide only simple answer supervision, leaving the thinking process as a black box. To construct explicit reasoning traces, we leverage the powerful capabilities and knowledge of VLMs. Specifically, we provide the VLM with the scene context as a reference, together with simple Q-A pair for expansion by the following two steps:
Question Rewriting: In existing datasets, queries often explicitly mention target objects by name, which allows the LLM directly locate the object and makes the reasoning process trivial. To fully stimulate 3D-LLM's 's scene perception and reasoning ability, we need to prepare queries avoiding directly naming the target. Instead, the queries should reference attributes such as the object's functionality or contextual role. For example, "What color is the chair?" is reformulated as "What are the visual features of the object near the desk, how does its location relative to the surrounding furniture suggest its daily use?" In tasks related to spatial planning, VLM is further prompted to infer object states or simulate real-world conditions.
Answering with CoT: The VLM needs to provide step-by-ster reasoning chains to reach the answer, transforming concise responses into comprehensive logical explanations. To maintain quality and consistency, we prompt the VLM to answer with: "i) identify the task goal. ii) analyze the current state or constraints and extract the object and scene content. üi) leverage scene information to form the basis for spatial analysis or planning. iv) finally, provide the answer."
Asking CoT Explicitly Grounded on Scene Context. Naive CoT has a risk of hallucination and superficial reasoning, where generated rationales may appear logically consistent but are detached
5

---

Published as a conference paper at ICLR 2026
Table 2: Results on SCoT- Perception tasks with or without perception CoT supervision. "CoT" means using CoT annotations in the perception tasks. The model is SCoT Reasoner.
ScanRefer
Multt
Scan2Cap
ScanQA
SQA3D
CoT
Acc@0.25
Acc
F1@0.25
F1
CIDEr@U.5
B-4@0.5
CIDEr
B-4
EM
EM
58.4
53.4
60.1
55.7
79.6
38.2
87.9
15.0
23.0
55.8
56.4
51.3
53.5
49.3
75.7
35.8
73.4
13.1
21.9
47.4
Table 3: Overall performance comparison on SCoT-Analysis tasks requiring text output. "R.". "M.", "Exp.", "Fai." and "Tru." are abbreviations for "ROUGE-L", "METEOR". "Explainability" "Faithfulness" and "TTrusworttiness" respectively. Note that the score range for "Explainability" "Faithfulness", and ""Trustorthiness" metrics from 10. "(CoT)" means the method trained with "Full SCoT Setting", , and the counterpart without it is trained with "Answer-Only Setting".
Object Andlysis
Relationskip Analysis
Scene Analyzis
Method
R.
M.
Exp.
Fai.
Tru.
R.
M.
Exp.
Fai.
Tru.
R.
M.
Exp.
Fai.
Tru.
Char3D V2
24.91
15.42
6.71
5.15
6.08
26.03
13.48
4.71
3.45
4.10
20.38
13.16
6.73
6.48
6.52
Char3D V2 (CoT)
25.18
15.76
6.93
5,50
6.35
26.23
14.91
4.87
4.07
4.54
21.55
13.97
6.74
6.43
6.58
Video LLM
19.72
12.93
6.84
5,02
5.97
25.71
14.07
4.42
3,03
3.98
22.01
12.79
7.28
7.01
6.92
Video 3D LLM (Col)
20.57
14.07
7.02
5:43
6.24
26.90
14.92
4.74
3.11
4.02
22.85
13.06
7.72
7.33
7.37
26.11
15.57
6.35
5.32
5.63
27.17
15.63
4.98
4.32
4.45
21.84
14.04
7.71
7,20
7.23
(CuT)
26.98
15.80
6.75
6.07
28.40
16.15
5.62
5.30
22.72
15.10
7.80
7.49
SCofT-Rcasoner
27.34
15.62
6.67
5.89
29.71
16.41
5.19
4.23
22.59
14.68
7.82
7.32
7.45
SCoT-Reasoner (Col)
27.22
16.17
7.04
6.15
6.41
30.69
16.60
5.77
5,34
5.41
23.48
15.29
7.95
7.55
7.68
from actual scene content, even scene contexts were provided. We thus introduce <SI> (Scene Information) tagging mechanism CoT generation process, the VLM is strictly prompted to explicitly insert <SI> at every step in the reasoning chain where scene-derived information (e.g.. object properties, spatial relations or scene layout) is utilized, as illustrated in Fig.
On the one hand, this approach enforces grounded reasoning, requiring the reasoning process to remain faithful to the visual context; samples with insufficient or incorrect use of <SI> tags are discarded. On the other hand, it enhances traceability, enabling users and developers to verify which elements of the scene underpin each inference.
3.3 QUALITY CONTROL
Besides the above <SI> check, to improve reliability of SCoT, we perform LLM.-based cross-check. We first employ ChatGPT-4.1 (Achiam et al., 2023), Qwen (Bai et al. (022)) and DeepSeek (Liu et al. 2024a) as independent agents. During cross-check process, if any of the agents identify issues (e.g.. poo-pd question, incorrect or misleading CoT, ambiguous answer, or inconsistencies between the scene facts and answer or CoT), these samples are flagged and removed. This strict procedure ensures that the SCoT dataset maintains the highest standards of data quality and factual consistency.
Afterwards, we conduct manual accuracy evaluation. We randomly select 50 samples for each task among trainset, resulting total of 500 samples to undergo manual checking. After excluding problematic cases, 447 samples are retained, yielding an acceptance rate of approximately 90%.
4 TRAINING 3D- 3D-LLMS WITH SCoT
3D-LLM Selection. To more comprehensively validate SCo'T, we train and evaluate various 3D-LLM architectures on SCoT, including 3D VG Transformer (Zhao al. (202T), Chat 3D V2 (Huang et al.)
2023), Chat Scene (Huang et al.) 2024), and Video 3D-LLM (Zheng et al. 2025). Introductions to these baselines are provided in the Appendix ASec.A.8). Moreover, we find that most prior methods focus on a single spatial modality (e.g., point clouds or images). To more comprehensively evaluate the effectiveness of SCoT, we additionally design SCoT-Reasoner, a unified framework supporting multimodal inputs such as images and point clouds. Its architecture and implementation details are provided in the Appendix ASec.A.2).
Training Strategy. We adopt two-stage training strategy to effectively integrate structured reasoning into 3D.LLMs. In the first stage, the model is trained on spatial perception samples from SCoT to establish a robust understanding of object properties, relationships, and scene structure. In the second stage, the model is fine-tuned on spatial analysis and planning tasks, where it learns to generate explicit, step-by-step reasoning chains grounded in scene information.

---

Published as a conference paper at ICLR 2026
Table 4: Overall performance comparison on SCoT-Plaminn. "R.", "M.", "Exp.", "Fai." and "Tru." are abbreviations for "ROUGE-L", "METEOR", "Explainability". "Faithfulneess and "Trustworthiness", respectively. Note that the score range for "Explainability" "FFithfulness"" and "Trustworthiness" metrics is from I to 10, "(CoT)" means the method trained with "Full SCoT Setting", and the counterpart without is trained with "Answer-Only Setting".
Situated Planning Exp.
Un-situated Planning
Method
Exp.
Fai.
Chat3D V2
23.93
12:21
6.38
6.18
6.33
28.69
16.44
6,08
5,88
6.03
Chat3D V2 (CoT)
24.95
13.01
6.65
6.34
6,56
28.99
16,23
6,35
6,04
6.26
Video 3D LLM
21.66
11.79
6.02
5.87
5.91
26.82
14.29
6,20
6,07
5.74
Video 3D I.IM (CaT)
21.34
12.35
6.31
6.10
6.00
27.29
15,90
7.01
6.45
6.19
Chat Scene
24.87
12.92
6.47
6.49
6.35
27.73
16.09
6.80
6.94
6.72
Chat (Col)
25.02
13.26
7.02
7.05
6.95
27.24
6,96
7.08
6.91
SCoT-Reasoner
24.21
12.37
6.64
6.09
6.30
28.01
18.35
6.97
6,93
6.76
SCoT-Reasoner (CoT)
25.13
13.06
7.38
6.94
7.14
29.04
18.11
7.27
7.29
7.15
5 EXPERIMENTS
5.1 EXPERIMENTAL PROTOCOLS
Settings. To ensure a comprehensive evaluation of SCoT, each model in Section is re-trained in two distinct configurations: (a) Full SCoT Setting: Models are trained using the complete SCoT data, including both answers and structured CoT sequences. (b) Answer-Only Setting: Models are trained using only the final answers from SCoT, with CoT sequences removed, to ablate and evaluate the specific contribution of CoT supervision. All the experiments are implemented with PyTorrch on a single NVIDIA A100 GPU. Training process of the first and second stage takes about 6 and 28 hours to converge, respectively. All other settings are guaranteed the same for a fair comparison.
Datasets. To facilitate comprehensive evaluation of model capabilities across different cogni- tive levels, we split the SCoT test set into three dedicated subsets: (a) SCoT-Perception is curated from existing public datasets, incorporating simple Q-A pairs from ScanQA. SQA3D, ScanRefer, Multi3DRefer, and Scan2Cap. It focuses on tasks such as visual question answering, caption gener- ation, and explicit visual grounding, serving evaluate basic scene perception and understanding. (b) SCoT-Analysis aims to assess model capabilities in object-centric, celationship-orinnte oriented, and scene-level reasoning, as well implicit detection. This test set contains only final answers without chain-of-thought sequences, enabling pure outcome-based evaluation. (c) SCoT-Plamning is designed to evaluate un-situated and situated planning abilities. Similarly, provides only final answers without intermediate reasoning chains, focusing the assessment on plan correctness and feasibility.
Metries. For Traditional Metries, we report Acc 20.25 and Acc@0.50 for grounding tasks. For text- based tasks, we adopt metrics including BLEU, ROUGE-I, METEOR, EM, and CIDEr. However, these metrics are proven unstable for only capturing text similarities (Qi et al. 2025
Lovin 2025).
Thus, we further adopt more reliable Comprehensive Metrics. Motivated by recent works (Qi et al., 2025 Lovin 2025), we introduce three comprehensive criteria evaluated by LLMs: (a) Explainability: Measures the clarity and logical coherence of the response, ensuring the reasoning process is transparent and easy to follow; (b) Faithfulness: Evaluates whether the response remains strictly grounded in the provided scene evidence without introducing unsupported claims hallucinations; (c) Trustworthiness: Assesses the overall credibility and reliability of the answer, considering both factual accuracy and presentation style. We prompt ChatGPT,, Qwen, and DeepSeek evaluators. Each model receives the generated response, ground truth, and relevant scene context. Each criterion rated on scale of 10, with final scores averaged across all evaluators. For MSQA, we report the average correctness score across test sets following Linghu et al. (2024).
5.2 QUANTITATIVE RESULTS
SCoT-Perception. Consistent with analysis in Sec.3.1, supervising CoT construction for simple perception tasks leads to 4.9% performance drop, as shown in Table 2 In these cases, linguistic priors may override scene-gres grounded cues, while the forced multi-step reasoning introduces unnecessary steps that further increase hallucinations. More results on SCoT-Perception with multiple basclines provided in the Appendix (Sec.A.5).
7

---

Published as a conference paper at ICLR 2026
What distinctive feotures does vereen areennhhariig device have, and where posttioned relatve to ambtent farniture and the room Iapond?
aa
Coll= (1) The tat gool is to deseribe the distinctie featuees the serennbbaurng
desice and to position relunie to the room leyour (2) The target object io montoor
(3) <St> The montor has rectangular thape with dark, frams and black screen
right corner of moo.. indicating worhopace muuly other furmiture nuch chairs and shelves.
surfoce <SI> itS. placed on desk locuted sear the
-St- This derk it surrosndedd
observing spatial design and placement suggest finetiions us display device for compuinn the office.
rectungulas sereen with a dark frame sereen surface. It is positioned on a desk near the upper rightt corner of the room, adjacest to other farniuwe like chairs and shelves. it likely serves as display device for computing takes stAnswers
<Aumerr> object
black
What
laynut support, how spatial arangement
forninue tipes contribute multfunctional of
(b)
surrondedd muitilee chairs, indioating setup conducive group diveusstras or seetinnss <Si> Along the walls. there are booksbelves and blackboard, which support actvitics such as roaling or preseutatiou. <SI> The spotial orangemon clutters soating around the tuble. facilitatin face-to-fuce intercction, while paripheral furiiture like the baokshelf and blackhoord aflow for individani small supports both collaborative and soliurry effectively <oCol>
SSI= room comtins
tasks This muliffurciions layont
<Annwer> The iayont napporty grou discusions or meetings the to chustered seating around a central mble. Periphoral clemous like bookshsives and biackboard enable roading presentations making the space multifuuc tional both collaborative and iudividuul tooks s/Answer"
spatial relatonnhhp, including relatve cabinets (located [3 2.0ja) the microwase (at (3.1, 0.6, 1.7jm)
CoT>> Calrudate eucludvan
destaee wring thesee too abject cocndioates: (3.1mm-1mm 1aa--006j---2000000000 12.00m-177m'
3m. Then. based the direcs cabinets (ar (3.1, 2.0ja)
comparison coordmattes
the
micromave (ai (3.1. 0.6 1.7ja).
</Col>
<Anrwer> The kicheen cubinets 0. away from the micronaee, positioned directly above dightly siArrerrs
(d)
(e)
Which blue iee usel for coffecting recyclable the area for cating postiioned placed
Which object, aeedd for the with its handic posiiioaed right locuned the left the window cuvering?
object a rectangular shape is pianed for viewing content round mefocce near a lighting source?
countrriop. floor sovwing?
<Col> <SI-The object <OBJ001> with a dark recannuuur shope is sdentified as a monitar, which hysically used for viewing content <SP> It pastiooed round surface, likely desk table. lighting
CCof= The object is a recycling characterized by its blue color nitablle for collecting recyclable mater inis. <SI= And pariiioned
<CoT> <SD> The abject wed for entering or eniting the room must be a door with hamdle positioned on right sude, <SI- It should be located
the dining eating countertop floor cnvering. These suanial categorical confiom larget </CoT>
left of the tynicallty
and anjucent
window covoring curtain. object idonsiiid fits this dese feseription, confirming correct choloe. <.Col>
lamp. align
This placement perfoctly description.
correct
<Answer = -B8001>> =tAmmer?>
AAamerr <ORR054- </Ansour>
AAseners <ORJ002- 4/Answor>
Figure 4: Representative qualitative results on spatial analysis tasks with CoT setting, (a) object analysis, (b) scene analysis, (c) relationship analysis, (d-f) implicit detection.
SCoT-Analysis and SCoT-Planning Table and Table summarize the performances of baselines trained with or without CoT supervision. Consistent with conclusions obtained in previous stud- ies (Wei et al. (022). , models with smaller parameter scales, such as 7B, exhibit minimal differences in held-in-text similarity metrics. For instance, the SCoT-Reasoner with CoT setting yields only modest enhancement over the non-CoT setting in text similarity metrics, with average improvement of 0.74% in ROUGE-L and 0.36% METEOR. However, training with CoT consistently leads to substantial gains in comprehensive scores, with average increases of 6.21% % 11.74%, and 10.02% in explainability. faithfulness and trustworthiness, indicating the response answers generated after CoT reasoning process align better with human-lit cognition and achieve higher accuracy.
We further observe that CoT plays an indispensable role in the implicit detection task, as it enables the model to infer the target object in implicit queries with statements that do explicitly mention object
8

---

Published as a conference paper at ICLR 2026
it possible to tuble if light above turn on, methods can improve visibiiity for cleaning task.?
<Cat> &S The kask gool clean elevated table despite the light above shows
various furniture cham, dressen, curtaies and shavves <St> Natual light can outer through windows open ourtains, which can provide suffictent tlamination if the overhend fight falls. <SI- Partable light sourcns like lamps or flashlights con be used in enhance visibiiity <S1> natural light iusufficiont, reposttioing ameself closer briginer ensures better viasbliity Thus, by utllcing alterativv light sources repasitioing, cleaning task remains feasible despite the now coneffuccionall overhead light. </CoT>
The
hed,
of
-Anawer's Yes, you can proceed by using a partable light souvce or opening nearhy curtains to increase natural light Alterntiveyy reponttion yourself cleser to brighter spots the
improve visibility while cleaning. siAnserrs
I'm standing near a seating area with two couches and a small table. want to prepare my worh setup and sturt video call on the screen pasitroned around workspace. How I arrunge and the equipment from this position achieve that?
<CoT> <SI>> The seating area includes two coucker and a small table, However, the current locution is not in the workepoce, so video call cannot be started direcily <St>> The screen is located on the desk in the workspace, and easily reachable from the souting area. <SI> Since there are no visible obstructions between the seuting urea and the desk with the screen, one can move directly to the desk to arrange the expippennt noeded for the video cull <S1> Than adjacting the camera angle of the veroon on the dask, ensires clear virihiiity during the video call. </CoT=
Answers From your position near woating area, move to the desk where the screen is located. Adjust the camera angle if neaded, then start the video call to communicate. </Annwer>
Figure 5: Results on planning tasks with CoT setting. (a) un-situated planning, (b) situated planning.
names before outputting the result. As shown in Table 5 the CoT setting significantly outperforms the non-CoT setting, with improvements of 23.6% and 20.7% of Acc@0.25 and Acc@0.50 metrics for SCoT-Reasoner, respectively.
5.3 QUALITATIVE RESULTTS
Figure 3 presents qualitative results of SCoT- Reasoner on the SCoT-Analysis test set, with addi- tional cases provided in the appendix. For object analysis in Fig. (a), the model begins by interpret- ing task objective in COT. then extracts reliable scene information, including object appearance, spatial po- sition, and nearby objects, and uses this evidence to infer the object's potential function. For scene analy- sis in Fig. 4 (b), its CoT thoroughly analyzes furniture composition and spatial layout in CoT to infer the room's functional type. For relationship analysis, the reasoning can also estimate inter-object distances and relative orientations directly from numerical coordi- nates, as shown in Fig. (c). Fig. 4 (d-f) presents qualitative results for implicit detection. When the query does not explicitly specify the object name, the CoT can infer its category, evaluates its alignment with the description across multiple attributes, and ultimately identify the correct target object.
Table 5: Quantitative results on implicit de- tection task in SCoT-Analysis.
Implicit Detection Acc@0.25
Method
Acc@0.50
312 VG Transformer
3.0
2.7
31> VG Transformer (Col)
12.5
10.9
Scene
6.7
Chat Scene (C6T)
26.9
23.2
Video 3D LLM
6.1
4.9
Video 3D LLM (CoT)
24.1
20.9
SCoT-Reasoner
8.6
7.7
SCoT-Reasoner (CoT)
32.2
28.4
For un-situated planning shown in Fig. (a), when the light fails to function properly during the cleaning task the reasoning observes a curtained window and suggests opening it for natural light or using a portable power source. For situated planning shown in Fig. (b), CoT generates a multi-step plan starting from the current location mentioned in the prompt to accomplish a complex goal.
5.4 GENERALIZABLE REASONING FROM SCoT
We further verify whether SCoT has truly learned transferable scene reasoning patterns rather than merely memorizing ScanNet scenes or language templates. Based on the MSQA benchmark (Linghu al. 2024), evaluate its generalizability from three perspectives.
9

---

Published as a conference paper at ICLR 2026
Table Evaluation results on the MSQA test under ScanNet and ARKitScenes environments. indicates zero-shot test. Methods annotated with "(-)" indicate the 3D-VL dataset used for model training (e.g.. "(SCoT)" "(3D-R1)"), then is evaluated on MSQA assess its generalization across
different 3D environments.
Method
Counting
Existence
Attributes
Spatial
Navigation
Others
Overall
MSQn-ec
(a) LEO
32.5
88.5
58.7
44.2
39.6
81.4
54.8
(b) MSR3D
32.3
93.1
50.0
46.5
54.1
75.6
54.2
LEO-VL
39.3
92.7
56.9
59.3
59.7
82.8
61.7
GPT-4o
32.3
79.3
79.0
37.0
31.7
91.6
52.3
Chat Scene (3D-R1)
35.3
41.9
34.4
45.1
25.2
86.2
43.1
(f) Chat Scene (SCoT)
35.5
64.8
44.5
42.7
39.8
84.8
47.6
(g) SCoT-Reasoner (SCoT)
41.5
54.8
48.9
43.0
91.2
54.4
SeeeeesssSssssss
(h) GPT-4o
37.5
55,2
48.1
37.7
21.0
60.7
41.0
(i) Qwen-VL.
43.2
46.0
44.5
25.3
26.9
70.1
39.7
(j) SCoT-Reasoner | (SCoT)
31.2
47.5
52.6
40.5
29.5
65.9
41.2
Q-A Format Pattern Generalization (ScanNet + MSQA-SccnNee).. Although the underlying scenes remain the same, the question formats, instruction modes, and description structures of MSQA-ScanNee differ completely from SCoT. The results in indicate that SCoT-Reasone (g), trained on SCoT dataset, achieves score of 54.4, which is comparable to in-domain supervised models like LEO (a) and MSR3D (b). Furthermore, it surpasses the zero-shot performance of large- scale GPT-4o (d) by 2.1 points. These findings demonstrate that the inference process does not rely on the language templates annotation frameworks used during training; instead, it suggests that SCoT has learned genuine scene logic and reasoning structures rather than mere textual patterns.
Cross-Scene Generalization (ScanNet - > ARKitScenes). Scenes in MSQA-ARKitScena dataset have a completely different environment, sensor characteristics, and geometric quality from SeanNet. As shown in Table 6 SCoT-Reasoner (j) trained with SCoT achieves total of 41.2 MSQA--ARKiScenne
outperforming the zero-shot capabilities of larger-scale LVLMs such as GPT-4o (h) and VI. (i). Notably, it secures +3.3 improvement in reasoning-intensive categories like Attributes, Spatial, and Navigation. Qualitative results in Fig. 10 of Appendix further reveal that the model can execute stable multi-step spatial reasoning even amidst major changes in geometric structure and viewpoint. This indicates that the supervision provided by SCoT facilitates the learning of spatial reasoning mechanisms that are transferable across domains, rather than structural memorization of ScanNet.
Compared Other Reasoning Datasets (SCoT vs. 3D-R1). To eliminate differences in models and training strategies, we trained thiry-party Chat Scene model (Huang et al.) 2024) using 3D- RI (Huang et al. 2025c) and SCoT under strictly consistent configurations and performed zero-shot evaluation the out-of-domain results show that SCoT significantly outperforms 3D-R1, particularly in categories requiring complex reasoning, such as Navigation (+14.6) that needs multi-step state tracking, spatial relationship understanding, and grounded inference. This evidence proves that SCoT provides higher-quality and more generalizable reasoning supervision.
SSQQ--ecnnNet dataset (as shown in Table 6 (c) and (f)). The
CONCLUSION
We introduced SCoT, million-scale Chain-of- Thought dataset annotated supervise 3D-LLMs with structured, ceene-grounded reasoning chain. SCoT covers diverse tasks spanning perception, analysis, and planning. Our experiments demonstrate substantial gains over counterparts trained without CoT annotation and supervision, especially for analysis and planning, highlighting the value of explicit reasoning supervision for advancing complex spatial intelligence in 3D environments. Our future work will involve improving the inference efficiency in real-time applications through model compression (c.g., quantization and lightweight distillation), hardware aardware-wwar aware optimization, and task-adaptive reasoning modes.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
This work does not involve human subjects, personally identifiable information, sensitive data. The datasets used are publicly available 3D-LLM benchmarks and self-constructed SCoT, and all data collection and processing strictly comply with their respective licenses. No potential conflicts of interest or ethical concerns were identified in relation to this study.
REPRODUCIBILITY STATEMENT
Our work consists of dataset SCoT and model SCoT-Reasoner. To ensure reproducibility, provide in the appendix detailed descriptions of the dataset construction pipeline, including data generation procedures and prompts, which enable others replicate the dataset. The model design details are also comprehensively documented in the appendix. Furthermore, we plan to release the dataset and related resources publicly to facilitate reproducibility and future research.
ACKNOWLEDGMENTS
This research was jointly supported by the National Natural Science Foundation of China (General Program) (Grant No. 42571521), the NSFC Basic Research Project for Ph.D. Candidate (Grant No. 424B2012) and the Wuhan Natural Science Foundation Project (2025041000010363))
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXi::2303.08774 2023. 3.3
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-graine tne-grained 3d object identification real-world scenes. In European conference computer vision, pp. 422-440. Springer, 2020. 2
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19129 19129919139, 2022. 2 A.1.1
A.5
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023. 3.3
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference on computer vision, pp. 202-221. Springer, 2020.
A.I.I A.5
Yanjun Chen, Yirong Sun, Xinghao Chen, Jian Wang, Xiaoyu Shen, Wenjie Li, and Wei Zhang. Integrating tffingof-thought for multimodal alignment: A study on 3d vision-language learning. arXiv preprint arXiv:200..06232, 2025. 2
Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang. Scan2cap: Context-aware dense captioning rgh-d scans. In Proceedings of the JEFF/CVF conference on computer vision and pattern recognition, pp. 3193-3203, 2021. A.I.I A.5
Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon- Moon-Young Lee. Tracking anything with decoupled video segmentation. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1316-1326, 2023, A.2
Jiajun Deng, Tianyu He, Li Jiang, Tianyu Wang, Feras Dayoub. and Ian Reid. 3d-llava: Towards generalist 3d imms with omni superpoint transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 3772-3782, 2025.

---

Published as a conference paper at ICLR 2026
Ziyang Gong, Wenhao Li, Oliver Ma, Songyuan Li, Jiayi Ji, Xue Yang, Gen Luo, Junchi Yan, and Rongrong Ji. Space- 10: A comprehensive benchmark for multimodal large language models in compositional spatial intelligence. arXiv preprint arXiv:2506.07990 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 2025. 2
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint
000.:22300000655, 2023. 2
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. A.3
Haifeng Huang, Zehan Wang, Rongjie Huang, Luping Liu, Xize Cheng, Yang Zhao, Tao Jin, and Zhou Zhao. Chat-3d v2: Bridging 3d scene and large language models with object identifiers.
CoRR, 2023. 4
Haifeng Huang, Yilun Chen, Zehan Wang, Rongje Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, al. Chat-scene: Bridging 3d scene and large language models with object identifiers. Advances in Neural Information Processing Systems, 37:1 13991-
114017, 2024.
5.4
Jiangyong Huang, Baoxiong Jia, Yan Wang, Ziyu Zhu, Xiongkun Linghu, Qing Li, Song-Chun Zhu, and Siyuan Huang. Unveiling the mist over 3d vision- Lannugg understanding: Object-centric evaluation with chain-of-analysie In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24570-24581, 2025a.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, 2025b.
Ting Huang, Zeyu Zhang, and Hao Tang. 3d-r1: Enhancing reasoning in 3d vims for unified scene understanding. arXiv preprint arXiv:2507.23-777 2025c.
5.4
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.067/4 067499 2025d. 2
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camney, et al. Openai ol system card. arXiv preprint
arXiv:2412.16720, 2024. 2
Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. In European Conference on Computer Vision, pp. 289-310. Springer, 2024. 2
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:
22199-22213, 2022. 2
Mingsheng Li, Xin Chen, Chi Zhang, Sijin Chen, Hongyuan Zhu, Fukun Yin, Gang Yu, and Tao Chen. M3dbench: Let's instruct large models with multi-modal 3d prompts. arXiv preprint
arXiv:2312.10763. 2023. 2
Xiongkun Linghu, Jiangyong Huang, Xuesong Niu, Xiaojian Shawn Ma, Baoxiong Jia, and Siyuan Huang. Multi-modal situated reasoning in 3d scenes. Advances in Neural Information Processing Systems,
2024.
2 5.1 5.4
12

---

Published as a conference paper at ICLR 2026
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a. 3.3
Chengzhi Liu, Zhongxing Xu, Qingyue Wei, Juncheng Wu, James Zou, Xin Eric Wang, Yuyin Zhou, and Sheng Liu. More thinking, less seeing? assessing amplified hallucination in multimodal reasoning models. arXiv preprint arXiv:2505.2/523, 2025a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-33916, 2023. 2
Ryan Liu, Jiayi Geng, Addison J Wu, Ilia Sucholutsky. Tania Lombrozo, and Thomas L Griffiths. Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse. arXiv preprint arXiv:2410.2133:
2024b. |
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.0178. 2025b. 2
Maria-Flavia Lovin. Lims to support a domain specific knowledge assistant. arXiv preprint arXiv:2502.04095, 2025. 5.1
Ruiyuan Lyu, Jingli Lin, Tai Wang. Shuai Yang, Xiaohan Mao, Yilun Chen, Runsen Xu, Haifeng Huang, Chenming Zhu, Dahua Lin, et al. Mmscan: A multi-modal 3d scene dataset with hierar- chical grounded language annotations. Advances in Neural Information Processing Systems, 37:
50898-50924, 2024. 2
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022. 2
A.5
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. A.2
Kun Ouyang, Yuanxin Liu, Haoning Wu, Yi Liu, Hao Zhou, Jie Zhou, Fandong Meng, and Xu Sun. Spacer: Reinforcing milms video spatial reasoning. arXiv preprint WxXiv:250.00800 2025. P
Yingzhe Peng. Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.075:3 2025.
Zhangyang Qi. Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision-language models. arXiv preprint arXiv:2501.01 00128, 2025. [.]]
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. arXiv preprint
1.Xi::2110.0315,,, 2022. A.2
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao. Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable rl rl-style large vision-language model. arXiv preprint wwXiv:2504.071.1 2025.
Tai Wang, Xiaohan Mao, Chemming Zhu, Runsen Xu, Ruiyuan Lyu. Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 19757-19767, 2024. 2
Zehan Wang, Haifeng Huang. Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data-efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint (xXXi:2308.08769
2023. 2
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of thouhht prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-24883 2022. 5.2
13

---

Published as a conference paper at ICLR 2026
Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan. Spatial-mllm: Boosting milm capabiliities in visual-bases spatial intelligence. arXiv preprint arXiv:2505.23747, 2025.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointll:
Empowering large language models to understand point clouds. In European Conference on Computer Vision, pp. 131-147. Springer, 2024. 2
Jianing Yang, Xuweiyi Chen, Nikhil Madaan, Madhavan Iyengar, Shengyi Qian, David F Fouhey, and Joyce Chai. 3d-grand: A million-scale dataset for 3d-llms with better grounding and less hallucination. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29501-29512, 2025a. 2
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503. 10615, 2025b. 2
Jiahui Zhang, Yurui Chen, Yanpeng Zhou, Yueming Xu, Ze Huang, Jilin Mei. Junhui Chen, Yu-Jic Yuan, Xinyue Cai, Guowei Huang, al. From flatland to space: Teaching vision-language models
to perceive and reason in 3d. arXiv preprint arXiv:2503.2299 2025. 2
Yiming Zhang, ZeMing Gong, and Angel X Chang. Multi3drefer: Grounding text description to multiple 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15225-15236, 2023.
A.5
Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. advg-transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2928-2937, 2021. 4
Lichen Zhao, Daigang Cai, Jing Zhang, Sheng, Dong Xu, Rui Zheng, Yinjie Zhao, Lipeng Wang, and Xibo Fan. Toward explainable 3d grounded visual question answering: A new benchmark and strong baseline. IEEE Transactions on Circuits and Systems for Video Technology. 33(6):
2935-2949, 2022. 2
Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d llm: Learning position-aware video represen- tation for 3d scene understanding. In Proceedings of the Computer Vision and Pattern Recognition
Conference, pp. 8995-9006. 2025. 2 A.5
Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. arXiv preprint arXiv:2310.0677: 2023. A.2
Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu. Llava-3d: A simple yet effective pathway to empowering imms with ad-awareness. arXiv preprint wxXiv2240.188255 2024. 2
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
In the appendix, we provide more statistical information about SCoT dataset, detailed introduction of SCoT-Reasoner, and quantitative results in SCoT-Perception dataset. Moreover, we present additional visualization results to further demonstrate the effectiveness of the CoT setting.
A.1 SCOT DATASET
A.1.1 STATISTICAL INFORMATION OF SCOT DATASET
Table 7: Statistical overview of SCoT dataset. We report the number of samples, average question length (Que. Len.), average CoT length (CoT Len.), and average answer length (Ans. Len.) across different task categories. Note that in these tasks, the answer for implicit grounding task is only an object placeholder, hence the average word length of the answer is always one.
Words
Tokens
Task Category
Samples
Que. Len.
CoT Len.
Ans. Len.
Que. Len.
CoT Len.
Ans. Len.
Spatial Perception"
270k
22.7
0.0
15.7
28.7
0.0
28.1
Implicit Detection
65k
35.0
51.4
1.0
39.8
59.4
7.0
Object Analysis
60k
27.1
85.4
45.5
29.9
105.4
53.3
Relationship Analysis
55k
25.7
32.1
27.8
92.1
42.0
Scene Analysis
280k
16.8
82.6
36.0
19.2
105.9
44.1
Un-situated Planning
210k
22.3
92.0
31.6
24.9
114.9
37.1
Situated Planning
180k
37.7
86.1
33.6
42.4
105.7
38.9
As shown Table 7 we present the detailed statistics of SCoT dataset. Spatial perception tasks are primarily derived from simple Question-Answer datasets (e.g.. ScanRefer (Chen et al. 2020). ScanQA (Azuma et al.) 2022), SQA3D (Ma et al. 2022). Scan2Cap (Chen et al., 2021). and MultiDDRefer (Zhang et al. 2023)), which are designed establish basic spatial understanding for 3D LLMs. However, these spatial perception tasks feature short responses, with an average of 15.7 words and 28.1 tokens, and contain explicit CoT reasoning. In contrast, spatial analysis (object analysis, relationship analysis, scene analysis) and spatial planning (un-situated planning and situated planning) tasks with text-based output produce substantially longer answer responses. The shortest responses in these categories average 27.8 words and 37.1 tokens, supported by explicit CoT annotations that provide abundant ceeneggoounded evidence and transparent reasoning processes.
A.1.2 PROMPT TEMPLATE FOR DATA GENERATION
As shown in Fig. and Fig. 7 the prompt templates are guided through carefully structured prompt that integrates task instructions, additional spatial information, and multi-vier images. The system prompt defines the overall role of LLM, requiring it to produce natural language question, a coherent reasoning chain, and a precise answer in strictly formatted manner. The task description emphasizes the need to reason over spatial relationships, such as Euclidean distance and relative orientation, while ensuring clarity and logical progression in the output. Additional structured information, including object identifiers, coordinates, and bounding box dimensions, is incorporated to provide model with explicit geometric cues for accurate reasoning. Finally, this enriched textual context is paired with multi- view images, ensuring that the model not only grounds its reasoning in visual evidence but also delivers results that are both interpretable and verifiable. This unified prompting strategy enables reliable 3D spatial understanding and consistent generation of detailed CoT explanations.
A.2 SCOT-REASONER FOR MULTI-MODAL SPATIAL INPUTS
In this paper, we introduce a new 3D-LLM architecture named "CoT-Reasonee to process multi-modal inputs (c.g., text, point cloud, and video) and support spatial perception, analysis, and planning. We incorporate the "Object-Redd -eee--eeeaatit-eeeeeeeee Scene" refinement module into SCoT-Reasons to enhance the model's ability for spatial understanding.
15

---

Published as a conference paper at ICLR 2026
You are or odvenced Al spatial reosoning system capable genercting object level question, detailed onewers and reliable choir-of-Thoughh (Co1)
content__rompp # This needs to be changed depending the different tosks.
[1) You must complete thres steps:
Step
Generate multifaceted suestion that ask about the torget object. without
class names, semontic cotegories reeeeee,ss,ee numbers.
(+) Generate question
the spatial peothonshp (incuuding distonce and reletive direction) between two objects, using abject nome and location.
Shep 2:
Generate o precise onswer for the corresenndng questien, may include:
plousible inf inference objects furction (ii) space property oaject Euclidean distance between objects and their relatie spetial orlentation (eg. "Object is meters to left and alightly in front Object 8*)
object nane
appeor
Step 3: Generate a relibble CoT using okject name, and detailed calculetion or reasoning analysis proceas should be emphasizzed in CoT.
identiffy the tesk goal it) andlyze current stane or construints and extract the object and scene content. Iil) leverage scene infornction to form the basis for
spatial analysis planning in) finelly. prowid answer, Nate,
soone informction (eg. object properties, spatial relationships, structural
attributes) utilized, nust be cited with on tay proceding the corresponding sentence.
[2] You must follow these rules.
The queat ian and onswer should not fecus on phroses like "This mmye. or AAridl view of.." or "The ebject highlighted by the red box.". Avsid using terms like highlighted object' in question answer.
inclide explonations nete-comeentary about tesk inself. Finally, sutput question needs pr preceded by a correspanding dentiffer for mork QQuestion and the of the question needs hav on </Questions identifier mork, the output answer needs to be precaded by a comnesonning identifiar mark cArewers and the and of the onswer needs to have </Arpwrr- identifiir mork. The output CoT neads to be preceded conrasponding identifier mark ido and the end of the CaTl needs have </ To identifier mork Meess coreful to have words such to words. Please note that
"top down view' or 'BEV Image". Note that keep longth orswer 20 words. Note that keep longth the question cornot refer to object class nomes, senontic cotsgories, or ID rumbers
additional_ine Leattnaliiofoomm information prommpt # This needs to be changed depending the different tasks. The "Object Ib' and correspancing 'Nome' of the target jbjtct highlighted by green box in wwwhhiieew__naaeeeeeeeee and the "Object ID' and corresponding 'Nome' of the neughborhood object highlighted by red box in olultiView Images.
The center str(Ginttnce__ borudd__ouddnnbbe) atrtenstorce__hiie * "m.
coordinates the torget object " strikinstance_ccom cotprmse_coddddtteeeoe ''n The length. width and height bounding one
The span the torget object is str(nox(inetonnt__ee bounddogooodddd
height the target object
The center point coardinates of the reighborhodd object are *+ str(crel wttssiintttecoooooooooo coordinatess)) + 'n. The length, width and height of bounding box are" str(mnei, instunntte boundingbso)) "'n span of the neighoorhoed object "* strimanl nei, rei_instance__aa banniiggoon The height of the naihtarraadd object is str(+ nei _nc.naccccchhhhth)) 'm.
messoges:
"role" "systen" "content" systen_prompt
"roke"/sser" "content" content, promptt additional aaddtione__nformmttinn informatiinn_pranee
"typ": mage "itage_urt" ('url' inrg'i joegbase64, (base54 [udtses__Mdtiiiie.. Inage"")
Figure 6: Prompt templates for data generation on object analysis and relationship analysis tasks. Note that annotation statements are represented in green font, and non-textual (multi- modality) variables are represented in blue font.
system_ prompt:
You are advanced AI systen espoble of generating complex reasoning question, onswer and reliable Chain-af- Thought (Col) based on the 3b spatial configuration. Please generats question and pairs (eg.. Scene layout retionaiity Scene spotial copocitt. Scene funchionality and Scone redesign suggestion, Score planning..... ..) that perforn scene level reasoning and planning.
content_ partent_ppppttttt. # This needs to be changed depending on the different tasks. You must complete three steps:
Step Senerate the questions that invovve higher-orden reasoning about the entire scene laysut. such os:
(1) Scone loyout rationality: Is the currest furniture or object anrangement practical efficient?
(2) Scone spatial capacity: How nuch usable unused space remais??
(3) Score functionality What functions (e dining, diesping, working) does the soene currently support?
(4) Scere redesign suggestion: What charges coul be proposed to improve the scang?
(5) Somne plaenin Balee on the scone context and stote standing sitting, halding dbject, welking), accomplish highnleoel task, such cal exercising.
cooking, clearing, relaxing, working.
Step 2: Generate preciss cntwers, and the onswers must be detailed, ransoned, and based on spetid emargements, scere utilization, and functionality. The
answer may include information such as object deraity, spatial zones used, renaining uboble space, and suggestions for improvement.
coloslation reasoning onalysis precess i) iddetlify taak gool il) unalyze curnent state or construiets and extract object and scere content, lil) levenage scone information form besis for speiial onalysis or planning v) findlly, provide the enswer. Nate that whenever soone information (s.g. object properties, spotial reletionships, structurol attributes) is utilized, it must be cited with on -DD toy proceding the corresponding sentence.
melnble using object nome,
amphosized in CoT
[2) You follow these rules:
The question and enswer should not focus on phroses like "This image." or "Aerid view of.. or "Tee object highlighted by the red box.." Aveid using ferms like highlighted object' in the question and answer. Do not include explenations or meto commentory about the tesk itself. Finally, the output question needs to be preceded corresponding idontifier mark (eig. -Question Scere loyout rationality)) and the end the QQuestion: Scere loyout rationdlity)) the cutput answer needs to preceded a corresponding idertifier mark (e.g. sivwer Scene laysut not tytlonality)) the and of the onswer needs to have on fien mark (e.g. </Arswer Scone layout ratherdity). Note that keep the length of arswer to 40 to 50 words, Pleose nete that the question connet refer object class names, semantic categeries, or ID numbers.
needs have identifier mork
additional information prompt: # This needs to be changed depending on the different tasks, There ane <object_io, sobject 2t... robject_N this scanario. The Object ID' and correspondingg 'Name' of the target object highlighted by green bax carrespanding 'aane'
<MultiView_ Iwwgge the "Object adaition, there interrogative statement about task the onginal dateset (s.g. SQAIb, 3D-LL datsset) that used as a reference for information extruction. The query is +Question_Rete Reference, The anaver is vtnswer_Rate Reference.. messages:
reighborheed object highlighted red box Miuutiiiee Inajje.
"role" "Systen" "cantent" systen_prompt
"roee" "content": content content_prompt- additionel additinnl_iffommation www.o/nnttoooprrrre
"Iyp"! inageuur "inaye_urt" ('url" imngee/ hatl'nnnnnjgggaaaeeee jpegbbas644, (base54_ Muddiiiio___ncee]
Figure 7: Prompt templates for data generation on scene analysis and spatial planning tasks. Note that annotation statements are represented in green font, and non-textual (multi amulti-modality variables are represented in blue font.
16

---

Published as a conference paper at ICLR 2026
3D Scene
Object Proposals
Video Frames
Segment
240k
Perception Data
Images Extroction
Language Instruction
460k +
3D Encoder
2D Encoder (DINO v2)
I stand near on entrance, intending to prepore
Reasoning Data
meal and than relax by sitting on couch. this room?
Obbjec-Reelttonshiin Refinement Module
Tokenizer
3D Projector
2D Projector
390k + Planning Data
Large Language Model
<CoT> <SD The scene contoins on entronce area near the kitchen counter and Adjacent to this ketchen cobinets, which suitable mools preporation.
space, there is couch and coffee table, offering confortable spot to sit and nelex. These objects cleanly accassible from the entrance, making transition feasible. The uncrowded floor space and lock of obstructions further support case of movement between these zones, Consequently, both prepaning meal and then sitting couch can be accomplished from the entrance location. </CoT>
Frozen Parometer
<Answer> It is feasible to prepare a meal using the wide counter and cabinets nearby, then
the soft sesting to and relex comfortably. s/Answers
Starting Position
Trainable Parameter
Figure 8: Model architecture of Rees-Reas. SCoT-Reasoner receives language instruction, object
proposals segmented from 3D scene and video frames as input, then performs step-by hep-by-step reasoning and analysis based on the object-grounded or scene-grounded facts, and ultimately generates reliable accurate and verifiable answers.
Text Tokenizer, We employ Vicuna-7B to process text token sequences. An embedding look-up table is used to map these tokens into text embeddings.
3D Encoding. To extract geometric and spatial attributes from 3D point clouds, we utilize Mask3D (Schult et al. 2022) to obtain 3D object proposals. For each object P,, Uni3D (Zhou et al. 2023 is used derive object-centric 3D feature vector FSD 1
Video Encoding. For input video frame, we adopt open-vocabulary video segmentation model DEVA (Cheng al., (023) to extract 2D object proposals. For each detected region, we compute object-centric 2D feature vector F:D
E IR 1x 1924 using DINOv2 (Oquab et al. 2023).
Refinement Module. We design the ORS) refinement module that integrates absolute spatial locations and pairwise relational cues.
" Obbject-RRlltionnsip-Seeee
scenne"
Spatial Graph Construction. Each 3D object proposal is encoded by Uni3D into E:D e 1012 while each tracked 2D object from video frames is encoded by DINOv2 into F?2 R 1024 We retain each object's 3D bounding box B; E IR 1x6 We extract the object-center coordinates in gixs and compute their pairwise differences to obtain a 3D offset vector, which is processed by linear layer to generate 3D offset embedding in R 1024 In the spatial graph, each object is represented as a node, while the offset embeddings constitute the edges.
ORS Representation Optimization. Given the spatial graph constructed above, we apply Graph- Transformer to refine object-centric representations. The attention mechanism augmented with a learaalle spatial bias derived from offset embeddings, enabling each object to selectively aggregate information from spatially relevant neighbors, while preserving direction and distance cues. The spatially-refined feature Fi E TR 1=111021 obtained from the Graphrranformmrr is concatenated with the original F?D E3D to produce an enhanced representation FOOn E IR 1x2204 The set of enhanced features {FORS) ns N forms the "object-relationship scenee representation. which is subsequently projected into the L.M embedding space via modality-specine projectors.
17

---

Published as a conference paper at ICLR 2026
(a) Object AnjecyA Analysis
(b) Relationship Analysis
(c) Scene Analysis
GPT
0.300
0.703
3,000
0.209
0 8.99
A75
GPT
1,000
0.491
0:597
925
DeepSeck Qwren
0.304
1 0.00
0.171
0.289
1.000
0.285
Quien
0.491
1,000
0.440
-025
-909
DoepSook
DeepSSeck
0,763
0.692
0.285
1,000
0.:97
0.440
1,000
-1.0
-0.0
GPT
Qwen
DeepSeak
GPT
Qwen
DeepSeck
GPT
Qwen
DeepSeek
(d) Un-situated Planning
(c) Situated Planning
1,000
0.386
0:
0.737
GPT
1,000
0.652
0.715
6:55
Qwen
1,000
1.000
0.654
4.25
-0.5
DeepSock
DappSeak
0.348
0.654
-11
-0.1
Qwen
DeepSeek
Qwen
DeepSeek
Figure 9: The correlation heatmap of LLM scores derived from ChatGPT-4.1, Qwen, DeepSeek across object analysis, relationship analysis, scene analysis, un-situated planning and situated planning tasks.
A.3 IMPLEMENTATION DETAILS
We build SCoT-Reasoner upon the pretrained Vicuna-7B (v1.5) backbone, with additional projection layers for multimodal inputs. We add scene and image tokens to the input sequence and enable object identifiers, and the maximum number of objects capped 200. All fine-tuning process targeting the attention projections and feed-forward components is conducted with the LoRA layers (Hu et al. 2022). The LoRA rank to 64 with an value of 16 and dropout rate of 0.05. Both the textual embedding layer and LoRA parameters are trainable, while the image projection module remains frozen. For optimization, we use the Adam optimizer with learning rate of x 10-3, 3 /1 0.9, 32 0.999, and weight decay of 0.02. A warm-up phase is applied for the first 10% of training epochs.
A.4
CORRELATION ANALYSIS
As illustrated in Fig. 9 we present the correlation heatmap visualization across the three heterogeneous evaluators (ChatGPT-4.1 14.1, Qwen, and DeepSeek), demonstrating that their rating remain independent. Despite receiving the same information (answer, ground truth, and structured scene evidences) and rating prompts inputs, the scores derived from three evaluators show moderate but not absolute inter-evaluator correlation with the most values in 0.3 to 0.7, indicating that no shared model -pecciip preference dominates their evaluation process. By averaging the three scores, evaluator-specins bias are further reduced, yielding more stable and objective assessment.
A.5 QUANTITATIVE RESULTS SCOT-PERCEPTIC DATASET
As shown in Table 8 report the performances of #CoT-Perception test set. Note that here we do not apply the SCoT-style reasoning; we only evaluate and demonstrate the baseline perceptual abilities of different models.
and other baseline method on the
SCoT-Reasoner achieves the best accuracy across both single-object and multi-object grounding benchmarks. It reaches Acc@0.50 of 53.4% in the ScanRefer dataset (Chen al. (022)). while it achieves an Fl 50.50 55.7% in the Multi3DRefer dataset (Zhang et al.) 2023). In
addition, further evaluate the methods simple QA tasks such as ScanQA (Azuma et al. 2022
18

---

Published as a conference paper at ICLR 2026
Table Quantitative results on spatial perception tasks. "SCoT-R" the abbreviations for "SCoT- Reasoner". "(CoT)" means the method trained with "Full SCoT Setting", and "(w.o. CoT)) means the method trained with "Answer-Only Setting" without CoT reasoning.
ScunRefer
MRlfe
Scan2Cap
ScanQA
SQA3D
Method
Acc@0.25
A.c@0.50
F1
F1
CIDER@0.5
B-400.5
CIDEr
B-4
EM
3D-LLM
30.3
59.2
7.2
20.4
Chat 3D
53.2
6.4
Chat 3D
42.5
38.4
45.1
41.6
63.9
31.8
87.6
14.0
21.2
54.7
1.:3DA
76.8
Scene-LLM
80.0
11.7
25.6
53.6
Chat Scene
55.5
50.2
57.1
52.4
77.1
36.3
87.7
14.3
21.6
54.6
Video 3D LLM
58.1
51.7
58.0
52.7
83.8
41.3
102.1
16.1
30.1
58.6
SCoT-Reasoner
58.4
53.4
60.1
55.7
79.6
38.2
87.9
15.0
23.0
55.8
Table 9: Ablation experiments on different CoT settings and <SI> levels. "R.". "M.". "Exp.". "Fai." and "Tru."" are abbreviations for "ROUGE-L.", "METHOR". "Fxxlainabillit"" "Faithfulness" and "Trustworthiness" respectively. "CoT.Len.." means the average token length CoT, and "<SI>" means average number of <SI> identifier in COT. Relationship analysis is excluded from this comparison because its CoT mainly consists of numerical computation process.
Task Category
R.
M.
Exp.
Fai.
Tru.
<Sb
CoT.Len.
Time (s)
Object Analysis (No CoT)
27.34
15.62
6.67
5.59
5.89
6,51
Object Analysis (w.o Sce. in CoT)
26.93
16.04
6.92
5.94
6.27
1.83
64.99
10.97
Object Analysis (w.o. Obj. in CoT)
26.85
15.34
6.43
5.37
5.72
1.91
85.36
11.71
Object Analysis (Full CoT)
27.22
16.17
6.15
6.41
122.45
15.98
Scene Analysis (No CoT)
22.59
14.68
7.82
7.32
7.45
5.08
Scene Analysis (w.o. See. CoT)
22.50
14.37
7.67
7.40
7.37
2.45
66.98
10.26
Scene Analysis (w.o, Obj CoT)
22.91
14.50
7.35
7.60
1.97
$1.55
9.15
Scene Analysis (Full CoT)
23.48
15.29
7.95
7.55
7.68
3.42
95.57
13.30
Situated Planning (No CoT)
24.21
12.37
6.64
6.09
6.30
6.42
Situated Planning (w.o. Sce. in CoT)
24.04
11.96
6.70
6.21
6.55
2.56
75.89
12.53
Situated Planning (w.o. Obj. in CoT)
25.37
12.54
7.10
6.85
7.07
1.99
49.67
10.90
Situated Planning (Full CoT)
25.13
13.06
7.38
6.94
7.14
3.56
99.89
13.95
Un-situated Planning (No CoT)
28.01
18.35
6.97
6.93
6.76
6.15
Un-situated Planning (w.o. Sce. in CoT)
28.34
17.89
6.82
7.17
6.98
2.90
96.36
12.81
Un-situated Planning (w.o. Obj. in CoT)
28.85
18,40
7.20
7.05
7.31
1.82
$2.31
9.57
Un-situated Planning (Full CoT)
29.04
18.11 1
7.27
7.29
7.15
3.91
120.28
14.27
Scan2Cap (Chen et al. 2021), and SQA3D (M et al. 2022), where the answers are relatively short. these datasets, SCoT-Re RRRasoner achieves competitive performance, ranking slightly below Video 3D LLM (Zheng et al. 2025). which benefits from temporal cues and richer video-based training.
A.6 ABLATION STUDY FOR DIFFERENT COT SETTINGS AND <SI> LEVELS
As shown Table 9 to have a deep understanding SCoT reasoning process, we explore the effects of object-level reasoning and scene-level reasoning process. Specifically, we categorized, filtered, and ablated different types of <SI> annotations. "w/o Obj" denotes removing <SI> for object-centric information, while "w/o Sce" removes scene-level analysis <SI> from the CoT. The quantitative results indicate:
Object-level reasoning is critical for object-centric tasks; removing it consistently reduces perfor- mance (e.g., METEOR 16.17-+15.34 Explainability 7.04-6.443 Faithfulness 6.15 +5.37).
Scene-level reasoning is essential for holistic scene understanding and planning, with substantial drops when removed (c.g., Scene Analysis METEOR 15.29-> + Situated Planning Trustworthi- ness 7.14+x6.55)
19

---

Published as a conference paper at ICLR 2026
(b)
a
Question: For front of me, there's a line books and brown wooden bonkshelf. To my left, of a middle distance, there's a white bood rark and a Hight brown top, black legr stable 17s- What the inside <bookshelf-22 my d'ccicck
Question: Another black schair 260- of my 5 o'clock I am sitting on the chait What object black and can be used for säting at a'ccooc??
Quesiion: am removing to the cubinet. At o'clock, closed gray cabinet is prosent At 8 a'clack. there is prodommnouty white monitor and walt there closet door the scone?
<books-75>
Col: <S1> The chair positioned the o'dockk directioon relatie the seated person, <SI> To the right of this chair, there an object that marches the description of being black and miltable for sitting <St> This abject is identiftedd chair based and color, confirming it be used for sitting
voene contaier a closed goy cabinet positioned
o'clock prodonmuantl white monitur located at o'clock, with wail nearty. <SD> There is no visible closet door in the room, as is not adjacent to these objects near the walls Tharefore the presence of clovet door unlikell based spotial arrangement and object velatoonshis
Col: <SD> The scone shows a blue
bonks and brown woesen bookshell. white book rack and light brown top, black legs pasitioned the baokcshelf
shape
books inside the bookshelf are arranged neatty on shelses. indicating organized straaee for reading or study parpones.
The obyect at your 7 o'elack is chair which is black and designed for string. It is positioned the right of the providing comfirtabll. (Ground Truth: chaie)
Answer:
Answey: books incide the bookshell are likely wsed reading studying, they seatly arranged on thebies. This semup supports organization and retrioval of matwrials for eduraational parposes.
Aarwer: There is no viasbic closst door it the rcene however, there is closed gray cubinet at 12 a'cluck and a while mondtor and wall 8 o'clock. (Ground Tracke No.)
seat where you are
place
Gronnd Trunh: Reading.
(d)
Question: cubinet made wood with closed. What texture of chair?
the sefu, At 3 d'cloct. white stands vertiaal,
Question: To my right, for away. there's a storly uard brown leather sofa with smouth serture I am sining on the is the cubinet higher lower than the shelf my right?
CoT. The goal to desevibe tenure the chair the o'clach positin. (2) The tanget object is chair.
brown ioather safa with right of shelf -SI= The cabinct is located higher than the shelf on the right side of the sofo This spetial relationship confieme that the cabinet is elevated above the sholf, making the higher object.
CoT: The scene
smooth texture posiiiouen
(3) <SI> The chair is made of wood with a smooth seture, indicating it is likely Saayyyyo uphokseeeed or cushinned (4) <SI= ht varioaa orientation suggests it is designod for sitting rather than reclining, Therefore the chair has smooch woodon tesure tyiccal of upholteered aushioned warting farnitue
The cabunet right side safa. confirutin elevated placement relative shelf. (Grosnd Truth: Higher)
Anzver:
higher than the shelf
The object chaur with smooth woodor sanee, likely upholatoodd oukiowed. Posstiuns verncully near corner. serves youting furminrre moam for sitting rather reelining (Ground Trush Smooth. )
Answer:
Current Location
Figure 10: Qualitative generalization results of SCoT-Reasoner on different scenes in MSR3D (MSQA) dataset. (a-c): ScanNet, (d) and (e): ARKitScenes. Note that SCoT-Reasoner trained on SCoT dataset.
However, the metric improvements from CoT reasoning comes with higher inference latency. Infer- ence time increases by 2.0x-3.2x, depending on task complexity. For example, the inference time of scene analysis increases from 5.08s 13.30s, and un-situated planning increases from 6.15s to 14.27s on a single A100 GPU. These tasks inherently require deeper reasoning, making the additional overhead acceptable for offline planning but challenging for time-critical robotics applications.
A.7 ADDITIONAL QUALITATIVE RESULTS
As illustrated in Fig. the additional qualitative results of object analysis indicate how CoT enhances the interpretability and reliability of responses in reasoning-oriente providing a direct answer, SCoT-Reasoner first decomposes the query into structured reasoning steps that explicitly reference object attributes, spatial relationships, and contextual cues within the 3D)
tasks. Rather than
20

---

Published as a conference paper at ICLR 2026
What are the main visual features of the strac ture posttioned benoath the sink area, and how
is sizwated relative to wearly applinnces?
<CaT>> <St> The aructure in idenssiedd as o bitchen cabinet characterized by itr rectongulor and wooden finish. <SP> It is passtioned directly bonouth the sink indicuting fuactional placessett for convesient
atensils cloaning supplies =SI Noarby
appliances
a refrigeaatrr on the other sitsating cobiner cluse proxiniity cooking and fond storage rones within the kitchon loyout, This spatial relationship highiights its role in organizing hitchen essentials. </Col>
stove one
<Anserr> The object is a hitchen cabinet with rectangular shape and wooden finish. It lscated directly boneath the sink area, adjacent to a stove one side and a refrigerator on the other. It function likely pequaration. <iAnswers
utenvils and cleaning supplier eary during meat
What are the vinaal characteristics of the small dart how placed on the table, and how positioned relation tn the surroudding soating surtiing??
small dank trom placed on a soble and its position velative to nurronsting seating fornttre (2) The turget obyect laptop. (3) =SI> The laptop has a sleek, rectongular shape with black surface and slim profile. eSt It rests on a light-colored table sarrounded by cashioned seatling furninue such as sofas and chairs. (4) By observig the scene, the laptop's ' placement on the table indicater it server portable electrunic devicn for wook or untertaiament in this living space. </CaT=
<Cof> The tukk is to deseribe the visua characteristics
-Anwwer> The object slesh, rectunguiar device with black surface and stmw proftic, idvaiffind lapoop It serving as a portable electroic device for work entertainment this livingg space. </Arewer>
light-cntursi tubie amndst owhiowed wouting farniture Bieely
What are the key foatures of the wondos furuiture piece with three sections near the adge of the and how does relate to itr surronnding elemenns??
CoT> (1) The unk gont is to describe the key fentures of wooden farminre piece with three rections near the eige of the room and explam its velationskp surrounding elements. (2) The target object bookshelf (3) <St= The bookshelf is medimn-sized rectanpuios in shope, and construcied from wood with multiple thelves for storage. 'SD= positioned along the wall at edge, adjacent seating area consisting couches chairs. By abeersing to size. shape, material, and location relutive other furnisings the baokshelf's design and flunctio are clearly deduced </CoT>
<Auswer> > abject a medium-sizeed rectangular wooden bookshelf with sultppe shelies. placed againes the wall close to seoting area that includes couches and chairs. Hos fanction is likely to store books or decorative liems within reack of those soated nearby. ALAnnwer>
What the distieguishingg footwee the flat horisonaa surfuce in the contey, and how postiionel relutuse reating anound it?
<CoT> <SI= The abject is table characteized by its flat, rectangolar shope with light- colored nerface. -St= It centrally poemined within the surrounded by malliple office cheirs evenly spaced awoud <St> arraegement nuggests shble workspace or dining area, faeilitaingg group interaction and accessibility from all seated pasitious. </CoT=
communal
"Answer> The object is rechangular sable with light colored surface, surounded by mudtiple
office chains evenly spaced araun it Poszioneed contrally in the room, likely serves
workspace dining area, faclinuting group interaction and easy nccezs from all seated poritions. </Anower>
Figure 11: Additional qualitative results of SCo Reas-Rees on object analysis task.
scene. For instance, when asked about the features of a wooden furniture piece, the CoT guides the model to progressively identify its shape, size, material, and spatial positioning relative to surrounding objects, before coneluding its functional role. This stepwise reasoning not only ensures that the answer is grounded in observable evidence but also makes the logic of the response transparent in the object analysis task.
As illustrated in Fig. 12 the additional qualitative results of scene analysis demonstrate that CoT reasoning enables 3D LLMs to perform fine-grae ane-ggrined scene understanding and logical analysis. Given a raw 3D environment, SCoT-Reasoner first extracts structured information by identifying key spatial elements such as furniture composition, accessibility pathways, and object arrangements in the scene. Then, CoT guides the reasoning process by decomposing the question into interpretable steps, aligning the extracted spatial cues with the semantic intent of the query. For instance, when asked about improving workspace functionality, SCoT- not only detects the suboptimal desk placement and lighting but also infers practical modifications, such as repositioning furniture and optimizing light exposure. Similarly, when evaluating whether seating layout promotes social interaction, the model interprets both the geometry of furniture and the affordances of open space, and then articulates a balanced judgment regarding comfort and accessibility.
21

---

Published as a conference paper at ICLR 2026
What practioal modifisstions can be to inprove efficincy, comfort, or functionality this spatial configuration?
<CoTs <St> The room containe bed desk, chair bookshelf. dresset, and varvus scatlered items on the floor <SI> The desk is posiiioned away from the window liniting natural light exuowee which con affect comfort points, potentially obstructing smooth movewent <SD> Juvoduccng nuiifanctional furiiuue could cppiiiiee space waage without overounding,
productviity <Si> There is visible clutter the floor entry
Eshancing lighting by reposttoning the dosk closer window or adding lauga would imgrove confort worksaaee functionaliny These adjartments onhance suatial efficioncy and wsr experieace.
<oCaT>
AAureer: intruduce sultifunciooal fureinue to masimice uschle space, repusition desk cloner to natual light for confort redice chatter near eary points, and ethance lighting to luprove conffor and funciinality </Anwwe>
the curvent arrangrmen of seating and tables efficient for social interaation while allowing oury movemon within the space?
<Col>
-SP> The souting arangement consists of muitiplc couches and arachates postiioned auple
<St> open floor space betnees these faraiture pieces. easwring unobitracted pathoays for eary meremment throughont the room. <SI> The spacing avouds overcemating while maintaining sutimate grouptngs baiancing comfort and accersbiity effectively. This layout supports both social engagement and smooth novigation wilion the ypuce. </CoT>
cofire table, pronoding facs-toffrr
=Auwerr> > The seating and tables are araaned to promote face-tu-face interaction, with clear pathwaye alloninn case movement. The avads overronding engagmmo and efficinnt sovigation through the Ownall, layout bulanoes comfout accessibiiigy well. <lAnneer>
bouh sociol
What prinary activities does this room support baned on the arrangement objects and furniuue??
<CoTs <SD> The contaies contral table suroounedd by mudtple chaire, indicating
hayboard wall supports cosputer-based actiiitiss such working atddying These clemosts togother tham room designed primarily for collotorative todk involving discssion. presenations. and computer we <Cols
conducive group discussions
<SE- whiteboard
presentations brainstoringg ressions. <SD
suggest
The num mainfy supporty collaborative work or meetings, with reating arraged around central suffuce likely wed for discussoon or presentations Addiiiional equipment like monitors and keyboards support computer based makks making it suitable for working or rudying a setting </Answer-
<Answer'
current arrangement fistures and accessorics in compact room aptimized for wote of wovemeu and practicol use of each functional grea?
<CoT> -SI> The compact room contains essential flatures ruch as wollet sink, tosel dispenior, dispouos, tranh creating distict functional <SI= The floor largely opos, focilloting unthstrurted
these -SP- placemont of accertories like paper dispenser soap dispenser near sinks supports practical use without clutteiig pathosays. layout balances accessibilly and usubility within the confined space. </Col>
Annwer' The arrongement is practical, with essentiol fistures spaced to allow clear patkwars for movement. Accessories are posiiiined logisally near usage points oshanclng junctionalty without overrooding Ovevall, the layout balances accessihiitt and officimtt me of limited space. </Anower>
Figure 12: Additional qualitative results of SCoT-Reasoner on scene analysis task.
As illustrated in Fig. 13 the additional qualitative results of spatial planning highlight the pivotal role of CoT reasoning in enabling 3D LLMs to bridge perception and planning in complex environments. SCoT-Reasoner explicitly verbalizes the current situational state, such as the agent's 's relative position, available pathways, and the spatial organization of surrounding objects. This step-by-step reasoning allows the system to transform low-level geometry into structured semantic cues, ensuring that task-relevant details (c.g., seating layout, desk placement, or container accessibility) are accurately extracted. Building on this structured scene representation, CoT then decomposes user queries into interpretable subproblems, aligning semantic intent with environmental evidence. For instance, when tasked with organizing group meeting, the reasoning chain highlights seat arrangement, clutter removal, and lighting adjustments, collectively optimizing comfort and interaction. Similarly, when sorting recyclable items, SCoT-Reasone not only localizes the recycling bin but also justifies the choice through its visual features and unobstructed access path. By articulating these intermediate steps, CoT ensures that the final answers regarding action instructions not only correct but also interpretable, trustworthy, and grounded in 3D evidence, thereby enhancing both reasoning transparency and task reliability.
22

---

Published as a conference paper at ICLR 2026
I'm sunding bright surounded dorker fooring and want to set up comfortuble soating space small group rolas and chot. I've already cleared furnitue moved obssacles away. How should procest to create thi soating arrangement using avoilable elowens in area?
<CoT> <St> The rsom contains bright surranneed by darker flooring whick swiabll for setting up confortable reating space, <SI> Nearby furmstur such as chairs or couches can be rearanged to face the bright area, facllitating conversation and relatation. <SI> There are no visible harris obstracting acoess to these jurnituee pieces, making it fousible move them eloter together <SP> Arranging reats in this way creates a cory evvnnmment conducive social interaction without ueeding bring in additional furniture. <oCoT>
s A From your position amouge the weachy funiiung face the benght aven enawring enough space for people comforuhll. This will cozy spot for converation relesation without obstruction. </Anower>
standing near snating area with multiple cushiseed seats arrugge around contral neffcce. want to organize a small meeting for four people. kaving giready set some matersais on the contral neface, How can I best arrange the reating and space to create a confortable and producivve evinomen?
<CoT> <St> The verting area consists of mulriple eichioned vosts arranged around a contral surface, proniding ample space for group interaction, <SI> To crease confortabic and productive envennment, the cheirs shouldd he eveuly spaced to allon amung purticipunte <SD- Clearing
muvemcnt and communication unshetrutedd maturials during the meeting. <SI> Adjusinn the lighting cahance viilulitt and confort, making the space iwiiiing and conductre to focused dicussion. These step collectively aptimize soatng araagement auronnlings for un offective weeting stip. </CaT>
clutter
contral surface
<Annwrrs From your position orrange the seats evenly arsund the contral surface with enaugh spare between thom for ewey movemen Clear any chatie un the surface, adjent lighting of nended and onnrre good uisibiiity for all participants <hArwerr
I'm suanding veuting aren in with maltiple warkspaces, want organizz my digita documents soustunenoll check my Hone should apprsoch from heve?
<Col> The room contoins multpie worksaces with them <ST> veating area with concher her is positioned away from these workspaces, providing o confortable spat to sit and organize documents, <SI> Nearby, there is a screen on desk that can he usnd for anoil checking. To aecomplio both tukk afficionlly one should mose from the senting aver to the derk where the screen located, then you can sit down and use the computer This allons document organisation and email checking without interruttion <KoT>>
chous amanged around
<Answer> From your porittee neur the reating arrange your docoments, you con down we computer your simiiunoously <lhurner>
soward the desk with the screen
I'm stunding near the extrance of spacions room with varlous supplies arranged clong one side. vust organize the materials by disporing recyclable ttems How should and compleu diss soriing took?
locate
<CoTT> <SP= recycling is positioned along ride of spacious room sientiffbble by color and coctingular shape. <SI= From entrance, it feasible move divectly toroands container without obusocles blocking the parh <SI> Opening the bin' allowr easy disponal of recyclabie materials mniid. This direct approach ensures efficient sorting and proper disposnl of recyclabie tens. </CoY>
AAurerrs From your position near the entrancs, locam the contiiner with blue body and rectangniar shape for side the Open diepose recyciable Home proprryy </Answors
Figure 13: Additional qualitative results of SCoT-Reasoner on un-situated planning and situated planning tasks.
A.8 BASELINE INTRODUCTION
We re-train the baseline methods (including 3D VG Transformer, 3D-LLM, Chat 3D, Chat 3D V2, LL3DA, Scene-LLM, Chat Scene, and Video 3D LLM) using the same settings with our SCoT- Reasoner for fair comparison in spatial perception, analysis and planning tasks, respectively.
3D VG Transformer. 3D VG Transformer is a transformer-bawe method specifically designed for 3D visual grounding, leveraging diverse relations to facilitate cross- cooss-mdalltyy proposal disambiguation.
3D-LLM. 3D-LLM is the first LLM- model for 3D scene understanding, which takes 3D point clouds and their dense features derived from multi-view images input and perform a diverse set of 3D-related tasks.
Chat 3D. Ch integrates the perceptual strengths of pre- pre-tedi 3D representations with the conversational capabilities of large language models, establishing the first universal dialogue system for 3D scenes.
23

---

Published as a conference paper at ICLR 2026
Chat 3D V2. Chat 3D V2 incorporates 3D object representations into LLMs and assigns attribute- aware token and relation aware token for each object to capture the object's attributes and spatial relationships with surrounding objects in the 3D scene.
LL3DA. LL3DA is a large language- 3D assistant designed to take point clouds as direct input and generate responses both textual instructions and visual interactions. By incorporating visual interactions, LL3DA effectively understands human engagement within 3D environments, thereby resolving ambiguities that may arise from plain text alone.
Scene-LLM, Scene-LLM is a 3D vision-language model designed to enhance the reasoning capabili- ties of embodied agents. It employs hybrid 3D visual feature representation that captures dense spatial information while enabling dynamic scene state updates.
Chat Scene. Chat Scene improves Chat 3D V2 and receives multimodal representation input, it models the scene embeddings as sequence of explicit object-level embeddings derived from semantic-rich 2D and 3D representations, performing well in object-level and scene-level tasks.
Video 3D LLM. Video 3D LLM treats 3D scenes as dynamic videos, then incorporates 3D position encoding into video representations aligns video representations with real-world spatial contexts, therehy performing 3D- VI. tasks based on 2D) Video I.I.M.
A.9 LIMITATIONS
While SCoT raises strong reasoning and planning capabilities of 3D-LLMs, several limitations remain. First, the current framework is primarily validated on indoor datasets, and its performance in real-world, open-ended 3D environments requires further investigation. Second, although the CoT reasoning improves interpretability, it may introduce longer inference time and occasional inconsistencies across reasoning steps. Our future work will extend SCoT toward more diverse and dynamic city-scale scenarios.
A.10
THE USE OF LARGE LANGUAGE MODELS
We clarify the role of Large Language Models in the preparation of this work as follows:
Manuscript Writing: LLMs were only employed for improving grammar and expression during manuscript writing.
Research Design and Experiments: LLMs were not involved in idea conception, methodological design, experimental implementation.
Dataset Generation and Evaluation: LLMs were used solely as auxiliary tools for dataset generation and evaluation metries.
24


Reviews:
[{"review_id": "7lgZa6TsMJ", "reviewer": "Reviewer_sqpn", "summary": "This paper presents SCoT, a million-scale Chain-of-Thought (CoT) dataset designed to teach 3D Large Language Models (3D-LLMs) to reason spatially.\nSCoT organizes tasks into three tiers — Spatial Perception (“what is there”), Spatial Analysis (“what does it mean”), and Spatial Planning (“what should I do”) — and annotates CoT reasoning only where necessary.\nThe dataset introduces scene-grounded reasoning with explicit \\<SI> tags to ensure factual alignment with 3D context.\nExperiments on multiple baselines (Chat3D, ChatScene, Video3D-LLM) and the proposed SCoT-Reasoner show significant improvements in reasoning explainability, faithfulness, and planning accuracy.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "High-quality dataset: 1.1M diverse samples covering perception, reasoning, and planning with strong annotation rigor and cross-checking.\n\nInnovative design: The three-tier CoT taxonomy (perception–analysis–planning) effectively balances reasoning depth and hallucination control.\n\nGrounded CoT methodology: The \\<SI> tag mechanism enforces transparent, scene-based reasoning rather than textual hallucination.\n\nStrong empirical validation: Extensive quantitative and qualitative analyses demonstrate consistent gains in complex reasoning tasks.\n\nPractical impact: Provides a scalable framework for training reliable, interpretable 3D-LLMs relevant to embodied AI and robotics.\n\nSolid writing and clarity: The paper is well-organized, and figures (e.g., Fig. 1 & 3) effectively illustrate the framework and dataset pipeline.", "weaknesses": "The use of LLM-based evaluators (ChatGPT, Qwen, DeepSeek) for “Explainability,” “Faithfulness,” and “Trustworthiness” is well-motivated but inherently subjective.\nIt would strengthen credibility if the authors validated inter-evaluator consistency (e.g., correlation scores between evaluators).\n\nThe paper does not evaluate how models trained on SCoT generalize to unseen 3D environments or other datasets (e.g., ARKitScenes or Omni3D).\n\nThe paper could provide more detailed ablations: Comparing models trained with different CoT lengths or varying levels of \\<SI> grounding.；Analyzing which task types (object vs. scene vs. planning) benefit most from CoT supervision.；Evaluating performance trade-offs when CoT is partially removed during inference.\n\nAlthough Table 5 highlights hallucination in perceptual CoT, the causes (e.g., linguistic priors vs. visual overfitting) are not deeply analyzed.\nThis discussion could offer more insight into why CoT sometimes harms visual fidelity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2oxAFSeqcW", "reviewer": "Reviewer_qGaY", "summary": "This paper propose SCoT, a large-scale CoT datasets for spatial reasoning. It spanning three levels including spatial perception, spatial analysis and spatial planning. SCoT annotates intermediate reasoning grounded in scene cues. It introduces<SI> token let CoT grounded in scenes context, and reduce hallucinations. Results shows model trained on their proposed datasets benefits complex analysis and planning.", "questions": "Refer to weakness.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper constructed a valuable large-scale datasets which including diverse scenes with their classified three-tier taxonomy of 3D task.\n\n2. This paper propose a detailed data construction pipeline which provides empirical insights for spatial reasoning data construction.", "weaknesses": "1. The novelty is not very clear in this paper, could the author emphasize the key differences between this work and previous spatial reasoning CoT studies, as listed in Table 1 such as 3D-R1 and SpaceR-151k, they are also including multi tasks and reasoning, so what's the biggest advantage of your dataset compared with them? Have you compared with these datasets in controlled settings?\n\n2. The base model used to train is too weak to verify the usefulness of the proposed method. I don't understand why using Vicuna-7B as the pretrained model even in 2025 today. \n\n3. I doubt the validity of the evaluation. If I understand correctly, part of the evaluation data are self-constructed by yourself? (In the table 2, and table 3.) The public evaluation data only appear in Table 7, but over half of results in Table 7 your proposed method can't beat other methods. I think it's extremely unfair to compare with other methods if using your self-built data in the main results, considering we don't know whether the performance gain comes from in distribution benefits. Are training and test scenes completely disjoint?\n\n4. Using same models (GPT-4.1, DeepSeek, Qwen) generating training data as evaluator, which will further introduce bias in evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LRyPnAxZB5", "reviewer": "Reviewer_A8Pi", "summary": "This work introduces a million-scale chain-of-thought dataset aimed at improving the spatial reasoning capabilities of existing 3D LLMs, along with a baseline model, SCoT-Reasoner. Experiments show that models fine-tuned on the SCoT dataset exhibit performance improvements across various 3D visual question answering benchmarks. Additionally, these fine-tuned models exhibit reasoning processes that are transparent, faithful to the scene, and inherently more trustworthy.", "questions": "1. Throughout the paper, the three main tasks are introduced in the order of SCoT-Perception, SCoT-Analysis, and SCoT-Planning. Why is a different order used in the results section (Section 5.2)? Maintaining consistency would improve readability.\n\n2. It is unclear which model is evaluated in Table 5. The authors should clarify the model configuration or variant used for these results.\n\n3. SCoT-Reasoner appears to be a key model in the evaluation, yet most of its technical details are relegated to the appendix. I recommend moving more of these details into the main Method section to help readers better understand the approach.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed SCoT dataset covers a diverse set of spatial tasks and is large in scale. The paper provides sufficient details to enable the research community to reproduce the data generation pipeline.\n2. The paper is well-presented and easy to follow.", "weaknesses": "1. SCoT is based on only 800 scenes from ScanNet, which is a relatively small scale. While the number of CoT examples is large, I’m concerned whether such a limited range of scene samples can genuinely enhance models' spatial reasoning ability on unseen real-world 3D environments.\n2. In Section 3, the author describes the SCoT format as “Query–CoT–Answer.” However, the spatial perception data seems to include only QA pairs without explicit reasoning. Clarification is needed on how these are treated as CoT examples.\n3. Most experiments are conducted on the SCoT test set. It would strengthen the paper if the authors included additional evaluations on out-of-domain datasets such as MSR3D [1] and Hypo3D [2] to assess generalization.\n4. The authors are encouraged to provide more discussion, ideally supported by results, demonstrating that fine-tuning on SCoT offers greater benefits for spatial reasoning compared to fine-tuning on previous 3D SCoT datasets.\n\n[1] Linghu, Xiongkun, et al. \"Multi-modal situated reasoning in 3d scenes.\" NeurIPS 2024.\n\n[2] Mao, Ye, et al. \"Hypo3D: Exploring Hypothetical Reasoning in 3D.\" ICML 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YOCUW4G0Oa", "reviewer": "Reviewer_VyFW", "summary": "This paper introduces SCOT, a new million-scale dataset designed to teach 3D-LLMs to reason spatially. To address the limitations of existing datasets that lack explicit reasoning steps, SCOT provides structured Chain-of-Thought annotations across a three-level taxonomy of tasks: Spatial Perception, Spatial Analysis, and Spatial Planning. A key contribution is the use of scene-grounded annotations, which force the model to base its reasoning on verifiable 3D evidence, thus improving the transparency and accuracy of complex analysis and planning tasks. Experiments show that this method significantly enhances the model ability to perform complex spatial reasoning, while also demonstrating that overuse of CoT for simple perception can be detrimental.", "questions": "What is the impact of generating detailed CoT reasoning on inference efficiency, and how does this affect the suitability for real-time applications such as robotics?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces SCOT, a new million-scale dataset designed to teach 3D-LLMs to reason spatially. To address the limitations of existing datasets that lack explicit reasoning steps, SCOT provides structured Chain-of-Thought annotations across a three-level taxonomy of tasks: Spatial Perception, Spatial Analysis, and Spatial Planning. A key contribution is the use of scene-grounded annotations, which force the model to base its reasoning on verifiable 3D evidence, thus improving the transparency and accuracy of complex analysis and planning tasks. Experiments show that this method significantly enhances the model ability to perform complex spatial reasoning, while also demonstrating that overuse of CoT for simple perception can be detrimental.", "weaknesses": "The scene-grounded reasoning lacks strong evidence, which relies on LLM judge evaluations rather than direct verification against ground-truth data, which may lead some problems on robustness and objectivity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)