SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Model
Authors: Yue Zhang
Zhiyang Xu
Ying Shen
Parisa Kordjamshidi
Lifu Huang
Venue: ICLR 2025 Poster (2025)
Keywords: Situated Understanding in 3D Scen
3D VL
LLM
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Integrating the 3D world into large language models (3D-based LLMs) has been a promising research direction for 3D scene understanding. However, current 3D-based LLMs fall short in situated understanding due to two key limitations: 1) existing 3D datasets are constructed from a global perspective of the 3D scenes and lack situated context.
2) the architectures of the current 3D-based LLMs lack an explicit mechanism for aligning situated spatial information between 3D representations and natural language, limiting their performance in tasks requiring precise spatial reasoning. 
In this work, we address these issues by introducing a scalable situated 3D dataset, named Spartun3D, that incorporates various situated spatial information.
In addition, we propose a situated spatial alignment module to enhance the learning between 3D visual representations and their corresponding textual descriptions. Our experimental results demonstrate that both our dataset and alignment module enhance situated spatial understanding ability.

Published as a conference paper at ICLR 2025
SPARTUN3D: SITUATED SPATIAL UNDERSTANDING OF 3D WORLD IN LARGE LANGUAGE MODELS
Yue Zhang' Zhiyang Xu² Ying Shen
Parisa Kordjamshidi"
Lifu Huang
"Michigan State University
Virginia Tech
3 University of Illinois at Urbana- Champaaan
*UC Davis
zhan1 624 4 msu. edu, zhiyangxevt. edu, ying228111inos..en
ABSTRACT
Integrating the 3D world into large language models (3D-hased L.MMs) has been a promising research direction for 3D scene understanding. However, current 3D- based LLMs fall short in situated understanding due two key limitations: 1) ex- isting 3D datasets are constructed from a global perspective of the 3D scenes and lack situated context. 2) the architectures existing 3D- LLMs lack explicit alignment between the spatial representations of 3D scenes and natural language, limiting their performance tasks requiring precise spatial reasoning. address these issues by introducing a scalable situated 3D dataset, named Spartun3D, that incorporates various situated spatial reasoning tasks. Furthermore, we propose Spartun3D-LLMM built
existing 3D-based LLM but integrated with novel situated spatial alignment module, aiming to enhance the alignment between 3D visual representations and their corresponding textual descriptions. Experimental results demonstrate that both our proposed dataset and aligmment module signifi- cantly enhance the situated spatial understanding of 3D-based LLMs.
INTRODUCTION
Recent advances in large language models (LLMs) have demonstrated their remark- able reasoning and communication capa- bilities across various tasks and modali-
Question what should you do to wash hands?
ties (Achiam et al.) 2023; Alayrac et al.
2022 Zhang et 2023aj Rubenstein
stank
et al. 2023). Building on these break- throughs, there has been growing interest in extending LLMs to the 3D world (3D-
bln
hin
Situationt: You ane standinng beside Irash 19t8 while there is litollen in froni
Situation2: You standing beside
based LLMs) (Hong et al.) 20233 Huang
trash bis while there toalet behind you..
eet al. 2023 2023| Chen al. [2024] Zhen et al.
Serub your hands soap and water.
3D LLMs: your hands.
2024
Wang et al.) 2023). ). Existing stud-
ies mainly focus integrating various
Spartun3D: You can use the Link oni the right.
Spartun3D: You use the sink the left.
3D scene representations into L.Mss, en- abling the models to perform 3D ground- ing and spatial reasoning through natural language. For example, 3D-LLM (Hong) et al. 2023) utilizes multi-view images to represent 3D scenes, pioneering a new direction in this field, while LEO (Huang et al. 2023) further pushes the boundary by directly injecting 3D point clouds into LIMss, aiming to develop generalist embodied agent capable of 3D grounding, embodied reasoning. and action planning.
Figure 1: Illustration of situated scene understanding of Spartun3D-LLM compared to other 3D-based LLMs.
Despite the promising progress, current 3D-baaed LLMs still fall short in situated understanding. fundamental capability for completing embodied tasks, such as Embodied Question Answering (Das)
al. 2018), Vision and Language Navigation (Anderson al. 2018 Zhang et al. 2024c), robotic manipulation (Shridhar et al. 2022). and many others. Situated understanding refers to the ability to
"Co-supervision.

---

Published as a conference paper at ICLR 2025
Situated Captioning
Situation: Stonding beside blue octagon wide bed that is messy while there is window on the left.
Situated QA
Object Artribute and Relation
Q Dese Bee the socue froms your curcent poetitu.
whhre is the dek bosted? A: Behind you
A: In frunt themi netaryalar has and
Nearly, chere's tall oightstand beskte bod. To the right. there's a tall pillow
Object SegeetAfffrre Affordance
dose to anocher pillon und blknket.
Q: need chack my apperraee Where should gu?
Behind, polabed merree
flosble
should
mirvor you.
daslve. On the left, themls aetificsl lamp and dakk dosc to soda chair and ted his, cuttiin another lamp, and wardisbe now othiect.
Siruated Planning
Q: went to make the rocm beginer and then ett donn to rdlax. What should do?
A.
your left
light, abse atjuudng cartume hand towaaks the sofn chale on your left side di and relas, You may need to pass other chatrss and yeur parh.
Other Vana
Figure 2: Examples of Spartun3D. Green box and arrow show the standing point and orientation. interpret and reason about 3D scene from a dynamic egocentric perspective, where the agent must continuously adjust understanding based on its changing position and evolving environment around it. This capability is crucial because agent's 's reasoning and response to the same question can vary depending on its current situation. For example, as shown in Fig | given the same question "What should you do to wash hands?", the agent might need to answer "use the sink the left/right" based on the agent's 's current perspective and location relative to the "sink".
However, achieving such situated understanding remains challenging for current 3D-based LLMs, and we identify two primary reasons. First, most existing 3D datasets (Chen et al.) 2021 Azuma et al. 2022; Zhu al.) 2023 Huang al. 2023) are constructed from global perspective of 31> scenes, lacking the situated information necessary for training models to reason from an agent's perspective. As a result, models fine-tuned on these datasets struggle to develop situated reasoning ability. While the introduction of SQA3D (Ma et al. (022) has made progress by providing a situated 3D dataset, the dataset is mainly anman-annotated, making it expensive and difficult to scale for the large-scale training needed by 3D-based LLMs. Second, existing 3D-based LLMs inject 3D representations into LLMs by simply concatenating tokens from different modalities (e.g., text, images, 3D point clouds). While this allows for basic cross-modal interaction, it lacks an explicit mechanism to align the situated spatial information from the 3D scene with natural language. Therefore, the models struggle capture the critical spatial relationships for situated understanding.
To address the aforementioned issues, we propose two key innovations: we first introduce a scal- able, LLM-generated dataset named Spartun3D, consisting of approximately 133k examples. Dif- ferent from datasets used by previous 3D-based LLMs (Zhu et al., 2023 Huang et al., 2023 Hong et al. 2023). Spartun3D incorporates various situated spatial information conditioned on the agent's standing point and orientation within the environment, consisting of two situated tasks: situated captioning and situated QA. Situated captioning is our newly proposed task that requires generating descriptions of the surrounding objects and their spatial direction based on the agent's 's situation. Sit- uated QA is designed with different types of questions targeting various levels of spatial reasoning ability for embodied agents. Furthermore, based on Spartun3D, we propose a new 3D-based LLM, Spartun3D-LLLM which is built on the most recent state-of-the-art 3D-based LLM, LEO (Huang et al.) 2023), but integrated with novel situated spatial alignment module that explicitly aligns 3D visual objects, their attributes and spatial relationship to surrounding objects with corresponding textual descriptions, with the goal of better bridging the gap between the 3D and text spaces.
We conduct extensive experiments across variety of tasks, including Spartun3D, SQA3D, and MP3D Nav (Savva et al. 2019). Our results demonstrate that our model, trained on Spartun3D, exhibits strong generalization to other tasks, specifically in zero-shot settings, highlighting the ef- fectiveness of our proposed dataset. Additionally, Spartun3D-LLM outperforms the baseline on nearly all tasks, indicating that incorporating direct text supervision improves the model's spatial understanding ability. This is further supported by our observation that the explicit alignment mod- ule improves the generation of more fine-grained, context-aware spatial information.
2 RELATED WORK
Situated Scene Understanding is essential for various embodied tasks, including embodied QA (Das et al. 2018 Wijmans et al. 2019), vision-and-language navigation (Aderson al. 2018
2

---

Published as a conference paper at ICLR 2025
3D Scene
a) Situation Design
b) Situated Scene Graph
e) Situated Tasks
Sofa
Soda:
Situated Captioning
Attribbute (oocur (yelon)) Situtioo eclation EI
Discribee the object your
Rett/ngae//onn wwtpg////oo////aaaaaeooe (aachssodd) from the Anose lightst ongic
"iffumbamee's store tteme "Attr (widd
1,5
"Amgle - 1e5 degree 'Distanee"' 1.5 'Phasshy' (ubblog)
Go right uife tukle sevesaso costral pance sdoa for placin timm, string
baskuerds...
Table
LLMs Prompting
Comsonate mentrright guamtze-aeeer paire braed enferventon quantions
stfec relutisoss offerdacee Shotted Plyening
Cabinet
Systemo Message
Situaled Scune Graph
Situation:
Standing beside silli while
Situation Tasks Instruction
ohper Atsssn:A A.amg
there is babke behind
Figure 3: Illustration of Spartun3D Dataset Construction Process. Given 3D scene: a) we first
select pivot object (e.g,, sofa) and referent object (e-g., table) to define the situation; b) we create situated scene graph based on situation, incorporating various spatial relationships (See Fig. 5): c) we use the scene graph to prompt GPT-4o (shown in gray box) to generate data; e) we utilize different prompting strategies for generating situated captions and QA pairs.
Zhang et al. 2021 Zhang & Kordjamshidi 2022a b 2024) and robotic manipulation (Shridhar al., 2022| lang et al. Driess Driess et al. 2023). Recently, SQA3D (Ma al. 2022) introduces
human-annotated dataset where the model generates answers based questions and given sit- uations. SIG3D (Man et al. 2024) highlights the situated awareness via a situation-gres
VL reasoning architecture. While SQA3D is crucial in 3D vision-language situation-grounded grounded 3D learning (Zhu et al., 2023 Huang et al.) 2023). its reliance on human annotations makes it costly and difficult to scale for the large-scale training required by 3D-based LLMs. In contrast, use the LLM to design an automated pipeline to generate situated questions and answers, enhancing both the scalability and diversity in data collection procedure.
Grounded 3D Scene Understanding. Compared to 2D Lision-language tasks (Song et al.) 2024
Antol et al. 2015 Zhang al.) 22024a Guo et al., 2024a Xu et al., [024, Wang et al.) 2024 Qi et al. 2024 Guo et al. (025), 3D scenes introduce additional dimensions of knowledge that is more
challenging align with text modalities. Early studies this area primarily focused on grounding
language to individual objects within 3D environments (Achlioptas et al., 2019 2020) Chen et al.)
2020 2019 2021). Recently, 3D Vista (Zhu et al.) 2023) proposes a pre-trained VL Transformer for 3D vision and text alignment, and few works utilize LLMSs in understanding 3D scenes (Hong) jet al,, 2023 Huang al. 2023) Chen al., [024] Zhen et al., 2024, Yang et al.) 2024). How- these works are less effective in situated understanding tasks that require generating answers from the agent's 's dynamic perspectives. MSQA (Linghu et al.] (025) is a concurrent work that also explore situated understanding of 3D-based LLMs. However, our approach differs in both dataset construction and model design. While MSQA focuses on leveraging images to assist in describing situation, our method emphasizes directly understanding the situation from textual descriptions, leading to distinct contributions in the field.
SPARTUN3D DATASET CONSTRUCTION
To better equip 3D-based LLMs with the capability of understanding situated 3D scenes, we intro- duce Spartun3D, diverse and scalable situated 3D dataset. To ensure the scalability of Spartun we carefully design an automatic pipeline that leverages the strong capabilities of GPT-4o (OpenAl, 2024). with three key stages as shown in Fig. 3 (1) Designing diverse situations that specify the agent's standing point and orientation given a 3D scene as input (Sec.B.1) (2) Constructing situated scene graphs to describe the spatial relationships between the agent and objects in the environment
conditioned on the agent's situations (Sec. 3.2 and (3) Prompting L.MMs to generate dataset based
on situated scene graphs (Sec. 3.3
3.1 SITUATION DESIGN
The 3D scenes in Spartun3D are taken from 3RScan Wu et al. 2021). which provides a diverse set of realistic 3D environments. Given particular 3D scene with all the objects labeled by humans from 3RScan, such as the example shown in Fig. B3, our first step is to generate diverse situations for the agent. To construct the situation, we begin by identifying the standing point and orientation and
3

---

Published as a conference paper at ICLR 2025
then complete situation description accordingly using the following template: "You are standing beside {pivot object name), and there is {referent object name) on the [lethiighfron//abbwarr The elements within {} specify the key components that together define the situation. Below, define the agent's standing point and orientation and explain how these elements are obtained to construct diverse and reliable situations.
Standing Point and Orientation. We begin with determining the agent's standing point and orientation within the 3D scene. Our approach is to place the agent beside an object, ensuring clear reference for orientation when interacting with the environment. Specifically, we project all objects from 3D space onto 2D plane, focusing only on the x and y coordinates to construct a bird-eye- view of the scene. From this projected 2D scene, we randomly select an object from the set of segmented objects within the 3D scene. To ensure the constructed situation remains realistic, we exclude objects that are positioned too high to avoid unnatural sit- uations like "standing beside the lamp on the ceiling". ". As a result, we limit the selection to objects whose z-axis is below the average height of all objects in the scene. Then, we choose midpoint from two sides of the selected object's bounding box that are closest to the center of the scene, as shown in Fig. 4 By prioritizing the side closest to the center, we minimize this risk and keep the agent within the scene's 's boundaries. Finally. the selected midpoint will be used as the agent's 's standing point. In addition, we need to determine the agent's 's orientation. We assume the agent's orientation is always facing forward the center of the selected object. This guarantees that the selected object remains within the agent's field of view.
Pivot Object
arientation
orientation
Scrne
Center
Standing Points
Figure 4: Standing Point and Orientation Selection.
Pivot and Referent Object. Once the agent's 's standing point and orientation are determined, we re- fer to the object that the agent stands beside as the "pivor object", and other objects surrounding the pivot object are potential referent objects. referent object then randomly selected, and its rel- ative position (leftrighhffrontbbackwwd) with respect to the agent's standing point and orientation, is used to generate the description of the situation.
3.2 SITUATED SCENE GRAPH CONSTRUCTION
Building on the agent's situation, we further construct situated scene graph that captures the com- prehensive spatial relationships between the agent and its surrounding objects. Existing 3D-based LLMs (Zhu et al., 2023) Huang et al., 2023) represent scenes in structured manner using JSON- formatted scene graphs including detailed scene context of object attributes and relative spatial relationships between objects. However, their spatial relations are based on global view, such as bird-view-eye perspective (as shown in Fig. 55) To enable situated understanding, we introduce a situated-xcene-grapp adapted from the original global scene graph to capture all relative spatial relationships between the agent's 's standing point and surrounding objects as follows:
Rotation Angles. We calculate rotation angles that reorient the agent from its orientation to the sur- rounding objects. Specifically, we first calculate the horizontal angle between the standing point and the center of the pivot object. Next, we calculate the horizontal angle between the stand- ing point and surrounding object. The rotation angle is determined by the difference between these two angles. We further normalize the rotation angles such that larger values correspond to a greater degree of rightward rotation.
Direction. We classify the object's rotation angles to the agent into four directional categories according to a predefined standard: [front, right, backward, left] (see Fig. (b)). For instance, an object is categorized as "right" if the turn angle falls within the range of [45-135] degrees relative to the agent's 's forward-facing orientation.
Distance. We compute the Euclidean distance between the agent's standing point and the center of the bounding boxes of surrounding objects.
Passby Objects. We assess whether the agent can move freely from its standing point to other objects. We draw straight line from the agent's 's standing point to the center of the referenced object. If this line intersects any other objects in the scene, those objects are considered "passby objects". For example, as illustrated in (d), the "table" is passby object between the agent and the "kitchen cabiner". We explictly include the information of passby objects to help the agent build awareness of objects that might influence its path while navigating.
4

---

Published as a conference paper at ICLR 2025
Global
aJRotation Angle
byDirection
Distanoo
djPasshy Objects ejSitusted Somne
PRUNT
Graph
Cabinet
Kitthen Counter
(315-360,0-53
0-45
Cabinet
Cabinet
Sofar
Backwards.
dagres
TEFT (235 S15)
RIGHT
(cabinent:
Tuble
Table
Table
Tuble
Angle: 165 degree
Distance: 1.5
diggres
160
BACKWARKD WARD (195,225)
1.2ml
Phodyy
Passhy: [tuble]))
RIGHT
Situation:
"Sundinng
[sataj
(n on yourr Iripitt..
TV
TV
Scene
Traditional Method
Figure 5: Spatial information in Situated Scene Graph. The red dot and green arrow show the standing point and orientation, respectively. In this example, the pivot object is the "sofa", the referent object is the "TV",, and the surrounding objects include the "table" and "cabinet".
After gathering the spatial information described above, we organize it into JSON format (shown in Fig. (e)), which is then used input to prompt LLMs generate datasets.
3.3 LLMS PROMPTING
We design specific instructions prompt GPT-40 (OpenAī 2024 for situated tasks: Situated Captioning and Situated QA. For both tasks, we ask GPT-4o to provide responses considering situated spatial information. Detailed prompts for each task are provided in Tab. i0 the Appendix, and examples of the generated dataset are shown in Fig. 2
Situated Captioning is our newly introduced task, aiming to generate brief situated descriptions of the surrounding objects as the agent performs a 360 clockwise rotation starting from its standing point and orientation. The motivation for introducing this task stems from its crucial role embod- ied tasks, such as navigation, where the agent must interpret and reason about its environment from 360% panoramic views to make decisions about movement and interaction (Zhou al., 2024| Zhang & Kordjamshidi 2023 Zhang et al. 2024b). Therefore, we guide GPT-4o to generate descriptions progressively, starting from lower rotation angles and moving toward higher angles in each direction.
Situated QA. We design three types of questions for the Situated QA task, each targeting differ- ent aspect of spatial reasoning for embodied agents. Unlike previous works that rely on a single generic prompt for all question types, we develop tailored prompting strategies for each question type, encouraging LLM to generate QA pairs focusing on different levels of reasoning.
Object Atribute and Relations include questions about objects attributes, such as color, shape, and size, while also incorporating situated spatial information. For instance, the questions to identify "the color of the table positioned the left". and determine "how many pictures hanging on the wall to the right".
Object Affordance focuses the function utility of the objects, often based common sense knowledge about how objects are used. Similarly, we require situated spatial information to be part of the answer. For example, when asked "Where can you check your appearance?"" the correct answer should be "mirror on your left". specifying both the object name (mirror) and its spatial location from the agent.
Situated Planning is the most challenging task, as it requires the agent to perform multi-hop situated spatial reasoning. The agent execute a series of actions across multiple steps. where each subsequent action depends on the outcome of the previous one. In our dataset, we implement 2-hop reasoning. which requires the agent to perform sequence of two continuous actions. For example, given the example in Fig 2 "make the room brighter and then down to relax.. the agent needs first turn left from its orientation to face and move toward the window, open it to brighten the room, then based on its new position, the agent continues turning left toward the sofa chairs and sits down.
only recognize its surroundings but also plan and
5

---

Published as a conference paper at ICLR 2025
Eituation
Question
Anawer
BQASD SpartunSD
Cord-prompt
Spa-Prompt
4:1
624
03
94
0.77
0,33
Captioning Atte. Rel. Alkordance Planning
What
How
Cen
Which
Otwex
Ceplioning Attr. & Rel. Affordance Planning (el Percentage of valid examples generated from two different prompts.
(4) Arg. human score for Spartun31
(6) Comparison between SQA3D
and Spartun3..
Figure 6: Human Evaluation of Spartun3D.
3.4 DATASET STATISTICS AND QUALITY EVALUATION
In total, we collect approximately 10k situated captions and 123k QA pairs. For the tasks of object attribute and relation and the tasks of affordance, sampled around 10 situations per scene. For captioning and planning tasks, we sample around 5 situations per scene due to the increasing cost of longer token sequences required for these tasks. For each task, we split the data instances training and test set. Table shows the statistics of our dataset.
We conduct human evaluation assess the quality of Spartun3D, introducing scores based on two key criteria: language naturalness, which evaluates whether the text reads as if it were naturally written by a human, and spatial fidelity, which ensures that the data accurately reflects the 3D scene with correct spatial rela- tionships. Detailed explanations of the criteria are in Sec. A.I Each criterion is rated on a scale from to 5, and the average of these two scores is the overall human score. We randomly select 50 examples from each task and compute human scores of situation, question, and answer, respectively. As shown in Fig- 6 (a), the average scores align with the complexity of each task, with relatively lower scores for captioning and plan- ning tasks. To evaluate how our generated data compares human-annotated data, we sampled 50 examples from SQA3D and combined them with our dataset. Our data shows a similar trend in hu- man evaluation results across different question types observed SQA3D (shown in Fig. (b)). We finally evaluate how different prompting strategies influence the quality of the data. We ex- periment with two types of prompts for representing spatial information to prompt GPT-4o: Cord- prompt, which consists of object center coordinates, standing point, orientation, and instructions for calculating distances and rotation angles, and Spa-prompt, consisting of the calculated angles and distance based on the approaches
Table I: Dataset statistics of Spartun3D and hu- man validation results.
Tasks
# of Examples
Train/Test
Captioning
10K
8,367 350
Attr. & Rel.
62K
61, 254/8, 168
Affordance
40K
35, 070/5, 017
Planning
~ 21K
19, 434/2, 819
described in Sec. 3.3 An example of each type of prompt can be found in Tab. II Fig (c) shows the percentage of examples with high human scores (2 4) for each prompt across tasks. The results indicate that
yields unsatisfactory results, revealing that LLMs lack strong 3D spatial reasoning when interpreting raw spatial coordinates, which is consistent with their struggles spatial reasoning across text and 2D images (Zhang al.) 2024d, Premsri & Kordjamshidi 2024 2025). Our Spa-prompt significantly improves the quality of the generated dataset by providing qualitative spatial relations (e.g. distance, direction).
MODEL ARCHITECTURE
In addition to enhancing the situated understanding of 3D-based LLMs with Spartun3D, we also propose new 3D-based LLM,, named Spartun3D-LLLMM which integrates novel Situated Spatial Alignment module to strengthen the alignment between the situated 3D visual features and their corresponding textual descriptions. Spartun3D-LLM is built upon LEO (Huang et al. (2023), which represents the most recent and state-of-the-ard 3D-based LLM, and directly takes 3D point cloud data as input, making it well-suited for spatial reasoning tasks in 3D environments. Fig. iillustattss the overview architecture of Spartun3D-LLMM
4.1 BACKGROUND
Problem Formulation. We formally define the input triple C,S,Q Q >, where C is the 3D scene context, S is the situation, and Q is a question. The situation S can be further denoted as S =<< S1,SP SP 2, where St is textual situation deseription, and SD and S" the standing

---

Published as a conference paper at ICLR 2025
points and orientation, respectively. Specifically, 5 is 3D coordinate in the form V. and S" is the quaternion < q2, q9, q2, w >, where < qu, qy q > is the rotation axis and w the define 2 = 0 to calculate the rotation angle task is to generate a textual answer, denoted as A, given scene context C, situation S, and question Q. During training, S and 5 are provided the agent to rotate and translate the environment, while during testing, only questions and situations are provided.
rotation angle. For simplicity,
2D plane. The
Backbone. LEO takes text, 2D im- age (optional), and 3D point clouds as input and formulate comprehen- sive 3D tasks as autoregressive se- quence generation. Specifically, data from different modalities are con- verted into a sequence of tokens as input to the LLM. The text tokens include system messages (e.g., "You Al visual assistant situated in a 3D scene."), "), situations, and ques- tions. These tokens then embed- ded into vector representations using an embedding look-up table. For 3D point clouds, LEO first applies seg- mentation masks extract the point clouds of individual objects the 3D scenes. Then, the sam- pled points of each object are input into a object-centris point cloud encoder, PointNet++ (Qi et al.) 2017) pre-trate pre-trained on ScanNet (Dai et al. 2017), to obtain the object-level representations.
staniin Edeend, fure ar sledl
caure pidare da vall..
large
cottr
WF4
Situated Spotial
Allgument
Selfortettion
Large Languogt Model
Situtiio stand behind e clbix faxing tabli and don't have momiter talle.
Waalow
Quoction: What the first object Left!
Figure 7: Spartun3D-LLM Model Architecture.
Formally. we denote the representation of input text tokens as W [W] W2.... War] E IR MxD where M denotes the number of input tokens, and /) represents the dimensionality each to- ken's embedding. Additionally, the input object visual representations are expressed as O [01, .2.... ok] E RKD,, where K is the number extracted objects from the scene. Finally, the output answer represented as A [a1, ag, ..... av], where N is the number of tokens in the response. The model's objective is to generate the answer given these combined inputs. The loss for generating the i-th token of the output answer is formulated follows:
a..]
(i) Σ (ogpe(a./a4) Ws.
LEO can integrate various LLM backbones, including OPT1.3B (Zhang et al. 2023b) and Vieuna- 7B (Chiang et al. (2023). In our experiments, we fine-tune LEO with different L.M backbones on our proposed dataset via LoRA (Hu et a 202T).
4.2 SITUATED SPATIAL ALIGNMENT MODULE
Situated tasks require robust spatial reasoning abilities to comprehend the position, orientation, and spatial relationships of objects within a 3D environment. Existing 3D-based LLMS typically process inputs by concatenating output representations from various modality encoders. While this method facilitates the integration of data across different modalities, it does not inherently ensure that the 3D visual representations encode situated spatial information or effectively align with textual de- scriptions, which potentially limits the model's ability to perform tasks that require precise spatial understanding. To tackle this challenge, introduce novel Situated Spatial Alignment Module to improve the alignment between the object-centric 3D visual representations and their situated textual descriptions. The process begins by generating detailed situated textual descriptions for each object. Subsequently, an alignment loss is introduced, which directs the model in effectively learning the 3D visual representations based on these situated textual descriptions.
Situated Textual Descriptions. For each object, we construct a comprehensive situated textual description based on a template that captures the object's name, attributes, and spatial relations with nearby objects, as "Stand besides {object name} and facing the center of the {object name), in front, there are {a list of nearby objects); }; on the right. behind... ...: and on the left..." The object's attributes also considered (e.g.. "white chair"). We consider up to five objects per direction. If no object is present in a specific direction, the description explicitly states this, ensuring to provide complete information about the 3D environment.

---

Published as a conference paper at ICLR 2025
Table 2: Experimental Results on Spartun3D Situated QA Tasks, represents the model initialized with LEO instruction-tuns tuned weights. [Keys: C: CIDER; B-4: BLEU-4; M: METEOR; R: ROUGE; Sim: Sentence Similarity; EM: Exact Match; Bold: best results].
Models
LLMs
Attributes and Relations
Affordance
Situated Planning
B-4
M
R
EM
C
B-4
M
R
S
C
B-4
M
R
LEO
zero-shot
100.3
0.00
17.4
39.1
42.7
13.3
0.00
3,00
5.00
32.3
0.00
0.00
7.00
15.3
50.2
LEO+Spartun3D
OPT138
20.1
24,9
66.9
229.7
44.8
32.1
Vicuna78
125.4
10.1
22.1
46.7
02.1
238.9
32.1
24.4
65.0
68.3
242.1
46.5
35.2
63.1
54.3
SparttnnD
VicunaBB
129.2
10.4
20.0
48.1
60.2
211.3
32.1
24.6
55.0
07.8
247.1
47.5
36.2
65.1
85.5
SparumeiD-LLM
3P
124.1
9.2
21.0
47.3
49.4
227
31.4
26.3
54.1
08.2
232.3
46.2
33.2
62.1
$5.4
Vicuma7B
131.2
10.3
24.
18.8
53.7
240.4
52.
25.0
55.3
68.7
241.0
47.1
36.4
64.0
86.6
Vicrma88
26.7
09.7
47.6
36.2
65.4
Table 3: Experimental Results on SQA3D given the 3D objects from Mask3D and Ground-truth.
#
Methods
Mask3D Schult et al.
2023
GT
c
R
M
EM
LEO Huang et al,
2023
14.2
6.4
8.2
12.4
15.3
6.7
8.6
13.9
Zero
2
LFOFSpartunn
82.3
14.2
32.8
34.7
83.1
15.2
33.7
35.9
3
Spartun3D-LLM LLMMM
83.5
15.7
34.7
36.2
85.6
16.6
35.8
37.1
4
3D> Vista Zhu al.
2023
48.5
5
3D-LLM (oone
50.2
Fine-tune
LEO [Hiang or 2023
132.0
33.0
49.2
52.4
132.3
34.3
51.4
52.5
131.0
31.6
52.2
135.3
31.2
52.1
54.2
Spartun3D-LLMM
138.2
35.3
53.4
64.8
138.3
35.4
53.7
55.0
3D Object- Text Alignment. Inspired by the success of 2D Visual-Language models, which effec- tively leverage semantically aligned text and visual features to excel in downstream tasks (Radford al. 2021; Li et al., 2022 2023). we aim enhance the 3D visual representations so that they can better encode the situated spatial information and effectively align with the textual descriptions. Specifically, we introduce 3D object-text alignment loss guide the learning process of point cloud encoders within 3D-based LLMs, leveraging the robust language representations captured by pre-trained text encoders. We experiment with various text encoders, and CLIP achieved the best performance. For more details, please refer to the Sec. A.4 in the Appendix.
More formally, we obtain the text representations of situated textual description for each object from pre-trained text encoders, denoted as W = [W1, W2,.... W&] R For object visual representations O, we employ spatial self-attention layers (Chen et al. 2022) to learn spatial-aware object representations. Specifically, pairwise spatial feature matrix F E IR k x x5 is introduced to represent relative spatial relations between objects. For example, for each object pairs o, and oj. we construct pairwise spatial feature fy [daj. sim(fx), cos(Or), sin(O,), cos().)] IR lx5 where d: is Euclidean distance between two objects, and 6, and 6, are horizontal and vertical angles connecting bounding box centers of o; and oj. respectively. Then, we inject F into the self-attention of the object as,
O' =sofmaax QKT
(LP where Q = WqO; KOIKKWWWO,, WRO,V WvO,
(2)
dp
where O' E RRXD denotes the spatial-aware object representation; Then we use a Mean Squared Error (i.e., MSE) as the objective function to minimize the distance between the object representation O' and the corresponding situated textual embedding W, denoted as Calign = MSE wt). The model is trained to jointly optimize both the alignment loss and the language modeling loss (Eq. I
as L = LLM Lalign-
5
EXPERIMENT
5.1 EXPERIMENTAL SETUP
To demonstrate the effectiveness of our proposed Spartun3D-LLLM, we conduct experiments on two situated understanding datasets: Spartun31[T and SQA3D. For SQA3D, we evaluate under two conditions: object proposals in 3D are derived either from Mask3D (Schult et al. (023) or ground- truth annotations. Also, we assess the transfcrability of our method on the navigation task using
and
MP3D ObjNav (Savva 2019). Following LEO (Huang et al. 2023), we report the performance
https:///i www../o////aamueeeccc.oomm in In/Spartunn.
8

---

Published as a conference paper at ICLR 2025
using standard generation metrics, including CIDEr, METEOR, BLEU-4, and ROUGEL,, sentence similarity (Reimers, 2019) for captioning task. For SQA3D and situated QA tasks of questions about attributes and relations, we also report additional metric of exact-match accuracy. More details about metrics are introduced in Appendix A.2.1 We also provide implementation details in Sec. AA.3 We leverage LEO baseline. Since the training stage in LEO has covered most of the evaluation tasks, we experiment with models initialized from scratch to ensure a fair comparison in the zero-shot setting, For other settings, we report the performance of models initialized both from scratch and from the instruction- tuned LEO. To distinguish between the two, models initialized from the instruction-tune tuned LEO are marked with an asterisk (*).
5.2 EXPERIMENTAL RESULTS
Spartun3D Benchmark. We evaluate the performance both the LEO model and Spartun3D-LLM after fine-tuning them on our proposed Spartun3D dataset. The fine-tuned LEO model is referred to LEO+Spartun3D. Table _ and Table 4 show the experimental results captioning and QA tasks, respectively. We experiment with two different LLM backbones: OptL.3B and Vicuma7B. Our experiments show that Spartun3D-LLM consistently outperforms LEO+Spartun3DD across all question types (around 2%-3% across all metrics), regardless of the LLM backbone used, indicating the effectiveness of our explicit alignment module. We observe that initializing our model with LEO pre-trained weights improves performance. Notably, without fine-tuning, LEO performs reasonably well on attribute and relation questions in zero-shot setting but struggles with other situated tasks.
SQA3D Performance. We evaluate our method on the SQA3D dataset, whose scenes are derived from ScanNet (Dai jet al.] 2017). Their scenes differ from those in Spartun3D, which are sourced from 3RScan. We experiment with two settings: zero-shot and fine-tuning, In the zero-shot setting, we re-trained LEO on their dataset (row#1) only constructed from 3RScan excluding all dataset con- structed from ScanNet to ensure a fair comparison with our method. As shown in Table 3 LEO performs poorly on SQA3D in the zero-shot setting, suggesting its limitations in learning situated understanding from its dataset. In contrast, LEO trained on Spartun3D (row#2) shows significant improvement, demonstrating the effectiveness of our dataset. Further comparisons of Spartun3I--LL.
Table 4: Experimental Results on Spartun3D Situated Captioning Task.
Models
LLMs
C
B-4
M
R
S
LEO
zero-shot
0,00
0.00
9.00
15.3
51.9
LEO-SpurtnnDD
OPT1.3B
15.3
17.7
31.2
67.3
Vicuna7B
6.7
15.8
18.7
32.3
70.4
LEO+ SpartunnD
Vicuma7B
14.1
17.2
22.6
32.1
76.3
SpartunD-LLLM
OPT1.B
6.4
15.7
18.5
31.2
68.6
VicumaBB
8.5
16.4
19.6
32.5
72.5
Spartun3DDLLLMM
Vicuna7B
14.6
19.3
23.3
33.4
78.1
with LEO+Spartun3) demonstrate better zero-shot learning (i.e., generaliza- tion) capability of our model. In the fine-tuning setting, Spartun3DD-LLM continues to outperform LEO across all metrics. Table the Appendix provides breakdown of the fine-tuned performance across various question types, where Spartun3D-LLM shows consistent improvement.
Navigation Performance. To demonstrate the effectiveness of our approach on downstream embodied tasks, we evaluate it on the ob- ject navigation tasks. Specifically, we randomly select 5 scenes that contain around 1000 examples from the MP3D ObjNav dataset. In this task, additionally input 2D ego-centric images to both LEO and Spartun3D-LLLMM for comparison. There are four types of nav- igation actions: turn left, turn right, move forward, and stop. We evaluate whether the model gen- erates correct action at each step. We conduct the experiment in a zero-shot setting, and Table 5 shows the accuracy of the model's performance. The baseline model, LEO, struggles to generate the required action-related text to guide navigation steps without fine-tuning specifically for navi- gation tasks. In contrast, our model demonstrates strong transferability to generate correct actions. Fig 8 (e) showcases qualitative example, illustrating how our model effectively generates accurate navigation actions without task task-specific fine-tuning.
Table 5: Performance on Nav- igation. (Accuracy %)
LEO
Spartun3D-IJJM
Zero-shot 0
20.3
5.3 ABLATION STUDY AND EXTRA ANALYSIS
Explicit Alignment Enhances General Spatial Understanding. We evaluate the effectiveness of our proposed situated spatial alignment module on general scene understanding tasks, such as Scan2Cap (Chen et al., 202T) and ScanQA (Azuma et al. 2022). In line with approach for

---

Published as a conference paper at ICLR 2025
Spa3D-LLM (wee allgn:):
your the
towards
adjust
Situation:
You ure sunuline Scaide rriendl thrre to pilloun
Sitution: You are slunsing besidde sink whis thrre in ported plant
Situnioo:: Fre stunaling bogide todlr whis chere is trosk bii
Spa3D-LLM Thur to right and head towards the sink dlaan it. Thhs, tirn siighlly your right and hrod towsand the dumaerr curtain to aefjost
Queetion: hose many engecttoonn white to pastinght?
Quaration: Whree dirty lnsndry?
olean the and then adtur the whourr cartoin. too should you prvensd"?
LED
launday hasdeet.
(c)
SpaaD-LLM
Spa7D-LLM lourddry
'Target: Find the chest drawes.
lefh.
(4)
(b)
Siouation: You ure stunding boskl stool
Describe printer
lof.
position.
In front of mo, there is obrect/To To my right, theme
LEEY
LEFT
RIGHY
dloed undangglar glms satdoos
Jachens onteale It my left, then'ss plactng hott
Tarn to po left head youar Jolk, hoad
tonne cacit
Sowanda chokk You
Yern yuurratgts
chasc
chent drawwer.
(d) Figure 8: Qualitative Examples. (a), (c), (c) situated qa examples of object attribute and relation, affordance, and planning, respectively. (d) situated captioning. (e) navigation in zero-shot setting. Our model generates both actions and descriptions of surrounding objects while navigating.
(e)
situated tasks, we construct textual descriptions for each object based on its attributes and spatial relations to others from a top-view perspective. As shown in Tab. 6 by incorporating the explicit spatial aligmment module, our model shows better results, indicating that our proposed alignment module not only improves situated understanding but also enhances general 3D scene understanding.
Improved Situated Understanding. To analyze the model's situated understanding ability further, we visu- alize the distribution of model responses generated for questions requiring strong spatial understanding from extract questions starting with "which direction" ". Fig. I0 illustrates the distribution of generated "directions" " including "left", "right", "for- ward" and "backward". We observe that LEO is biased towards generating "left" 97% of the time. However, the
Table 6: Spatial Alignment Evaluation on Other Benchmarks. The metric is Sentence Similarity (%).
SQA3D. Specifically,
Methods
Scan2Cap
ScanQA
LEO Spartun3D
54.2
Spartun3D-LLM
55.7
48.6
ground-truth distribution of "left" and "right" should be balanced, suggesting that LEO may have limited understanding of situated spatial relationships. The bias is significantly mitigated when LEO is trained on our dataset (LEO+Spartun3D) While adding our alignment loss (Spartun3D- LLM) helps futher, our dataset is the primary factor in addressing the bias.
Scaling Performance. We conduct scaling experiments to demonstrate how model performance improves with the addition of Spartun3D datasets. As shown Fig. g we evaluate performance on SQA3D and observe consistent improvement as the dataset scales, highlighting the potential for dataset expansion using our proposed method.
Qualitative Examples. In Fig. 8 we showcase several successful examples to demonstrate the effectiveness of Spartun3D-ILLM across various situated tasks. Notably, in Fig (c), the model without explicit alignment module tends to generate more general or vague spatial descriptions, such as "turn around". contrast, with the alignment module, the model produces more specific details, including terms like "turn slightly right". To verify this, we examine 30 examples from both situated planning and situated captioning tasks and observe this phenomenon in 17 of them. This highlights how the proposed spatial alignment module enhances the generation of fine- ine-grained spatial information, leading to more precise and contextually accurate outputs.
6 DISCUSSION AND CONCLUSION
In this work, our goal is to address the limitation of situated understanding of the 3D-based LLMs from two perspectives. First, we propose method to construct an LLM-generated dataset based on our designed situated scene graph. Then, propose explicit situated spatial alignment on the 3D-LLM to encourage the model to learn aligmment between 3D object and their textual representa- tions directly. Finally, we provide comprehensive experiments to show situated understanding of SQA3D and navigation. We also provide analysis to show our proposed explicit alignment module helps spatial understanding.
benchmark improve
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This project is partially supported by the Office of Naval Research (ONR) grant N00014-23-1-24117, the award No. 2238940 from the Faculty Early Career Development Program (CAREER) of the National Science Foundation (NSF) and the U.S. DARPA FoundSci Program #HR001 12490370. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Office of Naval Research.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 2xxx:2333.007774 2023.
Panos Achlioptas, Judy Fan, Robert Hawkins, Noah Goodman, and Leonidas J Guibas. Shape- glot: Learning language for shape differentiation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8938-8988 0888-997,44 2019.
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-gre teee-raaied 3d object identification in real-world scenes. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Pro- ceedings, Part 16, pp. 422-440. Springer, 2020.
Bean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting desuall--roundder navigation instructions in real environments. In Proceedings of the IEEE Con- ference Computer Vision and Pattern Recognition (CVPR), 2018.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question an- swering for spatial scene understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19129-19133 2022.
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European conference computer vision, pp. 202-221.. Springer, 2020.
Kevin Chen, Christopher Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese. Text2shape: Generating shapes from natural language by learning joint embeddings. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part III 14, pp. 100-116. Springer, 2019.
Shizhe Chen, Pierre-Loute Louis Guhur, Makarand Tapaswi. Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. Advances in neural information processing systems, 35:20522-20523 2022.
Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning.
Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 26428-26438 2024.
Zhenyu Chen, Ali Gholami, Matthias Nießner, and Angel X Chang, Scan2cap: Context-aware dense captioning in rgb-d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3193-3203, 2021.

---

Published as a conference paper at ICLR 2025
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. httpst::
llmmsss. http://r//bb//g//////// og/pl23-000000000 03 10-000----000000000000000000000- 30-300viii -0
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias
Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017..
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embod- ied question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-10, 2018.
Danny Driess, Fei Xia, Mehdi SM Sajadi,, Corey Lynch. Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.03377 2023.
Xiao Guo, Xiaohong Liu, lacopo Masi, and Xiaoming Liu. Language-guisd hierarchical fine- grained image forgery detection and localization. International Journal of Computer Vision, pp- 1-22, 2024a.
Xiao Guo, Manh Tran, Jiaxin Cheng, and Xiaoming Liu. Dense-face: Personalized face generation model via dense annotation prediction. arXiv preprint arXiv:2412. 18149, 2024b.
Xiao Guo, Xiufeng Song, Yue Zhang, Xiaohong Liu, and Xiaoming Liu. Rethinking vision-language model for deepfake detection: Multi-modal interpretable forged face detection. In CVPR, 2025.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances Neural Information Processing Systems, 012220022244999 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1Xiv:2106.09685 2021.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.1287 2023.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yonggiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint S..iv:2210000099 2(3):6, 2022.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman image pre- training for unified lasion-language understanding and generation. In International conference on machine learning, pp. 12888 12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Xiongkun Linghu, Jiangyong Huang, Xuesong Niu, Xiaojian Shawn Ma. Baoxiong Jia, and Siyuan Huang. Multi-modal situated reasoning in 3d scenes. Advances Neural Information Processing Systems, 37:1 97:140903-140936 2025.
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022.
Yunze Man, Liang-Yee Yan Gui, and Yu-Xiong Wang. Situational awareness matters in 3d vision lan- guage reasoning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13678-13660 2024.
OpenAI. Hello gpt-4o, 2024. URL https:://ppeei..co///// httppp/s//ppgan....c/////eeeeee/ 1oto--pp-400
12

---

Published as a conference paper at ICLR 2025
Tanawan Premsri and Parisa Kordjamshidi. Neuro-symbolic training for reasoning spatial lan- guage, 2024. URL
wwtp//////aaaa/aooooo....o...a/t /al//bb//22/ 2406. 13828
Tanawan Premsri and Parisa Kordjamshidi. Forest: Frame reference evaluation in spatial reason- ing tasks, 2025. URL
17775
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, Jin Di, Yu Cheng, Qifan Wang, and Lifu Huang. Rora- vlm: Robust retrieval augmeeted vision language models, 2024. URL https: org btrg/abs//0 2410. 08876
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
N Reimers. Sentence-ber bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908. 10084, 2019.
Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.1290 2023.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
9339-9337, 2019.
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 8216-8223, IEEE, 2023.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning, pp. 894-906. PMLR, 2022.
Xiufeng Song, Xiao Guo, Jiache Zhang, Qirui Li, Lei Bai, Xiaoming Liu, Guangtao Zhai, and Xiaohong Liu. On learning multi-modal forgery representation for diffusion generated video detection. In Proceeding of Thirty- Conference on Neurul Information Processing Systems, 2024.
Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan Wang, Weifeng Ge, and Lifu Huang. Grounded-videeollm: Sharpening fine-grained temporal grounding in video large language models, 2024. URL
03290
Zehan Wang, Haifeng Huang, Yang Zhao, Ziang Zhang, and Zhou Zhao. Chat-3d: Data- efficiently tuning large language model for universal dialogue of 3d scenes. arXiv preprint arXiv:2308. 08769, 2023.
Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied question answering in photorealistic envi- ronments with point cloud perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6659-6668. 2019.
Shun-Chena Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scene- graphfusion: Incremental 3d scene graph prediction from rgb-d sequences. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7515-7525, 2021.
Zhiyang Xu, Mingian Liu, Ying Shen, Joy Rimchala, Jiaxin Zhang, Qifan Wang, Yu Cheng, and Lifu Huang. Lateralization lora: Interleaved instruction tuning with modality-specialize adaptations, 2024. URL htI.stttps://eet
13

---

Published as a conference paper at ICLR 2025
Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. LIm-grounder, open-vocabulary 3d visual grounding with large language model as an agent. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp- 7694-7701. IEEE, 2024.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities. arXiv preprint arXiv:2305. 1000, 2023a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models, 2022. URL https://arxiv, wgg/abs/2205000068 3:19-0, 2023b.
Yue Zhang and Parisa Kordjamshidi. Explicit object relation alignment for vision and language navigation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pp. 322-331, 2022a.
Yue Zhang and Parisa Kordjamshidi. Lovis: Learning orientation and visual signals for vision and language navigation. In Proceedings of the 29th International Conference on Computational Linguistics. pp. 5745-5754, 2022b.
Yue Zhang and Parisa Kordjamshidi. VIn-trans: Translator for the vision and language navigation agent. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguis- tics (Volume I: Long Papers), pp. 13219-13233, 2023.
Yue Zhang and Parisa Kordjamshidi. Narrowing the gap between vision and action in navigation. arXiv preprint arXiv:2408. 200.::200..00888, 2024.
Yue Zhang, Quan Guo, and Parisa Kordjamshidi. Towards navigation by reasoning over spatial configurations. arXiv preprint arXiv:2/05.06839 2021.
Yue Zhang, Ben Colman, Ali Shahriyari, and Gaurav Bharaj. Common sense reasoning for deep fake detection. arXiv preprint 0rXiv:2402.001266 2024a.
Yue Zhang, Quan Guo, and Parisa Kordjamshidi. Navhint: Vision and language navigation agent with a hint generator. Association for Computational Linguisties, 2024b.
Yue Zhang, Ziqiao Ma, Jialu Li, Yanyuan Qiao, Zun Wang, Joyce Chai, Qi Wu, Mohit Bansal, and languuug navigation today and tomorrow: A survey in the era
Parisa Kordjamshidi. of foundation models. arXiv preprint arXiv:2407.000033 2024c.
Zheyuan Zhang, Fengyuan Hu, Jayjun Lee, Freda Shi, Parisa Kordjamshidi. Joyce Chai, and Ziqiao Ma. Do vision-language models represent space and how? evaluating spatial frame of reference under ambiguities. ArXiv, 173885 2024d. URL https:://api.
Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d lision-langunge-accionn generative world model. arXiv preprint 1030:2403.00631,, 2024.
Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and and-uang naviga- tion with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7641-7649 2024.
Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre- trained transformer for 3d vision and text alignment. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 2911-2921, 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX SECTION
A.1 HUMAN EVALUATION
Human Scores. To evaluate the quality of Spartun3D based on human scores, we mainly consider the following two aspects.
Language Naturalness evaluates the flow and syntax of the language, ensuring it reads naturally as if written by human and adheres to common-sense knowledge in practical scenarios. Examples of different scoring levels for situations, questions, and answers are provided in [ For instance, a score of 1 might be assigned to scenario like "Stand- ing beside a blanket while there a toiler in the background," which is uncommon a typical setting. Similarly, questions such as "Which is cleaner, the trash bin or the bed?" are unlikely to be asked by humans, reflecting low language naturalness.
Spatial Fidelity is critical to ensure that the generated data accurately represents the infor- mation in 3D scenes and adheres to factual details. This includes verifying the presence of objects within the scene and ensuring that spatial relationships between objects are cor- rectly represented. Additionally, common sense knowledge must be considered, especially when unusual objects or scenarios are mentioned the questions or answers. For example, as shown in Fig. [2 a score of is assigned the instance where "clothes are hung on mirror." This error arises because the ror's affordance as "hanging," " which misled the GPT model into generating an incorrect dataset.
data from 3RScan labeled the mir-
Error Analysis. We randomly sampled 50 examples from each task (200 in total) to validate the quality of our automatically generated data and manually assess the quality from the aspects of language naturalness and spatial fidelity. Language naturalness evaluates whether the generated texts are natural or written by a human, while spatial fidelity ensures that data accurately reflects the 3DD scene. We observe 26 emror in total and summarize them into the following categories.
Semantic Errors: The generated sentences may contain semantic mistakes. For instance, the answer " You should go in front you to water the potted plant.".
The generated content may occasionally conflict with basic common sense knowledge, such as producing unusual questions and answers. For example, it might generate a question like "If want to store items, which object is most accessible?" with the answer being "trash bin." This issue arises because the human-annotated data in- cludes an affordance for trash bins as objects for storing items. Such annotations inadver- tently influence GPT-4o to generate QA pairs that conflict with common sense knowledge.
Common Sense Violations:
Spatial Inconsistencies: Errors in capturing or reasoning about spatial relationships in the 3D environment primarily occur in Situated Planning tasks, which demand complex two- step spatial reasoning. These errors often arise because the second action depends on the outcome of the first action, and inaccuracies sometimes occur during the generation of the second action.
Misalignment between visual context and textual descriptions. some cases, the agent's view is obstructed due to the room layout or object size. For example, consider the situation: "Standing beside the sofa, there is a closet on your left." However, the closet is actually located in another room and cannot be seen from the agent's current standpoint. To address this issue, we designed a scenario where the agent stands beside pivot object and consistently faces the center of the pivot object, rather than facing a random object that could potentially be obstructed. Additionally, we incorporated pass-by spatial information to enhance the agent's 's awareness of surrounding objects. providing a more comprehensive sense of the environment.
15

---

Published as a conference paper at ICLR 2025
SQA3D
55
Ground-truth
LEO
LEO SpartonnD
SpartunSD-LLAN
54.5
52.5
right left buckwward forward
52
Figure 10: SQA3D Labels Distribution
Figure 9: Scaling Effect on SQA3D.
Scores
Situation
Question How many sofa chairs are to my left?
Answer
5
You are standing beside the nightstand while there picture on your left.
You can use the lamp behind you, but careful of the stools you will pass by.
4
You are standing beside lamp while there is box your left.
Is the bag closer to than the trash bin behind me?
You can place items on the cab- inet your left, which is the closest option. Alternatively. you can place them on the bed to your left.
3 You are standing beside the radiator while there is cur- tain rod on your back.
want to read a book. Should I walk to the window or sit in the chair?
You can go to the clothing hang- ing on the door behind you.
2 You are standing beside salt while there is a kitchen cab- inet on your left.
What are the charac- teristics of the win- dow to my left?
Turn slightly back to your right and head towards the lamp move it closer to your bed. You may pass by the curtain rod. You should go in front of you water the potted plant.
1
You standing beside blanket while there is a toi- let your back.
Which is cleaner, the trash bin behind me the bed to my left?
Table 7: Examples to Evaluate Language Naturalness.
A.2 EXPERIMENTAL RESULTS
A.2.1 EVALUATION METRICS
CIDEr (Consensus-basse Image Description Evaluation): A metric that measures the sim- ilarity between generated and reference descriptions by assessing n-gram overlap, tailored for image captioning but applicable in NLP for structured comparisons.
METEOR (Metric for Evaluation of Translation with Explicit ORdering): A metric that evaluates machine translation and other language generation tasks, considering precision, recall, and harmonic mean, while accounting for synonyms and stemming.
ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation, Longest Common Subse- quence): This metric focuses on the longest common subsequence between generated and reference texts, providing insight into recall and precision in longer sequences.
EM: An evaluation metric that checks whether the predicted output matches the reference exactly, often used in tasks like question answering.
A.3 IMPLEMENTATION DETAILS.
The maximum context length and output length of LLM are both set to 256. For each 3D scene, we sample up to 60 objects with 1024 points per object. During training, the pre-trained 3D point cloud encoder and the I.LM are frozen. We set rank and a in LoRA to be 16 and dropout rate to
16

---

Published as a conference paper at ICLR 2025
What
Is
How
Can
Which
Otbers
CLIPBERT Ma 2022
39.7
65.0
10.5
$5.6
36.1
35.4
SQA3D Aarer
31.6
63.5
46.0
69.5
43.9
45.3
47.0
3D-Vista
740 Br 20.3
34.8
63.3
45.4
69.8
47.2
48.5
LE0
Huang er al
2023
46.8
64.1
47.0
60.8
44.2
54.3
Table 8: Exact Match Performance SQA3D Across Various Question Types
plvce
Ahve
Atyeera Eive
Q:
hanging door without turming around?
Score5
JN
12t
No
Q: object divectly of
REC
ahjece We
A: Mirror
Score3
Q: mnve.arything
nighttand.
volor
abjes
Score2
Q: Can waeh something my lef??
Situation: Standing beside blue octagon wide bed that messy while there window the left.
Q: If want hang my cloches, where should
your
Figure 11: Spartun3D Word Distri- bution.
Figure 12: Examples Evaluate Spatial Fidelity.
be During inference, we employ beam search to generate the textual response, and the number of beams is 5. The model is trained on 6 NVIDIA RTX A6000 GPU for around 30 hours with 15 epochs. The learning rate is - 5, and the batch size is 24.
A.4 EXTRA EXPERIMENTAL RESULTS
Different Text Encoder. We conducted an ablation study eval- uate the impact of different text encoders on enhancing 3D visual representations. Table shows the results on object attribute and re- lation tasks. Our results indicate that text encoders from VL models outperform pure text encoders, highlighting the importance of mul- timodal learning for 3D object understanding. Among the tested encoders, the CLIP encoder yields the best results.
Text Encoder
EM
BERT Reeimers 2019
53.5
BLIP Lr 2022
54.2
RRaiond 2021
Table 9: Different Text En- coder Evaluation.
17

---

Published as a conference paper at ICLR 2025
Table 10: Prompts for Different Tasks.
Prompts for Situated Captioning: Provide a summary of scene focusing on object types and attributes ON MY LEFT/RIGHT/FRON//BBAA
Describe the scene also considering common sense, such as how objects can he used by humans and human activities in the left part of the scene.The description should conform to the given seene information. You don't 't need to describe each object in the scene, pick some objects of the scene for summary. You can also summarize the room's function, style, and comfort level based on the arrangement and color of objects within the room. Your summary must be one paragraph, not exceeding 300 words. Don't use IDs of the objects in the summary. Don't 't use turn degrees or distance meters in the summary.
Prompts for Object Attributes and Relations: Ask questions about object lypes, and counting. The questions related to attributes are better be asked when multiple objects contain the same al- tributes and the answer can be specified based spatial relations. You also ask about spatial positions between me and other objects or spatial relations between objects by comparing the an- gles. Based on 'relations' in the scene graph, you can ask about other relations between objects. Prompts for Object Affordance: Ask questions about object affordance and object utility based on common sense. The answer should consider the best option that follows common sense knowledge and is closer to me. If plan to go some objects, other objects are blocking my way; please specify them. There are several examples:
Q: Where should I go to quickly put something down?A: You can use the chair in front of you.
Q: I want read book. Should I walk to the window or sit in the chair? A: Sit in the chair, which is closer.
Q: If want to reach the kitchen counter, what object will be passed by? Answer: tables with chairs.
Q: What should do if want to cook? A: You can go to the kitchen area, but be careful about the tables and chairs you will pass by.
Prompts for Situated Planning: You need to generale meaningful question-answer pairs that require multi-hop reasoning and planning based on the scene information.Ask questions about ob- ject affordance and object utility based on common sense and path planning. The question must be answered based on my position. If I plan to go some objects, other objects are blocking my way. please specify them. The turn action should be considered based angles if plan go to multiple places. Do not use the number of turn degrees or distance meters the question and answer. There are several examples:
1. Question: want to dim the lights and take a nap; What should I do? Answer: Turn to your right and head towards the lamp. Dim the lights, then turn slightly to your left and head towards the sofa to lie down.
2. Question: want to light up the area near the kitchen counter to prepre some food. How should proceed? Answer: Turn slightly to your left and head towards the blinds on your left to adjust them. Then, turn slightly back to your right and head towards the kitchen cabinet in front of you.
3. Question: need to adjust the lighting to make the room brighter. What should I do? Answer:
Turn to your left and head towards the lamp. Adjust the lighting. Then, turn slightly back to your right and ensure the curtains or blinds are open to let in more light.
4. Question: need to adjust the lighting to make the room brighter and then prepare snack on
the kitchen counter. How should proceed? Answer: Turn to your left and head towards the lamp to adjust the lighting. After adjusting the lighting, turn slightly back to your right and head towards the kitchen counter. You may pass tables and chairs on your way.
18

---

Published as a conference paper at ICLR 2025
Table 11: Prompts used to Generate Situated QA related to Object Attribute and Relation.
You need to generate at least 10 meaning- ful question-answer pairs based the scene information. Ask questions about object types, and counting. The questions related attributes are better be asked when multiple objects contain the same attributes and the answer can be specified based on spatial relations. You can also ask about spatial positions between me and other objects or spatial relations between objects by comparing the angles. Based on 'relations' in the scene graph, you can ask about other relations between objects. You need provide the queried object. Do not consider the object's utility and affordance. Do not use the number of turn degrees or distance meters in the question and answer. Do not use IDs of the objects in the question and answer. The question-anse answer pair should be following format: Q: <question>>1> T: <queried object_id(s) A: <Answer>, You can answer the question according to the queried ohject(s). If there is no information about the question, the <Answer>>should he "unkown". There several examples:
General Prompts for Object Attribute and Relation:
Q: What the object closest to the left of me? THlmpp Answer: lamp.
Q: How many stools are on my left? T:stool3 A: One.
Q: There are multiple chairs, what is the size of the chair left of me? T.chair. chair 2 A: low chair.
Q: Is the cabinet far from
the sofa far from me? A: sofa.
Q: How many black objects to my right? Two, a towel and a toilct brush.
Q: Where is the trash bin? A: Behind you.
Q: What color is the trash bin in front of me? A: black (one white left of me and one black in front
of me)
Q: the mirror right of the shower curtain based on my standing position? A: Yes.
Q: the light on my right on off?
Q: What the object to the left of the white heater to my right?
Q: Is there picture to my right?
Q: Is the door in front of me the same color as the cabinet my right?
Q: The your 11 clack direction on;
false ?
Q: Can black ohjects are to my right be divided by three? Coordinate prompt:
Spatial Prompt:
You are standing beside the white toilet_6, and the initial 3d coordinate is [-1.15, 0.29, 0.48] toilet.6. You are facing the center of the toilet. 6, and the center coordinate of toi- let_6 is [-1.36, 0.28, 0.48]. The scene con- tains some objects, which compose scene graph JSON format deseribing objects, such as object coordinate, color, size, shape, and state. You can calculate object distance and rotation angle related to your stand- ing point and orientation using coordinates. If the rotation angle is 360,0-45] is defined as Front, [45-135] is RIGHT, [135-225] is BACK, and [225-315] is right.
You are standing beside white toilet_6. The scene contains some objects, which compose a seene graph in JSON format with four keys: "left", "right". "front". "backwards", indicating objects in the corresponding direction. Each entity in the scene graph denotes object instance with class label and an object The "distance" indicates the meters between you and the object. The "angle" represents the degrees compared to your current di- rection, where your direction in front is O degrees. further right. The "af- fordance" the motion activity related to this oh- ject. The "attributes" describe the object's charac- teristics, such as 'color' and "size". The "relations" describe the spatial relationships with other objects. The "passby" indicates other objects in your path if you walk toward it from your current position.
The larger angles are
range [315-
For example, from the scene graph
"table.8"; "soordinate".
[1,1,1], "affordances": ("placing items on"], "attributes": "color":"red" "relations" ["close by chair.36").
For example, from the seene graph "distance":
"Left" " "table__" 2.6, "passby": ["chair,21"), "afoordances"; ("placing
items on"], ] "attributes":
We know that the coordinate of table_8 is [1.1.]], there is table.8 that is close by chair_36. You can place items this ble_8.
"color":"red", "angle": 257.48,
"relations"; ["close by chair chair_36"].
We know that on my left 257.48 degrees and 2.6 meters, there is a table 8 that is close by chair 36. You can place items on table 8. If you go to table 8, you could pass by chair.21.
19


Reviews:
[{"review_id": "HvmfCFD9ml", "reviewer": "Reviewer_d7E6", "summary": "This paper introduces Spartun3D, a scalable dataset designed to enhance situated 3D understanding. The authors construct a situated scene graph to facilitate the generation of situated captions and QA pairs via LLM prompting. Additionally, they incorporate a situated spatial alignment module into a baseline 3D LLM to improve scene understanding. Experimental results demonstrate the effectiveness of their approach.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The situated scene graph is well-crafted and contributes to a higher-quality dataset that supports future research in situated 3D scene understanding.\n2. The paper is well-written and easy to follow.", "weaknesses": "1. The paper could benefit from including more examples of generated data, particularly those with errors, to provide better insight into the dataset's quality. It would be valuable to discuss potential methods for correcting inaccurate captions or QA pairs generated by the LLM, as well as the impact of these errors on model performance.\n2. There is a lack of a detailed ablation study on the proposed modules, specifically the Situated Textual Description and 3D Object-Text Alignment, which would help clarify their individual contributions.\n3. The evaluation of object navigation relies on only four simple actions, which may weaken the findings, although it does showcase some zero-shot capabilities of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uTLPwMX7C8", "reviewer": "Reviewer_jrTE", "summary": "The paper addresses the task of situated reasoning where an agent is located at a certain location in the 3D space and needs to reason about the situation (caption or answer questions) from it's spatial location. The paper generates a new dataset called Spartan3D, relying on GPT-4o that lets them scale the size of the data; and proposes an explicit spatial alignment module where the object tokens are aligned to a desciption of spatial situations/relations of the corresponding object obtained via a template. The method is tested on situation question answering benchmark of SQA3D and the newly proposed Spartan3D dataset. The results show that the explicit spatial alignment module helps in question answering as well as other captioning tasks; and the additional Spartan3D dataset helps performance on all datasets.", "questions": "- Some additional discussion / proof on why the additional automated data is useful. Scaling curves with varying amount of Spartan3D-LLM data used in training would help -- on real world benchmarks like SQA3D.\n\n- Clarification on the newly constructed dataset from 3RScan for zero-shot LEO baselines, and generalization capabilities of the proposed model\n\n- More details on the navigation experiments in Table-5; specifically regarding the training datasets used for the proposed model and the baselines.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is very well written and clearly lays down the problem and the proposed solution (with informative figures)\n- The Spartan3D dataset will be useful to the community\n- The proposed method obtains better results than prior state of the art on SQA3D and the proposed explicit alignment loss helps performance in all tasks (by about 1-2%)", "weaknesses": "- A big claim of the paper is that SQA3D is human collected dataset; and the proposed pipeline to generate SPARTUN3D can be very useful to scale up the dataset size. However, it is unclear how effective this automatically generated dataset is in comparison to human generated dataset. Specifically, the current experiments of Table-3 show that on real-world SQA3D benchmarks, using additional Spartan3D dataset on top of human collected SQA3D dataset helps performance by about 1-2% despite much larger dataset size. Additionally, just training on Spartan3D dataset results in significantly worse performance on SQA3D (by about 20%). Moreover, it is unclear if the automated data shows strong scaling behaviors -- which, if true, would push the community to scale up the datasets in a similar way that this paper proposed instead of collecting more human annotations.\n\n - L460-465 \"In the zeroshot setting, we re-trained LEO exclusively on a dataset constructed from 3RScan to ensure a fair\ncomparison with our method. As shown in Table 3, LEO performs poorly on SQA3D in the zeroshot setting, suggesting its limitations in learning situated understanding from its original dataset. In contrast, LEO trained on Spartun3D shows significant improvement, demonstrating the effectiveness of our dataset and the generalization ability of our model.\"\n\nThe above set of lines are confusing to me. \n- What exactly is this dataset constructed from 3RScan which is constructed for the zero-shot LEO baseline? Without this information, currently the conclusion seems to be that Spartan3D dataset is better than some other way of constructing a training dataset.\n- I did not follow how obtaining good performance from LEO trained on Spartun3D leads to the \"generalization ability of our model\" conclusion? Table-3 does not show how the proposed model works with the \"newly constructed 3RScan data\" instead of the Spartan3D dataset. \n\n- For navigation experiments in Table-5. what is Spartan3D-LLM trained on? Is LEO and Spartan3D-LLM identical except the explicit spatial alignment module for this experiment? I am trying to understand the main reason why LEO does not work at all for navigation while Spartan3D-LLM show some non-zero performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p7mQYiZGwX", "reviewer": "Reviewer_D9vh", "summary": "This paper introduces SPARTUN3D, a dataset aimed at enhancing the spatial reasoning abilities of 3D-based large language models (LLMs) by providing situated context. The dataset includes tasks like situated captioning and QA, challenging models to respond based on the agent's dynamic perspective. Additionally, the authors propose a novel alignment module in Spartun3D-LLM, which improves alignment between 3D scene representations and textual descriptions. The paper demonstrates the model’s generalization on multiple 3D tasks, outperforming baselines in spatial understanding and navigation accuracy.", "questions": "1) By using GPT4 API, how to ensure the quality and accuracy of datasets? Can you provide more information on quality control measures for SPARTUN3D? Specifically, how are errors from the automated pipeline identified and handled?\n2) How does the Situated Spatial Alignment Module impact computational resources, particularly in training time and memory usage? Would the module be feasible for real-time applications?\n3) Have you considered evaluating Spartun3D-LLM on other embodied tasks beyond QA and navigation, such as robotic manipulation, multi-step reasoning in dynamically changing environments or some other 3D tasks like visual grounding, 3D object detection or dense caption?\n4) Your chosen baselines like Leo,3D-Vista and 3D-LLM, were relesed in 2023,  the paper lacks a broader comparison with recent advances, please show some comparisons of other advances of 3D-LLMs with SPARTUN3D-LLM?\n5) As the alignment is one of your contributions of this paepr. Are there other alignment strategies that were considered or experimented with?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Innovative Dataset: SPARTUN3D addresses a key limitation in 3D-based LLMs by providing an extensive dataset that incorporates situated reasoning. This dataset, generated by leveraging GPT-4o, is both scalable and diverse, which is essential for training models in 3D scene understanding from dynamic perspectives. This is a well-motivated and relevant contribution to the field.\n\n2) Effective Alignment Module: The Situated Spatial Alignment Module enhances model performance by aligning 3D visual representations with their textual counterparts. This module appears to be a critical contribution as it enables more accurate spatial understanding and has implications for embodied tasks. The use of spatial self-attention and a 3D object-text alignment loss to improve object-text coherence is particularly novel.\n\n3) Comprehensive Experiments: The experimental evaluation is thorough, covering both in-domain tasks (Spartun3D and SQA3D) and out-of-domain tasks (MP3D ObjNav), with significant improvements shown in zero-shot navigation accuracy. Ablation studies and visualizations further confirm the efficacy of the alignment module in improving spatial reasoning capabilities.\n\n4) Practical Applications: Situated spatial understanding is valuable for real-world applications, such as navigation and human-robot interaction. This paper makes a strong case for the applicability of SPARTUN3D in these areas.", "weaknesses": "1) The SPARTUN3D dataset is automatically generated with GPT-4o, which, while scalable, may lack the nuanced spatial and contextual fidelity found in human-annotated data. This synthetic nature raises questions about the dataset’s generalizability and its ability to represent real-world scenarios accurately.\n\n2) The evaluation primarily focuses on situated QA, captioning, and navigation tasks, which, although useful, may not fully capture the complexity of embodied tasks (e.g., robotic manipulation or multi-step planning). Expanding the evaluation to include a wider variety of tasks would provide a more comprehensive assessment of the model's situated understanding abilities.\n\n3) While the Situated Spatial Alignment Module is an interesting addition, the paper lacks a rigorous theoretical foundation for its design. Details on why specific techniques (e.g., spatial self-attention and MSE for alignment) were chosen and how they uniquely contribute to spatial alignment are not thoroughly explained, which could weaken the perception of this module’s novelty. Also, the added complexity of the alignment module may lead to increased computational demands, yet the paper does not discuss or benchmark these costs. For practical deployment in real-time applications or on resource-limited systems, it is essential to understand the trade-offs between the module’s benefits and its computational impact.\n\n4) While comparisons to the LEO (2023.11 released) baseline and other similar models are present, the paper lacks a broader comparison with recent advances in spatial reasoning, navigation models or 3D tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AOWwT86CqS", "reviewer": "Reviewer_kEmt", "summary": "SPARTUN3D addresses key limitations in current 3D LLMs:\n\n- Existing 3D datasets are lack of situated context\n- Current 3D LLM's architecture lack explicit alignment between the spatial representations of 3D scenes and natural language.\n\nTo tackle these issues, this paper introduces a novel dataset, SPARTUN3D, along with a model enhancement named SPARTUN3D-LLM, featuring a dedicated spatial alignment module. Experimental results validate that both the dataset and model improvements significantly advance the situated spatial reasoning capabilities of 3D LLMs.", "questions": "(1). The input to the model is triple <C, S, Q>, where C is the 3D scene context, S is the situation, and Q is a question. The situation S can be further denoted as $S^t$, $S^p$, $S^r$, where $S^p$ is the 3D coordinate in the form <x, y, z> and $S^r$ is the quaternion. In the dataset, line 182, authors assume the agent's orientation is always facing forward to the center of the selected object. How is this selected object chosen? Will this introduce any bias?\n\n(2). Can author show how they make inference of SPARTUN3D? More specifically, what is S, C, and Q respectively? Is it possible to provide an example?\n\n(3). Why does finetuned 3D-LLM only has EM score? I understand 3D-Vista is a BERT but 3D-LLM seems not?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1). This paper is very well-motivated. The importance of situational reasoning in 3D-LLM is beyond doubt.\n\n(2). The writing of the paper is sound. \n\n(3). This paper covers a range of tasks that are less explored by 3D-LLMs.", "weaknesses": "(1). The motivation behind this dataset is strong, and I acknowledge the considerable effort put into curating it. However, the dataset’s quality remains unverified. While the authors took steps to avoid unnatural configurations (see line 176), I am not yet convinced of its robustness. Specifically, the input scene graph includes angular relationships calculated using the centroids of three objects. Given the variability in object sizes, it is uncertain whether GPT-4o can accurately interpret and annotate these spatial relations. What evidence supports the notion that these spatial relationships are meaningful to GPT-4o, and do these annotations align with human perception?\n\n(2). GPT-4o’s capacity to generate large volumes of data makes it ideal for annotating 3D scenes at scale. However, the paper lacks scaling experiments to demonstrate how model performance improves as more data is generated using this pipeline. Including such experiments would provide valuable insight into the benefits of scaling for this approach.\n\n(3). The input to the model is triple <C, S, Q>, where C is the 3D scene context, S is the situation, and Q is a question. The situation S can be further denoted as $S^t$, $S^p$, $S^r$, where $S^p$ is the 3D coordinate in the form <x, y, z> and $S^r$ is the quaternion. Can author provide more analysis on how <x, y, z> and $S^r$ would affect models' performance?\n\n(4). The setting is egocentric situational 3D-LLM, but SPARTUN3D takes mostly text information as input. Is it possible to directly takes a 2D image as egocentric condition?\n\n(5). For the experiments Navigation Performance in zero-shot manner, LEO's training data is very simple. It will be interesting to see a comparison between finetuned LEO with SPARTUN3D and Sparttun3D-LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)