SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Model

## Abstract

Abstract Integrating the 3D world into large language models (3D-based LLMs) has been a promising research direction for 3D scene understanding. However, current 3D-based LLMs fall short in situated understanding due to two key limitations: 1) existing 3D datasets are constructed from a global perspective of the 3D scenes and lack situated context.
2) the architectures of existing 3D-based LLMs lack explicit alignment between the spatial representations of 3D scenes
and natural language, limiting their performance in tasks requiring precise spatial reasoning.
We address these issues by introducing a scalable situated 3D dataset, named Spartun3D , that incorporates various situated spatial reasoning tasks.
Furthermore, we propose Spartun3D-LLM , built on an existing 3D-based LLM but integrated with a novel situated spatial alignment module, aiming to enhance the alignment between 3D visual representations and their corresponding textual descriptions. Experimental results demonstrate that both our proposed dataset and alignment module significantly enhance the situated spatial understanding of 3D-based LLMs.

## 1 Introduction

Figure: Figure 1: Illustration of situated scene understanding of Spartun3D-LLM compared to other 3D-based LLMs.
Refer to caption: x1.png

Recent advances in large language models (LLMs) have demonstrated their remarkable reasoning and communication capabilities across various tasks and modalities (Achiam et al., ; Alayrac et al., ; Zhang et al., ; Rubenstein et al., ). Building on these breakthroughs, there has been a growing interest in extending LLMs to the 3D world (3D-based LLMs) (Hong et al., ; Huang et al., ; Chen et al., ; Zhen et al., ; Wang et al., ).
Existing studies mainly focus on integrating various 3D scene representations into LLMs, enabling the models to perform 3D grounding and spatial reasoning through natural language. For example, 3D-LLM (Hong et al., ) utilizes multi-view images to represent 3D scenes, pioneering a new direction in this field, while LEO (Huang et al., ) further pushes the boundary by directly injecting 3D point clouds into LLMs, aiming to develop a generalist embodied agent capable of 3D grounding, embodied reasoning, and action planning.

Despite the promising progress, current 3D-based LLMs still fall short in situated understanding, a fundamental capability for completing embodied tasks, such as Embodied Question Answering (Das et al., ), Vision and Language Navigation (Anderson et al., ; Wang et al., ; Zhang et al., ), robotic manipulation (Shridhar et al., ), and many others.
Situated understanding refers to the ability to interpret and reason about a 3D scene from a dynamic egocentric perspective, where the agent must continuously adjust understanding based on its changing position and evolving environment around it.
This capability is crucial because an agent’s reasoning and response to the same question can vary depending on its current situation.
For example, as shown in Fig 1, given the same question “What should you do to wash hands?”, the agent might need to answer “use the sink on the right” or “use the sink on the left”, depending on the agent’s current perspective and location relative to the “sink”.

However, achieving such situated understanding remains challenging for current 3D-based LLMs, and we identify two primary reasons.
First, most existing 3D datasets (Chen et al., ; Azuma et al., ; Zhu et al., ; Huang et al., ) are constructed from a global perspective of 3D scenes, lacking the situated information necessary for training models to reason from an agent’s perspective.
As a result, models fine-tuned on these datasets struggle to develop situated reasoning ability.
While the introduction of SQA3D (Ma et al., ) has made progress by providing a situated 3D dataset, the dataset is mainly human-annotated, making it expensive and difficult to scale for the large-scale training needed by 3D-based LLMs.
Second, existing 3D-based LLMs inject 3D representations into LLMs by simply concatenating tokens from different modalities (e.g., text, images, 3d point clouds). While this allows for basic cross-modal interaction,
it lacks an explicit mechanism to align the situated spatial information from the 3D scene with natural language. Therefore, the models struggle to capture the critical spatial relationships for situated understanding.

To address the aforementioned issues, we propose two key innovations: we first introduce a scalable, LLM-generated dataset named Spartun3D, consisting of approximately $133$k examples.
Different from datasets used by previous 3D-based LLMs (Zhu et al., ; Huang et al., ; Hong et al., ), Spartun3D incorporates various situated spatial information conditioned on the agent’s standing point and orientation within the environment, consisting of two situated tasks:
situated captioning and situated QA.
Situated captioning is our newly proposed task that requires generating descriptions of the surrounding objects and their spatial direction based on the agent’s situation.
Situated QA is designed with different types of questions targeting various levels of spatial reasoning ability for embodied agents.
Furthermore, based on Spartun3D, we propose a new 3D-based LLM, Spartun3D-LLM, which is built on the most recent state-of-the-art 3D-based LLM, LEO (Huang et al., ), but integrated with a novel situated spatial alignment module that explicitly aligns 3D visual objects, their attributes and spatial relationship to surrounding objects with corresponding textual descriptions, with the goal of better bridging the gap between the 3D and text spaces.

Figure: Figure 2: Examples of two tasks in Spartun3D Benchmark. The green bounding box and arrow in the 3D scene demonstrate the standing point and orientation.
Refer to caption: x2.png

We conduct extensive experiments across a variety of tasks, including Spartun3D, Situated QA (Ma et al., ), and MP3D Nav (Savva et al., ).
Our results demonstrate that our model, trained on Spartun3D, exhibits strong generalization to other tasks, specifically in zero-shot settings, highlighting the effectiveness of our proposed dataset. Additionally, Spartun3D-LLM outperforms the baseline on nearly all tasks, indicating that incorporating direct text supervision improves the model’s spatial understanding ability. This is further supported by our observation that the explicit alignment module improves the generation of more fine-grained, context-aware spatial information.

## 2 Related Work

#### Situated Scene Understanding

is essential for various embodied tasks, including embodied QA (Das et al., ; Wijmans et al., ; Majumdar et al., ), vision-and-language navigation (Anderson et al., ; Savva et al., ; Gu et al., ; Zhang & Kordjamshidi, ; Zhang et al., ; Hong et al., ; Zhang & Kordjamshidi, ; ) and robotic manipulation (Shridhar et al., ; Jiang et al., ; Driess et al., ).
Recently, SQA3D (Ma et al., ) introduces a human-annotated dataset where the model generates answers based on questions and given situations.
SIG3D (Man et al., ) highlights the situated awareness via a situation-grounded 3D VL reasoning architecture.
While SQA3D is crucial
in 3D vision-language learning (Zhu et al., ; Huang et al., ; Hong et al., ), its reliance on human annotations makes it costly and difficult to scale for the large-scale training required by 3D-based LLMs.
In contrast, we use the LLM to design an automated pipeline to generate situated questions and answers, enhancing both the scalability and diversity in data collection procedure.

#### Grounded 3D Scene Understanding.

Compared to 2D vision-language tasks (Lin et al., ; Antol et al., ; Zhang et al., ), 3D scenes introduce additional dimensions of knowledge
that is more challenging to align with text modalities.
Early studies in this area primarily focused on grounding language to individual objects within 3D environments (Achlioptas et al., ; ; Chen et al., ; ; ).
Recently, 3D Vista (Zhu et al., ) proposes a pre-trained VL Transformer for 3D vision and text alignment, and a few works utilize LLMs in understanding 3D scenes (Hong et al., ; Huang et al., ; Chen et al., ; Zhen et al., ; Yang et al., ).
However, these works are less effective in situated understanding tasks that require generating answers from the agent’s dynamic perspectives, which is the key challenge that this work is tackling.

Figure: Figure 3: Illustration of Spartun3D Dataset Construction Process. Given a 3D scene: a) we first select a pivot object (e.g., sofa) and a referent object (e.g., table) to define the situation; b) we create situated scene graph based on situation, incorporating various spatial relationships (See Fig. 5); c) we use the scene graph to prompt GPT-4o (shown in gray box) to generate data; e) we utilize different prompting strategies for generating situated captions and QA pairs.
Refer to caption: x3.png

## 3 Spartun3D Dataset Construction

To better equip 3D-based LLMs with the capability of understanding situated 3D scenes, we introduce Spartun3D, a diverse and scalable situated 3D dataset.
To ensure the scalability of Spartun3D, we carefully design an automatic pipeline that leverages the strong capabilities of GPT-4o (OpenAI, ), with three key stages as shown in Fig. 3:
(1) Designing diverse situations that specify the agent’s standing point and orientation given a 3D scene as input (Sec. 3.1); (2) Constructing situated scene graphs to describe the spatial relationships between the agent and objects in the environment conditioned on the agent’s situations (Sec. 3.2) ; and (3) Prompting LLMs to generate dataset based on situated scene graphs (Sec. 3.3).

### 3.1 Situation Design

The 3D scenes in Spartun3D are taken from 3RScan (Wu et al., ), which provides a diverse set of realistic 3D environments. Given a particular 3D scene with all the objects labeled by humans from 3RScan, such as the example shown in Fig. 3, our first step is to generate diverse situations for the agent. To construct the situation, we begin by identifying the standing point and orientation and then complete a situation description accordingly using the following template:
“You are standing beside {pivot object name}, and there is {referent object name} on the {left/right/front/backward}.”
The elements within {} specify the key components that together define the situation.
Below, we define the agent’s standing point and orientation and explain how these elements are obtained to construct diverse and reliable situations.

Figure: Figure 4: Standing Point and Orientation Selection.
Refer to caption: x4.png

#### Standing Point and Orientation.

We begin with determining the agent’s standing point and orientation within the 3D scene. Our approach is to place the agent beside an object, ensuring a clear reference for orientation when interacting with the environment. Specifically, we project all objects from 3D space onto a 2D plane, focusing only on the x and y coordinates to construct a bird-eye-view of the scene. From this projected 2D scene, we randomly select an object from the set of segmented objects within the 3D scene. To ensure the constructed situation remains realistic, we exclude objects that are positioned too high to avoid unnatural situations like “standing beside the lamp on the ceiling”.
As a result, we limit the selection to objects whose z-axis is below the average height of all objects in the scene.
Then, we choose a midpoint from two sides of the selected object’s bounding box that are closest to the center of the scene, as shown in Fig. 4.
By prioritizing the side closest to the center, we minimize this risk and keep the agent within the scene’s boundaries. Finally, the selected midpoint will be used as the agent’s standing point.
In addition, we need to determine the agent’s orientation. We assume the agent’s orientation is always facing forward to the center of the selected object. This guarantees that the selected object remains within the agent’s field of view.

#### Pivot and Referent Object.

Once the agent’s standing point and orientation are determined, we refer to the object that the agent stands beside as the “pivot object”, and other objects surrounding the pivot object are potential referent objects. A referent object is then randomly selected, and its relative position (left/right/front/backward), with respect to the agent’s standing point and orientation, is used to generate the description of the situation.

### 3.2 Situated Scene Graph Construction

Building on the agent’s situation, we further construct a situated scene graph that captures the
comprehensive spatial relationships between the agent and its surrounding objects.
Existing 3D-based LLMs (Zhu et al., ; Huang et al., ) represent scenes in a structured manner using JSON-formatted scene graphs, including detailed scene context of object attributes and relative spatial relationships between objects.
However, their spatial relations are based on a global view, such as a bird-view-eye perspective (as shown in Fig. 5).
To enable situated understanding, we introduce a
situated-scene-graph adapted from the original global scene graph to capture all relative spatial relationships between the agent’s standing point and surrounding objects as follows:

- •
Rotation Angles. We calculate rotation angles that reorient the agent from its orientation to the surrounding objects. Specifically, we first calculate the horizontal angle between the standing point and the center of the pivot object. Next, we calculate the horizontal angle between the standing point and a surrounding object. The rotation angle is determined by the difference between these two angles. We further normalize the rotation angles such that larger values correspond to a greater degree of rightward rotation.
- •
Direction. We classify the object’s rotation angles to the agent into four directional categories according to a predefined standard: [front, right, backward, left] (see Fig. 5 (b)). For instance, an object is categorized as “right” if the turn angle falls within the range of [45-135] degrees relative to the agent’s forward-facing orientation.
- •
Distance. We compute the Euclidean distance between the agent’s standing point and the center of the bounding boxes of surrounding objects.
- •
Passby Objects. We assess whether
the agent can move freely from its standing point to other objects. We draw a straight line from the agent’s standing point to the center of the referenced object. If this line intersects any other objects in the scene, those objects are considered “passby objects”. For example, as illustrated in Fig 5 (d), the “table” is a passby object between the agent and the “kitchen cabinet”. We explictly include the information of passby objects to help the agent build awareness of objects that might influence its path while navigating.

Figure: Figure 5: Spatial information in Situated Scene Graph. The red dot and green arrow show the standing point and orientation, respectively. In this example, the pivot object is the “sofa”, the referent object is the “TV”, and the surrounding objects include the “table” and “cabinet”.
Refer to caption: x5.png

After gathering the spatial information described above, we organize it into a JSON format (shown in Fig. 5 (e)), which is then used as input to prompt LLMs to generate our datasets.

### 3.3 LLMs prompting

We design specific instructions to prompt GPT-4o (OpenAI, ) for two situated tasks: Situated Captioning and Situated QA.
For both tasks, we ask GPT-4o to provide responses considering situated spatial information.
Detailed prompts for each task are provided in Tab. 8 in the Appendix, and examples of the generated dataset are shown in Fig. 2.

#### Situated Captioning

is our newly introduced task, aiming to generate brief situated descriptions of the surrounding objects as the agent performs a $360^{\circ}$ clockwise rotation starting from its standing point and orientation.
The motivation for introducing this task stems from its crucial role in embodied tasks, such as navigation, where the agent must interpret and reason about its environment from $360^{\circ}$ panoramic views to make decisions about movement and interaction (Zhang & Kordjamshidi, ; Zhou et al., ; Zhang et al., ).
Therefore, we guide GPT-4o to generate descriptions progressively, starting from lower rotation angles and moving toward higher angles in each direction.
However, in practice, certain directions may not contain any objects, leading GPT-4o to omit those areas. To address this, we employ a template-based approach that enforces the model to generate descriptions for every direction. If there is no object in a certain direction, the description should mention the absence of objects. Each direction description is then combined with a sequence of directions: front, right, backward, and left, following a clockwise rotation.

#### Situated QA.

We design three types of questions for the Situated QA task, each targeting a different aspect of spatial reasoning for embodied agents.
Unlike previous works that rely on a single generic prompt for all question types, we develop tailored prompting strategies for each question type, encouraging LLM to generate QA pairs focusing on different levels of reasoning.

- •
Object Attribute and Relations include questions about objects attributes, such as color, shape, and size, while also incorporating situated spatial information. For instance, the questions to identify “the color of the table positioned to the left”, and determine “how many pictures are hanging on the wall to the right”.
- •
Object Affordance focuses on the function utility of the objects, often based on common sense knowledge about how objects are used. Similarly, we require situated spatial information to be part of the answer.
For example, when asked “Where can you check your appearance?”, the correct answer should be  “mirror on your left”, specifying both the object name (mirror) and its spatial location from the agent.
- •
Situated Planning is the most challenging task, as it requires the agent to perform multi-hop situated spatial reasoning.
The agent must not only recognize its surroundings but also plan and execute a series of actions across multiple steps, where each subsequent action depends on the outcome of the previous one.
In our dataset, we implement 2-hop reasoning, which requires the agent to perform a sequence of two continuous actions. For example, given the example in Fig. 2, “make the room brighter and then sit down to relax.”,
the agent needs to first turn left from its orientation to face and move toward the window, open it to brighten the room, then based on its new position, the agent continues turning left toward the sofa chairs and sits down.

### 3.4 Dataset Statistics and Quality Evaluation

In total, we collect approximately 10k situated captions and 123k QA pairs. For the tasks of object attribute and relation and the tasks of affordance, we sampled around $10$ situations per scene. For captioning and planning tasks, we sample around $5$ situations per scene due to the increasing cost of longer token sequences required for these tasks. For each task, we split the data instances into a Training and Test set. Table 1 shows the statistics of our dataset.

**Table 1: Dataset statistics of Spartun3D and human validation results.**
| Tasks | # of Examples | Train/Test | Cord-prompt | Spa-prompt |
| --- | --- | --- | --- | --- |
| Captioning | $\sim 10$K | $8,367$/$1,350$ | $27\%$ | $86\%$ |
| Attr. & Rel. | $\sim 62$K | $61,254$/$8,168$ | $57\%$ | $90\%$ |
| Affordance | $\sim 40$K | $35,070$/$5,017$ | $40\%$ | $87\%$ |
| Planning | $\sim 21$K | $19,434$/$2,819$ | $33\%$ | $87\%$ |

Figure: Figure 6: Spartun3D-LLM Model Architecture.
Refer to caption: x6.png

To evaluate the quality of Spartun3D, we randomly sampled approximately $50$ examples for each task and manually assessed their validity. Each example is evaluated based on whether the generated captions or answers are accurate given the 3D scene and the agent’s situation.
We experiment with two types of prompts for representing spatial information to prompt GPT-4O
: Cord-prompt, which consists of
object center coordinates, standing point, orientation, and instructions for calculating distances and rotation angles,
and Spa-prompt, consisting of the
calculated angles and distance based on the approaches we described in Sec. 3.3. An example of each type of prompt can be found in Tab. 10 in the Appendix.
Tab. 1 shows the percentage for valid outputs based on different prompts. The results indicate that Cord-prompt yields unsatisfactory results, revealing that LLMs lack strong 3D reasoning capabilities when tasked with interpreting raw spatial coordinates. Spa-prompt helps improve the quality of the generated dataset by providing more straightforward spatial information. The high valid percentage based on Spa-prompt also demonstrates the high quality of our dataset.

## 4 Model Architecture

In addition to enhancing the situated understanding of 3D-based LLMs with Spartun3D, we also propose a new 3D-based LLM, named Spartun3D-LLM, which integrates a novel
Situated Spatial Alignment module to strengthen the alignment between the situated 3D visual features and their corresponding textual descriptions. Spartun3D-LLM is built upon LEO (Huang et al., ), which represents the most recent and state-of-the-art 3D-based LLM, and directly takes 3D point cloud data as input, making it well-suited for spatial reasoning tasks in 3D environments.
Fig. 6
illustrates the overview architecture of Spartun3D-LLM.

### 4.1 Background

#### Problem Formulation.

We formally define the input as a triple $<C,S,Q>$, where $C$ is the 3D scene context, $S$ is the situation, and $Q$ is a question. The situation $S$ can be further denoted as $S=<S^{t},S^{p},S^{r}>$, where $S^{t}$ is a textual situation description, and $S^{p}$ and $S^{r}$ are the standing points and orientation, respectively. Specifically, $S^{p}$ is a 3D coordinate in the form $<x,y,z>$ and $S^{r}$ is the quaternion $<qx,qy,qz,w>$, where $<qx,qy,qz>$ is the rotation axis and $w$ is the rotation angle. For simplicity, we define $z=0$ to calculate the rotation angle on a 2D plane.
The task is to generate a textual answer, denoted as $A$, given scene context $C$, situation $S$, and question $Q$. During training, $S^{p}$ and $S^{r}$ are provided to the agent to rotate and translate the environment, while during testing, only questions and situations are provided.

#### Backbone.

LEO (Huang et al., ) takes the text, 2D image (optional), and 3D point clouds as input and formulate comprehensive 3D tasks as autoregressive sequence generation. Specifically, data from different modalities are converted into a sequence of tokens as input to the LLM.
The text tokens include system messages (e.g., “You are an AI visual assistant situated in a 3D scene.”), situations, and questions.
These tokens are then embedded into vector representations using an embedding look-up table.
For 3D point clouds, LEO first applies segmentation masks to extract the point clouds of individual objects in the 3D scenes.
Then, the sampled points of each object are input into a object-centric point cloud encoder, PointNet++ (Qi et al., ) pre-trained on ScanNet (Dai et al., ), to obtain the object-level representations.

Formally, we denote the representation of input text tokens as $\mathbf{W}=[\mathbf{w}_{1},\mathbf{w}_{2},...,\mathbf{w}_{M}]\in\mathbb{R}^{M
\times D}$
, where $M$ denotes the number of input tokens, and $D$ represents the dimensionality of each token’s embedding. Additionally, the input object visual representations are expressed as $\mathbf{O}=[\mathbf{o}_{1},\mathbf{o}_{2},...,\mathbf{o}_{K}]\in\mathbb{R}^{K
\times D}$, where
$K$ is the number of extracted objects from the scene. Finally, the output answer are represented as $\mathbf{A}=[\mathbf{a}_{1},\mathbf{a}_{2},...,\mathbf{a}_{N}]$, where $N$ is the number of tokens in the response.
The model’s objective is to generate the answer given these combined inputs. The loss for generating the $i$-th token of the output answer is formulated as follows:

$$ $\mathcal{L}_{\texttt{LM}}(\theta)=\sum_{i}logp_{\theta}(\mathbf{a}_{i}|\mathbf {a}_{i-1},\mathbf{W_{S}},\mathbf{o}).\vspace{-4mm}$ (1) $$

LEO can integrate various LLM backbones, including OPT1.3B (Zhang et al., ) and Vicuna-7B (Chiang et al., ).
In our experiments, we fine-tune LEO with different LLM backbones on our proposed dataset via LoRA (Hu et al., ).

### 4.2 Situated Spatial Alignment Module

Situated tasks require robust spatial reasoning abilities to comprehend the position, orientation, and spatial relationships of objects within a 3D environment.
Existing 3D-based LLMs typically process inputs by concatenating output representations from various modality encoders.
While this method facilitates the integration of data across different modalities, it does not inherently ensure that the 3D visual representations encode situated spatial information or effectively align with textual descriptions, which potentially limits the model’s ability to perform tasks that require precise spatial understanding.
To tackle this challenge, we introduce a novel Situated Spatial Alignment Module to improve the alignment between the object-centric 3D visual representations and their situated textual descriptions.
The process begins by generating detailed situated textual descriptions for each object.
Subsequently, an alignment loss is introduced, which directs the model in effectively learning the 3D visual representations based on these situated textual descriptions.

#### Situated Textual Descriptions.

For each object, we construct a comprehensive situated textual description based on a template that captures the object’s name, attributes, and spatial relations with nearby objects, as “Stand besides {object name} and facing the center of the {object name}, in front, there are {a list of nearby objects}; on the right, …; behind …; and on the left…”. The object’s attributes are also considered (e.g., “white chair”).
We consider up to five objects per direction. If no object is present in a specific direction, the description explicitly states this,yueensuring to provide complete information about the 3D environment.

#### 3D Object-Text Alignment.

Inspired by the success of 2D Visual-Language models, which effectively leverage semantically aligned text and visual features to excel in downstream tasks (Radford et al., ; Li et al., ; ; Wang et al., ), we aim to enhance the 3D visual representations so that they can better encode the situated spatial information and effectively align with the textual descriptions.
Specifically, we introduce a 3D object-text alignment loss to guide the learning process of point cloud encoders within 3D-based LLMs, leveraging the robust language representations captured by pre-trained text encoders. We experiment with various text encoders, and CLIP achieved the best performance. For more details, please refer to the Sec. A.3 in the Appendix.

More formally, we obtain the text representations of situated textual description for each object from pre-trained text encoders, denoted as $\mathbf{W}=[\mathbf{w}_{1},\mathbf{w}_{2},...,\mathbf{w}_{k}]\in\mathbb{R}^{K
\times D}$.
For object visual representations $\mathbf{O}$, we employ spatial self-attention layers (Chen et al., ) to learn spatial-aware object representations. Specifically, a pairwise spatial feature matrix $\mathbf{F}\in\mathbb{R}^{K\times K\times 5}$ is introduced to represent relative spatial relations between objects.
For example, for each object pairs $\mathbf{o}_{i}$ and $\mathbf{o}_{j}$, we construct pairwise spatial feature as $\mathbf{f}_{ij}=[d_{ij},sin(\theta_{h}),cos(\theta_{h}),sin(\theta_{v}),cos(
\theta_{v})]\in\mathbb{R}^{1\times 5}$, where $d_{ij}$ is Euclidean distance between two objects, and $\theta_{h}$ and $\theta_{v}$ are horizontal and vertical angles connecting bounding box centers of $\mathbf{o}_{i}$ and $\mathbf{o}_{j}$, respectively.
Then, we inject $\mathbf{F}$ into the self-attention of the object as,

$$ $\mathbf{O}^{\prime}=\texttt{softmax}(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d_{ h}}}+\texttt{MLP}(\mathbf{F}))\mathbf{V},\text{where}\,\,Q=W_{Q}O;K=W_{K}O,V=W _{V}O,$ (2) $$

where $\mathbf{O}^{\prime}\in\mathbb{R}^{K\times D}$ denotes the spatial-aware object representation;
Then we use a Mean Squared Error (i.e., MSE) as the objective function to minimize the distance between the object representation $\mathbf{O}^{\prime}$ and the corresponding situated textual embedding $\mathbf{W}$, denoted as
$\mathcal{L}_{\texttt{align}}=\texttt{MSE}(\mathbf{o^{\prime}},\mathbf{w}_{t})$. The model is trained to jointly optimize both the alignment loss and the language modeling loss (Eq. 1) as $\mathcal{L}=\mathcal{L}_{LM}+\mathcal{L}_{\texttt{align}}$.

## 5 Experiment

**Table 2: Experimental Results on Spartun3D Situated QA Tasks. $*$ represents the model initialized with LEO instruction-tuned weights. [Keys: C: CIDER; B-4: BLEU-4; M: METEOR; R: ROUGE; Sim: Sentence Similarity; EM: Exact Match; Bold: best results].**
| Models | LLMs | Attributes and Relations | Affordance | Situated Planning |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C | B-4 | M | R | EM | C | B-4 | M | R | S | C | B-4 | M | R | S |  |  |
| LEO | zero-shot | $100.3$ | $0.00$ | $17.4$ | $39.1$ | $42.7$ | $13.3$ | $0.00$ | $3.00$ | $5.00$ | $32.3$ | $0.00$ | $0.00$ | $7.00$ | $15.3$ | $59.2$ |
| LEO+Spartun3D | OPT1.3B | $121.3$ | $7.0$ | $20.1$ | $45.3$ | $47.7$ | $224.6$ | $30.6$ | $24.9$ | $53.2$ | $66.9$ | $229.7$ | $44.8$ | $32.1$ | $60.9$ | $83.8$ |
| Vicuna7B | $125.4$ | $10.1$ | $22.1$ | $46.7$ | $52.1$ | $238.9$ | $32.1$ | $24.4$ | $55.0$ | $68.3$ | $242.1$ | $46.5$ | $35.2$ | $63.1$ | $84.3$ |  |
| LEO*+Spartun3D | Vicuna7B | $129.2$ | $10.4$ | $23.0$ | $48.1$ | $53.2$ | $211.3$ | $32.1$ | $24.6$ | $55.0$ | $67.8$ | $247.1$ | $47.5$ | $36.2$ | $65.1$ | $85.8$ |
| Spartune3D-LLM | OPT1.3B | $124.1$ | $9.2$ | $21.0$ | $47.3$ | $49.4$ | $227.2$ | $31.4$ | $26.3$ | $54.1$ | $68.2$ | $232.3$ | $45.2$ | $33.2$ | $62.1$ | $85.4$ |
| Vicuna7B | $131.2$ | $10.3$ | $24.3$ | $48.8$ | $53.7$ | $240.4$ | $32.1$ | $25.0$ | $55.3$ | $68.7$ | $244.0$ | $47.1$ | $36.4$ | $64.0$ | $86.8$ |  |
| Spartune3D-LLM* | Vicuna7B | $\mathbf{135.4}$ | $10.7$ | $24.9$ | $51.3$ | $\mathbf{56.9}$ | $\mathbf{254.7}$ | $\mathbf{32.9}$ | $\mathbf{26.7}$ | $\mathbf{57.3}$ | $\mathbf{69.7}$ | $\mathbf{252.1}$ | $47.6$ | $36.2$ | $\mathbf{65.4}$ | $\mathbf{88.7}$ |

### 5.1 Experimental Setup

To demonstrate the effectiveness of our proposed Spartun3D-LLM, we conduct experiments on two situated understanding datasets, including Spartun3D and SQA3D (Ma et al., ). For SQA3D, we evaluate under two conditions: object proposals in 3D are derived either from Mask3D (Schult et al., ) or ground-truth annotations. Also, we assess the transferability of our method on the navigation task using MP3D ObjNav (Savva et al., ). Following LEO (Huang et al., ), we report the performance using standard generation metrics, including CIDEr, METEOR, BLEU-4, and ROUGE_L, sentence similarity (Reimers, ) for captioning task. For SQA3D and situated QA tasks of questions about attributes and relations, we also report an additional metric of exact-match accuracy.
More details about metrics are introduced in Appendix A.1.
We also provide implementation details in Sec. A.2. We leverage LEO as baseline.
Since the training stage in LEO has covered most of the evaluation tasks, we experiment with models initialized from scratch to ensure a fair comparison in the zero-shot setting. For other settings, we report the performance of models initialized both from scratch and from the instruction-tuned LEO.
To distinguish between the two, models initialized from the instruction-tuned LEO are marked with an asterisk (^∗).

**Table 3: Experimental Results on SQA3D given the 3D objects from Mask3D and Ground-truth.**
|  | Methods | Mask3D (Schult et al., ) | GT |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | C | M | R | EM | C | M | R | EM |
| zero-shot | LEO (Huang et al., ) | $14.2$ | $6.4$ | $8.2$ | $12.4$ | $15.3$ | $6.7$ | $8.6$ | $13.9$ |
| LEO+Spartun3D | $82.3$ | $14.2$ | $32.8$ | $34.7$ | $83.1$ | $15.2$ | $33.7$ | $35.9$ |  |
|  | Spartun3D-LLM | $83.5$ | $15.7$ | $34.7$ | $36.2$ | $85.6$ | $16.6$ | $35.8$ | $37.1$ |
| Fine-tune | 3D-Vista (Zhu et al., ) | - | - | - | $48.5$ | - | - | - | - |
| 3D-LLM (Hong et al., ) | - | - | - | $50.2$ | - | - | - | - |  |
| LEO (Huang et al., ) | $132.0$ | $33.0$ | $49.2$ | $52.4$ | $132.3$ | $34.3$ | $51.4$ | $52.5$ |  |
| LEO*+Spartun3D | $134.0$ | $34.6$ | $52.2$ | $53.5$ | $135.3$ | $34.2$ | $52.1$ | $54.2$ |  |
| Spartun3D-LLM* | $\mathbf{138.2}$ | $\mathbf{35.3}$ | $\mathbf{53.4}$ | $\mathbf{54.8}$ | $\mathbf{138.3}$ | $\mathbf{35.4}$ | $\mathbf{53.7}$ | $\mathbf{55.0}$ |  |

### 5.2 Experimental Results

**Table 4: Experimental Results on Spartun3D Situated Captioning Task.**
| Models | LLMs | C | B-4 | M | R | S |
| --- | --- | --- | --- | --- | --- | --- |
| LEO | zero-shot | $0.00$ | $0.00$ | $9.00$ | $15.3$ | $51.9$ |
| LEO+Spartun3D | OPT1.3B | $5.9$ | $15.3$ | $17.7$ | $31.2$ | $67.3$ |
| Vicuna7B | $6.7$ | $15.8$ | $18.7$ | $32.3$ | $70.4$ |  |
| LEO+Spartun3D* | Vicuna7B | $14.1$ | $17.2$ | $22.6$ | $32.1$ | $76.3$ |
| Spartun3D-LLM | OPT1.3B | $6.4$ | $15.7$ | $18.5$ | $31.2$ | $68.6$ |
| Vicuna7B | $8.5$ | $16.4$ | $19.6$ | $32.5$ | $72.5$ |  |
| Spartun3D-LLM* | Vicuna7B | $\mathbf{14.6}$ | $\mathbf{19.3}$ | $\mathbf{23.3}$ | $\mathbf{33.4}$ | $\mathbf{78.1}$ |

#### Spartun3D Benchmark

We evaluate the performance of both the LEO model and Spartun3D-LLM after fine-tuning them on our proposed Spartun3D dataset. The fine-tuned LEO model is referred to as LEO+Spartun3D.
Table 2 and Table 4 show the experimental results on Situated Captioning and Situated QA tasks, respectively.
We experiment with two different LLM backbones: Opt1.3B and Vicuna7B.
Our experiments show that Spartun3D-LLM consistently outperforms LEO+Spartun3D across all question types (around $2\%-3\%$ across all metrics), regardless of the LLM backbone used, indicating the effectiveness of our explicit alignment module.
We observe that initializing our model with LEO pre-trained weights improves performance. Notably, without fine-tuning, LEO performs reasonably well on attribute and relation questions in a zero-shot setting but struggles with other situated tasks.

#### SQA3D Performance.

We further evaluate our method on the SQA3D dataset, whose scenes are derived from ScanNet (Dai et al., ). Their scenes differ from those in Spartun3D, which are sourced from 3RScan. We experiment with two settings: zero-shot and fine-tuning.
In the zero-shot setting, we re-trained LEO exclusively on a dataset constructed from 3RScan to ensure a fair comparison with our method.
As shown in Table 3, LEO performs poorly on SQA3D in the zero-shot setting, suggesting its limitations in learning situated understanding from its original dataset.
In contrast, LEO trained on Spartun3D shows significant improvement, demonstrating the effectiveness of our dataset and the generalization ability of our model.
Furthermore, in the fine-tuning setting,
Spartun3D-LLM continues to outperform LEO across all metrics.
Table 9 in the Appendix provides a breakdown of the fine-tuned performance across various question types, where Spartun3D-LLM shows consistent improvement across nearly all question types.

**Table 5: Performance on Navigation. (Accuracy %)**
|  | LEO | Spartun3D-LLM |
| --- | --- | --- |
| Zero-shot | 0 0 | $\mathbf{20.3}$ |

#### Navigation Performance.

To demonstrate the effectiveness of our approach on downstream embodied tasks, we evaluate it on the object navigation tasks. Specifically, we randomly select $5$ scenes that contain around $1000$ examples from the MP3D ObjNav dataset.
In this task, we additionally input 2D ego-centric images to both LEO and Spartun3D-LLM for comparison.
There are four types of navigation actions: turn left, turn right, move forward, and stop.
We evaluate whether the model generates correct action at each step.
We conduct the experiment in a zero-shot setting, and Table 5 shows the accuracy of the model’s performance.
The baseline model, LEO, struggles to generate the required action-related text to guide navigation steps without fine-tuning specifically for navigation tasks.
In contrast, our model demonstrates strong transferability to generate correct actions.
Fig 7 (e) showcases a qualitative example, illustrating how our model effectively generates accurate navigation actions without task-specific fine-tuning.

Figure: Figure 7: Qualitative Examples. (a), (c), (c) situated qa examples of object attribute and relation, object affordance, and situated planning, respectively. (d) situated captioning example. (e) navigation example in a zero-shot setting. Our model generates both actions and descriptions of surrounding objects while navigating.

### 5.3 Ablation Study and Extra Analysis

#### Explicit Alignment Enhances General Spatial Understanding.

We evaluate the effectiveness of our proposed situated spatial alignment module on general scene understanding tasks, such as

**Table 6: Spatial Alignment Evaluation on Other Benchmarks. The metric is Sentence Similarity (%).**
| Methods | Scan2Cap | ScanQA |
| --- | --- | --- |
| LEO + Spartun3D | $54.2$ | $46.3$ |
| Spartun3D-LLM | $\mathbf{55.7}$ | $\mathbf{48.6}$ |

Scan2Cap (Chen et al., ) and ScanQA (Azuma et al., ).
In line with our approach for situated tasks, we construct textual descriptions for each object based on its attributes and spatial relations to others from a top-view perspective.
As shown in Tab. 6, by incorporating the explicit spatial alignment module, our model shows better results, indicating that our proposed alignment module not only improves situated understanding but also enhances general 3D scene understanding.

#### Improved Situated Understanding.

To analyze the model’s situated understanding ability further, we visualize the distribution of model responses generated for questions requiring strong spatial understanding from SQA3D. Specifically, we extract questions starting with “which direction”. Fig. 9 illustrates the distribution of generated “directions”, including “left”, “right”, “forward” and “backward”. We observe that LEO is biased towards generating “left” $97\%$ of the time. However, the ground-truth distribution of “left” and “right” should be balanced, suggesting that LEO may have a limited understanding of situated spatial relationships. In contrast, our model produces a distribution that closely matches the ground truth, demonstrating improved situated understanding.

#### Qualitative Examples.

In Fig. 7, we showcase several successful examples to demonstrate the effectiveness of Spartun3D-LLM across various situated tasks.
Notably, in Fig 7 (c), the model without an explicit alignment module tends to generate more general or vague spatial descriptions, such as “turn around”. In contrast, with the alignment module, the model produces more specific details, including terms like “turn slightly right”. To verify this, we examine $30$ examples from both situated planning and situated captioning tasks and observe this phenomenon in $17$ of them.
This highlights how the proposed spatial alignment module enhances the generation of fine-grained spatial information, leading to more precise and contextually accurate outputs.

## 6 Discussion and Conclusion

In this work, our goal is to address the limitation of situated understanding of the 3D-based LLMs from two perspectives. First, we propose a method to construct an LLM-generated dataset based on our designed situated scene graph. Then, we propose an explicit situated spatial alignment on the 3D-LLM to encourage the model to learn alignment between 3D object and their textual representations directly. Finally, we provide comprehensive experiments to show our own benchmark and improve situated understanding of SQA3D and navigation. We also provide analysis to show our proposed explicit alignment module helps spatial understanding.

## Appendix A Appendix Section

### A.1 Evaluation Metrics

- •
CIDEr (Consensus-based Image Description Evaluation): A metric that measures the similarity between generated and reference descriptions by assessing n-gram overlap, tailored for image captioning but applicable in NLP for structured comparisons.
- •
METEOR (Metric for Evaluation of Translation with Explicit ORdering): A metric that evaluates machine translation and other language generation tasks, considering precision, recall, and harmonic mean, while accounting for synonyms and stemming.
- •
ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation, Longest Common Subsequence): This metric focuses on the longest common subsequence between generated and reference texts, providing insight into recall and precision in longer sequences.
- •
EM: A stringent evaluation metric that checks whether the predicted output matches the reference exactly, often used in tasks like question answering.

Figure: Figure 8: Word distribution of questions of object attribute and relation.
Refer to caption: x7.png

### A.2 Implementation Details.

The maximum context length and output length of LLM are both set to $256$. For each 3D scene, we sample up to $60$ objects with $1024$ points per object. During training, the pre-trained 3D point cloud encoder and the LLM are frozen. We set rank and $\alpha$ in LoRA to be $16$ and dropout rate to be 0 0. During inference, we employ beam search to generate the textual response, and the number of beams is $5$.
The model is trained on a $6$ NVIDIA RTX A6000 GPU for around $30$ hours with $15$ epochs. The learning rate is $3e-5$, and the batch size is $24$.

**Table 7: Different Text Encoder Evaluation.**
| Text Encoder | EM |
| --- | --- |
| SBERT (Reimers, ) | $53.5$ |
| BLIP (Li et al., ) | $54.2$ |
| CLIP (Radford et al., ) | $\mathbf{56.9}$ |

### A.3 Different Text Encoder.

We conducted an ablation study to evaluate the impact of different text encoders on enhancing 3D visual representations. Table 7 shows the results on object attribute and relation tasks. Our results indicate that text encoders from VL models outperform pure text encoders, highlighting the importance of multimodal learning for 3D object understanding. Among the tested encoders, the CLIP encoder yield the best results.

**Table 8: Prompts for Different Tasks.**
| Prompts for Situated Captioning:<br>Provide a summary of a scene focusing on object types and attributes ON MY LEFT/RIGHT/FRONT/BACK. Describe the scene also considering common sense, such as how objects can be used by humans and human activities in the left part of the scene.The description should conform to the given scene information. You don’t need to describe each object in the scene, pick some objects of the scene for summary. You can also summarize the room’s function, style, and comfort level based on the arrangement and color of objects within the room. Your summary must be one paragraph, not exceeding 300 words. Don’t use IDs of the objects in the summary. Don’t use turn degrees or distance meters in the summary. |
| --- |
| Prompts for Object Attributes and Relations: Ask questions about object types, and counting. The questions related to attributes are better be asked when multiple objects contain the same attributes and the answer can be specified based on spatial relations. You can also ask about spatial positions between me and other objects or spatial relations between objects by comparing the angles. Based on ’relations’ in the scene graph, you can ask about other relations between objects. |
| Prompts for Object Affordance:<br>Ask questions about object affordance and object utility based on common sense. The answer should consider the best option that follows common sense knowledge and is closer to me. If I plan to go to some objects, other objects are blocking my way; please specify them. There are several examples:<br>Q: Where should I go to quickly put something down?A: You can use the chair in front of you.<br>Q: I want to read a book. Should I walk to the window or sit in the chair? A: Sit in the chair, which is closer.<br>Q: If I want to reach the kitchen counter, what object will be passed by? Answer: tables with chairs.<br>Q: What should I do if I want to cook? A: You can go to the kitchen area, but be careful about the tables and chairs you will pass by. |
| Prompts for Situated Planning: You need to generate 6 meaningful question-answer pairs that require multi-hop reasoning and planning based on the scene information.Ask questions about object affordance and object utility based on common sense and path planning. The question must be answered based on my position. If I plan to go to some objects, other objects are blocking my way, please specify them. The turn action should be considered based on angles if I plan to go to multiple places. Do not use the number of turn degrees or distance meters in the question and answer.<br>There are several examples:<br>1. Question: I want to dim the lights and take a nap; What should I do? Answer: Turn to your right and head towards the lamp. Dim the lights, then turn slightly to your left and head towards the sofa to lie down.<br>2. Question: I want to light up the area near the kitchen counter to prepre some food. How should I proceed? Answer: Turn slightly to your left and head towards the blinds on your left to adjust them. Then, turn slightly back to your right and head towards the kitchen cabinet in front of you.<br>3. Question: I need to adjust the lighting to make the room brighter. What should I do?<br>Answer: Turn to your left and head towards the lamp. Adjust the lighting. Then, turn slightly back to your right and ensure the curtains or blinds are open to let in more light.<br>4. Question: I need to adjust the lighting to make the room brighter and then prepare a snack on the kitchen counter. How should I proceed? Answer: Turn to your left and head towards the lamp to adjust the lighting. After adjusting the lighting, turn slightly back to your right and head towards the kitchen counter. You may pass tables and chairs on your way. |

Figure: Table 9: Exact Match Performance on SQA3D Across Various Question Types
Refer to caption: x8.png

**Table 10: Prompts used to Generate Situated QA related to Object Attribute and Relation.**
| General Prompts for Object Attribute and Relation:<br>You need to generate at least 10 meaningful question-answer pairs based on the scene information. Ask questions about object types, and counting. The questions related to attributes are better be asked when multiple objects contain the same attributes and the answer can be specified based on spatial relations. You can also ask about spatial positions between me and other objects or spatial relations between objects by comparing the angles. Based on ’relations’ in the scene graph, you can ask about other relations between objects. You need to provide the queried object. Do not consider the object’s utility and affordance. Do not use the number of turn degrees or distance meters in the question and answer. Do not use the IDs of the objects in the question and answer. The question-answer pair should be following format: Q: <question>T: <queried object_id(s)>A: <Answer>. You can answer the question according to the queried object(s).<br>If there is no information about the question, the <Answer>should be “unkown”.<br>There are several examples:<br>Q: What is the object closest to the left of me?<br>T:lamp_1<br>Answer: a lamp.<br>Q: How many stools are on my left? T:stool_3<br>A: One.<br>Q: There are multiple chairs, what is the size of the chair left of me?<br>T:chair_1, chair_2<br>A: low chair.<br>Q: Is the cabinet far from me or the sofa far from me? A: sofa.<br>Q: How many black objects are to my right? A: Two, a towel and a toilet brush.<br>Q: Where is the trash bin? A: Behind you.<br>Q: What color is the trash bin in front of me? A: black (one white left of me and one black in front of me)<br>Q: Is the mirror right of the shower curtain based on my standing position? A: Yes.<br>Q: Is the light on my right on or off?<br>Q: What is the object to the left of the white heater to my right?<br>Q: Is there a picture to my right?<br>Q: Is the door in front of me the same color as the cabinet to my right?<br>Q: The tv to your 11 o ’ clock direction on; true or false ?<br>Q: Can black objects are to my right be divided by three? |  |
| --- | --- |
| Coordinate prompt:<br>You are standing beside the white toilet_6, and the initial 3d coordinate is [-1.15, 0.29, 0.48] toilet_6. You are facing the center of the toilet_6, and the center coordinate of toilet_6 is [-1.36, 0.28, 0.48]. The scene contains some objects, which compose a scene graph in JSON format describing objects, such as object coordinate, color, size, shape, and state.<br>You can calculate object distance and rotation angle related to your standing point and orientation using coordinates. If the rotation angle is in the range [315-360,0-45] is defined as Front, [45-135] is RIGHT, [135-225] is BACK, and [225-315] is right.<br>For example, from the scene graph<br>"table_8": "coordinate": [1,1,1], "affordances": ["placing items on"], "attributes": "color":"red", "relations": ["close by chair_36"].<br>We know that the coordinate of table_8 is [1,1,1], there is a table_8 that is close by chair_36. You can place items on this table_8. | Spatial Prompt:<br>You are standing beside a white toilet_6. The scene contains some objects, which compose a scene graph in JSON format with four keys: “left”, “right”, “front”, “backwards”, indicating objects in the corresponding direction. Each entity in the scene graph denotes an object instance with a class label and an object ID. The “distance” indicates the meters between you and the object. The “angle” represents the degrees compared to your current direction, where your direction in front is 0 degrees. The larger angles are means further right. The “affordance” is the motion activity related to this object. The “attributes” describe the object’s characteristics, such as ’color’ and “size”. The “relations” describe the spatial relationships with other objects. The “passby” indicates other objects in your path if you walk toward it from your current position.<br>For example, from the scene graph<br>"Left":"table_8": "distance": 2.6, "passby": ["chair_21"], "affordances": ["placing items on"], "attributes": "color":"red", "angle": 257.48, "relations": ["close by chair_36"].<br>We know that on my left 257.48 degrees and 2.6 meters, there is a table_8 that is close by chair_36. You can place items on table_8. If you go to table_8, you could pass by chair_21. |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weakness section.
soundness: 3
strengths: 1. The situated scene graph is well-crafted and contributes to a higher-quality dataset that supports future research in situated 3D scene understanding.
2. The paper is well-written and easy to follow.
summary: This paper introduces Spartun3D, a scalable dataset designed to enhance situated 3D understanding. The authors construct a situated scene graph to facilitate the generation of situated captions and QA pairs via LLM prompting. Additionally, they incorporate a situated spatial alignment module into a baseline 3D LLM to improve scene understanding. Experimental results demonstrate the effectiveness of their approach.
weaknesses: 1. The paper could benefit from including more examples of generated data, particularly those with errors, to provide better insight into the dataset's quality. It would be valuable to discuss potential methods for correcting inaccurate captions or QA pairs generated by the LLM, as well as the impact of these errors on model performance.
2. There is a lack of a detailed ablation study on the proposed modules, specifically the Situated Textual Description and 3D Object-Text Alignment, which would help clarify their individual contributions.
3. The evaluation of object navigation relies on only four simple actions, which may weaken the findings, although it does showcase some zero-shot capabilities of the model.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Some additional discussion / proof on why the additional automated data is useful. Scaling curves with varying amount of Spartan3D-LLM data used in training would help -- on real world benchmarks like SQA3D.

- Clarification on the newly constructed dataset from 3RScan for zero-shot LEO baselines, and generalization capabilities of the proposed model

- More details on the navigation experiments in Table-5; specifically regarding the training datasets used for the proposed model and the baselines.
soundness: 3
strengths: - The paper is very well written and clearly lays down the problem and the proposed solution (with informative figures)
- The Spartan3D dataset will be useful to the community
- The proposed method obtains better results than prior state of the art on SQA3D and the proposed explicit alignment loss helps performance in all tasks (by about 1-2%)
summary: The paper addresses the task of situated reasoning where an agent is located at a certain location in the 3D space and needs to reason about the situation (caption or answer questions) from it's spatial location. The paper generates a new dataset called Spartan3D, relying on GPT-4o that lets them scale the size of the data; and proposes an explicit spatial alignment module where the object tokens are aligned to a desciption of spatial situations/relations of the corresponding object obtained via a template. The method is tested on situation question answering benchmark of SQA3D and the newly proposed Spartan3D dataset. The results show that the explicit spatial alignment module helps in question answering as well as other captioning tasks; and the additional Spartan3D dataset helps performance on all datasets.
weaknesses: - A big claim of the paper is that SQA3D is human collected dataset; and the proposed pipeline to generate SPARTUN3D can be very useful to scale up the dataset size. However, it is unclear how effective this automatically generated dataset is in comparison to human generated dataset. Specifically, the current experiments of Table-3 show that on real-world SQA3D benchmarks, using additional Spartan3D dataset on top of human collected SQA3D dataset helps performance by about 1-2% despite much larger dataset size. Additionally, just training on Spartan3D dataset results in significantly worse performance on SQA3D (by about 20%). Moreover, it is unclear if the automated data shows strong scaling behaviors -- which, if true, would push the community to scale up the datasets in a similar way that this paper proposed instead of collecting more human annotations.

 - L460-465 "In the zeroshot setting, we re-trained LEO exclusively on a dataset constructed from 3RScan to ensure a fair
comparison with our method. As shown in Table 3, LEO performs poorly on SQA3D in the zeroshot setting, suggesting its limitations in learning situated understanding from its original dataset. In contrast, LEO trained on Spartun3D shows significant improvement, demonstrating the effectiveness of our dataset and the generalization ability of our model."

The above set of lines are confusing to me. 
- What exactly is this dataset constructed from 3RScan which is constructed for the zero-shot LEO baseline? Without this information, currently the conclusion seems to be that Spartan3D dataset is better than some other way of constructing a training dataset.
- I did not follow how obtaining good performance from LEO trained on Spartun3D leads to the "generalization ability of our model" conclusion? Table-3 does not show how the proposed model works with the "newly constructed 3RScan data" instead of the Spartan3D dataset. 

- For navigation experiments in Table-5. what is Spartan3D-LLM trained on? Is LEO and Spartan3D-LLM identical except the explicit spatial alignment module for this experiment? I am trying to understand the main reason why LEO does not work at all for navigation while Spartan3D-LLM show some non-zero performance.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1) By using GPT4 API, how to ensure the quality and accuracy of datasets? Can you provide more information on quality control measures for SPARTUN3D? Specifically, how are errors from the automated pipeline identified and handled?
2) How does the Situated Spatial Alignment Module impact computational resources, particularly in training time and memory usage? Would the module be feasible for real-time applications?
3) Have you considered evaluating Spartun3D-LLM on other embodied tasks beyond QA and navigation, such as robotic manipulation, multi-step reasoning in dynamically changing environments or some other 3D tasks like visual grounding, 3D object detection or dense caption?
4) Your chosen baselines like Leo,3D-Vista and 3D-LLM, were relesed in 2023,  the paper lacks a broader comparison with recent advances, please show some comparisons of other advances of 3D-LLMs with SPARTUN3D-LLM?
5) As the alignment is one of your contributions of this paepr. Are there other alignment strategies that were considered or experimented with?
soundness: 3
strengths: 1) Innovative Dataset: SPARTUN3D addresses a key limitation in 3D-based LLMs by providing an extensive dataset that incorporates situated reasoning. This dataset, generated by leveraging GPT-4o, is both scalable and diverse, which is essential for training models in 3D scene understanding from dynamic perspectives. This is a well-motivated and relevant contribution to the field.

2) Effective Alignment Module: The Situated Spatial Alignment Module enhances model performance by aligning 3D visual representations with their textual counterparts. This module appears to be a critical contribution as it enables more accurate spatial understanding and has implications for embodied tasks. The use of spatial self-attention and a 3D object-text alignment loss to improve object-text coherence is particularly novel.

3) Comprehensive Experiments: The experimental evaluation is thorough, covering both in-domain tasks (Spartun3D and SQA3D) and out-of-domain tasks (MP3D ObjNav), with significant improvements shown in zero-shot navigation accuracy. Ablation studies and visualizations further confirm the efficacy of the alignment module in improving spatial reasoning capabilities.

4) Practical Applications: Situated spatial understanding is valuable for real-world applications, such as navigation and human-robot interaction. This paper makes a strong case for the applicability of SPARTUN3D in these areas.
summary: This paper introduces SPARTUN3D, a dataset aimed at enhancing the spatial reasoning abilities of 3D-based large language models (LLMs) by providing situated context. The dataset includes tasks like situated captioning and QA, challenging models to respond based on the agent's dynamic perspective. Additionally, the authors propose a novel alignment module in Spartun3D-LLM, which improves alignment between 3D scene representations and textual descriptions. The paper demonstrates the model’s generalization on multiple 3D tasks, outperforming baselines in spatial understanding and navigation accuracy.
weaknesses: 1) The SPARTUN3D dataset is automatically generated with GPT-4o, which, while scalable, may lack the nuanced spatial and contextual fidelity found in human-annotated data. This synthetic nature raises questions about the dataset’s generalizability and its ability to represent real-world scenarios accurately.

2) The evaluation primarily focuses on situated QA, captioning, and navigation tasks, which, although useful, may not fully capture the complexity of embodied tasks (e.g., robotic manipulation or multi-step planning). Expanding the evaluation to include a wider variety of tasks would provide a more comprehensive assessment of the model's situated understanding abilities.

3) While the Situated Spatial Alignment Module is an interesting addition, the paper lacks a rigorous theoretical foundation for its design. Details on why specific techniques (e.g., spatial self-attention and MSE for alignment) were chosen and how they uniquely contribute to spatial alignment are not thoroughly explained, which could weaken the perception of this module’s novelty. Also, the added complexity of the alignment module may lead to increased computational demands, yet the paper does not discuss or benchmark these costs. For practical deployment in real-time applications or on resource-limited systems, it is essential to understand the trade-offs between the module’s benefits and its computational impact.

4) While comparisons to the LEO (2023.11 released) baseline and other similar models are present, the paper lacks a broader comparison with recent advances in spatial reasoning, navigation models or 3D tasks.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: (1). The input to the model is triple <C, S, Q>, where C is the 3D scene context, S is the situation, and Q is a question. The situation S can be further denoted as $S^t$, $S^p$, $S^r$, where $S^p$ is the 3D coordinate in the form <x, y, z> and $S^r$ is the quaternion. In the dataset, line 182, authors assume the agent's orientation is always facing forward to the center of the selected object. How is this selected object chosen? Will this introduce any bias?

(2). Can author show how they make inference of SPARTUN3D? More specifically, what is S, C, and Q respectively? Is it possible to provide an example?

(3). Why does finetuned 3D-LLM only has EM score? I understand 3D-Vista is a BERT but 3D-LLM seems not?
soundness: 2
strengths: (1). This paper is very well-motivated. The importance of situational reasoning in 3D-LLM is beyond doubt.

(2). The writing of the paper is sound. 

(3). This paper covers a range of tasks that are less explored by 3D-LLMs.
summary: SPARTUN3D addresses key limitations in current 3D LLMs:

- Existing 3D datasets are lack of situated context
- Current 3D LLM's architecture lack explicit alignment between the spatial representations of 3D scenes and natural language.

To tackle these issues, this paper introduces a novel dataset, SPARTUN3D, along with a model enhancement named SPARTUN3D-LLM, featuring a dedicated spatial alignment module. Experimental results validate that both the dataset and model improvements significantly advance the situated spatial reasoning capabilities of 3D LLMs.
weaknesses: (1). The motivation behind this dataset is strong, and I acknowledge the considerable effort put into curating it. However, the dataset’s quality remains unverified. While the authors took steps to avoid unnatural configurations (see line 176), I am not yet convinced of its robustness. Specifically, the input scene graph includes angular relationships calculated using the centroids of three objects. Given the variability in object sizes, it is uncertain whether GPT-4o can accurately interpret and annotate these spatial relations. What evidence supports the notion that these spatial relationships are meaningful to GPT-4o, and do these annotations align with human perception?

(2). GPT-4o’s capacity to generate large volumes of data makes it ideal for annotating 3D scenes at scale. However, the paper lacks scaling experiments to demonstrate how model performance improves as more data is generated using this pipeline. Including such experiments would provide valuable insight into the benefits of scaling for this approach.

(3). The input to the model is triple <C, S, Q>, where C is the 3D scene context, S is the situation, and Q is a question. The situation S can be further denoted as $S^t$, $S^p$, $S^r$, where $S^p$ is the 3D coordinate in the form <x, y, z> and $S^r$ is the quaternion. Can author provide more analysis on how <x, y, z> and $S^r$ would affect models' performance?

(4). The setting is egocentric situational 3D-LLM, but SPARTUN3D takes mostly text information as input. Is it possible to directly takes a 2D image as egocentric condition?

(5). For the experiments Navigation Performance in zero-shot manner, LEO's training data is very simple. It will be interesting to see a comparison between finetuned LEO with SPARTUN3D and Sparttun3D-LLM.

Meta-review: **Summary**

The paper aims to study the ability of 3D-LLMs to perform situated spatial reasoning, where given 3D scene and an agent's location and pose, the agent needs to either provide descriptions of surrounding objects (situated captioning) or answer questions (situated question-answering).  To do so, the paper introduces Spartun3D, a situated 3D dataset consisting of 133K examples generated using LLMs for evaluating captioning and QA.  The paper also proposes Spartun3D-LLM, which adds a situated spatial alignment module to a recent 3D-based LLM (LEO). Experiments show that the proposed module improves performance on the proposed Spartun3D benchmark as well as other 3D language tasks.

The main contributions is the Spartun3D dataset, the proposed situated spatial alignment module, and the experiments demonstrating the effectives of the proposed approach.

**Strengths**

Reviewers noted the following strengths of the work:
- The dataset is useful for the community [d7E6,jrTE,D9vh]
- Reviewers found the paper to be well-written and easy to follow [d7E6,jrTE,kEmt].  The AC also find the problem formulation, dataset, and proposed module to be clearly described.
- Experiments show proposed module is effective [jrTE,D9vh] 
- Tasks investigated by this work is underexplored [kEmt]

**Weaknesses**

Reviewers noted the following weaknesses:
- Concerns about the use of LLM to generate the data [jrTE,D9vh,kEmt]  including 
  - Quality of the data compared to human generated data
  - Whether scaling the data up actually can help train better models 
- Some aspects where not initially clear [jrTE,kEmt] 
- The evaluation can include more tasks and recent methods [D9vh]
- Missing discussion of computational costs [D9vh]
- Reviewer also requested additional details such as:
  - More examples of generated data [d7E6]
  - Detailed ablation study [d7E6]

The main common concern across reviewers was the quality of the generated data and request for additional information on how scaling the data affects the model performance.  This concern (as well as other concerns) where addressed by the authors during the author response period.

**Recommendation**

Overall reviewers are slightly positive on this work.  The AC believes the dataset can be useful, and the paper is clear and so recommends acceptance.

Decision: Accept (Poster)