SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Model

Authors: Yue Zhang, Zhiyang Xu, Ying Shen, Parisa Kordjamshidi, Lifu Huang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Situated Understanding in 3D Scen, 3D VL, LLM

Abstract:
Integrating the 3D world into large language models (3D-based LLMs) has been a promising research direction for 3D scene understanding. However, current 3D-based LLMs fall short in situated understanding due to two key limitations: 1) existing 3D datasets are constructed from a global perspective of the 3D scenes and lack situated context.
2) the architectures of the current 3D-based LLMs lack an explicit mechanism for aligning situated spatial information between 3D representations and natural language, limiting their performance in tasks requiring precise spatial reasoning. 
In this work, we address these issues by introducing a scalable situated 3D dataset, named Spartun3D, that incorporates various situated spatial information.
In addition, we propose a situated spatial alignment module to enhance the learning between 3D visual representations and their corresponding textual descriptions. Our experimental results demonstrate that both our dataset and alignment module enhance situated spatial understanding ability.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730269906123

thread_type:
official_review

<thread_object>
creator: reviewer_kemt

<content>
summary:
SPARTUN3D addresses key limitations in current 3D LLMs:

- Existing 3D datasets are lack of situated context
- Current 3D LLM's architecture lack explicit alignment between the spatial representations of 3D scenes and natural language.

To tackle these issues, this paper introduces a novel dataset, SPARTUN3D, along with a model enhancement named SPARTUN3D-LLM, featuring a dedicated spatial alignment module. Experimental results validate that both the dataset and model improvements significantly advance the situated spatial reasoning capabilities of 3D LLMs.

strengths:
(1). This paper is very well-motivated. The importance of situational reasoning in 3D-LLM is beyond doubt.

(2). The writing of the paper is sound. 

(3). This paper covers a range of tasks that are less explored by 3D-LLMs.

weaknesses:
(1). The motivation behind this dataset is strong, and I acknowledge the considerable effort put into curating it. However, the dataset’s quality remains unverified. While the authors took steps to avoid unnatural configurations (see line 176), I am not yet convinced of its robustness. Specifically, the input scene graph includes angular relationships calculated using the centroids of three objects. Given the variability in object sizes, it is uncertain whether GPT-4o can accurately interpret and annotate these spatial relations. What evidence supports the notion that these spatial relationships are meaningful to GPT-4o, and do these annotations align with human perception?

(2). GPT-4o’s capacity to generate large volumes of data makes it ideal for annotating 3D scenes at scale. However, the paper lacks scaling experiments to demonstrate how model performance improves as more data is generated using this pipeline. Including such experiments would provide valuable insight into the benefits of scaling for this approach.

(3). The input to the model is triple <C, S, Q>, where C is the 3D scene context, S is the situation, and Q is a question. The situation S can be further denoted as $S^t$, $S^p$, $S^r$, where $S^p$ is the 3D coordinate in the form <x, y, z> and $S^r$ is the quaternion. Can author provide more analysis on how <x, y, z> and $S^r$ would affect models' performance?

(4). The setting is egocentric situational 3D-LLM, but SPARTUN3D takes mostly text information as input. Is it possible to directly takes a 2D image as egocentric condition?

(5). For the experiments Navigation Performance in zero-shot manner, LEO's training data is very simple. It will be interesting to see a comparison between finetuned LEO with SPARTUN3D and Sparttun3D-LLM.

questions:
(1). The input to the model is triple <C, S, Q>, where C is the 3D scene context, S is the situation, and Q is a question. The situation S can be further denoted as $S^t$, $S^p$, $S^r$, where $S^p$ is the 3D coordinate in the form <x, y, z> and $S^r$ is the quaternion. In the dataset, line 182, authors assume the agent's orientation is always facing forward to the center of the selected object. How is this selected object chosen? Will this introduce any bias?

(2). Can author show how they make inference of SPARTUN3D? More specifically, what is S, C, and Q respectively? Is it possible to provide an example?

(3). Why does finetuned 3D-LLM only has EM score? I understand 3D-Vista is a BERT but 3D-LLM seems not?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730463283260

thread_type:
official_review

<thread_object>
creator: reviewer_d9vh

<content>
summary:
This paper introduces SPARTUN3D, a dataset aimed at enhancing the spatial reasoning abilities of 3D-based large language models (LLMs) by providing situated context. The dataset includes tasks like situated captioning and QA, challenging models to respond based on the agent's dynamic perspective. Additionally, the authors propose a novel alignment module in Spartun3D-LLM, which improves alignment between 3D scene representations and textual descriptions. The paper demonstrates the model’s generalization on multiple 3D tasks, outperforming baselines in spatial understanding and navigation accuracy.

strengths:
1) Innovative Dataset: SPARTUN3D addresses a key limitation in 3D-based LLMs by providing an extensive dataset that incorporates situated reasoning. This dataset, generated by leveraging GPT-4o, is both scalable and diverse, which is essential for training models in 3D scene understanding from dynamic perspectives. This is a well-motivated and relevant contribution to the field.

2) Effective Alignment Module: The Situated Spatial Alignment Module enhances model performance by aligning 3D visual representations with their textual counterparts. This module appears to be a critical contribution as it enables more accurate spatial understanding and has implications for embodied tasks. The use of spatial self-attention and a 3D object-text alignment loss to improve object-text coherence is particularly novel.

3) Comprehensive Experiments: The experimental evaluation is thorough, covering both in-domain tasks (Spartun3D and SQA3D) and out-of-domain tasks (MP3D ObjNav), with significant improvements shown in zero-shot navigation accuracy. Ablation studies and visualizations further confirm the efficacy of the alignment module in improving spatial reasoning capabilities.

4) Practical Applications: Situated spatial understanding is valuable for real-world applications, such as navigation and human-robot interaction. This paper makes a strong case for the applicability of SPARTUN3D in these areas.

weaknesses:
1) The SPARTUN3D dataset is automatically generated with GPT-4o, which, while scalable, may lack the nuanced spatial and contextual fidelity found in human-annotated data. This synthetic nature raises questions about the dataset’s generalizability and its ability to represent real-world scenarios accurately.

2) The evaluation primarily focuses on situated QA, captioning, and navigation tasks, which, although useful, may not fully capture the complexity of embodied tasks (e.g., robotic manipulation or multi-step planning). Expanding the evaluation to include a wider variety of tasks would provide a more comprehensive assessment of the model's situated understanding abilities.

3) While the Situated Spatial Alignment Module is an interesting addition, the paper lacks a rigorous theoretical foundation for its design. Details on why specific techniques (e.g., spatial self-attention and MSE for alignment) were chosen and how they uniquely contribute to spatial alignment are not thoroughly explained, which could weaken the perception of this module’s novelty. Also, the added complexity of the alignment module may lead to increased computational demands, yet the paper does not discuss or benchmark these costs. For practical deployment in real-time applications or on resource-limited systems, it is essential to understand the trade-offs between the module’s benefits and its computational impact.

4) While comparisons to the LEO (2023.11 released) baseline and other similar models are present, the paper lacks a broader comparison with recent advances in spatial reasoning, navigation models or 3D tasks.

questions:
1) By using GPT4 API, how to ensure the quality and accuracy of datasets? Can you provide more information on quality control measures for SPARTUN3D? Specifically, how are errors from the automated pipeline identified and handled?
2) How does the Situated Spatial Alignment Module impact computational resources, particularly in training time and memory usage? Would the module be feasible for real-time applications?
3) Have you considered evaluating Spartun3D-LLM on other embodied tasks beyond QA and navigation, such as robotic manipulation, multi-step reasoning in dynamically changing environments or some other 3D tasks like visual grounding, 3D object detection or dense caption?
4) Your chosen baselines like Leo,3D-Vista and 3D-LLM, were relesed in 2023,  the paper lacks a broader comparison with recent advances, please show some comparisons of other advances of 3D-LLMs with SPARTUN3D-LLM?
5) As the alignment is one of your contributions of this paepr. Are there other alignment strategies that were considered or experimented with?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702964291

thread_type:
official_review

<thread_object>
creator: reviewer_jrte

<content>
summary:
The paper addresses the task of situated reasoning where an agent is located at a certain location in the 3D space and needs to reason about the situation (caption or answer questions) from it's spatial location. The paper generates a new dataset called Spartan3D, relying on GPT-4o that lets them scale the size of the data; and proposes an explicit spatial alignment module where the object tokens are aligned to a desciption of spatial situations/relations of the corresponding object obtained via a template. The method is tested on situation question answering benchmark of SQA3D and the newly proposed Spartan3D dataset. The results show that the explicit spatial alignment module helps in question answering as well as other captioning tasks; and the additional Spartan3D dataset helps performance on all datasets.

strengths:
- The paper is very well written and clearly lays down the problem and the proposed solution (with informative figures)
- The Spartan3D dataset will be useful to the community
- The proposed method obtains better results than prior state of the art on SQA3D and the proposed explicit alignment loss helps performance in all tasks (by about 1-2%)

weaknesses:
- A big claim of the paper is that SQA3D is human collected dataset; and the proposed pipeline to generate SPARTUN3D can be very useful to scale up the dataset size. However, it is unclear how effective this automatically generated dataset is in comparison to human generated dataset. Specifically, the current experiments of Table-3 show that on real-world SQA3D benchmarks, using additional Spartan3D dataset on top of human collected SQA3D dataset helps performance by about 1-2% despite much larger dataset size. Additionally, just training on Spartan3D dataset results in significantly worse performance on SQA3D (by about 20%). Moreover, it is unclear if the automated data shows strong scaling behaviors -- which, if true, would push the community to scale up the datasets in a similar way that this paper proposed instead of collecting more human annotations.

 - L460-465 "In the zeroshot setting, we re-trained LEO exclusively on a dataset constructed from 3RScan to ensure a fair
comparison with our method. As shown in Table 3, LEO performs poorly on SQA3D in the zeroshot setting, suggesting its limitations in learning situated understanding from its original dataset. In contrast, LEO trained on Spartun3D shows significant improvement, demonstrating the effectiveness of our dataset and the generalization ability of our model."

The above set of lines are confusing to me. 
- What exactly is this dataset constructed from 3RScan which is constructed for the zero-shot LEO baseline? Without this information, currently the conclusion seems to be that Spartan3D dataset is better than some other way of constructing a training dataset.
- I did not follow how obtaining good performance from LEO trained on Spartun3D leads to the "generalization ability of our model" conclusion? Table-3 does not show how the proposed model works with the "newly constructed 3RScan data" instead of the Spartan3D dataset. 

- For navigation experiments in Table-5. what is Spartan3D-LLM trained on? Is LEO and Spartan3D-LLM identical except the explicit spatial alignment module for this experiment? I am trying to understand the main reason why LEO does not work at all for navigation while Spartan3D-LLM show some non-zero performance.

questions:
- Some additional discussion / proof on why the additional automated data is useful. Scaling curves with varying amount of Spartan3D-LLM data used in training would help -- on real world benchmarks like SQA3D.

- Clarification on the newly constructed dataset from 3RScan for zero-shot LEO baselines, and generalization capabilities of the proposed model

- More details on the navigation experiments in Table-5; specifically regarding the training datasets used for the proposed model and the baselines.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730742006244

thread_type:
official_review

<thread_object>
creator: reviewer_d7e6

<content>
summary:
This paper introduces Spartun3D, a scalable dataset designed to enhance situated 3D understanding. The authors construct a situated scene graph to facilitate the generation of situated captions and QA pairs via LLM prompting. Additionally, they incorporate a situated spatial alignment module into a baseline 3D LLM to improve scene understanding. Experimental results demonstrate the effectiveness of their approach.

strengths:
1. The situated scene graph is well-crafted and contributes to a higher-quality dataset that supports future research in situated 3D scene understanding.
2. The paper is well-written and easy to follow.

weaknesses:
1. The paper could benefit from including more examples of generated data, particularly those with errors, to provide better insight into the dataset's quality. It would be valuable to discuss potential methods for correcting inaccurate captions or QA pairs generated by the LLM, as well as the impact of these errors on model performance.
2. There is a lack of a detailed ablation study on the proposed modules, specifically the Situated Textual Description and 3D Object-Text Alignment, which would help clarify their individual contributions.
3. The evaluation of object navigation relies on only four simple actions, which may weaken the findings, although it does showcase some zero-shot capabilities of the model.

questions:
Please refer to the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732420235565

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely appreciate the reviewers' constructive comments and feedback on our work. We are grateful that all reviewers acknowledged the motivation behind our research and its contributions to the community, as well as the well-organized presentation of our work.

At the same time, we have carefully addressed the valuable suggestions provided to improve our paper. Below, we summarize the key revisions made in our paper (labeled as blue front):

- **Human Evaluation (Reviewers d7E6, D9vh, and kEmt)**: We have included human evaluation results in Section 3.4, assessing the quality of Spartun3D in terms of language naturalness and spatial fidelity.
- **Scaling Effect (Reviewers jrTE and kEmt)**: We have added an analysis of the scaling effect in Section 5.3 to highlight the impact of dataset expansion on performance.
- **Clarification of SQA3D Performance (Reviewer jrTE)**: We have clarified the explanation of SQA3D performance in Section 5.2 to address the reviewer's concerns.
- **Error Analysis (Reviewer d7E6)**: Updates related to error analysis can be found in Appendix A.1, providing further insights into the error categories.

We hope that these revisions address the reviewers’ concerns and provide a stronger foundation for re-evaluating our submission. If there are any additional questions or comments, we are happy to address them.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732573814652

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers,

We sincerely appreciate the time and effort you’ve devoted to reviewing our work. We understand that your schedule may be quite busy, and we are truly grateful for your valuable feedback. As the Author-Reviewer discussion phase is extended, we would greatly value the opportunity to engage in further discussion with you during this period of time. Our aim is to gain insights into whether our responses effectively address your concerns and to ascertain if there are any additional questions or points you would like to discuss.

We look forward to the opportunity to discuss this further with you. Thank you for your thoughtful consideration.

Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733281325848

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Area Chair and Reviewers,

We would like to express our sincere gratitude for your efforts in facilitating the discussion regarding our paper.
We sincerely thank all reviewers (d7E6, jrTE, D9vh, kEmt) for recognizing the **innovative contribution of our dataset** in addressing current limitations in 3D-based LLMs for **situated understanding**. We are grateful to reviewers jrTE and D9vh for acknowledging **the effectiveness of our alignment strategy** and to reviewers D9vh and kEmt for highlighting **the strengths of our comprehensive experiments and practical applications.** Additionally, we appreciate all reviewers' feedback affirming that our paper is **well-written, clear, and sound**.


We also appreciate the reviewers' constructive suggestions to improve the quality of our work. Below, we summarize the key points addressed in our discussion:

**1) Quality Control and Error Analysis of Spartun3D:**
We appreciate the reviewers highlighting this common concern (d7E6, D9vh, and kEmt) to improve the quality of our paper.  Although we included some human evaluation in the original submission, we agree with the reviewer on the need for deeper evaluation and error analysis.
To address this, we revised Section 3.4 to include comprehensive human evaluations focusing on both language naturalness and spatial fidelity.  Additionally, we analyzed how different prompting strategies influence the dataset quality. We also provided the error analysis in Appendix A.1.
Following these updates, reviewers d7E6 and kEmt confirmed that their concerns had been addressed.
Although reviewer D9vh did not participate in the discussion period, we believe our new experiments have addressed the reviewer's main concern as well.

**2) Scaling Effects:** We sincerely appreciate the reviewers (jrTE and kEmt) for highlighting the need for scaling effect experiments, which we agree are essential to include. To address this concern, we have added a scaling effect experiment in Section 5.3. Both reviewers, jrTE and kEmt, have confirmed that we have addressed their concerns.


**3) Clarification of SQA3D Performance:** We sincerely thank Reviewer jrTE for this valuable feedback, which helped us identify and address the writing issues that caused confusion regarding our experimental results. We have revised the relevant part in Section 5.2 to clarify the effectiveness of the dataset and our designed alignment module. Reviewer jrTE has confirmed that these doubts have been resolved and expressed support for accepting our paper.

We have carefully refined our work and incorporated all suggested improvements in the revised submission, with updates clearly marked in blue. These changes significantly enhance the clarity and completeness of our paper. We thank the reviewers and area chair for their time and valuable suggestions in helping us improve this work.

Best,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
**Summary**

The paper aims to study the ability of 3D-LLMs to perform situated spatial reasoning, where given 3D scene and an agent's location and pose, the agent needs to either provide descriptions of surrounding objects (situated captioning) or answer questions (situated question-answering).  To do so, the paper introduces Spartun3D, a situated 3D dataset consisting of 133K examples generated using LLMs for evaluating captioning and QA.  The paper also proposes Spartun3D-LLM, which adds a situated spatial alignment module to a recent 3D-based LLM (LEO). Experiments show that the proposed module improves performance on the proposed Spartun3D benchmark as well as other 3D language tasks.

The main contributions is the Spartun3D dataset, the proposed situated spatial alignment module, and the experiments demonstrating the effectives of the proposed approach.

**Strengths**

Reviewers noted the following strengths of the work:
- The dataset is useful for the community [d7E6,jrTE,D9vh]
- Reviewers found the paper to be well-written and easy to follow [d7E6,jrTE,kEmt].  The AC also find the problem formulation, dataset, and proposed module to be clearly described.
- Experiments show proposed module is effective [jrTE,D9vh] 
- Tasks investigated by this work is underexplored [kEmt]

**Weaknesses**

Reviewers noted the following weaknesses:
- Concerns about the use of LLM to generate the data [jrTE,D9vh,kEmt]  including 
  - Quality of the data compared to human generated data
  - Whether scaling the data up actually can help train better models 
- Some aspects where not initially clear [jrTE,kEmt] 
- The evaluation can include more tasks and recent methods [D9vh]
- Missing discussion of computational costs [D9vh]
- Reviewer also requested additional details such as:
  - More examples of generated data [d7E6]
  - Detailed ablation study [d7E6]

The main common concern across reviewers was the quality of the generated data and request for additional information on how scaling the data affects the model performance.  This concern (as well as other concerns) where addressed by the authors during the author response period.

**Recommendation**

Overall reviewers are slightly positive on this work.  The AC believes the dataset can be useful, and the paper is clear and so recommends acceptance.