3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Referred Object Grounding

Authors: Tatiana Zemskova, Dmitry Yudin
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Referred Object Grounding, 3D Scene Graph, LLM, 3D Referred Object Grounding, 3D Scene Understanding

Abstract:
A 3D scene graph represents a compact scene model, storing information about the objects and the semantic relationships between them, making its use promising for robotic tasks. When interacting with a user, an embodied intelligent agent should be capable of responding to various queries about the scene formulated in natural language. Large Language Models (LLMs) are beneficial solutions for user-robot interaction due to their natural language understanding and reasoning abilities. Recent methods for creating learnable representations of 3D scenes have demonstrated the potential to improve the quality of LLMs responses by adapting to the 3D world. However, the existing methods do not explicitly utilize information about the semantic relationships between objects, limiting themselves to information about their coordinates. In this work, we propose a method 3DGraphLLM for constructing a learnable representation of a 3D scene graph.
The learnable representation is used as input for LLMs to perform 3D vision-language tasks. In our experiments on popular ScanRefer, RIORefer, Multi3DRefer, ScanQA, Sqa3D, and Scan2cap datasets, we demonstrate the advantage of this approach over baseline methods that do not use information about the semantic relationships between objects.

Decision: reject
Reviewer scores: 3, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730472815146

thread_type:
official_review

<thread_object>
creator: reviewer_cndl

<content>
summary:
This paper proposes 3DGraphLLM, a multimodal language model that takes into account the object relationships in a 3D scene and achieves better performance in referred object grounding tasks. 3DGraphLLM takes as input a scene point cloud, segmented into object instances using an off-the-shelf 3D object segmentation model. The features for each 3D object and their relationships are also extracted by pretrained models, projected to LLM token space, and input to the LLM as part of the prompt. The LLM and the projectors are finetuned on the training sets of 3D datasets. Experiments show that 3DGraphLLM outperforms baselines and several ablation studies show the effectiveness of design decisions.

strengths:
1. 3DGraphLLM converts object relationships to LLM token space and incorporates them into the prompt for 3D grounding tasks. In this way, the LLM receives more complete information about the 3D scene graph and gets a better understanding of the 3D environment. 
2. Most design decisions have ablation studies and their effectiveness is verified. 
3. The paper is well-written and easy to follow.

weaknesses:
1. LLMs are known for capturing a significant amount of common sense knowledge and show out-of-distribution generalization ability. However, in the design of 3D GraphLLM, I cannot see how such power of LLMs is being utilized or demonstrated at all, and thus **what's the point of using a pretrained LLM there?** Ideally, by using an LLM, I expect to see that in addition to taking the new modality (3D scene graph) as input, the adapted LLM also shows that its common sense knowledge is also being exploited in a non-trivial way, e.g. out-of-distribution generalization, achieving tasks other than grounding. At least I want to see if the entire model is trained from scratch (without pretraining on text), and whether the performance will be worse. 

With the above being said, I'm very concerned that in Table 2, the proposed method has worse performance than the expert models on ScanRefer, which seems to suggest that the LLM pretraining is not really helping. 

2. The proposed method relies heavily on the performance of the underlying 3D segmentation and scene graph generation models used to encode 3D objects and relationships into features. Even if the LLM is able to generalize OOD, it's still limited to these models and cannot recover from their failures. 

This concern is verified by Table 4, where using GT segmentation, 3DGraphLLM is better than the baseline by 5.6 percent, but less than 2.0 percent using predicted instance segmentation. 

3. Why is the flat graph representation object-centric (i.e. obj1, obj1-NN1, obj1-NN2, obj2, obj2-NN1, obj2-NN2...), instead of further flattened (i.e. obj1, obj2, obj3, ..., rel1, rel2, rel3, ...)? In the latter format, the set of relationships to keep can still be limited to the k-NN of each object instead of the dense graph. I'm asking because in Table 1 (L273), seems the $F_1^v$ is being repeated k times. The repetition will be even greater if one object is considered as NN of other object nodes. Therefore, there seems to be still a lot of waste in the proposed tokenization scheme. 

4. This paper proposes to train the model in a two-stage manner (first on GT instance segmentation, then on predictions), but this is not being ablated. 

5. The design minimum distance filter suggests that there are duplicate detections in the 3D detection results. I think these duplicates should be handled using filtering techniques like 3D NMS rather than the proposed naive minimum distance filter on the scene graph edges.

questions:
Please see the "Weakness" section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730490780214

thread_type:
official_review

<thread_object>
creator: reviewer_adra

<content>
summary:
The paper considers the problem of generating a learnable representation of a 3D scene to be processed by Large Language Models (LLMs). Compared to existing object-centric latent scene representations, the proposed approach incorporates an additional relational edge encoding. Both node (objects) and edge (relations) encodings are obtained by passing the off-the-shelf point cloud encoder outputs through a trainable projection head. Experiments show that this modification improves the performance on visual grounding tasks across multiple datasets.

strengths:
1. The idea to add object semantic relationship into the latent scene representation for LLMs is well-motivated.
2. The proposed approach is simple and effective.

weaknesses:
The proposed modification introduces a parameter of minimum distance for filtering the nearest neighbors. It requires manual tuning, as is set to be 1 cm as in the paper. This requirement might potentially make it hard to apply this approach to new domains, such as a tabletop scene or a larger multi-floor multi-room indoor scene.

questions:
1. What is the difference between the “3DGraphLLM-0” (in table 3, 4) and “Chat-3D v2” baseline (in table 2)? It is said in Section 4.4 that they are equivalent (“The 3DGraphLLM version with zero nearest neighbors…, equivalent to the Chat3Dv2 approach..”). However, under the ScanRefer benchmark, the 3DGraphLLM-0 (in table 4) outperforms Chat3Dv2 (in table 2) by more than 10% margin in $Acc\@0.25$. Similarly, why is the performance of 3DGraphLLM-0 under GT segmentation not consistent in table 3 and 4? 
2. The proposed approach uses Uni3D for object encoding and VL-SAT for edge encoding. Since VL-SAT also contains an object encoder, which might be even more compatible with its own edge encoder,  why is the Uni3D encoder used instead?
3. What are the common failure cases of the method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730554660079

thread_type:
official_review

<thread_object>
creator: reviewer_9kjd

<content>
summary:
The paper introduces a novel method called 3DGraphLLM, which combines semantic graphs and large language models (LLMs) for 3D referred object grounding in scenes. The method constructs a learnable representation of a 3D scene graph to be used as input for LLMs to perform 3D referred object grounding tasks. Experiments demonstrate its advantages over baseline methods that do not utilize semantic relationships between objects on popular datasets like ScanRefer, RIORefer, and Multi3DRefer.

strengths:
1. This paper notes that the semantic relationship between object pairs within a 3D scene is beneficial for visual grounding and other tasks such as visual captioning and visual question answering. It proposes 3DGraphLLM, which constructs a learnable representation of a 3D scene graph and uses it as input to a large language model (LLM) to perform the 3D referred object grounding task.

2. The authors have identified some technical issues (such as the domain gap between 3Rscan and ScanNet, and the presence of duplicate objects in the nearest neighbors), and I believe this paper is technically thorough.

3. The authors provide relatively sufficient and convincing experimental results.

weaknesses:
1. In the Edge Feature Encoder, the important issue is raised that the semantic relationships recognition methods are all trained on 3RScan scenes, while the visual grounding tasks are typically tested on ScanNet scenes. I believe this is indeed a challenge. The authors claim that the solution is to use VL-SAT due to its good generalization in the new domain. I am very familiar with VL-SAT, and currently, there is no evidence showing that this method has good generalization in the new domain to address the mentioned challenge.

2. Similarly regarding VL-SAT, the authors seem to have used the VL-SAT pre-trained on 3RScan to test on the ScanNet dataset (e.g., ScanRefer, Multi3DRefer, etc.). This is equivalent to the results of RIORefer->ScanRefer in Cross3DVG. In Figure 5, when the number of nearest neighbors = 2, the Acc@0.5 reaches an astonishing 49.0, which not only significantly surpasses the result of RIORefer->ScanRefer in Cross3DVG (13.34) but also shows a remarkable difference compared to the RIORefer->RIORefer test result (20.21) given in Cross3DVG. This could be explained by 1) using GT instance segmentation yields better results, and 2) 3DGraphLLM performs exceptionally well on RIORefer, surpassing the Expert model. However, 1) based on the results in Table 4, it is difficult to foresee such a significant gap, and 2) the performance of the model on similar datasets like ScanRefer and Multi3DRefer indicates that it is challenging to significantly exceed the existing SOTA Expert model. I hope the authors can provide some explanations regarding the above points and present more experimental results on RIORefer during the rebuttal period. A reasonable response to this weakness would make me consider raising my score.

3. Table 4 needs to be completed (i.e., providing the results of 3DGraphLLM-2 with OneFormer3D segmentation when the minimal distance = 1cm), as this is crucial for validating the effectiveness of the minimal distance strategy.

4. The results of VL-SAT are based on training with 160 objects and 27 relationships. I would like to know how much these 27 relationships help in ScanRefer. As far as I know, these 27 relationships do not seem to have a strong correlation with the descriptions in ScanRefer. I understand that this is essentially due to the limitations of dataset annotations, but I hope the authors can explore whether the annotations in the 3D Scene Graph Generation dataset are related to the descriptions in visual grounding.

questions:
1. Please address the questions in Weaknesses 1. I have concerns about the generalization between the two datasets.

2. Please respond to the questions in Weaknesses 2. I would like to see more experimental results on RIORefer and hope the authors can explain the significant differences in the results I mentioned in Weaknesses 2.

3. I hope the authors can answer the questions in Weaknesses 4, as I believe the correlation between the two datasets is very important.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696331026

thread_type:
official_review

<thread_object>
creator: reviewer_qz74

<content>
summary:
3DGraphLLMs proposes to improve referential grounding performance of LLM based methods by supplying an explicit spatial graph representation of the scene as an input to the LLM (in addition to the task and the object information). Specifically, the paper proposes to model relationships between objects using an off-the-shelf 3D scene graph model. Crucially, these relationships are only modelled among k nearest neighbors of the object to reduce the number of input tokens to the LLM. To prevent very neaby object proposals to be considered as neighbors, the paper use heurestics such as ignoring boxes that are too close to each other (1cm threshhold). The proposed model is tested on ScanRefer and Multi3DRefer grounding datasets and obtain better performance than other LLM based grounding models.

strengths:
The paper is well-written and easy to follow. It adds onto an interesting line of work in using LLMs for 3D referential grounding, The ablation study is well-designed and covers several of the relevant questions about design choices.

weaknesses:
The quantitative numbers of the papers are weak:
- The LLM used by this paper is the recent LLAMA3-8B-Instruct while other baselines use different and older LLMs. for eg. Chat3Dv2 and Grounded 3D LLM uses Vicuana-7B (based on LLAMA 2). This makes the comparison unfair with prior models. 
- The paper does not report results on other benchmarks like ScanQA, SQA3D and Scan2Cap while still training on it, and showing ablations on it. The paper mentions that the focus is on visual grounding tasks, and that is why the evaluations are only shown in these benchmarks; however the key idea of using scene graph representation might also help for these other spatial reasoning tasks and if not, should atleast be reported for completeness in my opinion. 
- The boost in performance by using the proposed scene graph technique is only around 1-2% (Table-4) in the setup when models uses masks from a detector like Mask3D (which represents the realistic setup). This boost comes at a cost of additional complexities of making the scene graphs and querying the LLMs with additional tokens (additional ~800 tokens per query). 
- The paper claims to achieve SOTA results on ScanRefer and Multi3DRefer benchmarks in the introduction. These claims do not look correct -- as per Table-2, 3D Vista and M3DRef-CLIP outperforms the proposed model on ScanRefer. Besides several stronger baselines like BUTD-DETR [1] and PQ3D [2] outperform 3DGraphLLMs by large margins. Perhaps, the intention was to say that 3D GraphLLMs is SOTA LLM-based visual grounding model -- the claim in introduction may likely need to be revised appropriately. 3DGraphLLMs outperformM3DRef-CLIP baseline on Multi3DRefer benchmark. However, the baseline is trained on much lesser data compared to 3DGraphLLMs.

questions:
The improvement from using 3DGraph representations seems somewhat marginal when compared in apples-to-apples setting with the predicted masks (Table-4). The results in Table-2 look good in the first glance, but due to the use of a stronger base LLM by the proposed method,  the results are hard to interpret directly. I would be happy to increase my rating if provided with more evidences / reasons of 3DGraphLLMs helping the grounding tasks (while not hurting the QA and captioning tasks by also reporting numbers on those).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732355603482

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Revision of 3DGraphLLM

comment:
We thank the reviewers for their detailed and thoughtful feedback on our work. We are pleased that the reviewers acknowledged the motivation behind the proposed idea of incorporating semantic edges of the scene graph to enhance the quality of LLM responses in the 3D referred object grounding task. We also appreciate the reviewers' comments on the quality of the paper's writing and the ablation experiments conducted.
The main concerns raised by the reviewers focused on two points:
 - The ability of 3DGraphLLM to leverage the pre-trained LLM and the scene graph for solving other 3D vision-language tasks (raised by r. qz74 and r. cnDL).
- The design choice for filtering duplicate objects in the graph based on the minimum distance between nearest neighbors (raised by r. aDrA and r. cnDL).

We have addressed these concerns by modifying the method for generating the sequence describing an object, adding a 2D object token next to the object identifier token. This modification, inspired by the Chat-Scene method [1], improved the stability of the 3DGraphLLM's convergence without altering the core contribution of this work: developing a method for creating a learnable representation of the 3D semantic scene graph and an algorithm for generating a flat sequence from the graph based on the k-nearest neighbors of the object.
Additionally, we conducted further experiments with the Non-Maximum Suppression filter and an ablation study on the proposed training pipeline using GT pre-training.

We have revised the text of our paper based on the reviewers' comments. 

Below, we provide the tables referenced in our responses to the reviewers. C denotes the CIDEr metric.

**Table 1**. Ablation study on semantic edges role and training pipeline.
| Methods | Pre-train | Number of edges | ScanRefer A@0.5 | Multi3DRefer F1@0.5 | ScanQA C | ScanQA B-4 | Sqa3D EM | Scan2Cap C@0.5 | Scan2Cap B-4@0.5 |
|-------------|------------|--------------|----------------------|--------------------------|---------------|----------------|---------------|---------------------|-----------------------|
| 3DGraphLLM-0 Vicuna1.5 | ✗ | 0 | 50.2 | 52.4 | 87.7 | 14.3 | 54.6 | 77.1 | 36.3 |
| 3DGraphLLM-2 Vicuna1.5 | ✗ | 2 | 50.1 | 52.7 | 92.2 | 15.5 | 54.7 | 80.4 | 36.9 |
| 3DGraphLLM-2 Vicuna1.5 | ✓ | 2 | 51.3 | 55.4 | 87.6 | 12.1 | 53.1 | 81.2 | 36.3 |
| 3DGraphLLM-0 LLAMA3-8B | ✗ | 0 | 52.0 | 55.1 | 84.0 | 15.8 | 53.8 | 80.0 | 37.5 |
| 3DGraphLLM-2 LLAMA3-8B | ✗ | 2 | 54.3 | 57.3 | 87.4 | 14.9 | 54.5 | 85.6 | 39.6 |
| 3DGraphLLM-2 LLAMA3-8B | ✓ | 2 | 54.6 | 58.2 | 83.1 | 12.5 | 55.2 | 82.9 | 37.8 |


**Table 2**.  Performance comparison of 3DGraphLLM with state-of-the-art approaches for 3D vision-language tasks. The ellipsis in the table indicates omitted baselines that were given in the original version of the paper.
|  |   | ScanRefer | Multi3DRefer |  |  | ScanQA | | Sqa3D | Scan2Cap |  |
| ---------- | --------- | --------- | ----------- | -------- | -------- | --------- | -------- | --------- | --------- | ---------- |
|    | Methods | A@0.25 | A@0.5 | F1@0.25 | F1@0.5 | C | B4 | EM | C@0.5 | B-4@0.5 |
| Expert models | ... | - | - | - | - | - | - | - | - | - |
|  | M3DRef-CLIP | 51.9 | 44.7 | 42.8 | 38.4 | - | - | - | - | - |
|  | Scan2Cap | - | - | - | - | - | - | - | 35.2 | 22.4 |
|  | ScanQA | -  | - | - | - | 64.9 | 10.1| - | -| - |
|  | Sqa3D | - | - | - | - | - | - | 47.2 | - | - |
|  | 3D-VisTA | 50.6 | 45.8 | - | - | 72.9 | 13.1 | 48.5 | 66.9 | 34.0 |
|  | BUTD-DETR | 52.2 | 39.8 | - | - | - | - | - | - | - |
|  | PQ3D | - | 51.2 | - | 50.1 | 87.8 | - | 47.1 | 80.3 | 36.0 |
| LLM-based models | ... | - | - | - | - | - | - | - | - | - |
|  | Chat-3D v2 | 35.9 | 30.4 | - | - | 77.1 | 7.3 | - | - | - |
|  | Scene-LLM | - | - | - | - | 80.0 | 12.0 | 54.2 | - | - |
|  | LL3DA | - | - | - | - | 76.8 | 13.5 | - | 65.2 | 36.8 |
|  | Grounded 3D-LLM | 47.9 | 44.1 | 45.2 | 40.6 | 72.7 | 13.4 | - | 70.6 | 35.5 |
|  | Chat-Scene | 55.5 | 50.2 | 57.1 | 52.4 | 87.7 | 14.3 | 54.6 | 77.1 | 36.3 |
|  | 3DGraphLLM Vicuna-1.5 | 57.0 | 51.3 | 60.1 | 55.4 | 87.6 | 12.1 | 53.1 | 81.2 | 36.3 |
|  | 3DGraphLLM LLAMA3-8B | 60.2 | 54.6 | 63.0 | 58.2 | 83.1 | 12.5 | 55.2 | 82.9 | 37.8 |

**Table 3**. Ablation study on semantic edges role depending on quality of instance segmentation.
| Methods | Instance segmentation | Number of edges | Minimal distance (cm) | ScanRefer A@0.25 | ScanRefer A@0.5 |
| ---------- | ------------------------- | ------------------- | ------------------------- | ----------------------- | ---------------------- |
| 3DGraphLLM-0 | Mask3D | 0 | - | 46.0 | 34.2 |
| 3DGraphLLM-2 | Mask3D | 2 | 0 | 47.3 (+ 1.3%) | 35.6 (+ 1.4%) |
| 3DGraphLLM-2 | Mask3D | 2 | 1 | 48.0 (+ 2.0%) | 36.2 (+ 2.0%) |
| 3DGraphLLM-2 | Mask3D (+ NMS) | 2 | 1 | 48.1 (+ 2.1%) | 36.5 (+ 2.3%) |

[1] Huang, H., et al. (2024). Chat-scene: Bridging 3d scene and large language models with object identifiers. In The 38 Annual Conference on Neural Information Processing Systems.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732997280405

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank the reviewers for their feedback and for raising their scores. We are pleased that our responses addressed their initial concerns. We would like to emphasize that most of the changes made based on the reviewers' feedback have already been incorporated into the revised version of the manuscript. The remaining results obtained during the discussion will be included in the final version of the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes 3DGraphLLM to combine semantic graphs and LLMs for 3D referred object grounding in scenes. Initially, the reviewers raised concerns about the experimental comparisons and the marginal performance gains. The authors provided a strong rebuttal addressing many of these issues. However, several reviewers noted that the experiments in the revised submission differ significantly from the original, and the paper's writing requires substantial improvement.

While the paper demonstrates notable merit, it is not recommended for acceptance in its current form. The authors are encouraged to incorporate the reviewers' feedback when revising the paper for submission to other venues.