CityAnchor: City-scale 3D Visual Grounding with Multi-modality LLMs

Authors: Jinpeng Li, Haiping Wang, Jiabin chen, Yuan Liu, Zhiyang Dou, Yuexin Ma, Sibei Yang, Yuan Li, Wenping Wang, Zhen Dong, Bisheng Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D Visual Grounding, Large language model, multi-modality language model

Abstract:
In this paper, we present a 3D visual grounding method called CityAnchor for localizing an urban object in a city-scale point cloud. Recent developments in multiview reconstruction enable us to reconstruct city-scale point clouds but how to conduct visual grounding on such a large-scale urban point cloud remains an open problem. Previous 3D visual grounding system mainly concentrates on localizing an object in an image or a small-scale point cloud, which is not accurate and efficient enough to scale up to a city-scale point cloud. We address this problem with a multi-modality LLM which consists of two stages, a coarse localization and a fine-grained matching. Given the text descriptions, the coarse localization stage locates possible regions on a projected 2D map of the point cloud while the fine-grained matching stage accurately determines the most matched object in these possible regions. We conduct experiments on the CityRefer dataset and a new synthetic dataset annotated by us, both of which demonstrate our method can produce accurate 3D visual grounding on a city-scale 3D point cloud.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730305763605

thread_type:
official_review

<thread_object>
creator: reviewer_p5jv

<content>
summary:
The paper presents CityAnchor, a novel 3D visual grounding method designed for city-scale scenes, utilizing a two-stage multi-modality large language model (LLM). The first stage focuses on coarse localization of potential object regions based on text descriptions, while the second stage involves fine-grained matching to determine the best object matches. Specifically, 
- Coarse Localization Module (CLM)  takes a city-scale colored point cloud, projects it onto a 2D map, and uses the text description to regress possible regions of the target on this 2D map.
- Fine-grained Matching Module (FMM) performs fine-grained comparisons between each candidate object identified by the CLM and the text description by predictiong the similarity between the text and each candidate object, and select the object with the highest similarity score.
Additionally, the paper introduces the CityAnchor dataset, a synthesized benchmark for 3D visual grounding. The method demonstrates significant improvements in accuracy and efficiency over existing techniques on the CityRefer and CityAnchor datasets.

strengths:
+ The paper introduces CityAnchor, a novel approach to 3D visual grounding specifically tailored for city-scale scenes. Its use of a two-stage multi-modality large language model (LLM) for both coarse localization and fine-grained matching is effective and advance in the field. The proposed approach effectively integrates spatial context and object features, showcases the ability in addressing the complexities of urban environments, setting it apart from existing methods that may not adequately handle such challenges.
+ The paper provides a comprehensive evaluation of the model's performance, demonstrating significant improvements in accuracy compared to existing methods. Additionally, the introduction of the CityAnchor dataset as a synthesized benchmark for 3D visual grounding further enhances the quality of the research, providing a valuable resource for future studies.
+ The paper is well-structured and easy to follow, clearly stating the objectives, methodology, and results of the research.

weaknesses:
- Limitations of 2D Mapping: Projecting 3D space onto a 2D map often results in a loss of spatial information, particularly in cases of occlusions or overlapping instances. This pipeline, especially the CLM component, operates under the significant assumption that all objects are nearly flat, which inherently limits its applicability to more complex environments, such as indoor spaces or densely populated urban areas like NYC.

- Dependence on Point Segmentation Model: The pipeline requires users to first segment the scene point cloud into objects using a pretrained 3D segmentation model. As a result, it heavily depends on the granularity and accuracy of this pretrained model, setting an upper performance limit for the proposed approach.

- Justification for the Proposed Dataset: While introducing a new dataset is valuable, the necessity of the CityAnchor dataset for this task or the proposed method requires further clarification. It is essential to address whether the method’s performance would significantly degrade without this dataset, whether it enhances diversity or robustness, and if it offers any unique attributes that the CityRefer dataset lacks. Providing these details would strengthen the contribution of the dataset to the paper.

- Inference time and hyperparameter tuning: These two limitations were briefly discussed in the paper.

questions:
- Effectiveness of CLM: In Figure 4, it appears that CLM is not highly effective, as it predominantly highlights houses and roads. Could the authors provide additional results or offer an explanation for this behavior?

- Prompt Engineering Requirements: It seems that users must invest effort in engineering their prompts to achieve accurate results, which may pose usability challenges. The authors could consider conducting a user study to gather prompts from a diverse set of users to assess the method’s robustness when different descriptions target the same object.

- Progressive Prompting Approach: It would be intuitive to structure prompts in a coarse-to-fine manner—beginning with a broad query that identifies multiple potential candidates and gradually introducing constraints to precisely locate the target.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720926214

thread_type:
official_review

<thread_object>
creator: reviewer_ynjm

<content>
summary:
The paper addresses the challenge of object localization in city-scale 3D point clouds with a method called CityAnchor, which leverages a multi-modal large language model. Using a two-stage approach of coarse localization and fine-grained matching, CityAnchor efficiently and accurately identifies target objects in large urban scenes. Experimental results show that CityAnchor significantly outperforms existing methods in accuracy and speed, with promising applications in urban planning and geospatial analysis.

strengths:
S1: The method’s coarse-to-fine design is well-structured for efficient large-scale 3D visual grounding.

S2: Experimental results show substantial and objective improvements over existing methods.

S3: The proposed model demonstrates clear practical potential for applications in urban planning and geospatial analysis.

weaknesses:
W1: The method seems heavily reliant on large language models, and the need for manual adjustment of the Region of Interest (RoI) threshold in the coarse localization stage suggests a lack of generalizability.

W2: While the coarse-to-fine localization design is effective, it builds on existing multi-stage approaches without introducing fundamentally new concepts in 3D visual grounding, which suggests that the method's strengths lack a degree of innovation.

Other:
The layout could be improved; for instance, all teaser images should be embedded as PDFs rather than PNGs to ensure the text within images is selectable.

questions:
It’s unclear how the method addresses dynamic objects in urban environments, such as moving vehicles or pedestrians, which could affect the reliability of grounding results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720953370

thread_type:
official_review

<thread_object>
creator: reviewer_8f4q

<content>
summary:
In the paper, the authors introduced CityAnchor, a 3D visual grounding method tailored for city-scale scenes. The CityAnchor is based on a two-stage multi-modal LLM. In the coarse stage, CityAnchor predicts candidate objects that match the query text descriptions on projected 2D maps derived from urban point clouds, allowing us to efficiently filter out redundant regions and concentrate on likely objects. Next, it performs fine-grained matching between text descriptions and the candidate objects to establish the final grounding results by encoding both text descriptions and object features into an LLM. To do that, the authors created a new dataset and evaluated the proposed method on both CityAnchor and CityRefer datasets to demonstrate the effectiveness of the proposed method on city-scale point clouds.

strengths:
First of all, it is interesting that a highly intuitive idea is proposed to address two major issues in existing methods concerning how to improve the multi-modality feature extraction in the large-scale visual grounding and how to efficiently localize the object in a large-scale point cloud.

Although it lacks technical novelty, it effectively implements the necessary modules to address two main problems. More specifically, a coarse location firstly finds possible regions on a projected 2D map of the point cloud while the fine-grained matching accurately determines the most similar region with the given test description.

In experimental section, the proposed method achieved SoTA results in city-level localization tasks, demonstrating its effectiveness. The ablation study is highly analytical for each level module and feature representation.

weaknesses:
One concern is the validity of the coarse localization module. As shown Fig.4, I am curious whether CLM operates as intended. It appears to activate too many candidates, as if it were merely distinguishing between urban and non-urban areas. This raises doubts about whether it significantly aids in the efficient operating in the overall framework.

Experiment analysis and description are not specific and descriptive. For example, a detailed explanation based on specific examples is needed to clarify the novel objects and novel descriptions, whether the reason they are not in the training data is due to an out-of-distribution (OOD) scenario.

questions:
I mentioned all comments including reasons and suggestions in the above sections. I recommend that the author will provide all the concerns, and improve the completeness of the paper. If the rebuttal period resolves the above-mentioned concerns, I will gladly raise my score. Also, there are little vague sentences and grammatical errors in the paper. I recommend that the author will revise the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730897798573

thread_type:
official_review

<thread_object>
creator: reviewer_6qax

<content>
summary:
This paper proposes a multi-modal LLM for city-scale 3D visual grounding named CityAnchor. CityAnchor adopts a coarse-to-fine searching strategy. First, a 3D segmentation model generates the potential objects from point clouds. Next, in the coarse localization stage, a LLaVA generates the <RoI> token and a SAM generates the RoI heapmap indicating the candidates of the target object. At last, in the fine-grained matching stage, another LLaVA select the best candidate as the target object. Besides, this work also presents a new dataset for 3D visual grounding. Experiments show that CityAnchor outperforms the previous methods by a large margin.

strengths:
1. The paper is well-written and easy to follow.
2. The experiment results are impressive.
3. The proposed method is well motivated and technically sound.

weaknesses:
1. As also mentioned in Sec.5, the efficiency of the proposed method is a major concern. It involves two LLMs and multiple forward passes in the fine-grained matching stage. Are there any possible solutions to improve the efficiency?
2. How to choose the positive and the negative samples during the training of FMM?
3. The objects are generated by a pretrained 3D segmentation model which can only recognize a closed set of targets. So I expect to see the generality of CityAnchor to unknown objects.
4. Some important details are missing:
- Line 157, what is $T_c$?
- Line 138 & 209, how to determine whether an object has a landmark name?
- Line 206, what is $c$ in $E^s_x$?
- Line 232, which encoder is the concatenated feature vector fed into?

questions:
Please address the questions in the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732165141444

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Response from Authors

comment:
We sincerely appreciate all reviewers for their detailed feedback and constructive suggestions. We are glad that the reviewers recognize the effectiveness of CityAnchor in addressing the 3D city-scale visual grounding problem with positive comments like the well-structured coarse-to-fine design (6qAx, 8f4q, YNjM, P5JV), the impressive experimental results demonstrating significant performance improvements (6qAx, 8f4q, YNjM, P5JV) compared to the existing baseline methods, and the introduction of the CityAnchor dataset can serve as a valuable resource for future research (P5JV).

After carefully improving the quality of our submission, we present here a revised main paper and supplementary materials, and the pdf has been updated. The modifications in the main paper have been highlighted in red, the added experiments and necessary analysis supplemented in the appendix have been highlighted in blue.

Changes in the main paper include:

$\bullet$ Detailed meaning of $c$ and encoding process for FMM in Sec.3.3.1.

$\bullet$ Improved writing: StreetSgin $\rightarrow$ StreetSign in Sec.4.1.1.

$\bullet$ Specific examples added for "Novel Objects" and "Novel Descriptions" settings in Sec.4.1.3.

$\bullet$ Clarification of fixed RoI threshold (0.3) and positive and negative sample ratio (1:3) in Sec.4.1.4.

$\bullet$ More in-depth discussion for limitations and future work in Sec.5.

Changes in the revised supplementary material include:

$\bullet$ Additional Results (Figure A.2) and discussion on CityRefer dataset with RoI map in Sec.A.1.

$\bullet$ Results (Figure A.7) and analysis on diverse text prompts in Sec.A.4.

$\bullet$ Generality experiment (Figure A.8 and Figure A.9) and discussion for unknown objects in Sec.A.5.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces an approach that localizes an object in a city-scale point cloud. The key of the proposed idea has two stages: one that roughly locates possible positions in the 2D map of the point cloud using LLM, and the next step accurately localizes these potential candidates. The authors conduct experiments with the CityRefer dataset and the newly annotated synthetic dataset, and the authors demonstrate superior accuracy on these two datasets. Given the consensus from reviewers for the paper's acceptance, AC confirms that the proposed approach introduced interesting ideas in this field and proposed a challenging dataset for the task.