CityAnchor: City-scale 3D Visual Grounding with Multi-modality LLMs
Authors: Jinpeng Li
Haiping Wang
Jiabin chen
Yuan Liu
Zhiyang Dou
Yuexin Ma
Sibei Yang
Yuan Li
Wenping Wang
Zhen Dong
Bisheng Yang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Visual Grounding
Large language model
multi-modality language model
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present a city-scale 3D visual grounding system to accurately find targets in point clouds from text descriptions.

Abstract: In this paper, we present a 3D visual grounding method called CityAnchor for localizing an urban object in a city-scale point cloud. Recent developments in multiview reconstruction enable us to reconstruct city-scale point clouds but how to conduct visual grounding on such a large-scale urban point cloud remains an open problem. Previous 3D visual grounding system mainly concentrates on localizing an object in an image or a small-scale point cloud, which is not accurate and efficient enough to scale up to a city-scale point cloud. We address this problem with a multi-modality LLM which consists of two stages, a coarse localization and a fine-grained matching. Given the text descriptions, the coarse localization stage locates possible regions on a projected 2D map of the point cloud while the fine-grained matching stage accurately determines the most matched object in these possible regions. We conduct experiments on the CityRefer dataset and a new synthetic dataset annotated by us, both of which demonstrate our method can produce accurate 3D visual grounding on a city-scale 3D point cloud.

Published as a conference paper at ICLR 2025
CITYANCHOR: CITY-SCALE 3D VISUAL GROUNDING WITH MULTI-MODALITT LLMs
Jinpeng Li Haiping Wang Jiabin Chen' Yuan Liu Zhiyang Dou' Yuexin Ma Sibei Yang' Yuan Li Wenping Wang Zhen Dong" Bisheng Yang!
Hong Kong University of Science and Technology 3 University of Pennsylvania "ShanghaiTech University 'Sun Yat-Sen University
"LISMARS, Wuhan University
Texas A&M University
CityAnchor
buiking that muttifeeed section, shusted at intersection point too road named Watsal road and Regino Drive. The building has lange parking aree.
One yellow car between gray car and red enc, that has blnck car in front of Rt the parking lct betwwen the BMTI building.
This query match 0.97.
This the cabys, the probabiliy that object
probabilly
Walsall larpe factory has a white and blue roof.
The big cval green ground that sumounded the mual brown nearby BornGreynound Stadium.
brown
This 5 cobj- you are
cobi, probabiltty object query match
querying, the probobility that object and match each other is
We present Cichon a multi multi-modality LLM, that can accurately localize a target in a city-scale point cloud from some text descriptions of the target. City Anchor achieves this by extracting features from the point cloud to grasp the intricate attributes and spatial relationships of urban objects. Then, CityAnchor comprehends the text descriptions and searches in the urban-scale point cloud for the objects corresponding the input text descriptions.
Figure 1:
ABSTRACT
In this paper, present 3D visual grounding method called Cityonchor for localizing an urban object in a city-scale point cloud. Recent developments in multiview reconstruction enable to reconstruct city-scale point clouds but how conduct visual grounding on such large-scale urban point cloud remains an open problem. Previous 3D visual grounding system mainly concentrates on localizing an object in an image or a small-scale point cloud, which is not accurate and efficient enough to scale up to a city-scale point cloud. We address this problem with multi-modality LLM which consists of two stages, coarse localization and fine-grained matching. Given the text descriptions, the coarse localization stage locates possible regions on projected 2D map of the point cloud while the fine-grained matching stage accurately determines the most matched object these possible regions. We conduct experiments the CityRefer dataset and a new synthetic dataset annotated by us, both of which demonstrate our method can produce accurate 3D visual grounding city-scale 3D) point cloud. The source code is available at wwwpgitthub.oooeeeeoooooeeeeeeee www.ooom/---SSSSSCCCAAAAttn
INTRODUCTION
3D visual grounding is a critical task in computer vision with transformative applications in robotics, AR/VR (Anderson et al., 2018 Lee et al. (02T), and autonomous driving (Najibi et al., 2023). Taking this to the next level by scaling 3D visualization to city-scale point clouds opens up thrilling new possibilities. Imagining the power to analyze entire cities detail, we could revolutionize
'The first two authors contribute equally.
1 "oorrsponning Authors.

---

Published as a conference paper at ICLR 2025
urban planning and infrastructure development. This leap in scale can propel us into of city-wide analysis, enhancing decision making processes and fostering innovative solutions for urban challenges. Accurate city-scale 3D visual grounding is set to inspire the next generation of map products and drive advancements in GeoScience, offering an unprecedented view of urban environments.
Scaling 3D visual grounding to a city scale is a challenging task. Existing work CityRefer (Miyanishi al. 2023) has tried to tackle this task by creating large-scale point cloud visual grounding dataset and training a neural network to embed both text descriptions and point clouds within the same feature space. Then, in inference time, given the text description and 10 possible candidate objects, CityRefer exhaustively compares the provided candidate objects to find the best match, which demonstrates impressive performance in this city-scale visual grounding task. However, training a multi-modality network from scratch to extract features for both text descriptions and point clouds shows the limited capacity for complex input texts. Meanwhile, CityRefer requires 10 candidates as inputs and exhaustively matching all the objects in the city is too time-consuming. Thus, how to improve the multi-modality feature extraction in the large-scale visual grounding and how to efficiently localize the object in large-scale point cloud remain two open problems.
A promising direction to improve visual grounding is to design a multi multi-modality Large Language Model (LLM) (Touvron et al. 2023 Lai ct al. 2023 Hong al. 2023b) to process both the text prompts and 3D point clouds. LLMs show a strong ability to understand the text descriptions and by aligning the features of point clouds with the feature space of LLMs, we are able to conduct visual grounding with the help of LLMs. There are already some pioneer works (Yang et al.) [023bb Liu et al. 2024a Yang et al. 2023a a Zhu al.) 2023 Jia al. 2024) following this path to incorporate LLMS with point clouds for the small-scale 3D visual grounding task. Most of these works concentrate on visual grounding in small-scale indoor point clouds containing less than 10 objects so they can exhaustively match each object with the text description by their multi molliityy LLMss. However, such an exhaustive matching too costly for city-scale visual grounding with hundreds or thousands of objects.
In this paper, we propose CityAnchor for the city-scale 3D visual grounding task, as shown in Fig. [ CityAnchor addresses this challenging large-scale visual grounding task with two designs. First, we finetune construct a multi-modality LLM based on a pretrained LLM (Liu et al. 2024b) to simultaneously process the 2D maps, 3D point clouds, and text descriptions. By aligning the features of 2D maps and 3D point clouds with the cell-established language feature space, City is able to accurately find the target objects and generalize well to unseen complex text descriptions. Second, we design a coarse-to-fine searching strategy for efficient visual grounding among hundreds or thousands of objects in city-scale scene. The coarse searching strategy of CityAnchor efficiently identifies a set of possible regions for the target objects by predicting a heatmap on the projected 2D maps. Then, the objects in these possible regions are extracted and compared with text descriptions to predict similarity scores. This coarse-to-fine fine searching strategy greatly reduces the number of finegrriied comparisons by LLMs and enables our method scale up to urban point cloud.
To evaluate the performance of CityAnchor, we conduct experiments on the CityRefer dataset and a new synthetic anlftannot dataset. We directly evaluate the visual grounding ability of City Anchor in the city scale rather than selecting from 10 candidate objects. Results demonstrate that City achieves state-of-the-an 3D visual grounding performance on the public CityRefer (Miyanishi et al., 2023) dataset and self-annotated dataset, surpassing the baselines by 30% 43% in grounding accuracy (Acc) on average. Moreover, CityAnchor requires only ~ 32 seconds to search for an object in city-scale point cloud with more than 300 objects, which is 1.3x faster than the baseline CityRefer. The improved performances and efficiency in such a city-scale visual grounding can greatly benefit downstream geoscience applications to analyze large-scale scenes.
2 RELATED WORK
Grounding in indoor scenes. ScanRefer (Chen et al. 2020) and Referit3D (Achlioptas al. 2020
first propose indoor datasets for 3D visual grounding that contain free-form object-annotation pairs on ScanNet (Dai et al.) 2017) dataset. On this basis, numerous point cloud-based visual grounding efforts have subsequently sprung up. Early works (Huang et al., [00TT] Gu et al. (023) resort to manually designed scene graph construction methods delineate spatial relationships among object
2

---

Published as a conference paper at ICLR 2025
proposals for locating. Recent methods (He et al. 2021 Roh et al.
2022
Zhao et al. 2021 Huang
jet al. 2022] Yang et al., 2024 2021: CChnn et al. 2023b Guo et al.
have pivoted towards the for visual encoding, text-visual
2023
development of large transtormer networks Vaswani et al. 2017)
feature alignment, and visual grounding. With the ascension of Large Language Models (LLMss,
there has been burgeoning interest (Chen al. 2023a Yang al. 2023a Hong al. 2023h Fu
et al. 2024 Chen et al.] [2224] Zhang et al.) 2024 Li et al. [024,] Tang et al. (024)) in exploring the
capabilities of LLMS to interpret the complex 3D indoor scenes for 3D visual tasks such as grounding
and planning. However, the above methods designed for indoor scenes can not naively scale up to
city-scale grounding task for their limited grounding capability or memory constraints.
Grounding in city-scale scenes. Research towards 3D visual grounding for open outdoor scene remains nascent. Earlier works including TouchDown (Chen et al.) 2019) and KITTI360Pose (Kol- met et al. 2022) aim at grounding in point clouds collected by constrained roadside environments. CityRefer (Miyanishi et al., 2023) annotated the large-scale SensatUrban (Hu et al. Ib) dataset to train simple baseline for city-scale 3D visual grounding. In City CityAnchor, we leverage the strong knowledge prior from pre-trained LLMs to achieve much better performances.
system, which is
Multi-Modality Large Language Models. Large Language Models (LLMs) like GPT-4 (Achiam
ct al. 2023) and LLaMA (Touvron et al., 2023), extensively trained on vast textual datasets through self self-supervised learning paradigms, exhibit versatile capabilities in addressing diverse language- related tasks while demonstrating robust generalization capacities. Inspired by the exceptional reasoning capabilities of LLMs, researchers are actively investigating methodologies to extend these competencies into understanding and generating other modalities (Wang et al. [2024] [ai et al. [023 Hong et al.) 2023b, Yang et al., 2023a Chen el al.) 2023a; Ma et al.) [2024] Yin el al.) 2024 Wu
al. 2023a), including visual and dimensiiaan spatial contexts, called multi-modality LLMs.
Aichor is built upon the well-known MM-LLM LISA (Lai et al., 2023) and 2D vision LLM
VA (Liu et al. 2024b).
3 METHOD
3.1 OVERVIEW
City Anchoo aims to locate a 3D target within a city-scale colored point cloud S based on given text description t. CityAnchor is a multi-modal large language model (LLM) that operates in two stages: a coarse localization stage and a fine- ame-grained matching stage. In the data preprocessing stage, we first segment S into objects using pretrained 3D segmentation model (Vu et al. 2022) and assign the street or building names, so-called landmarks, obtained from OpenStreetMap to each object following CityRefer (Miyanishi et al., 2023). Then, in the first stage, City Anchor employs Coarse Localization Module (CLM) to regress the possible regions of the target on a 2D map projected from the city point cloud, effectively filtering out irrelevant objects. In the second stage, Aichhr uses a Fine-grained Matching Module (FMM) to perform fine-grained comparisons between each candidate object in the above regions and the text description, ultimately selecting the most similar object as the grounding result.
3.2 COARSE LOCALIZATION STAGE
The coarse localization stage of Aichonchor is shown in Fig. 2 Given city-scale colored point cloud S, we first project the point cloud onto the XoY plane to obtain a 2D map (Liu et al.) 2024a). denoted as I. Subsequently, both / and / are fed into so-called Coarse Localization Module (CLM), which then predicts a heatmap indicating the possible locations of the target object.
The architecture of CLM follows LISA (Lai et al. 2023), including 2D vision LLM initialized from LLaVA (Liu et al.) 2024b), and an image segmentation model initialized from SAM (Kirillvv et al. 2023). We extend the original vocabulary of LLaVA with token, called < Rol >, the abbreviation for "Region of Interest".
Specifically, we first feed the projected 2D map and the text description into CLM and fine-tune the CLM to produce output texts T, that include the < Rol > token. Next, we extract the last-layer feature in the LLaVA of the CLM model corresponding to the < Rol token. This feature, along
3

---

Published as a conference paper at ICLR 2025
object segmentation
Coarse Localization Module (CLM)
Top-View
Projection
Cige-leeel
2D Map (RGB)
LLaVA + LORA
Image
Heurmap H
Point Cloud
<Rll>
Segmentor
B T.
(SAM)
Fiberieg
Description
Located heart of the vibrunt city. affacent to verdast park, this hote! boans spacions swimming pool and offers nccoremoiaaioon buildings with blue ficude.
Frozen Parameters
Trainable Parameters
Candidate Objects X
Figure 2: Coarse localization stage of Cichor where City Anchor accepts a text description and 2D map projected from the city-scale point cloud and predicts a heatmap of the correlation score with the input text descriptions. The heatmap helps us to filter out set of candidate objects.
with I, is input into the image segmentation model to generate a heatmap that contains the correlation score of every pixel. Then, for each target in S, we compute its correlation score using the score of its center point in H. If this value greater than the threshold 0, we consider it candidate object. All candidate objects together form set of candidates X.
Discussion. This CLM takes advantage of the strong feature extraction ability of LLaVA to under- stand both the input text descriptions and the image, which improves the generalization ability to unseen complex input texts than vanilla feature extraction network. The regressed heatmap filters out irrelevant objects and enables concentrating on just relevant objects.
3.3 FINE-GRAINED MATCHING STAGE
Given a candidate object x E X and the text description f, we train multi-modality LLM as our Fine-grained Matching Module (FMM) to predict the similarity between them. Then, the object with the largest similarity is selected as the final grounding result for the text description. Our FMM also adapted and fine-tuned from LLaVA (Liu al. 2024b). The original LLaVA is designed to handle text and 2D image inputs rather than the 3D visual grounding of point clouds. We thus modify LLaVA enable it process 2D maps, 3D point clouds, and text descriptions inputs and design a novel output module on the output of LLaVA to predict the similarity score between the text descriptions and the 3D object point cloud.
3.3.1 OBJECT ENCODING
To enable FMM accept the candidate object x and the text description for comparison, need to encode the feature of x and align it with the well-established feature space of LLaVA. Specifically, we tokenize and embed the input text description t set of feature vectors Exst E R as the original LLaVA, where is pre-defined sentence length and d = 1024 is the feature dimension.
For one candidate object æ E X represented by a point cloud, we extract three kinds of features for this object as the input to the LLaVA.
Attribute feature. Attribute features are the encoded features about the attributes of the object like colors and shapes. Since LLaVA enables CLIP features input, we also want our attribute features to aligned with CLIP feature space. achieve this, we adopted two methods to encode the attribute features of input object æ. First, we adopt Uni3D (Liu et al. (022a)) to directly extract CLIP features from 3D point clouds. We sample 4096 points from and feed them into Uni3D to obtain the 3D feature EY 1xd Second, we project the point cloud of x to the XoY plane to get an image and feed it to the image encoder of CLIP (Radford et al. 202 model to obtain a projected 2D feature E Rexd c denotes feature length of 2D CLIP feature.
Landmark feature. An object in a city typically has its own name, such as "Clare College", which we call landmark. Landmarks are frequently mentioned and critical in the city-scale visual grounding. For example, we will describe building "next to Clare College". Therefore, if the object has a landmark name, we embed the name into E E IR Ixd using a BiGRU (Chung et al. 2014) model and set F, to zeros otherwise.
Spatial context feature. visual grounding task, text descriptions often include relationships between the target object and its neighboring objects. To provide such spatial context information,
4

---

Published as a conference paper at ICLR 2025
Candidate objects
Attribute feature
Trainable Parameters
Feature Embedding
Frozen Parameters
Landmark fouture
SCR Grounding Result
Spatial context feature
0.57
Text description t:
00O COO
Fine-grained Matching
0.87
of city, adjacent verdint park. boses a spacioue swimming pool and offers accommodation buildings with blue façade.
Tokemize & Embedding
C...
Module(FMM)
0..0
COO
Figure 3: Fine-grained matching stage of City City Aichho scores the similarity between the text description and each candidate object with the LLM-driven Fine-grained Matching Module. The object with the highest similarity score is chosen as the grounding result.
we extract K nearest neighboring objects of x denoted as Y, {ve. i = =1, 1,.,, K} and collect their attribute features and landmark features as {E Y,, Eg Kxd] and {E, ly Ya, Fy IR Kxxd for x.
Then, we concatenate the above feature into feature vector E. = E E [((+c+2K)xd where -|I- means the concatenating on the feature dimension. Then, further con- catenate it with the text embedding Etst E Rx to get feature vector of length (2 c 2K +1) x d, which serves as the input the transformer encoder of FMM.
3.3.2 SIMILARITY SCORE PREDICTION
To enable FMM to predict the similarity between text description and object æ, expand the vocabulary of LLaVA with a new placeholder token, denoted as < SCR >. Subsequently. we train FMM generate output texts Ty containing the < SCR > token. We then extract the last layer feature FMM corresponding to the < SCR > the response. This feature is fed into an MI.P to regress a score s, quantifying the similarity between the text description and the input object.
3.4 TRAINING LOSSES
The training of City Anchor involves two stages. For CLM, we adopt an auto-regress regressiee cross-entrow (ACE) loss (Liu al. 2024b) ensure that L.LVVA generates reasonable text responses T, containing < RoI >. We further introduce segmentation loss (Lai et al. (023) to supervise the heatmap H
output by the segmentation model in CLMM The loss for CLM training is formulated by
LCLM Laat (Tey Te) beg (H, ff),
where F. is the annotated ground truth text answer containing Rol > token, H denotes the ground truth binary heatmap for candidate object selection in CLM with the ground truth region of the I and other pixels set O.
grounding target
For FMM, we also adopt the ACE loss to ensure it generates text answers Ty with the < SCR > the binary cross-entropy loss to supervise the regressed score of FMM to be
token. Then, the same as the ground truth score &, where å is set to if the input object corresponds to the text description and 0 otherwise. The loss for FMM training is formulated by
LFMM Ata (Ty, TT) + Ascr (s,
(2)
where T is the annotated ground truth text answer containing SCR > token.
4 EXPERIMENTS
4.1 EXPERIMENTAL PROTOCOLS
4.1.1 DATASETS
To demonstrate the effectiveness of our design, we adopt the CityRefer (Miyanishi et al., 2023 dataset and self-annotated dataset called City dataset for evaluation.
5

---

Published as a conference paper at ICLR 2025
30m
Rasterized Map
(RGB)
lel There is secdion
A hetge building with dank gray moo, that attached larpe parking
baakk building that
one house the lf
the intersection with
lot norh four moss cans,
off the Filznry Laae, buildnng three chimneys
bessie
g
(c) It multi molond
(d) The wiiee caron the nggt sdd the kig perking lot abova the Geallon West buildin. Tarm's sed-black
(6) blck corin between a rel car and
I11 itis mo peised the culemosi grax parking cst thruga
[gt (g Imogitar groan
[bl The recoangullrr
building with dan's
ground is on the edge of victont steet, 12n space between the rad and the
parking lot the
rof that has souar ponels In locited North
cmppy space and a white car the parking oot the Gridin West builing
Fibmay Lami, with no paclodd in there
cumplttlyy covered
tiee
Terrace and Brammock cottigis,
trees There
parking
Fisery
inother green
builinng with
Lishaee buiding.
commer
rettunpllrr shape.
Figure 4: Qualitative results on the CityRefer (Miyanishi et al. 2023) dataset. The projected 2D map obtained from the city-scale point cloud by top-view projection. The candidate objects from CLM are represented by red masks. In the query text, the target object is marked in red, the landmark name marked in blue, and the neighborhood description is marked in green. Grounding results shown in red boxes.
CityRefer MMyannishi et al. 2023) is a 3D visual grounding dataset annotated from city-scale dataset urban of more than 6km² and comprises over 35,000 natural language descriptions of 3D objects, alongside more than 5,000 landmark labels annotated using OpenStreetMap data. The grounding objects mainly fall into categories: Building, Car, Ground, and Parking. We use 85% of them for training and 15% of them for evaluation.
SensatUrban (Hu al. 2021bb dataset. The dataset
CityAnchor is sity- 3D visual grounding dataset. We use 25 city-scale point clouds of STPLS3D (Chen et al. (2022) dataset and manually annotate them with text prompts. There are 1448 text-object pairs. 80% of these pairs are used in training while the rest are used in tests. The objects in the CityAnchor dataset cover 9 different categories including Building, Vegetation, Aircraft, Truck, Vehicle, LightPole, Fence, StreetSign and Bike.
4.1.2 BASELINES
We re- re-train and evaluate all baselines with the same settings as CityAnchor for a fair evaluation.
InstanceRefer Yuan et al., 2021) is matching based framework designed for visual grounding point clouds, which leverages panoptic segmentation alongside linguistic cues to identify candidate instances in point clouds.

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative results on the CityRefer (Miyanishi et al., 2023) and the Cichor datasets. "NO" and "ND" are abbreviations for "Novel Objects" and "Noee Descriptions" respectively.
CityRefer-NO
CityRffor-D
CityAnchor-Ne
CityAachor-NV
Method
Acc@0.25
A.c@0.30
Acc@0,25
Acc@0.50
Aco@0.25
Acc@0,50
A.c
Acc@0.50
InstanceRefer
4.09
3.64
1.93
1.76
3.04
2.31
TDVe-Traasforrme
7.73
5.69
9.64
8.12
4.16
2.38
6.25
4.17
EDA
6.96
5,53
8,39
5.84
5.15
3.09
7.14
4.29
CityRefer
8.34
7.47
5.07
3.49
5.73
4.16
6.07
3.95
CityAnchor
50.69
46.86
53.17
50.37
41.23
35.11
47.81
43,40
Table 2: Additional comparisons with CityRefer (Miyanishi et al. 2023 method. "Time" means the time used in conducting visual grounding once.
CityRefer
CityAnchor
Method
Top-1
Top-3
Top-5
Time(s)
Top-1
Top-3
Top-5
Time(s)
CityRefer
7.47
10.24
12.19
42.27
4.16
6,91
11.05
96.91
CityAnchor
46.86
57.12
59.83
32.45
35.11
41.98
48.09
51.72
3DVG- Traasfranfrrrrr (Zaao et al. 2021) is transformer-bai method specifically designed for 3D visual grounding, comprehensively considering diverse relations enhance proposal generation and facilitate cross-modality proposal disambiguation.
EDA (Wu et al. 2023b) a one-step, explicit, dense-aligned approach 3D visual grounding tasks. This approach systematically decomposes text into multiple semantic components and achieves dense alignment with corresponding visual features. It employs position-aligned and semantic-aligned loss functions to facilitate fine-grained fusion of visual-text features, thereby mitigating issues of information blurring and imbalance commonly encountered existing methods.
RefeR (Miyanishi et al. 2023) is baseline designed for 3D visual grounding in city-scale scenes. CityRefer segments the whole scene into Slass-agnostic objects, compares the query text with all urban objects iteratively, and selects the most similar as the grounding result.
4.1.3 METRICS AND EXPERIMENTAL SETTING
We use the Intersection over Union (IoU) between our grounding results and the ground truth objects for evaluation. We report the ratio of grounding results with an IoU larger than 0.25 and 0.5, denoted as Acc@0... 0.25 and Acc @ @.50 respectively. In evaluation, previous methods including CityRefer (Miyanishi et al. 2023) typically first select 10 candidate objects and only conduct visual grounding among these candidates. Since our target is to evaluate the performance of city-scale visual grounding, we directly apply the grounding algorithm among all hundreds of objects in a scene of around 400mx400m without manually selecting any candidates. The CityRefer dataset contains 41 scenes while the City Anchor dataset contains 25 scenes. We evaluate the performance in two settings "Novel Objects" and "Novel Descriptions". "Novel Objects" setting means the objects in the test set are unseen in the training set. For example, a blue car object test set would not appear in training set. "Novel Descriptions" setting indicates that the evaluation objects the training set, but the query text descriptions are different. For example, an object may be described "A building with white roof near the road." "in training set but "Along with the street, there is white-roofed building" in test set.
seen in
4.1.4 IMPLEMENTATION DETAILS
All the experiments are implemented with Py Torch on a single NVIDIA A100 GPU (40 GB). In the coarse localization stage, we obtain the input RGB map by rasterizing the tip-view projection of the point cloud with a resolution m. We use the pre-trained weights from LISA (Lai et al.) 2023) to initialize our Coarse Localization Module. For our Fine-grained Matching Module, use the weights of Vicuma-7b-v1.3 (Zheng et al.) 2024) to initialize the modules inherited from LLaVA architecture. All fine-tuning is conducted with the LoRA layers (Hu al. 2021a). We use the Adam optimizer with a batch size of 8 and learning rate decaying from 2e-5 to 2c-7 with a cosine annealing scheduler. The training of CLM and FMM takes about 12 and 15 hours converge. We
7

---

Published as a conference paper at ICLR 2025
set the threshold 0 for the candidate object detection in CLM to fixed 0.3 (except for the specialized
analysis of Rol threshold) and the number of neighboring objects K for spatial context-aware feature enhancement in FMM to 5. We select positive and negative samples in ratio of 1:3 for FMM training.
(a) Query:
(b) Quar::
(C) Query:
(c) Quary:
2is cray corss parkod middle row
quey and Back rofed balddag
The cre parkod in the Pevato
The grey mod buiding that is on
of the carpork bihind West Mid andi
located benwees Holdond Avenue and the
parking ot agsinat the sida the gray
Agho Rood, and it is womes the Exst
Pulice Cudtady Suile. has dack Nue
Manchodte-- Emomouth frotpoh, in froa of lhe lange car park off Halloni
huuse on the Alpha Boad. The car the
Hertorrd Street Alse the blue car the
for portedd to the rggtt (tacing the West
only one parked under the woe in tnott
wittto car are oomes Them small
Midiands Polios Custddy Suile! and
Avemun To the fron of the baildng is
dhr building slatt.
porking ane at the let side of the building
mooth
the
lageerrpet
Top-1
(b) Top
(T)
(c) Top-1
(a) Top-2 2
Top
(c) Top-2
(c) Top
(a) Top-3
(b) Top-
(c) 3
(cD Top 3 3
Figure 5: Top-3 objects retrieved by City CityAnchor. The bounding boxes of grounding results are displayed red. In the query text, the target object is marked in red, the landmark name is marked in blue, and the neighborhood statement is marked in green. CityAnchor distinguishes the correct target (Top-1) I) from quite similar candidates by considering the neighborhood information.
4.2 QUALITATIVE RESULTS
In Fig. we provide visual grounding results of our Aithon on the CityRefer dataset and more qualitative results and comparisons on both datasets are provided the Appendix. the first stage of CityAnchor can identify the candidate objects effectively. which provides good initialization for efficient object matching in the second stage. Fig. (d-f), we showcase the results of cars in the same city. It can be seen that City can accurately identify the target cars even when there are many similar candidate cars. The strong performance mainly comes from our robust object encoding that enables the LLM in Aithonchor to discriminatively match the candidate objects with the query text descriptions in terms of geometry, colors, and spatial contexts.
be seen that
4.3 QUANTITATIVE COMPARISON
The quantitative results on the two datasets are reported in Table Baselines (Yuan et al. 2021 Zhao et al. 2021 Wu et al. 2023b) manually design the network structures to learn features for 3D visual grounding, which does not perform well on the city-scale datasets containing more objects with complex spatial contexts. CityRefer (Miyanishi et al. 2023), designed for city-scale grounding. shows reasonable results on the city-scale visual grounding task. The proposed Aichon surpasses all baseline methods by 36% ~ 51% on Ace @0.25 and 31% ~ 48% on Ace@ 0.50 on two datasets, which demonstrates that methods effectively handle 3D visual grounding on city scale.
8

---

Published as a conference paper at ICLR 2025
Table 3: Ablation studies on the CityRefer dataset. When not using Stage I, we iterate all the objects to find the most matched one with the text description. Dx means the CLIP (Radford et al. 2021 feature extracted on the top-view of 3D object in FMM. Eg means the Uni3D (Liu et al. 2024a) feature extracted on 3D object in FMM. E means the encoding of the landmark names in object encoding of FMM. "Nei. Enh." means enhancing the object features with spatial context information in object encoding of FMM. "Time" means the time used in visual grounding inference.
ld Stage
Stage II
Acc@0.50
Time(s)
E
E
Nei. Enh.
Building
Car
Ground
Parking
Overall
a)
25.27
36.17
28.26
21.51
31.82
b)
V
26.71
45.48
30.43
21.25
38.01
c)
V
V
26.98
50.94
44.38
36.23
43.12
d)
V
V
24.91
23.65
40.91
37.16
25.10
83.26
e)
V
28.52
56.25
45.64
37.97
46.86
32.45
Table 4: Quantitative analysis of Rol thresholds of CLM using accuracy the metries. "Time" means the time used in conducting visual grounding once.
Rol threshold 0
Top-1
Top-2
Top-3
Top-5
Top-10
Time(s)
0.25
40.92
45.93
47.27
50.04
54.13
48.11
0.30
46.86
53.31
57.12
59.83
64.66
32.45
0.35
26.80
28.85
30.48
32.36
35.10
27.05
0.40
19.66
22.20
23.05
24.17
26.39
21.19
4.4
MODEL ANALYSIS
Analysis Top-k results. In Table 2 we further conduct top-k experiment in comparison with CityRefer (Miyanishi et al. 2023) on the two city-scale datasets. For each query description, we select in objects with top-k matching scores, and is set to 3, and 5. Then, ratio of Top-k objects, where a result is regarded as correct if top-k results contain an object with larger IoU than 0.5 with the ground-truth object, i.e. Acc @0.5. It can be seen that City Anchor consistently outperforms the baseline CityRefer by a large margin on all top-k accuracy. In Fig. 5 we visualize the top-3 grounding results produced by Cityonchor. predicted by CityAnchon are all reasonable and partially aligned with the text descriptions like in colors. CityAnchor is able to distinguish them by other text descriptions about the spatial contexts of the objects. For example, in Fig. 5 (a), "a mostly empty grassy space to the left" is critical to localize the most matched car rather than the other two cars. This mainly comes from Anchhr considering both the object properties and the spatial contexts in the second fine-graines matching stage. Also, we further report the time used by City once on both datasets. City Aichhr requires only 22.45s to localize an object in scene of the CityRefer dataset and is 1.3x faster than the CityRefer method. This is mainly attributed to our coarse-to-fine strategy, which filters out most irrelevant objects.
report the correctness
be seen that the top-3 results
Ablation Studies. In Table 3. we conduct ablation studies of designs and report Acc @0.50 on the CityRefer dataset. We validate the effectiveness of the candidate object detection in CLM and the feature extraction in FMM. Comparing a) and b), we observe that encoding 3D object features E using Uni3D ÇLiu et al., 2024a) and the projected 2D features E using CLIP (Radford) al. 202T both improve the grounding performance. Comparison between b) and c) highlights that encoding landmark names E into object features enables FMM to achieve more accurate grounding results, particularly for targets strongly related landmarks such as grounds and parking. Comparison between d) with e) reveals that utilizing the predicted candidate object selection of our CLM accelerates inference by 2.5x and significantly improves the grounding accuracy by ~ 21%. Comparing c) with e) demonstrates that incorporating spatial context information from neighboring objects improves the accuracy of localization the grounding task.
Analysis on ROI threshold. In the first stage, we use threshold 0 to determine the Rol output by CLM, and grounding is only conducted on objects within the Rol. We perform an analysis on the threshold using grounding accuracy and runtime as metrics. As shown in Table 4 a low threshold results in numerous candidate objects being involved in fine-grained comparisons, thereby leading to long grounding times. Conversely, setting stricter threshold results in the exclusion of correct target

---

Published as a conference paper at ICLR 2025
w/o Color
w/o Shape
w/o Nei.
Full Query Text
GroundTruth
small is
red
(C-1) small red
[D-1] A small red that is
(E-1) small red that blocked by a
blocked by small car along the mead behind it aoother small car.
blocked in by a wille car alony the road, behind it
that blocked by small
blocked by
white car along the road.
car along the road, behind it
car along the read behind it
aoother
mothhrr small red
another small not car.
(A-2) hiss square building with roof
(8-2) This is building with roof aloug the
(C-2) This is square building with white reof.
(D: 2) This square building with white roof along Walaal Rood, apposits of the One Stop Building
(C-2) This ts square building with white roof along Walsall Road, appasits: of the One Stop Ballding.
Walsall Road, of the One Stop Building.
Walsall Road, opposite of the
One Stup B salding.
Figure 6: Analysis for specific parts of text prompts two representative examples. "w/o Color", "w/o Shape" and "w/o Nei." mean removing the description related to color, shape and neighborhood information in query text, respectively.
objects, leading to a decrease in grounding accuracy. To strike balance between grounding accuracy and time consumption, we selected 0.3 as the Rol threshold for the CLM in the first stage.
Analysis on different text descriptions. To show the compatibility with different styles of text prompts, we perform extended experiments to evaluate the impact of different text descriptions on grounding performance. As shown in Fig. 6 we visualize the grounding results of systematically removing text descriptions related to color, shape, and neighborhood contextual information. Color description plays important role in visual grounding and the lack of color information often leads to incorrect results. In contrast, while shape descriptions are less critical for cars, they become more significant for buildings due to the larger variability of shapes. In a scene with many similar objects (e.g., red cars), color and shape descriptions are inadequate to distinguish these objects, and integrating neighborhood contextual information is vital for achieving accurate grounding.
5 LIMITATIONS AND CONCLUSION
Limitations. Though our method produces reasonable results on the city-scale 3D visual ground- ing task, there still limitations. The inference time is still long, which costs ~32s for text description. This inference process could be sped up by quantization and pruning as done in other LLM applications. We can incorporate landmark information into Aithonnchor external knowledge using Rerreeval-Augmenned Generation (RAG) (Gao et af. (023). utilizing landmark information for efficient grounding. Another limitation is that we need adjust the RoI threshold during the coarse localization stage, as the sizes and shapes of objects differ on the projected RGB map. Our future work will involve improving the efficiency and generalizability of Rol detection as well as extending CityAnchor to conduct grounding for complex and dynamic 3D city-scale point clouds.
We present Cichor 3D visual grounding method designed for city- scenes. The key component of Cityonchor is a two-stage multi-modality LLM. In the coarse stage, CityAn- chor predicts candidate objects that correspond to the query text description on the projected 21) maps from the urban point cloud, which enables us to quickly rule out irrelevant regions and focus on plausible objects. In the fine stage, City Anchor conducts fine-grained matching between the text descriptions and the possible objects in the above candidate objects to determine the final grounding results by encoding both text descriptions and the features of objects into an LLM. We manually annotate a new dataset called Cichonchor and evaluate method both the CityAnchor dataset and the CityRefer dataset. The results show that our method can produce accurate visual grounding on city-scale point clouds.
Conclusion.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
This research is jointly sponsored by Projects of the National Key Research and Development Program of the Ministry of Science and Technology, China 2022YFB3904102 and NSFC Basic Research Project for Ph.D. Candidate (No.424B2012).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report.
arXiv preprini arXiv:2303.08774. 2023. 2
Panos Achlioptas, Ahmed Abdelrcheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grate tne-grained 3d object identification real-world scenes. In Computer Asion-ECCYS 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt I 16, pp. 422-440. Springer, 2020. 2
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and- lan-lag navigation: Interpreting visually- grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3674-3683, 2018.
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization rgb-d scans using natural language. In European conference on computer vision, pp. 202-221 221- Springer, 2020. 2
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi Touchdown: Natural language navigation and spatial reasoning in visual street environments. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12538-12541 2019, 2
A.6
Meida Chen, Qingyong Hu, Zifan Yu, Hugues Thomas, Andrew Feng, Yu Hou, Kyle McCullough, Fengbo Ren, and Lucio Soibelman. Stpls3d: A large-scale synthetic and real aerial photogrammetry 3d point cloud dataset. arXiv preprint arXiv:2203.088 090065, 2022. 4.1.1 A.3 A.6
Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding, reasoning. and planning. arXiv preprint arXiv:231 1. 2023a.
Yiping Chen, Shuai Zhang. Ting Han, Yumeng Du Wuming Zhang, and Jonathan Li. Chat3d: Interactive understanding 3d scene-level point clouds by chatting with foundation model for urban ecological construction. ISPRS Journal of Photogrammetry and Remote Sensing, 212:181-192
2024. 2
Zhenyu Chen, Ronghang Hu, Xinlci Chen, Matthias Nießner, and Angel X Chang. Unit3d: A unified transformer for 3d dense captioning and visual grounding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 18109-18119, 2023b.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 15X55:1412.3555, 2014.
3.3.1
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotaal 3d reconstructions of indoor scenes. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017. -
Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403. 1401, 2024. 2
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval- augmennted generation for large language models: A survey. arXiv preprint arXiv:2312. 10997, 2023. 5
11

---

Published as a conference paper at ICLR 2025
Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: open-vocabulary 3d scene graphs for perception and planning. arXiv preprint arXtv:2309.166500 2023. 2
Ziyu Guo. Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint
wrxiv:2309.00615, 2023. 2
Dailan He, Yusheng Zhao, Junyu Luo, Tianrui Hui, Shaofei Huang, Aixi Zhang, and Si Liu. Transre- fer3d: Entity-an andereaionr aware transformer for fine- fine-graines grained 3d visual grounding. In Proceedings
of the 29th ACM International Conference on Multimedia, pp. 2344 2352, 2021. 2
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-20444 2023a.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lIm: Injecting the 3d world into large language models. Advances Neural Information Processing Systems, 21-20082-204944 2023b. 1
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprini arXiv:2106.0966 202
4.1.4
Qingyong Hu, Bo Yang. Sheikh Khalid, Wen Xiao, Niki Trigoni, and Andrew Markham. Towards semantic segmentation of urban-scale 3d point clouds: A dataset, benchmarks and challenges. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 4977-4987, 2021b.
Pin-Hao Huang, Han-Hung Lee, Twaan-Tzzog Chen, and Tyng-Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1610-1618, 2021.
Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 15524-15533, 2022. 2
Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang, Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d langual learning for grounded scene understanding. arXiv preprint arXiv:2401.094 093400 2024. 1
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014. A.2.2
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014. A.3.2
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023. 3.2
Manuel Kolmet, Qunjie Zhou, Aljoša Ošep, and Laura Leal-Taixé Text2pos: cext-to-point-clon
cross-modal localization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 6687-6696, 2022. 2 A.6
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308. 00692, 2023. [ 3.2 3.4
4 A.2.3
12

---

Published as a conference paper at ICLR 2025
Lik- Hik-Hang Lee. Tristan Braud, Pengyuan Zhou, Lin Wang, Dianlei Xu, Zijun Lin, Abhishek Kumar, Carlos Bermejo, and Pan Hui. All one needs to know about metaverse: complete survey on technological singularity, virtual ecosystem, and research agenda. arXiv preprint arXh::11000053352 2021.
1
Zeju Li, Chao Zhang, Xiaoyan Wang, Ruilong Ren, Yifan Xu, Ruifei Ma, and Xiangde Liu. 3dmit: 3d multi-modal instruction tuning for scene understanding. arXiv preprint arXiv:2401.03200 2024.
2
Dingning Liu, Xiaoshui Huang, Yuenan Hou, Zhihui Wang. Zhenfei Yin, Yongshun Gong, Peng Gao, and Wanli Ouyang. Uni3d-llm: Unifying point cloud perception, generation and editing with large language models. arXiv preprint urXiv:2402.03327 2024a.
3.3.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in
neural information processing systems, 36, 2024b. 2 2 3.2
3.3
3.4
Xianzheng Ma. Yash Bhalgat, Brandon Smart, Shuai Chen, Xinghui Li, Jian Ding, Jindong Gu, Dave Zhenyu Chen, Songyou Peng, Jia-Wang Bian, et al. When IIms step into the 3d world: A survey and meta-analysis of 3d tasks via multi-modal large language models. arXiv preprint arXiv:2405.10255, 2024. 2
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 11-20, 2016. A.2.2
Taiki Miyanishi, Fumiya Kitamori, Shuhei Kurita, Jungdae Lee, Motoaki Kawanabe, and Nakamasa Inoue. Cityrefer: Geography-avee aware 3d visual grounding dataset on city-scale point cloud data. arXiv preprint arXiv:2310.1877 2023.
A.2. A.6
Mahyar Najibi, Jingwei Ji, Yin Zhou, Charles R Qi, Xinchen Yan, Scott Ettinger, and Dragomir Anguelov. Unsupervised 3d perception with 2d vision- language distillation for autonomous driving. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8602 8612,
2023. 1
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763 PMLR, 2021. 3.3.1 4.4
Junha Roh, Karthik Desingh, Ali Farhadi, and Dieter Fox. Languagerefer: Spatial- model for 3d visual grounding. In Conference on Robot Learning, pp. 1046-1056. PMLR, 2022. 2
Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, and Min Chen. Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors. arXiv preprint arXiv:2405.01413. 2024. 2
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation
and fine-tuned chat models. arXiv preprint arXiv:2307.09286 2023. 2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 2
Thang Vu, Kookhoi Kim, Tung M Luu, Thanh Nguyen, and Chang D Yoo. Softgroup for 3d instance segmentation point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2708-2717 2022. 3.1 A.3.I
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024. [
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal
lim. arXiv preprini arXiv:2309.055/11 2023a. 2
13

---

Published as a conference paper at ICLR 2025
Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. Eda: Explicit text- decoupling and dense alignment for 3d visual grounding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19231-19242, 2023b. 4.1.2 4.3
Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. Lim-grounder: Open-vocabulary 3d visual grounding with large language model as an agent. arXiv preprint urXiv:2309. /2311, 2023a.
Li Yang, Ziqi Zhang, Zhongang Qi, Yan Xu, Wei Liu, Ying Shan, Bing Li, Weiping Yang. Peng Li, Yan Wang, al. Exploiting contextual objects and relations for 3d visual grounding. Advances in Neural Information Processing Systems, 36, 2024. 2
Senqiao Yang. Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint arXiv:2312. 140744 2023b. [
Zhengyuan Yang. Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat: 2d semanties assisted training for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1856-1866, 2021. 2
Zhenfei Yin, Jiong Wang, Jianjian Cao. Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, al. Lamm: Language-assise assisted multi-modal instruction-tuniin dataset, framework, and benchmark. Advances in Neural Information Processing Systems, 36, 2024. 2
Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contextual referring. In Proceedings of the IEEE/CVF International Conference
on Computer Vision, pp. 1791-1800, 2021. 4.1.2
Sha Zhang, Di Huang. Jiajun Deng, Shixiang Tang, Wanli Ouyang, Tong He, and Yanyong Zhang Agent3d-zero: An agent for zero-shot 3d understanding. arXiv preprint arXiv:2403. 1835, 2024.
Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. In Proceedings the IEEE/CVF International Conference Computer Vision, pp. 2928-2937, 2021. 2
Lianmin Zheng, Win Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et Judging Ilm-as-as au--ase with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. 4.1.4
Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, and Qing Li. 3d-vista: Pre- trained transformer for 3d vision and text alignment. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 291 1-2921, 2023.
7 APPENDIX
In the appendix section, to further demonstrate the effectiveness of our proposed city-scale visual grounding system Aityonchor, we present additional visualization of grounding results. Moreover, introduce the additional implementation details of CityAnchor and the self-annotated City Anchor dataset.
A.1 ADDITIONAL QUALITATIVE RESULTS FOR CITY-SCALE VISUAL GROUNDING
In A we provide additional qualitative results our method on the self-annotated City Anchor dataset in "Novel Objects" setting, In Fig. A.2 we provide additional qualitative results of our method on the CityRefer dataset in "Novel Objects" setting. The filter effectiveness of Rol output by CLM can vary depending the size and attribute of target objects. Large objects (e.g., factory, athletic field, parking lots) can be easily identified and excluded through the heat map output by CLM. In contrast, small objects (e.g., cars, residential buildings)
challenging distinguish, requiring
14

---

Published as a conference paper at ICLR 2025
20m
(b)
Rasterized Map
(RGB)
This building
[b) rectangulee
with moof,
ibuilding, and tallest
comperison
intersaation road and there also cyan oar and a red car
sursunding buildings.
The gool of the building
yallow sumounded by
parked cest to
1
(h)
(d)
(g)
parked nost building stroetlight, and them is another car not for from this
track
Use,, unusuall the road, leaves ar red, there are fiere buildings neal to
Them while building, green tree, and groen car nest
This buildig is not very lall but Dovers a large From the view. reqular U Uhappp..
building
parked the middle the road asd to many huildings.
of vacant int, and noxt to it banlding with
white roof.
Figure A.I: Qualitative results on the Aichor dataset "Novel Objects" setting. The projected 2D map is obtained from the city-scale point cloud by top-view projection. The candidate objects from CLM are represented by red masks. In the query text, the target object is marked in red, the landmark name marked in blue, and the neighborhood statement is marked in green. Grounding results are shown in red boxes.
further detailed comparison in the FMM. In Fig. A.3 provide qualitative comparisons between the proposed Cichonnchrr and the baseline method CityRefer, where CityAnchor can distinguish the correct target for similar erroneous objects by considering both the object attributes and the spatial context information. In Fig A.4 and Fig A.3 we provide additional qualitative results our method on the CityRefer dataset in Novel Descriptions" setting.
We have several observations from them. First, color information is frequently mentioned in each query. Encoding the RGB information of object with Uni3D and CLIP is particularly important for distinguishing similar vehicles or buildings, as the color of the car or roof can serve as the most immediate and intuitive visual cue for object description. In addition, landmark information is also indispensable for open-world navigation and localization. Roads and parking lots are typically feature- less but have landmark names. Therefore, incorporating landmarks into City Anchor promotes the ability to focus on specific objects through their proprietary names, thereby improving the accuracy of 3D visual grounding.
A.2 EVALUATION CANDIDATE OBJECT SELECTION CLM
In this section, we aim to evaluate the accuracy of the candidate objects selected by the coarse localization module. Here, adopt the same threshold to segment the heatmap. With this threshold,
15

---

Published as a conference paper at ICLR 2025
35m
Rasterized Map
(RGB)
(a)
(b)
(c)
(d)
(a)
(b)
(d)
(a) This is an athletic field, which has 400m track and soccer field. Nexc to is a road with wooded area.
(b) The large light gray building across the street, from short side of the National Probation
(c) Back section of a two- story multicolored white and gray home with brown roof, with light blue car facing it off Teddington Grove.
(d) This is large
triangular parking space
near the sports field and white building. T hhere several comi-trucks this parking space.
Service building.
Figure A.2: Additional Qualitative results on the CityRefer dataset in "Novel Objects" setting. The projected 2D map is obtained from the city-scale point cloud by top-view projection. The candidate objects from CLM are represented by red masks. In the query text, the target object is marked in red, the landmark name is marked in blue, and the neighborhood statement is marked in green. Grounding results are shown in red boxes.
we turn it into a binary heatmap and evaluate its IoU with the region of the ground-truth object on this 2D map. We call these activation regions on the heatmap Region of Interests (Rol).
A.2.1 MODEL TRAINING
The training data for the CLM is derived exclusively from CityRefer (Miyanishi et al. 2023) dataset and the CityAnchor dataset. We project the point cloud instances onto the XoY plane to obtain rasterized object masks and then densify them using traditional erosion and dilation operations. Finally, feed the cot-imagee-mak pairs into the CLM for training and validation. During the training process, the weights of the text generation loss Xux and the mask loss Aseg are set to 1.0 and 1.0, respectively. Besides, set the batch size as 2 and the number of epochs as 5.
16

---

Published as a conference paper at ICLR 2025
white house with beige roof on
This residential
This the blue car in parking lot to the right the BMTR building. There black, red car to the south this car, and the north.
A car next to blue car with a black top off Teddington Grove, front multicolored White and beige house with brown roof.
building located on Walsall Road, yard another house with red car parked in its front yard.
Text
Teddington Grove cul-de-
sac, There is black
empty.
parked in the driveway next to the house.
CityAnchor CityRefer GroundTutth
Figure A.3: Qualitative comparisons of the baseline method CityRefer and the proposed framework CityAnchor. The ground truth and predicted boxes displayed in green and red, respectively.
Table A.I: Quantitative analysis of Rol area segmentation on CityRefer and CityAnchor datasets.
CityRefer gloU
CityAnchor gloU
Method
cloU
cloU
LISA-7B
11.5
10.1
13.4
13.0
LISA-13B
16.2
13.7
17.1
14.9
A.2.2 EVALUATION METRICS
Following most previous works (Kazemzadeh et al. 2014 Mao et al. 2016) on image segmentation, adopt metrics: gloU and cloU. The gloU is defined the average of all per-image Intersection- over-Unions (loUs), while cloU determined by the cumulative intersection over the union. Given that cloU tends be highly biased toward large-range objects and exhibits significant fluctuations, gloU is the preferred metric.
A.2.3 CANDIDATE OBJECT SELECTION RESULTS
Visualization. As illustrated in Fig A.6 We further visualize the heat maps, on which have performed smoothing. The value of each pixel in the heatmap indicates the correlation degree with the text description (red color indicates a strong correlation and blue color indicates a weak correlation).
Quantitative analysis. As shown in Table A. we show the IoU of Rol on two datasets after model fine-tuning using different LLM backbones. this Rol regression task from a projected 2D map, LISA-13B (Lai et al. 2023) demonstrates superior performance compared to LISA-7B (Lai et al.) 2023) on both the Cityrefer and Cityanchor datasets, particularly when the query text descriptions long, which indicates that understanding text descriptions and extracting discriminative features remain performance bottlenecks. A more powerful LLM backbone could improve performance the segmentation task.
17

---

Published as a conference paper at ICLR 2025
25m [
(a) A small building with a gray roof on Chesterton Lane, it below the Ethelreda building to left Castlebrae.
(b) building with grey roof and white wall adjacent the building named JSG Wine Merchant, The object front of the building is Chesterton Road.
(c) This small parking
parking in front of the while colored building named Henry Giles House situated along the road named Carlyle Road.
space localed Hertford
Street between Marino
Place and Edward's Court.
There is car parked
this space.
(h)
The green ground with lots of tree, along the Floating Wier Safety B Barier, it the Chesterton Road, white and red car parked front of the ground.
(1) A of two lennis courts within large grass field the comer of the map, that larger four tennis courts beside the river.
red car is parked undera a tree front of the rundel House Hotel along the Chesterton road. nother red car parked behind it and white van
(h) The blue car Hertford Street that is across from the gray house that two houses the right of the Edward's Court.
to
parked front of it.
Figure A.4: Additional qualitative results of Scene the CityRefer dataset in "Novel Descriptions" setting. The bounding boxes resulting objects are drawn red. In the query texts, the target object marked in red, the landmark name is marked in blue, and the neighborhood statement is marked in green.
18

---

Published as a conference paper at ICLR 2025
25m
(c)
(d)
(a) This is one- mlool white building with multi colored gray and white roof, located between the Tuřno] building and the railroad,
(b) The parking is
(c) The blue localed
(d) A mamon parked
located at the comer building the Station Bridge Road and the Wellington Road, A nd this parking lot is almost full of cars.
second from the bottom in the rightmost column of
the front of the
parking lot behind the building Wellington
the parking lot, when viewed from above with the Walsall Road positioned below.
Road and Birchfield Road between light blue and a light gray car.
(e)
ff
(g)
(e) The orange car parked next to the white car on Wellington Road, by the blue building with white roof.
(0) The black car is parked middle left edge map, positioned betweet blue cars.
(g) A large brown multi- unit building complex with four brown and gray roofs sits between a lot filled with blue containers and the railroad at the edge the map.
(h) The gray rectangular building on the Birchfield Road that directly to the right of the Station Bridge,
Figure A.5: Additional qualitative results of Scene on the CityRefer dataset in "Novel Descriptions" setting. The bounding marked in red, the landmark name is marked in blue, and the neighborhood statement is marked in green.
boxes resulting objects are drawn red. In the query texts, the target object
19

---

Published as a conference paper at ICLR 2025
The area consisting of a green ground and red running track and tribunes around it, next to the Perry Barr Greyhound Stadium Building.
very large multi-building complex near the edge of the map, with Bruce that are either white, dark gray, that next parking
of terraced houses adjoining green space close Brunswick gardens, its has three large trees in line at the green side.
2D Map (RGB))
HeatMap
large athletic field, next to wooded area with many trees.
The large football pitch with distinctive striping in the field.
Pure white building with grassy area next
The building with flower bed next to it.
2D Map (RGB3
(b)
(d)
HeatMap
Low
High
Figure A.6: Visualisation of predicted heat maps.
20

---

Published as a conference paper at ICLR 2025
Table A.2: 3D instance segmentation results CityAnchor (STPLS3D) dataset.
Target
AP
AP 60
AP25 25
mRec
mRec50
mRec25
Ground
74.5
77.7
79.5
75.4
78.2
79.8
Vegetation
32.8
58.3
71.1
43.9
68.9
80.5
Building
32.4
48.9
59,4
41.2
60.7
76.9
Wall
27.2
30.8
34.5
28.0
31.4
35.7
Bridge
84.2
94.7
96.7
85.7
95.5
97.6
Parking
78.3
88.6
91.6
82.0
92.4
96.7
Rail
58.6
92.0
92.2
64.6
93.8
96.9
Traffic Road
72.2
80.2
82.9
86.8
94.2
97.1
Street Furniture
18.0
38.4
52.5
34.2
60.9
79.2
Car
54.9
79.0
88.0
61.3
83.0
91.7
Footpath
48.5
68.0
74.6
53.8
70.7
76.7
Bike
13.2
25.6
38.6
17.4
30.3
47.6
Water
48.2
61.5
66.9
58.0
73.7
84.5
Fence
19.4
40.1
71.6
26.2
50.5
78.8
Average
47.3
63.1
71.4
54.2
70.3
80.0
A.3 3D INSTANCE SEGMENTATION
In this section, we conduct an experiment on the STPLS3D (Chen et al. 2022) dataset to evaluate the model performance of 3D instance segmentation on point clouds. The primary objective is to accurately generate segmentation masks, each associated with specific category label, for each identifiable object.
A.3.1 ARCHITECTURE
We use SoftGroup (Vu al. 2022) as the primary network for our city-scale instance segmentation task the STPLS3D dataset. The methodology is split into two stages: bottom-up grouping and top-down refinement.
A.3.2 NETWORK TRAINING
3D instance segmentation city-level dataset was developed using the PyTorch deep learning framework. The SoftGroup model trained 10,000 iterations with Adam optimizer (Kingma & Ba 2014). The batch size was maintained at 32, and the initial learning rate was set at 0.05, modulated a cosine annealing schedule. The parameters such as voxel size and grouping bandwidth were configured at 1.0 meters and 2.0 meters, respectively.
A.3.3 EVALUATION METRICS
The evaluation metric for instance segmentation model performance is the standard average precision, we utilized average precision (AP) and mean recall (mRec) as the primary evaluation metrics for each class assessed. APso and AP denote the scores with IoU thresholds of 50% and 25%, respectively, AP denotes the averaged scores with IoU threshold from 50% to 95% with a step size 5%.
A.3.4 RESULTS ON STPLS3D DATASET
Table A.2 shows the 3D instance segmentation performances. In the STPLS3D dataset, the average A Pss of the SoftGroup method achieves 47.3%.
21

---

Published as a conference paper at ICLR 2025
Query I
Query II
Query III
Query IV
(A-1) big building with brown roof and white wall is situated the intersection point of two road named Walsall mad and
(A-11) The building with Tour rectangular brown roofs the
(A-III) building with brown roof and white wall big perking area around the building situated along road named Regina Drive connecting W Walsall road. big tree situsted the front. left side of the building.
(A-IV) This building has brown multitiered roof sections. W Walsall Road is in tront
corner of Walsail Road and
Regina Drive. There are three
Regina Dave is to the left the building. The building has a larpe parking
Regina Drive, There open parking space in front of the building
light colored cars parked in front of building and immediately
adjacent the intersectinn.
right side of the building
(B-II) blue porked in small square parking area inside the orange brick wall. white car at back, ectar parking by the mundshout.
(B-I) cur to the white oar that parked behind One Stop Building
(B-I(O) This is bloe parked next to Olee Stop. On its is white car there another white in its vicinity.
(B-IV) The dark con clustered with two white cans in mini porking
in comer the One Stop building, There several cars parked next to
comner of the One Stop building.
(C-D The only light browa-yellow roff building running course.
(C-II) large,
(C-110 There building with yellow roof near the street,
(C-IV) Two nee-section building on Alándge Road. huildings reddish, with pitched roof that is mostly tan.
building of the stadiuum. It has lan, yellow
oval
Figure A.7: Analysis for diverse text prompts toward same object three representative examples. Note that a green checkmark in the bottom right corner of the query text box indicates correct grounding case, whereas red cross indicates an incorrect grounding case.
A.4 ANALYSIS FOR DIVERSE TEXT PROMPTS TOWARD SAME OBJECT
In Fig. A.7 we provide qualitative results for diverse text prompts toward same object. addition to thorough object feature extraction, the text description plays a crucial role in visual grounding. For the successful grounding cases, text descriptions related color and category are often indispensable.
A.5 GENERALITY EXPERIMENT FOR UNKNOWN OBJECTS
As illustrated in Fig A.8 we conduct generality experiment to assess CithAnchor's ability to recognize unknown objects. CityRefer dataset, CityAnchor is trained with only four object categories: Building, Car, Parking, and Ground, and introduce unknown object categories (such as Road Intersection, Woodland and River) that CityAnchor has not seen before. We evaluate the City/Achor's ability predict the similarity between these unknown objects and both the correct text descriptions (Positive Sample) and incorrect text descriptions (Negative Sample). In Fig. A.9 we provide both the Rol map and grounding results for three representative unknown objects.
Given the Fine- Fine-grained Matching Module (FMM) in Aichon is fine-tuned from LLaVA, which possesses outstanding real-world recognition capabilities, Aichor effectively inherits these
22

---

Published as a conference paper at ICLR 2025
Road Intersection (Positive Sample)
Woodland (Positive Sample)
River (Positive Sample)
There is circular road intersection and connects four roads.
woodland area near parking lot with many trucks parked.
There river next meadow with fow boats docked along shore.
This cobjs, and the probability 0.97.
<objs, the probability is 0.62.
<gbj> you are qureving, the probabiiity 0.94.
Road Intersection (Negative Sample)
Woodland (Negative Sample)
River (Negative Sample)
building near the street, with and many cars next to this building.
There is white parking
square building black flat root, surrounded
let, next to the utility room.
red cars.
They and the matching probability is only
other,
They can match other, and the matching probablitty is only 0.04.
They can match each other, and the matching probability only
Figure A.8: Qualitative results unknown objects for pre- praited City Eithor. The positive sample represents the case where the object matches the query text, with the matching probability of in GroundTruth, while the negative sample represents the mismatched case, with the matching probability of 0 GroundTruthh.
Table A.3: Comparison of existing 3D visual grounding datasets in outdoor environment. Na means the number of text descriptions for 3D objects. Npoints means the number of points. UAV- means Unmanned Aerial Vehicle (UAV) Photogrammetry. SA-P means Synthetic Aerial (SA)
Photogrammetry.
Dataset
Na
Area
Source
N points
TouchDown
25,575
Roadside
Google Street View
KITT1360Pose
43,381
Roadside
KITTI360 (3D Scan)
1,000M
CityRefer 35,196
City Center
SensatUrban (UAV-P) STPLS3D (UAV-P + SA-P)
2,847M
City Aithon 1448
City
strengths when interpreting each candidate object. Furthermore, the query texts often reference surrounding objects (e.g., roads, rivers, trees, and woodlands) to facilitate successful grounding by providing neighborhood contextual information. This requires Anchor to comprehend a broader range of objects and their fundamental characteristics, even when these objects are not the target objects in CityRefer dataset. The grounding results for unknown objects demonstrate that CityAnchor exhibits the certain generalization capabilities, enabling it to perform 3D visual grounding across broader range objects.
A.6 CITYANCHOR DATASET
As shown in Table A.3 we provide the basic statistics of the CityAnchor dataset in comparison to
TouchDown (Chen et al. 2019). KITTI360Pose (Kolmet et al. 2022) and CityRefer (Miyanishi et al. 2023) datasets.
The TouchDown dataset is derived from open-access Google Street View, aiming at text- eext-guided navigation and spatial reasoning using real-life visual observations. KrTTI360Pose dataset is oriented towards outdoor traffic environments and specifically designed to enhance mobile robot localiza- tion using templated language descriptions that focus solely on positional data. However, both the TouchDown and KITTI360Pose datasets are derived from a vehicle-based system, which is constrained to roadside environments. Although the benchmark dataset CityRefer constructed on the basis of publicly available 2D map (OpenStreettMap), visual grounding in general may result in privacy issues or racial and gender biases. To this end, we propose CityAncho dataset, which is a synthesized city-scale 3D visual grounding benchmark, we annotated using 25 city-scale point clouds
23

---

Published as a conference paper at ICLR 2025
25m
Rasterized Map
(RGB)
(b)
(a)
(b)
(a) There is green soccer field next a large grassy and woodland area with many trees.
(a) This is a road intersection that connects three roads, the end of the Aldridge Road, next to a vacant lot.
(c) This is the access
road next to the parking lot, which located next the Parkside with several cars parked inside the road.
Figure A.9: Qualitative results for unknown objects the CityRefer dataset. The bounding boxes of resulting objects are drawn in red. In the query texts, the target object is marked in red, the landmark name is marked in blue, and the neighborhood statement is marked in green.
of STPLS3D (Chen al. 2022 dataset. Based on the annotated categories, we add text descriptions including specific numerical information (e.g. floor heights, coverage areas, distances, etc.).
A.7 POSSIBLE BROADER IMPACTS
City-scale visual grounding not well-studied in the literature. This technique may leak some privacy data to enable person to localize some sensitive targets. This could be addressed in the data preparation prevent this from appearing in the annotations.
A.8 LICENSE DATASETS
The CityRefer dataset is under MIT license while the STPLS3D dataset is under CC-BY-NC-SA 4.0 license. We will release our City Anchor dataset under the MIT license.
24


Reviews:
[{"review_id": "lZySq8JPIJ", "reviewer": "Reviewer_6qAx", "summary": "This paper proposes a multi-modal LLM for city-scale 3D visual grounding named CityAnchor. CityAnchor adopts a coarse-to-fine searching strategy. First, a 3D segmentation model generates the potential objects from point clouds. Next, in the coarse localization stage, a LLaVA generates the <RoI> token and a SAM generates the RoI heapmap indicating the candidates of the target object. At last, in the fine-grained matching stage, another LLaVA select the best candidate as the target object. Besides, this work also presents a new dataset for 3D visual grounding. Experiments show that CityAnchor outperforms the previous methods by a large margin.", "questions": "Please address the questions in the weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The experiment results are impressive.\n3. The proposed method is well motivated and technically sound.", "weaknesses": "1. As also mentioned in Sec.5, the efficiency of the proposed method is a major concern. It involves two LLMs and multiple forward passes in the fine-grained matching stage. Are there any possible solutions to improve the efficiency?\n2. How to choose the positive and the negative samples during the training of FMM?\n3. The objects are generated by a pretrained 3D segmentation model which can only recognize a closed set of targets. So I expect to see the generality of CityAnchor to unknown objects.\n4. Some important details are missing:\n- Line 157, what is $T_c$?\n- Line 138 & 209, how to determine whether an object has a landmark name?\n- Line 206, what is $c$ in $E^s_x$?\n- Line 232, which encoder is the concatenated feature vector fed into?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dKO4HfSJKo", "reviewer": "Reviewer_8f4q", "summary": "In the paper, the authors introduced CityAnchor, a 3D visual grounding method tailored for city-scale scenes. The CityAnchor is based on a two-stage multi-modal LLM. In the coarse stage, CityAnchor predicts candidate objects that match the query text descriptions on projected 2D maps derived from urban point clouds, allowing us to efficiently filter out redundant regions and concentrate on likely objects. Next, it performs fine-grained matching between text descriptions and the candidate objects to establish the final grounding results by encoding both text descriptions and object features into an LLM. To do that, the authors created a new dataset and evaluated the proposed method on both CityAnchor and CityRefer datasets to demonstrate the effectiveness of the proposed method on city-scale point clouds.", "questions": "I mentioned all comments including reasons and suggestions in the above sections. I recommend that the author will provide all the concerns, and improve the completeness of the paper. If the rebuttal period resolves the above-mentioned concerns, I will gladly raise my score. Also, there are little vague sentences and grammatical errors in the paper. I recommend that the author will revise the paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First of all, it is interesting that a highly intuitive idea is proposed to address two major issues in existing methods concerning how to improve the multi-modality feature extraction in the large-scale visual grounding and how to efficiently localize the object in a large-scale point cloud.\n\nAlthough it lacks technical novelty, it effectively implements the necessary modules to address two main problems. More specifically, a coarse location firstly finds possible regions on a projected 2D map of the point cloud while the fine-grained matching accurately determines the most similar region with the given test description.\n\nIn experimental section, the proposed method achieved SoTA results in city-level localization tasks, demonstrating its effectiveness. The ablation study is highly analytical for each level module and feature representation.", "weaknesses": "One concern is the validity of the coarse localization module. As shown Fig.4, I am curious whether CLM operates as intended. It appears to activate too many candidates, as if it were merely distinguishing between urban and non-urban areas. This raises doubts about whether it significantly aids in the efficient operating in the overall framework.\n\nExperiment analysis and description are not specific and descriptive. For example, a detailed explanation based on specific examples is needed to clarify the novel objects and novel descriptions, whether the reason they are not in the training data is due to an out-of-distribution (OOD) scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "teQPlp5jRt", "reviewer": "Reviewer_YNjM", "summary": "The paper addresses the challenge of object localization in city-scale 3D point clouds with a method called CityAnchor, which leverages a multi-modal large language model. Using a two-stage approach of coarse localization and fine-grained matching, CityAnchor efficiently and accurately identifies target objects in large urban scenes. Experimental results show that CityAnchor significantly outperforms existing methods in accuracy and speed, with promising applications in urban planning and geospatial analysis.", "questions": "It’s unclear how the method addresses dynamic objects in urban environments, such as moving vehicles or pedestrians, which could affect the reliability of grounding results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1: The method’s coarse-to-fine design is well-structured for efficient large-scale 3D visual grounding.\n\nS2: Experimental results show substantial and objective improvements over existing methods.\n\nS3: The proposed model demonstrates clear practical potential for applications in urban planning and geospatial analysis.", "weaknesses": "W1: The method seems heavily reliant on large language models, and the need for manual adjustment of the Region of Interest (RoI) threshold in the coarse localization stage suggests a lack of generalizability.\n\nW2: While the coarse-to-fine localization design is effective, it builds on existing multi-stage approaches without introducing fundamentally new concepts in 3D visual grounding, which suggests that the method's strengths lack a degree of innovation.\n\nOther:\nThe layout could be improved; for instance, all teaser images should be embedded as PDFs rather than PNGs to ensure the text within images is selectable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cBXIJ6eqXk", "reviewer": "Reviewer_P5JV", "summary": "The paper presents CityAnchor, a novel 3D visual grounding method designed for city-scale scenes, utilizing a two-stage multi-modality large language model (LLM). The first stage focuses on coarse localization of potential object regions based on text descriptions, while the second stage involves fine-grained matching to determine the best object matches. Specifically, \n- Coarse Localization Module (CLM)  takes a city-scale colored point cloud, projects it onto a 2D map, and uses the text description to regress possible regions of the target on this 2D map.\n- Fine-grained Matching Module (FMM) performs fine-grained comparisons between each candidate object identified by the CLM and the text description by predictiong the similarity between the text and each candidate object, and select the object with the highest similarity score.\nAdditionally, the paper introduces the CityAnchor dataset, a synthesized benchmark for 3D visual grounding. The method demonstrates significant improvements in accuracy and efficiency over existing techniques on the CityRefer and CityAnchor datasets.", "questions": "- Effectiveness of CLM: In Figure 4, it appears that CLM is not highly effective, as it predominantly highlights houses and roads. Could the authors provide additional results or offer an explanation for this behavior?\n\n- Prompt Engineering Requirements: It seems that users must invest effort in engineering their prompts to achieve accurate results, which may pose usability challenges. The authors could consider conducting a user study to gather prompts from a diverse set of users to assess the method’s robustness when different descriptions target the same object.\n\n- Progressive Prompting Approach: It would be intuitive to structure prompts in a coarse-to-fine manner—beginning with a broad query that identifies multiple potential candidates and gradually introducing constraints to precisely locate the target.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The paper introduces CityAnchor, a novel approach to 3D visual grounding specifically tailored for city-scale scenes. Its use of a two-stage multi-modality large language model (LLM) for both coarse localization and fine-grained matching is effective and advance in the field. The proposed approach effectively integrates spatial context and object features, showcases the ability in addressing the complexities of urban environments, setting it apart from existing methods that may not adequately handle such challenges.\n+ The paper provides a comprehensive evaluation of the model's performance, demonstrating significant improvements in accuracy compared to existing methods. Additionally, the introduction of the CityAnchor dataset as a synthesized benchmark for 3D visual grounding further enhances the quality of the research, providing a valuable resource for future studies.\n+ The paper is well-structured and easy to follow, clearly stating the objectives, methodology, and results of the research.", "weaknesses": "- Limitations of 2D Mapping: Projecting 3D space onto a 2D map often results in a loss of spatial information, particularly in cases of occlusions or overlapping instances. This pipeline, especially the CLM component, operates under the significant assumption that all objects are nearly flat, which inherently limits its applicability to more complex environments, such as indoor spaces or densely populated urban areas like NYC.\n\n- Dependence on Point Segmentation Model: The pipeline requires users to first segment the scene point cloud into objects using a pretrained 3D segmentation model. As a result, it heavily depends on the granularity and accuracy of this pretrained model, setting an upper performance limit for the proposed approach.\n\n- Justification for the Proposed Dataset: While introducing a new dataset is valuable, the necessity of the CityAnchor dataset for this task or the proposed method requires further clarification. It is essential to address whether the method’s performance would significantly degrade without this dataset, whether it enhances diversity or robustness, and if it offers any unique attributes that the CityRefer dataset lacks. Providing these details would strengthen the contribution of the dataset to the paper.\n\n- Inference time and hyperparameter tuning: These two limitations were briefly discussed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)