CityRefer: Geography-aware 3D Visual Grounding Dataset on  City-scale Point Cloud Data
Authors: Taiki Miyanishi
Fumiya Kitamori
Shuhei Kurita
Jungdae Lee
Motoaki Kawanabe
Nakamasa Inoue
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: 3D Visual Grounding
Geographic Information System
TL;DR: We introduce the CityRefer dataset, a new dataset for city-level visual grounding

Abstract: City-scale 3D point cloud is a promising way to express detailed and complicated outdoor structures. It encompasses both the appearance and geometry features of segmented city components, including cars, streets, and buildings that can be utilized for attractive applications such as user-interactive navigation of autonomous vehicles and drones. However, compared to the extensive text annotations available for images and indoor scenes, the scarcity of text annotations for outdoor scenes poses a significant challenge for achieving these applications. To tackle this problem, we introduce the CityRefer dataset for city-level visual grounding. The dataset consists of 35k natural language descriptions of 3D objects appearing in SensatUrban city scenes and 5k landmarks labels synchronizing with OpenStreetMap. To ensure the quality and accuracy of the dataset, all descriptions and labels in the CityRefer dataset are manually verified. We also have developed a baseline system that can learn encoded language descriptions, 3D object instances, and geographical information about the city's landmarks to perform visual grounding on the CityRefer dataset. To the best of our knowledge, the CityRefer dataset is the largest city-level visual grounding dataset for localizing specific 3D objects.

CityRefer: Geography-aware 3D Visual Grounding Dataset on City-scale Point Cloud Data
Taiki Miyanishi 1.3- Fumiya Kitamori Shuhei Kurita Jungdae Lee", Motoaki Kawanabe
Nakamasa Inoue
ATR, TTokyo Institute of Technology, 'RIKEN AIP
Abstract
City-scale 3D point cloud is promising way to express detailed and complicated outdoor structures. encompasses both the appearance and geometry features of segmented city components, including cars, streets, and buildings, that can be uti- lized for attractive applications such as user-interactive navigation of autonomous vehicles and drones. However, compared to the extensive text annotations avail- able for images and indoor scenes, the scarcity of text annotations for outdoor scenes poses significant challenge for achieving these applications. To tackle this problem, we introduce the CityRefer datase for city-level visual grounding. The dataset consists of 35k natural language descriptions of 3D objects appearing in SensatUrban [99] city scenes and 5k landmarks labels synchronizing with Open- StreetMap. To ensure the quality and accuracy of the dataset, all descriptions and labels in the CityRefer dataset are manually verified. We also have developed a baseline system that can learn encoded language descriptions, 3D object instances, and geographical information about the city's 's landmarks to perform visual ground- ing the CityRefer dataset. To the best of our knowledge, the CityRefer dataset is the largest city-level visual grounding dataset for localizing specific 3D objects.
Introduction
Advancements in urban 3D scanning technologies, such as unmanned aerial vehicle photogrammetry and mobile laser scanning, enable the creation of accurate and photorealistic large-scale 3D scene
datasets. Examples of such datasets include street-level datasets acquired by automobiles O 17
37
39
3 and city-level datasets acquired by aerial vehicles 19 36
44
52
660 However,
while level photorealistic 3D scans have become practical and applicable in various fields like autonomous driving and unmanned vehicle delivery, the technology to comprehend city scenes through human-interactiin linguistic representations is still in its early stages of development. The ability to ground linguistic expressions to urban components is highly desired for interactive and interpretable applications, such language-gun -uided autonomous driving and aerial drone navigation. Achieving this requires the development of a 3D visual grounding dataset based the city-scale point clouds, which presents a significant challenge.
3D Visual grounding is 3D and language task that involves localizing objects in 3D scenes based on textual referred expressions. Compared to its 2D visual grounding counterparts 23 29 3I 53), 3D visual grounding poses additional challenges. The expressions used in 3D visual grounding often require more information to localize object instances due to the rich context of 3D scenes, making the problem further complex. Recent studies have made remarkable progress in 3D visual
"qual contribution
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

City Dataset ia
5k+ labels for buildings and roads
320 Footpath:
Aylestone Road
TrafficRoad: Aylestone Road
35k+ natural language descriptions for city-level 3D visual grounding
the street from rod and white car on Belvoir Road that is parked near large live and Irout of yellow buikding.
The blue
The red car in the middle of the five cars that are parked diagonally the of Acrefield Drive.
The white buildingg with a gray roof, between two other similar looking buildings, Acrefleld Drive.just before Woodvale Lodge.
Figure 1: The CityRefer dataset for city-level 3D visual grounding.
grounding, focusing determining the precise position of 3D objects given natural language descriptions 77 I0,
However, many of these methods have been evaluated on 3D indoor datasets, which typically consist of point clouds of room scenes and labels for household objects 2 14]. Subsequently, Kolmet et al. [4 proposed district-based visual grounding dataset on --0 26. Nevertheless, the availability of 3D visual grounding datasets still limited, especially in the context of aerial city-level 3D point clouds. Therefore, aim to address this gap by creating a publicly available 3D visual grounding dataset based on aerial city-level 3D point clouds.
18
20
21
22
35
54
56
58
In this paper, we introduce the CityRefer dataset for city-level 3D visual grounding. Specifically, we provide 35k natural language descriptions to localize 3D objects in the SensatUrban [9] environment as well as 5k labels of objects such as buildings and roads. Three example descriptions of sample scene are shown in Figure ! As seen from the figure, city-level visual grounding is very challenging because a system is required to find objects from wide city area while understanding the description of the target object and the relationships between relevant objects. Although we used crowdsourcing scale up the annotation, we needed thoroughly filter out incorrect annotations by hand to finalize the dataset and ensure the quality of the annotations. The main contributions of the CityRefer dataset are summarized as follows.
1. We provide instance-wine wise segmentation masks for 5k objects including 1.8k landmark objects with their name labels. Examples include Kem River and Baker Street, Aylestone Road, and Belvoir Road. These labels were obtained from the spatial joint between SensatUrban and OpenStreetMap using our semi-automatic system (Section 3.1).
2. We provide 35k natural language descriptions for city-level visual grounding. These descriptions are manually attached using our interactive annotation system (Section 3.2).
3. We provide a baseline system that performs city-level 3D visual grounding. Because it is nontrivial to adapt previous visual grounding methods for our city-level dataset, we propose simple but effective method that narrows the search area to find the target object by using geographical information.

---

Table 1: Comparison of 3D visual grounding datasets. Ndesc : Number of natural language descrip-
tions. Lusss Average description length. Npoints Number of 3D points.
Human annot.
Dataset
Ndsse
Lidese
Area
Environment (Format)
Npoints
REVERIE
Yes
21,702
18.0
Rooms
Matterport3D (RGB)
Indoor
SUN-Spol
Yes
7,987
14.1
Rooms
SUN RGB-D 40
SUNRefer
17
Yes
38,495
14.1
Rooms
SUN RGB-D 40
Nr3D 2
Yes
41,503
11.4
Rooms
ScanNet (3D Scan) 14
242M
ScanRefer
Yes
51,583
20.3
Rooms
ScanNet (3D Scan) 14
242M
Outdoor
TouchDown
I
Yes 25,575
29.7
Roadside
Google Street View (RGB)
KITTI360Pose
24
No
43,381
7.6
Roadside
KITTI-360 (3D Scan) 26
1,000M
CityRefer (Ours)
Yes
35,196
26.3
City center
SensatUrban (3D Scan)
19
2,847M
2 Related Work
Visual Grounding Datasets for 3D Spaces. Over the years, there has been significant research interest in 3D visual grounding as summarized in Table [] We first introduce two types of 3D visual grounding datasets, each focusing on different level of the scene: the indoor and outdoor roadside. The 3D visual grounding dataset is created by annotating indoor or outdoor 3D datasets with linguistic descriptions.
(i) Indoor scene level. With the increasing availability of indoor 3D datasets B 5
14
46
40 40
several visual grounding datasets have been proposed to address the demand for 3D scene understanding. REVERIE 32 comprises 10,318 panorama images captured across 86 indoor scenes and a total of 4,140 target objects. This dataset also provides 21,702 language instructions with rich textual annotations for guiding virtual agents within indoor photorealistic scenes of Matterport3D [8] The SUN-Spot [30] and SUNRefer (27] datasets focus on object localization in single-view RGB-D images within indoor environments from the SUN RGB-D dataset 40) Both datasets provide detailed language annotations indicating the spatial extent and location of objects in the images including
object bounding boxes. Specifically, SUNRefer contains 38, 49599 language annotations for 7,699 objects in indoor RGB-D images. Nr3D 2 and ScanRefer 9 are standard 3D visual grounding datasets that are built on top of ScanNet [14, a real-world 3D scene dataset with extensive semantic annotations. However, these datasets mainly focus on the indoor visual grounding task.
(ii) Outdoor scene level. In recent years, a number of richly annotated outdoor 3D datasets have been |
created by scanning cities using sensors installed in cars and drones 66
17
25
371
4E 50 60)
While there have been efforts to annotate outdoor 3D datasets with language descriptions for 3D visual grounding, the availability of such datasets still limited compared to indoor ones. The TouchDown dataset [I] is aimed at text- eext-guided navigation and spatial reasoning using real-life visual observations. It contains 9,326 examples of instructions and spatial descriptions in the visual navigation environment drawn from Google Street View. KITTI360Pose outdoor 3D environment from the KITT1360 dataset [50). which provides nine static scenes obtained by LiDAR scans. notable that the linguistic descriptions of KITT1360Pose are automatically generated by sentence template with position description query pairs. While both TouchDown and KITTI360 datasets are based on the vehicle perspective and hence limited to the semantics from roadsides, our dataset is based on SensatUrban [191, which covers 3D semantics of the board city areas that are generated from aerial images by drones.
!Z4 is text- text-based position localization dataset in an
Learning Visual Grounding of 3D Scenes. To facilitate deeper understanding of 3D scenes through language, there have been many efforts to connect languages and the 3D visual world, including 3D dense captioning 12 47 555. 3D change detection (3 and 3D visual question 2 (59] Specifically, 3D visual grounding that aims to locate object in 3D space in response to a natural language query is the fundamental task the 3D and language field 2 48 Several approaches in the field of visual grounding use pre-computed 3D object detection or instance segmentation results, utilizing the point cloud features extracted from the corresponding 3D bounding boxes and segments 20 (66. However, the challenge of object recognition arises due to the low resolution of 3D data resulting from the reconstruction process. To overcome this limitation, recent
answering [ 4 15
3

---

CityRefer
SensatUrban
141 Building King's College Chape
3D point Geographical
OpenStreetMap
Landmark/rood labels
133, TraffioRoad Lensfield Road
Figure 2: Stage of dataset construction. We perform a spatial join between the 3D SrbsatUrban environment and the 2D OpenStreetMap. The CityRefer dataset contains 1,850 landmark/road labels and geographical data. Examples include the landmark name King's 's College Chapel and the road name Lansfield Road.
studies have proposed the integration of both 2D images and 3D data 51 pp
By combining derived from
the rich spatial information provided by 3D data with the detailed appearance 2D images, these hybrid approaches aim to enhance robustness in the context of visual grounding. Furthermore, there have been studies proposing methods that integrate 3D visual captioning and grounding, where both models are learned simultaneously to achieve synergistic effects 7 10 These current 3D visual grounding methods mainly rely on two widely used indoor 3D visual grounding remote sensing data but were limited For these reasons, the performance of 3D visual grounding on outdoor 3D datasets remains unexplored. One of the initial studies 24 attempted to identify regions within the 3D point cloud based on textual queries on city-level 3D datasets but were limited to artificial language descriptions and did not fully use the geographic information 3D map despite the sparsity of city-level 3D point cloud data. In contrast, method uses geographic information from 3D maps to achieve accurate city-level 3D visual grounding.
datasets 2 1.) Several studies proposed visual grounding
to 2D images 44Il 57].
3 Dataset Construction
The CityRefer dataset consists of 1) instance-wise segmentation masks each with label and geo- graphical information, and 2) natural language descriptions for visual grounding. The 3D environment is the SensatUrban 19), which consists of photogrammetric point clouds of two UK cities covering 6 km2 of the city landscape. Semantic segmentation masks are provided with the envi- ronment, and our study refines them to instance-level masks for visual grounding. The annotation proceeds in two stages: semi-automatic generation of instance-wise segmentation masks (Stage I) and manual language annotation (Stage 2).
3.1 Stage 1: Semi-automatic Generation of Instance-wise Segmentation Masks
The goal this stage is to generate segmentation masks for each 3D object as well as to attach labels of names with geographical data (longitude, latitude, and elevation) to each object. To achieve this, we perform spatial join between the 3D environment and OpenStreetMap, from which we can obtain names and locations. Figure shows an example result of this stage. As shown, regions of King's 's College Chapel and Lensfield Road are visualized precisely. The procedures are described in detail below.
Georeferencing. The georeferencing is performed in the following three steps. First, given block of the 3D city, we create a top view image as shown in Figure 3h, where the image size fixed to 2048 by 2048 pixels. Second, manually choose ten points of interest, such corner and landmark points, that are clearly visible in OpenStreetMap. In this step, we also extract an image of the 2D map as shown in Figure Bc. Note that the coordinates of the ten points are manually annotated on
We use 34 blocks (scenes) of Cambridge and Birmingham from SensatUrban.

---

SensatUrban (3D scene)
Result of georeferencing
OpenStreetMap (2D map)
Figure 3: Georefereneing of 3D environment and 2D map.
both images. Finally, we compute the geometric transformation between the two images by using the transformation function from the scikit-image library. We manually tune the syper-parameter of transformation by visually verifying the results of the georeferencing. Here, the ten points of interest also updated and tuned needed. Figure shows example result in which the 3D scene and the 2D map are precisely joined.
Generating instance-wise segmentation masks. Given semantic segmentation masks with respect to 13 object categories, we refine them to instance-level masks. We divide the categories into three groups as shown Table 2 For Group 1, we directly use geographic information obtained from OpenStreetMap to create filters for each instance. For example, for segment of Lansfield Road in Opp, we create filter in the 3D scene based on the result of the georeferencing. For Group 2. apply the DBSCAN algo- rithm, a clustering method, to 3D points. difficult obtain accurate boundaries between instances by clustering because ob-
Table 2: Three groups of categories.
Grp. Method
Categories
Ground, High Vegetation, Building, Bridge, Rail, TrafficRoad, Footpath, Water
jects in this group are small and located sep- arately. For Group 3, used YOLOv7 to detect cars. The detection results are not per- fect but are sufficient for creating the dataset. Note that we ask annotators to only use cor- rectly segmented instances the next stage.
Filtering
Wall, Parking. StreetFurniture, Bike
Clustering
3
Detection
Car
3.2 Stage 2:
Manual Language Annotation
The goal of this stage to collect natural language descriptions that describe the target object in 3D scene for visual grounding. We prepare two interfaces: the language annotation interface and the quality control interface. The former is used to collect descriptions, and the latter is used to verify whether the collected descriptions accurate. Below present details of the interface implementation, the language annotation task, and the quality control procedure.
Interface. Figure 4 shows the interface used for language annotation and verification. To interactively show 3D scenes to annotators, we implement the interface with Potree 38], an open- source WebGL- based point cloud renderer that can process large-scale point cloud data. In the figure, the target object is highlighted in red. We provide interactive features such as zooming and panning as well as the labels each object. The annotators can view the regions of each object by clicking the labels.
Language annotation. We ask annotators to describe the target object with the following instructions.
You will see geographic objects in different 3D outdoor scenes. Please describe the objects in the 3D scene so that the objects can be uniquely identified on the basis of your descriptions and what you observed when submitting your responses. Some of the information below (a combination is preferred) should be included in your description:
The object's appearance, e.g., colors, shapes, materials.
The object's 's location in the scene, e.g., the parking lot is in front of Birmingham Library.
5

---

geographic objects the scene!
Find an object corresponding to description
Thonk bry Bor medbley
Thin for
Phesse
Click
zoom up
Targeted object
Other objects
top view!
Figure 4: Stage 2 of dataset construction. We ask annotators to describe the target object in the 3D scene so that the object can be uniquely identified based on the description. With our interface, annotators can see each object's region. Functions such as zooming and panning are also available.
The spatial relation between this object and other objects, e.g., this building the second one from the left.
Imagine you and your friend live in a certain city, and you would like ask your friend to find geographic object in that city. Since there may be many similar geographical objects, the description of the object should be as unique as possible.
To efficiently collect data, we show at most three target objects to annotators in a 3D scene. We also ask annotators to report segmentation errors and incorrect labels via another free-form text box and checkboxes.
Quality control. To ensure the quality of the annotations, we ask another set of annotators manually perform visual grounding with the following instructions.
You will see 1-3 descriptions for different geographic objects. Please choose the geographical object that best matches the description from the 3D scene (candidate objects are circled in red.)
Based on the results, we remove incorrect descriptions. We measured the accuracy of the annotation with visual grounding again, and the correct response rate was 91.53%.
We used Amazon Mechanical Turk (MTurrk) for annotation and quality control. There were 918 hours of work with total cost of $9,699 (the estimated hourly rate paid was $10.56). The total number of participating workers was 282.
Dataset Statisties
This section provides basic statisties of the CityRefer dataset in comparison to ScanRefer [9] (an indoor dataset) and KITTI360Pose [4 (a roadside dataset), We summarize the statistics in Table and discuss them detail below.
Target objects and descriptions. The CityRefer dataset consists of 35,196 descriptions, each of which describes an object in the 3D scenes. The target objects fall under one four categories: Car, Building, Ground, and Parking. The distribution is shown in Figure 5 (left. Note that all of them unnamed objects; that means that landmark objects such as famous buildings and roads, which can be identified by their names on OpenStreetMap, are excluded from the target objects. The distribution of description lengths is presented in Figure 0 along with those of ScanRefer and KITTI360Pose for comparison. Here, KITTI360Pose exhibits a sharp length distribution of the descriptions because they automatically generated from database with templates, e.g., the pose is south of gray road. In contrast, our dataset provides moderate- length descriptions to perform visual grounding. To the best of our knowledge, the CityRefer dataset the first large-scale dataset with manually annotated descriptions of city-level 3D scenes.

---

Table 3: Comparison of datasets. Area:
Type of scanned area. Manual: Whether annotation
manual or not. Geo data: Availability of geographical data. Neesc: Number of descriptions. Nobj:
Number of objects. M Number of landmark objects. V: Vocabulary size.
Dataset
Area
Manual
Geo data
Ndesc
Nobj
Mandmark
V
ScanRefer
Indoor
51,583
11,046
0
4,197
KITTI360Pose 24
Roadside
43,381
6,800
0
41
CityRefer (Ours)
City center
35,196
5,866
1,850
6,683
Bridge 0.8% 1.0% Ground 1.7%
Water 0.8% Vegetation 0.3%
CityReeer ScanRefer
Parking 3.6%
Ground
Car 4731%
TrafficRoad 38.8%
KITTIPose360
Density
0.3-
Footpath 21.2%
35,196 descriptions
1,850 landmarks
0.2
0.14
Building 41.4%
Building 35.4%
20
60 80
100 120 140
Number of words description
Figure 5: Object category distribution.
Figure 6: Description lengths
LooT
dew
Trown house
front indind
Street eete
cree
car
corner
next
Jay
FOw aieeend black
tee whi
temp righti
cal
stree
g green
Front Foad
buildin
Lann
Cour
right
LATE
area HALL
Tronttii Teo
street
par
King
Lrggnnee
Hot
car
one
bur Buildng
buildi
right Se
Side
roade
white whiterm
dark lot regbehind
road tra
lare gray blueeueee nest
lef
Blan
kings
Je aaree
grass Fronddd
soligge
geessrous
tree
rastaruuaa
blue
building
behind side
who rou
or gad
broad
Building
(b) Car
Ground
(d) Parking
Figure 7: Visualization of word distributions.
Landmark objects. There are 1,850 landmark objects with their names across seven categories: TrafficRoad, Building, Footpath, Ground, Rail, Bridge, Water, and Vegetation. The distribution shown in Figure S (right). Examples include Senate House Hill, Wellhead Lane, and Parkside Police Station. They are used to refer to target objects, e.g., the gray rectangular building to the right of the parking lot next to St. John's 's College Chapel.
Words. The vocabulary size of the CityRefer dataset 6,683. Figure 7 shows the visualization of word distributions. We can observe that words that specify object locations in the following three categories are frequently used: 1) colors e.g., red, blue, gray, 2) relative positions e.g. right, left, across, and 3) nearby objects tree, street, road. Compared with the template-based position descriptions of KITT1360Pose, our free-form descriptions contain more natural descriptions.
Experiments
Finally, we conducted experiments on the CityRefer dataset, focusing on two tasks: instance seg- mentation and visual grounding. To ensure comprehensive evaluation, we divided the dataset into three subsets: training, validation, and testing. The data split is summarized in Table 4 providing the number descriptions (NVesc), objects (Nob)), and landmark objects (Nimark).
5.1 City-level instance segmentation
This experiment focuses on evaluating the segmentation performance, The objective is to generate segmentation masks with corresponding category labels for each object using 3D point clouds as input.

---

Table 4: Dataset split for training, val- idation, and testing
Table 5: Instance segmentation performance.
Target
AP
APsu
AP25
mRec
mRecse
mRec2s
Subset
Ndeec
Nobj
Minack
Ground
19.9
39.8
52.9
28.0
47.2
60.6
Train
23,586
3,931
1,106
Building
3.7
12.2
24.4
7.2
16.8
26.5
Val
5,934
989
243
Parking
5.9
17.0
48.3
15.1
30.8
59.0
Test
5,676
946
501
Car
35.3
55.0
69.4
42.2
58.9
70.9
Total
35,196
5,866
1,850
Average
16.2
31.0
48.7
23.1
38.4
54.2
Method. We provide PyTocc--base implementation of baseline method using the SoftGroup++ model E5]. Due to the larger number of 3D points (2,847M points) compared to previous datasets for randomly sample 2% of the 3D points and feed them into the model.
3D instance segmentation, To provide transparency and reproducibility, we summarize the training details along with manually tuned hyperparameters in the Appendix.
Evaluation metrics. We use average precision (AP) and mean recall (mRec) as our primary metrics. In addition, we report the top-N AP and recall, with N set to 50 and 25. Specifically, we refer to these metrics 50 AP25, mRecsp. and mRec25-
shows the resulting performance for the four target categories. We see that cars are relatively easier to segment compared to the segmentation of other objects. However, the overall segmentation performance remains modest. This is because the CityRefer dataset involves many similar instances placed near each other.
Results. Table
5.2 City-level visual grounding
In this experiment, we assess the visual grounding performance, which involves locating the target object within the 3D point clouds based on a given natural language description. To evaluate the performance of both instance segmentation and visual grounding separately, we assume that ground- truth instance segmentation masks provided. We provide ten candidate answers, including the correct answer, for each description.
Method. We modified the InstanceRefer model 56 to enable city-level visual grounding. The modified model follows four-step process to perform visual grounding. First, we extract object features from each candidate instance by applying four-layer sparse convolution network [[6] with average pooling layer to input 3D points. Second, we use one-layer bi-directional GRU (BiGRU) [3] to extract language features from the given description. Third, object features and language features concatenated and fed into another BiGRU obtain visual-language features that represent the relationship between the description and the 3D objects. Finally, we compute scores using two-layer MLP. These scores indicate the likelihood of a candidate instance being the correct grounding for the given description. We use the cross-entropy loss for training. For more comprehensive information regarding the model architecture and training process, please refer Appendix.
Evaluation metrics. We assess the accuracy of our predictions by comparing their intersection over union (IoU) with the ground truth values. Specifically, we focus on positive predictions that exhibit higher loU with the ground truth instances than a certain threshold k. We use the Acc kloU metric, which is commonly used in the field of indoor 3D visual grounding research For our experiments, we set the threshold value k for IoU to 0.25.
Results. Table 6 presents the visual grounding performance for the four target categories, comparing the results of Random (random guess) and Bascline (the method described above). Baseline + Land incorporates landmark features extracted from 3D points and the names of landmark objects, in addition to the object and language features. We see that the baseline methods significantly perform better than the random guess but there is still large gap between the system performance and human performance (Acc. 0.950). This demonstrates that city-level visual grounding is challenging task despite advances in learning technology. Developing large 3D-vision-language models for city-level visual grounding would be a potential future research direction. Figure illustrates qualitative examples of 3D visual grounding when incorporating landmark information.
8

---

Table 6: City-level 3D visual grounding performance.
Method
Building
Car
Ground
Parking
Overall
Random
0.103 + 0.008
0.103 = 0.006
0.091
0.017
0.094 + 0.010 0.835 + 0.034
0.101 = 0.005
Baseline
0.255 + 0.005
0.282 = 0.010
0.477
0.024
0.312 = 0.006
Bascline + Land
0.255 + 0.008
0.298 + 0.007
0.489 0.009
0.853 + 0.020
0.320 + 0.005
Humans
0.947
0.956
0.937
0.945
0.950
Building
Car
Ground
Parking
Eg
building Straffordshire Garden directly ffront drivefrom Staffordshire Street with pickup truck and a ladder lift in front
white parked diagonally the corner of parking lot, betwren another white bluish gray front an shaped Brown building with a gray roof Ashcroft
The square green piece of ground left of Emmanuel College Chapel and below Hall building right Andrew's Street.
The large parking lot on Livingstone Road front large white building and has large van handicup parking.
Figure 8: Qualitative examples.
The examples demonstrate how the inclusion of landmark information aids in accurately identifying the target objects. This observation further reinforces incorporating landmark information is promising way to improve the city-level visual grounding performance.
6 Conclusion
We have introduced the CityRefer dataset, dataset for city-level visual grounding tasks. This dataset offers a comprehensive 3D environment where instance-wise segmentation masks, along with geographic data and labels, are provided. The creation of this environment involved performing spatial joint between the SensatUrban environment and the OpenStreetMap, resulting in rich and realistic urban setting. We also provided 35k natural language descriptions to locate objects as well as bascline systems for instance segmentation and visual grounding.
Limitations and future work. In this work, we tackled visual grounding in the 3D environment of real cities. However, there are still gaps between the real world and the 3D environment. In particular, because all 3D scenes are static, the vocabulary of the CityRefer dataset is limited to words that specify static objects. To achieve extend the dataset to include dynamic environments, where objects and scenes can change over time. Furthermore, while our visual grounding system has demonstrated promising results, there still room for improvement. The main purpose of this work is to introduce new dataset and indeed it nontrivial to apply visual grounding systems for our dataset; however, developing more accurate systems would be worth pursuing the future. We hope that the CityRefer dataset promotes further research and development as well as discussion on geography-awan learning technologies.
comprehensive and real-world applicability, would be necessary
Broader impacts. Although the dataset is constructed on the basis of publicly available 2D map (OpenStreettMap), visual grounding in general may result in privacy issues or racial and gender biases. The natural language descriptions we collect are carefully checked so that they do not include private information or offensive text.
Acknowledgements
This work was supported by JST PRESTO JPMJPR22P8 and JPMJPR20C2, and JSPS KAKENHI
22K12159, 12159.

---

References
[1] A. Abdelreheem, K. Olszewski, H.-Y. Lee, P. Wonka, and P. Achlioptas. Scanents3d: Exploiting marase-to-3d-objes correspondences for improved visio-linguistic models in 3d scenes. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024.
[2] P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. J. Guibas. ReferIt3D: Neural listeners for fine- iieeggrined 3d object identification in real-world scenes. In Proceedings of the Springer European Conference on Computer Vision (ECCV), 2020.
[3] Armeni, S. Sax, Zamir, and Savarese. Joint 2d-3a 1a--3d-semmm semmenmatic data for indoor scene understanding, arXiv preprini 17Xiv:1702.01 105, 2017.
[4] D. Azuma, T. Miyanishi, S. Kurita, and M. Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 191999
June 2022.
[5] G. Baruch, Z. Chen, A. Dehghan, T. Dimry, Y. Feigin, P. Fu, T. Gebauer, B. Joffe, D. Kurz, A. Schwartz, and E. Shulman. ARKitscenes a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. In Proceedings of Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS D&B), 2021.
[6] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall. Se- manticKITTI: A dataset for semantic scene understanding of LiDAR sequences. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 9297-9307, 2019.
[7] D. Cai, L. Zhao, J. Zhang, L. Sheng, and D. Xu. 3djog: A unified framework for joint dense captioning and visual grounding on 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16464-16473, 2022.
[8] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niebner, M. Savva, S. Song, A. Zeng, and Y. Zhang. Matterport3D: Learning from RGB-D data in indoor environments. In Proceedings of the International Conference on 3D Vision (3DV), pages 667-676, 2017.
[9] D. Z. Chen, A. X. Chang, and M. Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In Proceedings of the Springer European Conference on Computer Vision (ECCV), 2020.
[10] D. Z. Chen, Q. Wu, M. Nießner, and A. X. Chang. D3net: A speaker-listener architecture for semi- supe-super dense captioning and visual grounding rgb-d scans. Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[11] H. Chen, A. Suhr, D. Misra, N. Snavely, and Y. Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street enviromments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[12] Z. Chen, A. Gholami, M. Nießner, and A. X. Chang. Scan2cap: Context-aware dense captioning in rgb-d seans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[13] Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. In Neural Information Processing Systems (NeurIPS) Workshop on Deep Learning, 2014.
[14] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly- annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[15] Y. Etesam, I.. Kochiev, and X. Chang. 3dvqa: Visual question answering for 3d environments. In IEEE Conference on Robots and Vision (CRV), pages 233-240, 2022.
10

---

[16] B. Graham, M. Engelcke, and L. van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[17] T. Hackel, N. Savinov, L. Ladicky, J. D. Wegner, K. Schindler, and M. Pollefeys. Seman- tic3D.Net: A new large-scale point cloud classification benchmark. ISPRS Annals of Pho- togrammetry. Remote Sensing & Spatial Information Sciences, 2017.
[18] D. He, Y. Zhao, J. Luo, T. Hui, S. Huang, A. Zhang, and S. Liu. Transrefer3d: Entity-and- relation aware transformer for fine-grained 3d visual grounding. In Proceedings of the ACM International Conference Multimedia (ACMMM), pages 2344-233 2021.
[19] Q. Hu, B. Yang. S. Khalid, W. Xiao, N. Trigoni, and A. Markham. Sensaturban: Learning semantics from urban-scale photogrammetric point clouds. Springer International Journal of Computer Vision (IJCV), 130:316-333 643,,,33444 2022.
[20] P.-H. Huang, H.-H. Lee, H.-T. Chen, and T.-L. Liu. Text-guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 35, pages 1610-1618, 2021.
[21] S. Huang, Y. Chen, J. Jia, and L. Wang. Multi-view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15524-15555 15533, 2022.
[22] A. Jain, N. Gkanatsios, I. Mediratta, and K. Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In Proceedings of the Springer European Conference on Computer Vision (ECCV), pages 417-433, 2022.
[23] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. ReferltGame: Referring to objects in photographs of natural scenes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
[24] M. Kolmet, Q. Zhou, A. OSep, and L. Leal-Taixé. Text2pos: Text-to-point-cloud cross-modal localization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6687-6696, 2022.
[25] X. Li, C. Li, Z. Tong, A. Lim, J. Yuan, Y. Wu. J. Tang, and R. Huang. Campus3D: A photogrammetry point cloud benchmark for hierarchical understanding of outdoor scene. In Proceedings of the ACM International Conference on Multimedia (ACMMM), 2020.
[26] Y. Liao, J. Xie, and A. Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Trunsactions on Paltern Analysis and Machine Intelligence (TPAMI), 45:3292-33111 1000. 2022.
[27] H. Liu, A. Lin, X. Han, L. Yang, Y. Yu, and S. Cui. Refer-it-in-rgbdi A bottom-up approach for 3d visual grounding in rgbd images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6032-608 00226041,220 2021.
(28) X. Ma, S. Yong, Z. Zheng, Q. Li, Y. Liang, S.-C. Zhu, and S. Huang. Sqa3d: Situated question answering in 3d scenes. In International Conference on Learning Representations (ICLR), 2023.
[29] Mao, J. Huang, A. Toshev, O. Camburu, A. Yuille, and K. Murphy. Generation and Com- prehension of Unambiguous Object Descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[30] C. Mauceri, M. Palmer, and C. Heckman. Sun-spot: An rgb-d dataset with spatial referring expressions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2019.
[31] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, Hockenmaier, and S. Lazebnik. Flickr30k Entities: Collecting Region-to-Phreen Correspondences for Richer -e--eentence Models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
11

---

[32] Y. Qi, Q. Wu, P. Anderson, X. Wang, W. Y. Wang, C. Shen, and A. van den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[33] Y. Qiu, S. Yamamoto, R. Yamada, R. Suzuki, H. Kataoka, K. Iwata, and Y. Satoh. 3d change localization and captioning from dynamic scans of indoor scenes. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision (WACV), pages 1176-1185, 2023.
[34] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. M. Turner, E. Undersander, W. Galuba, A. Westbury, A. X. Chang, M. Savva, Y. Zhao, and D. Batra. Habitat- maateattmatterpott 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AL. In Proceedings of Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS D&B), 2021.
[35] Roh, K. Desingh, A. Farhadi, and D. Fox. Languagerefer: Spatial-language model for 3d visual grounding. In Proceedings of the Conference on Robot Learning (CoRL), pages 1046-1056, 2022.
[36] Rottensteiner, G. Sohn, J. Jung, M. Gerke, C. Baillard, S. Benitez, and U. Breitkopf. The ISPRS benchmark on urban object classification and 3D building reconstruction. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 1-3 (2012). Nr. 1, 1(1):293-22) 2988 2012.
[37] X. Roynard, J.-E. Deschaud, and F. Goulette. Paris-Lille-3D: A large and high-quality ground- truth urban point cloud dataset for automatic segmentation and classification. The International Journal of Robotics Research, 37(6):545-557, 2018.
[38] M. Schütz al. Potree: Rendering large point clouds in web browsers. Technische Universität Wien, Wiederi, 2016.
[39] A. Serna, B. Marcotegui, F. Goulette, and J.-E. Deschaud. Paris-rue-made database: 3d mobile laser scanner dataset for benchmarking urban detection, segmentation and classification methods. In Proceedings of the International Conference on Pattern Recognition, Applications and Methods (ICPRAM), 2014.
[40] S. Song, S. P. Lichtenberg, and J. Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[41] Y. Sun, S. Feng, Li, Ye, J. Kang, and X. Huang. Visual grounding in remote sensing images. In Proceedings of the ACM International Conference on Multimedia (ACMMM), page 404 412, 2022.
[42] W. Tan, N. Qin, L. Ma, Y. Li. Du, G. Cai, K. Yang, and J. Li. Toronto-3D: A large-scale mobile LiDAR dataset for semantic segmentation of urban roadways. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 202-203 2020.
[43] B. Vallet, M. Brédif, A. Serna, B. Marcotegui, and N. Paparoditis. Terramobiliza/qquulss urban point cloud analysis benchmark. Elsevier Computers & Graphics, 49: 19:1133-3, 2015.
[44] N. Varney, V. Asari, and Q Graehling. DALES: A large-scale aerial LiDAR data set for semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pages 186-187, 2020.
[45] T. Vu, K. Kim, T. M. Luu, X. T. Nguyen, and C. D. Yoo. Softgroup for 3d instance segmentation on 3d point clouds. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[46] J. Wald, A. Avetisyan, N. Navab, F. Tombari, and M. Niessner, Rio: 3d object instance re- localization changing indoor environments. Proceedings of the IEEF/CVE International Conference Computer Vision (ICCV), 2019.
12

---

[47] H. Wang, C. Zhang, J. Yu, and W. Cai. Spatiality guiiteedyggudddddeeg transformer for 3d dense captioning on point clouds. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 1393-1400, 2022.
[48] Y. Wu, X. Cheng, R. Zhang, Z. Cheng, and J. Zhang. Eda: Explicit text-decoupling and dense alignment for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 19231-19242, 2023.
[49] Xia, A. R. Zamir, Y. He, A. Sax, J. Malik, and S. Savarese. Gibson Env: real-world perception for embodied agents. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[50] J. Xie, M. Kiefel, M.-T. Sun, and A. Geiger. Semantic instance annotation of street scenes by 3d to 2d label transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[51] Z. Yang. S. Zhang, L. Wang, and J. Luo. Sat: 2d semantics assisted training for 3d visual grounding. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), 2021.
[52] Z. Ye, Y. Xu, R. Huang, X. Tong, X. Li, X. Liu, K. Luan, L. Hoegner, and U. Stilla. LASDU: A large-scale aerial LiDAR dataset for semantic labeling in dense urban areas. ISPRS International Journal of Geo- leo-Information, 9(7):450, 2020.
[53] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling Context in Referring Expressions. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
[54] Z. Yuan, X. Yan, Z. Li, X. Li, Y. Guo, S. Cui, and Z. Li. Toward explainable and fine-grained 3d grounding through referring textual phrases. arXiv preprint arXiv:2207.0182// 2022.
[55] Z. Yuan, X. Yan, Y. Liao, Y. Guo, G. Li, S. Cui, and Z. Li. X-trans2cap: Cross-modal knowledge transfer using transformer for 3d dense captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8563-8573, 2022.
[56] Z. Yuan, X. Yan, Y. Liao, R. Zhang, S. Wang, Z. Li, and S. Cui. Instancerefer: Cooperative holistic understanding for visual grounding on point clouds through instance multi-level contex- tual referring. In Proceedings of the IEEE/CVF International Conference Computer Vision (CVPR), pages 1791-1800, 2021.
[57] Y. Zhan, Z. Xiong, and Y. Yuan. Rsvg: Exploring data and models for visual grounding on remote sensing data. IEEE Transactions on Geoscience and Remote Sensing (TGRS), 61:1-13, 2023.
[58] L. Zhao, D. Cai, L. Sheng, and D. Xu. 3dvg-transformer: Relation modeling for visual grounding on point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9928-9937, 2021.
[59] L. Zhao, D. Cai, J. Zhang, L. Sheng, D. Xu, R. Zheng. Y. Zhao, Wang, and X. Fan. Towards explainable 3d grounded visual question answering: new benchmark and strong baseline. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2022.
[60] S. Zolanvari, S. Ruano, A. Rana, A. Cummins, R. E. da Silva, M. Rahbar, and A. Smolic. DublinCity: Annotated LiDAR point cloud and its applications. Proceedings of the British Machine Vision Conference (BMVC), 2019.
13


Reviews:
[{"review_id": "zASW9YatiZ", "reviewer": "Reviewer_vjm6", "summary": "", "questions": "", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. As the text annotations for outdoor scenes are challenging to obtain, they devise a semi-automatic instance mask generation method and manually accompany it with language annotation.\n2. They provide a detailed analysis of the data statistics. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "m5dZ1fbdr2", "reviewer": "Reviewer_hP2w", "summary": "", "questions": "", "limitations": "1. As the authors wrote, the objects are restricted to static ones and scenes. By incorporating descriptions of dynamic objects like movement direction, speed, and actions, a broader research scope could potentially be opened up.\n\n2. Reasoning on a city-level point cloud requires significant memory consumption, which makes models strike a balance between their range of interest and the sparsity of the point clouds.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. CityRefer distinguishes itself from other outdoor datasets through its larger size, more diverse cityscape scenes, and the inclusion of natural language descriptions. \n2. The process of constructing the dataset is systematically organized, incorporating rigorous quality control measures.n the first phase of the data construction pipeline, the result is 2D semantic masks derived from a top-down view, rather than 3D masks, as can be observed in the provided figures. Additionally, the authors are capable of providing 3D bounding box labels, making the dataset applicable to fields like autonomous driving.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "3dHlfgPbe6", "reviewer": "Reviewer_hsUT", "summary": "", "questions": "", "limitations": "Creating city-level datasets might inadvertently capture sensitive or private information. Although the authors have manually filtered the dataset, the process's thoroughness in addressing privacy is not detailed.", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The introduction of the CityRefer dataset addresses a clear gap in city-level 3D visual grounding, making it a significant contribution to the field.\n* The ability to linguistically comprehend city scenes has broad applications, particularly in emerging areas like autonomous driving and aerial drone navigation.\n* The commitment to release code, data, and trained models enhances the paper's relevance by providing tangible resources for others to build upon.\n* It's interest to align a city-scale point cloud with openStreetMap and uses annotations from OSM.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "DpobvgzJTk", "reviewer": "Reviewer_vMMJ", "summary": "", "questions": "", "limitations": "The authors have addressed the limitations and broader impacts in Sec. 6.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- This paper proposes a new 3D visual grounding dataset for city-level tasks. The dataset first utilizes SensatUrban [19] and OpenStreetMap to perform the spatial join. Therefore, the annotation of building and street names can be directly mapped on the 3D dataset.\n\n- The authors refine the semantic masks in SensatUrban to instance-level masks. They divide semantic categories into three groups, using filtering, clustering (DBSCAN), and detection (YOLOv7) to process different groups separately.\n\n- To annotate the language description for target objects, the authors design an interface for annotation and verification. The quality control can effectively remove incorrect descriptions for verification. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "egpptlCnOK", "reviewer": "Reviewer_eXYx", "summary": "", "questions": "", "limitations": "The paper does a nice job of discussing the limitation that the data is static (unlike actual cities).", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The paper does a nice job of positioning the gathered data relative to existing work, e.g. using summary statistics.\n* The data seems interesting and the data collection is definitely a significant task, i.e. others would probably not collect this data as a side-project for a methods paper.\n* The manual verification of the data is helpful.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Five experts reviewed this paper with all accepted recommendations. The area chairs agree that this work makes a very important contribution by introducing a tool for 3D Captioning. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper.
Decision: Accept (Poster)