SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Method. We modified the InstanceRefer model [56] to enable city-level visual grounding. The modified model follows a four-step process to perform visual grounding. First, we extract object features from each candidate instance by applying a four-layer sparse convolution network [16] with an average pooling layer to input 3D points. Second, we use a one-layer bi-directional GRU (BiGRU) [13] to extract language features from the given description. Third, object features and language features are concatenated and fed into another BiGRU to obtain visual-language features that represent the relationship between the description and the 3D objects. Finally, we compute scores using a two-layer MLP. These scores indicate the likelihood of a candidate instance being the correct grounding for the given description. We use the cross-entropy loss for training. For more comprehensive information regarding the model architecture and training process, please refer to the Appendix.
Evaluation metrics. We assess the accuracy of our predictions by comparing their intersection over union (IoU) with the ground truth values. Specifically, we focus on positive predictions that exhibit a higher IoU with the ground truth instances than a certain threshold  $k$ . We use the Acc@kIoU metric, which is commonly used in the field of indoor 3D visual grounding research [9]. For our experiments, we set the threshold value  $k$  for IoU to 0.25.
Results. Table 6 presents the visual grounding performance for the four target categories, comparing the results of Random (random guess) and Baseline (the method described above). Baseline + Land incorporates landmark features extracted from 3D points and the names of landmark objects, in addition to the object and language features. We see that the baseline methods significantly perform better than the random guess but there is still a large gap between the system performance and human performance (Acc. = 0.950). This demonstrates that city-level visual grounding is a challenging task despite advances in learning technology. Developing large 3D-vision-language models for city-level visual grounding would be a potential future research direction. Figure 8 illustrates qualitative examples of 3D visual grounding when incorporating landmark information.
Table 6: City-level 3D visual grounding performance.
Method	Building	Car	Ground	Parking	Overall
Random	0.103 ± 0.008	0.103 ± 0.006	0.091 ± 0.017	0.094 ± 0.010	0.101 ± 0.005
Baseline	0.255 ± 0.005	0.282 ± 0.010	0.477 ± 0.024	0.835 ± 0.034	0.312 ± 0.006
Baseline + Land	0.255 ± 0.008	0.298 ± 0.007	0.489 ± 0.009	0.853 ± 0.020	0.320 ± 0.005
Humans	0.947	0.956	0.937	0.945	0.950
Parking
Figure 8: Qualitative examples.
The building at the end of Straffordshire Garden directly in front of the drivefrom Staffordshire Street with a red pickup truck and a ladder lift in the drive in front of it.
A white van parked diagonally in the corner of a parking lot, between another white van and a bluish gray car, in front of an L-shaped Brown building with a gray roof off Ashcroft Grove
The square green piece of ground left of Emmanuel College Chapel and below Hall building and right of Saint Andrew's Street.
The large parking lot on Livingstone Road in front of a large white building and has a large white van in the handicap parking.
The examples demonstrate how the inclusion of landmark information aids in accurately identifying the target objects. This observation further reinforces incorporating landmark information is a promising way to improve the city-level visual grounding performance.
We have introduced the CityRefer dataset, a dataset for city-level visual grounding tasks. This dataset offers a comprehensive 3D environment where instance-wise segmentation masks, along with geographic data and labels, are provided. The creation of this environment involved performing a spatial joint between the SensatUrban environment and the OpenStreetMap, resulting in a rich and realistic urban setting. We also provided 35k natural language descriptions to locate objects as well as baseline systems for instance segmentation and visual grounding.
Limitations and future work. In this work, we tackled visual grounding in the 3D environment of real cities. However, there are still gaps between the real world and the 3D environment. In particular, because all 3D scenes are static, the vocabulary of the CityRefer dataset is limited to words that specify static objects. To achieve more comprehensive and real-world applicability, it would be necessary to extend the dataset to include dynamic environments, where objects and scenes can change over time. Furthermore, while our visual grounding system has demonstrated promising results, there is still room for improvement. The main purpose of this work is to introduce a new dataset and indeed it is nontrivial to apply visual grounding systems for our dataset; however, developing more accurate systems would be worth pursuing in the future. We hope that the CityRefer dataset promotes further research and development as well as discussion on geography-aware learning technologies.
Broader impacts. Although the dataset is constructed on the basis of a publicly available 2D map (OpenStreetMap), visual grounding in general may result in privacy issues or racial and gender biases. The natural language descriptions we collect are carefully checked so that they do not include private information or offensive text.
This work was supported by JST PRESTO JPMJPR22P8 and JPMJPR20C2, and JSPS KAKENHI 22K12159.