SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Metrics. To evaluate our models on the task of 3D occupancy prediction, we need to convert the point-level semantic annotations from LiDAR to voxel-level annotations. We do this by taking the most-present label inside each voxel. As we aim at semantic segmentation, our main metric is mean Intersection over Union (mIoU), which we use in the evaluation protocol proposed in the next paragraph. Additionally, we measure the class-agnostic occupancy Intersection over Union (IoU). For the retrieval benchmark, we report the average precision (AP) for each query, the mean of which over all queries yields the mean average precision (mAP).
New benchmark for open-vocabulary language-driven 3D retrieval. To evaluate the retrieval capabilities, we collected a new language-driven 3D grounding & retrieval benchmark equipped with natural language queries. To build this benchmark, we annotated 3D scenes from various splits of the nuScenes dataset with the ground-truth spatial localization for a set of natural language open-vocabulary queries. The resulting set contains 105 samples in total, which are divided to 42/28/35 samples from train/val/test splits of the nuScenes dataset. The objective is, given the query, to retrieve all relevant 3D points from the LiDAR point cloud. Results are evaluated using the precision-recall curve; negative data are all the non-relevant 3D points in the given scene. For the evaluation purposes, we report numbers on a concatenated set consisting of samples from the validation and test splits
(63 samples). To annotate the 3D retrieval ground truth, we (1) manually provide the bounding box of the relevant object(s) in the image domain, (2) use Segment Anything Model [31] guided by our manual bounding box to produce a binary mask of this object, (3) project the LiDAR point cloud into the image, and (4) assign each 3D point a label corresponding to its projection into the binary mask. Furthermore, we use HDBSCAN [40] to filter points that are projected to the mask in the image but in fact do not belong to the object. This resolves the imprecisions caused by projection.
New evaluation protocol for 3D occupancy prediction. The relatively new task of 3D occupancy prediction has no established evaluation protocol yet. TPVFormer [26] did not introduce any evaluation protocol and provided only qualitative results. Having semantic labels only from LiDAR points, i.e., not in the target voxel space, makes it challenging to evaluate. Since voxel semantic segmentation consists of both occupancy prediction of the voxel grid and classification of occupied voxels, it is not enough to evaluate just at the points of ground-truth information from the LiDAR, as this does not take free space prediction into account.
To tackle this, we take inspiration from [7] and propose to obtain the evaluation labels from the available LiDAR point clouds, as depicted in Fig. 3 and described next. First, LiDAR rays passing through 3D space set the labels of intersected voxels to free. Second, voxels containing LiDAR points are assigned the most frequent semantic label of points lying within (or an occupied label in the case of class-agnostic evaluation). Third, all other voxels are ignored during evaluation, as they were not observed by any LiDAR ray and we are not certain whether they are occupied or not.
Figure 3: Validation labels: blue = free, red = occupied, and gray = ignored voxels.
Implementation details. We use the recent TPVFormer [26] as backbone for the 2D-3D encoder. It takes surround-view images on the input and produces a voxel grid of size  $100 \times 100 \times 8$ , which corresponds to the volume  $[-51.2\mathrm{m}, +51.2\mathrm{m}] \times [-51.2\mathrm{m}, +51.2\mathrm{m}] \times [-5\mathrm{m}, +3\mathrm{m}]$  around the car. For the language-image feature extractor, we use MaskCLIP + [64], which provides features of dimension  $C_{\mathrm{ft}}^{\mathrm{out}} = 512$ . If not mentioned otherwise, we use the default learning rate of 2e-4, Adam [30] optimizer, and a cosine learning rate scheduler with final learning rate 1e-6, and with linear warmup from 1e-5 learning rate for the first 500 iterations. We train our models on  $8 \times \mathrm{A}100$  GPUs. We use ResNet-101 as image backbone in the  $f_{3\mathrm{D}}$  encoder, and full-scale images on the input. Both prediction heads have two layers, i.e.,  $N_{\mathrm{occ}} = N_{\mathrm{ft}} = 2$ , and  $C_{\mathrm{occ}} = 512$  and  $C_{\mathrm{ft}} = 1024$  feature channels. With this architecture setup, we train our model on  $100\%$  of the nuScenes training data for 12 epochs. We put the same weight to the occupancy and feature losses, i.e., we set  $\lambda = 1$  in Eq. 8. We ablate these choices in Sec. 4.3.
4.2 Comparison to the state of the art
Here we compare our approach to four relevant methods: (i) the fully supervised (closed-vocabulary) TPVFormer [26] and the following three open-vocabulary image-based methods, namely to (ii) MaskCLIP+ [64], (iii) ODISE [60], and (iv) OpenScene [43], which require 3D LiDAR point clouds on the input during the inference. Please note that compared to methods (ii)-(iv), our P0P-3D does not require (1) strong manual annotations (either in the image or point cloud domain) or (2) having point clouds on the input during the inference. Details are given next.
Comparison to a fully-supervised TPVFormer [26]. In figure Fig. 4b, we compare our results to the supervised TPVFormer [26] in terms of class-agnostic IoU and  $(16 + 1)$ -class mIoU (16 semantic classes plus the empty class) on the nuScenes [10] validation set. Interestingly, our model outperforms its supervised counterpart in the class-agnostic IoU by 11.5 points, showing superiority in the prediction of the occupied space. This can be attributed to different training schemes of the two methods: in the fully-supervised case, the empty class competes with the other semantic classes, whereas in our case the occupancy head performs only class-agnostic occupancy prediction. Next, for the  $(16 + 1)$ -class semantic occupancy segmentation, we can see that our zero-shot approach reaches  $\approx 78\%$  of the supervised counterpart performance, which we consider as strong result given that the latter requires manually annotated point clouds for training. In contrast, our approach is zero-shot and does not require any manual point cloud annotations at training. These results pave the way
(a) LiDAR-based evaluation
Figure 4: Comparison to the state of the art. We compare our P0P-3D approach to different baselines using (a) the LiDAR-based evaluation, (b) occupancy evaluation, and (c) open-vocabulary language-driven retrieval. In (a), our zero-shot approach P0P-3D outperforms the strong MaskCLIP+ [64] (M.CLIP+) baseline, while closing the gap to the fully supervised. Other recent methods using supervision and requiring LiDAR points during inference (ODISE [60] and OpenScene [43]) are even better. All methods that require manual annotations during training are denoted by striped bars). In (b), our zero-shot approach P0P-3D surpasses the fully-supervised model [26] on occupancy prediction (IoU) while reaching  $78\%$  of its performance on semantic occupancy segmentation (mIoU). Finally, in (c) we present results of open-vocabulary language-driven retrieval on our newly composed dataset, where we compare our approach to the MaskCLIP+ baseline. We measure mAP on manually annotated LiDAR 3D points in the scene. Our P0P-3D outperforms the MaskCLIP+ approach on this task by  $3.5\mathrm{mAP}$  points.
(b) 3D occupancy prediction
(c) Retrieval (mAP)
for language-driven vision-only 3D occupancy prediction and semantic segmentation in automotive applications. We show qualitative results of our P0P-3D approach in Fig. 5 and in the supplementary materials.
Comparison to MaskCLIP+ [64]. In Fig. 4a we compare the quality of the 3D vision-language features learnt by our P0P-3D approach against the strong MaskCLIP+ [64] baseline. In detail, we project the 3D LiDAR points to the 2D image(s) space, sample MaskCLIP+ [64] features extracted from the 2D image at the projected locations and backproject those extracted features back to 3D via the LiDAR rays. Note that MaskCLIP+ features are used in our tri-modal training to represent the language modality so it is interesting to evaluate the benefits of our approach in comparison to directly transferring MaskCLIP+ features to 3D. For a fair comparison, we evaluate only the LiDAR points with a projection to the camera, i.e., this evaluation considers only the classification of the 3D points, not the occupancy prediction itself. We call this metric LiDAR mIoU. Our P0P-3D outperforms MaskCLIP+ (26.4 vs. 23.0 mIoU), i.e., our method learns better 3D vision-language features than its teacher, while also not requiring LiDAR data at test time (as MaskCLIP+ does). Finally, Fig. 4a shows that P0P-3D reaches  $\approx 84\%$  of the performance of the fully-supervised model [26].
Comparison to open-vocabulary methods that require additional supervision. Furthermore, we compare our approach to ODISE [60] and OpenScene [43], which both require manual supervision during training. ODISE requires panoptic segmentation annotations for training, while OpenScene uses features from either LSeg [33] or OpenSeg [20], which are two image-language encoders that are trained with supervision from manually provided segmentation masks. We report results using OpenSeg. As Fig. 4b shows, these methods perform best, which can be attributed to additional manual annotations available during training.
Open-vocabulary language-driven retrieval. The goal is, given a text query of the searched object, to retrieve all 3D points belonging to the object in the given scene. During the evaluation, to get the relevance of LiDAR points to the query text description, we follow the same approach as for the task of zero-shot semantic segmentation, i.e., we pass the images to our model, get features aligned with the text, and compute their relevance to the given text query. This gives a score for every 3D point in the scene. In the ideal case the points belonging to the target object should have the highest score. We compare our method with MaskCLIP+ and report results in Fig. 4c. Our approach exhibits superior mAP compared to MaskCLIP+, achieving 18.4 mAP while MaskCLIP+ obtains mAP of 14.9.
Here we study the sensitivity of our model to various hyperparameters. Except otherwise stated, for this study we use half-resolution input images, i.e.,  $450 \times 800$ , the ResNet-50 backbone, and train for 6 epochs using  $50\%$  of the nuScenes training data.
Table 1: Sensitivity analysis. We investigate here the impact of loss weight  $\lambda$  in the final loss function (a), the image resolution and image backbone (b) and the depth of the prediction heads (c).
(a) Loss weight  $\lambda$  impact
λ	mIoU	IoU
1.00	12.0	30.0
0.50	12.0	30.5
0.25	11.9	30.5
(b) Image resolution and backbone
image resolution	mIoU
RN50	RN101
450×800	12.0	15.1
900×1600	12.3	15.2
(c) Depth of prediction heads
Nocc / Nft	mIoU
2	3
2	15.4	15.3
3	15.3	15.5
Loss weight  $\lambda$ . In Tab. 1a we study the sensitivity of our model to the loss weight  $\lambda$  of  $\mathcal{L}_{ft}$  in Eq. 8. We see that the model's performance is not sensitive to  $\lambda$ . By default we use  $\lambda = 1$ .
Input resolution and image backbone. In Tab. 1b we experiment with (a) using half  $(450\times 800)$  or full  $(900\times 1600)$  input images, and (b) using ResNet-50 (RN50) or ResNet-101 (RN101) for the image backbone. Following [26], RN50 is initialized from MoCov2 [15] weights and RN101 from FCOS3D [56] weights. We see that it is better to use the RN101 backbone while the input resolution has small impact (with full resolution being better).
Depth of prediction head. In Tab. 1c we study the impact of the  $N_{\mathrm{occ}}$  and  $N_{\mathrm{ft}}$  hyperparameters that control the number of hidden layers on the occupancy prediction head  $g$  and 3D language head  $h$  respectively, using RN101 as backbone. We see that the depth of the two prediction heads does not play a major role and it is slightly better to be the same, i.e.,  $N_{\mathrm{occ}} = N_{\mathrm{ft}}$ . Therefore, we opt to use  $N_{\mathrm{occ}} = N_{\mathrm{ft}} = 2$  in our experiments, as it performs well and requires less compute.
4.4 Demonstration of open-vocabulary capabilities
In Fig. 6 we provide visualizations of language-based 3D object retrievals inside a scene using text queries like "building door" and "tire". For reference, green boxes denote locations of reference objects (cars), to ease the orientation in the scene. The results show that our model is able to localize in 3D space fine-grained language queries.
Limitations. First, given the low spatial resolution of the voxel grid our model does not discover well small objects. This is not a limitation of the method, but of the currently used backbone architecture and input data. Second, another limitation is that our architecture does not natively support sequences of images as input which might be beneficial for reasoning about semantic occupancy of occluded objects and areas appearing thanks to relative motion of objects in the scene.
In this paper we propose P0P-3D, a tri-modal self-supervised learning strategy with a novel architecture that enables open-vocabulary voxel segmentation from 2D images and at the same time improves the occupancy grid estimation by a significant margin over the state of the art. Our approach also outperforms the strong baseline of directly back-projecting 2D vision-language features into 3D via LiDAR and does not require LiDAR at test-time. This work opens up the possibility of large-scale open-vocabulary 3D scene understanding driven by natural language.
Figure 5: Qualitative results of zero-shot semantic 3D occupancy prediction on the 16 classes in the nuScenes [10] validation split. Please note how our method is able to quite accurately localize and segment objects in 3D including road (magenta), vegetation (dark green), cars (blue), or buildings (gray) from only input 2D images and in a zero-shot manner, i.e. only by providing natural language prompts for the target classes. Visualizations are shown on an interpolated  $300 \times 300 \times 24$  voxel grid.
Figure 6: Qualitative results showcasing the language-driven 3D grounding and retrieval. On left (in red) we can see the six input images passed to the P0P-3Dto get open-vocabulary 3D features (middle). Given the searched object name ("Black hatchback"), we compute the similarity with the 3D feature field and obtain similarity heatmap (right). Language-based 3D retrieval is not possible using existing close-vocabulary methods such as [26]. Please see additional results in the supplementary.
This work supported by the European Regional Development Fund under the project IMPACT (no. CZ.02.1.010.00.015_0030000468), by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90140), and by CTU Student Grant SGS21184OHK33T37. This research received the support of EXA4MIND, a European Union's Horizon Europe Research and Innovation programme under grant agreement  $\mathrm{N}^{\circ}$  101092944. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the granting authority can be held responsible for them. The authors have no competing interests to declare that are relevant to the content of this article. Antonin Vobecky acknowledges travel support from ELISE (GA no 951847). We acknowledge the support from Valeo.
[54] Hao Tian, Yuntao Chen, Jifeng Dai, Zhaoxiang Zhang, and Xizhou Zhu. Unsupervised object detection with lidar clues. In CVPR, 2021. 3
[55] Antonin Vobecky, David Hurych, Oriane Simeoni, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, and Josef Sivic. Drive&segment: Unsupervised semantic segmentation of urban scenes via cross-modal distillation. In ECCV, 2022. 3
[56] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In ICCV, 2021. 9
[57] Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer. Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud. In ICRA, 2019. 2
[58] Shun-Cheng Wu, Keisuke Tateno, Nassir Navab, and Federico Tombari. SCFusion: real-time incremental scene reconstruction with semantic completion. In 3DV, 2020. 2
[59] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In CVPR, 2019. 3
[60] Jiarui Xu, Sifei Liu, Arash Vahdatc, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023. 7, 8
[61] Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shuguang Cui, and Zhen Li. 2dpass: 2d priors assisted semantic segmentation on lidar point clouds. In ECCV, 2022. 2
[62] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Lianian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In CVPR, 2022. 3
[63] Brady Zhou and Philipp Krahenbuhl. Cross-view transformers for real-time map-view semantic segmentation. In CVPR, 2022. 2
[64] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, 2022. 2, 3, 7, 8
[65] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In CVPR, 2021. 2