POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images
Authors: Antonín Vobecký
Oriane Siméoni
David Hurych
Spyros Gidaris
Andrei Bursuc
Patrick Perez
Josef Sivic
Venue: NeurIPS 2023 poster (2023)
Keywords: open-vocabulary segmentation
voxel occupancy prediction
semantic segmentation
autonomous driving
language-image alignment
TL;DR: Open-Vocabulary 3D occupancy prediction by leveraging language-image pre-training using surround-view cameras available in autonomous driving setups.

Abstract: We describe an approach to predict open-vocabulary 3D semantic voxel occupancy map from input 2D images with the objective of enabling 3D grounding, segmentation and retrieval of free-form language queries. This is a challenging problem because of the 2D-3D ambiguity and the open-vocabulary nature of the target tasks, where obtaining annotated training data in 3D is difficult. The contributions of this work are three-fold. 
First, we design a new model architecture for open-vocabulary 3D semantic occupancy prediction.  The architecture consists of a 2D-3D encoder together with occupancy prediction and 3D-language heads. The output is a dense voxel map of 3D grounded language embeddings enabling a range of open-vocabulary tasks. 
Second, we develop a tri-modal self-supervised learning algorithm that leverages three modalities: (i) images, (ii) language and (iii) LiDAR point clouds, and enables training the proposed architecture using a strong pre-trained vision-language model without the need for any 3D manual language annotations. 
Finally, we demonstrate quantitatively the strengths of the proposed model on several open-vocabulary tasks:
Zero-shot 3D semantic segmentation using existing datasets; 3D grounding and retrieval of free-form language queries, using a small dataset that we propose as an extension of nuScenes. You can find the project page here https://vobecant.github.io/POP3D.

POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images
Antonin Vobecky1.223
Oriane Siméoni'
David Hurych
Spyros Gidaris'
Andrei Bursuc
Patrick Pérez!
Josef Sivic
valeo.ai, Paris, France
2 CIRC CTU in Prague
FEE CTU in Prague
Abstract
We describe an approach to predict open-vocabulary 3D semantic voxel occu- pancy map from input 2D) images with the objective of enabling 3D grounding. segmentation and retrieval of free-form language queries. This is a challenging problem because of the 2D-3I) ambiguity and the open-vocabulary nature of the target tasks, where obtaining annotated training data in 3D is difficult. The con- tributions of this work are three-fold. First, design new model architecture for open-vocabulary 3D semantic occupancy prediction. The architecture consists 2D-3D encoder together with occupancy prediction and 3D-lanuuage heads. The output is dense voxel map of 3D grounded language embeddings enabling a range opea-uocybulary tasks. Second, we develop tri-modal sufessuper learning algorithm that leverages three modalities: (i) images, (ii) language and (iii) LiDAR point clouds, and enables training the proposed architecture using strong pre-trained vision-language model without the need for any 3D manual language annotations. Finally, we demonstrate quantitatively the strengths of the proposed model on several open-vocabulary tasks: Zero-shot 3D semantic segmentation using existing datasets; 3D grounding and retrieval free-form language queries, using small dataset that we propose as extension of nuScenes. You can find the project page here https: //wobeaatt....eeeeeee/ee//o/...e wwwaaantttggttt...e github. io/OP3D..
Introduction
The detailed analysis of 3D environments -both geometrically and semantically- is fundamental perception brick in many applications, from augmented reality to autonomous robots and vehicles. It usually conducted with cameras and/or laser scanners (LiDAR). In its most complete version, called semantic 3D occupancy prediction, this analysis amounts to labelling each voxel of the perceived volume as occupied by a certain class of object or empty. This extremely challenging since both cameras and LiDAR only capture information about visible surfaces, which may be projected from 3D into 2D without the loss of information, but not for every point in the 3D space. This one extra dimension makes prediction arduous and hugely complicates the manual annotation task.
Recent works, e.g., [26], propose leverage manually manually-annotate annotated LiDAR data produce partial annotation of the 3D occupancy space. However, relying on manual semantic annotation of point clouds remains difficult scale, even if sparse, and limits the learned representation to encode solely a closed vocabulary, i.e., limited predefined set of classes. In this work, we tackle these challenges and propose an open- -pec-vocabulary approach 3D semantic occupancy prediction that relies only on
"Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Our method: POP 3D
Examples of open-vocabulary tasks
surround view images
3D-text-aligned voxel features
zero-shot occupancy segmentation
text-based retrivall
PUP
open-vocabulary queries jeep, person, tree, trash bin...
similarity
rood
bus
vegetation
Q DIRE "jeep"
terrain
barrier
Figure I: Overview of the proposed method. Provided only with surround-view images as input, our model called POP-3D produces voxel grid of 3D text-aligned features that support open-vocabulary downstream tasks such zero-shot occupancy segmentation or text-based grounding and retrieval.
unlabeled image-LiDAR data for training. In addition, our model uses only camera inputs at run time, bypassing altogether the need for expensive dense LiDAR sensor, in contrast with most 3D semantic perception systems (whether at point or voxel level).
To this end, we harness the progress made recently in supervised 3D occupancy prediction [26] and in language-image alignment [64], within a two-head image-only model that can be trained with aligned image-LiDAR raw data. Leveraging sparse 3D occupancy information that LiDAR scans provide for free, we first train a class-agnostic occupancy prediction head. Using this same LiDAR information along with pre-true pre-trained language-alignt aligned visual features at the corresponding locations in images, jointly train a second head that predicts the same type of features at the 3D voxel level. At run time, these features can be probed from text prompts to get open- vocabboocyylary semantic segmentation of voxels that are predicted as occupied (Fig. 1). To assess the effectiveness of our method for semantic 3D occupancy prediction, Through evaluation with this protocol on autonomous driving data, our method shown to achieve a strong performance relative to the fully-supervised approach.
introduce novel evaluation protocol specifically tailored to this task.
In a nutshell, we attack the difficult problem of 3D semantic occupancy prediction with the lightest possible requirements: manual annotation of the training data, no pre-defined semantic vocabulary, and no recourse to LiDAR readings at run time. As result, the proposed image-only 3D semantic occupancy model named POP-3D (for oPen-vocabulary Occupancy Prediction in 3D) provides training data scalability and operational versatility, while opening up new understanding capabilities for autonomous systems through language-driven scene perception.
2 Related work
Semantic 3D occupancy prediction. Automatic understanding of the 3D geometry and semantics of scene has been traditionally enabled through high precision LiDAR sensors and corresponding architectures. 3D semantic segmentation, i.e., point-level classification of a point cloud, can be addressed with different types of transformations of the point cloud: point-based, directly operating on the -imeessiiniio points [45, 46, 53], and projection- based, operating on different representation, e.g., two-dimensionall images [57, 32, 8] or three-dimensional voxel representations [61, 65, 52, 18]. However, they produce predictions as sparse as the LiDAR point cloud offering an incomplete understanding of the full scene. Semantic scene completion [50] aims for dense inference of 3D geometry and semantics of objects and surfaces within a given extent, typically leveraging rich geometry information the input extracted from depth [16, 35], occupancy grids [58, 49], point clouds [48], or mix of modalities, c.g., RGBD [11, 17]. In this line, MonoScene [12] is the first camera-basel method produce dense semantic occupancy predictions from single image by projecting image features into 3D voxels by optical ray intersection. Recent progress in multi- camera Bird's- --ye- View (BEV) projection [44, 25, 63, 38, 5, enables the recent TPVFrrer [26] to generate surrounding 3D occupancy predictions by effectively exploiting tri tri-perspeetive view representations [13] augmenting the standard BEV with two additional perpendicular planes to recover the full 3D. All prior methods are trained in a supervised manner requiring rich voxel-level semantic information, which costly to curate and annotate. While we build on [26], we forego manual label supervision and, instead, develop model able to produce semantic 3D occupancy

---

predictions using supervision from LiDAR and from an image- image-language model allowing our model to acquire open-vocabulary skills in the voxel space.
Multi-modal representation learning. Distilling signals and knowledge from one modality into another is an effective strategy to learn representations [2, 3] or to learn to solve tasks using only few [14, 42, 1] or no human labels [54, 55J. The interplay between images, language and sounds often used for self-supervised representation learning over large repositories of unlabeled data fetched from the internet [2-4, 41, 42]. Images can be paired with different modalities towards solving complex 2D tasks, e.g., semantic segmentation [55], detection of road objects [54] or sound- aligned models project images and text into shared representation space [21, 51, 34, 36, 19, 47, 28]. Contrastive image-language learning on many millions of image-text pairs [47, 28] leads to high-quality representations with impressive zero-shot skills from one modality to the other. We use CLIPP [47] for its appealing open-vocabulan property that enables the querying of visual content with natural language toward recognizing objects of interest without manual labels. POP-3D uses LiDAR supervision for precise occupancy prediction and learns to produce in the 3D space CLIP-like features easily paired with language.
emitting objects [14, 42, 1]. image-languss
Open-vocabulas semantic segmentation. The aim of zero-shot semantic segmentation is to segment object classes that are not seen during training [59, 9, 24J. The advent of CLIP [47], which is trained on abundant web data, has inspired a new wave of methods, dubbed open-vocabulary. for recognizing random objects via natural language queries. CLIP features can be projected into 3D meshes [27] and NeRFs [29] to enable language queries. Originally producing image- level embeddings. CLIPP can be extended to pixel-level predictions for open-vocabulary semantic segmentation by exploiting different forms of supervision from segmentation datasets, e.g., pixel- level labels [33] or class agnostic masks [20, 39, 62] coupled with region-word grounding [23], however with potential forgetting of originally learned concepts [27]. MaskCLIP+ [64] adjusts the attentive-pooling layer of CLIP to generate pixel-level CLIP features that are further distilled into an encoder-decoder semantic segmentation network. MaskCLIP+ [64] preserves the open-vocabulary properties of CLIP, and we exploit it here to distill its knowledge into POP-3D. We generate target 3D CLIP features by mapping MaskCLIP+ pixel-level features to LiDAR points observed in images. By being trained to match these distillation targets, POP-3D manages to learn 3D features with open- neen-vocabllary perception abilities, in contrast to prior work on 3D occupancy prediction that is limited to recognizing closed-set of visual concepts.
3 Open-vocabulary 3D occupancy prediction
Our goal is predict 3D voxel representations of the environment, given set of 2D input RGB images, that is amenable to open-vocabulary tasks such as zero-shot semantic segmentation or concept search driven by natural language queries. This is challenging problem as we need to address the following two questions. First, what is the right architecture to handle the 2D-to-3D ambiguity and the open-vocabulary nature of the task? Second, how to formulate the learning problem without requiring manual annotation of large amounts of 3D voxel data, which are extremely hard to produce.
To address these guestions propose the following two innovations. First, we design an architecture for vocabulayy 3D occupancy prediction (Fig. 2(a) and Sec. 3.1) that handles the 2D-to-3D prediction and open-vocabulary tasks with two specialized heads. Second, we formulate its training as a tri-modal self-supervised learning problem (Fig. 2(b) and Sec. 3.2) that leverages aligned (i) 2D images with (ii) 3D point clouds equipped with (iii) pre-trained language- -eguuagg-maageaaee image features the three input modalities (i.e. camera, LiDAR and language) without the need for any explicit manual annotations. The details of these contributions are given next.
3.1 Architecture for open-vocabulary 3D occupancy prediction
We are given a set of surround-view images captured from one camera location and our goal is to output 3D occupancy voxel map and to support language-driven driven tasks. To reach the goals, we propose an architecture composed of three modules (Fig. 2(a)). First, a 2D-3D encoder predicts a voxel feature grid from the input images. Second, the occupancy head decodes this entire voxel grid into an occupancy map, predicting which voxels are free and which are occupied. Finally, the 3D-language head is applied on each occupied voxel to output a powerful language embedding vector enabling a range of 3D open- vocabulayy tasks. The three modules are described next.
3

---

Architecture open vocabulary occupancy prediction (sec. 3.1)
occupancy
full grid
2D-3D encoder
suround-view images
fTD
voxel feature grid Wy Dy
vonea features
occupled
3D- languagee head
Tri-modal seif-supervised training (sec. 3.2)
Feature loss Cn.
LIDAR points (frininn only)
language-iman image encoder
get features af point projections
feeture spece
aocamied
vaxelize
voxel-level
Occupancy loss Ceee
Figure 2: Proposed approach. In (a), we show the architecture of the proposed method. Having only surround-view images the input, the model first extracts dense voxel feature grid that is then fed to two parallel heads: occupancy head g producing voxel-level occupancy predictions, and to 3D-language feature head á which outputs features aligned with text representations, In b), we show how we train our approach, namely the occupancy loss Lose used to train class-agnostic occupancy predictions, and the feature loss La that enforces the 3D-language head h to output features aligned with text representations.
2D-to-3D encoder fAD. The objective of the 2D-to-3D encoder is to predict dense feature voxel grid given one or more images captured at one location as input. The output voxel grid representation encodes 3D visual information captured by the cameras. In detail, given surround-vin view camera RGB images I and camera calibration parameters, the encoder San produces a feature voxel grid
V = SaD (I) E HRHyxWWxDDxCC.
where Hy, Wv, and Dy are the spatial dimensions of the voxel grid, and Cy is the feature dimension of each voxel. This feature voxel grid is then passed to two distinct prediction heads designed to perform class-agnostic occupancy prediction and text-aligned feature prediction tasks respectively. The two heads are described next.
Occupancy head g. Given the feature voxel grid V, the occupancy prediction head g aims classifying every voxel as 'empty' or 'occupied'. Following [26], this head is implemented as a non-linear network composed of Noce hidden blocks with configuration Linear -Seerr-Sffppls- --offpplus--innrr -Linerr, each with Chidden hidden features, and a final linear classifier outputting two logits, one per class. outputs the tensor
O = g (V) HyxxWyxDxx2
(2)
containing the occupancy prediction for each voxel.
3D language head ht In parallel, the voxel grid V is fed to a language feature extractor. This head processes each voxel feature output an embedding vector that is aligned vision-language representations, such as CL.IP [47], aiming to inherit their open-vocabulary abilities. This allows us to address the limitations of closed-vocabulary predictions encountered in supervised 3D occupancy prediction models, which are bound to set of predefined visual classes. In contrast, our representa- tion enables to perform 3D language-driven tasks such as zero-shot 3D semantic segmentation. Similarly to the occupancy head, the 3D-language head consists of N₂ blocks with config configuration where each linear layer outputs Chidden features, and final linear layer that outputs Cf dime--iioii vision language embedding for each voxel. It outputs the tensor
Linear-Softse
On = (V) R Dyx
containing the predicted vision-languag embedding of each voxel.

---

3.2 Tri-modal self-supervised training
The goal to train the network architecture described in Sec. 3.1 predict the 3D occupancy map together with language-awan aware features for each occupied voxel. In turn, this will enable 3D open-vocabulary tasks such 3D zero-shot segmentation or language-driven search. The main challenge is obtaining the appropriate 3DD 3ID-grounded language annotations, which is expensive to do manually. Instead, we propose a tri-modal self-supervised learning algorithm that leverages three modalities: (i) images, (ii) language and (iii) LiDAR point clouds. Specifically, we employ a pre-trained image-language network to generate image-language features for the input images. These features are then mapped to the 3D space using registered LiDAR point clouds, resulting in 3D grounded image-language features. These grounded features serve as training targets for the network. The training algorithm is illustrated in Fig. 2(b). The training is implemented via two losses that are used to train the two heads of the proposed architectures jointly with the 2D-to-3i encoder. The details are given next.
Occupancy loss. We guide the occupancy head g to perform a class-agnostic occupancy prediction by the available unlabeled LiDAR point clouds, which we convert to occupancy prediction targets Toce {0. 1). Each voxel location x containing at least one LiDAR point labeled as 'occupied' (i.e., Tace (z) 1) and as 'empty' otherwise (T))( = 0). Having these targets, we supervise the occupancy prediction head densely at all locations of the voxel grid. The occupancy loss Lace is combination of cross-entropy loss LcE and Lovász-softmar [6] loss LLov:
Luce (Ouse Toce) = LCB (O Toce) + LLav (Onee, Towe),
where Oxee the predicted occupancy tensor and Toze the tensor of corresponding occupancy targets.
Image-language distillation. Unlike the occupancy prediction head that is supervised densely at the level of voxels, supervise the 3D-language head at the level of points p.. E Peam which project to at least one of the cameras, i.e., Peam C P, where P is the complete point cloud. This is required in order obtain feature targets from the language-image pre-trained model fi-
To get a feature target for a 3D point Pn. E Peam in the voxel feature grid, we use the known camera
projection function II, that projects 3D point yn, into 2D point (un (x) (w)) where (x() (x) (x))
are (xr, y) coordinates point u, camera C:
Uh IT (pn)
This way. we get set of 2D points U = {II (p.)) } in the camera coordinates. To obtain feature targets Tn for 3D points in Poam with corresponding 2D projections U in camera c, we run the feature extractor fi on image I,,, and use the 2D projections' coordinates to sample from the resulting feature map, i.e.,
Tn = { (I.) (u 11
E N
(6)
n=1
where [2r, v] is an indexing operator in the extracted feature map.
To train the 3D language head, we use Lg mean squared error loss between the targets Tn and the predicted features Ön IR xox" computed from for the 3D point locations in Paam
Ln
NC Th Ón
where || || is the Frobenius norm.
Final loss. The final loss used train the whole network a weighted sum of the occupancy and image-language losses. We use single hyperparameter À to balance the weighting of the two losses:
L
Luce + XLa-
(8)
3.3 3D open-vocabulary test-time inference
Once trained, as described in Sec. 3.2, our model supports different 3D open-vocabulary tasks at test- time. We focus on the following two: (i) zero-shot 3D semantic segmentation and (ii) language-driven 3D grounding.
5

---

Zero-shot 3D semantic segmentation from images. Given an input test image, the 3D 3D-text-aligned voxel features produced by our model support zero-shot 3D segmentation for a target set of classes specified via input text queries (prompts), as illustrated in Fig. 1. Unlike supervised approaches that necessitate retraining when the set of target classes changes, our approach requires training the model only once. We can adjust the number of segmented classes effortlessly by providing a different set of input text queries. In detail, at test-time we proceed along the following steps. First, set of test surround-viee images from one location is fed into the trained POP-3D network, resulting in class-agnostic occupancy prediction Ooce via the occupancy head g. and language-aligned feature predictions On via the 3D-language head h. Next, as described in [22], we generate set of query sentences for each text query using predefined templates. These queries are then input into the pre-trate -reitrained language- -image encoder fex.. resulting in set of language features. To obtain single text feature per query, we compute the average of these features. Finally, considering M such averaged text features, one for each of the M target segmentation classes, we measure their similarity to the predicted language-aligned features On at occupied voxels obtained from Ouse. We assign the label with the highest similarity each occupied voxel.
Language-driven 3D grounding. The task of language-driven 3D grounding is performed in a similar manner. However, here only a single input language query is given. Once determining the occupied voxels from Occe- we compute the similarity between the input text query encoded via the language-ing -mmgge encoder fex and predicted language-aligned features On at the occupied voxels. The resulting similarity score can be visualized as a heat-map, as shown in Fig. 1, or thresholded to obtain the location of the target query.
4 Experiments
This section studies architecture design choices and demonstrates the capabilities of the proposed approach. First, in Sec. 4.1, we describe the experimental setup used, particularly the dataset, metrics, proposed evaluation protocol, and implementation details. Then, we compare model to the state of the art in Sec. 4.2. Next, we present a set of studies on training hyperparameter sensitivity in Sec. 4.3 and finally show qualitative results in Sec. 4.4.
4.1 Experimental setup
We test the proposed approach on autonomous driving data, which provides a challenging test-bed.
Dataset. We use the nuScenes [10] dataset composed of 1000 sequences in total, divided into 700/150/150 scenes for train/val/test splits. Each sequence consists of 30 - 40 scenes resulting in 28, 130 training and in 6, 019 validation scenes. The dataset provides 3D point clouds captured with 32-beam LiDAR, surround- -iwe images obtained from six cameras mounted at the top of the car, and projection matrices between the 3D point cloud and cameras. LiDAR point clouds are annotated with 16 semantic labels. When using subsets of the complete dataset for ablations, we sort the scenes by their timestamp and take every N-th scene, e.g., every second scene in the case of a 50% subset.
Metrics. To evaluate our models on the task of 3D occupancy prediction, we need to convert the point-level semantic annotations from LiDAR to voxel-level annotations. We do this by taking the most-present label inside each voxel. As we aim at semantic segmentation, our main metric is mean Intersection over Union (mloU), which we use in the evaluation protocol proposed in the next paragraph. Additionally, we measure the class-agnostic occupancy Intersection over Union (IoU). For the retrieval benchmark, we report the average precision (AP) for each query, the mean of which over all queries yields the mcan average precision (mAP).
New benchmark for open- vocabulayyy language-driven 3D retrieval. To evaluate the retrieval capabilities, we collected a new language-driven 3D grounding retrieval benchmark equipped with natural language queries. To build this benchmark, we annotated 3D scenes from various splits of the nuScenes dataset with the ground-truth spatial localization for a set of natural language open- vocabulary queries. The resulting set contains 105 samples in total, which are divided to 42/28/35 samples from train/val/test splits the nuScenes dataset. The objective is, given the query, to retrieve all relevant 3D points from the LiDAR point cloud. Results are evaluated using the rrccision-recal. curve; negative data are all the non-relevant 3D points in the given scene. For the evaluation purposes, report numbers a concatenated set consisting of samples from the validation and test splits

---

(63 samples). To annotate the 3D retrieval ground truth, we (1) manually provide the bounding box of the relevant object(s) in the image domain, (2) use Segment Anything Model [31] guided by our manual bounding box to produce a binary mask of this object, (3) project the LiDAR point cloud into the image, and (4) assign each 3D point a label corresponding to its projection into the binary mask. Furthermore, we use HDBSCAN [40] to filter points that are projected to the mask in the image but in fact do not belong to the object. This resolves the imprecisions caused by projection.
New evaluation protocol for 3D occupancy prediction. The relatively new task of 3D oc- cupancy prediction has no established evaluation protocol yet. TPVFormer [26] did not in- troduce any evaluation protocol and provided only qualitative results. Having semantic labels only from LiDAR points, i.e., not in the target voxel space, makes challenging to evalu- ate. Since voxel semantic segmentation consists of both occupancy prediction of the voxel grid and classification of occupied voxels, it is not enough to evaluate just at the points of ground- truth information from the LiDAR, as this does not take free space prediction into account. To tackle this, we take inspiration from [7] and pro- pose to obtain the evaluation labels from the available LiDAR point clouds, as depicted in Fig. 3 and de- scribed next. First, LiDAR rays passing through 3D space set the labels of intersected voxels to free. Sec- ond, voxels containing LiDAR points are assigned the most frequent semantic label of points lying within (or occupied label in the case of class-agnostic evaluation). Third, all other voxels are ignored dur- ing evaluation, as they were not observed by any LiDAR ray and we are not certain whether they are occupied or not.
Figure 3: Validation labels: blue = =free
red occupied, and gray ignored voxels.
Implementation details. We use the recent TPVFrrmrr [26] as backbone for the 2D-3D encoder. takes surround-view images on the input and produces a voxel grid of size 100 x 100 x 8, which corresponds to the volume [-51.2m, 2-5122mx[-51..1 the language-image
+51.2m [ ( +33] around the car. For
feature extractor, CRt = 512. If not mentioned otherwise, we use the default learning rate of 2c-4, Adam [30] optimizer, and cosine learning rate scheduler with final learning rate 1e-6, and with linear warmup from le-5 learning rate for the first 500 iterations. We train our models on 8x GPUs. We use ResNet-101 as image backbone in the faD encoder, and full-scale images the input. Both prediction heads have two layers, i.e., Noce = Na 2. and Case = 512 and Cn 1024 feature channels. With this architecture setup, we train our model on 100% of the nuScenes training data for 12 epochs. We put the same weight to the occupancy and feature losses, i.e., we set A = 1 in Eq. 8. We ablate these choices in Sec. 4.3.
MaskCLIP + [64]. which provides features of dimension
4.2 Comparison to the state of the art
Here we compare our approach to four relevant methods: (i) the fully supervised (closed-vocabulary) TPVFormer [26] and the following three open-vocabulas image-based methods, namely to (ii) MaskCLIP+ [64], (iii) ODISE [60]. and (iv) OpenScene [43], which require 3D LiDAR point clouds on the input during the inference. Please note that compared to methods (ii)-(iv,, our POP-3D does not require (1) strong manual annotations (either in the image point cloud domain) or (2) having point clouds on the input during the inference. Details are given next.
Comparison to a fully-supervised TPVFormer [26]. In figure Fig. 4b, we compare our results to the supervised TPVFormer [26] in terms of class-agnostic IoU and (16+1)-class mloU (16 semantic classes plus the empty class) on the nuScenes [10] validation set. Interestingly, our model outperforms its supervised counterpart in the class-agnostic IoU by 11.5 points, showing superiority in the prediction of the occupied space. This can be attributed to different training schemes of the two methods: in the sully-sup case, the empty class competes with the other semantic classes, whereas in our case the occupancy head performs only alass-agoostic occupancy prediction. Next, for the (16+1 1) cllass semantic occupancy segmentation, we can see that our zero-shot approach reaches 2 78% of the supervised counterpart performance, which we consider as strong result given that the latter requires manually annotated point clouds for training. In contrast, our approach is zero-shot and does not require any manual point cloud annotations at training. These results pave the way
7

---

40
34.7
38.8
40
37.7
20
18.4
31.3
14.9
LiDAR mloU
30
26.4
voxcl metric.
30
26.2
15
23
21.3
20
16.7
LiDAR
10
10
Fully-sup.
2010-shot (POP-3D)
M.CLIP+ PDP-3D sup. ODISE OpenSen.
mloU
loU
POP-3D M.CLIP+
(a) Liased evaluation
(b) 3D occupancy prediction
(c) Retrieval (mAP)
Figure Comparison to the state of the art. We compare our POP-3D approach to differ- ent baselines using (a) the -eAR-baased evaluation, (b) occupancy evaluation, and (c) open- vocabulary language-driven retrieval. (a), our zero-shot approach POP-3D outperforms the strong MaskCLIP+ [64] (M. (MCI)))) baseline, while closing the gap to the fully supervised. Other recent methods using supervision and requiring LiDAR points during inference (ODISE [60] and Open- Scene [43]) are better. All methods that require manual annotations during training denoted by striped bars). In (b), our zero-shot approach POP-30 3D surpasses the supey-sue model [26] on occupancy prediction (IoU) while reaching 78% of its performance mentation (mloU). Finally, in (c) we present results of open-vocabulary language-driven retrieval on newly composed dataset, where we compare
semantic occupancy seg-
approach to the MaskCLIP+ baseline. We measure mAP on manually annotated LiDAR 3D points in the scene. Our POP-3D outperforms the MaskCLIP+ approach this task by 3.5 mAP points.
for language-driven vision-only 3D occupance prediction and semantic segmentation in automotive applications. We show qualitative results of our POP-3D approach in Fig. 5 and in the supplementary materials.
Comparison to MaskCLIP+ [64]. In Fig. 4a we compare the quality of the 3D vision-language features learnt by our POP-3D approach against the strong MaskCL[P+[64] baseline. detail, we project the 3D LiDAR points to the 2D image(s) space, sample MaskCLIP+[64] features extracted from the 2D image the projected locations and backproject those extracted features back to 3D via the LiDAR rays. Note that CLIP+ features are used in our tri-modal training to represent the language modality so it interesting to evaluate the benefits of our approach comparison to directly transferring MaskCLIP+ features to 3D. For fair comparison, we evaluate only the LiDAR points with projection to the camera, i.e., this evaluation considers only the classification of the points, not the occupancy prediction itself. We call this metric LiDAR mloU, Our POP-3D outperforms MaskCLIP+ (26.4 vs. 23.0 mloU), i.c., our method learns better 3D vision-language features than its teacher, while also not requiring LiDAR data at test time (as MaskCLIP+ does). Finally, Fig, 4a shows that PDP-3D reaches = 84% of the performance of the fully-supervised model [26].
Comparison to open-vocabulary methods that require additional supervision. Furthermore, we compare our approach ODISE [60] and OpenScene [43], which both require manual supervision during training. ODISE requires panoptic segmentation annotations for training, while OpenScene uses features from either LSeg [33] or OpenSeg [20], which are two image-language encoders that are trained with supervision from manually provided segmentation masks. We report results using OpenSeg. As Fig. 4b shows, these methods perform best, which can attributed to additional manual annotations available during training.
Open-vocabular language-driven retrieval. The goal is, given a text query of the searched object, to retrieve all 3D points belonging to the object in the given scene. During the evaluation, to get the relevance of LiDAR points the query text description, we follow the same approach as for the task of shot semantic segmentation, i.e., we pass the images to our model, get features aligned with text, and compute their relevance to the given text query. This gives score for every 3D point in the scene. In the ideal case the points belonging to the target object should have the highest score. We compare our method with MaskCIIP+ and report results Fig. 4c. Our approach exhibits superior mAP compared to MaskCLIPt, achieving 18.4 mAP while MaskCLIP+ obtains mAP of 14.9.

---

4.3
Sensitivity analysis
Here we study the sensitivity of our model to various hyperparameters, Except otherwise stated, for this study we use resfuresslution input images, i.e., 450 x 800, the ResNet-50 backbone, and train for epochs using 50% of the nuScenes training data.
Table Sensitivity analysis. We investigate here the impact of loss weight the final loss function (a). the image resolution and image backbone (b) and the depth of the prediction heads (c).
(a)
Loss weight impact
(b) Image resolution and backbone
(c) Depth prediction heads
mloU
loU
image resolution
mloU
mloU
RN50
RN101
Noce
Nn
2
3
1.00
12.0
30.0
0.50
12.0
30.5
x50
12.0
15.1
2
15.4
15.3
0.25
11.9
30.5
900x 1600
12.3
15.2
15.3
15.5
Loss weight X. In Tab. Ia we study the sensitivity of our model to the loss weight À of L in Eq. 8. We see that the model's performance not sensitive A. By default we use À
Input resolution and image backbone. In Tab. lb we experiment with (a) using half (45)x800) or full (900x 1600) input images, and (b) using ResNet-50 (RN50) or ResNet-101 (RN101) for the image backbone. Following [26]. RN50 is initialized from MoCov2 [15] weights and RN101 from FCOS3D [56] weights. We see that it is better to use the RN101 backbone while the input resolution has small impact (with full resolution being better).
Depth of prediction head. In Tab. Ic we study the impact of the Nos and N₂ hyperparameters that control the number of hidden layers the occupancy prediction head g and 3D language head respectively, using RN101 as backbone. We see that the depth of the two prediction heads does not play major role and it is slightly better be the same, i.e, Noce = Nu. Therefore, we opt to use Noce Nn = 2 in our experiments, as it performs well and requires less compute.
4.4 Demonstration of open-vocabulary capabilities
In Fig. 6 we provide visualizations of language-based 3D object retrievals inside scene using text queries like "building door" and "tire". For reference, green boxes denote locations of reference objects (cars), ease the orientation in the scene. The results show that our model is able localize in 3D space fine-grained language queries.
First, given the low spatial resolution of the voxel grid our model does not discover well small objects. This is not limitation of the method, but of the currently used backbone architecture and input data. Second, another limitation is that our architecture does not natively support sequences of images as input which might be beneficial for reasoning about semantic occupancy of occluded objects and areas appearing thanks to relative motion of objects in the scene.
Limitations.
Conclusion
In this paper we propose POP-3D, a tri-modal self-supervised learning strategy with novel architec- ture that enables open-vocabulas voxel segmentation from 2D images and at the same time improves the occupancy grid estimation by significant margin over the state of the art. Our approach also outperforms the strong baseline of directly back-projecting 2D vision-language features into 3D via LiDAR and does not require LiDAR at test-time. This work opens-up the possibility of large-scale open-vocabulary 3D scene understanding driven by natural language.

---

Figure 5: Qualitative results of zero-shot semantic 3D occupancy prediction on the 16 classes in the nuScenes [10] validation split. Please note how our method able to quite accurately localize and segment objects in 3D including road (magenta), vegetation (dark green), cars (blue), or buildings (gray) from only imput 2D images and in zero-shot manner, i.e. only by providing natural language prompts for the target classes. Visualizations are shown interpolated 300x300x24 voxel grid.
"building door"
"tire" "
Figure 6: Qualitative results showcasing the language-driven 3D grounding and retrieval. On left (in red) we can see the six input images passed to the POP-3Dto get open- vocabulary 3D features (middle). Given the searched object name ("Black hatchback"), we compute the similarity with the 3D feature field and obtain similarity heatmap (right). Language-based 3D retrieval not possible using existing close-vocabular methods such as [26]. Please see additional results in the supplementary.
10

---

Acknowledgements
This work supported by the European Regional Development Fund under the project IMPACT (no. 5000000000)4) by the Ministry of Education, Youth and Sports of the Czech Repub- lic through the e-INFRA CZ (ID:90140), and by CTU Student Grant SGS21 1840HK33T37. This re- search received the support of EXA4MIND, European Union's Horizon Europe Research and Innova- tion programme under grant agreement N 101092944. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Com- mission. Neither the European Union nor the granting authority can be held responsible for them. The authors have competing interests to declare that are relevant to the content of this article. Antonin Vobecky acknowledges travel support from ELISE (GA no 951847). We acknowledge the support from Valeo.
References
[1] Triantafyllos Afouras, Yuki M Asano, Francois Fagan, Andrea Vedaldi, and Florian Metze. Self-supervised object detection from audio-visual correspondence. CVPR. 2022.
[2] Jean-Baptin Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjclovic, Jason Ramapuram, Jeffrey
De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Sclf-supervised multimodal versatile
networks. In NeurlPS, 2020. 3
[3] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran.
Self- supervised learning by cross-modal audio-video clustering. In NewrIPS, 2020. 3
[4] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In ICCV, 2017.
[5] Florent Bartoccioni, ćloi Zablocki, Andrei Bursuc, Patrick Pérez, Matthicu Cord, and Karteek Alahari. Lara: Latents and rays for multi-camera bird' seyee--ve semantic segmentation. In CoRL, 2022.
[6] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection over-union measure neural networks, In CVPR, 2018.
[7] Alexandre Boulch, Corentin Sautier, Björn Michele, Gilles Puy, and Renaud Marlet. Also: Automotive
lidar self-supervision by occupancy estimation. CVPR, 2022. 7
[8] Alexandre Boulch, B Le Saux, and Nicolas Audebert, Unstructured point cloud semantic labeling using
deep segmentation networks. In Eurographics 2017. 2
[9] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. In
NewrIPS, 2019. 3
[10] Holger Caesar, Varun Bankiti, Alex H Lang, Sourahh Vora, Venice Frin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijhom. nuscenes: A multimodal dataset for autonomous driving.
In CVPR, 2020. 6, 7, 10
[11] Yingjie Cai, Xuesong Chen, Chao Zhang. Kwan- Yee Lin, Xiaogang Wang, and Hongsheng Li. Semantic scene completion via integrating instances and scene in-the-loop. CVPR, 2021.
[12] Anh-Quan Cao and Raoul de Charette. Monoscene: Monocular 3d semantic scene completion. In CVPR,
2022. 2
[13] Eric Chan. Counor Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Samch Khamis, et al. Efficient geometry-aware 3d generative
adversarial networks. In CVPR, 2022. 2
[14] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In CVPR, 2021. 3
[15] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003..422
2020. 9
[16] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. BSP-Net: generating compact meshes via binary space
partitioning. In CVPR, 2020. 2
[17] lan Cherabier, Johannes L Schonberger, Martin R Oswald, Mare Pollefeys, and Andreas Geiger. Learning priors for semantic 3d reconstruction. In ECCV, 2018. 2
convnets: Minkowski [19] Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In
[18] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d
convolutional neural networks. In CVPR, 2019. 2
CVPR, 2021. 3
[20] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Yi Lin. Sealing open- veeabvocayllary image segmentation with image-level labels. In ECCV, 2022, 3, 8
[21]
Albert Gordo and Diane Larlus. Beyond instance-level image retrieval: Leveraging captions to learn a global visual representation for semantic retrieval. In CVPR, 2017. 3
[22] Xiuye Gu. Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022. 6
[23] Tammay Gupta, Arash Vahdat, Gal Chechik, Xiaodong Yang, Jan Kautz, and Derek Hoiem. Contrastive
learning for weakly supervised phrase grounding. In ECCV, 2020. 3
[24] Ping Hu, Stan Sclaroff, and Kate Saenko. Uncertainty aware learning for zero-shot semantic segmentation.
In NewrlPS, 2020. 3

---

[25] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. BEVDet: high high-performance multi-camera
3d object detection in bird-eye-view. arXiv preprint arXiv:2112.1 1790. 2021.
[26]
Yuanhui Huang, Wenzhao Zheng. Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspeetive view for vision-based 3d semantic occupancy prediction. In CVPR, 2023. 1, 2, 7, 8, 10
[27] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen. Shuang Li, Ganesh Iyer. Soroush Saryazdi. Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de
Melo. Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. ConceptFusion: open-set
multimodal 3d mapping. In RSS, 2023. 3
[28] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021. 3
[29]
Justin Kerr, Chung Min Kim, Ken Goldberg. Angjoo Kanazawa, and Matthew Tancik. LERF: language embedded radiance fields. In ICCV, 2023.
[30] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and
Yann LeCun, editors, JCLR, 2015. 7
[31] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tote Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything.
In ICCV, 2023. 7
[32] Felix Järemo Lawin, Martin Danelljan, Patrik Tosteberg. Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg, Deep projective 3d semantic segmentation. In ICA/P, 2017. 2
[33] Boyi Li, Kilian Q Weinberger, Serge Belongic, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In /CLR, 2022. 3,8
[34] Gen Li, Nan Duan, Yuejian Fang. Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross- modal pre-training. In AAAI, 2020. 3
[35] Jie Li, Kai Han, Peng Wang. Yu Liu, and Xia Yuan. Anisotropic convolutional networks for 3d semantic
scene completion. In CVPR, 2020. 2
[36] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu- Vision and language representation learning with momentum distillation. In
Hong Hoi. Align before fuse:
NeurIPS, 2021. 3
[37] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang. Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li.
Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In AAAI, 2023. 2
[38] Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: l.caning bird's-eye-viie s-cye-view representation from multi-camera images via spatiotemporal trans-
formers. In ECCV, 2022. 2
[39] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask- CLIP. In CVPR, 2022. 3 Leland McInnes, John Healy, and Steve Astels. hdbscan: Hicrarchical density based clustering. JOSS,
[40]
2017. 7
[41] Antoine Micch, Jean- bean-Baptiste Alayrae, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020. 3
[42] Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features.
In ECCV, 2018. 3
[43] Songyou Peng. Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Marc Pollefeys, and Thomas Funkhouser. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023. 7,8
[44] Jonah Philion and Sanja Fidler. Lift. splat, shoot: Encoding images from arbiurary camera rigs by implicitly unprojecting to 3d. In ECCV, 2020. 2
[45] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep lcarning on point sets for 3d classification and segmentation. In CVPR, 2017. 2
[46] Charles Ruizhongtai Qi. Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature
learning point in metric space. In NeurIPS, 2017. 2
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from
natural language supervision. In JCML, 2021. 3,
[48]
Christoph B Rist, David Emmerichs, Markus Enzweiler, and Dariu M Gavrila. Semantic scene completion
using local deep implicit functions on lidar data. TPAMI, 2021. 2 Roldao, Raoul de Charette, and Anne Verroust-Blondet. I Lmscnet: Lightweight multiscale 3d semantic
[49]
completion. In 3DV, 2020. 2
[50] Luis Roldao, Raoul De Charette, and Anne Verroust- Blondett. 3d semantic seene completion: A survey. JCV, 2022. 2
[51]
Mert Bülent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In ECCV, 2020. 3
[52] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. In ECCV, 2020. 2
[53] Hugues Thomas, Charles R Qi, Jean-Emmnnuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In ICCV, 2019. 2
12

---

[54] Hao Tian, Yuntao Chen, Jifeng Dai, Zhaoxiang Zhang, and Xizhou Zhu. Unsupervised object detection with lidar clues. In CVPR, 2021.
[55] Antonin Vobecky, David Hurych, Oriane Siméoni, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, and Josef Sivic. Deive&segment: Unsupervised semantic segmentation of urban scenes via cross-modal distillation.
In ECCV, 2022. 3
[56]
Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Feos3d: Fully convolutional one-stage monocular
3d object detection. In ICCV, 2021. 9
[57] Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer. Squeczesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud.
In ICRA, 2019. 2
[58] Shun Chenn Wu, Keisuke Tateno, Nassir Navab, and Federico Tombari. SCFusion: real-time incremental
scene reconstruction with semantic completion. In 3DV. 2020. 2
[59]
Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele. and Zeynep Akata. Semantic projection
network for zero-and few-label semantic segmentation. In CVPR, 2019. 3
[60] Jiarui Xu, Sifci Liu, Arash Vahdate, Wonmin Byeon, Xiaolong Wang, and Shalini De Mcllo. Open- vocabulary panoptic segmentation with rext-to-image diffusion models. In CVPR, 2023. 7,
[61]
Xu Yan, Jiantao Gao, Chaoda Zheng. Chao Zheng, Ruimao Zhang. Shuguang Cui, and Zhen Li. 2dpass:
priors assisted semantic segmentation on lidar point clouds. ECCV. 2022,
[62]
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liumian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- basdd language-image pretraining. In
CVPR, 2022. 3
[63]
Brady Zhou and Philipp Krähenbühh. Cross-view transformers for real-time map-view semantic segmenta-
tion. In CVPR, 2022. 2
[64]
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense lahels from clip. In ECCV, 2022. 2, 3, 7, 8 Xinge Zhu, Hui Zhou, Tai Wang. Fangzhou Hong. Yuexin Ma. Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In CVPR, 2021. 2
[65]
13


Reviews:
[{"review_id": "ZUelt14OxQ", "reviewer": "Reviewer_nph2", "summary": "This work presents a novel approach for zero-shot 3D occupancy prediction for autonomous driving applications. The key idea consists of three parts:\n- a 2D-3D encoder to create a 3D voxel feature grid, based on TPVFormer\n- An MLP-based voxel occupancy predictor that is class agnostic\n- an MLP-based feature encoder, called a \"3D language head\" that takes the TPV-former features that have positive occupancy, and learns to predict the corresponding features of an off-the-shelf MaskCLIP+ model. In effect this is distilling the 2D MaskCLIP+ features into 3D.\n\nAt inference time LiDAR information is not necessary because occupancy is predicted directly from RGB images. Further, the model can be used zero-shot at inference time by computing the similarty of the output of the 3D language head features with language features from a CLIP text encoder. During training, only direct supervision is needed derived from class-agnostic occupancy from the LiDAR data, the distillation loss only needs an off-the-shelf vision language model. The proposed approach obtains ~78% of the fully supervised TPVFormer, while needing no class label supervision.", "questions": "- Why is evaluation limited to only one dataset when there are other datasets that can be used for evaluation?\n- Why aren't existing open-vocabulary segmentation models, backprojected in 3D, used as baselines in addition to MaskCLIP? This is particularly important given the small scale experiments.", "limitations": "The authors have adequately addressed the limitations and the potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n- While there are a variety of works trying to leverage vision-language models, this is the first work to show promising results on open vocabulary 3D occupancy prediction. The proposed model is simple but highly effective taking advantage of existing LIDAR datasets, the TPVFormer architecture, and off the shelf vision language models. The simplicity of this technique opens up the potential for removing the dependence of expensive supervision in this problem domain and instead focusing on extracting information from existing vision-language models.\n\nQuality\n- The main claim of the paper is that the proposed system can perform zero-shot semantic occupancy prediction without using any 3D semantic labels at training time. This paper provides sufficient evidence for this claim by evaluating on nuScenes and obtaining ~78% of  the performance of the fully supervised model. This highlights the power of vision-language models.\n- The related work is detailed and positions the proposed work well relative to previous techniques semantic 3D occupancy prediction, multi-modal learning and open vocabulary segmentation.\n- Section 4.3 presents empirical evidence for the choice of hyperparameters, and the effect of image resolutions.\n\nClarity\n- The paper writing is excellent, it easy to follow and understand. Figure 2 is particularly helpful for following the technical description of the model in Section 3.", "weaknesses": "Quality - small scale evaluation\n- The evaluation is done only on one dataset. This is understandable given the computation costs for training and evaluating models, but having evaluation on more than one dataset would significantly improve the paper. A potential candidate dataset is https://pandaset.org/ which contains LIDAR and 5 wide angle cameras.\n- Evaluation is done only against one open-vocabulary baseline and one supervised basline. While TPVFormer appears to be the only fully supervised reasonable choice, there are baselines like ODISE (https://github.com/NVlabs/ODISE) or OVSeg (https://jeff-liangf.github.io/projects/ovseg/) for open vocabulary segmentation, whose predictions could get back-projected into the class-agnostic occupancy predictions, to obtain 3D semantic labels. These are relevant baselines in light of the comparison with MaskCLIP.\n- Open-vocabulary recognition is powerful because it allows tackling difficult long-tailed scenarios. Currently such a benchmark doesn't exist for occupancy prediction, limiting the evidence to qualitative examples, if we want to evaluate performance beyond the classes defined in nuScenes. It would be a great improvement to the paper if there were some annotated samples to set quantitative benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and suggestions. We address here each comment/question. \n\n**Datasets for evaluation**\n\nIn this work, we leverage TPVFormer for the multi-camera 2D to 3D projection. To show the effect of our contribution compared to a fully-supervised setting in a fair way, we follow the TPVFormer setup (training and evaluating on nuScenes train and val sets, respectively) and train with our self-supervised objective. The design and tuning of neural BEV projection methods (used in TPVFormer) is strongly dependent on the camera rig setup of the ego-vehicle (cameras' intrinsics, extrinsics, type and number, LiDAR density). Switching to another dataset with different sensor configurations would require specific tuning and training time. The rebuttal period was too short for such an experiment, but we would gladly consider the suggestion for extensions of this work.\n\n**Additional baselines**\n\nWe thank the reviewer for the suggestion. We note that ODISE [C] (which was published and made the code available two months prior to the NeurIPs submission deadline) needs for its training the panoptic segmentation mask annotations and OVSeg [D] a segmentation model already pre-trained with segmentation mask annotations. So, both of them use a labor-intensive form of supervision that renders a direct comparison with our unsupervised POP-3D method unfair towards our work. Yet, in Tab. B of the General Response, we provide new results with ODISE. ODISE obtains 34.7 mIoU when we get 26.4 mIoU without human-label supervision. We will include this comparison in the final version of our paper. Regarding OVSeg, we did not have enough time to run it in our evaluation framework, but we would be happy to include it too in the final version of our paper.\n\n**Quantitative benchmark**\n\nWe thank the reviewer for the comment. As suggested, we provide in Table A of the General Response an open-vocabulary natural language 3D grounding and retrieval benchmark, which is limited for now but which we will continue to extend. We can see that our method achieves better results than MaskCLIP+.\n\n[C] Xu et al., Open-vocabulary panoptic segmentation with text-to-image diffusion models, CVPR 2023\n\n[D] Liang et al., Open-vocabulary semantic segmentation with mask-adapted CLIP, CVPR 2023"}, {"review_id": "lbfO4Diltx", "reviewer": "Reviewer_yGBj", "summary": "This paper proposes a camera-only open-vocabulary 3D occupancy prediction method, named POP3D. POP3D consists of a 2D-3D encoder and 3D language heads, which can combine open-vocabulary segmentation with 3D occupancy prediction together. The proposed method can be trained in a self-supervised manner by leveraging images, language, and Lidar modalities. Experiments on the widely used nuScenes benchmark demonstrate the effectiveness of the proposed method.", "questions": "Please see weaknesses.", "limitations": "Please see weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1] The presentation of this paper is clear.\n[2] The motivation of this paper makes sense.\n[3] The proposed method has practical value in auto-labeling system.\n", "weaknesses": "The paper is well-writen overall. One of my concerns is that the evaluation protocol proposed by the authors may not accurately reflect the quality of occupancy prediction. Would you please provide more details about the groud truth generation? (e.g. How many lidar sweeps have you used? Because the Lidar used in nuScenes dataset is sparse than others.) I am concerned that inaccurate evaluations may lead to incorrect conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. We address here the concerns individually.\n\n**Details about ground-truth generation**\n\nWe happily provide more details about the ground-truth generation. We use a single LiDAR sweep (i.e., only a single LiDAR point cloud from a single time step without point cloud aggregation). Then we consider the volume  $[−51.2m,+51.2m]×[−51.2m,+51.2m]×[−5m,+3m]$ around the car and voxelize the space with cubic voxels of 1m size (see Sec. 4.1/Implementation details). Following [7], we cast rays from the LiDAR sensor mounted on the car to the points in the LiDAR point cloud (see Figure 3 in the paper for visualization). We use the same assumption as [7], i.e., we consider all the points along the ray as an empty space. We proceed with setting all the voxels that contain at least one point from the LiDAR point cloud as “occupied”. The remaining voxels are ignored, as we cannot be sure whether they are empty or occupied.\n\nWe understand the concern of the reviewer about the sparsity of the nuScenes LiDAR, which has only 32 beams, hence a low vertical density. However, we argue that this does not lead to inaccurate occupancy predictions because the resolution in the horizontal direction is sufficient and allows the sensor to capture all objects except those extremely thin. "}, {"review_id": "I06AV1tULN", "reviewer": "Reviewer_9pAV", "summary": "The paper presents a novel approach for predicting open-vocabulary 3D semantic voxel occupancy maps from 2D images. The objective is to enable 3D grounding, segmentation, and retrieval of free-form language queries, which is challenging due to the ambiguity between 2D and 3D representations and the difficulty of obtaining annotated 3D training data. The proposed model architecture consists of a 2D-3D encoder, occupancy prediction, and 3D-language heads. It generates a dense voxel map of 3D grounded language embeddings, facilitating various open-vocabulary tasks. The authors also introduce a tri-modal self-supervised learning algorithm that leverages images, language, and LiDAR point clouds to train the model without the need for manual 3D language annotations.", "questions": "Could the authors justify the paper's contribution given the weakness?", "limitations": "Please see weaknesses.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "## Strengths\n- Tri-Modal Self-Supervised Learning: This approach enables training the model using a pre-trained vision-language model without the need for any 3D manual language annotations.\n\n- Experiment results: The authors quantitatively evaluated the model's performance in zero-shot 3D semantic segmentation using existing datasets. Additionally, the model's effectiveness in tasks such as 3D grounding and retrieval of free-form language queries is demonstrated.\n", "weaknesses": "## Weakness:\n- The experiment is unable to demonstrate the advantage of the so-called “Open”-vocabulary. Only a few categories are demonstrated in the experiment. If we only need to understand a few categories, it would be better to just use standard object detection methods that have tens of common categories. At least hundreds of object categories need to be demonstrated to justify the “Open”, which is a key advantage that the authors claimed.\n- The method needs paired lidar-image data. This makes it less generalizable than the masked clip. \n- Although the proposed method is better than masked clip in some known categories, this might not be true for a wider range of categories unseen in training.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and suggestions. We address here each comment/question.\n\n**Evaluating open-vocabulary capabilities on hundreds of object categories**\n\nWe would like to highlight that open-vocabulary approaches for image-based 2D tasks (semantic segmentation, object detection) can show results on a high number of classes also due to the existence of several richly annotated datasets with numerous classes (COCO, LVIS, Objects365), whereas most autonomous driving datasets come with a small number of annotated classes. We emphasize that for the 16 classes we consider in our evaluation, no class information was used during the self-supervised training, making this evaluation open-vocabulary. In contrast, similar approaches in the image domain achieve wider open-vocabulary detection skills by leveraging different forms of annotations, typically for a set of base classes (LSEG [A], Detic [E], OWL-ViT [F], etc.) and are then evaluated on a set of novel classes. POP-3D does not use any human annotation and learns to predict 3D semantic voxel occupancy maps only from LiDAR information and distillation from image-language features.\n\nWe understand the reviewer is asking for the “open” aspect of the method to be more consistently evaluated. We were not able to gather a benchmark with hundreds of object categories in the one week of the rebuttal (no such dataset exists among automotive driving datasets with paired images and LiDAR scans). However, we produce a first small benchmark which we plan to expand. We refer the reviewer to the section \"Language-driven 3D grounding & retrieval evaluation\" in the General Response. We propose a new benchmark that aims to evaluate the \"open\" quality of methods. Although limited in terms of the number of queries, we observe that this preliminary benchmark demonstrates further the good performances of our method and better ones than MaskCLIP+.\n\n**Neccesity of paired LiDAR-image data and generalization compared to MaskCLIP**\n\nWe would like to clarify this point. Indeed, the reviewer is correct in saying that our method POP-3D is trained using LIDAR-image pairs. However, during inference, our method takes **only 2D images as input** (no LiDAR data is used as input) and produces a 3D semantic open vocabulary feature map as output. In contrast, MaskCLIP does not need any paired LiDAR-image data for training but needs such pairs at inference time for the re-projection of features from 2D to 3D space. The MaskCLIP+ baseline is rather an artificial baseline that we proposed to emphasize that POP-3D successfully learns not only to lift and predict image-language features in the 3D space but also to leverage its 3D perception skills in order to produce better predictions. We observe that POP-3D outperforms MaskCLIP+, which needs LiDAR at inference time. We would like to stress that the setup of POP-3D reflects current ways of capturing datasets for autonomous driving [G,H,I,J] and also industry standards, where it is common to have paired image-LiDAR data during the initial data capture in order to validate the perception algorithms (such as depth prediction from camera validated by ground-truth measurements from LiDAR scanner). Such data can be captured using only a small fleet of cars. On the other hand, it is expensive to have LiDAR scanners mounted on every deployed car due to the high price of the sensor.\n\n**Performance on categories unseen in training**\n\nWe would like to clarify that during training, POP-3D does not receive any explicit information about the “known” categories; the model “sees” the pixels of the objects in the images, their corresponding LiDAR points, and MaskCLIP+ features, but no labels. These categories are not known by the model and are given to it only at inference time in the form of text embeddings. We try to answer the lack of comparison in the long-tail classes by collecting a new benchmark for open-vocabulary language-driven 3D grounding and retrieval. We refer the reviewer to the corresponding section in the General Response. Table A there provides results with our method and MaskCLIP+. Our method achieves better retrieval performance.\n\nRegarding object categories that are not seen at all during training, i.e., not present in the images of the nuScenes dataset, we hypothesize that given the performance boosts over the MaskCLIP+ baseline, POP-3D learns to exploit both the structure of the CLIP feature space and the 3D word layout. As a result, it can deal to some extent with such categories depending on how far they are from the nuScenes data distribution.\n\n[E] Zhou et al., Detecting Twenty-thousand Classes using Image-level Supervision, ECCV 2022\n\n[F] Minderer et al., Simple Open-Vocabulary Object Detection with Vision Transformers, ECCV 2022\n\n[G] Sun et al., Scalability in perception for autonomous driving: Waymo open dataset, CVPR 2020\n\n[H] Geiger et al,. Are we ready for autonomous driving? the KITTI vision benchmark suite, CVPR 2012\n\n[I] Behley et al., SemanticKITTI: A dataset for semantic scene understanding of lidar sequences, ICCV 2019\n\n[J] Mao et al., One million scenes for autonomous driving: ONCE dataset, NeurIPS Datasets and Benchmarks 2021\n"}, {"review_id": "d1XnmdYDo4", "reviewer": "Reviewer_Vrgc", "summary": "This work presents a system that is able to achieve open-vocabulary 3D semantic volumetric predictions with multi-view image inputs. Given a set of images, the system learns a 3D voxel grids which decodes into binary occupancy and visual-linguistic embeddings. The embeddings are used for retrieval and semantic segmentation given language inputs. Comparisons are present against the fully-supervised TPVFormer and  MaskCLIP+ on nuScenes.", "questions": "see weankess", "limitations": "no issue found", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper is well-written, easy to read, and materials are clearly presented.\n- The presented system should be quite useful in downstream applications especially for outdoor driving scenarios.\n- The presented method is reasonable, well designed, and is new.", "weaknesses": "- The paper only compares to MaskCLIP+ as the zero-shot baseline (TPVFormer is fully supervised), which is insufficient, given the abundant related works like Lerf/Semantic-abs/OpenScene: 3D Scene Understanding with Open Vocabularies /CLIP-FO3D: Learning Free Open-world 3d Scene Representations from 2D Dense CLIP. Although these work may use NeRF/point clouds as the 3D representation, the reviewer finds the core technical challenges are shared.\n- Given these previous works, which are not even cited and discussed, I would consider the technocal novelity of this work is low.\n- No qualitative comparison to maskclip+/TVPFormer. Please show figures.\n\n\nSome minor questions\n- what's the design for f_3d? No explanation given for this image->3D lifting backbone.\n- In line 162, how about some points are occupied but not observed in any view?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback.\n\n**Comparison to other open-vocabulary methods: LeRF, Semantic-Abs, OpenScene, CLIP-FO3D**\n\nThank you for suggesting relevant works which tackle a similar problem to ours. We will integrate a discussion about those works in our related work section. \n\nFor now, we would like to point out important differences between each one of them and our POP-3D method. The first, LeRF, is a NeRF-based method, which needs to be trained on each scene independently and does not generalize to novel scenes. CLIP-FO3D is a 3D network processing point clouds and RGB images and learns to imitate MaskCLIP features via pairs of RGB images aligned with dense point clouds as input during inference, whereas we produce 3D open-vocabulary semantic occupancy maps only from images. OpenScene learns a representation of 3D point clouds and is therefore limited to the availability of 3D scanners at inference time, which is too costly to be mounted on every car. Moreover, the CLIP-like features in OpenScene are obtained using either LSeg [A] (which requires ground-truth semantic image annotations) or OpenSeg [B] methods (which requires class-agnostic segmentation masks and image captions) which both need much more annotation than MaskCLIP+ (which requires no additional annotation) exploited in our POP-3D. Therefore, it is expected that LSeg and OpenSeg features provide better pixel-text alignment. In Table B of the General Response, we provide results using the OpenScene method (adapted to our evaluation framework and without test-time augmentations). OpenScene achieves better results than our method.  Again, we would like to stress that OpenScenes uses image-language encoders that were trained with much more supervision than MaskCLIP+, which we use in our experiments. At the same time, our method is agnostic to the used image-language encoder and could hence exploit encoders from LSeg and OpenSeg (as OpenScene does) to improve its performance.\n\nWe would be interested to have the full reference to the \"Semantic-abs\" work mentioned by the reviewer, as we were not able to find it.\n\n**Technical novelty**\n\nWe would like to point out the tight/impossible timeline here, which is the reason why we were not aware/couldn't compare to works suggested by the reviewer. Although we agree those works are relevant, two were published on arXiv in March 2023 (CLIP-FO3D, LERF) (which is below the two months suggested by NeurIPS guidelines), and OpenScene's (presented at CVPR23, which happened in June 2023) code was provided only in mid-March 2023. \n\nNonetheless, as mentioned above, we thank the reviewer for pointing us out to those works and will discuss them in the related work section of the paper. \n\n**Qualitative results for MaskCLIP+ and TPVFormer**\n\nWe provide an additional qualitative comparison of our POP-3D framework to the fully-supervised TPVFormer and to MaskCLIP+ results projected from 2D to 3D ground-truth point cloud. These qualitative results can be found in the PDF within the General Response.\n\n**Design of f_3d image-to-3D lifting**\n\nWe use the TPVFormer [26] model, which builds upon the popular BEVFormer bird's-eye-view (BEV) lifting method [37]. BEVFormer is an attention-based lifting method using points from the BEV grid as queries for fetching visual information from the image encoder features of the different cameras. TPVFormer generalizes BEVFormer to the 3D space in a computationally efficient manner via a tri-perspective view (TPV) representation. Three axis-aligned orthogonal TPV planes (HW, DH, WD, with H,W,D denoting the resolution of the three planes across height, width, and depth dimensions) are learned with BEVFormer lifting. Voxels are modeled in the 3D space by summing their projected features on the three planes. We give information about the setup of the 3D backbone in the \"Implementation details\" subsection and will expand the provided description. Additional information can be found in the original TPVFormer paper.\n    \nWe would like to highlight that POP-3D is not specifically designed for use with TPVFormer backbone and could be used with other image-to-3D lifting strategies as long as they produce voxel-level representations, which is the case for all recent methods.\n\n**Occupied invisible points**\n\nFollowing the original implementation of TPVFormer, we set such voxels as empty. We are aware of potentially different solutions for setting such voxels as ignored, but we did not yet test this training setup."}, {"review_id": "NjlazBFM4i", "reviewer": "Reviewer_q7fa", "summary": "This paper proposes an approach, POP-3D, to achieving open-vocabulary 3D occupancy prediction from images by aligning 3D features with those 2D features from pre-trained MaskCLIP+. Specifically, in contrast to previous occupancy prediction methods, such as TPVFormer, this framework uses two heads to predict class-agnostic occupancy and semantic-related features, respectively and then uses the features to perform language-grounded open-vocabulary 3D perception. To evaluate the effectiveness of the method, the paper also introduces a new evaluation method tailored to the 3D occupancy prediction task. Promising experimental results show the efficacy of the proposed method.", "questions": "None", "limitations": "The authors have mentioned the potential limitations and provide simple clarifications and possible solutions.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The basic idea is easy to follow and the illustration figures are clear.\n- The motivation to address the problems of open-vocabulary 3D occupancy prediction is good and the proposed method is effective.\n- The design of decomposed heads for geometry and semantic prediction is reasonable.\n- The proposed self-supervised learning method does not need 3D manual annotations while achieving comparable performance with supervised methods.\n- The experiments compared with MaskCLIP+ baseline are interesting, giving a more comprehensive position of the proposed method among different solutions for the 3D occupancy prediction task.", "weaknesses": "- (Method) Although the proposed method can achieve open-vocabulary 3D perception, the basic capability is mainly borrowed from 2D features and the pre-trained MaskCLIP+. It is one of the reasonable solutions for open-vocabulary 3D perception but I have to say there is nothing new for exploring how to achieve that from 3D data. I can understand that this is also related to the task setting in this paper is 3D occupancy prediction from \"images\", but it makes the studied problem more like a \"pseudo-3D open-vocabulary\" one.\n\n- (Problem Setting) While this paper shows the open-vocabulary capability of the proposed method with the zero-shot occupancy prediction results and the visualization of the case study, it still does not provide a clearly useful background to perform 3D open-vocabulary occupancy prediction in \"driving scenarios\" (there are typically very few categories of interest, which are almost enough for safe planning and control), or at least provide a good playground/annotations to study this problem. This significantly weakens the value of this paper, because the foundation of studying this problem is not very clear, especially when this benchmark is not officially set up or popular enough such that we do not need to justify this.\n\n- (Evaluation) The introduced evaluation protocol mainly considers the weaknesses of \"sparse\" semantic occupancy prediction in TPVFormer and adds empty labels along a lidar ray. However, such problems do not exist in many recent occupancy benchmarks [1][2][3], where dense occupancy labels are available. Therefore, I think the contribution of the new evaluation protocol is a little incremental and does not solve the mentioned problem fundamentally compared to these new occupancy benchmarks.\n\n[1] Tian et. al., Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving\n\n[2] Tong et. al., Scene as Occupancy\n\n[3] Wei et. al., SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. \n\n**Method novelty**\n\nWe would like to thank the reviewer for the comment. Although it is true that POP-3D uses image-text aligned features provided by MaskCLIP+, we argue that exploiting such features is an interesting and non-trivial starting point to explore the hard task of Open Vocabulary 3D semantic occupancy.\n\nThere are indeed several recent works that distill CLIP features into another modality, e.g., audio, point clouds, etc., to inherit CLIP’s open-vocabulary capabilities. While POP-3D exploits information from LiDAR to understand the 3D geometry of a scene, the supervisory signal comes from the image domain where large-scale language-image pre-training datasets (such as those used for training CLIP) are available, it remains an image-based method. We argue that the novelty and difficulty of our endeavor is in reaching an effective interplay between the complementary information coming from multiple camera images, LiDAR geometry, and language-image features in the complex outdoor 3D space surrounding the vehicle, without using any manual labels. The quantitative results showing performance boosts in occupancy estimation (IoU - Figure 4b) and close performance on semantic predictions (mIoU - Figure 4b), while outperforming original MaskCLIP+ features (Figure 4a), confirm that POP-3D does not just imitate image-language features, but reaches non-trivial 3D perception capabilities. \n\n**Motivation of open vocabulary in driving scenarios**\n\nThe need for open vocabulary in driving scenarios is essential for statistical system validation. In contrast to the standard research dataset setup (majority of data for training, minority for validation and testing), testing, called system validation, of the Advanced Driving Assistance Systems (ADAS) requires vastly more data captures than training. As an example, a level 3 system validation requires tens of thousands of hours of real-world driving in order to satisfy requirements based on functional safety standards defined in ISO 26262. Open-vocabulary functionality is essential to understand the contents of the data for: (a) being able to find out if specific long tail distribution object classes are present in the data based on system requirements and (b) having the ability to recognize the root causes of failures (e.g., specific object types on which false positives appear), in order to have feedback to models training and adaptation to improve performance and increase safety. In short, the behavior of ADAS on objects from classes unseen at training time needs to be thoroughly tested for safety reasons, and such object classes are unknown in advance.\n\nWe argue that there are many non-frequent types of objects in driving scenarios that are essential for the autonomous driving perception system to work under all conditions. To support this, we follow the suggestion of the reviewer and prepare a small benchmark to study this problem. We refer the reviewer to the section “Language-driven 3D grounding & retrieval evaluation” in the General Response. So far, we gathered seven scenes with non-frequent objects, such as excavators, large trash bin, or police car. We present the results in Table A of the General Response. We aim to enlarge this benchmark and publish it alongside the final paper. We hope that this can serve as a playground to study this problem. \n\n**Limitation of evaluation protocol and new occupancy benchmarks**\n\nWe agree with the comments of the reviewer regarding the limitations of our evaluation protocol. We thank the reviewer for the pointers to the recent and elaborate benchmarks with dense labels that are relevant to our work. However, we would like to highlight that the suggested benchmarks were published on arXiv in March 2023 (SurroundOcc [3]), April 2023 (Occ3D [1]), and June 2023 (OccNet [2]), so below the two-month period suggested by NeurIPS guidelines. \n\nDue to differences in the granularity of the voxel size produced by POP-3D and the multiple ones from the benchmarks we were unable to adapt and evaluate our method for such an experiment during the short rebuttal time. However, we plan to improve our evaluation protocol by leveraging these new findings and conduct evaluations on them for the updated version of the paper."}]

Author rebuttal:
We thank the reviewers for their valuable comments, which will allow us to improve the quality of the paper. 

We are glad to note that they appreciated the clarity [*nph2, yGBj, Vrgc, q7fa*] of the manuscript, the well-defined motivation of the tackled problem [*yGBj, q7fa*], the originality [*nph2, Vrgc*] and soundness [*9pAV, Vrgc, q7fa*] of the method, its relevance [*9pAV, nph2, yGBj, Vrgc, q7fa*], and the quality of the results [*nph2, 9pAV, q7fa*]. 


In order to address several comments on the experimental results, we report here additional quantitative and qualitative comparisons. First, we propose a new *open-vocabulary language-driven 3D grounding and retrieval* benchmark, which we plan to further extend and present in the revised paper *[nph2-BA, 9pAV-BR, q7fa-BA]*. Second, we compare to more recent related works, as required by reviewers *[nph2, Vrcg]*, even if they require much more manual annotation than ours.


## Language-driven 3D grounding & retrieval evaluation

As suggested by the reviewers, we have built a small (due to limited time) benchmark to evaluate the "open" capability of our method quantitatively. 

### The benchmark
We have collected an initial version of our *Language-driven 3D grounding & retrieval* benchmark with natural language queries. To build this benchmark, we have manually annotated 3D scenes from the validation split of the nuScenes dataset for a set of natural language open vocabulary queries. This initial set contains 7 queries, due to the time limitation of the rebuttal, but we continue to extend it. The spatial grounding for each query was obtained by manually annotating the relevant set of voxels in the scene. The objective is, given the query, to retrieve all relevant voxels in the scene. Results are evaluated using the precision-recall curve;  negative data are all the non-relevant voxels/points in the given scene. In the table below, we report the average precision (AP) for each query, further aggregated into a mean average precision (mAP) score corresponding to the mean performance across all queries. We believe this dataset, when further extended, would be an important step toward open-vocabulary analysis of driving scenes. 

### How do we form the text queries
The text category descriptions used as queries in every experiment are shown in the table below. To get feature corresponding to the query text description, we follow the same approach as presented in the main paper.

| experiment name | category text description | MaskCLIP+ | POP-3D (ours) |
| ---------------:| ------------------------: | ---------:| -------------:|
|     excavator-0 | excavator                 |      14.0 |          12.8 |
|      delivery-0 | delivery vehicle          |       6.9 |           9.9 |
|      delivery-1 |                           |      15.2 |          15.5 |
|      delivery-2 |                           |      63.9 |          65.5 | 
|        police-0 | police car                |      54.1 |          55.6 |
|     trash-bin-0 | trash bin                 |       9.7 |          12.6 |
|       backhoe-0 | backhoe                   |       3.3 |           4.1 |
|        **mean** |                           |  **23.9** |      **25.1** |

**Table A: AP results on our Language-driven 3D grounding & retrieval benchmark**

### Discussion about the results

We compare the performance of our method with MaskCLIP+ and report results in the table above. Our approach exhibits superior AP outcomes for 6 out of the 7 natural language queries. Overall, our POP-3D method attains a mAP of 25.1, surpassing MaskCLIP+'s mAP of 23.9.

 
## Comparison to more related works

We extend Fig. 4 of the main paper with more related works, which we present below in Table B. We additionally evaluate the ODISE (CVPR'23) and OpenScene (CVPR'23) methods on our task. Both methods can only be evaluated with the LiDAR-based evaluation because they do not produce 3D occupancy. We denote "2D->3D" in the table the setup where we assign 2D features to the 3D points projected into the images.

We see that both ODISE and OpenScene outperform previously reported results. However, they both use manual annotation while we don't. For instance, ODISE requires panoptic segmentation annotations for training, while OpenScene uses features from either LSeg [A] or OpenSeg [B], which are two image-language encoders that are trained with supervision. We compare here to OpenScene with OpenSeg [B] (noted "OpenScene-OS"), which requires both class-agnostic segmentation masks and image captions.

Note that we cannot evaluate MaskCLIP+, ODISE, and OpenScene for the task of image-based 3D occupancy prediction since they work either only in the 2D space (MaskCLIP+ and ODISE) or directly in the space of 3D point clouds (OpenScene), therefore do not predict the occupancy from images.

| Method                           | Type of supervision | LiDAR-based evaluation | 3D occupancy prediction from images (mIoU/IoU) |
| -------------------------------- | --------| ---------------------- | ----------------------- |
| MaskCLIP+ (2D->3D) |  none | 23.0 |      not adapted                   |
| POP-3D (ours)                    | none| 26.4                  |    16.7 / 37.7                |
| TPVFormer (supervised)           | full LiDAR segmentation | 31.3                      |  21.3 / 26.2 |
| ODISE (2D->3D)     | panoptic masks annotations | 34.7                      |        not adapted                  |
| OpenScene-OS (ensemble of 3D and 2D->3D features)                               | w. OpenSeg encoder trained w. class-agnostic seg. masks & image captions  |           38.8             |              not adapted             | 

**Table B: Extension of Fig. 4 of the main paper with more related works**

[A] Li et al., Language-driven Semantic Segmentation, ICLR 2022

[B] Ghiasi et al., Scaling Open-Vocabulary Image Segmentation with Image-Level Labels, ECCV 2022

Meta-review: The submission initially received mixed reviews. The authors did a good job during the discussion period, after which all reviewers became positive. The AC agrees with the recommendation. However, the authors are strongly encouraged to incorporate the feedback from the reviewers in the camera-ready version.
Decision: Accept (poster)