SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Antonin Vobecky1,2,3
Oriane Simeoni
David Hurych
Spyros Gidaris<sup>1</sup>
Andrei Bursuc<sup>1</sup>
Patrick Pérez<sup>1</sup>
Josef Sivic
$^{1}$  valeo.ai, Paris, France
2 CIIRC CTU in Prague
3 FEE CTU in Prague
We describe an approach to predict open-vocabulary 3D semantic voxel occupancy map from input 2D images with the objective of enabling 3D grounding, segmentation and retrieval of free-form language queries. This is a challenging problem because of the 2D-3D ambiguity and the open-vocabulary nature of the target tasks, where obtaining annotated training data in 3D is difficult. The contributions of this work are three-fold. First, we design a new model architecture for open-vocabulary 3D semantic occupancy prediction. The architecture consists of a 2D-3D encoder together with occupancy prediction and 3D-language heads. The output is a dense voxel map of 3D grounded language embeddings enabling a range of open-vocabulary tasks. Second, we develop a tri-modal self-supervised learning algorithm that leverages three modalities: (i) images, (ii) language and (iii) LiDAR point clouds, and enables training the proposed architecture using a strong pre-trained vision-language model without the need for any 3D manual language annotations. Finally, we demonstrate quantitatively the strengths of the proposed model on several open-vocabulary tasks: Zero-shot 3D semantic segmentation using existing datasets; 3D grounding and retrieval of free-form language queries, using a small dataset that we propose as an extension of nuScenes. You can find the project page here https://vobecant.github.io/POP3D.
The detailed analysis of 3D environments –both geometrically and semantically– is a fundamental perception brick in many applications, from augmented reality to autonomous robots and vehicles. It is usually conducted with cameras and/or laser scanners (LiDAR). In its most complete version, called semantic 3D occupancy prediction, this analysis amounts to labelling each voxel of the perceived volume as occupied by a certain class of object or empty. This is extremely challenging since both cameras and LiDAR only capture information about visible surfaces, which may be projected from 3D into 2D without the loss of information, but not for every point in the 3D space. This one extra dimension makes prediction arduous and hugely complicates the manual annotation task.
Recent works, e.g., [26], propose to leverage manually-annotated LiDAR data to produce a partial annotation of the 3D occupancy space. However, relying on manual semantic annotation of point clouds remains difficult to scale, even if sparse, and limits the learned representation to encode solely a closed vocabulary, i.e., a limited predefined set of classes. In this work, we tackle these challenges and propose an open-vocabulary approach to 3D semantic occupancy prediction that relies only on
$^{2}$ Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague
Figure 1: Overview of the proposed method. Provided only with surround-view images as input, our model called POP-3D produces a voxel grid of 3D text-aligned features that support open-vocabulary downstream tasks such as zero-shot occupancy segmentation or text-based grounding and retrieval.
unlabeled image-LiDAR data for training. In addition, our model uses only camera inputs at run time, bypassing altogether the need for expensive dense LiDAR sensor, in contrast with most 3D semantic perception systems (whether at point or voxel level).
To this end, we harness the progress made recently in supervised 3D occupancy prediction [26] and in language-image alignment [64], within a two-head image-only model that can be trained with aligned image-LiDAR raw data. Leveraging sparse 3D occupancy information that LiDAR scans provide for free, we first train a class-agnostic occupancy prediction head. Using this same LiDAR information along with pre-trained language-aligned visual features at the corresponding locations in images, we jointly train a second head that predicts the same type of features at the 3D voxel level. At run time, these features can be probed from text prompts to get open-vocabulary semantic segmentation of voxels that are predicted as occupied (Fig. 1). To assess the effectiveness of our method for semantic 3D occupancy prediction, we introduce a novel evaluation protocol specifically tailored to this task. Through evaluation with this protocol on autonomous driving data, our method is shown to achieve a strong performance relative to the fully-supervised approach.
In a nutshell, we attack the difficult problem of 3D semantic occupancy prediction with the lightest possible requirements: no manual annotation of the training data, no pre-defined semantic vocabulary, and no recourse to LiDAR readings at run time. As a result, the proposed image-only 3D semantic occupancy model named P0P-3D (for oPen-vocabulary Occupancy Prediction in 3D) provides training data scalability and operational versatility, while opening up new understanding capabilities for autonomous systems through language-driven scene perception.
Semantic 3D occupancy prediction. Automatic understanding of the 3D geometry and semantics of a scene has been traditionally enabled through high precision LiDAR sensors and corresponding architectures. 3D semantic segmentation, i.e., point-level classification of a point cloud, can be addressed with different types of transformations of the point cloud: point-based, directly operating on the three-dimensional points [45, 46, 53], and projection-based, operating on a different representation, e.g., two-dimensional images [57, 32, 8] or three-dimensional voxel representations [61, 65, 52, 18]. However, they produce predictions as sparse as the LiDAR point cloud offering an incomplete understanding of the full scene. Semantic scene completion [50] aims for dense inference of 3D geometry and semantics of objects and surfaces within a given extent, typically leveraging rich geometry information at the input extracted from depth [16, 35], occupancy grids [58, 49], point clouds [48], or a mix of modalities, e.g., RGBD [11, 17]. In this line, MonoScene [12] is the first camera-based method to produce dense semantic occupancy predictions from a single image by projecting image features into 3D voxels by optical ray intersection. Recent progress in multicamera Bird's-Eye-View (BEV) projection [44, 25, 63, 38, 5, 37] enables the recent TPVFormer [26] to generate surrounding 3D occupancy predictions by effectively exploiting tri-perspective view representations [13] augmenting the standard BEV with two additional perpendicular planes to recover the full 3D. All prior methods are trained in a supervised manner requiring rich voxel-level semantic information, which is costly to curate and annotate. While we build on [26], we forego manual label supervision and, instead, develop a model able to produce semantic 3D occupancy
predictions using supervision from LiDAR and from an image-language model allowing our model to acquire open-vocabulary skills in the voxel space.
Multi-modal representation learning. Distilling signals and knowledge from one modality into another is an effective strategy to learn representations [2, 3] or to learn to solve tasks using only few [14, 42, 1] or no human labels [54, 55]. The interplay between images, language and sounds is often used for self-supervised representation learning over large repositories of unlabeled data fetched from the internet [2-4, 41, 42]. Images can be paired with different modalities towards solving complex 2D tasks, e.g., semantic segmentation [55], detection of road objects [54] or sound-emitting objects [14, 42, 1]. Image-language aligned models project images and text into a shared representation space [21, 51, 34, 36, 19, 47, 28]. Contrastive image-language learning on many millions of image-text pairs [47, 28] leads to high-quality representations with impressive zero-shot skills from one modality to the other. We use CLIP [47] for its appealing open-vocabulary property that enables the querying of visual content with natural language toward recognizing objects of interest without manual labels. P0P-3D uses LiDAR supervision for precise occupancy prediction and learns to produce in the 3D space CLIP-like features easily paired with language.
Open-vocabulary semantic segmentation. The aim of zero-shot semantic segmentation is to segment object classes that are not seen during training [59, 9, 24]. The advent of CLIP [47], which is trained on abundant web data, has inspired a new wave of methods, dubbed open-vocabulary, for recognizing random objects via natural language queries. CLIP features can be projected into 3D meshes [27] and NeRFs [29] to enable language queries. Originally producing image-level embeddings, CLIP can be extended to pixel-level predictions for open-vocabulary semantic segmentation by exploiting different forms of supervision from segmentation datasets, e.g., pixel-level labels [33] or class agnostic masks [20, 39, 62] coupled with region-word grounding [23], however with potential forgetting of originally learned concepts [27]. MaskCLIP+ [64] adjusts the attentive-pooling layer of CLIP to generate pixel-level CLIP features that are further distilled into an encoder-decoder semantic segmentation network. MaskCLIP+ [64] preserves the open-vocabulary properties of CLIP, and we exploit it here to distill its knowledge into P0P-3D. We generate target 3D CLIP features by mapping MaskCLIP+ pixel-level features to LiDAR points observed in images. By being trained to match these distillation targets, P0P-3D manages to learn 3D features with open-vocabulary perception abilities, in contrast to prior work on 3D occupancy prediction that is limited to recognizing a closed-set of visual concepts.
3 Open-vocabulary 3D occupancy prediction
Our goal is to predict 3D voxel representations of the environment, given a set of 2D input RGB images, that is amenable to open-vocabulary tasks such as zero-shot semantic segmentation or concept search driven by natural language queries. This is a challenging problem as we need to address the following two questions. First, what is the right architecture to handle the 2D-to-3D ambiguity and the open-vocabulary nature of the task? Second, how to formulate the learning problem without requiring manual annotation of large amounts of 3D voxel data, which are extremely hard to produce.
To address these questions we propose the following two innovations. First, we design an architecture for open-vocabulary 3D occupancy prediction (Fig. 2(a) and Sec. 3.1) that handles the 2D-to-3D prediction and open-vocabulary tasks with two specialized heads. Second, we formulate its training as a tri-modal self-supervised learning problem (Fig. 2(b) and Sec. 3.2) that leverages aligned (i) 2D images with (ii) 3D point clouds equipped with (iii) pre-trained language-image features as the three input modalities (i.e. camera, LiDAR and language) without the need for any explicit manual annotations. The details of these contributions are given next.
3.1 Architecture for open-vocabulary 3D occupancy prediction
We are given a set of surround-view images captured from one camera location and our goal is to output a 3D occupancy voxel map and to support language-driven tasks. To reach the goals, we propose an architecture composed of three modules (Fig. 2(a)). First, a  $2D-3D$  encoder predicts a voxel feature grid from the input images. Second, the occupancy head decodes this entire voxel grid into an occupancy map, predicting which voxels are free and which are occupied. Finally, the 3D-language head is applied on each occupied voxel to output a powerful language embedding vector enabling a range of 3D open-vocabulary tasks. The three modules are described next.
Figure 2: Proposed approach. In (a), we show the architecture of the proposed method. Having only surround-view images on the input, the model first extracts dense voxel feature grid that is then fed to two parallel heads: occupancy head  $g$  producing voxel-level occupancy predictions, and to 3D-language feature head  $h$  which outputs features aligned with text representations. In b), we show how we train our approach, namely the occupancy loss  $\mathcal{L}_{\mathrm{occ}}$  used to train class-agnostic occupancy predictions, and the feature loss  $\mathcal{L}_{\mathrm{ft}}$  that enforces the 3D-language head  $h$  to output features aligned with text representations.
2D-to-3D encoder  $f_{3\mathrm{D}}$ . The objective of the 2D-to-3D encoder is to predict a dense feature voxel grid given one or more images captured at one location as input. The output voxel grid representation encodes 3D visual information captured by the cameras. In detail, given surround-view camera RGB images I and camera calibration parameters, the encoder  $f_{3\mathrm{D}}$  produces a feature voxel grid
$$
\mathbf {V} = f _ {\mathrm {3 D}} (\mathbf {I}) \in \mathbb {R} ^ {H _ {\mathrm {v}} \times W _ {\mathrm {v}} \times D _ {\mathrm {v}} \times C _ {\mathrm {v}}}, \tag {1}
$$
where  $H_{\mathrm{V}}, W_{\mathrm{V}}$ , and  $D_{\mathrm{V}}$  are the spatial dimensions of the voxel grid, and  $C_{\mathrm{V}}$  is the feature dimension of each voxel. This feature voxel grid is then passed to two distinct prediction heads designed to perform class-agnostic occupancy prediction and text-aligned feature prediction tasks respectively. The two heads are described next.
Occupancy head  $g$ . Given the feature voxel grid  $\mathbf{V}$ , the occupancy prediction head  $g$  aims at classifying every voxel as 'empty' or 'occupied'. Following [26], this head is implemented as a non-linear network composed of  $N_{\mathrm{occ}}$  hidden blocks with configuration Linear-Softplus-Linear, each with  $C_{\mathrm{occ}}^{\mathrm{hidden}}$  hidden features, and a final linear classifier outputting two logits, one per class. It outputs the tensor
$$
\mathbf {O} _ {\mathrm {o c c}} = g (\mathbf {V}) \in \mathbb {R} ^ {H _ {\mathrm {V}} \times W _ {\mathrm {V}} \times D _ {\mathrm {V}} \times 2}, \tag {2}
$$
containing the occupancy prediction for each voxel.
3D language head  $h$ . In parallel, the voxel grid  $\mathbf{V}$  is fed to a language feature extractor. This head processes each voxel feature to output an embedding vector that is aligned to vision-language representations, such as CLIP [47], aiming to inherit their open-vocabulary abilities. This allows us to address the limitations of closed-vocabulary predictions encountered in supervised 3D occupancy prediction models, which are bound to a set of predefined visual classes. In contrast, our representation enables us to perform 3D language-driven tasks such as zero-shot 3D semantic segmentation. Similarly to the occupancy head, the 3D-language head consists of  $N_{\mathrm{ft}}$  blocks with configuration Linear-Softplus-Linear, where each linear layer outputs  $C_{\mathrm{ft}}^{\mathrm{hidden}}$  features, and a final linear layer that outputs  $C_{\mathrm{ft}}^{\mathrm{out}}$ -dimensional vision language embedding for each voxel. It outputs the tensor
$$
\mathbf {O} _ {\mathrm {f t}} = h (\mathbf {V}) \in \mathbb {R} ^ {H _ {\mathrm {v}} \times W _ {\mathrm {v}} \times D _ {\mathrm {v}} \times C _ {\mathrm {f t}} ^ {\text {o u t}}}, \tag {3}
$$
containing the predicted vision-language embedding of each voxel.
3.2 Tri-modal self-supervised training
The goal is to train the network architecture described in Sec. 3.1 to predict the 3D occupancy map together with language-aware features for each occupied voxel. In turn, this will enable 3D open-vocabulary tasks such as 3D zero-shot segmentation or language-driven search. The main challenge is obtaining the appropriate 3D-grounded language annotations, which is expensive to do manually. Instead, we propose a tri-modal self-supervised learning algorithm that leverages three modalities: (i) images, (ii) language and (iii) LiDAR point clouds. Specifically, we employ a pre-trained image-language network to generate image-language features for the input images. These features are then mapped to the 3D space using registered LiDAR point clouds, resulting in 3D grounded image-language features. These grounded features serve as training targets for the network. The training algorithm is illustrated in Fig. 2(b). The training is implemented via two losses that are used to train the two heads of the proposed architectures jointly with the 2D-to-3D encoder. The details are given next.
Occupancy loss. We guide the occupancy head  $g$  to perform a class-agnostic occupancy prediction by the available unlabeled LiDAR point clouds, which we convert to occupancy prediction targets  $T_{\mathrm{occ}} \in \{0,1\}$ . Each voxel location  $x$  containing at least one LiDAR point is labeled as 'occupied' (i.e.,  $T_{\mathrm{occ}}(x) = 1$ ) and as 'empty' otherwise ( $T_{\mathrm{occ}}(x) = 0$ ). Having these targets, we supervise the occupancy prediction head densely at all locations of the voxel grid. The occupancy loss  $\mathcal{L}_{\mathrm{occ}}$  is a combination of cross-entropy loss  $\mathcal{L}_{\mathrm{CE}}$  and Lovász-softmax [6] loss  $\mathcal{L}_{\mathrm{Lov}}$ :
$$
\mathcal {L} _ {\text {o c c}} \left(\mathbf {O} _ {\text {o c c}}, \mathbf {T} _ {\text {o c c}}\right) = \mathcal {L} _ {\text {C E}} \left(\mathbf {O} _ {\text {o c c}}, \mathbf {T} _ {\text {o c c}}\right) + \mathcal {L} _ {\text {L o v}} \left(\mathbf {O} _ {\text {o c c}}, \mathbf {T} _ {\text {o c c}}\right), \tag {4}
$$
where  $\mathbf{O}_{\mathrm{occ}}$  is the predicted occupancy tensor and  $\mathbf{T}_{\mathrm{occ}}$  the tensor of corresponding occupancy targets.
Image-language distillation. Unlike the occupancy prediction head that is supervised densely at the level of voxels, we supervise the 3D-language head at the level of points  $p_n \in P_{\mathrm{cam}}$  which project to at least one of the cameras, i.e.,  $P_{\mathrm{cam}} \subset P$ , where  $P$  is the complete point cloud. This is required in order to obtain feature targets from the language-image pre-trained model  $f_{\mathrm{I}}$ .
To get a feature target for a 3D point  $p_n \in P_{\mathrm{cam}}$  in the voxel feature grid, we use the known camera projection function  $\Pi_c$  that projects 3D point  $p_n$  into 2D point  $u_n = (u_n^{(x)}, u_n^{(y)})$ , where  $(u_n^{(x)}, u_n^{(y)})$  are  $(x, y)$  coordinates of point  $u_n$  in camera  $c$ :
$$
u _ {n} = \Pi_ {c} \left(p _ {n}\right). \tag {5}
$$
This way, we get a set of 2D points  $U = \{\Pi_c(p_n)\}_{n=1}^N$  in the camera coordinates. To obtain feature targets  $\mathbf{T}_{\mathrm{ft}}$  for 3D points in  $P_{\mathrm{cam}}$  with corresponding 2D projections  $U$  in camera  $c$ , we run the language-image-aligned feature extractor  $f_1$  on image  $\mathbf{I}_c$ , and use the 2D projections' coordinates to sample from the resulting feature map, i.e.,
$$
\mathrm {T} _ {\mathrm {f t}} = \left\{f _ {\mathrm {I}} \left(\mathbf {I} _ {c}\right) \left[ u _ {n} ^ {(x)}, u _ {n} ^ {(y)} \right] \right\} _ {n = 1} ^ {N} \in \mathbb {R} ^ {N \times C _ {\mathrm {f t}} ^ {\text {o u t}}}, \tag {6}
$$
where  $[x,y]$  is an indexing operator in the extracted feature map.
To train the 3D language head, we use  $L_{2}$  mean squared error loss between the targets  $\mathbf{T}_{\mathrm{ft}}$  and the predicted features  $\tilde{\mathbf{O}}_{\mathrm{ft}} \in \mathbb{R}^{N \times C_{\mathrm{ft}}^{\mathrm{out}}}$  computed from  $h$  for the 3D point locations in  $P_{\mathrm{cam}}$ :
$$
\mathcal {L} _ {\mathrm {f t}} = \frac {1}{N C _ {\mathrm {f t}} ^ {\text {o u t}}} \left\| \mathbf {T} _ {\mathrm {f t}} - \tilde {\mathbf {O}} _ {\mathrm {f t}} \right\| ^ {2}, \tag {7}
$$
where  $\| \cdot \|$  is the Frobenius norm.
Final loss. The final loss used to train the whole network is a weighted sum of the occupancy and image-language losses. We use a single hyperparameter  $\lambda$  to balance the weighting of the two losses:
$$
\mathcal {L} = \mathcal {L} _ {\text {o c c}} + \lambda \mathcal {L} _ {\mathrm {f t}}. \tag {8}
$$
3.3 3D open-vocabulary test-time inference
Once trained, as described in Sec. 3.2, our model supports different 3D open-vocabulary tasks at test-time. We focus on the following two: (i) zero-shot 3D semantic segmentation and (ii) language-driven 3D grounding.
Zero-shot 3D semantic segmentation from images. Given an input test image, the 3D-text-aligned voxel features produced by our model support zero-shot 3D segmentation for a target set of classes specified via input text queries (prompts), as illustrated in Fig. 1. Unlike supervised approaches that necessitate retraining when the set of target classes changes, our approach requires training the model only once. We can adjust the number of segmented classes effortlessly by providing a different set of input text queries. In detail, at test-time we proceed along the following steps. First, a set of test surround-view images  $\mathbf{I}$  from one location is fed into the trained P0P-3D network, resulting in class-agnostic occupancy prediction  $\mathbf{O}_{\mathrm{occ}}$  via the occupancy head  $g$ , and language-aligned feature predictions  $\mathbf{O}_{\mathrm{ft}}$  via the 3D-language head  $h$ . Next, as described in [22], we generate a set of query sentences for each text query using predefined templates. These queries are then input into the pre-trained language-image encoder  $f_{\mathrm{text}}$ , resulting in a set of language features. To obtain a single text feature per query, we compute the average of these features. Finally, considering  $M$  such averaged text features, one for each of the  $M$  target segmentation classes, we measure their similarity to the predicted language-aligned features  $\mathbf{O}_{\mathrm{ft}}$  at occupied voxels obtained from  $\mathbf{O}_{\mathrm{occ}}$ . We assign the label with the highest similarity to each occupied voxel.
Language-driven 3D grounding. The task of language-driven 3D grounding is performed in a similar manner. However, here only a single input language query is given. Once determining the occupied voxels from  $\mathbf{O}_{\mathrm{occ}}$ , we compute the similarity between the input text query encoded via the language-image encoder  $f_{\mathrm{text}}$  and predicted language-aligned features  $\mathbf{O}_{\mathrm{ft}}$  at the occupied voxels. The resulting similarity score can be visualized as a heat-map, as shown in Fig. 1, or thresholded to obtain the location of the target query.
This section studies architecture design choices and demonstrates the capabilities of the proposed approach. First, in Sec. 4.1, we describe the experimental setup used, particularly the dataset, metrics, proposed evaluation protocol, and implementation details. Then, we compare our model to the state of the art in Sec. 4.2. Next, we present a set of studies on training hyperparameter sensitivity in Sec. 4.3 and finally show qualitative results in Sec. 4.4.
We test the proposed approach on autonomous driving data, which provides a challenging test-bed.
Dataset. We use the nuScenes [10] dataset composed of 1000 sequences in total, divided into 700/150/150 scenes for train/val/test splits. Each sequence consists of  $30 - 40$  scenes resulting in 28, 130 training and in 6, 019 validation scenes. The dataset provides 3D point clouds captured with 32-beam LiDAR, surround-view images obtained from six cameras mounted at the top of the car, and projection matrices between the 3D point cloud and cameras. LiDAR point clouds are annotated with 16 semantic labels. When using subsets of the complete dataset for ablations, we sort the scenes by their timestamp and take every  $N$ -th scene, e.g., every second scene in the case of a  $50\%$  subset.