SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ayca Takmaz $^{1*}$
Marc Pollefeys1,2
Elisabetta Fedele $^{1*}$
Federico Tombari3
Robert W. Sumner<sup>1</sup>
Francis Engelmann<sup>1,3</sup>
$^{1}$ ETH Zürich  $^{2}$ Microsoft  $^{3}$ Google
openmask3d.github.io
"footrest"
"side table with a flower vase on it"
"angel"
Figure 1: Open-Vocabulary 3D Instance Segmentation. Given a 3D scene (top) and free-form user queries (bottom), our OpenMask3D segments object instances and scene parts described by the open-vocabulary queries.
We introduce the task of open-vocabulary 3D instance segmentation. Current approaches for 3D instance segmentation can typically only recognize object categories from a pre-defined closed set of classes that are annotated in the training datasets. This results in important limitations for real-world applications where one might need to perform tasks guided by novel, open-vocabulary queries related to a wide variety of objects. Recently, open-vocabulary 3D scene understanding methods have emerged to address this problem by learning queryable features for each point in the scene. While such a representation can be directly employed to perform semantic segmentation, existing methods cannot separate multiple object instances. In this work, we address this limitation, and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D instance segmentation. Guided by predicted class-agnostic 3D instance masks, our model aggregates per-mask features via multi-view fusion of CLIP-based image embeddings. Experiments and ablation studies on ScanNet200 and Replica show that OpenMask3D outperforms other open-vocabulary methods, especially on the long-tail distribution. Qualitative experiments further showcase OpenMask3D's ability to segment object properties based on free-form queries describing geometry, affordances, and materials.
3D instance segmentation, which is the task of predicting 3D object instance masks along with their object categories, has many crucial applications in fields such as robotics and augmented reality. Due to its significance, 3D instance segmentation has been receiving a growing amount of attention in recent years. Despite the remarkable progress made in 3D instance segmentation methods [8, 9, 39, 58, 77, 78], it is noteworthy that these methods operate under a closed-set paradigm, where the set of object categories is limited and closely tied to the datasets used during training.
We argue that there are two key problems with closed-vocabulary 3D instance segmentation. First, these approaches are limited in their ability to understand a scene beyond the object categories seen during training. Despite the significant success of 3D instance segmentation approaches from recent years, these closed-vocabulary approaches may fail to recognize and correctly classify novel objects. One of the main advantages of open-vocabulary approaches, is their ability to zero-shot learn
categories that are not present at all in the training set. This ability has potential benefits for many applications in fields such as robotics, augmented reality, scene understanding and 3D visual search. For example, it is essential for an autonomous robot to be able to navigate in an unknown environment where novel objects can be present. Furthermore, the robot may need to perform an action based on a free-form query, such as "find the side table with a flower vase on it", which is challenging to perform with the existing closed-vocabulary 3D instance segmentation methods. Hence, the second key problem with closed-vocabulary approaches is their inherent limitation to recognize only object classes that are predefined at training time.
In an attempt to address and overcome the limitations of a closed-vocabulary setting, there has been a growing interest in open-vocabulary approaches. A line of work [20, 43, 45] investigates open-vocabulary 2D image segmentation task. These approaches are driven by advances in large-scale model training, and largely rely on recent foundation models such as CLIP [55] and ALIGN [33] to obtain text-image embeddings. Motivated by the success of these 2D open-vocabulary approaches, another line of work has started exploring 3D open-vocabulary scene understanding task [24, 32, 52], based on the idea of lifting image features from models such as CLIP [55], LSeg [43], and OpenSeg [20] to 3D. These approaches aim to obtain a task-agnostic feature representation for each 3D point in the scene, which can be used to query concepts with open-vocabulary descriptions such as object semantics, affordances or material properties. Their output is typically a heatmap over the points in the scene, which has limited applications in certain aspects, such as handling object instances.
In this work, we propose OpenMask3D, an open-vocabulary 3D instance segmentation method which has the ability to reason beyond a predefined set of concepts. Given an RGB-D sequence, and the corresponding 3D reconstructed geometry, OpenMask3D predicts 3D object instance masks, and computes a mask-feature representation. Our two-stage pipeline consists of a class-agnostic mask proposal head, and a mask-feature aggregation module. Guided by the predicted class-agnostic 3D instance masks, our mask-feature aggregation module first finds the frames in which the instances are highly visible. Then, it extracts CLIP features from the best images of each object mask, in a multiscale and crop-based manner. These features are then aggregated across multiple views to obtain a feature representation associated with each 3D instance mask. Our approach is intrinsically different from the existing 3D open-vocabulary scene understanding approaches [24, 32, 52] as we propose an instance-based feature computation approach instead of a point-based one. Computing a mask-feature per object instance enables us to retrieve object instance masks based on their similarity to any given query, equipping our approach with open-vocabulary 3D instance segmentation capabilities. As feature computation is performed in a zero-shot manner, OpenMask3D is capable of preserving information about novel objects as well as long-tail objects better, compared to trained or fine-tuned counterparts. Furthermore, OpenMask3D goes beyond the limitations of a closed-vocabulary paradigm, and allows segmentation of object instances based on free-form queries describing object properties such as semantics, geometry, affordances, and material properties.
- We introduce the open-vocabulary 3D instance segmentation task in which the object instances that are similar to a given text-query are identified.
- We propose OpenMask3D, which is the first approach that performs open-vocabulary 3D instance segmentation in a zero-shot manner.
- We conduct experiments to provide insights about design choices that are important for developing an open-vocabulary 3D instance segmentation model.
Closed-vocabulary 3D semantic and instance segmentation. Given a 3D scene as input, 3D semantic segmentation task aims to assign a semantic category to each point in the scene [2-4, 9, 14, 15, 22, 28, 29, 31, 38, 42, 44, 46, 47, 53, 54, 63, 64, 66, 68, 70, 73]. 3D instance segmentation goes further by distinguishing multiple objects belonging to the same semantic category, predicting individual masks for each object instance [13, 16, 25, 27, 34, 41, 58, 62, 65, 67, 74]. The current state-of-the-art approach on the ScanNet200 benchmark [10, 57] is Mask3D [58], which leverages a transformer architecture to generate 3D mask proposals together with their semantic labels. However, similar to existing methods, it assumes a limited set of semantic categories that can be assigned to an object instance. In particular, the number of labels is dictated by the annotations provided in the training datasets, 200 – in the case of ScanNet200 [57]. Given that the English language encompasses
numerous nouns, in the order of several hundred thousand [69], it is clear that existing closed-vocabulary approaches have an important limitation in handling object categories and descriptions.
Foundation models. Recent multimodal foundation models [1, 7, 21, 33, 55, 75, 76] leverage large-scale pretraining to learn image representations guided by natural language descriptions. These models enable zero-shot transfer to various downstream tasks such as object recognition and classification. Driven by the progress in large-scale model pre-training, similar foundation models for images were also explored in another line of work [5, 51], which aim to extract class-agnostic features from images. Recently, steps towards a foundation model for image segmentation were taken with SAM [36]. SAM has the ability to generate a class-agnostic 2D mask for an object instance, given a set of points that belong to that instance. This capability is valuable for our approach, especially for recovering high-quality 2D masks from projected 3D instance masks, as further explained in Sec. 3.2.2.
Open vocabulary 2D segmentation. As large vision-language models gained popularity for image classification, numerous new approaches [11, 20, 23, 26, 32, 40, 43, 45, 48, 56, 71, 72, 79, 80] have emerged to tackle open-vocabulary or zero-shot image semantic segmentation. One notable shift was the transition from image-level embeddings to pixel-level embeddings, equipping models with localization capabilities alongside classification. However, methods with pixel-level embeddings, such as OpenSeg [20] and OV-Seg [45], strongly rely on the accuracy of 2D segmentation masks, and require a certain degree of training. In our work, we rely on CLIP [55] features without performing finetuning or any additional training, and compute 2D masks using the predicted 3D instance masks.
Open-vocabulary 3D scene understanding. Recent success of 2D open-vocabulary segmentation models such as LSeg [43], OpenSeg [20], and OV-Seg[45] has motivated researchers in the field of 3D scene understanding to explore the open vocabulary setting [6, 12, 19, 24, 30, 32, 35, 37, 49, 52, 59, 60]. OpenScene [52] uses per-pixel image features extracted from posed images of a scene and obtains a point-wise task-agnostic scene representation. On the other hand, approaches such as LERF [35] and DFF [37] leverage the interpolation capabilities of NeRFs [50] to extract a semantic field of the scene. However, it is important to note that all of these approaches have a limited understanding of object instances and inherently face challenges when dealing with instance-related tasks.
Overview. Our OpenMask3D model is illustrated in Fig. 2. Given a set of posed RGB-D images captured in a scene, along with the reconstructed scene point cloud  $①$ , OpenMask3D predicts 3D instance masks with their associated per-mask feature representations, which can be used for querying instances based on open-vocabulary concepts  $④$ . Our OpenMask3D has two main building blocks: a class agnostic mask proposal head  $②$  and a mask-feature computation module  $③$ . The class-agnostic mask proposal head predicts binary instance masks over the points in the point cloud. The mask-feature computation module leverages pre-trained CLIP [55] vision-language model in order to compute meaningful and flexible features for each mask. For each proposed instance mask, the mask-feature computation module first selects the views in which the 3D object instance is highly visible. Subsequently, in each selected view, the module computes a 2D segmentation mask guided by the projection of the 3D instance mask, and refines the 2D mask using the SAM [36] model. Next, the CLIP encoder is employed to obtain image-embeddings of multi-scale image-crops bounding the computed 2D masks. These image-level embeddings are then aggregated across the selected frames in order to obtain a mask-feature representation. Sec. 3.1 describes the class agnostic mask proposal head, and Sec. 3.2 describes the mask-feature computation module.
The key novelty of our method is that it follows an instance-mask oriented approach, contrary to existing 3D open-vocabulary scene understanding models which typically compute per-point features. These point-feature oriented models have inherent limitations, particularly for identifying object instances. Our model aims to overcome such limitations by introducing a framework that employs class agnostic instance masks and aggregates informative features for each object instance.
Input. Our pipeline takes as input a collection of posed RGB-D images captured in an indoor scene, and the reconstructed point cloud representation of the scene. We assume known camera parameters.
3.1 Class agnostic mask proposals
The first step of our approach involves generating  $M$  class-agnostic 3D mask proposals  $\mathbf{m}_1^{3\mathbf{D}},\ldots ,\mathbf{m}_{\mathbf{M}}^{3\mathbf{D}}$ . Let  $\mathbf{P}\in \mathbb{R}^{P\times 3}$  denote the point cloud of the scene, where each 3D point is represented with its corresponding 3D coordinates. Each 3D mask proposal is represented by a
Figure 2: An overview of our approach. We propose OpenMask3D, the first open-vocabulary 3D instance segmentation model. Our pipeline consists of four subsequent steps:  $①$  Our approach takes as input posed RGB-D images of a 3D indoor scene along with its reconstructed point cloud.  $②$  Using the point cloud, we compute class-agnostic instance mask proposals.  $③$  Then, for each mask, we compute a feature representation.  $④$  Finally, we obtain an open-vocabulary 3D instance segmentation representation, which can be used to retrieve objects related to queried concepts embedded in the CLIP [55] space.
binary mask  $\mathbf{m}_{\mathrm{i}}^{3\mathrm{D}} = (m_{i1}^{3D},\dots,m_{iP}^{3D})$  where  $m_{ij}^{3D}\in \{0,1\}$  indicates whether the  $j$ -th point belongs to  $i$ -th object instance. To generate these masks, we leverage the transformer-based mask-module of a pre-trained 3D instance segmentation model [58], which is frozen during our computations. The architecture consists of a sparse convolutional backbone based on the MinkowskiUNet [9], and a transformer decoder. Point features obtained from the feature backbone are passed through the transformer decoder, which iteratively refines the instance queries, and predicts an instance heatmap for each query. In the original setup, [58] produces two outputs: a set of  $M$  binary instance masks obtained from the predicted heatmaps, along with predicted class labels (from a predefined closed set) for each mask. In our approach, we adapt the model to exclusively utilize the binary instance masks, discarding the predicted class labels and confidence scores entirely. These class-agnostic binary instance masks are then utilized in our mask-feature computation module, in order to go beyond semantic class predictions limited to a closed-vocabulary, and obtain open-vocabulary representations instead. Further details about the class-agnostic mask proposal module are provided in Appendix A.1.
3.2 Mask-feature computation module
Mask-feature computation module aims to compute a task-agnostic feature representation for each predicted instance mask obtained from the class-agnostic mask proposal module. The purpose of this module is to compute a feature representation that can be used to query open-vocabulary concepts. As we intend to utilize the CLIP text-image embedding space and maximally retain information about long-tail or novel concepts, we solely rely on the CLIP visual encoder to extract image-features on which we build our mask-features.
As illustrated in Fig. 3, the mask-feature computation module consists of several steps. For each instance mask-proposal, we first compute the visibility of the object instance in each frame of the RGB-D sequence, and select the top- $k$  views with maximal visibility. In the next step, we compute a 2D object mask in each selected frame, which is then used to obtain multi-scale image-crops in order to extract effective CLIP features. The image-crops are then passed through the CLIP visual encoder to obtain feature vectors that are average-pooled over each crop and each selected view, resulting in the final mask-feature representation. In Sec. 3.2.1, we describe how we select a set of top- $k$  frames for each instance. In Sec. 3.2.2, we describe how we crop the frames, based on the object instance we want to embed. In Sec. 3.2.3, we describe how we compute the final mask-features per object.
3.2.1 Frame selection
Obtaining representative images of the proposed object instances is crucial for extracting accurate CLIP features. To achieve this, we devise a strategy to select, for each of the  $M$  predicted instances, a subset of representative frames (Fig. 3,  $\langle \mathbf{a}\rangle$ ) from which we extract CLIP features. In particular, our devised strategy selects frames based on their visibility scores  $s_{ij}$  for each mask  $i$  in each view  $j$ . Here, we explain how we compute these visibility scores.
Figure 3: Mask-Feature Computation Module. For each instance mask,  $①$  we first compute the visibility of the instance in each frame, and select top-  $k$  views with maximal visibility. In  $⑥$  , we compute a 2D object mask in each selected frame, which is used to obtain multi-scale image-crops in order to extract effective CLIP features.  $⑥$  The image-crops are then passed through the CLIP visual encoder to obtain feature vectors that are average-pooled over each crop and  $④$  each selected view, resulting in the final mask-feature representation.
Given a mask proposal  $\mathbf{m}_{\mathrm{i}}^{3\mathbf{D}}$  corresponding to the  $i$ -th instance, we compute the visibility score  $s_{ij} \in [0,1]$  for the  $j$ -th frame using the following formula:
$$
s _ {i j} = \frac {\operatorname {v i s} (i , j)}{\max  _ {j ^ {\prime}} \left(\operatorname {v i s} (i , j ^ {\prime})\right)}
$$
Here,  $vis(i,j)$  represents the number of points from mask  $i$  that are visible in frame  $j$ . Note that we assume that each mask is visible in at least one frame. We compute  $vis(i,j)$  using the following approach. First, for each 3D point from the  $i$ -th instance mask, we determine whether the point appears in the camera's field of view (FOV) in the  $j$ -th frame. To achieve this, we use intrinsic  $(\mathbf{I})$  and extrinsic  $(\mathbf{R}|\mathbf{t})$  matrices of the camera from that frame to project each 3D point in the point cloud to the image plane. We obtain corresponding 2D homogeneous coordinates  $\mathbf{p}_{2\mathbf{D}} = (u,v,w)^{\top} = \mathbf{I}\cdot (\mathbf{R}|\mathbf{t})\cdot \mathbf{x}$ , where  $\mathbf{x} = (x,y,z,1)^{\top}$  is a point represented in homogeneous coordinates. We consider a point to be in the camera's FOV in the  $j$ -th frame if  $w\neq 0$ , and the  $\frac{u}{w}$  value falls within the interval  $[0,W - 1]$  while  $\frac{v}{w}$  falls within  $[0,H - 1]$ . Here  $W$  and  $H$  represent the width and height of the image, respectively. Next, it is important to note that 3D points that are in the camera's FOV are not necessarily visible from a given camera pose, as they might be occluded by other parts of the scene. Hence, we need to check whether the points are in fact visible. In order to examine whether a point is occluded from a given camera viewpoint, we compare the depth value of the 3D point deriving from the geometric computation (i.e.  $w$ ) and the measured depth, i.e. depth value of the projected point in the depth image. We consider points that satisfy the inequality  $w - d > k_{\mathrm{threshold}}$  as occluded, where  $k_{\mathrm{threshold}}$  is a hyper-parameter of our method. If the projection of a point from mask  $i$  is not occluded in the  $j$ -th view, it contributes to the visible point count,  $vis(i,j)$ . Once we compute mask-visibility scores for each frame, we select the top  $k_{\mathrm{view}}$  views with the highest scores  $s_{ij}$  for each mask  $i$ . Here,  $k_{\mathrm{view}}$  represents another hyperparameter.
3.2.2 2D mask computation and multi-scale crops
In this section, we explain our approach for computing CLIP features based on the frames selected in the previous step. Given a selected frame for a mask, our goal is to find the optimal image crops from which to extract features, as shown in (Fig. 3, ⑥). Simply considering all of the projected points of the mask often results in imprecise and noisy bounding boxes, largely affected by the outliers (please see Appendix A.2.2). To address this, we use a class-agnostic 2D segmentation model, SAM [36], which predicts a 2D mask conditioned on a set of input points, along with a mask confidence score.
SAM is sensitive to the set of input points (see Appendix A.2.3, A.2.4). Therefore, to obtain a high quality mask from SAM with a high confidence score, we take inspiration from the RANSAC algorithm [18] and proceed as described in Algorithm 1. We sample  $k_{\text{sample}}$  points from the projected points, and run SAM using these points. The output of SAM at a given iteration  $r$  is a 2D mask  $(\mathbf{m}_{\mathbf{r}}^{2\mathrm{D}})$  and a mask confidence score  $(\text{score}_r)$ . This process is repeated for  $k_{\text{rounds}}$  iterations, and the 2D mask with the highest confidence score is selected.
Next, we use the resulting mask  $\mathbf{m}_{*}^{2\mathbf{D}}$  to generate  $L = 3$  multi-level crops of the selected image. This allows us to enrich the features by incorporating more contextual information from the surroundings.
Algorithm 1 - 2D mask selection algorithm
$\mathbf{score}_{*}\gets 0,\mathbf{m}_{*}^{2D}\gets 0,r\gets 0$    
while  $r <   k_{rounds}$  do Sample  $k_{sample}$  points among the projected points at random Compute the mask  $\mathbf{m_r^{2D}}$  and the score score  $r$  based on the sampled points using SAM if  $\mathrm{score}_r > \mathrm{score}_*$  then  $\mathrm{score}_{*}\leftarrow \mathrm{score}_{r},\mathrm{m}_{*}^{2D}\leftarrow \mathrm{m}_{r}^{2D}$  end if  $r\gets r + 1$    
end while
Specifically, the first bounding box  $\mathbf{b}^1 = (x_1^1, y_1^1, x_2^1, y_2^1)$  with  $0 \leq x_1^1 < x_2^1 < W$  and  $0 \leq y_1^1 < y_2^1 < H$  is a tight bounding box derived from the 2D mask. The other bounding boxes  $\mathbf{b}^2$  and  $\mathbf{b}^3$  are incrementally larger. In particular, the 2D coordinates of  $\mathbf{b}^i$  for  $i = 2, 3$  are obtained as follows.
$$
x _ {1} ^ {l} = \max  \left(0, x _ {1} ^ {1} - \left(x _ {2} ^ {1} - x _ {1} ^ {1}\right) \cdot k _ {e x p} \cdot l\right)
$$
$$
y _ {1} ^ {l} = \max  \left(0, y _ {1} ^ {1} - \left(y _ {2} ^ {1} - y _ {1} ^ {1}\right) \cdot k _ {e x p} \cdot l\right)
$$
$$
x _ {2} ^ {l} = \min  \left(x _ {2} ^ {1} + \left(x _ {2} ^ {1} - x _ {1} ^ {1}\right) \cdot k _ {e x p} \cdot l, W - 1\right)
$$
$$
y _ {2} ^ {l} = \min  \left(y _ {2} ^ {1} + \left(y _ {2} ^ {1} - y _ {1} ^ {1}\right) \cdot k _ {e x p} \cdot l, H - 1\right)
$$
Note that here  $l$  represents the level of the features and  $k_{exp} = 0.1$  is a predefined constant.
3.2.3 CLIP feature extraction and mask-feature aggregation
For each instance mask, we collect  $k \cdot L$  images by selecting top-  $k$  views and obtaining  $L$  multi-level crops as described in Sec. 3.2.1 and Sec. 3.2.2. Collected image crops are then passed through the CLIP visual encoder in order to extract image features in the CLIP embedding space, as shown in (Fig. 3, ©). We then aggregate the features obtained from each crop that correspond to a given instance mask in order to get an average per-mask CLIP feature (Fig. 3, @). The computed features are task-agnostic, and can be used for various instance-based tasks by encoding a given text or image-based query, using the same CLIP model we employed to encode the image crops.
In this section, we present quantitative and qualitative results from our method OpenMask3D. In Sec 4.1, we quantitatively evaluate our method, and compare OpenMask3D with supervised 3D instance segmentation approaches as well as existing open-vocabulary 3D scene understanding models we adapted for the 3D instance segmentation task. Furthermore, we provide an ablation study for OpenMask3D. In Sec. 4.2, we share qualitative results for open-vocabulary 3D instance segmentation, demonstrating potential applications. Additional results are provided in the Appendix.
4.1 Quantitative results: closed-vocabulary 3D instance segmentation evaluation