SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Once we obtain the class-agnostic masks, our primary objective is to determine the best CLIP features for each of them. We operate under the assumption that CLIP is capable of extracting meaningful features from instances when provided with a favorable viewpoint, and in our feature computation process, we prioritize selecting views that offer a higher number of visible points. In practice, to assess the quality of a particular viewpoint for each mask, we compute the proportion of points of the 3D instance mask that are visible from that view. Next, we rank the views based on their visibility scores, and we select the top  $k$  views with the highest scores, where  $k$  is a hyper parameter. Fig. 6 illustrates this concept, where two views of a given instance are compared based on the number of visible points. The view that has a higher number of visible points from the instance is selected (outlined in green).
Figure 6: Visual comparison between a selected view (marked with a green outline) for the corresponding chair instance, and a discarded view. The green points in the images represent the 3D instance points projected onto the 2D image. In the selected view the instance is fully visible, and mostly occluded in the discarded view.
A.2.2 Why do we need SAM?
In order to get the CLIP image features of a particular instance in the selected views, we need to crop the image section containing the object instance. This process involves obtaining an initial tight crop of the instance projected onto 2D, and then incrementally expanding the crop to include some visual context around it.
A straightforward approach to perform the cropping would be to project all of the visible 3D points belonging to the mask onto the 2D image, and fit a 2D bounding box around these points. However, as we do not discard any mask proposals, the masks can potentially include outliers, or the masks might be too small as an outcome of the DBSCAN algorithm described in Sec. A.1. As depicted in Fig. 7, this can result in bounding boxes that are either too large or too small. Using CLIP-image features obtained from crops based solely on projected points can lead to inferior instance mask-features.
To address this challenge, and to improve the robustness of our model against noisy instance masks, we propose using a 2D segmentation method that takes a set of points from the mask as input, and produces the corresponding 2D mask, together with its confidence score. The Segment Anything Model (SAM) [36] precisely fulfills this task, allowing us to generate accurate 2D masks in a class-agnostic manner.
Figure 7: Difference between the bounding boxes obtained by tightly cropping around the projected points from the 3D instance mask (left), and the bounding box obtained from the 2D mask generated by SAM (right).
A.2.3 Which points should we give as input to SAM?
When using SAM for predicting a 2D mask, we need to determine a set of points on which the mask generation process will be conditioned. Initially, we attempted to input all the visible points projected from the 3D mask. However, this approach results in poor quality masks due to the inaccuracies of the projections and the noise present in the 3D masks, as illustrated in Figure 8.
Figure 8: Output of SAM, using all of the visible points from the projected 3D mask as input.
To address this issue, we explore an alternative approach. Instead of using all of the visible points as input for SAM, we randomly sample a subset of points from the projected 3D mask. Interestingly, this method produces much cleaner masks, as it can be seen in Fig. 9.
Figure 9: Output of SAM, using only 5 randomly sampled points (visualized as green dots) of the projected 3D mask as input.
A.2.4 Why do we need to run SAM for multiple rounds?
Relying solely on a small set of random points as input for SAM might be unreliable, particularly when the sampled points are outliers, or too concentrated in a particular area of the instance we want to segment.
To address this limitation, we implemented a sampling algorithm (as outlined in the main paper) inspired by RANSAC [18]. In this approach, we perform  $k_{\text{round}} = 10$  sampling rounds, and in each round we randomly sample  $k_{\text{sample}} = 5$  points from the projected 3D instance mask. SAM returns the predicted 2D mask along with the corresponding mask confidence score for each set of sampled points. Based on the confidence scores returned by SAM, we select the mask with the highest score. Fig. 10 illustrates an example where the points sampled in one round are concentrated in a small spatial range, resulting in an incorrect mask prediction by SAM.
Figure 10: Output of SAM, using only 5 randomly sampled points of the mask as input. Here the sampled points (the green points visualized in the image) are concentrated in a small spatial range and cause SAM to predict an incorrect mask, which does not include the legs of the chair.
However, by employing our sampling algorithm and running SAM for multiple rounds, we achieve improved results. The iterative process allows us to select the mask with the highest score achieved among the rounds, as shown in Fig. 11. In this particular example, the selected mask, i.e. the one with the highest SAM confidence score, accurately segments the chair.
Figure 11: Illustration of the results from the round which gives the highest confidence score. On the left, we visualize the 5 sampled points which are given as input to SAM. On the right, we visualize the SAM mask prediction, showing a perfect segmentation of the chair.
B.1 Ablation Studies
In addition to our ablation experiments provided in the main paper, here we share additional results. First, in Table 6, we share results obtained from using a varying number of views  $(k)$  in the top-  $k$  view selection process. This analysis is conducted on the ScanNet200 dataset. Additionally, we provide an ablation study evaluating the hyperparameters of the multi-scale cropping in our method. This analysis is conducted on the Replica dataset, and is presented in Table 7.
Top-k	AP	AP50	AP25	head(AP)	common(AP)	tail(AP)
1	13.6	17.6	20.8	15.5	12.5	12.8
5	15.4	19.9	23.1	17.1	14.1	14.9
10	15.4	20.0	23.2	16.4	14.0	15.8
Table 6: Ablation study of the top-  $k$  frame selection parameter  $k$  . This analysis is conducted on the ScanNet200 validation set.
Levels	Ratio of Exp.	AP	\( AP_{50} \)	\( AP_{25} \)
1	0.1	11.3	16.0	20.2
3	0.1	13.1	18.4	24.2
5	0.1	12.8	17.6	22.6
3	0.05	12.9	18.1	23.5
3	0.1	13.1	18.4	24.2
3	0.2	12.8	17.7	22.9
Table 7: Ablation study of the multi-scale cropping hyperparameters on the Replica dataset. We analyze the effect of varying number of levels, and the ratio of expansion.
B.2 Evaluation on Replica without RGB-D images
Our approach OpenMask3D requires images as input, as it depends on visual-language models that operate on images in combination with text. In this work, we prioritized the ability to recognize uncommon/long-tail objects over generalization across different modalities. Using vision-language models on images provides an excellent opportunity to preserve this generalization capability. Nevertheless, even when only the 3D scan of a scene is available, it could still be possible to render images from the 3D scan, and use those synthetic images as input to our OpenMask3D pipeline. We tried this on the Replica [61] dataset, and rendered RGB-D images from the dense scene point clouds (as illustrated in Fig. 12). Our results using these rendered synthetic views of the scene point cloud are presented in Tab. 8. Even without any RGB-D images as an input, OpenMask3D performs relatively well, and we observe that the performance decreases by only  $-1.5$  AP. Furthermore, OpenMask3D shows stronger performance compared to the 3D baseline (OpenScene [52], 3D Distill).
C Details on baseline experiments
Quantitative baseline experiments
As mentioned in the main paper, to compare our OpenMask3D method with the 3D open-vocabulary scene understanding approach OpenScene [52], we adapted OpenScene for the 3D instance segmentation task. As
SAM score  $= 0.985$
Model	AP	\( AP_{50} \)	\( AP_{25} \)
OpenScene [52] (3D Distill) + masks	8.2	10.5	12.6
OpenMask3D w. rendered RGB-D images	11.6	14.9	18.4
OpenMask3D	13.1	18.4	24.2
Table 8: 3D instance segmentation results on the Replica dataset. Comparing OpenMask3D performance when using original RGB-D images vs. RGB-D images rendered from the point cloud.
(a) RGB image from the Replica dataset
(b) RGB image rendered from the point cloud
(c) RGB image from the Replica dataset
(d) RGB image rendered from the point cloud
Figure 12: Visualization of the Replica RGB images. Original RGB images from the Replica dataset (left), and RGB images rendered from the scene point cloud (right).
OpenScene computes a per-point feature representation for each point in the point cloud, we use our proposed class-agnostic instance masks to aggregate OpenScene features for each mask. This way, we can associate each object instance mask with a "per-mask" feature computed from OpenScene features.
OpenScene provides 3 different model variants, namely 2D fusion, 3D distill, and 2D/3D ensemble. We primarily compare with the OpenScene models using OpenSeg [20] features, whereas we also experiment with the 2D fusion model using LSeg [43] features. The main reason why we primarily compare against the models using OpenSeg features is that the features have a dimensionality of 768, relying on the same CLIP architecture we employ in our work - ViT-L/14 (336px) [55] -, whereas the LSeg features have a dimensionality of 512. For our closed-vocabulary experiments with the OpenScene model using LSeg features, we embed each text category using ViT-B/32 CLIP text encoder, and we assign object categories using these embeddings. In all other experiments, we embed text queries using the CLIP architecture ViT-L/14 (336px).
Qualitative baseline experiments
In addition to the quantitative results presented in the main paper, we conducted qualitative comparisons to further evaluate our mask-based open-world representation in comparison to the point-based representation provided by OpenScene [52].
In Fig. 13 and Fig. 14, we provide these qualitative comparisons. In Fig. 13, we show results from the OpenScene [52] 2D fusion model using OpenSeg [20] features, and from OpenMask3D. In Fig. 14, we compare features from OpenMask3D with features from several OpenScene variants (2D fusion, 3D distill, 2D/3D ensemble). To visualize results from OpenScene, for a given query, we compute the similarity score between each point-feature and the query text embedding, and visualize the similarity scores for each point as a heatmap over the points. On the other hand, OpenMask3D results are based on mask-features, i.e., each mask has an associated feature vector, for which we compute a similarity score. Hence, we visualize the similarity scores for each mask as a heatmap over the instance masks, and each point in a given instance mask is assigned to the same similarity score.
Query: "a dollhouse"
Query: "a desk with a backpack on it"
Query: "Cetaphil"
Figure 13: Qualitative comparisons. Heatmaps illustrating the similarity between the CLIP embeddings of a specific query and the open-vocabulary representation of the scene. A comparison is made between the point-based OpenScene approach proposed by Peng et al. [52] (left) and our mask-based approach OpenMask3D (right). Dark red means high similarity, and dark blue means low similarity with the query.
As evident from Fig. 13 and Fig. 14, while similarity computation between the query embedding and each OpenScene per-point feature vector results in a reasonable heatmap, it is challenging to extract object instance information from the heatmap representation. Our method OpenMask3D, on the other hand, computes the similarity between the query embedding and each per-mask feature vector. This results in crisp instance boundaries, particularly suitable for the use cases in which one needs to handle object instances.
In this section, we provide additional qualitative results.
In Fig. 15, we show results from object categories that are not present in the ScanNet200 label set. Objects from these novel categories are successfully segmented using our OpenMask3D approach. Thanks to its zero-shot learning capabilities, OpenMask3D is able to segment a given query object that might not be present in common segmentation datasets on which closed-vocabulary instance segmentation approaches are trained.
In Fig. 16, we show open-vocabulary 3D instance segmentation results using queries describing various object properties such as affordances, color, geometry, material type and object state. This highlights that our model is able to preserve information about such object properties and go beyond object semantics, contrary to the capabilities of existing closed-vocabulary 3D instance segmentation approaches.
(a) Ours (per-mask)
(b) OpenScene (2D Fusion, per-point)
(c) OpenScene (3D Distill, per-point)
(d) OpenScene (2D/3D Ensemble, per-point)
Figure 14: Feature similarity for the query "the table with a flower vase on it".
"dollhouse"
"Cetaphil"
"Roomba"
"flip-flops"
"gym bench"
"rice cooker"
Figure 15: Qualitative results from OpenMask3D. We show open-vocabulary instance segmentation results using arbitrary queries involving object categories that are not present in the ScanNet200 dataset labels. In each scene, we visualize the instance with the highest similarity score for the given query embedding. These predictions show the zero-shot learning ability of our model, highlighting the open-vocabulary capabilities.
"sit"
"wash"
"an orange pillow"
"a yellow pillow"
"acirculartable"
"arctangular table"
"aleatherchair
"afabricchair
"an empty table"
"aside tablewithaflower vaseonit"
State
Figure 16: Qualitative results from OpenMask3D, queries related to object properties. We show open-vocabulary 3D instance segmentation results using queries describing various object properties such as affordances, color, geometry, material type, and object state. In each row, we show two different queries per category, for which our method successfully segments different object instances. These results highlight the strong open-vocabulary 3D instance segmentation capabilities of our model, going beyond object semantics.
"a full garbage bin"
"an empty garbage bin"