SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We evaluate our approach on the closed-vocabulary 3D instance segmentation task. We conduct additional experiments to assess the generalization capability of OpenMask3D.
4.1.1 Experimental setting
Data. We conduct our experiments using the ScanNet200 [57] and Replica [61] datasets. We report our ScanNet200 results on the validation set consisting of 312 scenes, and evaluate for the 3D instance segmentation task using the closed vocabulary of 200 categories from the ScanNet200 annotations. Rozenberszki et al. [57] also provide a grouping of ScanNet200 categories based on the frequency of the number of labeled surface points in the training set, resulting in 3 subsets: head (66 categories), common (68 categories), tail (66 categories). This grouping enables us to evaluate the performance of our method on the long-tail distribution, making ScanNet200 a natural choice as an evaluation dataset. To assess the generalization capability of our method, we further experiment with the Replica [61] dataset, and evaluate on the office0, office1, office2, office3, office4, room0, room1, room2 scenes.
Metrics. We employ a commonly used 3D instance segmentation metric, average precision (AP). AP scores are evaluated at mask overlap thresholds of  $50\%$  and  $25\%$ , and averaged over the overlap
Model	Image Features	AP	\( AP_{50} \)	\( AP_{25} \)	head (AP)	common (AP)	tail (AP)
Closed-vocabulary, fully supervised
Mask3D [58]	-	26.9	36.2	41.4	39.8	21.7	17.9
Open-vocabulary
OpenScene [52] (2D Fusion) + masks	OpenSeg [20]	11.7	15.2	17.8	13.4	11.6	9.9
OpenScene [52] (3D Distill) + masks	OpenSeg [20]	4.8	6.2	7.2	10.6	2.6	0.7
OpenScene [52] (2D/3D Ens.) + masks	OpenSeg [20]	5.3	6.7	8.1	11.0	3.2	1.1
OpenScene [52] (2D Fusion) + masks	LSeg [43]	6.0	7.7	8.5	14.5	2.5	1.1
OpenMask3D (Ours)	CLIP [55]	15.4	19.9	23.1	17.1	14.1	14.9
Table 1: 3D instance segmentation results on the ScanNet200 validation set. Metrics are respectively: AP averaged over an overlap range, and AP evaluated at  $50\%$  and  $25\%$  overlaps. We also report AP scores for head, common, tail subsets of ScanNet200. Mask3D [58] is fully-supervised, while OpenScene [52] is built upon on 2D models (LSeg [43], OpenSeg [20]) trained on labeled datasets for 2D semantic segmentation. Since OpenScene [52] does not provide instance masks, we aggregate its per-point features using class-agnostic masks. OpenMask3D outperforms other open-vocabulary counterparts, particularly on the long-tail classes.
range of  $[0.5:0.95:0.05]$  following the evaluation scheme from ScanNet [10]. Computation of the metrics requires each mask to be assigned a prediction confidence score. We assign a prediction confidence score of 1.0 for each predicted mask in our experiments.
OpenMask3D implementation details. We use posed RGB-depth pairs for both the ScanNet200 and Replica datasets, and we process 1 frame in every 10 frames in the RGB-D sequences. In order to compute image features on the mask-crops, we use CLIP [55] visual encoder from the ViT-L/14 model pre-trained at a 336 pixel resolution, which has a feature dimensionality of 768. For the visibility score computation, we use  $k_{threshold} = 0.2$ , and for top-view selection we use  $k_{view} = 5$ . In all experiments with multi-scale crops, we use  $L = 3$  levels. In the 2D mask selection algorithm based on SAM [36], we repeat the process for  $k_{rounds} = 10$  rounds, and sample  $k_{sample} = 5$  points at each iteration. For the class-agnostic mask proposals, we use the Mask3D [58] model trained on ScanNet200 instances, and exclusively use the predicted binary instance masks in our pipeline. We do not filter any instance mask proposals, and run DBSCAN [17] to obtain spatially contiguous clusters, breaking down masks into smaller new masks when necessary. This process results in a varying number of mask proposals for each scene. For further implementation details about OpenMask3D, please refer to Appendix A. Computation of the mask-features of a ScanNet scene on a single GPU takes 5-10 minutes depending on the number of mask proposals and number of frames in the RGB-D sequence. Note that once the per-mask features of the scene are computed, objects can be queried in real-time ( $\sim 1 - 2$  ms) with arbitrary open-vocabulary queries.
Methods in comparison. We compare with Mask3D [58], which is the current state-of-the-art on the ScanNet200 3D instance segmentation benchmark. We also compare against recent open-vocabulary 3D scene understanding model variants (2D Fusion, 3D Distill, 2D/3D Ensemble) from OpenScene [52]. Note that since OpenScene only generates a per-point feature vector (without any per-instance aggregation), it is not possible to directly compare it with our method. To address this, we extended OpenScene by averaging its per-point features within each instance mask generated by our mask module (see Appendix C for more details).
Class assignment. Our open-vocabulary approach does not directly predict semantic category labels per each instance mask, but it instead computes a task-agnostic feature vector for each instance, which can be used for performing a semantic label assignment. In order to evaluate our model on the closed-vocabulary 3D instance segmentation task, we need to assign each object instance to a semantic category. Similar to OpenScene [52], we compute cosine similarity between mask-features and the text embedding of a given query in order to perform class assignments. Following Peng et al. [52], we use prompts in the form of “a {} in a scene”, and compute text-embeddings using CLIP model ViT-L/14(336px) [55] for each semantic class in the ScanNet200 dataset. This way, we compute a similarity score between each instance and each object category, and assign instances to the category with the closest text embedding.
3D closed-vocabulary instance segmentation results. We quantitatively evaluate our approach on the closed-vocabulary instance segmentation task on the ScanNet200 [10, 57] and Replica [61] datasets. Closed-vocabulary instance segmentation results are provided in Tab. 1 and Tab. 2.
State-of-the-art fully supervised approach Mask3D [58] demonstrates superior performance compared to open-vocabulary counterparts. While this gap is more prominent for the head and common categories, the difference is less apparent for the tail categories. This outcome is expected, as Mask3D benefits from full-supervision using the closed-set of class labels from the ScanNet200 dataset. Furthermore, as there are more training samples from the categories within the head and common subsets (please refer to [57] for the statistics), the fully-supervised approach is more frequently exposed to these categories - resulting in a stronger performance. When we compare OpenMask3D with other open-vocabulary approaches, we observe that our method performs better on 6 out of 6 metrics. Notably, OpenMask3D outperforms other open-vocabulary approaches especially on the tail categories by a significant margin. Our instance-centric method OpenMask3D, which is specifically designed for the open-vocabulary 3D instance segmentation task, shows stronger performance compared to other open-vocabulary approaches which adopt a point-centric feature representation. Differences between the feature representations are also illustrated in Fig. 5.
How well does our method generalize? As the mask module is trained on a closed-set segmentation dataset, ScanNet200 [57], we aim to investigate how well our method generalizes beyond the categories seen during training. Furthermore, we aim to analyze how well our method generalizes to out-of-distribution (OOD) data, such as scenes from another dataset, Replica [61]. To demonstrate the generalization capability of our approach, we conducted a series of experiments. First, we analyze the performance of our model when we use class-agnostic masks from a mask-predictor trained on the 20 original ScanNet classes [10], and evaluate on the ScanNet200 dataset. To evaluate how well our model performs on "unseen" categories, we group the ScanNet200 labels into two subsets: base and novel classes. We identify ScanNet200 categories that are semantically similar to the original ScanNet20 classes (e.g. chair and folded-chair, table and dining-table), resulting in 53 classes. We group all remaining object classes that are not similar to any class in ScanNet20 as "novel". In Tab. 3, we report results on seen ("base") classes, unseen ("novel") classes, and all classes. Our experiments show that the OpenMask3D variant using a mask proposal backbone trained on a smaller set of object annotations from ScanNet20 can generalize to predict object masks from a significantly larger set of objects (ScanNet200), resulting in only a marginal decrease in the performance. In a second experiment, we evaluate the performance of OpenMask3D on out-of-distribution data from Replica, using a mask predictor trained on ScanNet200. The results from this experiment, as presented in Tab. 2, indicate that OpenMask3D can indeed generalize to unseen categories as well as OOD data. OpenMask3D using a mask predictor module trained on a smaller set of objects seems to perform reasonably well in generalizing to various settings.
Model	AP	\( AP_{50} \)	\( AP_{25} \)
OpenScene [52] (2D Fusion) + masks	10.9	15.6	17.3
OpenScene [52] (3D Distill) + masks	8.2	10.5	12.6
OpenScene [52] (2D/3D Ens.) + masks	8.2	10.4	13.3
OpenMask3D (Ours)	13.1	18.4	24.2
Table 2: 3D instance segmentation results on the Replica [61] dataset. To assess how well our model generalizes to other datasets, we use instance masks from the mask proposal module trained on ScanNet200, and test it on Replica scenes. OpenMask3D outperforms other open-vocabulary counterparts on the Replica dataset.
Method	Mask Training	Novel Classes	Base Classes	All Classes
AP	\(AP_{50}\)	\(AP_{25}\)	AP	\(AP_{50}\)	\(AP_{25}\)	AP	tail (AP)
OpenScene [52] (2D Fusion) + masks	ScanNet20	7.6	10.3	12.3	11.1	15.0	17.7	8.5	6.1
OpenScene [52] (3D Distill) + masks	ScanNet20	1.8	2.3	2.7	10.1	13.4	15.4	4.1	0.4
OpenScene [52] (2D/3D Ens.) + masks	ScanNet20	2.4	2.8	3.3	10.4	13.7	16.3	4.6	0.9
OpenMask3D (Ours)	ScanNet20	11.9	15.2	17.8	14.3	18.3	21.2	12.6	11.5
OpenMask3D (Ours)	ScanNet200	15.0	19.7	23.1	16.2	20.6	23.1	15.4	14.9
Table 3: 3D instance segmentation results using masks from mask module trained on ScanNet20 annotations, evaluated on the ScanNet200 dataset [57]. We identify 53 classes (such as chair, folded chair, table, dining table ...) that are semantically close to the original ScanNet20 classes [10], and group them as "Base". Remaining 147 classes are grouped as "Novel". We also report results on the full set of labels, titled "All".
Ablation study. In Tab. 4, we analyze design choices for OpenMask3D, i.e., multi-scale cropping and 2D mask segmentation. 2D mask segmentation refers to whether we use SAM [36] for refining
2D Mask	Multi-Scale	AP	\( AP_{50} \)	\( AP_{25} \)	head(AP)	common(AP)	tail(AP)
✘	✘	12.9	16.6	19.5	15.1	12.2	11.3
✘	✓	14.3	18.4	21.1	16.1	13.6	12.9
✓	✘	14.1	18.3	21.5	16.0	13.0	13.4
✓	✓	15.4	19.9	23.1	17.1	14.1	14.9
Table 4: OpenMask3D Ablation Study. 2D mask and multi-scale crop components. 2D mask refers to whether SAM [36] was employed for computing 2D masks. Results are reported on the ScanNet200 [10] validation set.
Model	Oracle	Img. Feat.	AP	head(AP)	common(AP)	tail(AP)
Closed-vocabulary, full sup.
Mask3D [58]	×	-	26.9	39.8	21.7	17.9
Mask3D [58]	✓	-	35.5	55.2	27.2	22.2
Open-vocabulary
OpenScene [52] (2D Fusion) + masks	✓	OpenSeg [20]	22.9	26.2	22.0	20.2
OpenScene [52] (2D Fusion) + masks	✓	LSeg [43]	11.8	26.9	5.2	1.7
OpenMask3D (Ours)	✓	CLIP [55]	29.1	31.1	24.0	32.9
Table 5: 3D instance segmentation results on the ScanNet200 validation set, using oracle masks. We use ground truth instance masks for computing the per-mask features. We also report results from the fully-supervised close-vocabulary Mask3D (row 2) whose predicted masks we match to the oracle masks using Hungarian matching, and assign the predicted labels to oracle masks. On the long-tail categories, OpenMask3D outperforms even the fully supervised Mask3D with oracle masks.
the 2D mask from which we obtain a 2D bounding box. When SAM is not used, we simply crop the tightest bounding box around the projected 3D points. The ablation study shows that both multi-scale cropping and segmenting 2D masks to obtain more accurate image crops for a given object instance positively affect the performance. Additional experiments analyzing the importance of the number of views (k) used in top-k view selection are provided in Appendix B.1.
How well would our approach perform if we had perfect masks? Another analysis we conduct is related to the class-agnostic masks. As the quality of the masks plays a key role in our process, we aim to quantify how much the performance could improve if we had "perfect" oracle masks. For this purpose, we run OpenMask3D using ground truth instance masks from the ScanNet200 dataset instead of using our predicted class-agnostic instance masks. In a similar fashion for the baselines, we use the oracle masks to aggregate OpenScene per-point features to obtain per-mask features. We first compare against the fully-supervised Mask3D performance (Tab. 5, row 1). In a second experiment, we also supply oracle masks to Mask3D (Tab. 5, row 2) to ensure a fair comparison. For this experiment, we perform Hungarian matching between the predicted masks and oracle masks discarding all class-losses, and we only match based on the masks. To each oracle mask, we assign the predicted class label of the closest-matching mask from Mask3D. In Tab. 5, it is evident that the quality of the masks plays an important role for our task. Remarkably, our feature computation approach that is not trained on any additional labeled data, when provided with oracle masks, even surpasses the performance of the fully supervised method Mask3D (by  $+15.0\%$  AP) and Mask3D with oracle masks (by  $+10.7\%$  AP) on the long-tail categories. Overall, this analysis indicates that our approach has indeed promising results which can be further improved by higher quality class-agnostic masks, without requiring any additional training or finetuning with labeled data.
4.2 Qualitative results
In Fig. 4, we share qualitative results from our approach for the open-vocabulary 3D instance segmentation task. With its zero-shot learning capabilities, OpenMask3D is able to segment a given query object that might not be present in common segmentation datasets. Furthermore, object properties such as colors, textures, situational context and affordances are successfully recognized by OpenMask3D. Additional qualitative results in this direction are provided in Appendix D.
In Fig. 5, we provide qualitative comparisons between our instance-based open-vocabulary representation and the point-based representation provided by OpenScene [52]. Our method OpenMask3D computes the similarity between the query embedding and per-mask feature vectors for each object instance, which results in crisp instance boundaries. This is particularly suitable for the use cases in which one needs to identify object instances. Additional analysis and further details about the visualizations are provided in Appendix C.
"pepsi"
"pool"
"an armchair with floral print"
"abedwithoutclothesonit"
"cool down"
"agreenseat"
Figure 4: Qualitative results from OpenMask3D. Our open-vocabulary instance segmentation approach is capable of handling different types of queries. Novel object classes as well as objects described by colors, textures, situational context and affordances are successfully retrieved by OpenMask3D.
Query: "a dollhouse"
Figure 5: Heatmaps showing the similarity between given text queries and open-vocabulary scene features. Input 3D scene and query (left), per-point similarity from OpenScene (middle) and per-mask similarity from OpenMask3D (right). Dark red means high similarity, and dark blue means low similarity with the query text.
OpenScene [52]
OpenMask3D (Ours)
4.3 Limitations.
The experiments conducted with oracle masks indicate that there is room for improvement in terms of the quality of 3D mask proposals which will be addressed in future work. Further, since the per-mask features originate from images, they can only encode scene context visible in the camera frustum, lacking a global understanding of the complete scene and spatial relationships between all scene elements. Finally, evaluation methodologies for systematically assessing open-vocabulary capabilities still remain a challenge. Closed-vocabulary evaluations, while valuable for initial assessments, fall short in revealing the true extent of open-vocabulary potentials of proposed models.
We propose OpenMask3D, the first open-vocabulary 3D instance segmentation model that can identify objects instances in a 3D scene, given arbitrary text queries. This is beyond the capabilities of existing 3D semantic instance segmentation approaches, which are typically trained to predict categories from a closed vocabulary. With OpenMask3D, we push the boundaries of 3D instance segmentation. Our method is capable of segmenting object instances in a given 3D scene, guided by open-vocabulary queries describing object properties such as semantics, geometry, affordances, material properties and situational context. Thanks to its zero-shot learning capabilities, OpenMask3D is able to segment multiple instances of a given query object that might not be present in common segmentation datasets on which closed-vocabulary instance segmentation approaches are trained. This opens up new possibilities for understanding and interacting with 3D scenes in a more comprehensive and flexible manner. We encourage the research community to explore open-vocabulary approaches, where knowledge from different modalities can be seamlessly integrated into a unified and coherent space.
Acknowledgments and disclosure of funding. Francis Engelmann is a postdoctoral research fellow at the ETH AI Center. This project is partially funded by the ETH Career Seed Award "Towards Open-World 3D Scene Understanding", and Innosuisse grant (48727.1 IP-ICT). We sincerely thank Jonas Schult for helpful discussions, and Lorenzo Liso for the help with setting up our live demo.
[73] Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters. In European Conference on Computer Vision (ECCV), 2018.
[74] Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, and Niki Trigoni. Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds. In Neural Information Processing Systems (NeurIPS), 2019.
[75] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive Captioners are Image-Text Foundation Models. Transactions on Machine Learning Research, 2022.
[76] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
[77] Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann. Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[78] Yuanwen Yue, Sabarinath Mahadevan, Jonas Schult, Francis Engelmann, Bastian Leibe, Konrad Schindler, and Theodora Kontogianni. AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation. arXiv preprint arXiv:2306.00977, 2023.
[79] Nir Zabari and Yedid Hoshen. Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples. arXiv preprint arXiv:2112.03185, 2021.
[80] Chong Zhou, Chen Change Loy, and Bo Dai. Extract Free Dense Labels from CLIP. In European Conference on Computer Vision (ECCV), 2022.
The Appendix section is structured as follows: In Sec. A, we provide further details about our OpenMask3D method, and we explain certain design choices. In Sec. B we present additional results, such as ablation studies on hyperparameters and insights into the performance of OpenMask3D. In Sec. C, we provide further details about how the baseline experiments are performed. In Sec. D, we share additional qualitative results.
A OpenMask3D details
A.1 Class-agnostic mask proposal module
As OpenMask3D is an instance-centric approach, we first need to obtain instance mask proposals. Since we are operating in an open-vocabulary setting, these instance masks are not associated with any class labels, they are class agnostic.
We use the mask module of the transformer-based Mask3D [58] architecture as the basis for our class-agnostic mask proposal module. Specifically, for our experiments on ScanNet200 [57], we use [58] trained on the training set of ScanNet200 [57] for instance segmentation. For our experiments on the Replica [61] dataset, we use the [58] module trained on the training and validation sets of the ScanNet200 [57] dataset, but without segments (see [58] for details on segments). We keep the weights of the mask proposal module frozen. Unlike Mask3D, our mask proposal module discards class predictions and mask confidence scores (which are based on class likelihoods), and we only retain the binary instance mask proposals.
The architecture consists of a sparse convolutional backbone based on the MinkowskiUNet [9], and a transformer decoder. Point features obtained from the feature backbone are passed through the transformer decoder, which iteratively refines the instance queries, and predicts an instance heatmap for each query. The query parameter specifies the desired number of mask proposals from the transformer-based architecture. We set the number of queries to 150, following the original implementation of Mask3D. This choice enables us to obtain a sufficient number of mask proposals for our open-vocabulary setting. The output from the final mask-module in the transformer decoder is a set of binary instance masks.
The original implementation of the Mask3D [58] model first ranks the proposed instance masks based on their confidence scores, and retains the top  $k$  masks based on this ranking. As the confidence scores are guided by class likelihoods, we do not utilize such scores to rank the masks, and do not filter out any of the proposed instance masks. Furthermore, since we aim to retain as many mask proposals as possible, we do not perform mask-filtering based on object sizes, or pairwise mask overlaps. In other words, we keep all masks proposed by the mask module as each of these masks might correspond to a potential open-vocabulary query.
As the model may occasionally output mask proposals that are not spatially contiguous, the original implementation of Mask3D employs the DBSCAN clustering algorithm [17] to break down such non-contiguous masks into smaller, spatially contiguous clusters. We follow this practice, and perform DBSCAN clustering on the instance masks, setting the epsilon parameter,  $\epsilon$ , to 0.95. This procedure often increases the final number of instance masks. As such, our OpenMask3D model generates class-agnostic instance masks, for which we compute open-vocabulary mask-features as described next.
A.2 Mask-feature computation module details
In the main paper, we outlined the steps to extract per-mask features in the CLIP [55] space. In this section, we will delve into the motivations behind specific design choices made throughout the process.
A.2.1 Per-mask view ranking: assumptions and visualizations