OpenMask3D: Open-Vocabulary 3D Instance Segmentation
Authors: Ayça Takmaz
Elisabetta Fedele
Robert Sumner
Marc Pollefeys
Federico Tombari
Francis Engelmann
Venue: NeurIPS 2023 poster (2023)
Keywords: open-world
open-vocabulary
3D vision
point cloud
instance segmentation
3D instance segmentation

Abstract: We introduce the task of open-vocabulary 3D instance segmentation. Current approaches for 3D instance segmentation can typically only recognize object categories from a pre-defined closed set of classes that are annotated in the training datasets. This results in important limitations for real-world applications where one might need to perform tasks guided by novel, open-vocabulary queries related to a wide variety of objects. Recently, open-vocabulary 3D scene understanding methods have emerged to address this problem by learning queryable features for each point in the scene. While such a representation can be directly employed to perform semantic segmentation, existing methods cannot separate multiple object instances. In this work, we address this limitation, and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D instance segmentation. Guided by predicted class-agnostic 3D instance masks, our model aggregates per-mask features via multi-view fusion of CLIP-based image embeddings. Experiments and ablation studies on ScanNet200 and Replica show that OpenMask3D outperforms other open-vocabulary methods, especially on the long-tail distribution. Qualitative experiments further showcase OpenMask3D’s ability to segment object properties based on free-form queries describing geometry, affordances, and materials.

OpenMask3D: Open-Vocabulaa 3D Instance Segmentation
Ayça Takmaz
Elisabetta Fedele
Robert W. Sumner!
Marc Pollefeys .22
Federico Tombari
Francis Engelmann' 1.3
ETH Zürich
Microsoft
Google
openmask3d.githu..i
"Sootrest"
"side tuble with a.flower vase on it"
"angel"
Figure 1: Open-Vonn Vocabulary 3D Instance Segmentation. Given 3D scene (top) and free-form user queries (bottom), our OpenMask3D segments object instances and scene parts described by the apen-vocabulag queries.
Abstract
We introduce the task of open-vocabulary 3D instance segmentation. Current approaches for 3D instance segmentation typically only recognize object cat- egories from a pre-defined closed set classes that are annotated in the training datasets. This results in important limitations for real-wor well-world applications where might need to perform tasks guided by novel, open-vocabulary queries related to a wide variety of objects. Recently, open-vocabulary 3D ods have emerged to address this problem by learning queryable features for each point in the scene. While such representation semantic segmentation, existing methods cannot separate multiple object instances. In this work, we address this limitation, and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D instance segmentation. Guided by predicted class-agnostic 3D instance masks, model aggregates per-mask fea- tures via multi-view fusion of CLIP- based image embeddings. Experiments and ablation studies on ScanNet200 and Replica show that OpenMask3D outperforms other open- voca-vocabllary methods, especially on the long-tail distribution. Qualitative experiments further showcase OakkMas'3D's ability to segment object properties based on free-form queries describing geometry, affordances, and materials.
understanding meth-
be directly employed to perform
Introduction
3D instance segmentation, which is the task of predicting 3D object instance masks along with their object categories, has many crucial applications fields such as robotics and augmented reality. Due to its significance, 3D instance segmentation has been receiving growing amount of attention recent years. Despite the remarkable progress made 3D instance segmentation methods [8, 9, 39, 58, 77, 78), it is noteworthy that these methods operate under a closed-set paradigm, where the set of object categories limited and closely tied to the datasets used during training.
We argue that there are two key problems with closed-vocabulary 3D instance segmentation. First, these approaches are limited in their ability understand a scene beyond the object categories seen during training. Despite the significant success of 3D instance segmentation approaches from recent years, these closeu-vocabulary approaches may fail recognize and correctly classify novel objects. One of the main advantages of open- vocabulocyylarry approaches, is their ability to zero-shot learn
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

categories that are not present at all in the training set. This ability has potential benefits for many applications in fields such as robotics, augmented reality, scene understanding and 3D visual search. For example, it is essential for an autonomous robot to be able to navigate in an unknown environment where novel objects can be present. Furthermore, the robot may need to perform an action based on free-form query, such as "find the side table with a flower vase on it", which is challenging to perform with the existing fooeedvocabuuarg 3D instance segmentation methods. Hence, the second key problem with flosed-vocabulat approaches is their inherent limitation to recognize only object classes that are predefined at training time.
In an attempt to address and overcome the limitations of a vossel-vccabllary setting, there has been a growing interest in open- vocabulary approaches. A line of work [20, 43, 45] investigates open-vocabulary 2D image segmentation task. These approaches are driven by advances in large-scale model training, and largely rely on recent foundation models such as CL.IP [55] and ALIGN [33] to obtain text-image embeddings. Motivated by the success of these 2D open-vocabulary approaches, another line of work has started exploring 3D open- vocabulary scene understanding task [24, 32, 52], based on the idea of lifting image features from models such as CLIP [55], LSeg [43], and OpenSeg [20] to 3D. These approaches aim to obtain a aask-sge feature representation for each 3D point in the scene, which can be used to query concepts with open-vocabulary descriptions such object semantics, affordances or material properties. Their output is typically a heatmap over the points in the scene, which has limited applications in certain aspects, such as handling object instances.
In this work, we propose OpenMask3D, an open-vocabulary 3D instance segmentation method which has the ability to reason beyond a predefined set of concepts. Given an RGB-D sequence, and the corresponding 3D reconstructed geometry, OpenMask3ID predicts 3D object instance masks, and computes mask-feature representation. Our two-stage pipeline consists of a class-agnostic mask proposal head, and mask-feature aggregation module. Guided by the predicted class-agnostic 3D instance masks, our mask-feature aggregation module first finds the frames in which the instances are highly visible. Then, it extracts CLIP features from the best images of each object mask, in a multi- scale and crop-based manner. These features are then aggregated across multiple views to obtain feature representation associated with each 3D instance mask. Our approach is intrinsically different from the existing 3D open- vocabulary scene understanding approaches [24, 32, 52] as we propose an instance-bart based feature computation approach instead of a point-based one. Computing a mask-feature per object instance enables us retrieve object instance masks based on their similarity to any given query, equipping our approach with open-vocabulary 3D instance segmentation capabilities. As feature computation is performed in zero-shot manner, OpenMask3I) is capable of preserving information about novel objects as well as long-tail objects better, compared to trained or fine- tuned counterparts. Furthermore, OpenMask3D goes beyond the limitations of closed-vocabulary paradigm, and allows segmentation of object instances based on free-form queries describing object properties such as semantics, geometry, affordances, and material properties.
Our contributions are fhree-fold:
We introduce the veccbbbuayl 3D instance segmentation task in which the object instances that are similar to a given text-query are identified.
We propose OpenMask3D, which is the first approach that performs open- vocabbucyy 3D instance segmentation in zero-shot manner.
We conduct experiments to provide insights about design choices that are important for developing an open-vocabulary 3D instance segmentation model.
2 Related work
Closed-vocabularyy 3D semantic and instance segmentation. Given 3D scene as input, 3D semantic segmentation task aims to assign a semantic category to each point in the scene [2-4, 9, 14, 15, 22, 28, 29, 31, 38, 42, 44, 46, 47, 53, 54, 63, 64, 66, 68, 70, 73]. 3D instance segmentation goes further by distinguishing multiple objects belonging to the same semantic category. predicting individual masks for each object instance [13, 16, 25, 27, 34, 41, 58, 62, 65, 67, 74]. The current of-te-of-the-ar approach on the ScanNer200 benchmark [10, 57] is Mask3D [58]. which leverages a transformer architecture to generate 3D mask proposals together with their semantic labels. However, similar to existing methods, it assumes limited set of semantic categories that can be assigned to an object instance. In particular, the number of labels is dictated by the annotations provided in the training datasets, 200- in the case of ScanNet200 [57]. Given that the English language encompasses

---

numerous nouns, in the order of several hundred thousand [69]. it is clear that existing closed- vocabulary approaches have an important limitation in handling object categories and descriptions.
Foundation models. Recent multimodal foundation models [1. 21, 33, 55, 75, 76] leverage large- scale pretraining to learn image representations guided by natural language descriptions. These models enable zero-shot transfer to various downstream tasks such as object recognition and classification. Driven by the progress large-scale model pre-training, similar foundation models for images were also explored in another line of work [5, 51], which aim to extract class-agnostic features from images. Recently, steps towards foundation model for image segmentation were taken with SAM [36]. SAM has the ability to generate a class- class-agnosie agnostic 2D mask for an object instance, given a set of points that belong to that instance. This capability is valuable for our approach, especially for recovering high-quality 2D masks from projected 3D instance masks, as further explained in Sec. 3.2.2.
Open vocabulary 2D segmentation. As large vision-language models gained popularity for image classification, numerous new approaches [11, 20, 23, 26, 32, 40, 43, 45, 48, 56, 71, 72, 79, 80] have emerged to tackle open-vocabulary or zero-shot image semantic segmentation. One notable shift was the transition from image-level embeddings to pixel-level embeddings, equipping models with localization capabilities alongside classification. However, methods with pixel-level embeddings, such as OpenSeg [20] and OV-Seg [45], strongly rely on the accuracy of 2D segmentation masks, and require a certain degree of training. In our work, we rely on CLIP [55] features without performing finetuming or any additional training, and compute 2D masks using the predicted 3D instance masks.
Open-vocabulary 3D scene understanding. Recent success of 2D apen-vocabulary segmentation models such as I.Seg [43], OpenSeg [20], and OV-Seg(45] has motivated researchers in the field of 3D scene understanding explore the open vocabulary setting [6, 12, 19, 24, 30, 32, 35, 37, 49, 52, 59, 60J. OpenScene [52] uses per-pixel image features extracted from posed images of a scene and obtains a point-wise task-agnostic scene representation. On the other hand, approaches such as LERF [35] and DFF [37] leverage the interpolation capabilities of NeRFs [50] to extract a semantic field of the scene. However, it is important to note that all of these approaches have a limited understanding of object instances and inherently face challenges when dealing with instance-related tasks.
3 Method
Overview. Our OpenMask3D model is illustrated in Fig. 2. Given a set of posed RGB-D images captured in scene, along with the reconstructed scene point cloud ①, OpenMask3D predicts 3D instance masks with their associated per-mask feature representations, which can be used for querying instances based on open-vocabulary concepts @ Our OpenMask3D has two main building blocks: a class agnostic mask proposal head @ and mask-feature computation module @. The class-agnostic mask proposal head predicts binary instance masks over the points in the point cloud. The mask-feature computation module leverages pre- pre-trained CLIP [55] Dision-languss model in order to compute meaningful and ficxible features for each mask. For each proposed instance mask, the mask-feature computation module first selects the views in which the 3D object instance is highly visible. Subsequently, in each selected view, the module computes a 2D segmentation mask guided by the projection of the 3D instance mask, and refines the 2D mask using the SAM [36] model. Next, the CLIP encoder is employed to obtain emnedddin of multi-scale image-crope bounding the computed 2D masks. These image-level embeddings are then aggregated across the selected frames in order to obtain a mask-feature representation. Sec. 3.1 describes the class agnostic mask proposal head, and Sec. 3.2 describes the mask-feature computation module.
The key novelty of our method is that it follows an instance-mask oriented approach, contrary to existing 3D open- vocabulary scene understanding models which typically compute per-point features. These point-feature oriented models have inherent limitations, particularly for identifying object instances. Our model aims overcome such limitations by introducing framework that employs class agnostic instance masks and aggregates informative features for each object instance.
Input. Our pipeline takes as input collection of posed RGB-D images captured in an indoor scene, and the reconstructed point cloud representation of the scene. We assume known camera parameters.
3.1 Class agnostic mask proposals
The first step of our approach involves generating M class-agnostic 3D mask proposals m@D SDD Let P E Px3 denote the point cloud of the scene, where each 3D point is represented with its corresponding 3D coordinates. Each 3D mask proposal is represented by
3

---

Input
Open-Yocabulag Instunce Segmentation
Query: "witch a moviee
3D Mask
Proposal Network
( Class Agnostic Instance Mask Proposals
recomstructed goometry
F
selecting upp views
2D segment crop
pet-crop CLIP features
per-mask features
posed RGB-D frames
mask .4
queryable features for each 3D instance mask
Mask-Feature Computation for Each Instance
Figure 2: An overview of our approach. We propose MasnMa the first segmentation model. Our pipcline consists of four subsequent steps: ! Our approach takes as input posed RGB-D images of 3D indoor scene along with its reconstructed point cloud. 2 Using the point cloud, we compute class-agnostic instance mask proposals. 3 Then, for each mask, we compute feature representation. 4 Finally, we obtain open-vocabulary 3D instance segmentation representation, which can be used to retrieve objects related to queried concepts embedded in the CLIP [55] space.
3D instance
binary mask m (m m where mi с {0, 13 indicates whether the j-th point belongs to i-th object instance. To generate these masks, leverage the transformer-basee mask-modulc of a pre-trained 3D instance segmentation model [58]. which is frozen during our computations. The architecture consists of sparse convolutional backbone based on the MinkowskiUNet [9], and a transformer decoder. Point features obtained from the feature backbone are passed through the transformer decoder, which iteratively refines the instance queries, and predicts an instance heatmap for each query. In the original setup. [58] produces two outputs: a set of M binary instance masks obtained from the predicted heatmaps, along with predicted class labels (from predefined closed set) for each mask. In our approach, we adapt the model to exclusively utilize the binary instance masks, discarding the predicted class labels and confidence scores entirely. These class-agnostic binary instance masks are then utilized in our mask-feature computation module, in order to go beyond semantic class predictions limited to a closed-vocabulary, and obtain open-vocabulay representations instead. Further details about the class-agnostic mask proposal module are provided in Appendix A.
3.2 Mask-feature computation module
Mask-feature computation module aims to compute task-agnostic feature representation for each predicted instance mask obtained from the class-agnostic mask proposal module. The purpose of this module is to compute a feature representation that can be used to query open-vocabulary concepts. As we intend to utilize the CLIP text-image embedding space and maximally retain information about long-tail or novel concepts, we solely rely the CLIP visual encoder to extract image-features on which we build our mask-features.
As illustrated in Fig. 3, the mask-feature computation module consists of several steps. For each instance mask-proposal, we first compute the visibility of the object instance in each frame of the RGB-D sequence, and select the top-k views with maximal visibility. In the next step, we compute 2D object mask in each selected frame, which then used obtaín multi-scale image-crops in order to extract effective CL.IP features. The image-crops are then passed through the CL.IP visual encoder to obtain feature vectors that are average-pooles over each crop and each selected view, resulting in the final mask-featur representation. In Sec. 3.2.1, we describe how we select set of top-k frames for each instance. In Sec. 3.2.2, we describe how we crop the frames, based on the object instance we want to embed. In Sec. 3.2.3, we describe how we compute the final mask-features per object.
3.2.1 Frame selection
Obtaining representative images of the proposed object instances is crucial for extracting accurate CLIP features. To achieve this, we devise a strategy to select, for each the M predicted instances, a subset of representative frames (Fig. 3, @ from which we extract CLIP features. In particular, our devised strategy selects frames based on their visibility scores Sij for each mask each view j. Here, we explain how we compute these visibility scores.

---

Select top views
e 2D segment & crop
Extract per-erop CLIP features aggregate
Agrrgatte features per each instance mask
CLIP visual encoder
1o-4
20
60-11
Multi-scale m
VicW
Computation Module. For each instance mask, @ we first compute the visibilityy of the instance each frame, and select top-k views with maximal visibility. In ①. compute 2D object mask each selected frame, which is used to obtain multi-scale image-crops order to extract effective CLIP features. @ The image-crops are then passed through the CLIP visual encoder obtain feature vectors that are average-pooled over each crop and @ cach selected view, resulting in the final mask-fcature representation.
Figure 3:
Given a mask proposal m 3D corresponding to the i-th instance, we compute the visibility score Sij [0, 1] for the j-th frame using the following formula:
8ij vis (isj)
maxy (vis (vis(i, (i)
Here, (i) represents the number of points from mask that are visible in frame j. Note that we assume that each mask is visible in at least one frame. We compute vis(i,)) using the following approach. First, for each 3D point from the i-th instance mask, we determine whether the point appears in the camera's field of view (FOV) in the j-th frame. To achieve this, we use intrinsic (I) and extrinsic (R)t) matrices of the camera from that frame to project each 3D point in the point cloud to the image plane. We obtain corresponding 2D homogeneous coordinates P2D ( [u, v, w) = (R|() x. where x (x, y, 1)T is point represented in homogeneous coordinates. We consider a point to be in the camera's s FOV the j-th frame if w * O, and the w value falls within the interval [o, 1] while 2 falls within [0, H 1]. Here W and H represent the width and height of the 10 image, respectively. Next, it is important to note that 3D points that are in the camera's 's FOV are not necessarily visible from a given camera pose, as they might be occluded by other parts of the scene. Hence, we need to check whether the points are in fact visible. In order to examine whether point is occluded from given camera viewpoint, we compare the depth value of the 3D point deriving from the geometric computation (i.e. w) and the measured depth, i.e. depth value of the projected point in the depth image. We consider points that satisfy the inequality w > kitreshold as occluded, where kirreshodd is a hyper-paramete of our method. If the projection of a point from mask i is not occluded in the j-th th view, contributes to the visible point count, vis(i, (i.j). Once we compute visibiisibilltty scores for each frame, we select the top kview views with the highest scores siz for each mask i. Here, kview represents another hyperparameter.
3.2.2 2D mask computation and multi-scale crops
In this section, we explain our approach for computing CLIP features based the frames selected in the previous step. Given a selected frame for a mask, our goal is to find the optimal image crops from of the projected points of
which to extract features, as shown (Fig. 3. Q). Simply considering the mask often results in imprecise and noisy bounding boxes, largely affected by the outliers (please see Appendix A.2.2). To address this, we use a aaass-ag 2D segmentation model, SAM [36], which predicts a 2D mask conditioned on a set of input points, along with a mask confidence score.
SAM is sensitive the set of input points (see Appendix A.2.3, A.2.4). Therefore, obtain high quality mask from SAM with a high confidence score, we take inspiration from the RANSAC algorithm [18] and proceed described in Algorithm 1. We sample kaample points from the projected points, and run SAM using these points. The output of SAM at a given iteration r is a 2D mask (m?D) and mask confidence score (score,). This process is repeated for krounds iterations, and the 2D mask with the highest confidence score is selected.
Next, we use the resulting mask 2D to generate
multi-level crops of the selected image. This allows us to enrich the features by incorporating more contextual information from the surroundings.
5

---

Algorithm 2D mask selection algorithm
score, 0, . 0
while < kyounds do
Sample ksampte points among the projected points random Compute the mask Dn? and the score score, based on the sampled points using SAM if score, > score, then score, score,, m, m.
end if r' + r+11 end while
Specifically, the first bounding box b = (z), 2i. 12) with S æ < æ < W and S < y < H is tight bounding box derived from the 2D mask. The other bounding boxes b² and bd incrementally larger. In particular, the 2D coordinates of b² for i 2, are obtained as follows.
2 max (0, . - (x) - -x1) keap" 1)
v1 max(0, v1 (3) 1/1) Kezz 1)
2/ = min(2) + (a) x1) keep 1, -W
3/2 = min(yá (v2 - -i) Keep -1H
Note that here / represents the level of the features and keep = 0.1 is predefined constant.
3.2.3 CLIP feature extraction and mask-feature aggregation
For each instance mask, we collect k- L images by selecting top-k views and obtaining L multi-level crops as described Sec. 3.2.1 and Sec. 3.2.2. Collected image crops are then passed through the CLIP visual encoder in order to extract image features in the CI.IP embedding space, as shown in given
(Fig. 3, @). We then aggregate the features obtained from each crop that correspond instance mask in order to get average per-mask CLIP feature (Fig. 3, @). The computed features are aann-tgn,,,,c, and can be used for various mnstance-based tasks by encoding given text or image-based query, using the same CLIP model employed to encode the image crops.
4 Experiments
In this section, we present quantitative and qualitative results from our method OpenMask3D. In Sec 4.1, we quantitatively evaluate our method, and compare OpenMask3D with supervised 3D instance segmentation approaches as well as existing open-vocabulary 3D scene understanding models we adapted for the 3D instance segmentation task. Furthermore, we provide an ablation study for OpenMask3I. In Sec. 4.2, we share qualitative results for open-vocabulary 3D instance segmentation, demonstrating potential applications. Additional results are provided in the Appendix.
4.1
Quantitative results: closed-vocabulary 3D instance segmentation evaluation
We evaluate our approach on the closed- vocabulary 3D instance segmentation task. We conduct additional experiments to assess the generalization capability of OpenMask3ID.
4.1.1
Experimental setting
Data. We conduct our experiments using the ScanNer200 [57] and Replica [61] datasets. We report our ScanNet200 results on the validation set consisting of 312 scenes, and evaluate for the 3D instance segmentation task using the closed vocabulary of 200 categories from the ScanNet200 annotations. Rozenberszki et al. [57] also provide a grouping of ScanNet200 categories based on the frequency of the number of labeled surface points in the training set, resulting in 3 subsets: head (66 categories), common (68 categories), tail (66 categories). This grouping enables us to evaluate the performance of evaluation dataset.
method on the long-tail distribution, making ScanNet200 natural choice To assess the generalization capability of our method, we further experiment with the Replica [61] dataset, and evaluate the office0, officel, office2, office3, office4, roomO, rooml, room2 scenes.
Metrics. We employ a commonly used 3D instance segmentation metric, average precision (AP). AP scores are evaluated at mask overlap thresholds of 50% and 25%, and averaged over the overlap

---

Model
Image Features
AP
APso
AP25
head (AP)
common (AP)
tail (AP)
Cloxed-vocabulory fully supervised
Mask3D [58]
26.9
36.2
41.4
21.7
17.9
Open-vocabulays
OpenScene [52] (2D Fusion) masks
OpenSeg [20]
11.7
15.2
17.8
134
11.6
9.9
OpenScene [52] (3D Distill) masks
OpenSeg [20]
4.8
6.2
7.2
10.6
2.6
0.7
OpenScene [52] (2D/3D Ens.) + masks
OpenSeg [20]
5.3
6.7
8.1
11.0
3.2
1.1
OpenSeene [52] (2D Fusion) masks
LSeg [43]
6.0
7.7
8.5
14.5
2.5
1.1
OpenMask3D (Ours)
CLIP [55]
15.4
19.9
23.1
17.1
14.1
14.9
Table 1:
3D instance segmentation results the ScanNet200 validation set. Metrics respectively: AP averaged over an overlap range, and AP evaluated at 50% and 25% overlaps. We also report AP scores for head, common, tail subsets of ScanNet200. Mask3D [58] is fully-supervised, while OpenScene [52] is built upon on 2D models (LSeg [43], OpenSeg [20] trained on labeled datasets for 2D semantic segmentation. Since OpenScene [52] does provide instance masks, aggregate its per-point features using class-agnostic masks. OpenMask3D outperforms other open-vocabulary counterparts, particularly on the long-tail classes.
range of [0.5 : 0.95: 0.05] following the evaluation scheme from ScanNet [10]. Computation of the metries requires each mask to be assigned prediction confidence score. We assign prediction confidence score of 1.0 for each predicted mask in our experiments.
OpenMask33D implementation details. We use posed RGB-depth pairs for both the ScanNer200 and Replica datasets, and we process frame in every 10 frames the RGB-D sequences. order to compute image features on the mask-crops, we use CLIP [55] visual encoder from the ViT-L/14 336 pixel resolution, which has a feature dimensionality of 768. For the visibility score computation, we use kehreshald = 0.2, and for top-view selection we use kniww = 5. In all experiments with multi-scale crops, we use = levels. In the 2D mask selection algorithm based on SAM [36], we repeat the process for krounds = 10 rounds, and sample Ksample 5 points at each iteration. For the class-agnostic mask proposals, we use the Mask3D [58] model trained on ScanNet200 instances, and exclusively use the predicted binary instance masks in our pipeline. We do not filter any instance mask proposals, and run DBSCAN [17] to obtain spatially contiguous clusters, breaking down masks into smaller masks when necessary. This process results in varying number of mask proposals for each scene. For further implementation details about OpenMask3D, please refer to Appendix A. Computation of the mask-features of a ScanNet scene single GPU takes 5-10 minutes depending on the number mask proposals and number of frames in the RGB-D sequence. Note that the per-mask features of the scene computed, objects can be queried in real-time (~ 1-2 ms) with arbitrary open- vec-bvoayylaary queries.
model pre-trained
Methods in comparison. We compare with Mask3D [58]. which is the current state-of-the-art on the ScanNer200 3D instance segmentation benchmark. We also compare against recent open-vocabulary 3D scene understanding model variants (2D Fusion, 3D Distill, 2D/3D Ensemble) from OpenScene [52]. Note that since OpenScene only generates a per- per-point feature vector (without any per-instance aggregation), it is not possible to directly compare it with our method. To address this, we extended OpenScene by averaging its per- per-point features within each instance mask generated by our mask module (see Appendix C for more details).
Class assignment. Our per each instance mask, but it instead computes a task-agnostic feature vector for each instance, which can be used for performing semantic label assignment. In order to evaluate our model on the flosed-oocabulary 3D instance segmentation task, we need to assign each object instance to semantic category. Similar to OpenScene [52], we compute cosine similarity between mask-features and the text embedding of a given query in order to perform class assignments. Following Peng et al. [52]. we use prompts in the form of "a {) in a scene", and compute text-embeddings using CLIP model ViT-L/14(336xx) [55] for each semantic class in the ScanNet200 dataset. This way, compute a similarity score between each instance and each object category, and assign instances to the category with the closest text embedding.
approach does not directly predict semantic category labels
4.1.2 Results and analysis
3D dosed-vocabulare instance segmentation results. We quantitatively evaluate our approach on the datasets. Closed- vocabulary instance segmentation results are provided in Tab. I and Tab. 2.
instance segmentation task on the ScanNe1200 [10, 57] and Replica [61]
7

---

State-of of aa- fully supervised approach Mask3D [58] demonstrates superior performance compared to open- ppen-vocauulary counterparts. While this gap more prominent for the head and common categories, the difference is less apparent for the tail categories. This outcome is expected, as Mask3D benefits from full-supervision using the closed-set of class labels from the ScanNer200 dataset. Furthermore, as there are more training samples from the categories within the head and common subsets (please refer to [57] for the statistics), the fully-supervised approach is more frequently exposed to these categories resulting in a stronger performance. When we compare OpenMask3D with other open open-vocabulary approaches, we observe that our method performs better on 6 out of 6 metrics. Notably, OpenMask3D outperforms other open-vocabulle approaches especially on the tail categories by a significant margin. Our instance-centric method OpenMask3D, which is specifically designed for the open-vocabulary 3D instance segmentation task, shows stronger performance compared to other open-vocabulary approaches which adopt a point-centric feature representation. Differences between the feature representations also illustrated in Fig. 5.
How well does our method generalize? As the mask module is trained on a closed-set segmentation dataset, ScanNet200 [57]. we aim to investigate how well our method generalizes beyond the categories seen during training. Furthermore, we aim to analyze how well our method generalizes to oullof-diitribution (OOD) data, such scenes from another dataset, Replica [61]. To demonstrate the generalization capability of our approach, we conducted a series of experiments. First, we analyze the performance of our model when use -annostg masks from mask-predicter trained on the 20 original ScanNet classes [10], and evaluate on the ScanNet200 dataset. To evaluate how well our model performs on "unseen" categories, we group the ScanNet200 labels into two subsets: base and novel classes. We identify ScanNer200 categories that are semantically similar to the original ScanNer20 classes (e.g. chair and folded-chair, table and dining-table), resulting in 53 classes. We group all remaining object classes that are not similar to any class in ScanNet20 as "novel". In Tab. 3, we report results on seen ("base") classes, unseen ("novel") classes, and all classes. Our experiments show that the OpenMask3D variant using a mask proposal backbone trained on smaller set of object annotations from ScanNet20 can generalize set of objects (ScanNet200), resulting in only a marginal decrease in the performance. In a second experiment, we evaluate the performance of OpenMask3D on -iit--i-iititttioonn data from Replica, using a mask predictor trained on ScanNet200. The results from this experiment, as presented in Tab. 2, indicate that OpnnMask3D can indeed generalize to unseen categories as well as OOD data. OpenMask3D using mask predictor module trained on smaller set of objects seems to perform reasonably well in generalizing to various settings.
predict object masks from a significantly larger
Model
AP
AP50
AP25
OpenScene [52] (2D Fusion) + masks
10.9
15.6
17.3
OpenScene [52] (3D Distill) + masks
8.2
10.5
12.6
OpenScene [52] (2D/3D Ens.) + masks
8.2
10.4
13.3
OpenMask3D (Ours)
13.1
18.4
24.2
Table 2: 3D instance segmentation results on the Replica [61] dataset. To assess how well our model generalizes to other datasets, we use instance masks from the mask proposal module trained on ScanNet200, and test it on Replica scenes. OpenMask3D outperforms other open-vocabulary counterparts on the Replica dataset.
Novel Classes
Base Classes
All Classes
Method
Musk Training
AP
APso
AP26
AP
AP50
AP2s
AP
tail (AP)
OpenScene [52] (21) Fusion) masks
Ner20
7.6
10.3
12.3
11.1
15.0
17.7
6.1
OpenSeene [52] (3D Distill) masks
ScanNer20
1.8
2.3
2.7
10.1
13.4
15.4
4.1
0.4
OpenScene [52) (2D/3D Ens.) masks
ScunNet20
2.4
2.8
3.3
10.4
13.7
16.3
4.6
0.9
OpenMask3D (Ours)
NeunNet20
11.9
15.2
17.8
14.3
18.3
21.2
12.6
11.5
OpenMask3D (Ours)
ScunNet200
15.0
19.7
23.1
16.2
20.6
23.1
15.4
14.9
Table 3: 3D instance segmentation results using masks from mask module trained on ScanNet20 annota- tions, evaluated on the ScanNet200 dataset [57]. We identify 53 classes (such as chair, folded chair, table, dining table... ...) that are semantically close to the original ScanNer20 classes [10], and group them as "Base". Remaining 147 classes are grouped "Novel". We also report results on the full set of labels, titled "All".
Ablation study. In Tab. 4, we analyze design choices for OpenMask3D, i.e., multi-scale cropping and 2D mask segmentation. 2D mask segmentation refers to whether we use SAM [36] for refining
8

---

2D Mask Multi-Scale
AP
APso
AP25
head(AP)
common(AP)
tail(AP)
12.9
16.6
19.3
15.1
12.2
11.3
14.3
18.4
21.1
16.1
13.6
12.9
14.1
18.3
21.5
16.0
13.0
13.4
15.4
19.9
23.1
17.1
14.1
14.9
Table 4: OpenMask33D Ablation Study. 2D mask and multi-scale crop components. 21) mask refers whether
SAM [36] was employed for computing 2D masks. Results are reported the ScanNer200 [10] validation
Model
Oracle
Img. Feat.
AP
head(AP)
common(AP)
taik(AP)
Cloved-vocabulary, full sup.
Mask3D [58]
3
26.9 35.5
39.8
21.7
17.9
Mask3D [58]
55.2
27.2
22.2
Open-vocabull
OpenScene [52] (2D Fusion) musks
OpenSeg [20]
22.9
26.2
22.0
20.2
OpenScene [52] (2D Fusion) musks
LSeg [43]
11.8
26.9
5.2
1.7
OpenMask3D (Ours)
CLIP [55]
29.1
31.1
24.0
32.9
Table 5: 3D instance segmentation results on the ScanNet200 validation set, using oracle masks. We use ground truth instance masks for computing the per-mask features. We also report results from the fully- supervised close-vocabllary Mask3D (row 2) whose predicted masks we match to the oracle masks using Hungarian matching, and assign the predicted labels to oracle masks. On the long-tail categories, OpenMask3D outperforms even the fully supervised Mask3D with oracle masks
the 2D mask from which we obtain a 2D bounding box. When SAM is not used, we simply crop the tightest bounding box around the projected 3D points. The ablation study shows that both multi-scale cropping and segmenting 2D masks obtain more accurate image crops for a given object instance positively affect the performance. Additional experiments analyzing the importance of the number of views (k) used in top-k view selection provided in Appendix B.I.
How well would our approach perform if we had perfect masks? Another analysis we conduct is related to the class-agnostic masks. As the quality of the masks plays key role in our process, we aim to quantify how much the performance could improve we had "perfect" oracle masks. For this purpose,
Mas33as33D using ground truth instance masks from the ScanNer200 dataset instead of using our predicted alass-agnostic instance masks. In a similar fashion for the baselines, the oracle masks to aggregate OpenScene per-point features to obtain per-mask features. We first compare against the fully-supervised Mask3D performance (Tab. 5, row 1). In a second experiment, we also supply oracle masks Mask3D (Tab. 5, row to ensure fair comparison. For this experiment, we perform Hungarian matching between the predicted masks and oracle masks discarding all class-losses, and we only match based on the masks. To each oracle mask, we assign mask from Mask3D. In Tab. 5, it is evident that
the predicted class label of the closest-matchin the quality of the masks plays an important role for our task. Remarkably, our feature computation approach that is not trained on any additional labeled data, when provided with oracle masks, even surpasses the performance of the fully supervised method Mask3D (by +15.0% AP) and Mask3D with oracle masks (by +10.7% AP) on the long- tail categories. Overall, this analysis indicates that our approach has indeed promising results which can be further improved by higher quality class-agnostic masks, without requiring any additional training or finetuning with labeled data.
4.2 Qualitative results
In Fig. 4, we share qualitative results from our approach for the open-vocabulary 3D instance segmentation task. With its zero-shot learning capabilities, OpenMask3D is able segment given query object that might not be present in common segmentation datasets. Furthermore, object properties such as colors, textures, situational context and affordances are successfully recognized by OpenMask3D. Additional qualitative results in this direction are provided in Appendix D.
In Fig. 5, we provide qualitative comparisons between instance-basee based open-vocabulas represen- tation and the point-based representation provided by OpenScene [52]. Our method OpenMask3D computes the similarity between the query embedding and per-mask feature vectors for each object instance, which results in crisp instance boundaries. This is particularly suitable for the use cases in which one needs to identify object instances. Additional analysis and further details about the visualizations are provided in Appendix C.

---

"epsii"
"pool"
"an armchair with floral print"
"a bed without clothes on it"
"cool down"
"a green seut"
Figure 4: Qualitative results from OpenMask3D. Our open- open-vocabulary instance segmentation approach capable of handling different types of queries. Novel object classes as well as objects described by colors, textures, situational context and affordances are successfully retrieved by OpenMask3D.
Query: "a dollhouse"
OpenScene [52]
OpenMask3D (Ours)
Figure 5: Heatmaps showing the similarity between given text queries and vocabullyy features. Input 3D scene and query (left), per-point similarity from OpenScene (middle) and per-mask similarity from OpenMask3D (right). Dark red means high similarity, and dark blue means low similarity with the query text.
4.3 Limitations.
The experiments conducted with oracle masks indicate that there is room for improvement in terms of the quality of 3D mask proposals which will be addressed future work. Further, since the per-mask features originate from images, they can only encode scene context visible in the camera frustum, lacking a global understanding the complete scene and spatial relationships between all scene elements. Finally, evaluation methodologies for systematically assessing open- -occbuuayy capabilities still remain a challenge. dlosed-vocabulars evaluations, while valuable for initial assessments, fall short in revealing the true extent of open-vocabulary potentials of proposed models.
Conclusion
We propose OpenMask3D, the first open-vocabulary 3D instance segmentation model that can identify objects instances in 3D scene, given arbitrary text queries. This is beyond the capabilities of existing 3D semantic instance segmentation approaches, which are typically trained to predict categories from closed vocabulary. With OpenMask3ID, we push the boundaries of 3D instance segmentation. Our method is capable of segmenting object instances in a given 3D scene, guided by open-vocabulary queries describing object properties such semantics, geometry, affordances, material properties and situational context. Thanks to its zero-shot learning capabilities, OpenMask3D is able to segment multiple instances of given query object that might be present in common segmentation datasets on which glosed-vocabulary instance segmentation approaches are trained. This opens up new possibilities for understanding and interacting with 3D scenes more comprehensive and flexible manner. We encourage the research community to explore opcc-vocayylary approaches, where knowledge from different modalities can be seamlessly integrated into unified and coherent space.
10

---

Acknowledgments and disclosure of funding, Francis Engelmann is a postdoctoral research fellow the ETH AI Center. This project is partially funded by the ETH Career Seed Award "Towards WrendWorld 3D Scene Understanding", and Innosuisse grant (48727.1 IP-ICT). We sincerely thank Jonas Schult for helpful discussions, and Lorenzo Liso for the help with setting up our live demo.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc. Antoine Miech, lain Barr, Yana Hasson, Karel Lenc. Arthur Mensch, Katherine Millican, Malcolm Reynolds. Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro, Jacob Menick, Sebastian Borgeaud. Andrew Brock, Aida Nematzadoh, Sahand Sharifzadch, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a Visual Language Model for Few-Shot Learning. In Neural Information Processing Systems (NeurIPS), 2022.
[2] Abhishek Anand, Hema Swetha Koppula, Thorsten Joachims, and Ashutosh Saxena. Contextually Guided Semantic Labeling and Search for 3D Point Clouds. In International Journal Robotics Research (IJRR), 2011.
[3] Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Joannis Brilakis, Martin Fischer, and Silvio Savarese. 3D Semantic Parsing of Large-Scale Indoor Spaces. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[4] Matan Atzmon, Haggai Maron, and Yaron Lipman. Point Convolutional Neural Networks by Extension Operators. In ACM Transactions On Graphics (TOG), 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Suuee Vision Transformers. International Conference on Computer Vision (ICCV). 2021.
[6] Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S. Ryoo, Austin Stone, and Daniel Kappler. Open-vocabulary Queryable Scene Representations for Real World Planning. arXiv preprint (tt::22000098777 2022.
[7] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-inage learning. arXiv preprint 23 2022.
[8] Julian Chibane, Francis Engelmann, Tuan Anh Tran, and Gerard Pons-Moll. Box2Mask: vised 3D Semantic Instance Segmentation using Bounding Boxcs. European Conference on Computer Vision (ECCV), 2022.
Weakly Super-
[9] Christopher Choy, Young Gwak, and Silvio Savarese. 4D Spatio- Temporal ConyNets: Minkowski Convolutional Neural Networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[10] Angela Dai, Angel X Chang, Manolis Savva, Macicj Halher, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-Annotan 3D Reconstructions of Indoor Scenes. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[11] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling Zero-Shot Semantic Segmentation. In CVPR, 2022.
[12] Runyu Ding, Jihan Yang. Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. PLA: Language-Driven Open-Vocabulle 3D Scene Understanding. In CVPR, 2023.
[13] Cathrin Elich. Francis Engelmann, Theodora Kontogianni. and Bastian Leibe. 3D-BEVIS: Birds-Eye-Viee Instance Segmentation. In German Conference on Pattern Recognition (GCPR).201)
[14] Francis Engelmann, Theodora Kontogianni, Alexander Hermans, and Bastian Leibe. Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds. In International Conference on Computer Vision (ICCV) Workshops, 2017.
[15] Francis Engelmann, Theodora Kontogianni, Jonas Schult, and Bastian Leibe. Know What Your Neighbors Do: 3D Semantic Segmentation of Point Clouds. In European Conference on Computer Vision (ECCV) Workshops, 2018.
11

---

[16] Francis Engelmann. Martin Bokeloh, Alireza Fathi, Bastian Leibe. and Matthias NieBner. 3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation. In Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[17] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A Density-Basee Algorithm for Dis- covering Clusters in Large Spatial Databases with Noise. In Knowledge Discovery and Data Mining, 1996.
[18] Martin A. Fischler and Robert C. Bolles. Random sample consensus: a paradigm for model fitting with applications image analysis and automated cartography. Commun. ACM, 24:381-395, 1981.
[19] Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Clip on wheels: Zero-shot object navigation as object localization and exploration. arXiv preprtint aXXiv:200.004429 2022.
[20] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling vpen-Vocabullay Image Segmentation with Image-Level Labels. In European Conference Computer Vision (ECCV), 2021.
[21] Rohit Girdhar, Alaacldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. ImageBind: One Embedding Space To Bind Them All. Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[22] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks. In Conference Computer Vision and Puttern Recognition (CVPR), 2018.
[23] Xiuye Gu, Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary Object Detection via Vision and Language Knowledge Distillation. In International Conference on Learning Representations (ICLR), 2022.
[24] Huy Ha and Shuran Song. Semantic Abstraction: Open- 3D Scene Understanding from 2D Vision- Language Models. In CORL, 2022.
[25] Lei Han, Tian Zheng, Lan Xu, and Lu Fang, OccuSeg: Occupancy- 3D Instance Segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[26] Sunan He, Taian Guo, Tao Dai, Ruizhi Qiao, Bo Ren, and Shu- Tao Xia. Open- Vhen-Vocabulary Multi-Labe Classification via Multi-modal Knowledge Transfer. In AAAI, 2023.
[27] Ji Hou, Angela Dai, and Matthias Nießner. 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[28] Zeyu Hu, Xuyang Bai, Jiaxiang Shang. Runze Zhang, Jiayu Dong, Xin Wang, Guangyuan Sun, Hongho Fu, and Chiew Lan Tai. VMNet: toxel-Mesh Network for Geodesic-Awart 3D Semantic Segmentation. In International Conference Computer Vision (ICCV), 2021.
[29] Binh-Son Hua, Minh-Khoi Tran, and Sai-Kit Yeung. Point-wise Convolutional Neural Network. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[30] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual Language Maps for Robot Navigation. In Proceedings of the IEFE International Conference on Robotics and Automation (ICRA), London, UK, 2023.
[31] Jing Huang and Suya You. Point Cloud Labeling Using 3D Convolutional Neural Network. In International Conference on Pattern Recognition (ICPR), 2016.
[32] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer. Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. ConceptFusion: Open-Set Multimodal 3D Mapping. In Robotics: Science and Systems (RSS), 2023.
[33] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hieu Pham, Quoe V. Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. In International Conference on Machine Learning (ICML), 2021.
[34] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi- Wing Fu, and Jiaya Jia. PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
12

---

[35] Justin Kerr. Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. LERF: Language Embedded Radiance Fields. In ICCV, 2023.
[36] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao, Spencer Whitehead, Alexander C. Berg. Wan-Yen Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In (CCV,2023
[37] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing NeRF for Editing via Feature Field Distillation. Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[38] Hema Koppula, Abhishek Anand, Thorsten Joachims, and Ashutosh Saxena, Semantic Labeling of 3D Point Clouds for Indoor Scones. In Neural Information Processing Systems (NewrIPS), 2011.
[39] Lars Kreuzberg, Idil Esen Zulfikar, Sabarinath Mahadevan, Francis Engelmann. and Bastian Leibe. 4D-STOP: Panoptic Segmentation 4D Lidar using Opatio-Tempont Object Proposal Generation and Aggregation. In European Conference on Computer Vision (ECCV) Workshops, 2022.
[40] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. Open- Vocabulary Object Detec- tion upon Frozen Vision and Language Models, In International Conference on Learning Representations (ICLR), 2023.
[41] Jean Lahoud, Bernard Ghanem, Marc Pollefeys, and Martin R. Oswald. 3D Instance Segmentation via Multi-task Metric Learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[42] Loic Landrieu and Martin Simonovsky. Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs. In Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
[43] Boyi Li, Kilian Q. Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranfil, Language-driven Semantic Segmentation. In International Conference on Learning Representations (ICLR), 2022.
[44] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. PointCNN: Convolution on X-transformed Points. In Neural Information Processing Systems (NeurIPS), 2018.
[45] Feng Liang. Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Vecab-oocaulary Semantic Segmentation with Mask -asktadd CLIP. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[46] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully Convolutional Networks for Semantic Segmen- tation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[47] Yan Lu and Christopher Rasmussen. Simplified Markov Random Fields for Efficient Semantic Labeling of 3D Point Clouds. In International Conference Intelligent Robots and Systems (ICIRS). 2012.
[48] Chao Ma, Hu-Hao Yang. Yanfeng Wang, Ya Zhang, and Weidi Xie. Open-vocabulay Semantic Segmenta- tion with Frozen Vision Vision-Language Models. In British Machine Vision Conference (BMVC), 2022.
[49] Kirill Mazur, Edgar Sucar, and Andrew Davison. Feature- Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understanding. In International Conference on Robotics and Automation (ICRA), 2023.
[50] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NcRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[51] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang, Hu Xu, Vasu Sharma, Shang-Wer Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. arXiv preprint arXiv:2304.00 02.30040719,, 07193, 2023.
[52] Songyou Peng. Kyle Genova, Chiyu "Max" Jiang, Andrea Tagliasacchi, Mare Pollefeys, and Thomas Funkhouser. OpenScene: 3D Scene Understanding with Open Vocabularies. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[53] Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR). 2017.
[54] Charles R. Qi. Li Yi, Hao Su, and Leonidas J. Guibas. PointNet++: Deep Hierarchical Feature Learning Point Sets in Metric Space. In Neural Information Processing Systems (NeurIPS), 2017.
13

---

[55] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. International Conference on Machine Learning (ICML), 2021.
[56] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. DenseCLIP: Language-Guide Guided Dense Prediction with Context-Awar Prompting. In Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
[57] David Rozenberszki, Or Litany, and Angela Dai. Language- Language-Grounded Indoor 3D Semantic Segmentation in the Wild. In European Conference on Computer Vision (ECCV), 2022.
[58] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3D: Mask Transformer for 3D Semantic Instance Segmentation. In International Conference on Robotics and Automation (ICRA), 2023.
[59] Nur Muhammad Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur D. Szlam. CLIP- Fields: Weakly Supervised Semantic Fields for Robotic Memory. 2023.
[60] Dhruv Shah, Blazej Osinski, Brian lehter, and Sergey Levine. Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action. 2022.
[61]
Julian Straub, Thomas Whelan. Lingni Ma. Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel. Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge. Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe, The Replica dataset: preprint arXiv: 057977 2019.
digital replica of indoor spaces. arXiv
[62] Ayça Takmaz, Jonas Schult, Irem Kaftan, Mertean Akçay, Bastian Leibe, Robert Sumner, Francis Engel- mann, and Siyu Tang. 3D Segmentation of Humans in Point Clouds with Synthetic Data. In International Conference on Computer Vision (ICCV), 2023.
[63] Lyne P. Tchapmi, Christopher B. Choy, Iro Armeni, Young Gwak, and Silvio Savarese. SEGCloud: Semantic Segmentation of 3D Point Clouds. In International Conference on 3D Vision (3DV), 2017.
[64] Hugues Thomas, Charles R. Qi, dean-Fmmanuul Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas. KPConw: Flexible and Deformable Convolution for Point Clouds. In International Conference on Computer Vision (ICCV), 2019.
[65] Thang Vu, Kookhoi Kim, Tung M. Luu, Xuan Thanh Nguyen, and Chang D. Yoo. SoftGroup for 3D Instance Segmentation on 3D Point Clouds. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[66] Tianyi Wang, Jian Li, and Xiangjing An. An Efficient Scene Semantic Labeling Approach for 3D Point Cloud. In IEEE International Conference on Intelligent Transportation Systems (ITSC), 2015.
[67] Weiyue Wang, Ronald Yu, Qiangui Huang, and Urrihh Neumann. SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[68] Silvan Weder, Hermann Blum, Francis Engelmann, and Mare Pollefeys. LabelMakor: Automatic Semantic Label Generation from RGB-D Trajectories. In International Conference on 3D Vision (3DV), 2024.
[69] Tianhan Wei, Xiang Li. Yau Pun Chen, Yu- Wing Tai, and Chi- Keung Tang. FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation. Conference Computer Vision and Pattern Recognition (CVPR). 2020.
[70] Daniel Wolf, Johann Prankl, and Markus Vincze. Fast Semantic Segmentation of 3D Point Clouds using a Dense CRF with Learned Parameters. In International Conference on Robotics and Automation (ICRA), 2015.
[71] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Bycon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. GroupViT: Semantic Segmentation Emerges from Text Supervision. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[72] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. ODISE: Open- Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models. In CVPR, 2023.
14

---

[73] Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters. In European Conference Computer Vision (ECCV), 2018.
[74] Bo Yang. Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, and Niki Trigoni. Learning Object Bounding Boxes for 3D Instance Segmentation Point Clouds. In Neural Information Processing Systems (NeurlPSS. 2019.
[75] Jiahui Yu, Zirui Wang. Vijay Vasudevan, Legg Yeung. Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive Captioners are Image- Text Foundation Models. Transactions on Machine Learning Research, 2022.
[76] Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao. Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: new foundation model for computer vision. arXiv preprint arxiv::111.... 1.N::2111.1142... 2021.
[77] Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann. Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[78] Yuanwen Yue, Sabarinath Mahadevan, Jonas Schult, Francis Engelmann, Bastian Leibe, Konrad Schindler, and Theodora Kontogianni. AGILE3D: Attention Guided Interactive Multi-objeet 3D Segmentation. arXiv preprint arXiv:2306. 00977. 2023.
[79] Nir Zabari and Yedid Hoshen. Semantic Segmentation In-the-Win Wild Without Seeing Any Segmentation Examples. arXiv preprint arXiv:2112.031885 2021.
[80] Chong Zhou, Chen Change Loy, and Bo Dai. Extract Free Dense Labels from CLIP. In European Conference on Computer Vision (ECCV), 2022.
15

---

Appendix
The Appendix section is structured as follows: In Sec. A. provide further details about our OpenMask3D method, and we explain certain design choices. In Sec. B we present additional results, such ablation studies on hyperparameters and insights into the performance of OpenMask3D. In Sec. C, we provide further details about how the baseline experiments are performed. In Sec. D, we share additional qualitative results.
A OpenMask3D details
A.1
Class-agnostic mask proposal module
As OpenMask3D is an instance-centric approach, we first need obtain instance mask proposals. Since we are operating in an open- open-vocabulary setting, these instance masks are not associated with any class labels, they are class agnostic.
We use the mask modulc of the transformer-basee Mask3D [58] architecture as the basis for our class-agnostic mask proposal module. Specifically, for our experiments ScanNer200 [57], we use [58] trained on the training set of ScanNet200 [57] for instance segmentation. For our experiments on the Replica [61] dataset, we the [S8] module trained on the training and validation sets of the ScanNer200 [57] dataset, but without segments (see [58] for details on segments). We keep the weights of the mask proposal module frozen. Unlike Mask3D, our mask proposal module discards class predictions and mask confidence scores (which are based on class likelihoods), and only retain the binary instance mask proposals.
The architecture consists of sparse convolutional backbone based on the MinkowskiUNet [9], and a transformer decoder. Point features obtained from the feature backbone are passed through the transformer decoder, which iteratively refines the instance queries, and predicts an instance heatmap for each query. The query parameter specifies the desired number of mask proposals from the transformer- hased architecture. We set the numher of queries to 150, following the original implementation of Mask3D. This choice enables us to obtain sufficient. number of mask proposals for open-vocabulary setting. The output from the final mask-module in the transformer decoder a set binary instance masks.
The original implementation
the Mask3D [58] model first ranks the proposed instance masks based on their confidence scores, and retains the top lo masks based on this ranking. As the confidence scores are guided by class likelihoods, wc do not utilize such scores to rank the masks, and do not filter out any of the proposed instance masks. Furthermore, since aim retain as many mask proposals possible, we do not perform mask-filterngg based on object sizes, or pairwise mask overlaps. In other words, keep all masks proposed by the mask module as each these masks might correspond to potential open-vocabulary query.
As the model may occasionally output mask proposals that are not spatially contiguous, original imple- mentation of Mask3D employs the DBSCAN clustering algorithm [17] to break down such non-contiguous masks into smaller, spatially contiguous clusters. We follow this practice, and perform DBSCAN clustering on the instance masks, setting the epsilon parameter, 0.95. This procedure often increases the final number of instance masks. As such, our OpenMask3D model generates class -ansotig instance masks, for which we compute open-vocabulary mask-featured as descrihed next.
A.2 Mask-feature computation module details
In the main paper, we outlined the steps to extract per-mask features in the CLIP [55] space. In this section, we will delve into the motivations behind specific design choices made throughout the process.
A.2.1 Per-mask view ranking: assumptions and visualizations
Once we obtain the class-agnostic masks, our primary objective is determine the best CLIP features for each of them. We operate under the assumption that CLIP is capable of extracting meaningful features from instances when provided with favorable viewpoint, and in our fcature computation process, we prioritize selecting views that offer higher number of visible points. In practice, to assess the quality of particular viewpoint for each mask, we compute the proportion of points of the 3D instance mask that are visible from that view. Next, we rank the views based on their visibility scores, and we select the top & views with the highest scores, where & a hyper parameter. Fig. 6 illustrates this concept, where two views of given instance are compared based on the number of visible points. The view that has higher number of visible points from the instance is selected (outlined in green).
16

---

1428 visible points
591 visible points
Figure Visual comparison between selected view (marked green outline) for the corresponding chair instance, and a discarded view. The green points the images represent the 3D instance points projected onto the 2D image. In the selected view the instance is fully visible, and mostly occluded in the discarded view.
A.2.2 Why do we need SAM?
In order get the CLIP image features of a particular instance in the selected views, we need crop the image section containing the object instance. This process involves obtaining initial tight crop of the instance projected 2D, and then incrementally expanding the crop to include some visual context around
straightforward approach perform the cropping would project of the visible 3D points belonging the mask onto the 2D image, and fit 2D bounding box around these points. However, as we do not discard any mask proposals, the masks can potentially include outliers, or the masks might be too small as an outcome of the DBSCAN algorithm described in Sec. A. As depicted in Fig. 7, this can result in bounding boxes that small. Using CLIP-image features obtained from crops based solely projected points can lead to inferior instance mask-features.
either large
address this challenge, using a 2D segmentation method that takes a set of points from the mask as input, and produces the corresponding 2D mask, together with its confidence score. The Segment Anything Model (SAM) [36] precisely fulfills this task, allowing generate accurate 2D masks in class- dans-aggnottic manner.
improve robustness our model against noisy instance masks, we propose
Figure 7: Difference between the hounding boxes obtained by tightly cropping around the projected points from the 3D instance mask (left), and the bounding box obtained from the 2D mask generated by SAM (right).
A.2.3 Which points should give as input to SAM?
When using SAM predicting 2D mask, need determine set points on which the mask generation process will conditioned. Initially, we attempted to input all visible points projected from the mask. However, this approach results in poor quality masks due to the inaccuracies of the projections and the noise present in the 3D masks, as illustrated in Figure 8.
17

---

SAM score 0.694
SAM
Figure 8: Output of SAM, using all of the visible points from the projected 3D mask input.
To address this issue, explore alternative approach. Instead of using all of the visible points input for SAM, we randomly sample subset of points from the projected 3D mask. Interestingly, this method produces much cleaner masks, as it can be seen in Fig. 9.
SAM score 0.978
SAM
Figure 9; Output of SAM, using only 5 randomly sampled points (visualized green dots) of the projected 3D mask input.
A.2.4 Why do we need run SAM for multiple rounds?
Relying solely small set of random points as input for SAM might be unreliable, particularly when the sampled points are outliers, or too concentrated a particular area of the instance we want to segment.
To address this limitation, we implemented a sampling algorithm (as outlined in the main paper) inspired by RANSAC [18]. In this approach, we perform krownd 10 sampling rounds, and in each round we randomly sample ksampte 5 points from the projected 3D instance mask. SAM returns the predicted 2D mask along with the corresponding mask confidence returned by SAM, select the mask with the highest score. Fig. 10 illustrates an example where the points sampled in one round are concentrated a small spatial range, resulting in an incorrect mask prediction by SAM.
for each set of sampled points. Based the confidence scores
SAM score 0.949
SAM
Figure 10: Output of SAM, using only 5 randomly sampled points of the mask input. Here the sampled points (the green points visualized in the image) are concentrated in small spatial range and cause SAM to predict incorrect mask, which does not include the legs of the chair.
However, by employing our sampling algorithm and rumning SAM for multiple rounds. we achieve improved results. The iterative process allows select the mask with the highest score achieved among the rounds, shown in Fig. 1I. In this particular example, the selected mask, i.e. the one with the highest SAM confidence score, accurately segments the chair.
18

---

SAM score 0.985
SAM
Figure 11: Illustration of the results from the round which gives the highest confidence On the left, visualize the 5 sampled points which are given input SAM. the right, visualize the SAM mask prediction, showing a perfect segmentation of the chair.
B
Additional results
B.1
Ablation Studies
In addition to our ablation experiments provided in the main paper, here share additional results. First, in Table we share results obtained from using varying number of views (k) the top-k view selection process. This analysis is conducted
ScanNet200 dataset. Additionally, we provide ablation study evaluating hyperparameters of the multi-scale cropping in our method. This analysis is conducted on the Replica dataset, and is presented in Table 7.
Top-k
AP
AP50
2P2s
head(AP)
common(AP)
tail(AP)
13.6
17.6
20.8
15.5
12.5
12.8
15.4
19.9
23.1
15.4
20.0
23.2
16.4
14.0
15.8
Table 6: Ablation study of the top-k frame selection parameter k. This analysis conducted on the ScanNer200 validation set.
Levels
Ratio of Exp.
AP
AP50
AP2s
1
0.1
11.3
16.0
20.2
3
0.1
13.1
18.4
24.2
0.05
12.9
18.1
23.5
0.1
13.1
18.4
24.2
3
0.2
12.8
17.7
22.9
Table 7: Ablation study the multi-scale cropping hyperparameters on the Replica dataset. We analyze the effect of varying number of levels, and the ratio of expansion.
B.2 Evaluation on Replica without RGB-D images
Our approach OpenMask3D requires images as input, as depends on visual-language models that operate on images in combination with text. In this work, we prioritized the ability to recognize muccommon/long-eee objects over generalization across different modalities. Using vision-language models images provides an when only the 3D of
excellent opportunity to preserve this generalization capability. Nevertheless, scene available, could still possible render images from the scan, and use those synthetic images as input to our OpenMask3D pipeline. We tried this on the Replica [61] dataset, and rendered RGB-D images from the dense scene point clouds (as illustrated in Fig. 12). Our results using these rendered synthetic views of the scene point cloud are presented in Tab. 8. Even without any RGB-D images relatively well, and observe that the performance decreases by only -1.5 AP. Furthermore, OpenMask3D shows stronger performance compared to the 3D baseline (OpenScene [52]. Distill).
input, OpenMask3D performs
C Details on baseline experiments
Quantitative baseline experiments
As mentioned in the main paper. to compare our OpenMask3D method with the 3D open- vocabuuay scene understanding approach OpenScene [52], adapted OpenScene for the 3D instance segmentation task. As
19

---

Model
AP
APso
AP25
OpenScene [52] (3D Distill) masks
8.2
10.5
12.6
OpenMask3D w. rendered RGB-D images
11.6
14.9
18.4
OpenMask3D
13.1
18.4
24.2
Table 8: 3D instance segmentation results on the Replica dataset. Comparing OpenMask3D performance when using original RGB-D images vs. RGB-D images rendered from the point cloud.
(a) RGB image from the Replica dataset
(b) RGB image rendered from the point cloud
(c) RGB image from the Replica datasct
(d) RGB image rendered from the point cloud
Figure 12: Visualization the Replica RGB images. Original RGB images from the Replica dataset (left),
and RGB images rendered from the scene point cloud (right).
OpenScene computes a per por-point feature representation for each point in the point cloud, we use our proposed class-agnostic instance masks to aggregate OpenScene features for each mask. This way. we can associate each object instance mask with "per-mask" feature computed from OpenScene features.
OpenScene provides 3 different model variants, namely 2D fusion, 3D distill, and 2D/3D ensemble. We primarily compare with the OpenScene models using OpenSeg [20] features, whereas also experiment with the 2D fusion model using L.Seg [43] features. The main reason why we primarily compare against models using OpenSeg features is that the features have a dimensionality of 768, relying on the same CLIP architecture we employ in our work ViT-L/14 (336px) [55] -, whereas the LSeg features have dimensionality of 512. For our closed-vocabee vocabullyy experiments with the OpenScene model using LSeg features, embed each text category using ViT-B/32 CLIP text encoder, and assign object categories using these embeddings. In other experiments, we embed text queries using the CLIP architecture ViT-L/14 (336px).
Qualitative baseline experiments
In addition the guantitative results presented in the main paper, conducted qualitative comparisons further evaluate our mask-based open-world representation in comparison the point-based representation provided by OpenScene [52].
Fig. 13 and Fig. 14, we provide these qualitative comparisons. Fig. 13, we show results from the OpenScenc [52] 2D fusion model using OpenSeg [20] features, and from OpenMask3D. In Fig. 14, we compare features from OpenMask33 with features from several OpenScene variants (2D fusion, 3D distill, 2D/3D ensemble). To visualize results from OpenScene, for given query, compute the similarity
between each point-feature the query text embedding, visualize the similarity scores each point a heatmap over the points. On the other hand, OpenMask3D results are based on mask-feature features, i.e., each mask has an associated feature vector, for which we compute similarity score. Hence, we visualize the similarity scores for each mask as heatmap over the instance masks, and each point in given instance mask is assigned the same similarity score.
20

---

Query: "a dollhouse"
OpenScenc [52]
OpenMask3D (Ours)
Query: "a desk with backpack on it"
OpenScene [52]
OpenMask3D (Ours)
Query: "Cetaphil"
OpenScene [52]
OpenMask3D (Ours)
Figure 13: Qualitative comparisons. Heatmaps illustrating the similarity hetween the CLIP embeddings specific query and the open- gpen-vocabulary representation of the scene. comparison is made between point-based OpenScene approach proposed by Peng et al. [52] (left) and our mask based approach OpenMask3D (right). Dark red means high similarity, and dark blue means low similarity with the query.
As evident from Fig. 13 and Fig. 14, while similarity computation between the query embedding and each OpenScene per-point feature vector results in reasonable heatmap, it is challenging extract object instance information from the heatmap representation. Our method OpenMask3D, on other hand, computes similarity between the query embedding and each per-mask feature vector. This results crisp instance boundaries, particularly suitable for the use cases in which one needs to handle object instances.
D Qualitative results
In this section, we provide additional qualitative results.
In Fig. 15, we show results from object categories that are not present in the ScanNer200 label set. Objects from these novel categories are successfully segmented using our OpenMask3D approach. Thanks to its zero-shot learning capabilities, OpenMask3D able segment given query object that might segmentation datascts on which dlsed-vvccaulle instance segmentation approaches are trained.
present common
Fig. 16, we show open-vocabular 3D instance segmentation results using queries describing various object properties such as affordances, color, geometry, material type and object state. This highlights that our model able to preserve information about such object properties and go beyond object semantics, contrary to the capabilities of existing vloseu-vocabulary 3D instance segmentation approaches.
21

---

(a) Ours (per-mask)
(b) OpenScene (2D Fusion, per-point)
(c) OpenScene (3D Distill, per-point)
(d) OpenScene (2D/3D Ensemble, per-point)
Figure 14: Feature similarity for query "the table with flower vase on it".
22

---

"dollhouse"
"Cetaphil"
"Roomba"
flopsflos""
gym bench"
"rice cooker" "
Figure 15: Qualitative results from OpenMask3D. We show open-vocabulary instance segmentation results using arbitrary queries involving object categories that are not present in the ScanNet200 dataset labels. In each scene, visualize the instance with the highest similarity for the given query embedding. These predictions show zero-shot learning ability of our model, highlighting the open- vocabulary capabilities.
23

---

Affordance
"sit"
"wash
Color
"an orange pillow"
"a yellow pillow"
Geometry
"a circular table" "
rectangular table" "
Material
"a leather chuir"
"a fabric chair"
State
"an empty table"
"a side table with a flower vase on it"
State
"a full garbage bin"
"an empty garbage bin"
Figure 16: Qualitative results from OpenMask3D, queries related to object properties. We show open- vocabulary 3D instance segmentation results using queries describing various object properties such af- fordances, color, geometry, material type, and object state. In each row, we show two different queries per category, for which our method successfully segments different object instances. These results highlight the strong open- opea-vocabulary 3D instance segmentation capabilities of our model, going beyond object semanties.
24


Reviews:
[{"review_id": "5y1g80A1jz", "reviewer": "Reviewer_X8Ds", "summary": "The paper proposes to solve open-vocabulary 3D instance segmentation. It uses a class-agnostic 3D instance segmentation model to obtain instance masks, then gather multi-scale image features from multiple frames by CLIP and SAM to do the open-vocabulary classification task.", "questions": "1. What is the latency and inference cost of the model during inference?\n2. How $k_{rounds}$ is chosen? Is there any ablation study about that? Furthermore, does it mean that SAM needs to inference $k_{rounds}\\times number of frames$, and CLIP need inference $k_{rounds}\\times number of frames \\times number of scales$\n3. How the confidence score $s_r$ is obtained?", "limitations": "See weakness", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes to use instance-level features for 3D open-vocabulary instance segmentation, which is not attempted by previous works\n2. The proposed module exhibits reasonable improvements, as shown in Tables 2&3.", "weaknesses": "1. The design of the framework may be complicated for real-world usages. It uses SAM to do segmentation for multiple frames, and then use multi-scale images for CLIP to inference. Each component like SAM and CLIP is a large foundation model and takes a while to inference, not mention that they are used multiple times.\n2. The idea is not so novel. Although such idea of using instance mask features is not attempted in 3D instance segmentation, it has been widely adopted in 2D open-vocabulary segmentation tasks, like OpenSeg[1], ODISE[2], ZegFormer[3].\n3. The experiments are not thorough. For example, the details of using SAM and RANSAC are not studied.\n\n\n[1] Scaling Open-Vocabulary Image Segmentation with Image-Level Labels, ECCV2022\n[2] ODISE: Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models, CVPR2023.\n[3] Decoupling Zero-Shot Semantic Segmentation.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback.\n\n### Q1: What is the latency and inference cost of the model during inference?\n\nIn the table below, we provide the runtime of our approach on ScanNet for generating the queryable 3D scene representation, and also for performing a query on that representation. For comparison, we additionally provide the runtimes of the original OpenScene [46]. All runtimes are measured on the same hardware.\n| Method | Generating 3D Scene Representation [s] | Querying 3D Scene Representation [ms] | Semantics? | Instances? |\n| --- | --- | --- | --- | --- |\n| OpenScene (2D Fusion) |  440 | 168.8 | $\\checkmark$| ✘  |\n| OpenScene (2D/3D Ensemble) |  524 | 168.8 | $\\checkmark$| ✘  |\n| OpenMask3D |  556 | **0.923** |  $\\checkmark$| $\\checkmark$ |\n| OpenMask3D (fast) |  **350** | **0.923** |  $\\checkmark$| $\\checkmark$ |\n\nWe also report the runtime of a faster variation of our model (called \"fast\") which runs CLIP only once per segment on a single crop and uses a smaller SAM backbone. The exact hyper-parameters are shown in the table below. This faster version performs almost as good as the original model (small performance drop of -1.3 AP$_{50}$, full table is available as Tab. 2 of the rebuttal PDF).\n\n| Model | top-k | $\\mathsf{L}$ | SAM backbone | \n| --- | --- | --- | --- | \n| OpenMask3D |  5 | 3 |  ViT-H| \n| OpenMask3D (fast) |  1 | 1 |  ViT-B| \n\n### Q2.a: Value of $k\\_{rounds}$\n\nThe reviewer asks about how $k\\_{rounds}$ (number of SAM runs in RANSAC) is selected and if there is an analysis study.\nWe tried increasing values for $k\\_{rounds}$ but saw only a marginal increase in performance, e.g, from 16.6 AP50 (k=1) to 16.8 AP50 (k=10). \nInterestingly, increasing the number of SAM runs has little impact on the runtime performance.\nIndeed, the table below shows (for SAM with different backbones), that the high computational cost of SAM comes when the image is set (which is done only once) and only little overhead is added when re-running the prediction based on a new set of ground truth points. Thus, performing multiple SAM predictions on the same image has no large impact on the runtime, but makes our model more robust.\n\n| Function | Backbone | time [s] |\n| --- | --- | --- | \n| SAM.set_image() |  ViT-H | 0.497 |  \n| SAM.predict() |   ViT-H  | 0.006 | \n| SAM.set_image() |   ViT-B | 0.109 |  \n| SAM.predict() |   ViT-B | 0.005 |  \n\n### Q2.b: How often is SAM and CLIP called?\n\nSince we only use the top k_view views for each mask, not all frames are utilized. \nTherefore, the number of iterations for SAM is $\\mathsf{k_{rounds}\\cdot \\mathbf{M \\cdot k_{view}}}$ and for CLIP, it is $\\mathsf{\\mathbf{M \\cdot k_{view}}\\cdot L}$, where $\\mathsf{M}$ is the number of 3D masks predicted by the class agnostic mask predictor and $\\mathsf{L}$ is the number of levels used for the multi-level crop. Overall this approach is comparable in runtime to OpenScene which needs to run OpenSeg over the full frame (which is slow), whereas here we run the much faster CLIP on small crops.\n\n### Q3: How is the confidence score $s_r$ obtained?\n\nThe confidence score $s_r$ is an output of the SAM model. For each 2D mask, SAM also predicts a confidence score $s_r$ (see main paper L201, L204-207). We will describe this more clearly in the paper. In the supplementary material (Sec, 1.2.), we provide further explanations on how we utilize SAM (L52-L92), and visualize 2D mask proposals along with confidence scores returned by SAM in Fig. 3 to Fig. 6.\n"}, {"review_id": "MddAo6lKpQ", "reviewer": "Reviewer_zuiw", "summary": "This paper addresses the problem of text-based 3D instance segmentation. To tackle this problem, the authors propose OpenMask3D, a zero-shot approach for 3D instance segmentation that utilizes class-agnostic 3D instance masks and multi-view fusion of CLIP-based image embeddings to aggregate per-mask features. The model's performance is evaluated through experiments and ablation studies on the ScanNet200 dataset, where it outperforms OpenScene in some metrics.\n", "questions": "\n- Please clearly outline the differences compared to OpenScene\n- The examples given in the paper are mainly for common objects \"armchair\", \"seat\", \"pool\", \"sofa\". Can you give an example where the method identifies objects that are truly from an \"open\" vocabulary (i.e. not common)?", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n**Clarity and quality**: The paper is well written and explains all the components with clarity and in detail. The figures are excellent. The results are very neatly presented and overall the paper is engaging to read.\n\n**Empirical Evaluation**: The paper includes extensive empirical evaluations on the ScanNet200 dataset. The paper not only demonstrates the performance of OpenMask3D but also presents ablation studies on the number of frames used, the use of 2d masks and multi-scale crops to understand the contribution of these components.\n\n**Relevance**: The work addresses a highly relevant challenge in computer vision and autonomous systems. As these technologies become increasingly prevalent, the ability to understand and interact with a diverse range of objects becomes critically important, underscoring the relevance of this research.", "weaknesses": "\n**Novelty of the task**: The paper makes strong claims about being the first to introduce \"open-vocabulary 3D instance segmentation\". However, given OpenScene and others I don't think it can be claimed that the problem of text-based 3D instance segmentation is novel. OpenScene might not have used the word \"instance\" but it does show results for \"Open-vocabulary 3D object search\" which is similar in meaning. \n\n**Significance of the Contribution**:  It is true that most existing segmentation approaches have relied on a closed-set of objects in 3D annotated datasets and thus a system that can perform open-vocabulary segmentation would be quite a significant contribution to the field of 3D scene understanding. However, the technical contribution of the proposed approach is somewhat limited considering its similarity to OpenScene. The main differences seem to be the frame selection and feature aggregation strategy, but if this is the case I would expect these components to be more prominently handled in the paper (rather than claiming a whole novel task).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing valuable feedback, and we are really glad that the reviewer found our paper enjoyable to read! We address the questions and concerns in the following:\n\n### Q1: Differences between OpenMask3D (instance segmentation) and OpenScene (semantic segmentation)\n\nThe reviewer asks about the differences between OpenScene and this work, in particular the reviewer suggests that OpenScene already solves the task of \"open-vocabulary 3D instance segmentation\". \nWe disagree: OpenScene mainly addresses *semantic* segmentation (i.e., it predicts per-point features), while OpenMask3D addresses *instance* segmentation (i.e., it predicts a set of object masks and associated features). Both tasks are fundamental computer vision problems, that are well defined and conceptually different. \"Open-vocabulary 3D object search\" in OpenScene primarily identifies a single point in the scene that matches the query the best. \"Image-based 3D object detection\" in OpenScene identifies the set of points that have close similarity to the given image-based query (e.g. a chair image), however it cannot differentiate between multiple instances of the same object class (e.g., it cannot predict a segmentation mask for Chair 1 and a different mask for Chair 2).\n\nIn the light of this, we would like to clearly outline the main conceptual differences between OpenMask3D and OpenScene. OpenMask3D is *mask-based*, while OpenScene is *point-based*. Specifically, OpenScene outputs a per-point feature representation (num_points, feature_dim). OpenMask3D outputs a set of binary instance masks (num_masks, num_points), and corresponding per-mask features (num_masks, feature_dim). We would like to draw the reviewer’s attention to Fig. 1.a and Fig. 1.c in the rebuttal PDF, and Fig. 7 in the supplementary material, which we believe are helpful for visualizing our following statements. \n\nBoth OpenMask3D and OpenScene output task-agnostic open-vocabulary features, but our method is tailored towards identifying object *instances*. While OpenScene returns heatmaps over the scene points describing similarity to the query, it cannot differentiate between two object instances that belong to the same query. In practice, it can either retrieve a set of points from the point cloud by using a given similarity score threshold (Sec. 5 of OpenScene paper, “Image-based 3D object detection”) or can identify a single point that best matches the query (Sec. 5 of OpenScene paper, “Open-vocabulary 3D object search”). In contrast, OpenMask3D is able to segment the *object instances* that suit the given open-vocabulary query, automatically separating the multiple instances from each other.\n\nSuppose now we want to find top-k objects which match an open-vocabulary query. Although, as the reviewer correctly underlined, OpenScene is able to perform this task for $k=1$ (Sec, 5 of OpenScene paper, \"Open-vocabulary 3D object search\"), it cannot retrieve multiple object matches when $k>1$, because it cannot distinguish whether a point belongs to a specific object or to another one.\n\nSuppose finally we want to find how many objects are present in a room such that the similarity with a given query is higher than a given threshold. In this case OpenScene has no direct way to provide such an answer as it can only return a per-point features/labels.\n\nIn brief, OpenScene and many recent 3D open-vocabulary approaches need consequent steps to identify and separate object instances, which poses many practical limitations. As the reviewer also highlighted, “the ability to understand and interact with a diverse range of objects becomes critically important” - particularly when coupled with the necessity to densely segment separate objects. The novelty in our work lies in its ability to directly identify object instances in an open-vocabulary setting, and its design that is tailored to efficiently focus on *instances*.\n\n**OpenScene uses OpenSeg, OpenMask3D uses CLIP**\n\nThis is another fundamental difference between both methods: to obtain pixel-aligned CLIP features, OpenScene relies on OpenSeg which is a fine-tuned version based on CLIP. However, during fine-tuning, it becomes less \"general\" than the original CLIP (used by OpenMask3D). This effect is also shown by the results in Table 1 of the main paper, particularly in long-tail categories.\n\n### Q2: Examples from an “open” vocabulary\n\nThe reviewer asks whether we could share examples where the method identifies objects that are truly from an “open” vocabulary. Our method is indeed capable of segmenting uncommon object instances, and we would like to draw attention to several figures from our submission. In Fig. 1 (main paper), we share examples using the queries “footrest” and “angel”. In Fig. 4 of the main paper, we share additional “uncommon” object queries, such as “pepsi”. We also would like to highlight sup. mat. Fig. 8, in which *all* examples are uncommon categories. This figure highlights examples such as “Cetaphil” (a soap brand), “Roomba” (a robot vacuum cleaner brand) and “dollhouse”. We apologize if we have not referenced the figures sufficiently within the text to highlight particularly novel object queries."}, {"review_id": "G1PlzgBICa", "reviewer": "Reviewer_cqVz", "summary": "This paper presents a method for 3D open-vocabulary instance segmentation. It proposes to use a class-agnostic Mask3D to get some instance mask proposals, project the instance points to 2D views to get some 2D segments, and extract some instance features based on the 2D segments by CLIP image encoder. Then, text descriptions can be used as queries to find the best instance proposals in an open-vocabulary way by comparing the similarities with the instance mask features. ", "questions": "Please refer to the weaknesses. More questions are listed below:\n1. More details of training a class-agnostic Mask3D are expected, for example, the scores used in the Hungarian matching process and the losses used in training. \n2. The paper will be better if there is a discussion of the domain gap between the cropped image and the images used to train CLIP. Also, considering the domain gap, a detailed ablation of the hyperparameters in multi-scale cropping is expected.\n3. The equations in L212 seem not accurate.\n \n", "limitations": "Some of the limitations are discussed in the last paragraph of the paper. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well organized with good figures and easy to understand. \n2. The paper is the first method for open-vocabulary instance segmentation in 3D scenes. \n3. The experiments and analysis in the paper show the great zero-shot ability of the method. ", "weaknesses": "1. The method relies on RGB-D images, and thus cannot generalize to 3D scenes (e.g., 3D scenes collected by LiDAR) without 2D images. \n2. The experiments are conducted only on the ScanNet dataset. Experiments on more datasets like the Matterport3D used in OpenScene are expected to prove the generalizability of the method. \n3. The inference speed of the whole framework seems slow as the method uses some heavy models (e.g., SAM and CLIP) for multiple views.  SAM model is even used multiple iterations for each mask. A comparison of the inference time between this method and OpenScene is expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. We are happy that the reviewer found our paper well-organized and easy to understand, and appreciated the zero-shot ability of our open-vocabulary 3D instance segmentation approach. Below, we hope to answer the questions raised by the reviewer.\n\n### W1/W2: Generalization to other datasets & 3D scenes without 2D images\n\nTo assess the generalization ability of our method to other datasets, we share additional results on the Replica dataset. For the OpenScene baselines, we use the pre-computed features from the OpenScene repository. We provide qualitative examples of open-vocabulary queries from our method in Fig. 1.d of the rebuttal PDF. Our approach outperforms other open-vocabulary counterparts on the Replica dataset as shown below:\n\n|Method|AP|AP$_{50}$|AP$_{25}$|\n|---|---|---|---|\n|OpenScene (2D Fusion)|10.9|15.6|17.3|\n|OpenScene (3D Distill) |8.2|10.5|12.6|\n|OpenScene (2D/3D Ens.)|8.2|10.4|13.3|\n| OpenMask3D |**13.1**|**18.4**|**24.2**|\n\nFurthermore, the reviewer highlights that our approach requires images as input. This is correct, and it is due to the fact that OpenMask3D, like OpenScene, depends on visual-language models that operate on images in combination with text. In this work, we prioritized the ability to recognize uncommon/long-tail objects over generalization across different modalities. Using vision-language models on images provides an excellent opportunity to preserve this generalization capability. Nevertheless, we would like to state that when only a 3D scan of a scene is available, it could still be possible to render images from the 3D scan. We tried this on Replica dataset, and rendered RGB-D images from the scene point clouds, which is illustrated in Fig. 1.b of the rebuttal PDF. With this approach, we obtained the scores shown in the table below, marked \"rendered RGB-D\".\n\n|Method|AP|AP$_{50}$|AP$_{25}$|\n|---|---|---|---|\n|OpenMask3D (rendered RGB-D) |11.6|14.9|18.4|\n| OpenMask3D |**13.1**|**18.4**|**24.2**|\n\nOverall the performance does decrease when using images rendered from the point cloud, but only by -1.5 AP. Yet, we agree that when no color images are available or when the scan is sparse (LiDAR) it might not be possible to easily render color images.\n\n\n### W3: Runtime\n\nIn the table below, we provide the runtime of our approach on ScanNet for generating the queryable 3D scene representation, and also for performing a query on that representation. For comparison, we additionally provide the runtimes of the original OpenScene [46]. All runtimes are measured on the same hardware.\n| Method | Generating 3D Scene Representation [s] | Querying 3D Scene Representation [ms] | Semantics? | Instances? |\n| --- | --- | --- | --- | --- |\n| OpenScene (2D Fusion) |  440 | 168.8 | $\\checkmark$| ✘  |\n| OpenScene (2D3D Ensemble) |  524 | 168.8 | $\\checkmark$| ✘  |\n| OpenMask3D |  556 | **0.923** |  $\\checkmark$| $\\checkmark$ |\n| OpenMask3D (fast) |  **350** | **0.923** |  $\\checkmark$| $\\checkmark$ |\n\nWe also report the runtime of a faster variation of our model (called ‘fast’) which runs CLIP only once per segment on a single crop and uses a smaller SAM backbone. The exact hyper-parameters are shown in the table below. This faster version performs almost as good as the original model (small performance drop of -1.2 AP).\n\n| Model | top k | $\\mathsf{L}$ | SAM backbone | \n| --- | --- | --- | --- | \n| OpenMask3D |  5 | 3 |  ViT-H| \n| OpenMask3D (fast) |  1 | 1 |  ViT-B| \n\n|Method|AP|AP$_{50}$|AP$_{25}$|\n|---|---|---|---|\n|OpenMask3D (fast) |11.9|17.1|23.3|\n| OpenMask3D |**13.1**|**18.4**|**24.2**|\n\n### Q1: Class-Agnostic Mask3D training details\nClass-Agnostic Mask3D closely follows Mask3D but ignores the semantics - main difference is how we discard class label and class confidence-based filtering stage. In the supplementary material Sec. 1.1 we provide a detailed explanation. We will extend the description in the main text to include further implementation details.\n\n### Q2: Discussion on the domain gap & ablation on multi-scale cropping parameters\nWe agree with the reviewer, and provide an ablation study regarding the hyperparameters used during the multi-scale cropping phase on the Replica dataset. In the table below, we ablate results both based on number of levels, and ratio of expansion between levels:\n\n| Levels | Ratio of Expansion|AP | AP50| AP25|\n| --- | --- | --- |--- |--- |\n| 1 | 0.1 |11.3 |16.0|20.2|\n| 3|  0.1 | **13.1**|**18.4**|**24.2**|\n| 5| 0.1 |12.8|17.6|22-6|\n| 3|  0.05| 12.9 |18.1 |23.5 |\n| 3|  0.1 |**13.1**|**18.4**|**24.2**|\n| 3|  0.2| 12.8 | 17.7 | 22.9 |\n\nRegarding the domain gap in w.r.t. CLIP, we also would like to highlight our \"rendered RGB-D\" experiment on Replica discussed earlier, which we believe highlights the robustness of our CLIP-based mask-feature module to image-quality and realism.\n\n\n### Q3: Equations in L212\nThe reviewer noticed a problem with the equations, we updated them as follows:\n\n- $x_1^l = \\max(0, x_1^1-(x_2^1-x_1^1)\\cdot k_{exp} \\cdot l)$\n- $y_1^l = \\max(0, y_1^1-(y_2^1-y_1^1)\\cdot k_{exp}\\cdot l)$\n- $x_2^l = \\min(x_2^1+(x_2^1-x_1^1)\\cdot k_{exp}\\cdot l, W-1)$\n- $y_2^l = \\min(y_2^1+(y_2^1-y_1^1)\\cdot k_{exp}\\cdot l, H-1)$\n"}, {"review_id": "xLK3JdPBdm", "reviewer": "Reviewer_7yua", "summary": "The paper proposes to perform open-vocabulary instance segmentation by utilizing class-agnostic 3D instance segmentation masks from a 3D instance segmentation model trained on scannet200 and generating class labels for it using CLIP. The paper proposed to first obtain class-agnostic instance masks from a supervised Mask3D model (without using class annotations). Then it finds the images where the objects are best visible. It then projects the 3D segmentation masks in those views and refine them using SAM. Finally they average features for each object from multiple views and at multiple scale to arrive at one feature vector per 3D instance mask. The class label can then be obtained by doing dot-product with the obtained feature vector and the language embedding of the class. The results show that the proposed method is superior than prior point-based methods when also supplied with class-agnostic instance segmentation mask in closed-vocabulary setting.  The paper also shows some qualitative results with free-flowing natural language.", "questions": "- As mentioned in the limitations, here are some suggestions which might help alleviate the concerns over instance-segmentation annotation bottleneck:\n    -  Does class-agnostic segmentation generalize beyond training categories? Performance on OOD dataset like Matterport3D and maybe instance segmentation model trained on scannet 18 classes and tested on scannet 200 class could help (compared against point-based methods). Another suggestion could be to hold out rare categories from Scannet200 for training class-agnostic instance segmentation model and evaluate on the held out categories at test time. Ofcourse, these are just suggestions and any other experiment that could help us get to the bottom of this will be highly appreciated.\n\n- Answers to \"unfounded claims\" as described in limitations would be super helpful too. \n\n- Could you give some insights on why is the performance of OpenScene 2D Fusion model so much better than 2D-3D ensemble models while the Openscene paper consistently showed better results with the ensemble version of their model? That was a bit strange to me.", "limitations": "The paper do not discuss limitations. I think the biggest one is their reliance on class-agnostic instance segmentation mask annotations which would be good to discuss in the ppaer.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written, I especially appreciate the helpful supplementary video and text content which made the nitty gritty details of the pipeline very clear. \n- Open-Vocabulary 3D instance segmentation is a very useful task which hasn't been tackled before -- this paper brings attention to it (that said, I have some concerns here as mentioned in weaknesses)\n- The proposed method obtains better results than prior point-based methods like OpenScene", "weaknesses": "- Open-Vocabulary: The proposed method relies on 3D instance segmentation predictions which in turn relies on 3D segmentation annotations. This, however, is not available for wide variety of objects, hence I am unsure if we can conclude that this model is indeed “open-vocabulary”. For example: If instead of scannet200, the proposed model uses class-agnostic masks from a model trained on 20 scannet classes, would it be able to achieve decent results on Scannet200? Would this model trained on scannet work on a different dataset like MatterPort3D? Additionally, since 3D datasets are significantly smaller than their 2D counterparts, doesn’t relying on 3D instance segmentation mask a serious bottleneck which wouldn’t scale? The point-based models are open-vocabulary in the sense that they are not bottlenecked by any 3D-specific annotations — at the same time I do agree with the point of this paper that they can only do semantic and not instance segmentation. However, using instance masks via 3D annotations might not as well lead to “open-vocabulary instance segmentation” proposed in this paper. Ideally, it should primarily have results on held out 3D categories that the model or any of its components have never seen during training.  \n- In continuation of the above, the results for 3D instance segmentation in open-vocabulary setting is only qualitative and not quantitative. I understand though that prior methods too show only qualitative open-vocabulary results, and so this is said as a minor point and not a major complaint. However, the lack of quantitative results on categories outside the training data is concerning — especially since this proposed model particularly used labels from scannet which may not generalize beyond the 200 categories they were trained on, and on out-of-domain 3D scenes. \n- Unfounded Claims: \n    - L304: The paper highlights that when given access to oracle masks at “test time” to their model, it outperforms the supervised Mask3D model on tail AP by 9.1%. As the paper concludes in L305-308, this result indicates that if somehow we are able to obtain high quality class-agnostic masks, we do not need supervision for class labelling as their method can outperform supervised Mask3D. In my opinion, this is a misleading claim because while their baseline “Mask3D” has access to ground truth masks and classes during training, it does not have access to oracle masks during test time. This makes the comparison unfair. AP is very sensitive to quality of the segmentation mask and if supplied oracle masks to Mask3D it may do much better. A very crude way to supply that would be — computing Hungarian matching between the predicted masks and ground truth masks (without class labels), and for each predicted mask replace it with the matched ground truth mask while keeping the class label same. In general though, this comparison of supervised mask3d and the proposed method needs much more care to make balanced conclusions.\n    - (Minor) L290-291: Claims that “the ablation study show that effect of 2D mask segmentation is less significant than the effect of multi-scale cropping”. Based on this, one might expect the row 2 of component analysis section to be (significantly) better than row 3. However, on some metrics row 2 wins while on others row 3 wins. And the difference is not that much.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the extensive feedback, we really appreciate the helpful suggestions for experimental setups! \n\n### Open-vocabulary Evaluation & Generalization Beyond Training Categories\nThe reviewer correctly highlights that Mask3D is trained on closed-set segmentation masks dataset, and is concerned that the class-agnostic Mask3D would not generalize beyond the masks seen during training. This is indeed a valid concern, and deserves further analysis. However -- as our additional experiments below indicate -- in practice, the model still manages to generalize quite well beyond the object masks seen during training. To demonstrate the capability of our approach beyond the mask-predictor training categories, we conducted a series of experiments following the suggestions by the reviewer.\n\n**Generalization to unseen categories**\n\nFirst, we analyze how well our model would perform if we use class-agnostic masks from a mask-predictor trained on the 20 original ScanNet classes, and evaluate on the ScanNet200 dataset. To evaluate how well our model performs on “unseen” categories, we classify the ScanNet200 labels into two subsets: *base* and *novel* classes. We identify ScanNet200 categories that are semantically similar to the original ScanNet20 classes (e.g. chair and folded-chair, table and dining-table, etc.), resulting in 53 classes. We group all remaining object classes that are not similar to any class in ScanNet20 as \"novel\". Below, we report results on seen (“base”) classes, unseen (“novel”) classes, and all classes. The full table is provided in Tab. 1 of the rebuttal PDF. \n\n|Method|Novel (AP$_{50}$)|Base (AP$_{50}$)|All (AP$_{50}$)|\n|---|---|---|---|\n| OpenScene (2D Fusion) |10.3|**15.0**|11.6|\n| OpenScene (3D Distill) |2.3|13.4|5.3|\n| OpenScene (2D/3D Ens.) | 2.8| 13.7|5.8|\n| OpenMask3D (Ours) |**12.9**| **15.0**| **13.5**|\n\nOur experiments show that the model trained on a smaller set of object annotations from ScanNet20 can generalize to predict object masks for a significantly larger set of objects (ScanNet200), resulting in only a marginal decrease in the performance. Particularly, we see that OpenMask3D, compared to other open-vocabulary counterparts - seem to better preserve information about uncommon objects. \n\n**Generalization to OOD data**\n\nFurthermore, we show results on out-of-distribution data from *Replica*, using a mask predictor trained on ScanNet.\nFor the OpenScene baselines, we use the pre-computed features from the OpenScene repository.\nReplica dataset contains high-quality *mesh* reconstruction of indoor scenes, and RGB-D images rendered from these meshes.\nIn order to assess the robustness of our CLIP-based mask-feature module to image-quality and realism, we conduct a second experiment where we render RGB-D images from the *point clouds* of Replica scenes (marked “rendered RGB-D” below, illustrated in Fig. 1.b, rebuttal PDF).\n\n|Method|AP|AP$_{50}$|AP$_{25}$|\n|---|---|---|---|\n|OpenScene (2D Fusion)|10.9|15.6|17.3|\n|OpenScene (3D Distill) |8.2|10.5|12.6|\n|OpenScene (2D/3D Ens.)|8.2|10.4|13.3|\n|OpenMask3D (rendered RGB-D) |11.6|14.9|18.4|\n| OpenMask3D |**13.1**|**18.4**|**24.2**|\n\nOur results demonstrate that OpenMask3D can indeed generalize to unseen categories as well as OOD data.\nNevertheless, we understand and agree with the reviewer’s concern in general, however, the above experiments seem to indicate that this is less of a problem than one might initially think. In particular, the mask predictor module trained on a smaller set of objects seems to perform reasonably well in various settings. Furthermore, several qualitative examples we provide in our submission show that our method can achieve good “open-vocabulary” results for objects that were not originally annotated e.g. Fig.1 \"angel\", Fig.4 \"pepsi\" - and more in the supplementary.\n\n### Oracle Mask Experiment\n\nThe question is about an experiment from the paper (Tab. 3) showing that OpenMask3D when given access to oracle masks has the potential of outperforming supervised Mask3D model on tail categories by 9.1% AP. The reviewer raised a concern about the fairness of our experiment, stating that Mask3D does not have access to oracle masks in test time unlike the open-vocabulary approaches presented in Tab. 3. We understand the concern, and we regret our unintendedly strong wording regarding the claim in L304-308. To address this, we conduct the suggested experiment in which we supply oracle masks to Mask3D. We perform Hungarian matching between the predicted masks and oracle masks discarding all class-losses, and only match based on the masks. For each oracle mask, we assign the class label of the matched mask from Mask3D. Our results are below:\n\n|Method|AP|head (AP)|common (AP)|tail (AP)|\n|---|---|---|---|---|\n|Mask3D (Hungarian M., oracle)|35.5|55.2|27.2|22.2|\n|OpenMask3D (oracle)|23.4|24.6|19.3|27.0|\n\nEven when we supply Mask3D with oracle masks, our approach surpasses the Mask3D performance on the tail categories by $+4.8$ AP. While these findings in fact confirm our initial claim, we will revise our text to make more careful conclusions.\n\n### Performance OpenScene 2D Fusion vs. 2D/3D Ensemble\n\nThe reviewer points out that OpenScene 2D performs better than the 2D/3D ensemble variant, which is not in line with the findings in OpenScene [46]. We also noticed this and show visualizations in Fig. 1.c of the rebuttal PDF: since the ensemble is a point-wise operator it seems to add inconsistent noise which might be the reason for the reduced performance.\n\n### Limitations\n\nWe agree with the reviewer that our paper would benefit from further discussing limitations, and we will extend our section (L320-323) to include a discussion on class-agnostic mask quality as also discussed in (L294-308). Although our findings above demonstrate that the 3D annotation bottleneck might not be as severe as initially thought, we understand that it still plays an important role and will be added to the limitations."}, {"review_id": "Aso3TdrkXd", "reviewer": "Reviewer_STxk", "summary": "This paper focuses on the task of open-vocabulary 3D instance segmentation, which involves predicting 3D object instance masks and their corresponding categories. The authors highlight the limitations of traditional closed-vocabulary approaches that operate within a predefined set of object categories, which restricts their ability to handle novel objects and free-form queries.\n\nTo address these limitations, the authors propose OpenMask3D, a zero-shot approach for open-vocabulary 3D instance segmentation. OpenMask3D utilizes predicted class-agnostic 3D instance masks and performs mask-feature aggregation using CLIP-based image embeddings. This allows the model to reason beyond pre-defined concepts and handle open-vocabulary queries.\n\nThis paper demonstrates its superiority over closed-vocabulary counterparts. The proposed method has the potential to enhance various applications such as robotics, augmented reality, scene understanding, and 3D visual search.", "questions": "The paper primarily evaluates the proposed OpenMask3D method on the ScanNet200 dataset. Evaluation on more datasets with diverse scenes should be performed.\n\nIt would be beneficial to include a comprehensive comparison of the computational resources and inference speed between OpenMask3D and existing closed-vocabulary and open-vocabulary 3D instance segmentation methods. \n\nThe structure of Experiment section should be better organized.\n\n", "limitations": "The paper briefly mentions limitations of  OpenMask3D. It would be valuable to provide a more thorough analysis of the limitations, and discuss potential method / direction that could be explored to solve it.\n\nPotential negative societal impact is not applicable for this work.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors propose OpenMask3D as the first zero-shot approach for open-world 3D instance segmentation, offering a unique solution that leverages predicted class-agnostic 3D instance masks and CLIP-based image embeddings. This combination of ideas and techniques demonstrates originality in problem formulation, model architecture, and feature aggregation for open-vocabulary 3D instance segmentation.\n\nThe methodology is presented in a clear and structured manner, allowing readers to understand the proposed approach easily. The experimental evaluation is conducted on the common benchmark datasets, and the authors perform ablation studies to gain insights into the design choices of the model. \n\nThe results demonstrate that OpenMask3D outperforms other open-vocabulary counterparts, particularly in scenarios with a long-tail distribution of objects, which advance the state-of-the-art 3D instance segmentation in the complex real world.\n", "weaknesses": "While the paper demonstrates several strengths, there are also some areas that could be improved to further enhance its contributions and impact. \n\nEvaluation on a Single Dataset: The paper conducts experiments and ablation studies on the ScanNet200 dataset, which may limit the generalizability of the findings. It would be beneficial to evaluate the proposed OpenMask3D method on multiple datasets, including those with varying object distributions, to assess its robustness and performance across different scenarios. A more diverse evaluation would strengthen the claims made in the paper.\n\nLong Pipeline: another weakness of this work is the long pipeline of the proposed OpenMask3D model. The pipeline consists of multiple stages, including a class-agnostic mask proposal head, mask-feature aggregation module, and multi-view fusion of CLIP-based image embeddings. The length of the pipeline may introduce computational complexity and potentially impact the efficiency and real-time applicability of the model.\n\nLack of Computation Cost Comparison: While the paper discusses the architecture and methodology of OpenMask3D in detail, it does not provide a thorough comparison of the computational resources required and the speed achieved between different methods.\n\nProblematic Structure of section 4.1: sec 4.1 is named as \"Closed-vocabulary 3D instance segmentation evaluation\", while both closed-set instance segmentation results and open-vocabulary instance segmentation results are presented in this section. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback, and aim to address the questions in the following:\n\n### Q1: Evaluation on more datasets and settings\n\nThe reviewer suggests evaluating on other datasets to show the generalizability of OpenMask3D.\nWe agree and additionally run experiments on Replica, besides ScanNet200.\nFor the OpenScene baselines, we use the pre-computed fused features from the OpenScene github repository. Scores are shown in the table below. The results show that our approach generalizes well to new datasets, we obtain comparable scores as on ScanNet200.\n\n| Method | ScanNet200 (mAP$_{50}$)  | Replica (mAP$_{50}$|\n| --- | --- | --- | \n| OpenScene (2D Fusion) | 15.2 | 15.6 |\n| OpenScene (3D Distill) | 6.2 | 10.5 |\n| OpenScene (2D/3D Ens.) | 6.7 | 10.4 |\n| OpenMask3D (Ours) | 16.8 | 18.4 |\n\nWe additionally provide an experiment to analyze the generalization capability of our approach beyond the training categories. We use class-agnostic masks from a mask-predictor trained on the 20 original ScanNet classes, and evaluate on the ScanNet200 dataset. As we intend to assess how well our model performs on “unseen” categories, we categorize objects into two subsets: *base* and *novel* classes. We identify ScanNet200 categories that are semantically close to the original ScanNet20 classes (folded-chair, table, dining table, cabinet, kitchen cabinet, bathroom counter etc.), resulting in 53 categories. Below, we report results on seen (“base”) classes, unseen (“novel”) classes, and all classes. Full table is provided in Tab. 1 of the rebuttal PDF. \n\n|Method|Novel (AP$_{50}$)|Base (AP$_{50}$)|All (AP$_{50}$)|\n|---|---|---|---|\n| OpenScene (2D Fusion) |10.3|**15.0**|11.6|\n| OpenScene (3D Distill) |2.3|13.4|5.3|\n| OpenScene (2D/3D Ens.) | 2.8| 13.7|5.8|\n| OpenMask3D (Ours) |**12.9**| **15.0**| **13.5**|\n\n### Q2: Comparison of Computational Resources\n\nWe compare the necessary computational resources of the original OpenScene from [46] and our OpenMask3D. We report the runtimes and GPU memory requirements.\n\n****1) Runtime****\n\nIn the table below, we provide the runtime of our approach on ScanNet for generating the queryable 3D scene representation, and also for performing a query on that representation. For comparison, we additionally provide the runtimes of the original OpenScene [46]. All runtimes are measured on the same hardware.\n| Method | Generating 3D Scene Representation [s] | Querying 3D Scene Representation [ms] | Semantics? | Instances? |\n| --- | --- | --- | --- | --- |\n| OpenScene (2D Fusion) |  440 | 168.8 | $\\checkmark$| ✘  |\n| OpenScene (2D3D Ensemble) |  524 | 168.8 | $\\checkmark$| ✘  |\n| OpenMask3D |  556 | **0.923** |  $\\checkmark$| $\\checkmark$ |\n| OpenMask3D (fast) |  **350** | **0.923** |  $\\checkmark$| $\\checkmark$ |\n\nWe also report the runtime of a faster variation of our model (called ‘fast’) which runs CLIP only once per segment on a single crop and uses a smaller SAM backbone. The exact hyper-parameters are shown in the table below. This faster version performs almost as good as the original model (small performance drop of -1.3 mAP$_{50}$).\n\n| Model | top k | $\\mathsf{L}$ | SAM backbone | \n| --- | --- | --- | --- | \n| OpenMask3D |  5 | 3 |  ViT-H| \n| OpenMask3D (fast) |  1 | 1 |  ViT-B| \n\n\n****2) Minimum GPU Memory Requirement to run the Model****\n\nIn the table below we indicate the minimum GPU memory requirements for OpenScene and OpenMask3D. OpenScene has much stronger GPU requirements since it depends on OpenSeg.\n\n| Method | GPU memory | main components |\n|---|---|---|\n| OpenScene |  32 GB | OpenSeg: 32GB, CLIP-text: 4GB, OpenScene: 32GB|\n| OpenMask3D | 10 GB | SAM: 8GB, CLIP-image: 4GB, CLIP-text:  4GB |\n| OpenMask3D-fast | 4 GB | SAM: 4GB, CLIP-image: 4GB, CLIP-text:  4GB|\n\n### Q3: Structure of Section 4.1\n\nThe reviewer highlights the interleaved presentation of both open-set and closed-set results in this section.\nThis is correct and we will separate both settings more clearly to improve the organization of the experiment section.\n\n### Limitations\nWe agree with the reviewer that our paper would benefit from discussing the limitations further, and we will extend our section (L320-323) to include a discussion on class-agnostic mask quality as also discussed in (L294-308). Another limitation we would like to mention is that current top-k view selection algorithm directly relies on the visibility of an object instance in each frame - and this can benefit from additional criteria to assess the *quality* of a frame so that more informative frames could be selected. We have observed that frame-selection plays an important role, particularly for avoiding scene context to be overly infused into our per-mask features. Lastly, we also agree that a systematic approach is needed to evaluate open-vocabulary performance quantitatively, and call for future work in this direction.\n\n\n"}]

Author rebuttal:
We thank all the reviewers for their valuable feedback, we appreciate their detailed suggestions. We reply to each reviewer’s questions and concerns in the individual responses, and we have added tables and figures in the attached rebuttal PDF, which we reference and explain in the responses. 

Here, we also would like to provide an overview of the material in the attached document:

**Table 1.** We provide 3D instance segmentation results using masks from mask module trained on ScanNet, evaluated on the ScanNet200 dataset. We identify classes (such as chair, folded chair, table, dining table ...) that are semantically close to the original ScanNet classes, and group them as “Base”. Remaining classes are grouped as “Novel”. We also report results on the full set of labels.

**Table 2** - We provide additional experiments on the **Replica** dataset. We further analyze different training setups quantitatively.

**Table 3** - We provide an ablation study on the hyperparameters related to the multi-scale cropping stage of our approach.

**Table 4 & Table 5** -  We provide an overview of the memory requirements of foundation models used in OpenMask3D and OpenScene, and the time requirements for atomic operations. 

**Figure 1.a & Figure 1.c** - Qualitative comparison of per-mask (OpenMask3D) and per-point (OpenScene) features.

**Figure 1.b** - Illustration of our additional experiment on the Replica dataset, in which we render RGB-D images from the point clouds, and use these images as an input to our pipeline.

**Figure 1.d** - Qualitative results from open-vocabulary queries on the Replica dataset, using our OpenMask3D approach.

Meta-review: This paper initially received mixed reviews.  An initially negative reviewer upgraded their opinion to positive in light of the clarifications and details in the authors' rebuttal responses.  Two other reviewers who were positive indicated that the discussion period alleviated their concerns and their post-rebuttal opinion remains positive with respect to acceptance.  One negative remaining reviewer expresses a concern regarding novelty that the AC does not find to be substantiated with sufficient specificity, while another negative reviewer has not fully engaged in the discussion period or expressed an opinion after the author rebuttal.  Therefore, the AC does not find a strong basis to reject the paper.  The AC recommends that the authors incorporate the clarifications and additional results provided during the discussion period when preparing the camera ready version of the paper.
Decision: Accept (poster)