Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation
Authors: Mohamed El Amine Boudjoghra
Angela Dai
Jean Lahoud
Hisham Cholakkal
Rao Muhammad Anwer
Salman Khan
Fahad Shahbaz Khan
Venue: ICLR 2025 Oral (2025)
Keywords: Open Vocabulary
3D point cloud instance segmentation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent works on open-vocabulary 3D instance segmentation show strong promise but at the cost of slow inference speed and high computation requirements. This high computation cost is typically due to their heavy reliance on aggregated clip features from multi-view, which require computationally expensive 2D foundation models like Segment Anything (SAM) and CLIP. Consequently, this hampers their applicability in many real-world applications that require both fast and accurate predictions. To this end, we propose a novel open-vocabulary 3D instance segmentation approach, named Open-YOLO 3D, that efficiently leverages only 2D object detection from multi-view RGB images for open-vocabulary 3D instance segmentation. 
 We demonstrate that our proposed Multi-View Prompt Distribution (MVPDist) method makes use of multi-view information to account for misclassification from the object detector to predict a reliable label for 3D instance masks. Furthermore, since projections of 3D object instances are already contained within the 2D bounding boxes, we show that our proposed low granularity label maps, which require only a 2D object detector to construct, are sufficient and very fast to predict prompt IDs for 3D instance masks when used with our proposed MVPDist.
 We validate our Open-YOLO 3D on two benchmarks, ScanNet200 and Replica, 
 under two scenarios: (i) with ground truth masks, where labels are required for given object proposals, and (ii) with class-agnostic 3D proposals generated from a 3D proposal network.
 Our Open-YOLO 3D achieves state-of-the-art performance on both datasets while obtaining up to $\sim$16$\times$ speedup compared to the best existing method in literature. On ScanNet200 val. set, our Open-YOLO 3D achieves mean average precision (mAP) of 24.7% while operating at 22 seconds per scene. github.com/aminebdj/OpenYOLO3D

Published as a conference paper at ICLR 2025
OPEN- YOLO 3D: TOWARDS FAST AND ACCURATE OPEN- VOCABULARY 3D INSTANCE SEGMENTATION
OPEN-YOLO
Mohamed EI Amine Boudjoghra TUM, MBZUAI
Angela Dai TUM
Jean Lahoud MBZUAI
Hisham Cholakkal MBZUAI
Rao Muhammad Anwer MBZUAI, Aalto University
Salman Khan MBZUAL, ANU
Fahad Shahbaz Khan MBZUAL, Linköping University
ABSTRACT
Recent works on open-vocabulary 3D instance segmentation show strong promise but at the cost slow inference speed and high computation requirements. This high computation cost is typically due to their heavy reliance aggregated clip features from multi-view, which require computationally expensive 2D foundation models like Segment Anything (SAM) and CLIP. Consequently, this hampers their applicability in many real-world applications that require both fast and accurate predictions. To this end, propose novel open-vocabulary 3D instance seg- mentation approach, named YOLO 3D, that efficiently leverages only 2D object detection from multi-view RGB images for open-vocabulary 3D instance segmentation. We demonstrate that our proposed Multi- Prompt Distribution (MVPDist) method makes use of multi-view information to account for misclassifi- cation from the object detector to predict reliable label for 3D instance masks. Furthermore, since projections of 3D object instances are already contained within the 2D bounding boxes, show that our proposed low granularity label maps, which require only 2D object detector to construct, are sufficient and very fast to predict prompt IDs for 3D instance masks when used with our proposed MVPDist. We validate our Open Open-YOLO YOLO 3D on two benchmarks, ScanNer200 and Replica, under two scenarios: (i) with ground truth masks, where labels are required for given object proposals, and (ii) with alass-agnostic 31) proposals generated from a 3D proposal network. Our Open- Open-YOLO YOLO 3D achieves state-of-the-art performance on both datasets while obtaining ing method in literature. On ScanNer200 val. set, our Open-YOLO YOLO 3D achieves mean average precision (mAP) of 24.7% while operating at 22 seconds per scene. ggthub.co//mmieeh/j//eeooYLL0000
to ~~66 speedup compared to the best exist-
INTRODUCTION
3D instance segmentation is a computer vision task that involves the prediction of masks for indi- vidual objects 3D point cloud scene. holds significant importance fields like robotics and augmented reality. Due to its diverse applications, this task has garnered increasing attention in recent years. Researchers have long focused on methods that typically operate within closed-set framework, limiting their ability to recognize objects not present in the training data. This constraint poses challenges, particularly when novel objects must be identified or categorized in unfamiliar environments. Recent methods Nguyen et al. (2024); Takmaz et al. (2023) address the problem of novel class segmentation, but they suffer from slow inference that ranges from 5 minutes for small scenes to 10 minutes for large scenes due to their reliance computationally heavy foundation models like SAM Kirillov et al. (2023) and CLIP Zhang et al. (2023) along with heavy computation for lifting 2D CLIP feature to 3D.
opeenvocaaullry 3D instance segmentation robotics tasks such as manipulating objects and inventory management require accurate predictions while being fast in the decision-making process. Further- more, these tasks alter the point cloud throughout time, where objects can be rearranged, removed, added; this would require open vocabulary 3D instance segmentation pipelines to re-run from

---

Published as a conference paper at ICLR 2025
Ground truth
Ours
Open3DIS
"Throw the leftover food in trash bin"
"Take the dishes to the
sink"
Figure I: Open-vocabulary 3D instance segmentation with our Open-YOLO 3D. The proposed Open-YOLON 3D is capable of segmenting objects in zero-shot manner. Here, We show the output for a ScanNer220 Rozenberszki et al. (2022) scene with various prompts, where our model yields improved performance compared to the recent Open3DIS Nguyen al. (2024). We show zoomed-in
images of hidden predicted instances in the colored boxes. Additional results are in suppl. material.
scratch for every updated scene. Thus, recent methods would be ill-suited for such robotics tasks due to their low speed. Motivated by recent advances 2D object detection Cheng et al. (2024), we look into alternative approach that efficiently leverages fast object detectors instead of utilizing computationally expensive foundation models that are adopted by recent methods.
This paper proposes novel open- vocabulary 3D instance segmentation method, named Open- Open-YOLO YOLO 3D, that utilizes efficient, joint 2D-3D information using bounding boxes and projections from 3D point clouds. We employ open-vocabulan 2D object detector to generate bounding boxes with their class labels for all frames corresponding to the 3D scene; on the other side, we utilize a 3D) instance segmentation network to generate 3D class-agnostic instance masks for the point clouds, which proves to be much faster than 3D proposal generation methods from 2D instances Nguyen et al. (2024 Lu al. (2023). Unlike recent methods Takmaz et al.) (2023) Nguyen et al. (2024) which use SAM and CLIP to lift 2D clip features to 3D for prompting the 3D mask proposal, we propose an approach that relies on the bounding box predictions from 2D object detectors which prove to be significantly faster. We use the predicted bounding boxes in all RGB frames corresponding to the point cloud scene to construct a Low Granularity (L.G) label map for every frame. One I.G label map a two-dimensional array with the same height and width the RGB frame, with the bounding box areas replaced by their predicted class label. Next, we use our proposed MVPDist to assign the best possible prompt ID to the 3D masks by using multi-view information, present example output of our method in Figure T Our contributions are the following:
We introduce 2D object detection- based approach for open- vpca-vocabulary labeling of 3D instances, which efficiently uses object detectors to greatly improve the results.
We propose a novel approach to scoring 3D mask proposals using only bounding boxes from 2D object detectors.
Our Open-YOLO YOLO 3D achieves superior performance on two benchmarks, while being considerably faster than existing methods in the literature. On ScanNer200 val. set, our Open-YOLO YOLO 3D achieves an absolute gain of 2.3% at mAP50 while being ~66x faster compared to the recent Open3DIS Ngguyen et al. (2024).
2 RELATED WORKS
Closed vocabulayy 3D segmentation: The 3D instance segmentation task aims at predicting masks for individual objects in 3D scene, along with class label belonging the set of known classes.
2

---

Published as a conference paper at ICLR 2025
Some methods use grouping- based approach in bottom-up manner, by learning embeddings in the
latent space to facilitate clustering of object points Chen et al. (2021); Han et al. (2020); ); He et al.
(2021) Jiang et al. (2020); Lahoud et al. (2019); Liang al. (2021): Wang al. (0118) Zhang &
Wonka 2021). Conversely, proposal-based methods adopt a top-down strategy, initially detecting 3D)
bounding boxes and then segmenting the object region within each box [Engelman et al. (2020): Hou
(ct al. (2019); et al. (2020); Yang et al. (2019); Yi et al. (2019). Notably, inspired by advancements in 2D works Cheng et al. [2022 [021), transformer designs Vaswani al.
(2017) have been recently
applied to 3D instance segmentation tasks Schult et al. 2023); Sun et al. (2023): Kolodiazhnyi et al.
(2024); AT Khatib et al. (2023); Dain al. (2024). Mask3D Schult al. (2023) introduces the first
hybrid architecture that combines Convolutional Neural Networks (CNN) and transformers for this task. It uses 3D CNN backbone to extract per-point features and a transformer-base
instance mask decoder to refine set of queries. Building on Mask3D, the authors of Khatib et al. (2023) show that using explicit spatial and semantic supervision at the level of the 3D backbone further improves the instance segmentation results. Oneformer3D Kolodiazhnyi et al. (2024) follows similar architecture and introduces learnable kernels in the transformer decoder for a unified semantic, ODiN instance, and panoptic segmentation. ODIN Dain al. (2024) proposes an architecture that uses 2D-3D fusion to generate the masks and class labels. Other methods introduce
alternatives to dense annotation approaches, aiming to reduce the annotation cost associated with 3D data Chibane et al. (2022); ; Hou et al. (2021), y; Xie et al. (2020). While these methodologies strive to enhance the quality of 3D instance segmentation, they typically rely on a predefined set of semantic labels. In contrast, our proposed approach aims at segmenting objects with both known and unknown class labels.
Open- vocabbaccyylary 2D recognition: This task aims at identifying both known and novel classes, where the labels of the known classes are available in the training set, while the novel classes are not encountered during training. In the direction of open-vocabulary object detection (OVOD),
several approaches have been proposed Zhong et al. (2022); |hhm et al. (2024); [iu et al.
2023):
Zang et al. (2022); WWang et al. (2023); Kaul et al. (2023); 3 Yao al. (2023); Cheng et al. (0024).
Another widely studied task is open- vocabulary segmentation (OVSS) Bucher et | (2019); Xu et al.
(002)) Li et al. 2021); : Ghiasi al. [022); Liang et al. (2023). Recent voca-vocayulary semantic segmentation methods Li et al. 2021 Ghiasi et al. 3 Liang et al. (2023) leverage pre-trained
CLIP Zhang al. (2023) perform open-vocabulary segmentation, where the model is trained to output a pixel-wise feature that is aligned with the text embedding in the CLIP space. Furthermore, framework where vanilla class names are first decomposed into various attribute descriptions, and then different attribute representations are aggregated into a final class representation. Open-vocabulary instance segmentation (OVIS) aims at predicting instance masks while preserving high zero-shot capabilities. One approach Huynh et al. (2022) proposes a cross-modal pseudo-labeling framework, where student model is supervised with pscudo-labels for the novel classes from teacher model. Another approach VS et al. (2023) proposes an annotation-free method where a pre-trained lision-language model is used to produce annotations at both the box and pixel levels. Although these methods show high zero-shot performance and real-time speed, they are still limited to 2D applications only.
AttrSeg Ma et al. (2024) proposes decomposition--aggeeaal
Open-vocabulary 3D segmentation: Several methods Huang al. (2024); Peng et al. (022) ; Gu
ct al. (202)): Hong et al. (2023) have been proposed to address the challenges of open- voca--ocabulary semantic segmentation where they use foundation models like clip for unknown class discovery, while the authors of Bouddoghraa et al. (2023) focus on weak supervision for unknown class discovery without relying on any 2D foundation model. OpenScene Peng et al. (2023) makes use of 2D open-vocabulary semantic segmentation models to lift the pixel-wise 2D CLIP features into the 3D space, which allows the 3D model to perform 3D open-vocabulary point cloud semantic segmentation. On the other hand, ConcepiGraphs Gu et al. (2023) relies on creating an open- vocc--ocyuulrry scene graph that captures object properties such as spatial location, enabling a wide range of downstream tasks including segmentation, object grounding, navigation, manipulation, localization, and remapping. In the direction of 3D point cloud instance segmentation, OpenMask3I) Takmaz et al. (2023) uses a 3D instance segmentation network to generate class-agnostic mask proposals, along with SAM Kirillov ct al. (2023) and CLIP Zhang et al. (2023), to construct 3D clip feature for each mask using RGB-D images associated with the 3D scene. Unlike OpenMask3D where a 3D proposal network is used, OVIR-3D Lu et al.] (2023) generates 3D proposals by fusing 2D masks obtained by a 2D instance segmentation model. Open3DIS Nguyen et al. (2024) combines proposals from 2D and 3D with novel 2D masks fusion approaches via hierarchical agglomerative clustering, and also proposes to
3

---

Published as a conference paper at ICLR 2025
Input
Output
Projection
ippa Cutsst
30 Saa
1D priposale
Nettork
Multvittrr raps
R08-0 segieice
2D OVOD
"chair: "table" "Taatrees"
Figure Proposed open-world 3D instance segmentation pipeline. We use 3D instance segmentation network (3D Network) for generating class-agnostic proposals. For open-vocabulary prediction, a 2D Open- Vocabulary Object Detector (2D OVOD) generates bounding boxes with class labels. These predictions are used to construct label maps for all input frames. Next, assign the top-k label maps to each 3D proposal based on visibility. Finally, we generate a Multi-View Prompt Distribution from the 2D projections of the proposals to match text prompt to every 3D proposal.
use wisnt-wise 3D CLIP features instead of mask-wise features. The two most recent approaches in Nguyen al. (2024) Takmaz et al. (2023) show promising generalizability in terms of novel class discovery Takmaz et al. (2023) and novel object geometries especially small objects Nguyen et al. (2024). However, they both suffer from slow inference speed, as they rely on SAM for 3D mask proposal clip feature aggregation in the case of OpenMask3D Takmaz et al. (2023). and for novel 3D proposal masks generation from 2D masks Nguyen et al. (2024).
3 PRELIMINARIES
Problem formulation: 3D instance segmentation aims at segmenting individual objects within 3D scene and assigning one class label to each segmented object. In the voca-vocayulary (OV) setting, the class label can belong to previously known classes in the training labels. To this end, let P denote a 3D reconstructed point cloud scene, where a sequence of RGB-D images was used for the reconstruction. We denote the RGB image frames as along with their corresponding depth frames D. Similar to recent methods Peng et al. (2023); Takmaz et al. (2023); Nguyen al. (0044) we assume that the poses and camera parameters are available for the input 3D scene.
well as new class
3.1 BASELINE OPEN- VONABULAR 3D INSTANCE SEGMENTATION
We base approach OpenMask3D Takmaz al. (2023). which is the first method that performs open- vpcabuocayylary 3D instance segmentation in a zero-shot manner. OpenMask3D has two main modules: a class-agnostic mask proposal head, and mask-feature computation module. The class- agnostic mask proposal head uses transformer-basee pre-trained 3D) instance segmentation model Schult et al. (2023) to predict a binary mask for each object in the point cloud. The mask-feature computation module first generates 2D segmentation masks by projecting 3D masks into views in which the 3D instances are highly visible, and refines them using the SAM Kirillov al. (2023) model. A pre-trained CLIP vision- vision-language model Zhang et al. (2023) is then used to generate image embeddings for the 2D segmentation masks. The embeddings are then aggregated across all the 2D frames to generate 3D mask-feature representation.
Limitations: OpenMask31) makes use of the advancements in 2D segmentation (SAM) and vision-
language models (CLIP) to generate and aggregate 2D feature representations, enabling the querying of instances according to open-vocabulary concepts. However, this approach suffers from a high computation burden leading to slow inference times, with processing time of 5-10 minutes per scene. The computation burden mainly originates from two sub-tasks: the 2D segmentation of the large number of objects from the various 2D views, and the 3D feature aggregation based on the object visibility. We next introduce our proposed method which aims at reducing the computation burden and improving the task accuracy.
4

---

Published as a conference paper at ICLR 2025
4 METHOD: OPEN- YOLO 3D
We here present our proposed 3D open-vocabulary instance segmentation method, Open- YOLO 3D, which aims at generating 3D instance predictions in an efficient strategy. Our proposed method introduces efficient and improved modules at the task level as well as the data level. Task Level: Unlike OpenMask3D, which generates segmentations of the projected 3D masks, we pursue more efficient approach by relying 2D object detection. Since the end target is to generate labels for the 3D masks, the increased computation from the 2D segmentation task is not necessary. Data Level: OpenMask3D computes the 3DD mask visibility in 2D frames by iteratively counting visible points for each mask across all frames. This approach is time-consuming, and we propose an alternative approach compute the 3D mask visibility within all frames once.
Motivation:
4.1 OVERALL ARCHITECTREE
Our proposed pipeline is shown in Figure 2 First, we generate a set of instance proposals M using 3D instance segmentation network; the proposals represented as binary masks, where every 3D mask has a dimension equal to the number of points as the input point cloud. For the open vocabulary prediction, we use a 2D open-vocabulary object detection model to generate set of bounding boxes denoted B, for every frame I,; the bounding boxes with their predicted labels are used construct a low-gran oowgranularity label map C; for every input frame I. To assign a prompt ID to the 3D mask proposals, we first start by projecting all N points in the point cloud scene /' onto the Ny frames, which results in Nj 2D projection with N points for each. Afterward, the 2D projections and the 3D mask proposals are used to compute the visibility of every mask in every frame using our proposed accelerated visibility computation (VAcc); the visibility is then used to assign top-k Low- Granularity label maps to each mask and to select the top-k 2D projections corresponding to every 3D mask proposal; for a single 3D mask we crop the (x. y) coordinates from the projections using the instance mask and filter out the points that are occluded (x, y) coordinates from the top-k frames are used to select per-point labels from their corresponding Low-Granularig label maps to finally construct Multi- View Prompt Distribution to predict the prompt ID corresponding to the 3D mask proposal.
outside the frame. The final cropped
4.2 3D OBJECT PROPOSAL
To generate class-agnostic 3D object proposals, we rely on the 3D instance generation approach
Mask3D Schult et al. (2023), ) which allows for faster proposal generation compared to 2D
3D proposal generation methods Netuyen et al. (2024); Lu et al. (2023). Mask3D is hybrid model that combines a 3D Convolutional Neural Network as a backbone for feature generation and a transformer-based model for mask instance prediction. The 3D CNN backbone takes the voxelized input point cloud scene as input, and outputs multi-level feature maps, while the transformer decoder takes the multi-level feature maps refine a set of queries through self and cross-attention. The final refined queries are used to predict instance masks. The 3D proposal network predicts a set K3D E N of 3D mask proposals M E z gKanXX x.N for given point cloud P R4x N with N points in homogeneous coordinate system, where Z2 {0, 1}.
4.3 Low GRANULARITY (LG) LABEL-MAPS
As discussed earlier, the focus of our approach is to generate fast and accurate open-vocabulary labels computationally intensive 2DD segmentation, we propose a 2D detection-based approach in our pipeline. For every RGB image I, we generate set of Ki, bounding boxes B, {(by, c2) | by E g Cj E N, Vj E (I. Ks,;}} using an open-vocabulary 2D object detector, where b, are the bounding boxes coordinates while ej is its predicted label. We assign weight wy byf + 6, for each output bounding box by. where 6/f and 6/ are the bounding box's height and width, respectively. The weights represent the bounding box size and help determine the order of bounding boxes when used to construct the LG label maps.
for the generated 3D proposals. Instead of relying
After obtaining the 2D object detections, represent the output of each 2D image frame I, as an LG label map L. E zw XHI To construct L., we start by initializing all of its elements as -1. In our notation, -1 represents no class and is ignored during prediction. Next, we sort all bounding
5

---

Published as a conference paper at ICLR 2025
boxes following their weights and replace the region of the bounding box by with its corresponding predicted class label çj in the label map, starting with the bounding box with the highest weight. The weight w choice is motivated by the fact that if two objects of different sizes appear in the same direction the camera is pointing to, the small object is visible the image if is closer to the camera than the large object.
4.4 ACCELERATED VISIBILITY COMPUTATION (VAcc)
In order to associate 2D label maps with 3D proposals, we compute the visibility of each 3D mask. To this end, we propose fast approach that is able compute 3D mask visibility within frames via tensor operations which are highly parallelizable.
Given Ky N 2D RGB frames I {Z+ 3 Vi (L, Ky)} associated with the 3D scene P, along with their intrinsic matrices I X4xx4 and extrinsic matrices E E RNNxx444 We denote the projection of the 3D point cloud P on frame with index p2D E R1xN and it can be computed as follows P.D 1; E, P. After stacking the projection operations of all frames, the projection of the scene onto all frames can be computed with GPU in single shot follows
P2D (1 E) P
where batch-matrix multiplication, matrix multiplication, and P2D E RRNxxXX defined as P2D
P2D 12D .... Ny
Furthermore, we compute the visibility V 2 N of all projected points within all frames as follows
v = 110 < P2D < W) O 1(0 < P2D < H)
where is the indicator function, W and H are the image width and height respectively @ is element wise multiplication, P2D E N and paD RN are the projected 3D points æ and y coordinates on all N, frames, respectively.
For occlusion, we define another visibility matrix as Vd En NNNN that is computed as follows
yd=1((P"" D2 < Tdepth)
where D, NNNN is the real depth of the 3D point cloud obtained from the depth maps, while P2D P2D
is the depth obtained from projecting the point cloud P, and |. is the
IIIINN absolute value. Finally, the per 3D mask proposal visibility {V RNNxXKan, computed in a single shot using GPU as follows
V S 1]. can be
== ((V @ Vd). M c M-
where E Z X is the matrix of 3D mask proposals generated by the 3D proposal network, and Moount E Naaa is the count of points per mask,
The percentage of visibility of mask in frame i is represented with the element V.5 E V.
4.5 MULTI-VIEW PROMPT DISTRIBUTION (MVPDIST)
Given an RGB sequence and their corresponding LG label-maps L (C, C, ezwx Vi
(1....,y,)) }, We define the label distribution D, E Z Ni.. of a 3D mask proposal M, as
D, = {C, [P2D Msa. P2D Ms:]
Vi e Px}
where Myt Vdd ViL_ My is the mask for non-oceluded in-frame points that belong to the jth instance, Ps is the set of frame indices where the jth 3D mask has top-k visibility, computed using the visibility matrix V, PP2 and p?D are the projected x and coordinates of the point cloud scene onto the bu frame, respectively, while [.,] ZW + Z is coordinate-brat selection operator with n as an arbitrary natural number.
We define the probability for the mask M, be assigned to class the occurrence of class within the distribution Dy; we demonstrate the approach in Figure for one 3D proposal. We assign the class with the highest probability to the mask M₁.

---

Published as a conference paper at ICLR 2025
i 20 Projection
Prbbbbiity baty Pradicted Clax chill Mean: $1,07
Coopee
Selected set offabels
OWOD
Top-K label mage
Text prompe D
Figure 3: Multi-View Prompt Distribution (MVPDist). After creating the LG label maps all frames, we select the top-k label maps based on the 2D projection of the 3D proposal. Using the (x, y) coordinates of the 2D projection, we choose the labels from the LG label maps to generate the MVPDist. This distribution predicts the ID of the text prompt with the highest probability.
4.6 INSTANCE PREDICTION CONFIDENCE SCORE
We propose in this section a way to score class-agnoste 3D instance mask proposals by leveraging 2D information from bounding boxes along with 3D information from 3D masks. Our rationale that good mask should lead to high class and mask confidence, unlike previous 3D instance segmentation methods, which use only class confidence. To achieve this, we define score of 3D mask proposal Sm $1oU Sclass- where Sclass is the probability of the class with the highest occurrence in MVPDist, while SToU is the average IoU across multi-views between the bounding box of the projected 3D mask and the bounding box with the highest IoU generated with 2D object detector in the view (More details are in the appendix).
5 EXPERIMENTS
Datasets: We conduct our experiments using the ScanNet200 Rozenberszki et al. (002) and Replica Straub et al. 2019) datasets. Our analysis ScanNet200 is based on its validation set, comprising 312 scenes. For the 3D instance segmentation task, we utilize the 200 predefined categories from the ScanNet200 annotations. ScanNer200 labels categorized into three subsets-head (66 categories), common (68 categories), and tail (66 categories)-base on the frequency of labeled points in the training set. This categorization allows to evaluate our method's performance across the long-tail distribution, underscoring ScanNet200 as suitable evaluation dataset. Additionally, to assess the generalizability our approach, we conduct experiments on the Replica dataset, which has categories. For the metrics, we follow the evaluation methodology in ScanNet Dai et al. (2017) and report the average precision (AP) at two mask overlap thresholds: 50% and 25% well as the average across the overlap range of (0.5:0.95:0.05).
Implementation details: We use RGB-depth pairs from the ScanNet200 and Replica datasets, processing every 10th frame for ScanNer200 and all frames for Replica, maintaining the same settings as OpenMask3D for fair comparison. To create LG label maps, we use the YOLO- Woold Cheng et al. (2024) extra-large model for its real-time capability and high zero-shot performance. We use Mask3DD/Schll et al. (2023) with non-maximum suppression to filter proposals similar to Open3DIS Nguyen et al. (2024), and avoid DBSCAN [Eter al. (1996) to prevent inference slowdowns. We use a single NVIDIA A A00 40GB GPU for all experiments.
5.1 RESULTS ANALYSIS
Npen-Vocabulary 3D instance segmentation on ScanNet200: We compare our method's perfor- mance against other approaches on the ScanNer200 dataset in Table [] We indicate whether each method uses 2D instances to generate 3D proposals and whether SAM is used for labeling the 3D masks. Our method achieves state-of-the-ar of-tt-- performance with proposals from only 3D instance segmentation network compared to methods from two settings (i) 3D mask proposals from only a 3D instance segmentation network (ii) using combination of 3D mask proposals from 3D network and 2D instances from a 2D segmentation model. Additionally, our method is ~u 16x faster compared to state-of-the-a ofette-rrt Open3DIS.
7

---

Published as a conference paper at ICLR 2025
Table 1: State-of-the-arr comparison on ScanNet200 validation set. We use Mask3D trained on the ScanNet200 training set to generate alass-ag mask proposals. Our method demonstrates better performance compared to those that generate 3D proposals by fusing 2D masks and proposals from a 3D network (highlighted in gray in the table), while outperforms offteof-the-- art methods by a wide margin under the same conditions using only proposals from 3D network.
SAM
3D proposals
Method
from 2D masks
3D mask
mAP
mAP50
mAP25
head
tail
time/scene
labeling
Mask3D (Closed Vocab.)
26.9
36.2
41.4
21.7
17.9
13:41
SAM3D
14.2
21.3
6.2
4.6
482. 60
OVIR-3D Open3DIS
13.0
24.9
32.3
14.4
12.7
11.7
466. 80
23.7
29.4
32.8
27.8
21.2
21.8
360.
OpenSeene (2D Fusion)
X
x
11.7
17.8
13.4
11.6
9.9
46.45
OpenSeene (3D Distill)
X
4.8
6.2
7.2
10.6
2.6
0.7
0.26
OcenSeene (2D-3D Ens.)
X
X
5.3
6.7
8.1
11.0
3.2
1.1
46.78
OpenMask3D Open3DIS
23.1
17.1
14.9
$53.87
18.6
23.1
27.3
24.7
16.9
13.3
57.68
Open-YOLO 3D (Ours)
X
X
24.7
31.7
36.2
27.8
24.3
21.6
21.8
Table 2: State-of-the-art comparison on Replica dataset. We use Mask3D trained on the Scan- Net200 training set to generate class-agnostic mask proposals. We show that our method generalizes better than state-of-the-art methods under the same setting with proposals from 3D networks only.
SAM for
3D proposals
Method
3D mask labeling
mAP
mAP50
mAP25
time/scene (s)
from 2D masks
OVIR-3D Open3DIS
11.1
27.5
52.74
18.5
24.5
28.2
187.97
OpenScenc (2D fusion)
10.9
15.6
17.3
317.327
OpenScene (3D Distill)
8.2
10.5
12.6
4.29
OpenScenc (2D-3D Ens.)
8.2
10.4
13.3
320.127
OpenMask3D Open3DIS
13.1
18.4
24.2
547.32
14.9
18.8
23.6
35.08
Open-YOLO 3D (Ours)
23.7
28.6
34.8
16.6
Generalizability to unseen dataset: To test the generalizability of our method, we use the 3D proposal network pre-trained on the ScanNer200 training set and evaluate on the Replica dataset; the results are shown in Table 2 Our method shows competitive performance with ~ 11 x speedup against state-of-thhe-ar models Nguyen et al.] (2024), which use CLIP features. We highlight that OpenScene (3D Distill) achieves the highest speed by using a trained 3D U-Net to directly predict per-point CLIP features in under second, whereas 2D prior-based methods are slower but offer better generalization.
Performance with given 3D masks: We further test our method for 3D proposal prompt- ing from text against existing methods in the literature in the case of ground truth 3D masks and report the results in Table For Mask3D, oracle masks are assigned class predictions from matched predicted masks using Hungarian matching. For open-vocabulary methods, we use ground-truth masks as input proposals. We show the results in thtat MVPDist out- perform
Table 5: Comparative results with oracle masks on ScanNet200 dataset. Our MVPDist significantly outperforms state-of-the art methods using CLIP features. We show that MVPDist with YOLO-WORD predictions achieve high zero-shot capabilities.
multi-view information to
2D OV Prior
Method
mAP
head
comm
tail
Closed Vocab. Mask3D
None
35.5
55.2
27.2
22.2
OpenSoenc OpenSeenc
2D LSeg
11.8
26.9
1.7
2D OpenSeg
22.9
26.2
22.0
20.2
OpenMask3D Open3DIS
Mask wise CLIP
29.1
31.1
24.0
32.9
Point wise CLIP
30.9
33.1
25.1
35.4
Open-YOL.O (Ours)
MVPDist
39.6
43.3
36.8
38.5
approaches in retrieving the correct 3D proposal masks from text prompts, due to the high zero-shot performance achieved by state-of-the-art open-vocabulary 2D object detectors.
8

---

Published as a conference paper at ICLR 2025
"Square Table"
"plant"
Open3DIS
Ours
Open3DIS
Ours
Figure 4: Qualitative results on scene office0 in the Replica dataset. We show instances with confidence score above 0.5 for both methods. We show that our method is much more precise when segmenting the object in the text compared to state-of-the-art method Open3DIS.
Table 3: Effect of crop generation prior on OpenMask3D. R1 corresponds to the original Open-
Mask3D, while we show the effect of replacing SAM with 2D object detector in rows R2 R4.
Labeling
time/scene
Row ID
2D Prior
MVPDist
VAce
mAP
method
(s)
RI
SAM+CLIP
Clip features
X
X
33.0
675.6
R2
YoLOV&+CLIP
Clip features
X
X
21.1
440.40
R3
RT-DETR+CLIP
Clip features
X
X
28.4
487.95
R4
CoLD
Clip features
X
X
32.5
384.29
Generalizability out of dis- tribution classes: We conduct an experiment similar to Open- Mask3D, and report the results in Table 6 In this experi- ment, evaluate model's performance using Mask3D pre- trained the ScanNet20 dataset to generate class-agnostic masks. The results are reported for Novel classes, comprising 147 classes from ScanNet200 that differ from the 18 classes in ScanNer20, and for Base classes, which consist of 47 classes that overlap with those in ScanNet20. Our method demonstrates better perfor- mance on both Novel and Base classes.
Table 6: Comparative results on ScanNet200 validation set with Mask3D pretrained on ScanNet20 training set. We show the results with method Novel and Base classes defined in Mpen paper Takmaz et al. (2023), with class agnostic masks generated by Mask3D pre-trained ScanNet20.
Model
mAP
mAPbas
mAP novel
time/scene (s)
OpenMask3D
12.6
14.3
11.9
511.41
Open3DIS
19.0
25.8
16.5
312.11
Yolo3D
19.6
26.1
17.1
22.7
Ablation over Replica dataset:
Table 7: Ablation study on deducted components and their impact mAP and inference time.
To evaluate object detectors as alternatives to SAM for generat- ing crops in 3D CLIP feature ag- gregation, tested three detec-
Row
Deducted Components
CLussification Prior
mAP
Time
Ours (MVPDist Vace)
YoloWorld
392.02
Ours (MVPDist & Vacc)
CLIP features
32.5
396,89
tors on the Replica dataset, using ground truth 3D mask proposals to compare labeling performance.
Ours Vace
YoloWorld
46.2
376.42
Ours
YoloWorld
17.86
Results shown in Table rows R2 to R4, with RI presenting baseline results from OpenMask3D
Takmaz et al. (2023). We generate class-agnostic bounding boxes with an object detector, then assign the highest IoU bounding box to each 3D instance as a crop, selecting the most visible views. For 3D CLIP feature aggregation, we follow OpenMask3D's approach, aggregating features from multiple levels and views. These experiments show that YOLO- W-rlld can generate crops nearly as good as SAM but with significant speed improvements. To show the effect of adding our contributions on the overall performance, we report the results in table
show in Row the zero-shot performance of proposed View Prompt Distribution with LG label maps compared to (i) Row O where assign the 3D mask a class label based on the Yolo World prediction with the highest IoU overlap and confidence across the views. This experiment in row relies solely Yorld predictions (ii) using the class prediction obtained from the CLIP feature of the corresponding matched bounding box. Additionally, Row 3 shows speed improvements using the GPU for 3DD mask visibility computation.

---

Published as a conference paper at ICLR 2025
Table Effeet of number of top-k views on the labeling performance with High Granularity (HG) and cow-Granllarity (LG) label maps. HG Label maps are generated by prompting SAM with the bounding boxes used to generate LG label maps.
Topk (-)
10 mAP/ time(s)
20 mAP/ time(s)
30 mAP/ time(s)
40 mAP/ time(s)
Map type (4)
mAP/ time(s)
HG label maps
16.3/113.93
22.5 115.16
23.9/ / 114.76
24.4 / 115.89
24.5 115.92
LG label maps
16.3 20.7
22.8 / 21.6
24.3 / 21.7
24.6 / 22.1
24.7 / 22.2
Table 8: Results with 3D masks lifted using hierarchical agglomerative clustering proposed in Open3DIS. For 2D) mask generation, we use G-DINO and SAM; then we compare the class label prediction capabilities for the 3D aggregated masks between point-wise features with clip in Open3DIS and our proposed Multi-View prompt distribution with YoloWorld.
Model
Proposal method
mAP
mAP50
mAP25
mAP
mAP. com
mAP
Time/scene (8)
Open3DIS
3D-Net 2D-Net
23.7
29.4
32.8
27.8
21.2
21.8
360.12
Ours
3D-Net 2D-Net
25.6
31.9
36.1
27.9
24.2
24.7
303.41
Top K analysis:
We show in Table that naively using YOLO- World with only one label-map with the highest visibility per 3D mask proposal results in sub-optimal results and using top-K label-maps result in better predictions since YOLO-World is also expected to make misclassifications in some views while generating correct ones in others. This approach assumes that YOLO-World makes a correct class prediction for the same 3D object in multiple views for it to be effective.
the distribution can provide better estimate across multiple frames,
High-Granularity (HG) vs. Low-Granularity (LG): Table shows that using SAM to generate HG label maps slightly reduces mAP, and slows down the inference by ~ 5 times. This is due to the nature of projected 3D instances into 2D, where the projection already holds 2D instance information as shown in Figure and SAM would just result in redundancy in the prediction.
How does Open Yolo3D compare other clip-based methods under different scenarios? In scenarios such as robot navigation where the RGB-D sequence and point cloud change over time or when the user has a set of classes that they want to extract in single shot, we show through our experiments that Yolo3D significantly outperforms other methods, such as OpenMask3D Takmaz et al. (2023) and Open3DIS Nguyen et al. (2024), where the entire pipeline must be executed from scratch for set of text prompt. However, in static scenes where users want to prompt objects sequentially, clip-based models can run once to extract clip features from multiple views, allowing them to respond new prompts within milliseconds. In contrast, method, with stored visibility matrix and visual features from the 2D detector backbone, takes to seconds for each new prompt under the later scenario for small scenes.
Our method makes use of a 3D proposal network only for proposal generation in order reach high speed. Other proposal generation methods Lu al. (2023): Nguyen al. (2024) fuse 2D instance masks from a 2D instance segmentation methods to generate rich 3D proposals even for very small objects, which are generally overlooked by 3D proposal networks like Mask3D Schult et al.] (2023) due to low resolution in 3D. Thus, fast 2D instance segmentation models like FastSAM Zhao et al. 2023) can be used to generate 3D proposals from the 2D images, which might further improve the performance of our method.
Limitations:
6 CONCLUSION
We present Open-YOLO 3D, novel and efficient open- veccbulaayulary 3D instance segmentation method, which makes use open-vocabulary 2D object detectors instead of heavy segmentation models. Our approach leverages 2D object detector for class-labeled bounding boxes and a 3D instance segmentation network for class-agnostic masks. We propose to use MVPDist generated from multi- view low granularity label maps to match text prompts to 3D class agnostic masks. Our proposed method outperforms existing techniques, with gains in mAP and inference speed. These results show new direction toward more efficient open-vocabulay 3D instance segmentation models.
10

---

Published as a conference paper at ICLR 2025
Acknowledgement The computations were enabled by resources provided by NAISS at Alvis partially funded by Swedish Research Council through grant agreement no. 2022-06725, LUMI hosted by CSC (Finland) and LUMI consortium, and by Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the NSC.
REFERENCES
Salwa Al Khatib, Mohamed EI Amine Boudjoghra, Jean Lahoud, and Fahad Shahbaz Khan. 3d instance segmentation via enhanced spatial and semantic supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 541-550, 2023.
Mohamed El Amine Boudjoghra, Salwa K. Al Khatib, Jean Lahoud, Hisham Cholakkal, Rao Muham- mad Anwer, Salman Khan, and Fahad Khan. 3d indoor instance segmentation in open-world. In Advances in Neural Information Processing Systems, 2023.
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. Advances in Neural Information Processing Systems. 32, 2019.
Shaoyu Chen, Jiemin Fang, Qian Zhang, Wenyu Liu, and Xinggang Wang. Hierarchical aggregation for 3d instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15467-15476, 2021.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification all you need for semantic segmentation. Advances in Neural Information Processing Systems, 14:17864-17877
2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1290-1299, 2022.
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open- vocabulary object detection. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2024.
Julian Chibane, Francis Engelmann, Tuan Anh Tran, and Gerard Pons-Moll. Box2mask: Weakly supervised 3d semantic instance segmentation using bounding boxes. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXI, pp. 681-699. Springer, 2022.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Rinhly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, and Matthias Nießner. 3d- mpa: Multi-proposal aggregation for 3d semantic instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9031-9040, 2020.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. density- based algorithm for dis- covering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD'96 96, pp. 226-231. AAAI Press, 1996.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Scaling open-vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pp. 540-557. Springer, 2022.
Qiao Gu, Alihusein Kuwajerwala, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Cor- ban Rivera, William Paul, Rama Chellappa, Chuang Gan, Celso M de Melo, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2nd Workshop on Language and Robot Learning: Language as Grounding, 2023.
Lei Han, Tian Zheng, Lan Xu, and Lu Fang. Occuseg: Occupancy-awan 3d instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2940-2949 2020.
11

---

Published as a conference paper at ICLR 2025
Tong He, Chunhua Shen, and Anton Van Den Hengel. Dyco3d: Robust instance segmentation of 3d point clouds through dynamic convolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 354-363, 2021.
Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 3d concept learning and reasoning from multi-view images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9202-9212, 2023.
Hou, Angela Dai, and Matthias Nießner. 3d-sis: 3d semantic instance segmentation of rgb-d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4421 4430, 2019.
Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15587-15597, 2021.
Zhening Huang, Xiaoyang Wu, Chen, Hengshuang Zhao, Lei Zhu, and Joan Lasenby. Openins3d: Snap and lookup for 3d open-vocabulan instance segmentation. European Conference Com- puter Vision, 2024.
Dat Huynh, Jason Kuen, Zhe Lin, Jiuxiang Gu, and Ehsan Elhamifar. Open-vocabulary instance segmentation via robust cross-modes pseudo-labcling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7020-7031, 2022.
Ayush Jain, Pushkal Katara, Nikolaos Gkanatsios, Adam W Harley, Gabriel Sarch, Kriti Aggarwal, Vishrav Chaudhary, and Katerina Fragkiadaki. Odin: A single model for 2d and 3d perception. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and Pattern recognition, pp. 4867-4876, 2020.
Prannay Kaul, Weidi Xie, and Andrew Zisserman. Multi-modal classifiers for open-vocabulary object detection. In International Conference on Machine Learning, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Maxim Kolodiazhnyi, Anna Vorontsova, Anton Konushin, and Danila Rukhovich. Oneformer3d: One transformer for unified point cloud segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Jean Lahoud, Bernard Ghanem, Mare Pollefeys, and Martin R Oswald. 3d instance segmentation via multi-task metric learning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 0266, 2019.
Jean Lahoud, Jiale Cao, Fahad Shahbaz Khan, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, and Ming- Hsuan Yang. 3d vision with transformers: A survey, 2022.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations, 2021.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open- vocanuocyy semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, and Kui Jia. Instance segmentation in 3d scenes using semantic superpoint tree networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2783-2792, 2021.
Shih-Hung Liu, Shang- Yi Yu, Chao-Chi Wu, Hwann- Soaan-Tzong Chen, and Tyng- Luh Liu. Learning gaussian instance segmentation in point clouds. arXiv preprint arXiv:2007.0986/ 2020.
12

---

Published as a conference paper at ICLR 2025
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023.
Shiyang Lu, Haonan Chang, Eric Pu Jing, Abdeslam Boularias, and Kostas Bekris. Ovir-3d: Open- vocabulary 3d instance retrieval without training on 3d data. In Conference on Robot Learning, pp. 1610-1620. PMLR, 2023.
Chaofan Ma, Yang Yuhuan, Chen Ju, Fei Zhang, Ya Zhang, and Yanfeng Wang. Open-vocabulary semantic segmentation via attribute caapmmoosiiionngggggattii Advances in Neural Information Processing Systems, 36, 2024.
Phuc DA Nguyen, Tuan Due Ngo, Chuang Gan, Evangelos Kalogerakis, Anh Tran, Cuong Pham, and Khoi Nguyen. Open3dis: Spen-vocabulary 3d instance segmentation with 2d mask guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Jing Nie, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang, and Ling Shao. Enriched feature guided refinement network for object detection. Proceedings of the IEEE/CVF international conference computer vision, pp. 9537-9546, 2019.
Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, al. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 815-824, 2023.
Chau Pham, Truong Vu, and Khoi Nguyen. Lp-ovod: Open-vocabulary object detection by linear probing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 779-788, 2024.
David Rozenberszki, Or Litany, and Angela Dai. tation in the wild. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
indoor 3d semantic segmen-
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3D: Mask Transformer for 3D Semantic Instance Segmentation. In International Conference on Robotics and Automation (ICRA), 2023.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The replica dataset: A digital replica of indoor spaces, 2019.
Jiahao Sun, Chunmei Qing, Junpeng Tan, and Xiangmin Xu. Superpoint transformer for 3d scene instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Ayca Takmaz, Elisabetta Fedele, Robert Sumner, Mare Pollefeys, Federico Tombari, and Francis Engelmann. Openmask3d: Open-vocabulary 3d instance segmentation. Advances in Neural Information Processing Systems, 36, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
Vibashan VS, Ning Yu, Chen Xing, Can Qin, Mingfei Gao, Juan Carlos Niebles, Vishal M Patel, and Ran Xu. Mask-free ovis: Open-vocabularry instance segmentation without manual mask annotations. In P Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp.
25339-23549, 2023.
Luting Wang, Yi Liu, Penghui Du, Zihan Ding, Yue Liao, Qiaosong Qi. Biaolong Chen, and Si Liu. Object-aware distillation pyramid for open-vocabulary object detection. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 11 111866117 1996, 2023.
13

---

Published as a conference paper at ICLR 2025
Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neumann. Sgpn: Similarity group proposal network for 3d point cloud instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2569-25767 2018.
Saining Xie. Jiatao Gu, Demi Guo, Charles Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part III 16, pp. 574-591. Springer, 2020.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18134-18144, 2022.
Bo Yang, Jianan Wang, Ronald Clark, Qingyong Hu, Sen Wang, Andrew Markham, and Niki Trigoni. Learning object bounding boxes for 3d instance segmentation on point clouds. Advances in neural information processing systems, 32, 2019.
Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23497-23506, 2023.
Li Yi, Wang Zhao, He Wang, Minhyuk Sung, and Leonidas Guibas. Gspn: Generative shape proposal network for 3d instance segmentation in point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3947-3956, 2019.
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In European Conference Computer Vision, pp. 106-122. Springer, 2022.
Biao Zhang and Peter Wonka. Point cloud instance segmentation using probabilistic embeddings. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8883-8892, 2021.
Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2048-2059, 2023.
Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv preprint arXiv:2306. 12/56, 2023.
Yiwu Zhong. Jianwei Yang. Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based aagguage-image pre- training. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16793-16803, 2022.
14


Reviews:
[{"review_id": "Qfehz0Hdsj", "reviewer": "Reviewer_qcNj", "summary": "This paper introduces an efficient 3D mask labeling method that leverages multi-view 2D label maps, referred to as Low Granularity (LG) Label Maps, created from 2D object bounding boxes to label 3D instances. The 3D instance (mask) proposals are generated using a pre-trained class-agnostic 3D segmentation method. To address object occlusion across different viewpoints, an Accelerated Visibility Computation (VACC) method is introduced, enabling rapid calculation of visibility matrices using intrinsic and extrinsic parameters.", "questions": "1. In Table 1, does the class-agnostic Mask3D model have access to mask annotations for the same classes as those in the validation set? Do the other methods use the same class-agnostic segmentation model?\n\n2. Since the proposed approach relies on a class-agnostic 3D instance generation model, what are the advantages of using only mask annotations, rather than both instance and label annotations, for training? I mean, are there practical scenarios where only mask annotations are available?\n\n3. What does the tag \"(Closed Vocab)\" mean in Table 1? Does it indicate that the Mask3D method uses both mask annotations and object class annotations for training?\n\n4. What is the performance of Mask3D (Closed Vocab.) on the Replica dataset?\n\nMinor：\n\n1. In line 092, a comma is missing after \"multi-view information\".", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized, and the ideas are clearly illustrated.\n2. This paper introduces a novel approach for efficient open-vocabulary 3D instance labeling by leveraging 2D bounding box priors from a fast 2D object detector, demonstrating superior performance and time efficiency in experimental results.\n3. A fast visibility computation algorithm (VAcc) is proposed to accelerate the process of associating 2D label maps with 3D proposals that may be occluded in some views.  This algorithm demonstrates both efficiency and robustness to variations in label map granularity.", "weaknesses": "1. The foundation of the proposed method is built upon the class-agnostic 3D segmentation model, Mask3D, which is used to generate 3D mask proposals. However, this paper lacks sufficient evidence to demonstrate Mask3D's effectiveness and generalizability for open vocabulary instance proposals.\n2. The experimental evaluation of the proposed method for open-vocabulary 3D instance segmentation is relatively limited (only Table 6).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e42gyaE7bs", "reviewer": "Reviewer_LA4L", "summary": "This paper primarily aims to achieve faster open-vocabulary 3D instance segmentation compared with existing methods like OpenMask3D. To realize this target, this work first uses a 3D instance segmentation network to generate segmentation proposals. Then, the output of an open-vocabulary 2D object detector as well as some designed 3D information is employed to derive the categories of these proposals.", "questions": "See weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **[Efficiency]** The experimental results suggest that the proposed method achieves high precision with a significantly better speed compared with most methods, and efficiency is important for practical deployment.\n\n2. **[Clearness]** This paper explains its main contribution, how to assign class predictions to 3D proposals, with great clarity. The implementation details are elaborated sufficiently.", "weaknesses": "1. **[Insufficient Academic Contributions]**: This work just combines the output of a 3D segmentation network and a well-implemented open-vocabulary 2D object detector to realize open-vocabulary 3D object detection (similar to existing open-world segmentation method, just with a replacement of the post network to 2D object detector), which is trivial. It is much faster than previous methods because previous methods are developed based on models like SAM and CLIP. This work employs more efficient and suitable existing models. Therefore, although this work is sound in terms of engineering, its real academic contribution and new insights are plain.\n\n2. **[Insufficient Ablation Study]** As the method is efficient because it makes good use of existing models, it is important to clearly analyze how these models contribute to the efficiency, which will guide future works on how to develop an efficient open-vocabulary pipeline. However, this work fails to do so.\n\n3. **[Misleading title]** The method name OPEN-YOLO 3D seems to be unsuitable. YOLO is a 2D object detector while the task is about 3D point cloud segmentation. Although the method utilizes the output of YOLO-World to generate class predictions, the method name is still a little misleading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nixKHPI5hD", "reviewer": "Reviewer_4bgM", "summary": "This paper proposes Open-YOLO 3D, which is an open-vocabulary 3D instance segmentation framework that efficiently combines 2D object detection and 3D mask generation. The key idea of this paper is its reliance on bounding box predictions from a 2D open-vocabulary object detector and the subsequent use of these predictions for efficient 3D mask proposal and labeling. Unlike prior methods that use computationally intensive models such as SAM and CLIP for feature lifting from 2D to 3D, this paper uses a novel Multi-View Prompt Distribution (MVPDist) and Accelerated Visibility Computation (VAcc) methods to speed up the segmentation process. The framework this paper proposed achieves up to 16x faster inference while keeping competitive or better accuracy.", "questions": "Could you provide a more detailed theoretical analysis or complexity comparison of VAcc with the conventional method?\nHow does the method perform when the 2D object detector encounters difficult conditions, such as poor lighting or significant occlusion, if there's any evaluation of the robustness under such conditions?\nWhat are the potential strategies for mitigating errors from misclassifications made by the 2D object detector, and how do they affect the 3D mask assignments?\nI'm also particularly interested in the discussion in your limitation section, I was wondering would integrating fast 2D segmentation models, as mentioned, be feasible within your current framework? How might this affect both performance and speed?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper proposed a novel framework, which uses 2D object detection for 3D instance segmentation. The model they presented reduced computational overhead significantly. The Accelerated Visibility Computation (VAcc) leverages tensor operations and GPU batch processing, enabling highly parallelized visibility computation. This contributes to the following speed improvements without compromising performance. By integrating a high-performing 2D open-vocabulary detector, the framework retains strong zero-shot performance, which is important for real-world applications that use new or unknown object types.\n\nIt also includes detailed experiments that showcase Open-YOLO 3D's speech and accuracy, and highlight its performance above state-of-the-art approaches like Open3DIS and OpenMask3D. The paper also includes comprehensive ablation studies to demonstrate the improvement of each component. \n\nThe overall writing is clear and the framework will be beneficial for related research.", "weaknesses": "I like the overall framework this paper presents and appreciate its contribution to 3D instance segmentation by introducing an inference-efficient model, but I still have some concerns about it:\n\nWhile the paper mentions that VAcc uses tensor operations, a deeper explanation or complexity analysis comparing it to conventional iterative methods would strengthen the understanding of its true computational advantage, and the reason why it can achieve faster inference speed. I believe the paper clearly demonstrates the operation of this proposed algorithm, however, more explanation about why it is efficient and how much computation cost it saves will better demonstrate the paper's contribution.\n\nThe method relies on the quality of the 2D object detector, and this might be an issue if the 2D views are suboptimal (for example poor lighting, and occlusions). A more extensive analysis or discussion on how 2D detection failures propagate through the pipeline would add value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jzqiTlMBYF", "reviewer": "Reviewer_C5qy", "summary": "The authors propose an efficient method for open-vocabulary 3D instance segmentation to enhance the real-time capability. Unlike existing methods that rely on obtaining 2D masks and category labels from 2D foundation models (like SAM and CLIP), the authors propose a novel approach, Open-YOLO3D, which only leverages bounding boxes generated by 2D object detectors. Moreover, the authors propose a Multi-View Prompt Distribution (MVPDist) method to endeavor promising performance in recognition. The experimental results demonstrate the promising real-time performance of the method proposed by the authors.", "questions": "Please see the weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe authors only utilize the bounding boxes from 2D object detectors to alleviate the redundancy brought by 2D masks, which demonstrates a significant improvement in inference speed compared to OpenMask3D.\n2.\tThe authors propose a Multi-View Prompt Distribution to obtain reliable category labels form 3D masks, the experimental results evaluation on the ScanNet200 and Replica datasets prove the efficiency of the method proposed by the authors.\n3.\tThe paper is well-structured, and the connection between the proposed method and the motivation is coincident.", "weaknesses": "1.\tThe improvements of the segmentation performance observed in Open-YOLO3D primarily arise from the enhanced category recognition, which is likely from the prior knowledge of the pre-trained YoloWorld model.\n2.\tThe challenges inherent in Open-YOLO3D closely resemble those faced by Open3DIS, as both methods rely on pre-trained models for generating 3D proposals. As discussed in Open3DIS, the pre-trained 3D models have limited capabilities when it comes to detecting uncommon categories. The representation of 3D data for open vocabulary instance segmentation might be uncultivated and limited.\n3.\tRecent studies [1] have indicated that OpenMask3D performs poorly on certain outdoor datasets, such as NuScenes. Does Open-YOLO3D face similar challenges in effectively identifying sparse-diverse and less common categories in outdoor environments?\n\n[1] Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V1Jl2kfaLG", "reviewer": "Reviewer_rQL1", "summary": "This paper aims to deal with the open-vocabulary 3D instance segmentation task with a fast and cost-effective approach by utilizing a YOLO-style design. A Multi-View Prompt Distribution method is proposed to effectively fuse the multi-view information. The low granularity label maps are proposed to only use 2D detectors to predict prompt IDs for 3D instance masks. Experimental results demonstrate the state-of-the-art performance of the proposed method. The speed of the proposed method is about 16 times faster than that of existing methods.", "questions": "Please draw my concerns listed in the Weaknesses part.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe proposed method is simple yet effective.\n2.\tThe two proposed designs are helpful for 3D instance segmentation with meaningful designs.\n3.\tThe experimental results show that the proposed method could achieve good performance while remaining very efficient.", "weaknesses": "1.\tIs it possible to extend the proposed method on panoptic segmentation of 3D scenarios? Please present your design briefly for this. \n2.\tAs shown in Table 1, the inference time of the proposed method is 21.8, which is slower than OpenScene (3D Distill). Please add the explanation for this phenomenon in the corresponding text (first paragraph of Section 5.1).\n3.\tIn Line 405, it should be 4.29 but not 04.29 for OpenScene (3D Distill).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)