Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation

## Abstract

Abstract Recent works on open-vocabulary 3D instance segmentation show strong promise, but at the cost of slow inference speed and high computation requirements. This high computation cost is typically due to their heavy reliance on 3D clip features, which require computationally expensive 2D foundation models like Segment Anything (SAM) and CLIP for multi-view aggregation into 3D. As a consequence, this hampers their applicability in many real-world applications that require both fast and accurate predictions. To this end, we propose a fast yet accurate open-vocabulary 3D instance segmentation approach, named Open-YOLO 3D, that effectively leverages only 2D object detection from multi-view RGB images for open-vocabulary 3D instance segmentation.
We address this task by generating class-agnostic 3D masks for objects in the scene and associating them with text prompts.
We observe that the projection of class-agnostic 3D point cloud instances already holds instance information; thus, using SAM might only result in redundancy that unnecessarily increases the inference time.
We empirically find that a better performance of matching text prompts to 3D masks can be achieved in a faster fashion with a 2D object detector.
We validate our Open-YOLO 3D on two benchmarks, ScanNet200 and Replica,
under two scenarios: (i) with ground truth masks, where labels are required for given object proposals, and (ii) with class-agnostic 3D proposals generated from a 3D proposal network.
Our Open-YOLO 3D achieves state-of-the-art performance on both datasets while obtaining up to ∼ similar-to \sim ∼ 16 × \times × speedup compared to the best existing method in literature.
On ScanNet200 val. set, our Open-YOLO 3D achieves mean average precision (mAP) of 24.7% while operating at 22 seconds per scene.
Code and model are available at github.com/aminebdj/OpenYOLO3D

## 1 Introduction

3D instance segmentation is a computer vision task that involves the prediction of masks for individual objects in a 3D point cloud scene. It holds significant importance in fields like robotics and augmented reality. Due to its diverse applications, this task has garnered increasing attention in recent years. Researchers have long focused on methods that typically operate within a closed-set framework, limiting their ability to recognize objects not present in the training data. This constraint poses challenges, particularly when novel objects must be identified or categorized in unfamiliar environments. Recent methods [, ] address the problem of novel class segmentation, but they suffer from slow inference that ranges from 5 minutes for small scenes to 10 minutes for large scenes due to their reliance on computationally heavy foundation models like SAM [] and CLIP [] along with heavy computation for lifting 2D CLIP feature to 3D.

Open-vocabulary 3D instance segmentation is important for robotics tasks such as, material handling where the robot is expected to perform operations from text-based instructions like moving specific products, loading and unloading goods, and inventory management while being fast in the decision-making process. Although state-of-the-art open-vocabulary 3D instance segmentation methods show high promise in terms of generalizability to novel objects, they still operate in minutes of inference time due to their reliance on heavy foundation models such as SAM. Motivated by recent advances in 2D object detection [], we look into an alternative approach that leverages fast object detectors instead of utilizing computationally expensive foundation models.

This paper proposes a novel open-vocabulary 3D instance segmentation method, named Open-YOLO 3D, that utilizes efficient, joint 2D-3D reasoning, using 2D bounding box predictions to replace computationally-heavy segmentation models. We employ an open-vocabulary 2D object detector to generate bounding boxes with their class labels for all frames corresponding to the 3D scene; on the other side, we utilize a 3D instance segmentation network to generate 3D class-agnostic instance masks for the point clouds, which proves to be much faster than 3D proposal generation methods from 2D instances [, ].
Unlike recent methods [, ] which use SAM and CLIP to lift 2D clip features to 3D for prompting the 3D mask proposal, we propose an alternative approach that relies on the bounding box predictions from 2D object detectors which prove to be significantly faster than CLIP-based methods. We utilize the predicted bounding boxes in all RGB frames corresponding to the point cloud scene to construct a Low Granularity (LG) label map for every frame. One LG label map is a two-dimensional array with the same height and width as the RGB frame, with the bounding box areas replaced by their predicted class label. Next, we use intrinsic and extrinsic parameters to project the point cloud scene onto their respective LG label maps with top-k visibility for final class prediction.
We present an example output of our method in Figure 1.
Our contributions are following:

- •
We introduce a 2D object detection-based approach for open-vocabulary labeling of 3D instances, which greatly improves the efficiency compared to 2D segmentation approaches.
- •
We propose a novel approach to scoring 3D mask proposals using only bounding boxes from 2D object detectors.
- •
Our Open-YOLO 3D achieves superior performance on two benchmarks, while being considerably faster than existing methods in the literature. On ScanNet200 val. set, our Open-YOLO 3D achieves an absolute gain of 2.3% at mAP50 while being $\sim$16x faster compared to the recent Open3DIS [].

## 2 Related works

Closed-vocabulary 3D segmentation: The 3D instance segmentation task aims at predicting masks for individual objects in a 3D scene, along with a class label belonging to the set of known classes. Some methods use a grouping-based approach in a bottom-up manner, by learning embeddings in the latent space to facilitate clustering of object points [, , , , , , , ]. Conversely, proposal-based methods adopt a top-down strategy, initially detecting 3D bounding boxes and then segmenting the object region within each box [, , , , ]. Notably, inspired by advancements in 2D works [, ], transformer designs [] have been recently applied to 3D instance segmentation tasks [, , , , ]. Mask3D [] introduces the first hybrid architecture that combines Convolutional Neural Networks (CNN) and transformers for this task. It uses a 3D CNN backbone to extract per-point features and a transformer-based instance mask decoder to refine a set of queries.
Building on Mask3D, the authors of [] show that using explicit spatial and semantic supervision at the level of the 3D backbone further improves the instance segmentation results.
Oneformer3D [] follows a similar architecture and introduces learnable kernels in the transformer decoder for a unified semantic, instance, and panoptic segmentation.
ODIN [] proposes an architecture that uses 2D-3D fusion to generate the masks and class labels.
Other methods introduce weakly-supervised alternatives to dense annotation approaches, aiming to reduce the annotation cost associated with 3D data [, , ]. While these methodologies strive to enhance the quality of 3D instance segmentation, they typically rely on a predefined set of semantic labels. In contrast, our proposed approach aims at segmenting objects with both known and unknown class labels.

Open-vocabulary 2D recognition: This task aims at identifying both known and novel classes, where the labels of the known classes are available in the training set, while the novel classes are not encountered during training.
In the direction of open-vocabulary object detection (OVOD), several approaches have been proposed [, , , , , , , ].
Another widely studied task is open-vocabulary segmentation (OVSS) [, , , , ].
Recent open-vocabulary semantic segmentation methods [, , ] leverage pre-trained CLIP [] to perform open-vocabulary segmentation, where the model is trained to output a pixel-wise feature that is aligned with the text embedding in the CLIP space.
Furthermore, AttrSeg [] proposes a decomposition-aggregation framework where vanilla class names are first decomposed into various attribute descriptions, and then different attribute representations are aggregated into a final class representation.
Open-vocabulary instance segmentation (OVIS) aims at predicting instance masks while preserving high zero-shot capabilities.
One approach [] proposes a cross-modal pseudo-labeling framework, where a student model is supervised with pseudo-labels for the novel classes from a teacher model. Another approach [] proposes an annotation-free method where a pre-trained vision-language model is used to produce annotations at both the box and pixel levels.
Although these methods show high zero-shot performance and real-time speed, they are still limited to 2D applications only.

Open-vocabulary 3D segmentation: Several methods [, , ] have been proposed to address the challenges of open-vocabulary semantic segmentation where they use foundation models like clip for unknown class discovery, while the authors of [] focus on weak supervision for unknown class discovery without relying on any 2D foundation model. OpenScene [] makes use of 2D open-vocabulary semantic segmentation models to lift the pixel-wise 2D CLIP features into the 3D space, which allows the 3D model to perform 3D open-vocabulary point cloud semantic segmentation.
On the other hand, ConceptGraphs [] relies on creating an open-vocabulary scene graph that captures object properties such as spatial location, enabling a wide range of downstream tasks including segmentation, object grounding, navigation, manipulation, localization, and remapping.
In the direction of 3D point cloud instance segmentation,
OpenMask3D [] uses a 3D instance segmentation network to generate class-agnostic mask proposals, along with SAM [] and CLIP [], to construct a 3D clip feature for each mask using RGB-D images associated with the 3D scene.
Unlike OpenMask3D where a 3D proposal network is used, OVIR-3D [] generates 3D proposals by fusing 2D masks obtained by a 2D instance segmentation model. Open3DIS [] combines proposals from 2D and 3D with novel 2D masks fusion approaches via hierarchical agglomerative clustering, and also proposes to use point-wise 3D CLIP features instead of mask-wise features. The two most recent approaches in [, ] show promising generalizability in terms of novel class discovery [] and novel object geometries especially small objects []. However, they both suffer from slow inference speed, as they rely on SAM for 3D mask proposal clip feature aggregation in the case of OpenMask3D [], and for novel 3D proposal masks generation from 2D masks [].

Figure: Figure 2: Proposed open-world 3D instance segmentation pipeline. We use a 3D instance segmentation network (3D Network) for generating class-agnostic proposals. For open-vocabulary prediction, a 2D Open-Vocabulary Object Detector (2D OVOD) generates bounding boxes with class labels. These predictions are used to construct label maps for all input frames. Next, we assign the top-k label maps to each 3D proposal based on visibility. Finally, we generate a Multi-View Prompt Distribution from the 2D projections of the proposals to match a text prompt to every 3D proposal.
Refer to caption: x2.png

## 3 Preliminaries

Problem formulation: 3D instance segmentation aims at segmenting individual objects within a 3D scene and assigning one class label to each segmented object. In the open-vocabulary (OV) setting, the class label can belong to previously known classes in the training set as well as new class labels. To this end, let $P$ denote a 3D reconstructed point cloud scene, where a sequence of RGB-D images was used for the reconstruction. We denote the RGB image frames as $\mathcal{I}$ along with their corresponding depth frames D. Similar to recent methods [, , ], we assume that the poses and camera parameters are available for the input 3D scene.

### 3.1 Baseline Open-Vocabulary 3D Instance Segmentation

We base our approach on OpenMask3D [],
which is the first method that performs open-vocabulary 3D instance segmentation in a zero-shot manner. OpenMask3D has two main modules: a class-agnostic mask proposal head, and a mask-feature computation module. The class-agnostic mask proposal head uses a transformer-based pre-trained 3D instance segmentation model [] to predict a binary mask for each object in the point cloud. The mask-feature computation module first generates 2D segmentation masks by projecting 3D masks into views in which the 3D instances are highly visible, and refines them using the SAM [] model. A pre-trained CLIP vision-language model [] is then used to generate image embeddings for the 2D segmentation masks. The embeddings are then aggregated across all the 2D frames to generate a 3D mask-feature representation.

Limitations:
OpenMask3D makes use of the advancements in 2D segmentation (SAM) and vision-language models (CLIP) to generate and aggregate 2D feature representations, enabling the querying of instances according to open-vocabulary concepts. However, this approach suffers from a high computation burden leading to slow inference times, with a processing time of 5-10 minutes per scene. The computation burden mainly originates from two sub-tasks: the 2D segmentation of the large number of objects from the various 2D views, and the 3D feature aggregation based on the object visibility. We next introduce our proposed method which aims at reducing the computation burden and improving the task accuracy.

## 4 Method: Open-YOLO 3D

Motivation:
We here present our proposed 3D open-vocabulary instance segmentation method, Open-YOLO 3D, which aims at generating 3D instance predictions in an efficient strategy. Our proposed method introduces efficient and improved modules at the task level as well as the data level.
Task Level: Unlike OpenMask3D, which generates segmentations of the projected 3D masks, we pursue a more efficient approach by relying on 2D object detection. Since the end target is to generate labels for the 3D masks, the increased computation from the 2D segmentation task is not necessary. Data Level: OpenMask3D computes the 3D mask visibility in 2D frames by iteratively counting visible points for each mask across all frames. This approach is time-consuming, and we propose an alternative approach to compute the 3D mask visibility within all frames at once.

### 4.1 Overall Architecture

Our proposed pipeline is shown in Figure 2. First, we generate a set of instance proposals $M$ using a 3D instance segmentation network; the proposals are represented as binary masks, where every 3D mask has a dimension equal to the number of points as the input point cloud. For the open vocabulary prediction, we use a 2D open-vocabulary object detection model to generate a set of bounding boxes denoted Bi for every frame $\mathcal{I}_{i}$; the bounding boxes with their predicted labels are used to construct a low-granularity label map $\mathcal{L}_{i}$ for every input frame $\mathcal{I}_{i}$. To assign a prompt ID to the 3D mask proposals, we first start by projecting all $N$ points in the point cloud scene $P$ onto the $N_{f}$ frames, which results in $N_{f}$ 2D projection with $N$ points for each. Afterward, the 2D projections and the 3D mask proposals are used to compute the visibility of every mask in every frame using our proposed accelerated visibility computation (VAcc); the visibility is then used to assign top-k Low-Granularity label maps to each mask and to select the top-k 2D projections corresponding to every 3D mask proposal; for a single 3D mask we crop the (x, y) coordinates from the projections using the instance mask and filter out the points that are occluded or outside the frame. The final cropped (x, y) coordinates from the top-k frames are used to select per-point labels from their corresponding Low-Granularity label maps to finally construct a Multi-View Prompt Distribution to predict the prompt ID corresponding to the 3D mask proposal.

### 4.2 3D Object Proposal

To generate class-agnostic 3D object proposals, we rely on the 3D instance generation approach Mask3D [], which allows for faster proposal generation compared to 2D mask-based 3D proposal generation methods [, ]. Mask3D is a hybrid model that combines a 3D Convolutional Neural Network as a backbone for feature generation and a transformer-based model for mask instance prediction. The 3D CNN backbone takes the voxelized input point cloud scene as input, and outputs multi-level feature maps, while the transformer decoder takes the multi-level feature maps to refine a set of queries through self and cross-attention. The final refined queries are used to predict instance masks.
The 3D proposal network predicts a set $K_{3D}\in\mathbb{N}$ of 3D mask proposals $M\in\mathbb{Z}_{2}^{K_{3D}\times N}$ for a given point cloud $P\in\mathbb{R}^{4\times N}$ with $N$ points in homogeneous coordinate system, where $\mathbb{Z}_{2}=\{0,1\}$.

### 4.3 Low Granularity (LG) Label-Maps

As discussed earlier, the focus of our approach is to generate fast and accurate open-vocabulary labels for the generated 3D proposals. Instead of relying on computationally intensive 2D segmentation, we propose a 2D detection-based approach in our pipeline. For every RGB image $\mathcal{I}_{i}$ we generate a set of $K_{b,i}$ bounding boxes $\text{B}_{i}=\{(b_{j},c_{j})\hskip 8.5359pt|\hskip 8.5359ptb_{j}\in\mathbb{R}^
{4},\hskip 8.5359ptc_{j}\in\mathbb{N},\hskip 8.5359pt\forall j\in(1,...,K_{b,i
})\}$ using an open-vocabulary 2D object detector, where $b_{j}$ are the bounding boxes coordinates while $c_{j}$ is its predicted label. We assign a weight $w_{j}=b_{j}^{H}+b_{j}^{W}$ for each output bounding box $b_{j}$,
where $b_{j}^{H}$ and $b_{j}^{W}$ are the bounding box’s height and width, respectively. The weights represent the bounding box size and help determine the order of bounding boxes when used to construct the LG label maps.

After obtaining the 2D object detections, we represent the output of each 2D image frame $\mathcal{I}_{i}$ as an LG label map $\mathcal{L}_{i}\in\mathbb{Z}^{W\times H}$.
To construct $\mathcal{L}_{i}$, we start by initializing all of its elements as -1. In our notation, -1 represents no class and is ignored during prediction. Next, we sort all bounding boxes following their weights and replace the region of the bounding box $b_{j}$ with its corresponding predicted class label $c_{j}$ in the label map, starting with the bounding box with the highest weight. The weight $w_{j}$ choice is motivated by the fact that if two objects of different sizes appear in the same direction the camera is pointing to, the small object is visible in the image if it is closer to the camera than the large object.

### 4.4 Accelerated Visibility Computation ( VAcc )

In order to associate 2D label maps with 3D proposals, we compute the visibility of each 3D mask. To this end, we propose a fast approach that is able to compute 3D mask visibility within frames via tensor operations which are highly parallelizable.

Given $K_{f}\in\mathbb{N}$ 2D RGB frames $\mathcal{I}=\{\mathcal{I}_{i}\in\mathbb{R}^{3\times W\times H}\hskip 8.5359pt|
\hskip 8.5359pt\forall i\in(1,...,K_{f})\}$ associated with the 3D scene $P$, along with their intrinsic matrices $I\in\mathbb{R}^{N_{f}\times 4\times 4}$ and extrinsic matrices $E\in\mathbb{R}^{N_{f}\times 4\times 4}$. We denote the projection of the 3D point cloud $P$ on a frame with index $i$ $P_{i}^{2D}\in\mathbb{R}^{4\times N}$, and it can be computed as follows $P_{i}^{2D}=I_{i}\cdot E_{i}\cdot P$. After stacking the projection operations of all frames, the projection of the scene onto all frames can be computed with a GPU in a single shot as follows

$$ $P^{2D}=(I\star E)\cdot P$ $$

where $\star$ is batch-matrix multiplication, $\cdot$ is matrix multiplication, and $P^{2D}\in\mathbb{R}^{N_{f}\times 4\times N}$ defined as $P^{2D}=[P_{1}^{2D},...,P_{N_{f}}^{2D}]$.

Furthermore, we compute the visibility $V^{f}\in\mathbb{Z}_{2}^{N_{f}\times N}$ of all projected points within all frames as follows

$$ $V^{f}=\mathbbm{1}(0<P_{x}^{2D}<W)\odot\mathbbm{1}(0<P_{y}^{2D}<H)$ $$

where $\mathbbm{1}$ is the indicator function, $W$ and $H$ are the image width and height respectively , $\odot$ is element wise multiplication, $P_{x}^{2D}\in\mathbb{R}^{N_{f}\times N}$ and $P_{y}^{2D}\in\mathbb{R}^{N_{f}\times N}$ are the projected 3D points $x$ and $y$ coordinates on all $N_{f}$ frames, respectively.

For occlusion, we define another visibility matrix as $V^{d}\in\mathbb{Z}_{2}^{N_{f}\times N}$ that is computed as follows

$$ $V^{d}=\mathbbm{1}(|P_{z}^{2D}-D_{z}|<\tau_{depth})$ $$

where $D_{z}\in\mathbb{R}^{N_{f}\times N}$ is the real depth of the 3D point cloud obtained from the depth maps, while $P_{z}^{2D}=P^{2D}_{1\leq i\leq N_{f},j=3,1\leq k\leq N}$ is the depth obtained from projecting the point cloud $P$, and $|\cdot|$ is the absolute value.
Finally, the per 3D mask proposal visibility $\{V\in\mathbb{R}^{N_{f}\times K_{3D}},\hskip 8.5359pt0\leq V\leq 1\}$, can be computed in a single shot using a GPU as follows

$$ $V=\left((V^{f}\odot V^{d})\cdot M^{T}\right)\odot M_{count}^{-1}$ $$

where $M\in\mathbb{Z}_{2}^{K_{3D}\times N}$ is the matrix of 3D mask proposals generated by the 3D proposal network, and $M_{count}\in\mathbb{N}^{K_{3D}}$ is the count of points per mask,

The percentage of visibility of a mask $j$ in a frame $i$ is represented with the element $V_{i,j}\in V$.

Figure: Figure 3: Multi-View Prompt Distribution (MVPDist). After creating the LG label maps for all frames, we select the top-k label maps based on the 2D projection of the 3D proposal. Using the (x, y) coordinates of the 2D projection, we choose the labels from the LG label maps to generate the MVPDist. This distribution predicts the ID of the text prompt with the highest probability.
Refer to caption: x3.png

### 4.5 Multi-View Prompt Distribution ( MVPDist )

Given an RGB sequence $\mathcal{I}$
and their corresponding LG label-maps
$\mathcal{L}=\{\mathcal{L}_{i}\hskip 8.5359pt|\hskip 8.5359pt\mathcal{L}_{i}\in
\mathbb{Z}^{W\times H},\hskip 8.5359pt\forall i\in(1,...,K_{f})\}$. We define the label distribution $\mathcal{D}_{j}\in\mathbb{Z}^{N^{j}_{dist}}$ of a 3D mask proposal $M_{j}$ as

$$ $\mathcal{D}_{j}=\{\mathcal{L}_{i}\left[P_{i,x}^{2D}\cdot M_{ji},P_{i,y}^{2D} \cdot M_{ji}\right]\hskip 8.5359pt|\hskip 8.5359pt\forall i\in\mathcal{P}_{k}\}$ $$

where $M_{ji}=V_{i}^{d}\cdot V_{i}^{f}\cdot M_{j}$ is the mask for non-occluded in-frame points that belong to the $j^{th}$ instance, $\mathcal{P}_{k}$ is the set of frame indices where the $j^{th}$ 3D mask has top-k visibility, computed using the visibility matrix $V$, $P_{i,x}^{2D}$ and $P_{i,y}^{2D}$ are the projected x and y coordinates of the point cloud scene onto the $i_{th}$ frame, respectively, while $[\cdot,\cdot]:\mathbb{Z}^{W\times H}\mapsto\mathbb{Z}^{n}$ is a coordinate-based selection operator with $n$ as an arbitrary natural number.

We define the probability for the mask $M_{j}$ to be assigned to a class $c$ as the occurrence of class $c$ within the distribution $\mathcal{D}_{j}$; we demonstrate the approach in Figure 3 for one 3D proposal. We assign the class with the highest probability to the mask $M_{j}$.

### 4.6 Instance Prediction Confidence Score

We propose in this section a way to score class-agnostic 3D instance mask proposals by leveraging 2D information from bounding boxes along with 3D information from 3D masks. Our rationale is that a good mask should lead to a high class and mask confidence, unlike previous Open-Vocabulary 3D instance segmentation methods, which use only class confidence. To achieve this, we define the score of a 3D mask proposal as $s_{m}=s_{IoU}\cdot s_{class}$,
where $s_{class}$ is the probability of the class with the highest occurrence in MVPDist, while $s_{IoU}$ is the average IoU across multi-views between the bounding box of the projected 3D mask and the bounding box with the highest IoU generated with a 2D object detector in the view (More details are in the appendix).

## 5 Experiments

Datasets: We conduct our experiments using the ScanNet200 [] and Replica [] datasets. Our analysis on ScanNet200 is based on its validation set, comprising 312 scenes. For the 3D instance segmentation task, we utilize the 200 predefined categories from the ScanNet200 annotations. ScanNet200 labels are categorized into three subsets—head (66 categories), common (68 categories), and tail (66 categories)—based on the frequency of labeled points in the training set. This categorization allows us to evaluate our method’s performance across the long-tail distribution, underscoring ScanNet200 as a suitable evaluation dataset. Additionally, to assess the generalizability of our approach, we conduct experiments on the Replica dataset, which has 48 categories.
For the metrics,
we follow the evaluation methodology in ScanNet [] and report the average precision (AP) at two mask overlap thresholds: 50% and 25%, as well as the average across the overlap range of [0.5:0.95:0.05].

Implementation details: We use RGB-depth pairs from the ScanNet200 and Replica datasets, processing every 10th frame for ScanNet200 and all frames for Replica, maintaining the same settings as OpenMask3D for fair comparison. To create LG label maps, we use the YOLO-World [] extra-large model for its real-time capability and high zero-shot performance. We use Mask3D [] with non-maximum suppression to filter proposals similar to Open3DIS [], and avoid DBSCAN [] to prevent inference slowdowns. We use a single NVIDIA A100 40GB GPU for all experiments.

Table: Table 1: State-of-the-art comparison on ScanNet200 validation set. We use Mask3D trained on the ScanNet200 training set to generate class-agnostic mask proposals. Our method demonstrates better performance compared to those that generate 3D proposals by fusing 2D masks and proposals from a 3D network (highlighted in gray in the table). It outperforms state-of-the-art methods by a wide margin under the same conditions using only proposals from a 3D network.

Table: Table 2: State-of-the-art comparison on Replica dataset. We use Mask3D trained on the ScanNet200 training set to generate class-agnostic mask proposals. We show that our method generalizes better than state-of-the-art methods under the same setting with proposals from 3D networks only.

Figure: Figure 4: Qualitative results on scene office0 in the Replica dataset. We show instances with a confidence score above 0.5 for both methods. We show that our method is much more precise when segmenting the object in the text compared to state-of-the-art method Open3DIS.
Refer to caption: x4.png

### 5.1 Results analysis

Table: Table 3: Ablation study on Replica dataset with ground truth masks. We show the effect of replacing SAM with a 2D object detector in rows R2 to R4 while we demonstrate the effect of replacing CLIP features with MVPDist with label maps from an OV 2D object detector.

Open-Vocabulary 3D instance segmentation on ScanNet200:
We compare our method’s performance against other approaches on the ScanNet200 dataset in Table 1. We indicate whether each method uses 2D instances to generate 3D proposals and whether SAM is used for labeling the 3D masks. Our method achieves state-of-the-art performance with proposals from only a 3D instance segmentation network compared to methods from two settings (i) 3D mask proposals from only a 3D instance segmentation network (ii) using a combination of 3D mask proposals from a 3D network and 2D instances from a 2D segmentation model. Additionally, our method is $\sim$ 16$\times$ faster compared to state-of-the-art Open3DIS.

Table: Table 4: Effect of number of top-k views on the labeling performance with High Granularity (HG) and Low-Granularity (LG) label maps. HG Label maps are generated by prompting SAM with the bounding boxes used to generate LG label maps.

Generalizability test:
To test the generalizability of our method, we use the 3D proposal network pre-trained on the ScanNet200 training set and evaluate on the Replica dataset; the results are shown in Table 2. Our method shows competitive performance with $\sim$ 11 $\times$ speedup against state-of-the-art models [], which use CLIP features.

Table: Table 5: Comparative results with oracle masks on ScanNet200 dataset. Our MVPDist significantly outperforms state-of-the-art methods using CLIP features. We show that MVPDist with YOLO-WORLD predictions can use multi-view information to achieve high zero-shot capabilities.

Performance with given 3D masks: We further test our method for 3D proposal prompting from text against existing methods in the literature in the case of ground truth 3D masks and report the results in Table 5. For Mask3D, oracle masks are assigned class predictions from matched predicted masks using Hungarian matching. For open-vocabulary methods, we use ground-truth masks as input proposals. We show the results in Table 5 that MVPDist can outperform CLIP-based approaches in retrieving the correct 3D proposal masks from text prompts, due to the high zero-shot performance achieved by state-of-the-art open-vocabulary 2D object detectors.

Ablation over Replica dataset: To test the ability of object detectors to replace SAM for generating crops for 3D CLIP feature aggregation, we conduct three experiments with different object detectors on the Replica dataset, using ground truth 3D mask proposals to compare our method’s labeling ability against others. The results are in Table 3, rows R2 to R4, with row R1 showing OpenMask3D [] base code results. We generate class-agnostic bounding boxes with an object detector, then assign the highest IoU bounding box to each 3D instance as a crop, selecting the most visible views. For 3D CLIP feature aggregation, we follow OpenMask3D’s approach, aggregating features from multiple levels and views. These experiments show that YOLO-World can generate crops nearly as good as SAM but with significant speed improvements. Row R5 demonstrates YOLO-World’s better zero-shot performance using our proposed Multi-View Prompt Distribution with LG label maps. Row R6 shows speed improvements using the GPU for 3D mask visibility computation.

Top K analysis: We show in Table 4 that naively using YOLO-World with only one label-map with the highest visibility per 3D mask proposal results in sub-optimal results and using top-K label-maps can result in better predictions as the distribution can provide better estimate across multiple frames, since YOLO-World is also expected to make misclassifications in some views while generating correct ones in others. This approach assumes that YOLO-World makes a correct class prediction for the same 3D object in multiple views for it to be effective.

High-Granularity (HG) vs. Low-Granularity (LG):
Table 4 shows that using SAM to generate HG label maps slightly reduces mAP, and slows down the inference by $\sim$ 5 times. This is due to the nature of projected 3D instances into 2D, where the projection already holds 2D instance information as shown in Figure 3, and SAM would just result in redundancy in the prediction.

Qualitative results: We show qualitative results on the Replica dataset in Figure 4 and compare it to Open3DIS.
Open3DIS shows good performance in recalling novel geometries that 3D proposal networks like Mask3D generally fail to capture (small-sized objects). However, it comes at the cost of very low precision due to the redundant masks with different class predictions.

Limitations: Our method makes use of a 3D proposal network only for proposal generation in order to reach high speed. Other proposal generation methods [, ] fuse 2D instance masks from a 2D instance segmentation methods to generate rich 3D proposals even for very small objects, which are generally overlooked by 3D proposal networks like Mask3D [] due to low resolution in 3D. Thus, fast 2D instance segmentation models like FastSAM [] can be used to generate 3D proposals from the 2D images, which might further improve the performance of our method.

## 6 Conclusion

We present Open-YOLO 3D, a novel and efficient open-vocabulary 3D instance segmentation method, which makes use of open-vocabulary 2D object detectors instead of heavy segmentation models. Our approach leverages a 2D object detector for class-labeled bounding boxes and a 3D instance segmentation network for class-agnostic masks.
We propose to use MVPDist generated from multi-view low granularity label maps to match text prompts to 3D class agnostic masks.
Our proposed method outperforms existing techniques, with gains in mAP and inference speed.
These results show a new direction toward more efficient open-vocabulary 3D instance segmentation models.

## Appendix A Appendix

Figure: Figure 5: Additional details on how IoU score $s_{IoU}$ is computed. We show that our method can provide a reliable mask score using Intersection Over Union (IoU) between the bounding boxes estimated using the 3D cropped instance 2D projection and the bounding boxes from a 2D object detector. We also demonstrate that it covers all three cases of different 3D mask proposals.
Refer to caption: x5.png

Figure: Figure 6: Additional details on how the High-Granularity label maps (HG label maps) are constructed. To get a detailed per-pixel label for the label maps, we prompt SAM with bounding boxes to get pixel-wise masks, and then we follow the same approach as in Section 4.3 to construct the label maps. The multi-View Prompt Distribution is estimated in the same way as in Section 4.5
Refer to caption: x6.png

Figure: Figure 7: Time breakdown of the average time for Open-Vocabulary 3D instance segmentation over the replica dataset. We report the time for each step that makes the whole Open-Vocabulary 3D instance segmentation pipeline for our method and the two most recent methods, OpenMask3D and Open3DIS. 3D proposal generation is the step of generating class agnostic mask proposals for the objects in the 3D point cloud scene 2D projection & Visibility computation is the step of projection of the point cloud scene onto the 2D frames, and computing the visibility of each mask proposal in the 2D frames. Text prompting with 2D models is the step of inference from the 2D models to prompt the 3D class agnostic masks with text.
Refer to caption: extracted/5680410/Figures/time_breakdown.png

Figure: Figure 8: Comparison between MVPDist with HG label maps and with LG label maps. We show that using HG label maps instead of LG label maps slightly increases the prediction class score but results in the same prediction. Thus, LG can give a similar performance with around 5 times speed up in terms of inference time.
Refer to caption: x7.png

Figure: Figure 9: Comparison between our method and Open3DIS. We show that our method is more precise than Open3DIS.
Refer to caption: x8.png

Figure: Figure 10: Comparison of prediction with the highest score between our method and Open3DIS on replica dataset.
Refer to caption: x9.png

Figure: Figure 11: Scoring quality comparison between ours and Open3DIS. (Top) We show on the left the two masks with top confidence scores $s_{m}$ computed as in section 4.6, notice that the second mask in blue has a portion that extends to another instance, which makes it worse in than the red mask. Our scoring technique captures this detail while scoring the mask. (Bottom) We show on the left the two masks with the highest scores by Open3DIS. Notice that using only semantic scores results in wrong mask confidence since it is aggregated from 2D clip features, where points with the most visibility across views contribute more than others to the final aggregated 3D clip features. Thus, this type of scoring assigns very high scores to bad masks that have points with more visibility.
Refer to caption: x10.png

Figure: Figure 12: Additional qualitative results on ScanNet200. with text prompt "Chair".
Refer to caption: x11.png

Figure: Figure 13: Additional qualitative results on ScanNet200. with text prompt "Table".
Refer to caption: x12.png

Figure: Figure 14: Additional qualitative results on ScanNet200. with text prompt "Television".
Refer to caption: x13.png

Figure: Figure 15: Additional qualitative results on ScanNet200. with text prompt "Fridge".
Refer to caption: x14.png

Figure: Figure 16: Additional qualitative results on ScanNet200. with text prompt "Snacks".
Refer to caption: x15.png

Figure: Figure 17: Additional qualitative results on ScanNet200. with text prompt "Computer".
Refer to caption: x16.png

Figure: Figure 18: Additional qualitative results on ScanNet200. with text prompt "Desk".
Refer to caption: x17.png

Figure: Figure 19: Additional qualitative results on ScanNet200. with text prompt "Bed".
Refer to caption: x18.png

Figure: Figure 20: Additional qualitative results on ScanNet200. with text prompt "Shoes".
Refer to caption: x19.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In Table 1, does the class-agnostic Mask3D model have access to mask annotations for the same classes as those in the validation set? Do the other methods use the same class-agnostic segmentation model?

2. Since the proposed approach relies on a class-agnostic 3D instance generation model, what are the advantages of using only mask annotations, rather than both instance and label annotations, for training? I mean, are there practical scenarios where only mask annotations are available?

3. What does the tag "(Closed Vocab)" mean in Table 1? Does it indicate that the Mask3D method uses both mask annotations and object class annotations for training?

4. What is the performance of Mask3D (Closed Vocab.) on the Replica dataset?

Minor：

1. In line 092, a comma is missing after "multi-view information".
soundness: 3
strengths: 1. The paper is well-organized, and the ideas are clearly illustrated.
2. This paper introduces a novel approach for efficient open-vocabulary 3D instance labeling by leveraging 2D bounding box priors from a fast 2D object detector, demonstrating superior performance and time efficiency in experimental results.
3. A fast visibility computation algorithm (VAcc) is proposed to accelerate the process of associating 2D label maps with 3D proposals that may be occluded in some views.  This algorithm demonstrates both efficiency and robustness to variations in label map granularity.
summary: This paper introduces an efficient 3D mask labeling method that leverages multi-view 2D label maps, referred to as Low Granularity (LG) Label Maps, created from 2D object bounding boxes to label 3D instances. The 3D instance (mask) proposals are generated using a pre-trained class-agnostic 3D segmentation method. To address object occlusion across different viewpoints, an Accelerated Visibility Computation (VACC) method is introduced, enabling rapid calculation of visibility matrices using intrinsic and extrinsic parameters.
weaknesses: 1. The foundation of the proposed method is built upon the class-agnostic 3D segmentation model, Mask3D, which is used to generate 3D mask proposals. However, this paper lacks sufficient evidence to demonstrate Mask3D's effectiveness and generalizability for open vocabulary instance proposals.
2. The experimental evaluation of the proposed method for open-vocabulary 3D instance segmentation is relatively limited (only Table 6).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness.
soundness: 2
strengths: 1. **[Efficiency]** The experimental results suggest that the proposed method achieves high precision with a significantly better speed compared with most methods, and efficiency is important for practical deployment.

2. **[Clearness]** This paper explains its main contribution, how to assign class predictions to 3D proposals, with great clarity. The implementation details are elaborated sufficiently.
summary: This paper primarily aims to achieve faster open-vocabulary 3D instance segmentation compared with existing methods like OpenMask3D. To realize this target, this work first uses a 3D instance segmentation network to generate segmentation proposals. Then, the output of an open-vocabulary 2D object detector as well as some designed 3D information is employed to derive the categories of these proposals.
weaknesses: 1. **[Insufficient Academic Contributions]**: This work just combines the output of a 3D segmentation network and a well-implemented open-vocabulary 2D object detector to realize open-vocabulary 3D object detection (similar to existing open-world segmentation method, just with a replacement of the post network to 2D object detector), which is trivial. It is much faster than previous methods because previous methods are developed based on models like SAM and CLIP. This work employs more efficient and suitable existing models. Therefore, although this work is sound in terms of engineering, its real academic contribution and new insights are plain.

2. **[Insufficient Ablation Study]** As the method is efficient because it makes good use of existing models, it is important to clearly analyze how these models contribute to the efficiency, which will guide future works on how to develop an efficient open-vocabulary pipeline. However, this work fails to do so.

3. **[Misleading title]** The method name OPEN-YOLO 3D seems to be unsuitable. YOLO is a 2D object detector while the task is about 3D point cloud segmentation. Although the method utilizes the output of YOLO-World to generate class predictions, the method name is still a little misleading.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Could you provide a more detailed theoretical analysis or complexity comparison of VAcc with the conventional method?
How does the method perform when the 2D object detector encounters difficult conditions, such as poor lighting or significant occlusion, if there's any evaluation of the robustness under such conditions?
What are the potential strategies for mitigating errors from misclassifications made by the 2D object detector, and how do they affect the 3D mask assignments?
I'm also particularly interested in the discussion in your limitation section, I was wondering would integrating fast 2D segmentation models, as mentioned, be feasible within your current framework? How might this affect both performance and speed?
soundness: 4
strengths: This paper proposed a novel framework, which uses 2D object detection for 3D instance segmentation. The model they presented reduced computational overhead significantly. The Accelerated Visibility Computation (VAcc) leverages tensor operations and GPU batch processing, enabling highly parallelized visibility computation. This contributes to the following speed improvements without compromising performance. By integrating a high-performing 2D open-vocabulary detector, the framework retains strong zero-shot performance, which is important for real-world applications that use new or unknown object types.

It also includes detailed experiments that showcase Open-YOLO 3D's speech and accuracy, and highlight its performance above state-of-the-art approaches like Open3DIS and OpenMask3D. The paper also includes comprehensive ablation studies to demonstrate the improvement of each component. 

The overall writing is clear and the framework will be beneficial for related research.
summary: This paper proposes Open-YOLO 3D, which is an open-vocabulary 3D instance segmentation framework that efficiently combines 2D object detection and 3D mask generation. The key idea of this paper is its reliance on bounding box predictions from a 2D open-vocabulary object detector and the subsequent use of these predictions for efficient 3D mask proposal and labeling. Unlike prior methods that use computationally intensive models such as SAM and CLIP for feature lifting from 2D to 3D, this paper uses a novel Multi-View Prompt Distribution (MVPDist) and Accelerated Visibility Computation (VAcc) methods to speed up the segmentation process. The framework this paper proposed achieves up to 16x faster inference while keeping competitive or better accuracy.
weaknesses: I like the overall framework this paper presents and appreciate its contribution to 3D instance segmentation by introducing an inference-efficient model, but I still have some concerns about it:

While the paper mentions that VAcc uses tensor operations, a deeper explanation or complexity analysis comparing it to conventional iterative methods would strengthen the understanding of its true computational advantage, and the reason why it can achieve faster inference speed. I believe the paper clearly demonstrates the operation of this proposed algorithm, however, more explanation about why it is efficient and how much computation cost it saves will better demonstrate the paper's contribution.

The method relies on the quality of the 2D object detector, and this might be an issue if the 2D views are suboptimal (for example poor lighting, and occlusions). A more extensive analysis or discussion on how 2D detection failures propagate through the pipeline would add value.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Please see the weakness.
soundness: 3
strengths: 1.	The authors only utilize the bounding boxes from 2D object detectors to alleviate the redundancy brought by 2D masks, which demonstrates a significant improvement in inference speed compared to OpenMask3D.
2.	The authors propose a Multi-View Prompt Distribution to obtain reliable category labels form 3D masks, the experimental results evaluation on the ScanNet200 and Replica datasets prove the efficiency of the method proposed by the authors.
3.	The paper is well-structured, and the connection between the proposed method and the motivation is coincident.
summary: The authors propose an efficient method for open-vocabulary 3D instance segmentation to enhance the real-time capability. Unlike existing methods that rely on obtaining 2D masks and category labels from 2D foundation models (like SAM and CLIP), the authors propose a novel approach, Open-YOLO3D, which only leverages bounding boxes generated by 2D object detectors. Moreover, the authors propose a Multi-View Prompt Distribution (MVPDist) method to endeavor promising performance in recognition. The experimental results demonstrate the promising real-time performance of the method proposed by the authors.
weaknesses: 1.	The improvements of the segmentation performance observed in Open-YOLO3D primarily arise from the enhanced category recognition, which is likely from the prior knowledge of the pre-trained YoloWorld model.
2.	The challenges inherent in Open-YOLO3D closely resemble those faced by Open3DIS, as both methods rely on pre-trained models for generating 3D proposals. As discussed in Open3DIS, the pre-trained 3D models have limited capabilities when it comes to detecting uncommon categories. The representation of 3D data for open vocabulary instance segmentation might be uncultivated and limited.
3.	Recent studies [1] have indicated that OpenMask3D performs poorly on certain outdoor datasets, such as NuScenes. Does Open-YOLO3D face similar challenges in effectively identifying sparse-diverse and less common categories in outdoor environments?

[1] Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understanding.

---

confidence: 5
contribution: 3
initial_rating: 10
presentation: 4
questions: Please draw my concerns listed in the Weaknesses part.
soundness: 3
strengths: 1.	The proposed method is simple yet effective.
2.	The two proposed designs are helpful for 3D instance segmentation with meaningful designs.
3.	The experimental results show that the proposed method could achieve good performance while remaining very efficient.
summary: This paper aims to deal with the open-vocabulary 3D instance segmentation task with a fast and cost-effective approach by utilizing a YOLO-style design. A Multi-View Prompt Distribution method is proposed to effectively fuse the multi-view information. The low granularity label maps are proposed to only use 2D detectors to predict prompt IDs for 3D instance masks. Experimental results demonstrate the state-of-the-art performance of the proposed method. The speed of the proposed method is about 16 times faster than that of existing methods.
weaknesses: 1.	Is it possible to extend the proposed method on panoptic segmentation of 3D scenarios? Please present your design briefly for this. 
2.	As shown in Table 1, the inference time of the proposed method is 21.8, which is slower than OpenScene (3D Distill). Please add the explanation for this phenomenon in the corresponding text (first paragraph of Section 5.1).
3.	In Line 405, it should be 4.29 but not 04.29 for OpenScene (3D Distill).

Meta-review: This paper presents an efficient approach to open-vocabulary 3D instance segmentation by leveraging 2D bounding box priors from a pre-trained open-vocabulary 2D object detector. The authors propose the Multi-View Prompt Distribution (MVPDist) method, which effectively utilizes multi-view information while addressing potential misclassification from the 2D object detector to generate reliable 3D instance masks. The experimental results demonstrate the promising real-time performance of the method.

Initially, the reviewers raised several concerns, including:

- The generalizability of Mask3D (qcNj)
- Insufficient evaluation (qcNj, LA4L)
- The engineering focus of the work with insufficient academic contribution (LA4L)
- Lack of explanation and complexity analysis of VAcc (4bgM)
- The robustness of the 2D object detector in extreme conditions (4bgM)
- Performance improvements primarily attributed to the pre-trained YoloWorld model (C5qy)
- Limited capabilities on uncommon categories (C5qy)
- Performance on outdoor datasets (C5qy)
- Feasibility of extending the method to 3D panoptic segmentation (rQL1)
- Slower inference times (rQL1)

The authors provided detailed responses to these concerns, and after the rebuttal, four reviewers (4bgM, rQL1, C5qy, qcNj) voted to accept the paper, while reviewer LA4L did not update their initial rating, which was borderline rejection. The AC noted that reviewer LA4L did not follow up during the author response period.

Upon reviewing the authors’ rebuttal to reviewer LA4L’s comments, the AC finds the authors’ responses to the second and third concerns - regarding the engineering focus and the academic contribution - particularly convincing. The first concern, regarding the academic contribution, is more subjective, but the AC agrees that the proposed Open-YOLO 3D method is effective for the open-vocabulary 3D instance segmentation task.

Given the resolution of most concerns and the strong experimental results, the AC recommends accepting this paper and believes it is appropriate to let the community assess its academic contribution.

Decision: Accept (Oral)