Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation

Authors: Mohamed El Amine Boudjoghra, Angela Dai, Jean Lahoud, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Fahad Shahbaz Khan
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Open Vocabulary, 3D point cloud instance segmentation

Abstract:
Recent works on open-vocabulary 3D instance segmentation show strong promise but at the cost of slow inference speed and high computation requirements. This high computation cost is typically due to their heavy reliance on aggregated clip features from multi-view, which require computationally expensive 2D foundation models like Segment Anything (SAM) and CLIP. Consequently, this hampers their applicability in many real-world applications that require both fast and accurate predictions. To this end, we propose a novel open-vocabulary 3D instance segmentation approach, named Open-YOLO 3D, that efficiently leverages only 2D object detection from multi-view RGB images for open-vocabulary 3D instance segmentation. 
 We demonstrate that our proposed Multi-View Prompt Distribution (MVPDist) method makes use of multi-view information to account for misclassification from the object detector to predict a reliable label for 3D instance masks. Furthermore, since projections of 3D object instances are already contained within the 2D bounding boxes, we show that our proposed low granularity label maps, which require only a 2D object detector to construct, are sufficient and very fast to predict prompt IDs for 3D instance masks when used with our proposed MVPDist.
 We validate our Open-YOLO 3D on two benchmarks, ScanNet200 and Replica, 
 under two scenarios: (i) with ground truth masks, where labels are required for given object proposals, and (ii) with class-agnostic 3D proposals generated from a 3D proposal network.
 Our Open-YOLO 3D achieves state-of-the-art performance on both datasets while obtaining up to $\sim$16$\times$ speedup compared to the best existing method in literature. On ScanNet200 val. set, our Open-YOLO 3D achieves mean average precision (mAP) of 24.7% while operating at 22 seconds per scene. github.com/aminebdj/OpenYOLO3D

Decision: accept (oral)
Reviewer scores: 8, 5, 8, 8, 10

Reviews and discussion:
<thread>
creation date:
1730538506653

thread_type:
official_review

<thread_object>
creator: reviewer_rql1

<content>
summary:
This paper aims to deal with the open-vocabulary 3D instance segmentation task with a fast and cost-effective approach by utilizing a YOLO-style design. A Multi-View Prompt Distribution method is proposed to effectively fuse the multi-view information. The low granularity label maps are proposed to only use 2D detectors to predict prompt IDs for 3D instance masks. Experimental results demonstrate the state-of-the-art performance of the proposed method. The speed of the proposed method is about 16 times faster than that of existing methods.

strengths:
1.	The proposed method is simple yet effective.
2.	The two proposed designs are helpful for 3D instance segmentation with meaningful designs.
3.	The experimental results show that the proposed method could achieve good performance while remaining very efficient.

weaknesses:
1.	Is it possible to extend the proposed method on panoptic segmentation of 3D scenarios? Please present your design briefly for this. 
2.	As shown in Table 1, the inference time of the proposed method is 21.8, which is slower than OpenScene (3D Distill). Please add the explanation for this phenomenon in the corresponding text (first paragraph of Section 5.1).
3.	In Line 405, it should be 4.29 but not 04.29 for OpenScene (3D Distill).

questions:
Please draw my concerns listed in the Weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730689635430

thread_type:
official_review

<thread_object>
creator: reviewer_c5qy

<content>
summary:
The authors propose an efficient method for open-vocabulary 3D instance segmentation to enhance the real-time capability. Unlike existing methods that rely on obtaining 2D masks and category labels from 2D foundation models (like SAM and CLIP), the authors propose a novel approach, Open-YOLO3D, which only leverages bounding boxes generated by 2D object detectors. Moreover, the authors propose a Multi-View Prompt Distribution (MVPDist) method to endeavor promising performance in recognition. The experimental results demonstrate the promising real-time performance of the method proposed by the authors.

strengths:
1.	The authors only utilize the bounding boxes from 2D object detectors to alleviate the redundancy brought by 2D masks, which demonstrates a significant improvement in inference speed compared to OpenMask3D.
2.	The authors propose a Multi-View Prompt Distribution to obtain reliable category labels form 3D masks, the experimental results evaluation on the ScanNet200 and Replica datasets prove the efficiency of the method proposed by the authors.
3.	The paper is well-structured, and the connection between the proposed method and the motivation is coincident.

weaknesses:
1.	The improvements of the segmentation performance observed in Open-YOLO3D primarily arise from the enhanced category recognition, which is likely from the prior knowledge of the pre-trained YoloWorld model.
2.	The challenges inherent in Open-YOLO3D closely resemble those faced by Open3DIS, as both methods rely on pre-trained models for generating 3D proposals. As discussed in Open3DIS, the pre-trained 3D models have limited capabilities when it comes to detecting uncommon categories. The representation of 3D data for open vocabulary instance segmentation might be uncultivated and limited.
3.	Recent studies [1] have indicated that OpenMask3D performs poorly on certain outdoor datasets, such as NuScenes. Does Open-YOLO3D face similar challenges in effectively identifying sparse-diverse and less common categories in outdoor environments?

[1] Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understanding.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693686544

thread_type:
official_review

<thread_object>
creator: reviewer_4bgm

<content>
summary:
This paper proposes Open-YOLO 3D, which is an open-vocabulary 3D instance segmentation framework that efficiently combines 2D object detection and 3D mask generation. The key idea of this paper is its reliance on bounding box predictions from a 2D open-vocabulary object detector and the subsequent use of these predictions for efficient 3D mask proposal and labeling. Unlike prior methods that use computationally intensive models such as SAM and CLIP for feature lifting from 2D to 3D, this paper uses a novel Multi-View Prompt Distribution (MVPDist) and Accelerated Visibility Computation (VAcc) methods to speed up the segmentation process. The framework this paper proposed achieves up to 16x faster inference while keeping competitive or better accuracy.

strengths:
This paper proposed a novel framework, which uses 2D object detection for 3D instance segmentation. The model they presented reduced computational overhead significantly. The Accelerated Visibility Computation (VAcc) leverages tensor operations and GPU batch processing, enabling highly parallelized visibility computation. This contributes to the following speed improvements without compromising performance. By integrating a high-performing 2D open-vocabulary detector, the framework retains strong zero-shot performance, which is important for real-world applications that use new or unknown object types.

It also includes detailed experiments that showcase Open-YOLO 3D's speech and accuracy, and highlight its performance above state-of-the-art approaches like Open3DIS and OpenMask3D. The paper also includes comprehensive ablation studies to demonstrate the improvement of each component. 

The overall writing is clear and the framework will be beneficial for related research.

weaknesses:
I like the overall framework this paper presents and appreciate its contribution to 3D instance segmentation by introducing an inference-efficient model, but I still have some concerns about it:

While the paper mentions that VAcc uses tensor operations, a deeper explanation or complexity analysis comparing it to conventional iterative methods would strengthen the understanding of its true computational advantage, and the reason why it can achieve faster inference speed. I believe the paper clearly demonstrates the operation of this proposed algorithm, however, more explanation about why it is efficient and how much computation cost it saves will better demonstrate the paper's contribution.

The method relies on the quality of the 2D object detector, and this might be an issue if the 2D views are suboptimal (for example poor lighting, and occlusions). A more extensive analysis or discussion on how 2D detection failures propagate through the pipeline would add value.

questions:
Could you provide a more detailed theoretical analysis or complexity comparison of VAcc with the conventional method?
How does the method perform when the 2D object detector encounters difficult conditions, such as poor lighting or significant occlusion, if there's any evaluation of the robustness under such conditions?
What are the potential strategies for mitigating errors from misclassifications made by the 2D object detector, and how do they affect the 3D mask assignments?
I'm also particularly interested in the discussion in your limitation section, I was wondering would integrating fast 2D segmentation models, as mentioned, be feasible within your current framework? How might this affect both performance and speed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730865783859

thread_type:
official_review

<thread_object>
creator: reviewer_la4l

<content>
summary:
This paper primarily aims to achieve faster open-vocabulary 3D instance segmentation compared with existing methods like OpenMask3D. To realize this target, this work first uses a 3D instance segmentation network to generate segmentation proposals. Then, the output of an open-vocabulary 2D object detector as well as some designed 3D information is employed to derive the categories of these proposals.

strengths:
1. **[Efficiency]** The experimental results suggest that the proposed method achieves high precision with a significantly better speed compared with most methods, and efficiency is important for practical deployment.

2. **[Clearness]** This paper explains its main contribution, how to assign class predictions to 3D proposals, with great clarity. The implementation details are elaborated sufficiently.

weaknesses:
1. **[Insufficient Academic Contributions]**: This work just combines the output of a 3D segmentation network and a well-implemented open-vocabulary 2D object detector to realize open-vocabulary 3D object detection (similar to existing open-world segmentation method, just with a replacement of the post network to 2D object detector), which is trivial. It is much faster than previous methods because previous methods are developed based on models like SAM and CLIP. This work employs more efficient and suitable existing models. Therefore, although this work is sound in terms of engineering, its real academic contribution and new insights are plain.

2. **[Insufficient Ablation Study]** As the method is efficient because it makes good use of existing models, it is important to clearly analyze how these models contribute to the efficiency, which will guide future works on how to develop an efficient open-vocabulary pipeline. However, this work fails to do so.

3. **[Misleading title]** The method name OPEN-YOLO 3D seems to be unsuitable. YOLO is a 2D object detector while the task is about 3D point cloud segmentation. Although the method utilizes the output of YOLO-World to generate class predictions, the method name is still a little misleading.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731077146514

thread_type:
official_review

<thread_object>
creator: reviewer_qcnj

<content>
summary:
This paper introduces an efficient 3D mask labeling method that leverages multi-view 2D label maps, referred to as Low Granularity (LG) Label Maps, created from 2D object bounding boxes to label 3D instances. The 3D instance (mask) proposals are generated using a pre-trained class-agnostic 3D segmentation method. To address object occlusion across different viewpoints, an Accelerated Visibility Computation (VACC) method is introduced, enabling rapid calculation of visibility matrices using intrinsic and extrinsic parameters.

strengths:
1. The paper is well-organized, and the ideas are clearly illustrated.
2. This paper introduces a novel approach for efficient open-vocabulary 3D instance labeling by leveraging 2D bounding box priors from a fast 2D object detector, demonstrating superior performance and time efficiency in experimental results.
3. A fast visibility computation algorithm (VAcc) is proposed to accelerate the process of associating 2D label maps with 3D proposals that may be occluded in some views.  This algorithm demonstrates both efficiency and robustness to variations in label map granularity.

weaknesses:
1. The foundation of the proposed method is built upon the class-agnostic 3D segmentation model, Mask3D, which is used to generate 3D mask proposals. However, this paper lacks sufficient evidence to demonstrate Mask3D's effectiveness and generalizability for open vocabulary instance proposals.
2. The experimental evaluation of the proposed method for open-vocabulary 3D instance segmentation is relatively limited (only Table 6).

questions:
1. In Table 1, does the class-agnostic Mask3D model have access to mask annotations for the same classes as those in the validation set? Do the other methods use the same class-agnostic segmentation model?

2. Since the proposed approach relies on a class-agnostic 3D instance generation model, what are the advantages of using only mask annotations, rather than both instance and label annotations, for training? I mean, are there practical scenarios where only mask annotations are available?

3. What does the tag "(Closed Vocab)" mean in Table 1? Does it indicate that the Mask3D method uses both mask annotations and object class annotations for training?

4. What is the performance of Mask3D (Closed Vocab.) on the Replica dataset?

Minor：

1. In line 092, a comma is missing after "multi-view information".
</content>
</thread_object>

</thread>
<thread>
creation date:
1732133607386

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all the reviewers (qcNj, LA4L, 4bgM, C5qy, rQL1 ) for their valuable feedback and constructive comments, which will undoubtedly enhance the clarity and overall quality of our work. A well-documented code with the pre-trained models will be publicly released. [qcNj]This paper introduces a novel approach for efficient open-vocabulary 3D instance labeling. [LA4L] The experimental results suggest that the proposed method achieves high precision with a significantly better speed compared with most methods, [4bgM] The overall writing is clear and the framework will be beneficial for related research. [C5qy] the connection between the proposed method and the motivation is coincident. [rQL1]The experimental results show that the proposed method could achieve good performance while remaining very efficient.

As requested by the reviewers, we uploaded a new revised version by including the following in the manuscript (the updated sections are highlighted in blue in the revised version):
- We organized and updated the ablation study in Table 7, line 469.
- We fixed the typo in line 092.
- We fixed the typo in Table 405.
- We added an explanation regarding the performance of Openscene 3D distill in line 413.
- We added a theoretical complexity analysis in the supplementary material from line 11 to line 25.
- We added robustness study and analysis in supplementary material from line 25 to line 61.
-  We provided additional details regarding the generalization of the proposal network from ScanNet200 to the Replica base/novel split in the supplementary material (Table 3).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents an efficient approach to open-vocabulary 3D instance segmentation by leveraging 2D bounding box priors from a pre-trained open-vocabulary 2D object detector. The authors propose the Multi-View Prompt Distribution (MVPDist) method, which effectively utilizes multi-view information while addressing potential misclassification from the 2D object detector to generate reliable 3D instance masks. The experimental results demonstrate the promising real-time performance of the method.

Initially, the reviewers raised several concerns, including:

- The generalizability of Mask3D (qcNj)
- Insufficient evaluation (qcNj, LA4L)
- The engineering focus of the work with insufficient academic contribution (LA4L)
- Lack of explanation and complexity analysis of VAcc (4bgM)
- The robustness of the 2D object detector in extreme conditions (4bgM)
- Performance improvements primarily attributed to the pre-trained YoloWorld model (C5qy)
- Limited capabilities on uncommon categories (C5qy)
- Performance on outdoor datasets (C5qy)
- Feasibility of extending the method to 3D panoptic segmentation (rQL1)
- Slower inference times (rQL1)

The authors provided detailed responses to these concerns, and after the rebuttal, four reviewers (4bgM, rQL1, C5qy, qcNj) voted to accept the paper, while reviewer LA4L did not update their initial rating, which was borderline rejection. The AC noted that reviewer LA4L did not follow up during the author response period.

Upon reviewing the authors’ rebuttal to reviewer LA4L’s comments, the AC finds the authors’ responses to the second and third concerns - regarding the engineering focus and the academic contribution - particularly convincing. The first concern, regarding the academic contribution, is more subjective, but the AC agrees that the proposed Open-YOLO 3D method is effective for the open-vocabulary 3D instance segmentation task.

Given the resolution of most concerns and the strong experimental results, the AC recommends accepting this paper and believes it is appropriate to let the community assess its academic contribution.