Kati8/ICLR_2023_2025 huggingface dataset:
authors: Mohamed El Amine Boudjoghra
authors: Angela Dai
authors: Jean Lahoud
authors: Hisham Cholakkal
authors: Rao Muhammad Anwer
authors: Salman Khan
authors: Fahad Shahbaz Khan
reviews: 
Comments: This paper aims to deal with the open-vocabulary 3D instance segmentation task with a fast and cost-effective approach by utilizing a YOLO-style design. A Multi-View Prompt Distribution method is proposed to effectively fuse the multi-view information. The low granularity label maps are proposed to only use 2D detectors to predict prompt IDs for 3D instance masks. Experimental results demonstrate the state-of-the-art performance of the proposed method. The speed of the proposed method is about 16 times faster than that of existing methods.
Strengths: 1.	The proposed method is simple yet effective.
2.	The two proposed designs are helpful for 3D instance segmentation with meaningful designs.
3.	The experimental results show that the proposed method could achieve good performance while remaining very efficient.
Weaknesses: 1.	Is it possible to extend the proposed method on panoptic segmentation of 3D scenarios? Please present your design briefly for this. 
2.	As shown in Table 1, the inference time of the proposed method is 21.8, which is slower than OpenScene (3D Distill). Please add the explanation for this phenomenon in the corresponding text (first paragraph of Section 5.1).
3.	In Line 405, it should be 4.29 but not 04.29 for OpenScene (3D Distill).
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 4
reviews: 
Comments: The authors propose an efficient method for open-vocabulary 3D instance segmentation to enhance the real-time capability. Unlike existing methods that rely on obtaining 2D masks and category labels from 2D foundation models (like SAM and CLIP), the authors propose a novel approach, Open-YOLO3D, which only leverages bounding boxes generated by 2D object detectors. Moreover, the authors propose a Multi-View Prompt Distribution (MVPDist) method to endeavor promising performance in recognition. The experimental results demonstrate the promising real-time performance of the method proposed by the authors.
Strengths: 1.	The authors only utilize the bounding boxes from 2D object detectors to alleviate the redundancy brought by 2D masks, which demonstrates a significant improvement in inference speed compared to OpenMask3D.
2.	The authors propose a Multi-View Prompt Distribution to obtain reliable category labels form 3D masks, the experimental results evaluation on the ScanNet200 and Replica datasets prove the efficiency of the method proposed by the authors.
3.	The paper is well-structured, and the connection between the proposed method and the motivation is coincident.
Weaknesses: 1.	The improvements of the segmentation performance observed in Open-YOLO3D primarily arise from the enhanced category recognition, which is likely from the prior knowledge of the pre-trained YoloWorld model.
2.	The challenges inherent in Open-YOLO3D closely resemble those faced by Open3DIS, as both methods rely on pre-trained models for generating 3D proposals. As discussed in Open3DIS, the pre-trained 3D models have limited capabilities when it comes to detecting uncommon categories. The representation of 3D data for open vocabulary instance segmentation might be uncultivated and limited.
3.	Recent studies [1] have indicated that OpenMask3D performs poorly on certain outdoor datasets, such as NuScenes. Does Open-YOLO3D face similar challenges in effectively identifying sparse-diverse and less common categories in outdoor environments?

[1] Open-Vocabulary SAM3D: Towards Training-free Open-Vocabulary 3D Scene Understanding.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper proposes Open-YOLO 3D, which is an open-vocabulary 3D instance segmentation framework that efficiently combines 2D object detection and 3D mask generation. The key idea of this paper is its reliance on bounding box predictions from a 2D open-vocabulary object detector and the subsequent use of these predictions for efficient 3D mask proposal and labeling. Unlike prior methods that use computationally intensive models such as SAM and CLIP for feature lifting from 2D to 3D, this paper uses a novel Multi-View Prompt Distribution (MVPDist) and Accelerated Visibility Computation (VAcc) methods to speed up the segmentation process. The framework this paper proposed achieves up to 16x faster inference while keeping competitive or better accuracy.
Strengths: This paper proposed a novel framework, which uses 2D object detection for 3D instance segmentation. The model they presented reduced computational overhead significantly. The Accelerated Visibility Computation (VAcc) leverages tensor operations and GPU batch processing, enabling highly parallelized visibility computation. This contributes to the following speed improvements without compromising performance. By integrating a high-performing 2D open-vocabulary detector, the framework retains strong zero-shot performance, which is important for real-world applications that use new or unknown object types.

It also includes detailed experiments that showcase Open-YOLO 3D's speech and accuracy, and highlight its performance above state-of-the-art approaches like Open3DIS and OpenMask3D. The paper also includes comprehensive ablation studies to demonstrate the improvement of each component. 

The overall writing is clear and the framework will be beneficial for related research.
Weaknesses: I like the overall framework this paper presents and appreciate its contribution to 3D instance segmentation by introducing an inference-efficient model, but I still have some concerns about it:

While the paper mentions that VAcc uses tensor operations, a deeper explanation or complexity analysis comparing it to conventional iterative methods would strengthen the understanding of its true computational advantage, and the reason why it can achieve faster inference speed. I believe the paper clearly demonstrates the operation of this proposed algorithm, however, more explanation about why it is efficient and how much computation cost it saves will better demonstrate the paper's contribution.

The method relies on the quality of the 2D object detector, and this might be an issue if the 2D views are suboptimal (for example poor lighting, and occlusions). A more extensive analysis or discussion on how 2D detection failures propagate through the pipeline would add value.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: This paper primarily aims to achieve faster open-vocabulary 3D instance segmentation compared with existing methods like OpenMask3D. To realize this target, this work first uses a 3D instance segmentation network to generate segmentation proposals. Then, the output of an open-vocabulary 2D object detector as well as some designed 3D information is employed to derive the categories of these proposals.
Strengths: 1. **[Efficiency]** The experimental results suggest that the proposed method achieves high precision with a significantly better speed compared with most methods, and efficiency is important for practical deployment.

2. **[Clearness]** This paper explains its main contribution, how to assign class predictions to 3D proposals, with great clarity. The implementation details are elaborated sufficiently.
Weaknesses: 1. **[Insufficient Academic Contributions]**: This work just combines the output of a 3D segmentation network and a well-implemented open-vocabulary 2D object detector to realize open-vocabulary 3D object detection (similar to existing open-world segmentation method, just with a replacement of the post network to 2D object detector), which is trivial. It is much faster than previous methods because previous methods are developed based on models like SAM and CLIP. This work employs more efficient and suitable existing models. Therefore, although this work is sound in terms of engineering, its real academic contribution and new insights are plain.

2. **[Insufficient Ablation Study]** As the method is efficient because it makes good use of existing models, it is important to clearly analyze how these models contribute to the efficiency, which will guide future works on how to develop an efficient open-vocabulary pipeline. However, this work fails to do so.

3. **[Misleading title]** The method name OPEN-YOLO 3D seems to be unsuitable. YOLO is a 2D object detector while the task is about 3D point cloud segmentation. Although the method utilizes the output of YOLO-World to generate class predictions, the method name is still a little misleading.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: This work addresses the open-vocabulary 3D instance segmentation challenge through an efficient and economical framework built on YOLO-inspired architecture. The authors introduce a Multi-View Prompt Distribution technique designed to effectively integrate information from multiple viewpoints. Additionally, they propose low granularity label maps that leverage 2D detectors to predict prompt identifiers for 3D instance masks. The experimental evaluation demonstrates competitive performance against existing methods, with the proposed approach achieving approximately 16-fold speedup compared to alternative techniques.
Strengths: 1. The proposed approach balances simplicity with strong empirical performance. 2. Both proposed components contribute meaningful innovations to the 3D instance segmentation task. 3. The experimental outcomes indicate that the method achieves competitive results while maintaining substantial computational efficiency.
Weaknesses: 1. Could the authors discuss the potential application of their method to 3D panoptic segmentation scenarios? A brief explanation of the proposed extension would be valuable. 2. As indicated in Table 1, the method's inference time of 21.8 is higher than OpenScene (3D Distill). An explanation for this discrepancy should be included in the primary text of Section 5.1. 3. Line 405 contains a formatting error; the value should read 4.29 rather than 04.29 for OpenScene (3D Distill).
Recommendation: 10
Reviewer confidence: 5
Soundness/correctness: 3
Clarity: 4