EmbodiedSAM: Online Segment Any 3D Thing in Real Time

Authors: Xiuwei Xu, Huangxing Chen, Linqing Zhao, Ziwei Wang, Jie Zhou, Jiwen Lu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3d instance segmentation; online 3d scene segmentation

Abstract:
Embodied tasks require the agent to fully understand 3D scenes simultaneously with its exploration, so an online, real-time, fine-grained and highly-generalized 3D perception model is desperately needed. Since high-quality 3D data is limited, directly training such a model in 3D is infeasible. Meanwhile, vision foundation models (VFM) has revolutionized the field of 2D computer vision with superior performance, which makes the use of VFM to assist embodied 3D perception a promising direction. However, most existing VFM-assisted 3D perception methods are either offline or too slow that cannot be applied in practical embodied tasks. In this paper, we aim to leverage Segment Anything Model (SAM) for real-time 3D instance segmentation in an online setting. This is a challenging problem since future frames are not available in the input streaming RGB-D video, and an instance may be observed in several frames so efficient object matching between frames is required. To address these challenges, we first propose a geometric-aware query lifting module to represent the 2D masks generated by SAM by 3D-aware queries, which is then iteratively refined by a dual-level query decoder. In this way, the 2D masks are transferred to fine-grained shapes on 3D point clouds. Benefit from the query representation for 3D masks, we can compute the similarity matrix between the 3D masks from different views by efficient matrix operation, which enables real-time inference. Experiments on ScanNet, ScanNet200, SceneNN and 3RScan show our method achieves state-of-the-art performance among online 3D perception models, even outperforming offline VFM-assisted 3D instance segmentation methods by a large margin. Our method also demonstrates great generalization ability in several zero-shot dataset transferring experiments and show great potential in data-efficient setting.

Decision: accept (oral)
Reviewer scores: 8, 8, 6, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730279050339

thread_type:
official_review

<thread_object>
creator: reviewer_jhbv

<content>
summary:
The paper presents EmbodiedSAM, a framework designed for online 3D instance segmentation, leveraging the Segment Anything Model (SAM). This method integrates SAM with a novel merging strategy that fuses geometric, contrastive, and semantic information to deliver high-efficiency segmentation of 3D objects in dynamic environments. By enabling real-time segmentation, EmbodiedSAM is positioned to enhance 3D perception tasks where immediate processing is critical. The framework is evaluated on several datasets, demonstrating its potential to perform effectively in real-world applications where timely response is essential.

strengths:
1. This paper is well-written and easy to follow.
2. The experiment part of this paper is sufficient.
3. This paper provides a nice summary of existing works.

weaknesses:
This paper is very interesting, I think it has reached the standard for acceptance at the conference, I only have a few simple questions.

1. The Geo-aware Pooling module is a common technology. So, I think it should not be one of the main contributions of this paper, the whole pipeline is the most important design.
2. As an online segmentation network, the author should report the time cost of each module, such as super point building, query refinement, and merging.
3. The results presented in Table 6 should include more detailed explanations regarding the performance impact of removing each of the three auxiliary tasks (geometric, contrastive, and semantic similarities) in the merging strategy. This clarification is necessary to understand the individual contribution of each auxiliary task.
4. The ablation study is conducted across different datasets, making it difficult to directly compare the results. It would be more effective to perform the entire ablation study on a single dataset to ensure clearer and more consistent analysis.

questions:
See the "weakness" part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701786504

thread_type:
official_review

<thread_object>
creator: reviewer_h4wa

<content>
summary:
The paper proposes a 3D instance segmentation method that is online, real-time, fine-trained and generalizable. Specifically it lifts the 2D segmentation masks generated by SAM to 3D using depth and uses the instance indices to group points into superpoints. It proposes a dula-level query decoder to use both points and superpoints for efficiently generating fine-grained masks. As an online method, it proposes a efficient way to merge masks at current timestamp to previous masks, which considers geometric similarity, contrastive similarity and semantic similarity. Because each superpoint has a feature vector, the similarities can be compuated as matrix operation for quick inference. Experiments were conducted on four baseline datasets and the proposed method outperforms both previous online methods and offline methods, and it also shows generalizability on zero-shot cases.

strengths:
1. The paper proposes a novel way to lift 2D segmentation masks to 3D that is geometry aware. It also proposes an algorithm to better, efficiently merge 3D masks from different timestamps.
2. The method can achieve better results on four datasets, most notably in a real-time way (can achieve 10 fps with FastSAM as backbone). This is very important for its application in real-world scenarios. 
3. The paper is well-organized and easy to follow.

weaknesses:
1. All qualitative results are on ScanNet200. Wondering if the proposed method also has clear visual improvements on other datasets used in the paper.
2. Since the method is targeted to be used in real-world embodied tasks and is supposed to generalize to unseen scenes, it would be more convincing if results on some real-world captures are included.
3. For generating superpoint features in equation (3), why the $z^{global}$ is added instead of concatenated to the pooled point features? Also why using the original point features instead of $z^{local}$

questions:
1. Typo: L280: position pair-> positive pair
2. All results included in the paper are class-agnostic. I am curious if the proposed method has advantages over all object categories equally or it works better on certain categories.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730978267900

thread_type:
official_review

<thread_object>
creator: reviewer_tbzg

<content>
summary:
This paper proposes a method for online 3D instance segmentation. Given a sequence of posed RGB-D images, they first use SAM to acquire 2D masks, and unproject them to 3D superpoints using the depth maps. Then they propose a novel framework to aggregate geometric features of superpoints, followed by a dual-level query decoder to predict 3D masks. They additionally propose to leverage bounding boxes IOU and semantics to assist the similarity computation, and use feature contrasting to further improve the performance.

strengths:
1. Motivation: The paper proposes a novel framework for online, efficient 3D instance segmentation from RGB-D sequences, which is of high potential for embodied tasks.
2. Contribution: The proposed framework is efficient in mask generation and query merging, and the use of the IOU and semantic are reasonable and effective.
3. Performance: The performance greatly surpasses the baselines.

weaknesses:
I'm not very familiar with the task of this paper and did not find obvious weaknesses in this paper. Please see the questions below.

questions:
1. According to Table 5 and 6, the performance gains mainly come from the merging strategies. An ablation of the geometric-aware query lifting and dual-level query decoder would better evaluate the necessity of the proposed framework.
2. According to Table 6, the computation of the similarity matrix in Eq.(7) plays a vital role, and it would be beneficial if can provide an ablation study on the weights of the different terms in Eq.(7). Besides, is directly adding them better than multiplying them?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731045371147

thread_type:
official_review

<thread_object>
creator: reviewer_bgw4

<content>
summary:
This paper proposes a geometry-aware query lifting module that represents 2D masks generated by SAM as 3D-aware queries. These queries are then iteratively refined using a similarity matrix between 3D masks from various viewpoints through a dual-level query decoder. EMBODIEDSAM partially addresses the offline and slow-processing issues present in most existing VFM-assisted 3D perception methods, making it more suitable for real-world embodied tasks.

strengths:
1. Using InsQuery instead of traditional handcrafted strategies achieves faster and geometrically consistent multi-frame mask merging, ultimately enabling real-time inference speed.
2. The paper is well-structured and clearly written.
3. The supplementary video demo effectively highlights the online, real-time, fine-grained, and highly generalized capabilities of EmbodiedSAM.

weaknesses:
1. How is recognition achieved for any 3D object category? Why are the results in Table 1 for class-agnostic 3D instance segmentation？
2. In Figure 4, it would be helpful to match the colors of predicted and ground truth instances for easier comparison.
3. Provide additional comparisons with more recent offline methods, such as OpenMask3D[1] and OpenIns3D[2].

Reference:

[1] Takmaz, Ayça, et al. "Openmask3d: Open-vocabulary 3d instance segmentation." arXiv preprint arXiv:2306.13631 (2023).

[2] Huang, Zhening, et al. "Openins3d: Snap and lookup for 3d open-vocabulary instance segmentation." arXiv preprint arXiv:2309.00616 (2023).

questions:
See the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731048426691

thread_type:
official_review

<thread_object>
creator: reviewer_qwsp

<content>
summary:
This paper introduces a real-time, online 3D instance segmentation method for embodied tasks, leveraging the Segment Anything Model (SAM) to address the lack of high-quality 3D data for direct training. Unlike existing methods, which are often too slow for practical use, this approach translates SAM’s 2D masks into fine-grained 3D shapes via a geometric-aware query lifting module and a dual-level query decoder. This enables efficient object matching across frames, achieving state-of-the-art results on benchmarks like ScanNet and demonstrating strong generalization in zero-shot and data-efficient scenarios.

strengths:
1. The writing is clear and easy to follow.
2. The method is technically sound.
3. The performance results are strong.

weaknesses:
1. The camera's intrinsic and extrinsic parameters are crucial for this task. In embodied environments, it is very important to obtain accurate and stable intrinsic and extrinsic parameters. How can you acquire them in embodied environments? Whether the unstable and incorrect  intrinsic and extrinsic parameters affect the final results? Additionally, in lines 139-140, the point clouds are generated by projecting the depth image into 3D space using pose parameters. So, how are two point clouds aligned and merged? In real-world scenarios, point cloud alignment is also a significant issue.

2. There is a lack of visualization for the point-wise weights predicted in Geometric-aware Pooling. Providing this visualization would clarify how the model weights features across different geometric regions.

3. Regarding the class-agnostic 3D instance segmentation results compared in Table 1, the fairness of the comparison could be questioned. Since Open3DIS does not use semantic and instance labels for supervision, it may not be entirely fair to compare it with fully-supervised methods. I think a comparison with fully-supervised online 3D instance segmentation methods, as done in Table 3, would be more appropriate.

questions:
See weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731142445370

thread_type:
official_review

<thread_object>
creator: reviewer_evni

<content>
summary:
This paper leverages SAM for 3D instance segmentation, incorporating a geometry-aware query lifting module that transforms 2D masks into fine-grained 3D shapes on point clouds. Using 3D queries, it efficiently merges 3D masks across frames through simple matrix operations. This design enables online, real-time 3D instance segmentation.

strengths:
Unlike approaches that apply SAM on individual images and project 2D masks onto 3D point clouds, this paper introduces a geometry-aware module that lifts 2D masks to 3D queries. This enables the prediction of temporally and geometrically consistent 3D masks using iterative query refinement.

This work eliminates the need for handcrafted merging strategies by instead identifying similarities between newly predicted and previously generated 3D masks, resulting in improved performance.

Achieving real-time inference, and outperforming not only online methods but also offline methods.

weaknesses:
The abstract and introduction quickly dive into the implementation without providing sufficient motivation to explain why this approach is taken. For instance, it lacks clarity on how 3D-aware is achieved, the role of queries, the need for iterative refinement, and the insight of the dual-level query decoder. Although the comparison with prior work is clear, the explanation of the proposed method lacks an overview, leaving out insights behind the approach.

questions:
Lack of discussion with respect to :
Gaussian Grouping: Segment and Edit Anything in 3D Scenes
LangSplat: 3D Language Gaussian Splatting
Segment Any 3D Gaussians
</content>
</thread_object>

</thread>
<thread>
creation date:
1732183511823

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thanks for all reviewers’ constructive comments and patience on our work. We are glad to receive positive feedbacks from all reviewers. We have provided detailed replies to the questions of each reviewer and refined our paper following the suggestions. The modification on our paper mainly including:
- Open-vocabulary 3D instance segmentation results in Appendix A.4.
- Visualization of point-wise weights predicted in Geometric-aware Pooling in Appendix A.5.
- Qualitative results on other datasets except for ScanNet200 in Appendix A.6.
- Some writing and presentation are improved.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper receives unanimous positive ratings of 8,8,6,8,8. The AC follows the recommendations of the reviewers to accept the paper. The reviewers think that the proposed method of using a geometry-aware module that lifts 2D mask to 3D queries is effective, the performance of the proposed method is strong and the proposed method can work in real-time inference. The proposed method can even outperform offline methods. The weaknesses mentioned by the reviewers are mostly clarifications, which are well-addressed by the authors in the discussion and rebuttal phases.