EmbodiedSAM: Online Segment Any 3D Thing in Real Time
Authors: Xiuwei Xu
Huangxing Chen
Linqing Zhao
Ziwei Wang
Jie Zhou
Jiwen Lu
Venue: ICLR 2025 Oral (2025)
Keywords: 3d instance segmentation; online 3d scene segmentation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We presented EmbodiedSAM, an efficient framework that leverages vision foundation models for online, real-time, fine-grained and generalized 3D instance segmentation.

Abstract: Embodied tasks require the agent to fully understand 3D scenes simultaneously with its exploration, so an online, real-time, fine-grained and highly-generalized 3D perception model is desperately needed. Since high-quality 3D data is limited, directly training such a model in 3D is infeasible. Meanwhile, vision foundation models (VFM) has revolutionized the field of 2D computer vision with superior performance, which makes the use of VFM to assist embodied 3D perception a promising direction. However, most existing VFM-assisted 3D perception methods are either offline or too slow that cannot be applied in practical embodied tasks. In this paper, we aim to leverage Segment Anything Model (SAM) for real-time 3D instance segmentation in an online setting. This is a challenging problem since future frames are not available in the input streaming RGB-D video, and an instance may be observed in several frames so efficient object matching between frames is required. To address these challenges, we first propose a geometric-aware query lifting module to represent the 2D masks generated by SAM by 3D-aware queries, which is then iteratively refined by a dual-level query decoder. In this way, the 2D masks are transferred to fine-grained shapes on 3D point clouds. Benefit from the query representation for 3D masks, we can compute the similarity matrix between the 3D masks from different views by efficient matrix operation, which enables real-time inference. Experiments on ScanNet, ScanNet200, SceneNN and 3RScan show our method achieves state-of-the-art performance among online 3D perception models, even outperforming offline VFM-assisted 3D instance segmentation methods by a large margin. Our method also demonstrates great generalization ability in several zero-shot dataset transferring experiments and show great potential in data-efficient setting.

Published as a conference paper at ICLR 2025
EMBODIEDSAM: ONLINE SEGMENT ANY 3D THING IN REAL TIME
Xiuwei Xu', Huangxing Chen', Linqing Zhao', Ziwei Wang', Jie Zhou', Jiwen Lul" 'Tsinghua University, "Nanyang Technological University
ABSTRACT
Embodied tasks require the agent to fully understand 3D scenes simultaneously with its exploration, so an online, real-time. fine-grained red--imee.ii--grruieee and highty-generalizen 3D perception model is desperately needed. Since high-quality 3D data is limited, directly training such a model in 3D is infeasible. Mcanwhile, vision foundation models (VFM) has revolutionized the field of 2D computer vision with superior performance, which makes the use of VFM to assist embodied 3D perception a promising direction. However, most existing VFM 3D perception methods are either offline or too slow that cannot be applied in practical embodied tasks. In this paper, we aim leverage Segment Anything Model (SAM) for real-time 3D instance segmentation in an online setting. This is challenging problem since future frames are not available in the input streaming RGB-D video, and an instance may be observed in several frames so efficient object matching between frames required. To address these challenges, we first propose a geometric-aware query lifting module to represent the 2D) masks generated by SAM by 3D-aware queries, which is then iteratively refined by a dual-level query decoder. In this way, the 2D masks are transferred to fine-grained shapes on 3D point clouds. Benefit from the query representation for 3D masks, we can compute the similarity matrix between the 3D masks from different views by efficient matrix operation, which enables real-time inference. Experiments on ScanNet, ScanNet200, SceneNN and 3RScan show method achieves state-of-the-art performance among online 3D perception models, even outperforming offline VFM-assisted 3D instance segmentation methods by large margin. Our method also demonstrates great generalization ability in several zero-shot dataset transferring experiments and show great potential in data eata-efficient setting. Code is available
INTRODUCTION
Embodied tasks, like robotic manipulation and navigation Mousavian et al. (2019): Chaplot et al. (2020); Zhang et al. (2023), require the agent to understand the 3D scene, reason about human instructions and make decisions with self-action. Among the pipeline, embodied visual perception is the foundation for various downstream tasks. In embodied scenarios, we hope the 3D perception model to be: (1) ontine. The input data is streaming RGB-D vidco rather than pre-collected one and visual perception should be performed synchronously with data collection; (2) real-time. High inference speed is needed for robot planning and control; f3ne--ggraine
It should recognize almost One model can be applied to different kinds of scenes and be compatible with different sensor parameters like camera intrinsies. As high-quality 3D data is limited, training such a model in pure 3D is almost infeasible.
any object appeared in the scene; (4) hiighly-generaliee
Inspired by the great achievements of large language models (LLMs) Zhang et al. (2022): Chowdhery et al. (203);; Achiam et al. (2023), a series of vision foundation models (VFMs) such as SAM|
al. (2023) and SEEM Zou et al. (2023). have emerged. VFMs are revolutionizing the field of 2D computer vision by their fine-grained, accurate and generalizable segmentation on image pixels. However, less studies have been conducted on developing VFMs for the 3D domain. Since there is much less high-quality annotated 3D data compared with 2D counterparts, it holds great promise to
"Corresponding author.
Project page: http:://wwwws.giinbiiofASAAA

---

Published as a conference paper at ICLR 2025
Segnentation
JP Segmenturion
20.2 4P
ALAAP Sive
J mii
20
Project
Hand-crafied Merging
Fast Laurnable Merging
10
Musk
Offline RGB-D video
Previous JD SAL
Ours
Streaming RGB-D video
Figure 1: Different from previous 3D SAM methods Yang et al. 2023 Xu et al. (2023): ): Yin et al. (2024) that project 2D masks to 3D and merge them with hand-crafted strategies, ESAM lifts 2D masks to 3D queries and iteratively refine them to predict accurate 3D masks. With 3D queries. ESAM is also able fastly merge 3D masks in different frames with simple matrix operations. Take SAM3D Yang et al. (2023) for comparison, our ESAM surpasses its performance by 23.2% AP with a more than 20x faster speed.
explore the adaptation or extension of existing 2D VFMs for embodied 3D perception. Recently, there are some works Yang et al. (2023); Yin et al. (2024); Lu et al. (2023) that adopt SAM to automatically generate masks on multi view images of a 3D scene and merge the masks in 3D with projection and iterative merging. While these approaches achieve fine-grained 3D instance segmentation with high generalization ability, they still face some serious problems that hinder their application: (1) they apply SAM on individual images and directly project the 2D masks to 3D point clouds with camera parameters. So the predictions are not geometric-awane results across different views; (2) they merge per-fraue per-frame mask predictions in 3D with hand-crafted strategy. E.g., computing geometric similarity between all pairs of masks and merge them according to threshold, which is inaccurate and very slow; (3) most of them offline methods based on pre-collected RGB-D frames with 3D reconstruction, requiring full data collection before perception.
which may produce inconsistent
In this paper, we propose VFM-assisted 3D instance segmentation framework namely Embodied- SAM (ESAM), which exploits the power of SAM to online segment anything in 3D scenes with high accuracy, fast speed and strong generalization ability. As shown in Figure T different from previous 3D SAM methods Yang et al. (2023); Xu et al. (2023); Yin et al. (2024) that project 2D masks to 3D and merge them with hand-crafted strategies, ESAM lifts 2D masks 3D queries and predicts temporal and coometric-consistene 3D masks with iterative query refinement. Benefit from the 3D query representation, ESAM is also able to fastly merge 3D masks in different frames with simple matrix operations. Specifically,
extract point-wise features from the point clouds projected from depth image. Then we regard the 2D masks generated by SAM as superpoints, which is used to guide mask-wise aggregation by our proposed geometric-aware pooling module, generating 3D queries with one-to-one correspondence to SAM masks. We further present a dual-level query decoder to iteratively refine the 3D queries, which makes the queries efficiently attend with superpoint-wisg features and generate fine- fine-grained point-wise masks. Since each 3D instance mask is associated with query, we can compute similarity between newly predicted 3D) masks and previous ones by efficient matrix multiplication in parallel and accurately merge them. To enhance the discriminative ability of query features, we design three representative auxiliary tasks for estimation of geometric, contrastive and semantic similarities. We conduct extensive experiments on ScanNet, ScanNer200, SceneNN and 3RScan datasets. ESAM achieves both leading accuracy and speed among online 3D perception models. Compared with offline VFM-assisted 3D instance segmentation methods, we improve the performance by a large margin while still remain strong generalization ability. Moreover, ESAM also shows great potential in data-efficient setting when trained with limited data.
2 RELATED WORK
VF VaM-sssisted 3D Scene Segmentation: In 2D realm, vision foundation models (VFM) Oquab et al.
(2023 Kirillov et al. (003)) Li et al. (023) have exploded in growth. Benefit from the large amount of annotated visual data, the 2D VFM shows great accuracy and very strong generalization ability, which makes them work well in zero-shot scenarios. Since there is much less high-quality annotated data in the field of 3D vision than the 2D counterpart, using 2D VFM to assist 3D scene perception
2

---

Published as a conference paper at ICLR 2025
becomes promising direction Rozenberszki et al. (2024); Yang et al. (003); Yin et al. (0224) 3 Rozenberszki et al. (2024) considers 2D self self-supervised features from DINO Oquab et al. (2023 to generate initial pseudo masks, which is then iteratively refined with self-traiming SAM3D Yang et al. (203)) adopts Kiriloll et al. (2023) to generate 2D instance masks, which then projected to 3D space by depth and camera parameters and merged according to the geometry. SAI3D Yin et al. (2024) generates 3D primitives on the reconstructed 3D mesh. Then it adopts semantic-SAM acquire 2D masks with semantic scores, which are connected with the 3D primitives and merged by a graph-based region growing strategy. Recently, there are some works Qin et al. (2024); Ye et al. (202)) leverage VFM to segment 3D scenes represented by 3D gaussian, which provide a new perspective for 3D instance segmentation. Our approach also utilizes SAM to assist 3D instance segmentation. Differently, we makes the process of 2D-to-3D projection and 3D mask merging learnable and online. In this way, our ESAM is able to predict more accurate 3D masks and be applied in practical real-tin real-time online tasks.
Online 3D Scene Perception: In persuit of embodied AI, real world applications like robotic navigation Chaplot et al. (2020); Zhang et al. (2023) and manipulation Mousavian et al. (2019) have received increasing attention. Online 3D scene perception, which precisely understands the surrounding 3D scenes from streaming RGB-D videos, becomes the visual basis of these robotic tasks. Early online 3D perception methods process 2D images separately and project the predictions to 3D point clouds, which is followed by a fusion step to merge the predictions from different frames McCormac et al. (2017); Narita al. (2019). However, the predictions on 2D image is not geometric and temporal aware, which makes the fusion step difficult and inaccurate. Fusion-aware 3D-Conv Zhang al. (2020) and SVCNN Huang al. (202T) construct data structures to maintain the information of previous frames and conduct point-based 3D aggregation to fuse the 3D features for semantic segmentation. INS- Liu et al. (2022) extends sparse convolution Graham et al.) (2018); Choy et al. (2019) to incremental CNN to efficiently extract global 3D features for semantic and instance segmentation. In order to simplify the design of online 3D perception model and leverage the power of the advanced offline 3D architectures, MemAda Xu et al. (2024) proposes new paradigm for online 3D scene perception, which empowers offline model with online perception ability by multimodal memory- -hoory adapters. Different from the previous works, ESAM lifts SAM-generater 2D masks to accurate 3D mask and corresponding queries, which enables us to efficiently merge the per-frame predictions with high accuracy.
3 APPROACH
Given sequence of RGB-D images X, [21, ..,,... . with known poses, goal is to segment any instance in the corresponding 3D scene. Formally, 2² = (Te, Pt) where I, is the color image and P. is the point clouds acquired by projecting the depth image to 3D space with pose parameters. Our method is required predict instance masks for the observed 3D scene S. U'_n P.. Furthermore, we want to solve this problem online; that is, at any time instant future frames 34, i >>> I are not known, and temporally consistent 3D instance masks of St should be predicted at each time instant.
The overview of our approach is shown in Figure 2 instance segmentation in an incremental manner achieve real-time processing. At time instant t, we only predict the instance masks M of current frame P.. Then we merge M. to the previous instance masks M of S and get the updated instance masks M of St.
Overview.
We solve the problem of online 3D
3.1 QUERY LIFTING AND REFINEMENT
Consider the model is receiving the 1-th RGB-D frame (T,, Pt), we first adopt SAM automatic mask generation to acquire 2D instance masks Mad from Tt. In this subsection, we ignore the subscript for clearer statement.
Geometric-aware Query Lifting. As SAM does not leverage the information from previous frames, nor does it exploit 3D information from the depth image, directly project 2d to P results in inaccurate and cumporal-inconsiste feature, which enables to further refine the queries for 3D instance mask generation. Since the 2D binary mask is less informative, here we propose to extract point cloud features from the scene and then regard the 2D masks indexs to cluster point clouds into superpoints, where queries can
3D masks. Instead, we aim to lift each 2D mask to a 3D query
3

---

Published as a conference paper at ICLR 2025
Tomperol-pwurc
Query
Fy
Projection
Point Mask
Coo dwar Pazling
104-1
Generation
Merge
Feed Fonward
SAM
Memory
Self-Attantion
Grupine
Guupore paccom ain
Superpoint Masked
Query Auek Dorneepiondeer melerrerc
Cross-Attentin
Most
2D Mark
Soperpaints
Scleet
IusQurry
Query Lifting
Query Refinement
Query Merging
Figure 2: Overview of ESAM. At a new time instant first adopt SAM to generate 2D instance masks M2d. We propose geometric-aware query lifting module to lift M to 3D queries Qr while preserving fine- ane-grained shape information. Q2 are refined by dual-Ievel decoder, which enables efficient cross-attentinn and generates fine- seeegrained point-wise masks Mr from Qt. Then M is merged into previous masks Mt" by fast query merging strategy.
be simply selected from the superpoint features. Assuming the point clouds P R and there are .M masks in 2d we first map M to according to the color-depth correspondence to get superpoint index S E Z.N, where each element in S falls in [0, M). Then feed P to a 3D sparse U-Net[Choo et al. 2019)
with memory-based adapter Xu et al. (2024) to extract temporal-aware 3D features Fp E R. With Fp and S, we can pool the point-wise features to superpoint features
Fs E IR MXC
However, naive operation such max average pooling may degrade the representation ability of Fs. To better preserve the point features inside each superpoint, we take the geometric shape of each superpoint into account. For superpoint p S P, E [o, .M., we compute the normalized relative positions p" of all points Pj E P with respect to the superpoint's center c;. In this way, the Pi
set P; = {p} max(p;) min(ny)
Pj E }i} represents the normalized shape of this superpoint with diameter of and center of origin. Then we compute the local and global features for each point:
x globel
Agg(z
RCC
local MLP(P.) E xP|xxC
where MLP performs on each individual point and Agg is the aggregation function implemented with channel-wise max-poolis max-pooling. The local and global features represent the relevance between points and shape, so we concat both features and feed them to another MLP to predict point-wise weight:
t0 Sigmoid (MT.((2) E IR (0.1)) 2j
[z]
locol
galobal
Finally, we aggregate point features Fp into the i-th superpoint with weighted average pooling:
F (FF)) + global G(F)) mean( * [wi,.., wip,il)
Note we enhance the pooled superpoint feature with zgiobal fully combine the shape-leved geometric feature and scene-level 3D U-Net feature. The computation for each superpoint can be parallelized with point-wise MIP and Scatter function Fey, so this geometrie-aware pooling is actually efficient.
Dual-level Query Decoder. After pooling, the M 2D instance masks 2d are lifted to 3D superpoint features Fs. Then we initialize series of 3D instance queries Qo from Fs. which are iteratively refined by several transformer-based query decoder layers and leveraged to predict 3D masks. During training, we randomly sample a proportion between 0.5 and of Fs to construct Qo for data augmentation. While at inference time we simply set Qu Fs..
Each qeury decoder employs masked cross-attention between queries and the scene representations to aggregate instance information for each query:
Q Q.K
KT + A1) V. A(ii.) -
if (i,j) True [ 0, 1.2 (4)
Q Softmax
VC
otherwise
4

---

Published as a conference paper at ICLR 2025
where indicates matrix multiplication, Q is the linear projection of Q1, K and V are the linear projection of the scene representations F. F can be point-wise features Ep or superpoint-wist features Fs. A₁ is the attention mask derived from the predicted 3D instance masks Mf in the I-th decoder layer. (i,j) indicates i-th query attending to j-th point or superpoint. Then we feed Qi to self-attention layer and feed forward network to get Q1+1, followed by mask generation module to predict the instance mask for each query:
Mfar ( F*)
9; / 0,3,
(5)
where Φ is linear layer. M point mask if F Fp, otherwise it is superpoint mask.
common practice Schult et al. (002)); Sun et al. 2023); Kolodiazhnyi et al. (2024b) for qeury
decoder is to adopt the same level of scene representations for cross- coss-attention and mask generation. However, since SAM has already outputs high-level semantic-aware masks, we observe M < N. we adopt the point- boint-wise scene representations Fp for query decoder, the cross-attention operation will be memory- Fs, the predicted 3D instance masks will only be the combination of superpoints and thus cannot be refined to finer granularity. To get the best of both worlds, our query decoder is designed to be dual-level. For cross-attention in Eq 4. we F Fs to achieve efficient interaction. While for mask prediction in Eq (5). we set F Fp for ane-igri mask generation. To support masked attention, we pool point mask to superpoint mask before Eq (4):
due to the large amount of points. While if we use superpoint features
(6) where G is the gcometric-aware pooling in Eq (3). We can reuse the pre-computed weights in Eq 2 to reduce computation. In this way, after 3x query decoders, we acquire accurate point masks M as well as the corresponding queries Qa, which denoted as M. and Q₂ in the following subsections. We perform mask-NMS Mc to filter out redundant masks as well as the corresponding queries.
M f- G(MF"T) 4
3.2 EFFICIENT ONLINE QUERY MERGING
Once lifting 2D masks M2d to accurate 3D masks M we then merge M to the previous instance masks M, to acquire Mf Note when = 1 we have Mire Mp initialization. However, the mainstream solution for merging in- stance masks in previous works Yang et al. (2023)
Auxiliary Task
Yin et al. (2024): Liu et al. (2022) Narita et al. (2019)
Lu et al. 2023) is to traverse over all masks in M
X
and compare each mask in Mf with all previous masks in M. This process is very slow, because order to accurately decide wheter new mask shoule be merged into previous mask, the geometric sim- ilarity such mask-IoU or CD-distance is computed on the point clouds of the two masks. The compu- tation of similarity involves all the points in each mask, which has high computation complexity. What worse, above operations are hard to be computed in parallel, since the number of points in each mask different and we need to pick out point clouds of each instance according to the mask one by one. To this end, we propose to represent each mask in fixed- size vectors and compute the similarity with efficient matrix operation.
X
X
Amsddal Bon
X
X
Pudh
x
X
Q-Cinnesst
X
Semuntic
Figure 3: Details of our efficient query merg- ing strategy. We propose three kinds of rep- resentative auxiliary tasks, which generates geometric, contrastive and semantic represen- tations in the form of vectors. Then the simi- larity matrix can be efficiently computed by matrix multiplication. We further prune the similarity matrix and adopt bipartite matching to merge the instances.
Benefit from our architecture, for each mask in Meur and M we have the corresponding query feature. The query feature itself is a fixed-size vector repre- sentation, but simply computing similarity between them is less informative, which gets very low performance. Therefore, set up several representative auxiliary tasks based on the query features to learn vector representations under different metrics, which are used to compute geometric, contrastive and semantic similarities.
First, for geometric similarity. we observe the model is able to learn the whole geometry with only partial observation. However, due to the restriction of segmentation that predictions only be made
5

---

Published as a conference paper at ICLR 2025
on existing points, the model cannot express its knowledge about the whole geometry. Therefore, make the model able to express its full knowledge by introducing an auxiliary task of bounding box prediction. We adopt MLP to predict the bounding box regression based on the center of each query (i.e. the center & of the corresponding superpoint) to get box B E R Then the geometric similarity between two masks can he computed by the IoU between the two boxes. We ignore box orientations since the IoU matrix between two sets of axis-aligned bounding boxes can be computed by simple matrix operation.
Second, for contrastive similarity, we aim to learn an instance-specific representation where features from the same instance should be pulled together and otherwise pushed away. This representation can be learned by contrastive training between two adjacent frames: we use MLP to map the query features Q: to contrastive feature ft. Then for an instance appears in the 1-th and (t + 1)-th frames, we choose the two features of this instance (ft fi+1) the positive pair, and sample features from other instances (Ji (5.21) as the negative pair. The detailed loss function is shown in the next subsection.
Finally, for semantic similarity, we simply adopt MLP to predict per-category probability distribution S S IR where K is the number of pre-defined categories. There are also other choices for this task. For example, if adopt semantic-SAN Li et al. (2023) instead of SAM, can directly utilize the semantic predictions for the 2D masks to serve as S for the corresponding queries.
In this way, the similarity matrix C between Mi and Mr can be efficiently computed with their corresponding geometric, contrastive and semantic representations:
ppre
JE
Sare
Seur
C = (B B.TT)= +
1 2
where loU(.,-) means the IoU matrix between two of axis-aligned bounding boxes. We prune C by setting elements smaller than threshold to -oo. Then bipartite matching with cost - is performed on M and Mr"r, which assigns each mask in M one of the masks in M If a new mask fails to match with any previous mask, we register new instance for this mask. Otherwise we merge the two masks as well as their B, and S. Mask merging can be simply implemented by taking union. -
While for other representations, we weighted average them by: [, - B n+I Be"r [j] and so on. We assume the j-th new mask is merged to the i-th previous mask. n is the count of merging, which indicates the number of masks that have been merged [[
3.3 LOss FUNCTION
We have semantic and instance labels on each RGB-D frame. In each RGB-D video, the instance labels of different frames are consistent. Given the annotations, we compute per-frame losses based the predictions from each query. Since the queries Q, are lifted from 2D SAM masks in one-to-one way, we ignore the complicated label assignment step and directly utilize the annotations on 2D mask to supervise the predictions from the corresponding query. We assume that 2D SAM mask can belong only to one instance, and thus we can acquire the ground mask for each query. We utilize the pixel correspondence with depth image map 2D instance mask to 3D point clouds, and compute ground-truth axis-aligned bounding box based on the 3D instance mask. With above annotations, we compute binary classification loss C. with cross-entropy to discriminate foreground and background instances. The predicted 3D mask is supervised by a binary cross-entropy Ci and a Dice loss Li The losses for bounding boxes and semantic predictions are defined as loU-loss Cim, and binary cross-entropy Lrem respectively.
semantic label and 2D instance
Apart from the above per-frame losses, we also formulate a contrastive loss between adjacent frames:
Z
Lcont
1-111
log
(8)
Ejet (I + ((((
i=1
where ( is cosine similarity. So finally the total loss is formulated as:
1 C T Σ (aL's + Lice + L'aice + BL + L.ee + Lcont 6++11 Lcont (+--1)
(9)
where Leont" T+T+1 and Leont 1+0 is set O.

---

Published as a conference paper at ICLR 2025
Table 1: Class-agnostic 3D instance segmentation results of different methods ScanNet200 dataset.
The unit of Speed is ms per frame, where the speed of VFM and other parts are reported separately.
Method
Type
VFM
AP
AP50
AP25
Speed
SAMPro3D
Offline
SAM
18.0
32.8
56.
Open3DIS
Offline
GroundedSAM
34.6
43.1
48.5
SAI3D
Offline
SemanticSAM
28.2
47.2
67.9
SAM3D
Online
SAM
20.2
35.7
55.5
1369+1518
ESAM
Online
SAM
42.2
63.7
79.6
1369+80
ESAM-E
Online
FastSAM
43.4
65.4
80.9
20+80
Table 2: Dataset transfer results of different methods from ScanNet200 SceneNN and 3RScan. We directly evaluate the models in Table I on other datasets to show their generalization ability.
BaanNer200->RRSaee
Method
Type
AP
AP50
AP25
AP
AP50
AP25
SAMPro3D
Offline
12.6
25.8
53.2
3.9
8.0
21.0
Open3DIS
Offline
18.2
32.2
48.9
9.5
21.8
47.0
SAI3D
Offline
18.6
34.7
65.7
8.1
16.9
37.0
SAM3D
Online
15.1
30.0
51.8
6.2
13.0
33,9
ESAM
Online
28.8
52.2
69.3
14.1
31.2
59.6
ESAM-E
Online
28.6
50.4
71.0
13.9
29.4
58.8
4 EXPERIMENT
In this section, we first describe our datasets and implementation details. Then we compare our method with state-of-the-arr VFM-assisted 3D instance segmentation methods online 3D segmentation methods to validate its effectiveness. We also apply ESAM in data-efficient setting to demonstrate its application potential. Finally conduct ablation studies to provide a comprehensive analysis on our design. More supplementary experiments can be found in appendix.
4.1 BENCHMARK AND IMPLEMENTATION DETAILS
We evaluate our method four datasets: ScanNet Dai et al. (2017).
Roleeeee200/Roonesrsrseeee et al.
(202)), SceneNN Hua et al. (016) and 3RScan Wald et al. (2019). ScanNet contains 1513 scanned scenes, out of which we use 1201 sequences for training and the rest 312 for testing. ScanNet200 provides more fine-grained annotations on the scenes of SeanNet, which contains more than 200 categories. SceneNN contains 50 high-quality scanned scenes with instance and semantic labels. Following Xu et al. (2024), we select 12 clean sequences for testing. 3RScan is more challenging indoor dataset where the RGB-D sequences are acquired by fast-moving cameras. We choose its test split for testing, which contains 46 scenes. Each dataset provide both posed RGB-D sequences and reconstructed point clouds with labels.
Benchmarks: We compare different methods on four benchmarks. First, we compare with VFM- assisted 3D instance segmentation methods in Table I We train different methods on ScanNet200 training set (if needed) and evaluate them on ScanNer200 validation set in class-agnostic manner. We also train and evaluate the methods in the same way on ScanNet, where the results are put in appendix (Table 7. For offline methods, the input of each scene is reconstructed point cloud and RGB-D video, where predictions are made on the reconstructed point clouds. For online methods, the input is streaming RGB-D video, and map the final predicted results on St to the reconstructed point clouds with nearest neighbor interpolation for comparison.
Since zero-shot methods like SAM3D do not require training. To fairly compare them with learnable methods, we further evaluate the models in Table on SceneNN and 3RScan without finetuning. This benchmark, shown in Table validates the generalization ability of different methods.
7

---

Published as a conference paper at ICLR 2025
Table 3: 3D instance segmentation results of different methods on ScanNet and SceneNN datasets.
ScanNet
SceneNN AP
Method
Type
FPS
AP
AP50
AP25
AP
AP25
TD3D
Offline
46.2
71.1
81.3
Oneformer3D
Offline
59.3
78.8
86.7
INS-Conv
Online
57.4
TD3D-MA
Online
39.0
60.5
71.3
26.0
42.8
59.2
3.5
ESAM-E
Online
41.6
60.1
75.6
27.5
48.7
64.6
10.0
ESAM-E+FF
Online 42.6
61.9
77.1
33.3
53.6
62.5
9.8
Finally, we compare with online 3D instance segmentation methods in Table Following previous works Liu et al. evaluate them on Net validate set and SceneNN.
(2022) Xu et al. (2024). we train different methods ScanNet training set and
Compared methods: On Table and Table we compare ESAM with SAM3D Yang al. 2023 SAMPro3D Xu et al. (2023), Open3DIS Nguyen et al. (2024) and SAI3D Yin et al. (2024). ). We adopt the 2D version of Open3DIS for fair comparison. Because its 3D version makes predictions directly on reconstructed point clouds, while other methods only make predictions on RGB-D frames. On Table 3 we compare with TD3D Kolodiazhnyi et al.] (2024a). Oneformer3D Kolodiazhnyi et al. 22024b), INS- INS-Conv[ Liu et al. (2022) and TD3D-MA Xu et al. (2024). In terms of VFM, above (2023), GroundedS Ren et al. (2024), SLM al. (2023) and FastSAM Zhao et al. (2023).
methods mainly adopt SAM Kirillov
Implementation details: Following Xu et al. (2024), we train ESAM in two stages. First we train single-view perception model on (20)))()((0)5 which is a subset of ScanNet(200) with individual RGB-D frames, without memory-based adapters and losses for the three auxiliary tasks. Next finetune the single-view perception model RGB-D sequences with the adapters and full losses. To reduce memory footprint, we randomly sample adjacent RGB-D frames for each scene at every iteration. For hyperparameters, the dual-level query decoder, we actually set F = Fs for the first two iterations of mask prediction, and then set F Fp. This smoothens the mask generation process with curriculum learning.
&p 0.5, c 1.75,
0.02, = 0.5 and 0.5. In
4.2 COMPARISON WITH STATE-OF-THE-ART
We compare our method with the top-performance VFM-assisted 3D instance segmentation methods and online 3D instance segmentation methods as described above. We provide three versions of ESAM, namely ESAM, ESAM-E and ESA.-E++F.. ESAM adopts SAM as the VFM while ESAM-E adopts FastSAM to achieve real-time inference. ESAM-E+FF not only adopts the 2D masks from FastSAM, but also fuses image features extracted by FastSAM backbone to point clouds following Rukhovich al. (002)). We also include some visualization results in for qualitative evaluation.
According to Table T on class-agnostic 3D instance segmentation task (i.e. the 3D "segment anything task"), our ESAM establishes new state-of-the-an compared with previous methods, even including the offline ones. Note that it is much more challenging for online methods to perceive the 3D scenes compared to offline alternatives, since offline methods directly process the complete reconstructed 3D scenes while online methods deal with partial and noisy frames. Despite the high accuracy, ESAM is also much faster than previous methods. It takes only 80ms process a frame due to the efficient architecture design and fast merging strategy, while methods like SAM3D that adopts hand-crafted merging strategy requires more than 1s per frame. When replacing SAM with the faster alternative FastSAM, ESAM-E can achieve real-time online 3D instance segmentation with about 10 FPS, while the accuracy still much higher than previous methods.
In terms of generalization ability, ESAM also demonstrates great performance. As shown in Table 2 when directly transferred to other datasets, ESAM still achieves state-of-the-a accuracy compared with zero-shot methods. We note offline methods perform worse on 3RScan dataset, this is because they highly rely on clean reconstructed 3D meshes with accurately aligned RGB frames. While in SCan the camera is moving fast and thus the RGB images and camera poses are blurry.
8

---

Published as a conference paper at ICLR 2025
Input Seene
Ground- Truth
SAM3D
SAL3D
Ours
Figure 4: Visualization results of different 3D instance segmentation methods ScanNet200 dataset. As highlighted in red boxes, SAM3D predicts noisy masks while SAI3D tends to over segment an instance into multiple parts.
We visualize the predictions of the above methods ScanNet200, as shown in Figure ESAM can predict accurate and fine-grained 3D instance segmentation masks, while being able to process streaming RGB-D video in real time. We also provide online visualization to further demonstrate the practicability of ESAM in Figure 6 appendix. More details can be viewed our video demo.
As shown in Table 3 ESAM also achieves state-of-the thrttrr performance compared with previous online 3D instance segmentation methods. Different from previous methods that only fuse 2D features 3D point clouds, our approach utilize both 2D features and 2D masks to better guide the learning of 3D representation.
4.3 ANALYSIS OF ESAM
Data-efficient learning. We reduce the training sam- ples by using only 20% or 50% training set and report the class-agnostic performance of ESAM on Scan- Net200 in Table 4 It is shown that the performance degradation ESAM is not significant with only half the training data. Moreover, ESAM still achieves state-of-the-art performance even with 10% training data (SAI3D: 28 28.2//4.2/67.99 This is because 2D VFM has already provided a good initialization, thus the learning part of ESAM is easy to converge.
Table 4: Performance of ESAM when trained with partial training set.
Proportion
AP
AP50
AP25
100%
42.2
63.7
79.6
50%
40.2
62.3
78.4
10%
32.8
54.1
73.9
Ablation study. We conduct ablation studies to validate the effectiveness of the proposed methods. For architecture design, we conduct experiments on ScanNet-25k. and report AP and average inference latency (ms) of each frame excluding VFM in Table It can be that geometric-aware pooling boosts the performance up to 1.3% while brings negligible computational overhead. Note that the prediction
single views will accumulate the whole scenes, high AP ScanNet-25k

---

Published as a conference paper at ICLR 2025
contributes lot to the final performance. We can also observe that the dual-level design in ESAM achieves comparable accuracy compared with the time-consune consumnnn F = Fp strategy, while only slightly increases the latency compared with the super-st F Fs strategy. For the merging strategies, we compare different design on ScanNet with AP reported, as shown in Table 6 It is shown that each auxiliary task is important for the quality of mask merging. We notice that the geometric similarity has the most significant influence on the final performance. This is because most mask pairs can be excluded based on distance.
Visualization of auxiliary tasks. We also visualize the predictions of our auxiliary tasks for com- prehensive understanding ESAM. From F (a), it can be observed that the model is able to predict the whole geometry of objects with only partial observation. The t-SNE visualization in F validates that the model successfully learns discriminative query representation for object matching. Finally the semantic segmentation results in Tigure5](c shows that our ESAM can learn satisfactory semantic representation and is extendable to 3D semantic segmentation task.
Table 5: Effects of the architecture design.
Table 6: Effects of the merging strategies.
Method
AP
Latency
Method
AP
Replace 9 with average pooling
45.9
43.6
Remove box representation
33.4
Set F = Fg only
34.5
43.1
Remove contrastive representation
36.9
Set F = Fp only
47.4
51.7
Remove semantic representation
37.6
The final model
47.2
45,4
The final model
41.6
30
20
5
1-
(a)
(c)
Figure 5: Visualization of the auxiliary tasks for our merging strategy. (a) 3D box prediction for geometric similarity. We visualize the bounding boxes of an object at different time instant. (b) t-SNE visualization of the instance-specifie representation for contrastive similarity. Different colors indicate different instances and different points indicate the instance feature at different frames. (c) Query-wise semantic segmentation for semantic similarity.
5 CONCLUDING REMARK
In this work, we presented ESAM, an efficient framework that leverages vision foundation models for online, real-time, fine-grained and generalized 31) instance segmentation. We propose to lift the 2D masks generated by VFM to 3D queries with geometric-aware pooling, which is followed by a dual-path query decoder to refine the queries and generate accurate 3D instance masks. Then with the query-mask correspondence, we design three auxiliary tasks to discriminatively represent each 3D mask, which enables fast mask merging with matrix operations. Extensive experimental results on four datasets demonstrates that ESAM achieves leading performance, online and real-time inference and strong generalization ability. We believe ESAM brings a new paradigm on how effectively leverage 2D VFM for embodied perception.
Potential Limitations. Despite of the satisfactory performance, there are still some limitations of ESAM. First, whether ES AS is real-time depends on the adopted VFM. Currently we adopt SAM and FastSAM, among which only FastSAM can achieve real-time inference. However, we believe there will be more efficient 2D VFM with better performance and more functions in the near future, and ESAM can be further improved along with the improvement of 2D VFM. Second, the 3D U-Net and memory- based adapters for feature extraction inference time for 3D part of ESAM. The speed of ESAM may be boosted to a higher level if we can make the backbone more efficient, which we leave for future work.
relatively heavy, which count for most of the
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 624B2076, Grant 62125603, Grant 62336004, and Grant 62321005, and in part by the Beijing Natural Science Foundation under Grant No. L247009.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint (:XXi:2303.08722 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. ICCV, pp. 9650- 9660, 2021.
Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. NeurIPS, 33:4247-42555 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. JMLR, 24(240): 1-113, 2023.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, pp. 3075-3084, 2019.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, pp. 5828 5839, 2017.
Pedro F Felzenszwalb and Daniel P Huttenlocher. Efficient graph-based image segmentation. IJCV, 59(2)1677181, 2004.
Matthias Fey. Pytorch extension library of optimized scatter operations. [EB/OL]. https :// \//
github. httaubbooorruuusteeeeeee wwt.m///wuatylllyyyytooo...tatte
Benjamin Graham, Martin Engeleke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In CVPR, pp. 2224-9232, 2018.
Yuenan Hou, Xinge Zhu, Yuexin Ma, Chen Change Loy, and Yikang Li. Point-to-voxel knowledge distillation for lidar semantic segmentation. In CVPR, pp. 1479-8488, 2022.
Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh-Khoi Tran, Lap-Fai Yu, and Sai-Kit Yeung. Scenenn: A scene meshes dataset with annotations. In 3DV, pp. 92-101, 2016.
Shi-Sheng Huang, Ze- Yu Ma, Tai-Jiang Mu, Hongbo Fu, and Shi-Min Hu. Supervoxel convolution for online 3d semantic segmentation. TOG, 40(3):1-15, 2021.
Zhening Huang, Xiaoyang Wu, Xi Chen, Hengshuang Zhao, Lei Zhu, and Joan Lasenby. Openins3d: Snap and lookup for 3d open- openbbocay instance segmentation. In ECCV, pp. 169-185. Springer, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In ICCV, pp. 4015-4026, 2023.
Maksim Kolodiazhnyi, Anna Vorontsova, Anton Konushin, and Danila Rukhovich. Top-down beats bottom-up in 3d instance segmentation. In WACV, pp.
2024a.
Maxim Kolodiazhnyi, Anna Vorontsova, Anton Konushin, and Danila Rukhovich. Oneformer3d: One transformer for unified point cloud segmentation. In CVPR, 2024b.
11

---

Published as a conference paper at ICLR 2025
Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, and Jianfeng Gao. Semantic- sam: Segment and recognize anything at any granularity. arXiv preprint arXiv:2307.04766 2023.
Hao Li, Asim Kadav, Igor Durdanovic, Haman Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprini arXiv:1608.088) 2016.
Leyao Liu, Tian Zheng, Yun-Jou Lin, Kai Ni, and Lu Fang. Ins-conv: Incremental sparse convolution for online 3d segmentation. In CVPR, pp.
2022.
Shiyang Lu, Haonan Chang, Eric Pu Jing, Abdeslam Boularias, and Kostas Bekris. Ovir-3d: Open- vocabulary 3d instance retrieval without training on 3d data. In CoRL, pp. 1610-1620. PMLR, 2023.
John McCormac, Ankur Handa, Andrew Davison, and Stefan Leutenegger. Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. In ICRA, pp. 4628-4635. IEEE, 2017.
Leland McInnes and John Healy, Accelerated hierarchical density based clustering, In ICDMW, pp. 33 33-42. IEEE, 2017.
Arsalan Mousavian, Clemens Eppner, and Dieter Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In ICCV, pp. 2901-2910, 2019.
Gaku Narita, Takashi Seno, Tomoya Ishikawa, and Yohsuke Kaji. Panopticfusion: Online volumetric semantic mapping at the level of stuff and things. In IROS, pp. 4205-4212. IEEE, 2019.
Phuc D. A. Nguyen, Tuan Duc Ngo, Evangelos Kalogerakis, Chuang Gan, Anh Tran, Cuong Pham, and Khoi Nguyen. Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance. In CVPR, 2024.
Lucas Nunes, Xicyuanli Chen, Rodrigo Marcuzzi, Aljosa Osep, Laura Leal-Taixé, Cyrill Stach- niss, and Jens Behley. Unsupervised class-agnosta instance segmentation of 3d lidar data for autonomous vehicles. RAL, 7(4):8713-8722 8720, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d
language gaussian splatting. In CVPR, pp. 20051-20060, 2024.
Tianhe Ren. Shilong Liu, Ailing Zeng. Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang. Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-worke world models for diverse visual tasks, 2024.
David Rozenherszki, Or Litany, and Angela Dai. Language-grounder indoor 3d semantic segmenta- tion in the wild. ECCV, pp. 125-141. Springer, 2022.
David Rozenberszki, Or Litany, and Angela Dai. Unscene3d: Unsupervised 3d instance segmentation for indoor scenes. In CVPR, 2024.
Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Tr3d: Towards real-time indoor 3d object detection. arXiv preprint 230.:02000008888.. 2023.
Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d for 3d semantic instance segmentation. arXiv preprint arXiv:2210.03 2022.
Jiahao Sun, Chunmei Qing, Junpeng Tan, and Xiangmin Xu. Superpoint transformer for 3d scene instance segmentation. In AAAI, volume 37, pp. 2393-2401, 2023.
Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Mare Pollefeys, Federico Tombari, and Fran- cis Engelmann. Openmask3d: Open-vocabulary 3d instance segmentation. arXiv preprint arXiv:2306. 13631, 2023.
12

---

Published as a conference paper at ICLR 2025
Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Nießner. Rio: 3d object instance re-localization in changing indoor environments. In ICCV, pp. 7658-7667, 2019.
Mutian Xu, Xingyilang Yin, Lingteng Qiu, Yang Liu, Xin Tong, and Xiaoguang Han. Sampro3d: Locating sam prompts 3d for zero-shot scene segmentation. arXiv preprint 1rXiv22311.17707, 2023.
Xiuwei Xu, Chong Xia, Ziwei Wang, Linqing Zhao, Yueqi Duan, Jie Zhou, and Jiwen Lu. Memory- based adapters for online 3d scene perception. CVPR, 2024.
Yunhan Yang, Xiaoyang Wu, Tong He, Hengshuang Zhao, and Xihui Liu. Sam3d: Segment anything in 3d scenes. arXiv preprint arXiv:2306. 03908, 2023.
Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. In ECCV, pp. 162-179. Springer, 2024.
Yingda Yin, Yuzheng Liu, Yang Xiao, Daniel Cohen-Or, Jingwei Huang, and Baoquan Chen. Sai3d: Segment any instance in 3d scenes. CVPR, 2024.
Jiazhao Zhang, Chenyang Zhu, Lintao Zheng. and Kai Xu. Fusion-aware point convolution for online semantic scene segmentation. In CVPR, pp. 4534-4543, 2020.
Jiazhao Zhang, Liu Dai, Fanpeng Meng, Qingman Fan, Xuelin Chen, Kai Xu, and He Wang. 3d-aware object goal navigation via simultaneous exploration and identification. In CVPR, pp. 6672-6682, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0000 010000 2022.
Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jínqiao Wang. Fast segment anything. arXiv preprint arXiv: 2556, 2023.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. NeurIPS, 36, 2023.
A APPENDIX
We provide more experimental results in the appendix.
A.1 ONLINE VISUALIZATION
We demonstrate the online 3D segmentation process in Figure 6 It is shown that ESAM can effectively merge partial segmentation results into whole object and generate one-grained 3D masks for the online reconstructed 3D scene.
A.2 ADDITIONAL EXPERIMENTS
Following Yin et al. 2024). we compare ESAM with conventional clustering methods McInnes & Healy (2017 y Nunes al. (2022); Felzenszwalb & Huttenlocher (2004); |Rozenberszki al. (2024); Caron et al. (222)) and VFM -asssssi 3D scene perception methods on ScanNet in Table Consistent with the results in Table [ ESAM also achieves leading performance and speed on ScanNet in the class-agnostic 3D instance segmentation setting.
A.3 ANALYSIS ON INFERENCE TIME
We decompose the inference time ESAM excluding VFM Table The temporal- aware backbone consists of sparse convolutional U-Net and several memory based adapters. The merging process consists of similarity computation, bipartite matching and mask representation updating. Due to the efficient design, the decoder and merging operation of ESAM only take small proportion of
13

---

Published as a conference paper at ICLR 2025
Figure 6:
Online visualization of ESAM on ScanNet200 dataset. Refer to the video demo in our project page for more details.
inference time. In terms of backbone, it is promising to adopt techniques like network pruning or knowledge distillation Li al. (2016); Hou et al. (2022) to accelerate its inference speed, which we leave for future work.
A.4 OPEN-VOCABULARY RESULTS
We further evaluate the open-vocabulary 3D instance segmentation ability on the 198 categories of Scan- Net200 in Table 99] Since ESAM outputs class- agnostic 3D masks, there are two methods to extend to open-vocabulary 3D segmentation. The first feed the class-agnostic 3D masks open- nocc-vocayylary mask classification model like OpenMask3D Takmaz
Table 9: Open-vocabulary 3D instance seg-
mentation results on ScanNer200 dataset.
Method
AP50
AP25
SAI3D
9.6
14.7
19.0
ESAM
13.7
19.2
23.9
al. (2023) and OpenIns3D Huang et al. (2024), which is adopted in the code of SAI3D Yin et al. (2024). The second to adopt open-vocabulary 2D segmentation model to acquire the category labels for each 2D mask. Since there is one-to-one correspondence between 3D mask and 2D mask in ESAM, we can acquire the category labels for each 3D mask accordingly. Here we follow SAI3D adopt the first method and compare with it.
14

---

Published as a conference paper at ICLR 2025
Table Class-agnostic 3D instance segmentation results of different methods ScanNet dataset. Following Yin et al. (2024), we compare with conventional clustering methods and VFM-assisted 3D scene perception methods. The unit of Speed ms per frame, where the speed of VFM and other parts are reported separately.
Method
Type
VFM
AP
AP 50
AP25
Speed
HDBSCAN
Offline
1.6
5.5
32.1
Nunes et al.
Offline
2.3
7.3
30.5
Felzenszwalb et al.
Offline
5.0
12.7
38.9
UnScene3D
Offline
DINO
15.9
32.2
58.5
SAMPro3D
Offline
SAM
16.7
31.5
57.9
Open3DIS
Offline
GroundedS
29.9
46.7
58.6
SAI3D
Offline
SemanticSAM
30.8
50.5
70.6
SAM3D
Online
SAM
20.2
34.0
53.3
1369+1518
ESAM
Online
SAM
48.2
70,3
85.3
1369+80
ESAM-E
Online
FastSAM
49.3
71.4
85.8
20+80
Table Decomposition of the inference time (ms) ESAM excluding VFM.
Backbone
Merging Matching
Decoder
Total
3D-Unet
Adapters
Similarity
Updating
41.0
28.0
5.0
0.7
0.3
5.0
80
A.5 VISUALIZATION ON GEOMETRIC-AWARE POOLING
We provide the visualization of point-wise weights predicted in Geometric-aware Pooling in Figure The points with high weights are shown in red, while ones with low weights are shown in blue. is shown that noisy boundary with inaccurate shape information will be assigned low weight, while regions contain objects (especially small objects) are assigned high weight for better segmentation.
A.6 QUALITATIVE RESULTS ON SCENENN AND 3RSCAN
further visualize different methods on SceneNN and 3RScan Figure Consistent with Figure ESAM predicts accurate and clean 3D masks. On the contrary, the prediction of SAM3D is noisy or incomplete while the prediction of SAI3D tends to over-segmented.
15

---

Published as a conference paper at ICLR 2025
Figure 7: Visualization of weights in Geometric-aware Pooling. We also provide the corresponding RGB images. The points with high weights are shown in red, while ones with low weights are shown in blue.
SAM3D
SAL3D
ESAM
SccneN
3RSean
Figure 8: Visualization results on SceneNN and 3RScan.
16


Reviews:
[{"review_id": "J77R9UilmD", "reviewer": "Reviewer_eVni", "summary": "This paper leverages SAM for 3D instance segmentation, incorporating a geometry-aware query lifting module that transforms 2D masks into fine-grained 3D shapes on point clouds. Using 3D queries, it efficiently merges 3D masks across frames through simple matrix operations. This design enables online, real-time 3D instance segmentation.", "questions": "Lack of discussion with respect to :\nGaussian Grouping: Segment and Edit Anything in 3D Scenes\nLangSplat: 3D Language Gaussian Splatting\nSegment Any 3D Gaussians", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Unlike approaches that apply SAM on individual images and project 2D masks onto 3D point clouds, this paper introduces a geometry-aware module that lifts 2D masks to 3D queries. This enables the prediction of temporally and geometrically consistent 3D masks using iterative query refinement.\n\nThis work eliminates the need for handcrafted merging strategies by instead identifying similarities between newly predicted and previously generated 3D masks, resulting in improved performance.\n\nAchieving real-time inference, and outperforming not only online methods but also offline methods.", "weaknesses": "The abstract and introduction quickly dive into the implementation without providing sufficient motivation to explain why this approach is taken. For instance, it lacks clarity on how 3D-aware is achieved, the role of queries, the need for iterative refinement, and the insight of the dual-level query decoder. Although the comparison with prior work is clear, the explanation of the proposed method lacks an overview, leaving out insights behind the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SnMcRYVayS", "reviewer": "Reviewer_qwSP", "summary": "This paper introduces a real-time, online 3D instance segmentation method for embodied tasks, leveraging the Segment Anything Model (SAM) to address the lack of high-quality 3D data for direct training. Unlike existing methods, which are often too slow for practical use, this approach translates SAM’s 2D masks into fine-grained 3D shapes via a geometric-aware query lifting module and a dual-level query decoder. This enables efficient object matching across frames, achieving state-of-the-art results on benchmarks like ScanNet and demonstrating strong generalization in zero-shot and data-efficient scenarios.", "questions": "See weakness above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing is clear and easy to follow.\n2. The method is technically sound.\n3. The performance results are strong.", "weaknesses": "1. The camera's intrinsic and extrinsic parameters are crucial for this task. In embodied environments, it is very important to obtain accurate and stable intrinsic and extrinsic parameters. How can you acquire them in embodied environments? Whether the unstable and incorrect  intrinsic and extrinsic parameters affect the final results? Additionally, in lines 139-140, the point clouds are generated by projecting the depth image into 3D space using pose parameters. So, how are two point clouds aligned and merged? In real-world scenarios, point cloud alignment is also a significant issue.\n\n2. There is a lack of visualization for the point-wise weights predicted in Geometric-aware Pooling. Providing this visualization would clarify how the model weights features across different geometric regions.\n\n3. Regarding the class-agnostic 3D instance segmentation results compared in Table 1, the fairness of the comparison could be questioned. Since Open3DIS does not use semantic and instance labels for supervision, it may not be entirely fair to compare it with fully-supervised methods. I think a comparison with fully-supervised online 3D instance segmentation methods, as done in Table 3, would be more appropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CQ16YQfwzJ", "reviewer": "Reviewer_bGw4", "summary": "This paper proposes a geometry-aware query lifting module that represents 2D masks generated by SAM as 3D-aware queries. These queries are then iteratively refined using a similarity matrix between 3D masks from various viewpoints through a dual-level query decoder. EMBODIEDSAM partially addresses the offline and slow-processing issues present in most existing VFM-assisted 3D perception methods, making it more suitable for real-world embodied tasks.", "questions": "See the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Using InsQuery instead of traditional handcrafted strategies achieves faster and geometrically consistent multi-frame mask merging, ultimately enabling real-time inference speed.\n2. The paper is well-structured and clearly written.\n3. The supplementary video demo effectively highlights the online, real-time, fine-grained, and highly generalized capabilities of EmbodiedSAM.", "weaknesses": "1. How is recognition achieved for any 3D object category? Why are the results in Table 1 for class-agnostic 3D instance segmentation？\n2. In Figure 4, it would be helpful to match the colors of predicted and ground truth instances for easier comparison.\n3. Provide additional comparisons with more recent offline methods, such as OpenMask3D[1] and OpenIns3D[2].\n\nReference:\n\n[1] Takmaz, Ayça, et al. \"Openmask3d: Open-vocabulary 3d instance segmentation.\" arXiv preprint arXiv:2306.13631 (2023).\n\n[2] Huang, Zhening, et al. \"Openins3d: Snap and lookup for 3d open-vocabulary instance segmentation.\" arXiv preprint arXiv:2309.00616 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pKps0hYpQT", "reviewer": "Reviewer_tbzg", "summary": "This paper proposes a method for online 3D instance segmentation. Given a sequence of posed RGB-D images, they first use SAM to acquire 2D masks, and unproject them to 3D superpoints using the depth maps. Then they propose a novel framework to aggregate geometric features of superpoints, followed by a dual-level query decoder to predict 3D masks. They additionally propose to leverage bounding boxes IOU and semantics to assist the similarity computation, and use feature contrasting to further improve the performance.", "questions": "1. According to Table 5 and 6, the performance gains mainly come from the merging strategies. An ablation of the geometric-aware query lifting and dual-level query decoder would better evaluate the necessity of the proposed framework.\n2. According to Table 6, the computation of the similarity matrix in Eq.(7) plays a vital role, and it would be beneficial if can provide an ablation study on the weights of the different terms in Eq.(7). Besides, is directly adding them better than multiplying them?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Motivation: The paper proposes a novel framework for online, efficient 3D instance segmentation from RGB-D sequences, which is of high potential for embodied tasks.\n2. Contribution: The proposed framework is efficient in mask generation and query merging, and the use of the IOU and semantic are reasonable and effective.\n3. Performance: The performance greatly surpasses the baselines.", "weaknesses": "I'm not very familiar with the task of this paper and did not find obvious weaknesses in this paper. Please see the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XLoORBU0qv", "reviewer": "Reviewer_H4Wa", "summary": "The paper proposes a 3D instance segmentation method that is online, real-time, fine-trained and generalizable. Specifically it lifts the 2D segmentation masks generated by SAM to 3D using depth and uses the instance indices to group points into superpoints. It proposes a dula-level query decoder to use both points and superpoints for efficiently generating fine-grained masks. As an online method, it proposes a efficient way to merge masks at current timestamp to previous masks, which considers geometric similarity, contrastive similarity and semantic similarity. Because each superpoint has a feature vector, the similarities can be compuated as matrix operation for quick inference. Experiments were conducted on four baseline datasets and the proposed method outperforms both previous online methods and offline methods, and it also shows generalizability on zero-shot cases.", "questions": "1. Typo: L280: position pair-> positive pair\n2. All results included in the paper are class-agnostic. I am curious if the proposed method has advantages over all object categories equally or it works better on certain categories.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel way to lift 2D segmentation masks to 3D that is geometry aware. It also proposes an algorithm to better, efficiently merge 3D masks from different timestamps.\n2. The method can achieve better results on four datasets, most notably in a real-time way (can achieve 10 fps with FastSAM as backbone). This is very important for its application in real-world scenarios. \n3. The paper is well-organized and easy to follow.", "weaknesses": "1. All qualitative results are on ScanNet200. Wondering if the proposed method also has clear visual improvements on other datasets used in the paper.\n2. Since the method is targeted to be used in real-world embodied tasks and is supposed to generalize to unseen scenes, it would be more convincing if results on some real-world captures are included.\n3. For generating superpoint features in equation (3), why the $z^{global}$ is added instead of concatenated to the pooled point features? Also why using the original point features instead of $z^{local}$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yt1bQNi9rP", "reviewer": "Reviewer_jHbV", "summary": "The paper presents EmbodiedSAM, a framework designed for online 3D instance segmentation, leveraging the Segment Anything Model (SAM). This method integrates SAM with a novel merging strategy that fuses geometric, contrastive, and semantic information to deliver high-efficiency segmentation of 3D objects in dynamic environments. By enabling real-time segmentation, EmbodiedSAM is positioned to enhance 3D perception tasks where immediate processing is critical. The framework is evaluated on several datasets, demonstrating its potential to perform effectively in real-world applications where timely response is essential.", "questions": "See the \"weakness\" part.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2. The experiment part of this paper is sufficient.\n3. This paper provides a nice summary of existing works.", "weaknesses": "This paper is very interesting, I think it has reached the standard for acceptance at the conference, I only have a few simple questions.\n\n1. The Geo-aware Pooling module \bis a common technology. So, I think it should not be one of the main contributions of this paper, the whole pipeline is the most important design.\n2. As an online segmentation network, the author should report the time cost of each module, such as super point building, query refinement, and merging.\n3. The results presented in Table 6 should include more detailed explanations regarding the performance impact of removing each of the three auxiliary tasks (geometric, contrastive, and semantic similarities) in the merging strategy. This clarification is necessary to understand the individual contribution of each auxiliary task.\n4. The ablation study is conducted across different datasets, making it difficult to directly compare the results. It would be more effective to perform the entire ablation study on a single dataset to ensure clearer and more consistent analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)