Online Segment Any 3D Thing as Instance Tracking
Authors: Hanshi Wang
caizijian
Jin Gao
Yiwei Zhang
Weiming Hu
Ke Wang
Zhipeng Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Online 3D Segmentation
Embodied Intelligence
Primary area: applications

Abstract: Online, real-time, and fine-grained 3D segmentation constitutes a fundamental capability for embodied intelligent agents to perceive and comprehend their operational environments. Recent advancements employ predefined object queries to aggregate semantic information from Vision Foundation Models (VFMs) outputs that are lifted into 3D point clouds, facilitating spatial information propagation through inter-query interactions.
Nevertheless, perception, whether human or robotic, is an inherently dynamic process, rendering temporal understanding a critical yet overlooked dimension within these prevailing query-based pipelines. This deficiency in temporal reasoning can exacerbate issues such as the over-segmentation commonly produced by VFMs, necessitating more handcrafted post-processing. Therefore, to further unlock the temporal environmental perception capabilities of embodied agents, our work reconceptualizes online 3D segmentation as an instance tracking problem (AutoSeg3D). Our core strategy involves utilizing object queries for temporal information propagation, where long-term instance association promotes the coherence of features and object identities, while short-term instance update enriches 
instant observations. Given that viewpoint variations in embodied robotics often lead to partial object visibility across frames, this mechanism aids the model in developing a holistic object understanding beyond incomplete instantaneous views. Furthermore, we introduce spatial consistency learning to mitigate the fragmentation problem inherent in VFMs, yielding more comprehensive instance information for enhancing the efficacy of both long-term and short-term temporal learning. The temporal information exchange and consistency learning facilitated by these sparse object queries not only enhance spatial comprehension but also circumvent the computational burden associated with dense temporal point cloud interactions. Our method establishes a new state-of-the-art, surpassing ESAM by 2.8 AP on ScanNet200 and delivering consistent gains on ScanNet, SceneNN, and 3RScan datasets, corroborating that identity-aware temporal reasoning is a crucial, previously underemphasized component for robust 3D segmentation in real-time embodied intelligence. Code is at https://github.com/AutoLab-SAI-SJTU/AutoSeg3D.

Online Segment Any 3D Thing as Instance Tracking
Hanshi Wang 12.3, Zijian Cai³, Jin G2 Yiwei Zhang Weiming Hu 1,2,566 Ke Wang", Zhipeng Zhang
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA
School of Artificial Intelligence, University of Chinese Academy of Sciences
8 Lat School of Artificial Intelligence, Shanghai Jiao Tong University *Anyverse Intelligence
*Bejjing Key Laboratory of Super Intelligent Security of Multi-Modal Information 6 School of Information Science and Technology, ShanghaiTech University "KargoBot
(hanshi wang. cv, zhipeng. zhang. (ronn..vv)@ooo ev)toollokk. com; jin. gin.gao@nlpr. aa.ac.a...............
Abstract
Online, real-time, and fine- fine-grained 3D segmentation constitutes fundamental capa- bility for embodied intelligent agents to perceive and comprehend their operational environments. Recent advancements employ predefined object queries aggregate semantic information from Vision Foundation Models (VFMs) outputs that are lifted into 3D point clouds, facilitating spatial information propagation through inter-query interactions. Nevertheless, perception, whether human or robotic, is an inherently dynamic process, rendering temporal understanding critical yet over- looked dimension within these prevailing query-based pipelines. This deficiency in temporal reasoning can exacerbate issues such the sver-segmentttion commonly produced by VFMs, necessitating more handcrafted post-processing. Therefore, further unlock the temporal environmental perception capabilities of embodied agents, our work reconceptualizes online 3D segmentation as instance track- ing problem (AutoSeg3D). Our core strategy involves utilizing object queries for temporal information propagation, where long-term instance association promotes the coherence features and object identities, while short-term instance update enriches instant observations. Given that viewpoint variations in embodied robotics often lead to partial object visibility across frames, this mechanism aids the model developing holistic object understanding beyond incomplete instantaneous views. Furthermore, we introduce spatial consistency learning to mitigate the fragmenta- tion problem inherent in VFMs, yielding more comprehensive instance information for enhancing the efficacy of both long-term and short-term temporal learning. The temporal information exchange and consistency learning facilitated by these sparse object queries not only enhance spatial comprehension but also circumvent the computational burden associated with dense temporal point cloud interactions. Our method establishes a new state-of-the-arr surpassing ESAM by 2.8 AP on Scan- Net200 and delivering consistent gains on ScanNet. SceneNN, and 3RScan datasets, corroborating that identity- -waare temporal reasoning is crucial, previously under- emphasized component for robust 3D segmentation in real time embodied intelli- gence. Code is at hihhtt:/// / thub. com/ Sat.aaa---SSSSSSSAAAaAAAaA------
Introduction
The ability to perform online, real-time, and fine-grained 3D instance segmentation a cornerstone for embodied intelligent agents to perceive and comprehend their operational environments. Autonomous robots and embodied assistants increasingly depend on such systems for exploring and interacting
"This work was completed during Hanshi's remote internship at SJTU and co-mentored by Prof. Zhipeng CCorespponiigg author.
Zhang.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

with complex scenes. Early approaches predominantly adopts an offline paradigm, which involved accumulating complete point clouds prior to processing, thereby incurring prohibitive latency and memory costs. In pursuit of faster and online perception capabilities, recent research has begun to explore paradigms assisted by Vision Foundation Models (VFMs) such as SAM [
Current online VFM-assisted models are engineered to process streaming inputs by initially predicting segmentation results with VFMs and subsequently lifting the generated masks and recorded depth to superpoint representations. However, these pipelines simply concatenate global point features across scans and omit instance level temporal modeling, which worsens fragmentation and over segmentation by VFMs. Post hoc mon-mmmimum suppression only partially corrects these errors and introduces the concurrent loss of valid information not as expected.
Seeking to address these limitations, we draw inspiration from established methodologies for main- taining temporal coherence in online perception. Classical multi-object tracking (MOT) methods, for instance, achieve consistent identity assignment by exploiting spatial continuity and appearance affinities to link detections across frames [2 3. Similarly, video instance segmentation frameworks like VisTR [4] and 3D detection models such as Sparse4D [5] employ query-based memory banks to propagate and update object features over time, enabling each instance to maintain a persistent representation robust to occlusion and partial views. The core design principle underpinning these diverse approaches is the explicit maintenance and evolution of instance- instance-specifie representations across temporal sequences. Inspired by this paradigm, we recast online 3D instance segmentation as an instance-traed trackennt task. By integrating object-level temporal priors directly into the segmentation pipeline, our approach aims to concurrently rectify over-segmentation errors and enforce identity consistency, thereby substantially enhancing overall segmentation performance and robustness.
More specifically, we introduce a novel, tracking-centric pipeline that directly addresses the two core limitations of VFM-based methods. Our framework decomposes into three lightweight and synergistic modules. First, the Long-Term Memory (LTM) maintains a bounded track bank and employs Hungarian assignment based on confidence-gated affinity matrix recover identities after prolonged occlusions with constant overhead. Second, the Short- Memory (STM) refines in- stance embeddings via distance- aware cross-frame attention to inject immediate temporal context while filtering out background noise. Third, Spatial Consistency Learning (SCL) merges high-affinity mask fragments at inference by jointly reasoning over 2D appearance and 3D geometry, while con- currently employing one-to-many fragment supervision during training to mitigate over-segmentation and generate coherent, high-fidelity queries for LTM and STM. Together, these components form cohesive, real-time 3D instance segmentation system that enforces consistent object identities across frames, injects immediate temporal context while filtering out background noise, and merges high-affinity fragments to directly counteract VFM over-segmentation. By integrating these modules, our framework preserves real-time throughput while delivering 2.8 AP gain over recent ESAM (6] on ScanNet200 77). Extensive evaluations on both ScanNet200 and ScanNet [8], as well zero-shot assessments on SceneNN [9] and 3RScan [0] demonstrate consistent performance gains.
In summary, our contributions are as follows: 1) We recast online 3D instance segmentation as continuous instance tracking problem by treating each VFM-derived mask as a track query within a unified framework. 2) We propose lightweight architecture with three synergistic modules where LTM propagates identities across frames to ensure continuity, STM injects short-term temporal context while filtering background noise, and SCL merges overlapping fragments to counteract over-segmentation and enrich instance embeddings. 3) Our framework achieves new state-of-the-ar results on ScanNet200, ScanNet, SceneNN, and 3RScan while sustaining real-time throughput, and ablation studies verify the contribution of each component.
2 Related Work
VFM-assisted 3D Scene Segmentation. Vision foundation models (VFMs) have emerged as promising cornerstone for 3D scene understanding embodied intelligence, especially in the con- struction and reasoning of 3D spatial information [[]
12
13
14
I
15
16
17 18]. Large-scale
pretrained VFMs such as SAM [ and CLIP [12] exhibit powerful vocabulayy segmentation and semantic alignment capabilities, which have been extensively leveraged in downstream 3D perception pipelines. SAM3D [19] first predicts 2D instance masks with SAM and then lifts them to 3D via depth and camera parameters, followed by geometric merging. CLIP2Scene [20] distills multimodal knowl-
2

---

edge from CLIP into a 3D backbone through semantic and spatio-temporal consistency regularization, enabling label-efficient scene parsing. OpenMask3D [2T] combines extraexx visual features with SAM-refined masks to generate discriminative per-instance embeddings for ppen-vocabulary 3D instance segmentation. Despite these advances, several studies have highlighted that the 21) masks produced by VFMs are often over-segmented. SAI3D [22], for example, decomposes the reconstructed mesh into 3D primitives, assigns semantic scores to 2D masks via Semantic-SAN [15], and aggregates the primitives through a graph-based region-growing algorithm. Nevertheless, existing approaches still rely on heuristic post-hoc fusion of projected 3D masks, which often proves brittle in cluttered or dynamically changing robotic environments. In this work, we propose learnable fusion module that jointly reasons over over-segmentationn hypotheses in both 2D and 3D spaces. By optimizing fusion in end-to-end manner, our method mitigates the impact of erroneous 2D masks and delivers more robust and scalable 3D scene understanding.
Online 3D Scene Perception. Driven by the rapid advancements in autonomous driving and embodied AI, robotic tasks are increasingly demanding higher levels of 3D scene understanding. In these scenarios, the ability to process information in real time, adapt to diverse conditions, and achieve perception is crucial. However, most of the common instance segmentation methods [23 330 31
24
32 [3 are offline. They can handle large-scale datasets but are highly dependent on the quality of preprocessing and data augmentation, which makes it difficult to apply them complex and ever-changing robotic environments. Recently, online 3D scene perceptions 34 35 36 37 38 39] have attracted increasing attention. INS-Conv [88] proposes an incremental sparse convolutional network for online 3D segmentation, which achieves efficient and accurate inference by processing only the residuals between consecutive frames and incorporating an uncertainty term to adaptively select which residuals to update. MemAda 40) proposes an adapter- model that equips mainstream offline frameworks with the competence to perform online scene perception, enabling them to process real-time RGB-D sequences efficiently. Building on this foundation, ESAM [6] further advances the field by achieving online scene segmentation and designing a dual-layer decoder along with auxiliary tasks facilitate the merging of 3D masks. While prevailing methods fuse dense features (e.g., raw point clouds) temporally, they often lack the semantic context crucial for instance-level tasks. We address this by recasting online segmentation as instance tracking, which allows us to propagate semantically rich instance information across frames. This focus on semantic consistency through time yields significantly more precise instance segmentation results, while also being computationally efficient.
25 26
28 29
21
3 Method
3.1 Overall Architecture
Fig. 3.1 illustrates our brecking-cenrrin online 3D segmentation framework. The design draws inspiration from the brain's 's complementary learning systems [4 hippocampus rapidly forms episodic memories, allowing quick adaptation to novel contexts and interaction with recent experience, whereas the neocortex consolidates these transient traces into durable representations through slow, cumulative learning, producing a stable store of knowledge. This dual mechanism not only enhances adaptability but also ensures the coherence and persistence of memory. Mirroring this division, we decompose our framework into long-term memory for instance association and short-term memory for instance update, realised by three lightweight yet synergistic modules: 1) Long-term memory (LTM), detailed in Sec. 3.2) matches instance identities over extended periods, enabling recovery after prolonged occlusion. 2) Short-term memory (STM), detailed in Sec. 3.3 recurrently updates each instance's 's representation with information from the immediately preceding frame. 3) Spatial Consistency Learning (SCL) includes Learning- -Base Mask Integration inference and Cestancc-Consistemma Mask Supervision during training, detailed in Sec. 3.4 respectively counteract VFM's intrinsic over-segmentation, thereby reducing query redundancy and furnishing STM and LTM with coherent, high-fidelity mask representations.
45 Specifically, the
3.2 Long-Term Memory for Instance Association
Online 3D segmentation requires that all point-cloud observations of the same instance, collected across successive frames, be fused into single temporally coherent instance. To improve the
3

---

RGB-D
SjotialkCondssturyy Leaming
Short-Term Memory Instance Update
Leng-Term Memory for Instance A ssociation
Pooling
Short-Tem eem Mumory
Transtemmer Decodar
Yaud fastares
cocke
ValilOuniio
Seguent Anything
Uplile
Laornable ssociation
Unrakied
2D Mask
DodENCN-NEE Albeliee
Doop Dood
Figure 1: This diagram delineates the operational mechanisms of our constituent modules. Spatial Consistency Learning (SCL) mitigates the over-segmentation tendencies of VFM by employ- ing a one-to-many supervision strategy during the training phase and utilizing learning- based mask integration at the inference stage. The Short-term Memory (STM) module enriches current instance representations by integrating observational data from prior frames. Furthermore, the Long-term Memory (LTM) module engineered to associate instances, segmented by the Visual Front-end Module (VFM), with established tracklets in memory, consequently enhancing temporal consistency.
temporal consistancy, recast instance aggregation an explicit instance tracking problem with supervised matching, confidence gating, and Hungarian assignment.
Concretely, at the first frame (t 1), we obtain N, instance queries that derived from 3D mask 5) and their corresponding embeddings Q1 E IRNixd that derived by applying a MLP to instance queries, and predicted 3D bounding boxes B, E IRNiX6, Each box is axis-aligned and specified by its minimum and maximum coordinates (zrmn:: ymmi: Amin; Tmax, Umax Æmax). Similarly, for every subsequent frame & containing N, segments, we obtain the instance embeddings Q. xd the corresponding boxes B, E N:6 and the instance embeddings from the tracklets in memory Q Trk R VT xd Here, N represents the number of active tracklets up to now. Critically, the embedding associated with each tracklet is not merely derived from the immediately preceding frame t 1, instead, it encapsulates richer temporal information accumulated across the sequence, thereby reflecting a more comprehensive long-term history (see Sec. 3.33for for more details). Then we measure the similarity between instances (segments) from current frames with tracklets by,
PPP
[i]
Q [rk
E MLP (I.U( T (D)), Egj = E E
(1)
where ESP Eseo E E R.N xd respectively correspond to the appearance, geometric, and fused affinity features. We then project the fused affinity features using learned w, w' E Rd,
exp(w Eg) Mj = Σj exeepw [ Eij'' Cj Cj= (w Eg), Ay = My Cij.
(2)
where Mj is a normaaelle affinity obtained via softmax, which represents the relative similarity that segmented instance corresponds to tracklet j. Since each row sums 1, every segment allocates its entire probability mass across the set of candidate tracklets. To modulate this raw affinity we introduce sigmoid-based confidence gate Cj a(-). which down- down-weights uncertain matches and suppresses spurious associations. To convert these probabilities into one-to-one correspondences, we formulate bipartite matching problem. Specifically, segments and tracklets form a bipartite graph with edges weighted by gated affinities Ag- Solving this assignment with the Hungarian algorithm selects the set of pairs () that maximises the summed weights while ensuring that every segment and every track is used at most once. For each matched pair, the track state is updated by,
B Tyk - B Trk [j] + [
Trk - Q Tokl [J] + Q.
aj + aj +1
(3)
03 +1
aj +1
where aj the track age. Unmatched instances initialise new tracklets with a 1. Tracks that remain unmatched for more than Tie frames are marked stale, removed from the active set, and pushed into a fixed-capacity queue, the oldest entry evicted when the buffer overflows. At every time step any segment that remains unassigned after the active-track matching stage is subsequently

---

matched against the LTM buffer with the identical Hungarian solver. A successful match reactivates the stored tracklet, restoring its state and resetting its age, thereby recovering instances that reappear after extended occlusion. By coupling confidence-gatt gatteeddeegged Hungarian assignment with bounded long-term memory, the proposed strategy suppresses spurious instances, maintains identities through prolonged occlusions, and guarantees constant computational overhead.
3.3 Short-Term Memory for Instance Update
In online 3D segmentation, the inherent scene continuity, where instances visible in frame often reappear in frame f, naturally motivates the use of fross-frame attention mechanisms to integrate historical context. For embodied agents, which frequently experience rapid and significant viewpoint variations, the effective fusion of object-level appearance information gathered from these diverse perspectives particularly crucial. This capability to integrate multi-view object-centric features allows the agent to build more robust and consistent understanding of instances over time, despite substantial changes in their observed appearance. Therefore, we design a instance update module that reuses and continually refines instance-centric embeddings.
We recognize that applying global cross-attention between all Nr current queries and the instance embeddings from the previous frame can introduce substantial noise. This occurs because background queries often form irrelevant associations with prior instance features, thereby degrading the fusion process. To instill explicit instance awareness and mitigate this, we introduce the distance-awarg Short- Term Memory (STM). Specifically, to filter out irrelevant interactions we adopt the distance-aware attention, which gates attention by Euclidean distance between instance centroids,
Qj K,K,,, diag(r.) (((-)))(()))))) VI-1, vd
Attn(Q', K,- Vt-1) Softmax
where Q E NNNxd denotes current instance queries, the memory key K₁- RN--xd and value Vt-1 N are derived from Qt-1, (t-l..)) N.N.N. stores pairwise centroid distances and T: [T...... TN.]" contains query-specific receptive-fiele scales. We predict these scales with shared linear layer, T Linear( QQ)). so each query adaptively narrows or widens its spatial scope. Large suppress attention to distant memory slots, encouraging local refinement, whereas small π retain a global context when necessary. By suppressing attention to remote regions and modulating each query's s receptive field, short-term memory yields temporally enhanced embeddings Q
3.4 Spatial Consistency Learning for Robust Association
As illustrated in Fig VFMs like SAM I frequently fragments single instance into several neighbouring masks. This fragmentation compromises effective fross-frame instance association. Previous methods 6 ignore this, resulting in degraded spatial coherence. To mitigate this gap, we introduce learning-based mask integration (LMI) at inference to merge high-affinity fragments and instance consistency mask supervision (ICMS) during training to apply one-to-many supervision.
Learning-Band -aandd Mask Integration. To recover coherent masks, we learn an affinity matrix that merges masks belonging to the same instance at every frame Given point cloud features P, and corresponding 2D masks set Mr, we can get query features Q₁ and position X, through
(Q., x.) Po. M. (m:)
N.
(5)
where Pool aggregates point features within each mask. We then predicts axis-aligned bounding boxes B, MLP Q N.6 since the boxes generated by corresponding 3D mask may not be a complete object 6 For each pair (i.j) we compute the affinity feature Eij and Ajj as in Sec. 3.2 We first perform hierarchical clustering to identify mask groups whose pairwise affinities Aij all exceed 6. We then merge the masks within each such group to form Mr, and finally re-pool features over these merged masks,
(Q', x;) = Pool (.).
(6)
The mask aggrregggtot module is invoked only at inference. During training we intentionally refrain from merging masks to leverage fragment diversity implicit data augmentation, detailed below.
Instance Consistency Mask Supervision. In addition to retaining fragmented masks as implicit data augmentation, each fragment can also provides complementary view of the same object. Therefore,
5

---

we supervise corresponding fragments (many) with each ground ground-truth truth instance (one). This strategy improves robustness to low-quality masks, yields more consistent predictions for fragmented queries, and simplifies duplicate removal, which is vital for instance association in Sec. 3.2
To formalise this supervision, consider ground-truth instance gk and its corresponding query
2x = {
qq
(((m))((g)) > 0.5),
[P(m.]
where P(.) denotes the pixel-set of a mask m; and its corresponding query q. We then enforce consistency across these fragments via
N LIN = Σ Σ (((()))
(8)
k=1 9.E
where / is the loss and Yk denotes the ground-truth. However, as shown in Tab. 8 naively replacing the original one-to-one loss with LINN leads to consistent drop in segmentation accuracy. Our experiments demonstrate that this modification erodes the model's capacity select the highest- quality fragment, which is supported by the self-attention mechanism. To satisfy both objectives, we configure the decoder two distinct branches. the first branch we enable self-attention and employ standard one-to-one supervision in order to preserve fragment selection capability. In the second branch we disable self-attention and apply one-to-many supervision order to strengthen robustness across diverse fragments. Notably, this dual-branch mechanism is active only during training and incurs no additional computational cost at inference time.
3.5 Loss Functions
Our framework is trained end-to-end by minimising:
L = Ceeg + Bttm Liem + Bagg Lagg
(9)
where the scalars B. weight the contribution of each term.
Segmentation loss Leeg. The dual-decoder architecture described in Sec. 3.4 yields three sub-losses:
Lang L1 + X Lin Lig:
(10)
where L111 enforces one-to-one assignment, LIN ensures consistency across masks through multi- target supervision, and Lbg penalizes background masks.
Long-term memory loss Lism. We introduce a matrix Bij, where Vij = query qi and track refer to the same ground-truth instance, and Yij otherwise. We compute one-to-one assignment n² by applying the Hungarian algorithm to the cost matrix - log Mj. The matching loss becomes
Lmatch
Σ
log My.
(11)
N
(I.))R
To generate the sigmoid gate Cj for confidence we add
Lcont N,N
Σ
Jun log Cj + +1 Muy log(1 Cy)]
(12)
The full long-term memory loss is then
Lutm Lmatch + Boonf boont:
(13)
Mask-aggregation loss Lagg. To supervise the affinity predictor in LMI, employ binary cross- entropy over positive pairs P and negative pairs N:
Lags
log
Au
1
Σ
log(1- - Aij.
(14)
W
(DDEP
(JDEN
Pairs whose masks overlap ground-trun truth instance over 50 % are positive, all others negative.

---

Table Class agnostag 3D instance segmentation results of different methods on ScanNet200 dataset.
Method
Present at
Type
VFM
AP
AP
AP25
FPS
SAMPro3D 46
20V5
Offline
SAM
18.0
32,8
56,1
Open3DIS 47
CV22'2024
Offline
GroundedSAM
34.6
43.1
48.5
SAI3D 22
CVVPR2002
Offline
SemanticSAM
28.2
47.2
67.9
SAM3D 19
2C23
Online
SAM
20.2
35.7
55.5
0.4
ESAM 66]
(CLR'2025
Online
SAM
42.2
63.7
79.6
0.7
AutoSeg3D (Ours)
Online
SAM
45.5
66.7
81.0
0.7
ESAM-E
ICLR'200 2025
Online
FMM
43.4
65.4
80.9
10.6
3 (Ours)
Online
FaMtSAM
46.2
67.9
81.7
10.1
Table 3D instance segmentation results of different methods on ScanNet and SceneNN datasets. denotes represent the results we reproduced following the official released config.
ScanNet
SceneNN
Method
Present at
Type
AP
APs0
AP25
AP
APau
AP20
TD3D 48
ICME'2004
Offline Offline
46.2
71.1
81.3
O
40
SVPR'2024
59.3
78.8
86.7
INS-ConV 38
CVPR'2022
Online
57.4
TD3D-MA
ICME' 2024
Online
39.0
60.5
71.3
26.0
42.8
59.2
ESAM 6
ICT.R'2025
Online
41.6
59.6
75.2
30.3
47.6
63.4
AutoSeg3D (Ours)
Online
43.4
62.5
77.4
33.1
52.6
63.8
4 Experiments
4.1
Experiment Settings
Following baseline ESAM (6). begin by training single-view perception model ScamNet(2000)255) a subset ScanNet200 77 with RGB-D frames. Then we fine-tune it on RGB-D sequences with full loss functions and randomly sample 8 RGB-D frames per scene cach training step. For the optimization settings, we use an AdamW optimizer with a learning rate of 0.0001 and weight decay of 0.05 and the batch size is 4. All experiments conducted using PyTorch on a single NVIDIA Tesla A100 GPU. Our experiments are conducted on ScanNet [8] ScanNet200 77, SceneNN [9, and 3RScan [10 datasets.
4.2 Comparison with State-of-the-arts
Results on ScanNet200 of Class-agnostic Setting. Tab. details the class-agnostic results on ScanNet200, demonstrating the superiority of our approach over existing state-of-the-re methods. Specifically, when SAM serves as the 2D segmentation model, our method achieves gains of 3.3 in AP, 3.0 in 50 and 1.4 in AP25 compared to the recent ESAM (6]. The consistent performance im- provements, even with a more lightweight 2D segmentation model such as FastSAM 50), underscore the effectiveness and generalizability of method.
Results on ScanNet and SceneNN. Following the experimental setup of ESAM (6, Tab. 2 reports the results of our method, which is trained on ScanNet and subsequently evaluated on both ScanNet and SceneNN to assess its generalization performance. The notable improvements across multiple evaluation metries and datasets strongly demonstrate the effectiveness and generalizability of our approach. Specifically, our method achieves significant gains of 1.8 in AP, 2.9 in APs0. and 2.2 in AP ScanNet evaluation compared ESAM.
Results on SceneNN and 3RScan. Tab. repoors the results our method, trained on ScanNer200 and evaluated on SceneNN and 3RScan, which again demonstrate its strong generalization capabilities. Our approach surpasses previous methods, achieving significantly higher AP5o and AP25 scores. This underscores the effectiveness and adaptability of our method for robotic applications.
7

---

Table Results of transferring different methods trained on ScanNet200 to SceneNN and 3RScan. "-E" indicates using FastSAM instead of SAM for 2D segmentation.
ScanNet200 >SceneNN
ScanNet200 +3RScan
Method
Present at
Type
AP
APau
AP20
AP
APso
AP20
SAMPro3D 46
3DV'2025
Offline
12.6
25.8
53.2
3.9
8.0
21.0
Open3DIS
CVPR 22024
Offline
18.2
32.2
48.9
9.5
21.8
47.0
SAI3D 22
CVPR 22024
Offline
18.6
34.7
65.7
8.1
16.9
37.0
SAM3D 19
ICCVW"2023
Online
15.1
30.0
51.8
6.2
13.0
33.9
ESAM 6
ICLR'20025
Online
28.8
52.2
69.3
14.1
31.2
59.6
A (Ours)
Online
29.7
53.6
71.9
16.0
32.4
60.7
ESAM-E 6
2C25
Online
28.6
50.4
71.0
13.9
29.4
58.8
AutoSeg3D (Ours)
Online
30.2
54.1
72.8
16.8
34.3
61.0
4.3 Ablation Studies and Further Analysis
Table 4: Component-wise ablation.
womponent-wise Ablation. To further investi- gate the effects of our designs, we conduct an ablation studies on the ScanNet200. As depicted in Tab. the introduction of long-term memory obtains gains of 2.5 and 2.9 in AP and APso (Ø) us. )), because of its effectiveness instance association. The integrating of short-term mem- ory enhances our model's ability to capture posi- tional and content details from previous frames, resulting in performance improvements of 1.3 and 0.9 in AP and APso respectively (3 vs. D).
LTM STM LMI ICMS AP
AP
AP2s
41.6
62.9
78.7
44.1
65.8
80.7
42.9
63.8
80.0
44.8
66.7
81.0
V
45.6
66.9
81.2
V
45.5
67.0
81.3
V
46.2 67.9
81.7
@ and @ proves the effectiveness of components
in the proved spatial consistency learning. The synergistic combination of all proposed elements
constitutes an effective tracking-centric 3D segmentation framework, as in
Long-Term Memory. As shown in Tab. 5 com-
Table 5: Ablation for LTM.
pared the baseline without LTM,, adding geo- metric and appearance cues leads to steady gains @@ us. @). Incorporating confidence estimation brings notable boost, while further combin- ing the recall mechanism achieves the highest scores, with AP increasing from 43.2 to 46.2 (+3.0), 30 from 64.8 to 67.9 1). and AP25 from 80.4 to 81.7 (+1.3) (ω @). The recall and confidence strategies enable the model to effectively handle challenging cases such long-term occlusions and ambiguous matches, resulting in more reliable temporal consistency throughout the sequence.
Strategy
AP
APs0
AP2s
1
w/o LTM
43.2
64.8
80.4
+ Geometric
43.7
65,4
80.5
+ Appearance
45.0
66.5
81.3
+ Confidence
45.5
67.2
81.5
Recall
46.2
67.9
81.7
Short-Teree Memory. As shown in Tab. 6 start- ing from the baseline without STM, solely in-
Table 6: Ablation for STM.
troducing cross-frame attention brings limited improvement due to potential noise from irrele- vant associations (2 us. D). By further incorpo- rating our distance-aware attention, which gates memory updates based on instance centroid dis- tances, we observe clear performance boost (3) D). Equipped with query- -peryfic receptive-
Strategy
AP
AP 50
AP15
w/o STM
44.7
66.4
81.3
+ cross
45.1
66.5
81.2
distance
45.8
67.5
81.6
scale
46.2
67.9
81.7
field scales, the final STM boosts AP from 44.7 to 46.2 (+1.5), APso from 66.4 to 67.9 (+1.5),
and AP25 from 81.3 81.7 (+0.4) (Ø us. @). These results demonstrate that explicitly modeling spatial proximity and adaptive receptive fields effectively suppresses noisy associations and enhances instance update accuracy dynamic scenes.

---

Scone
Gruud- GroundTTruth Truth
ESAM
Figure 2: Visualization of segmentation results on ScanNet200 dataset.
Table 7: Ablation for LMI.
Learning-Baser Mask Integration. As shown Tab. 7 applying Learning-Basee Mask Inte- gration (LMI) only during inference yields the best performance, improving AP from 45.6 to 46.2 (+0.6), AP from 66.9 to 67.9 (+1.0) (3 ws. D). By contrast, incorporating LMI during training degrades performance, early-stage inaccuracies introduce false mask fusions that hinder model convergence (2 ms. D).
Strategy
AP
APsu
AP25
w/o LMI
45.6
66.9
81.0
train and infer.
44.5
66.1
80.4
infer. only
46.2
67.9
81.7
Instance Consistency Mask Supervision. As shown in Tab. (8] introducing Instance Con- sistency Mask Supervision (IMCS) with dual branches and TopK=4 achieves the best perfor- mance, improving AP from 45.5 to 46.2 (+0.7) (Dvs.D). Here, setting K=4 indicating chosing the four masks exhibiting the highest similarity scores when compared to the ground truth of object. By contrast, the single-branch con- figuration incurs substantial drop in all metrics, underscoring the importance combining one-to-one and one-to-many supervision signals.
Table 8: Ablation of ICMS.
Strategy
Topk
AP
APsu
AP25
w/o ICMS
45.5
67.0
81.3
single-branch
4
44.2
65.8
80.6
dual-branch
46.2
67.4
81.4
dual-branch
46.2
67.9
81.7
dual-branch
46.1
67.3
81.3
dual-branch
8
46.1
67.2
81.4
Qualitative Analysis. We present a qualitative analysis conducted on the ScanNet validation set, with illustrative examples provided in Fig. 2 These results further substantiate the superior instance segmentation capabilities of our proposed model. The visualizations demonstrate that our model not only accurately segments target objects but also effectively rectifies over-segmented masks.
5 Conclusion and Limitation
Conclusion. In this paper, we present novel, tracking-centre framework for online, real-time, and fine-graine grainer 3D instance segmentation. By recasting the task as continuous instance tracking, our approach integrates Long-Term Memory for robust identity propagation, Short-Term Memory for immediate temporal context, and Spatial Consistency Learning to suppress over-segmentation. Extensive experiments multiple benchmarks demonstrate that our lightweight system achieves state-of-the-ar accuracy while maintaining real-time efficiency. Limitation. Both our and previous methods do not explicitly model relative motion of moving objects. We leave this for future research.

---

Acknowledgements
This work was supported in part by the Natural Science Foundation of China (Grant No. 6203601 1, 62422317, U22B2056, 62192782, 62503323), the Beijing Natural Science Foundation (Grant No. JQ22014, L223003). The work of Weiming Hu was also supported part by the Natural Science Foundation of China (Grant No. U2441241, U24A20331, 62202470, 62473363).
References
[1] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026 2023.
[2] Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang. Xiangyu Zhang, and Yiehen Wei. Motr: End-to-end multiple-object tracking with transformer. In European conference on computer vision, pages 659-675. Springer, 2022.
[3] Tim Meinhardt, Alexander Kirillov, Laura Leal- Taixe, and Christoph Feichtenhofer. Track- former: Multi-object tracking with transformers. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8844-8854, 2022.
[4] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8741-8750 2021.
[5] Xuewu Lin, Tianwei Lin, Zixiang Pei, Lichao Huang, and Zhizhong Su. Sparse4d: Multi-view 3d object detection with sparse spatial-temporal fusion. arXiv preprint arXiv:2211. 100888 2022.
[6] Xiuwei Xu, Huangxing Chen, Linqing Zhao, Ziwei Wang, Jie Zhou, and Jiwen Lu. Embodied- sam: Online segment any 3d thing real time. arXiv preprint arXiv:2408. 11811, 2024.
[7] David Rozenberszki, Or Litany, and Angela Dai. Language-grounded indoor 3d semantic segmentation in the wild. In European Conference on Computer Vision, pages 125-141. Springer, 2022.
[8] Angela Dai, Angel X Chang, Manolis Savva. Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly- 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839, 2017.
[9] Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh- Khoi Tran, Lap-Fai Yu, and Sai-Kit Yeung. Scenenn: A scene meshes dataset with annotations. In 2016 fourth international conference on 3D vision (3DV), pages 92-101. Ieee, 2016.
[10] Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Nießner. Rio:
3d object instance re-localization in changing indoor environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7658-7661, 2019.
[11] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and languul representation learning with noisy text supervision. In International conference on machine learning, pages PMLR, 2021.
[12] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-876888 PmL.R, 2021.
[13] Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, al. Dino-x: unified vision model for open- world object detection and understanding. arXiv preprint arXiv:2411.1434f 2024.
10

---

[14] Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.00 193, 2023.
[15] Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, Lei Zhang, and Jianfeng Gao. Segment and recognize anything at any granularity. In European Conference Computer Vision, pages 467-484. Springer, 2024.
[16] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In European conference on computer vision, pages 529-544. Springer, 2022.
[17] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
[18] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXivv2303..08774, 2023.
[19] Yunhan Yang, Xiaoyang Wu, Tong He, Hengshuang Zhao, and Xihui Liu. Sam3d: Segment anything in 3d scenes. arXiv preprint arXiv:2306.03908, 2023.
[20] Rumnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7020-7001 7000-703,2200 2023.
[21] Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann. Openmask3d: Open- vocabulary 3d instance segmentation. arXiv preprint wxXiv:2366.1366 13631, 2023.
[22] Yingda Yin, Yuzheng Liu, Yang Xiao, Daniel Cohen-Or, Jingwei Huang, and Baoquan Chen. Sai3d: Segment any instance in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3292-3302, 2024.
[23] Ji Hou, Angela Dai, and Matthias Nießner. 3d-sis: 3d semantic instance segmentation of rgb-d scans. In Proceedings of the IEEE/CVF conference on computer vision and pattenn recognition, pages 4421-4430, 2019.
[24] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings the IEEE/CVF international conference on computer vision, pages 6411-6420, 2019.
[25] Le Hui, Linghua Tang, Yaqi Shen, Jin Xie, and Jian Yang. Learning superpoint graph cut for 3d instance segmentation. Advances in Neural Information Processing Systems, 35:36804-36811 2022.
[26] Zhihao Liang, Zhihao Li, Songcen Xu, Mingkui Tan, and Kui Jia. Instance segmentation in 3d scenes using semantic superpoint tree networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2783-2792, 2021.
[27] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3d: Mask transformer for 3d semantic instance segmentation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 0216-8223. IEEE, 2023.
(28) Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 3075-3084, 2019.
[29] Lei Han, Tian Zheng, Lan Xu, and Lu Fang, Occuseg: Occupancy-awar 3d instance segmenta- tion. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 2940-2949, 2020.

---

[30] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni. and Andrew Markham, Randla-net: Efficient semantic segmentation of large-scale point clouds. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 11108-11117, 2020.
[31] Siqi Fan, Qiulei Dong, Fenghua Zhu, Yisheng Lv, Peijun Ye, and Fei-Yue Wang. Scf-net: Learning spatial contextual features for large-scale point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14504-14513,
2021.
[32] Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neumann. Sgpn: Similarity group proposal network for 3d point cloud instance segmentation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 2569-2578. 2018.
[33] Jiahao Sun, Chunmei Qing, Junpeng Tan, and Xiangmin Xu. Superpoint transformer for 3d scene instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 2393-2401, 2023.
[34] John McCormae, Ankur Handa, Andrew Davison, and Stefan Leutenegger. Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. In 2017 IEEE International Conference on Robotics and automation (ICRA), pages 4628 4635. IEEE, 2017.
[35] Gaku Narita, Takashi Seno, Tomoya Ishikawa, and Yohsuke Kaji. Panopticfusion: Online volumetric semantic mapping at the level of stuff and things. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4205-4212. IEEE, 2019.
[36] Shi-Sheng Huang, Ze- Yu Ma, Tai-Jiang Mu, Hongbo Fu, and Shi-Min Hu. Supervoxel convolu- tion for online 3d semantic segmentation. ACM Transactions Graphics (TOG), 40(3):1-15, 2021.
[37] Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, and Kai Xu. Fusion-aware point convolution for online semantic 3d scene segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4534-5443 2020.
[38] Leyao Liu, Tian Zheng, Yun-Jou Lin, Kai Ni, and Lu Fang. Ins-conv: Incremental sparse convolution for online 3d segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18975-18984, 2022.
[39] Yijie Tang, Jiazhao Zhang, Yuqing Lan, Yulan Guo, Dezun Dong, Chenyang Zhu, and Kai Xu. Onlineanyseg: Online zero-shot 3d segmentation by visual foundation model guided 2d mask merging. arXiv preprint arXiv:2503. 0009, 2025.
[40] Xiuwei Xu, Chong Xia, Ziwei Wang, Linqing Zhao, Yueqi Duan, Jie Zhou, and Jiwen Lu. Memory-based adapters for online 3d scene perception. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21604-21613, 2024.
[41] Rajesh PN Rao. A sensory-motor theory of the neocortex. Nature neuroscience, 27(7): 1235, 2024.
[42] Marina A Lynch. Long-term potentiation and memory. Physiological reviews, 84(1):87-136 2004.
[43] Randall C O'RReilly, Rajan Bhattacharyya, Michael D Howard, and Nicholas Ketz. Complemen- tary learning systems. Cognitive science, 38(6):1229: 220--128, 2014.
[44] Larry R Squire, Lisa Genzel, John T Wixted, and Richard G Morris. Memory consolidation. Cold Spring Harbor perspectives in biology, 78):10227 2015.
[45] Yadin Dudai, Avi Karni, and Jan Born. The consolidation and transformation of memory. Neuron, 88(1) 3:20-32, 2015.
[46] Mutian Xu, Xingyilang Yin, Lingteng Qiu, Yang Liu, Xin Tong, and Xiaoguang Han. Sampro3d: Locating sam prompts 3d for zero-shot scene segmentation. arXiv preprint arXiv:2311 177007 2023.
12

---

[47] Phuc Nguyen, Tuan Duc Ngo, Evangelos Kalogerakis. Chuang Gan, Anh Tran, Cuong Pham, and Khoi Nguyen. Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4018-4028, 2024.
[48] Jinglin Zhao, Debin Liu, Laurence T Yang. Ruonan Zhao, Zheng Wang. and Zhe Li. Td3d: rensor-based discrete diffusion process for 3d shape generation. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2024.
[49] Maxim Kolodiazhnyi, Anna Vorontsova, Anton Konushin, and Danila Rukhovich. Oneformer3d: One transformer for unified point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 20943-20953, 2024.
[50] Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv preprint arXiv:2306. 12156, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Sec. 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
Please refer to Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification:
Code and raw results will be publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpipps.ccppulli//uuuii wwwpc//ggue////eeeecoooc CoodeSubeiiieeee ssi asionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: It would be too computationally expensive to report Error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpt:htttt:ttte http://pp...ppp...../..cacaaaaaa
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and we confirm that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets used in this paper are credited and the license is respected.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Code and raw results will be publicly available upon acceptance and we will include details about training, license, limitations, etc.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research in this paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research in this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
19

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We have reviewed the LLM policy and confirm that the method development in this research does not involve LLMs as any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: postps:/nneuripeee /ttprippp...c/oooofeeeee/e////2 for what should or should not be described.
20


Reviews:
[{"review_id": "Bp2mYMBcIA", "reviewer": "Reviewer_Xyr3", "summary": "This paper reframes online 3D segmentation as an instance tracking task to enable identity-aware temporal reasoning for embodied agents. By leveraging object queries for long- and short-term temporal association and introducing spatial consistency learning, the method addresses over-segmentation and enhances holistic object understanding across frames.", "questions": "1. Has the method been profiled against ESAM in terms of inference latency and memory usage under a standard embodied setup?\n\n2. What are the conceptual and implementation-level differences between the proposed long-term memory mechanism and ESAM’s query merging?\n\n3. How sensitive is the learned mask integration (LMI) threshold to variation in fragment granularity? Is it feasible to learn this threshold adaptively rather than using fixed heuristics?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is clearly written and easy to follow, with good organization of methodology and experiments.\n2. Tackling instance-level 3D segmentation in the context of embodied AI is both meaningful and timely, especially in light of recent progress in open-vocabulary segmentation.\n3. The proposed method demonstrates competitive performance across benchmarks, with thorough ablations validating each module’s contribution.\n\nWeaknesses:\n1. In embodied scenarios, real-time inference is crucial for downstream decision-making and interaction. However, the paper lacks a direct runtime comparison with prior approaches such as ESAM or point cloud reconstruction-based methods, making it difficult to assess its practical efficiency in deployment settings.\n2. The dual-memory system (LTM + STM) introduces architectural complexity that could make training and inference more difficult, especially when deployed in real-time systems.\n3. Although the Learning-based Mask Integration (LMI) is supervised during training, the actual merging still relies on a manually chosen threshold for clustering, which may be brittle across scenarios.", "was_revised": true, "final_justification": "The authors have addressed my main concerns regarding runtime, architectural complexity, the conceptual differences with ESAM’s query merging, and threshold robustness. I am inclined to accept this paper.", "rebuttal": "Thank you for your detailed review and thoughtful comments. We’re very grateful to see your recognition of our writing quality, the significance of our research, and our performance.\n\n**Q1: Runtime comparison and memory usage with ESAM or** **point cloud** **reconstruction-based methods.**\n\n**A1:** We apologize for any confusion caused. \n\n1. In Table 1 of the manuscript, we **have** already provided a comparison of end-to-end inference runtime. \n\n2. To offer further clarity, the table below presents a more detailed analysis of efficiency and resource requirements. We infer that the \"point cloud reconstruction-based methods\" mentioned by the reviewer refer to offline paradigms, such as SAI3D in our Tab.1 of the paper. Given that these are not online streaming processes, we computed their average FPS across an entire sequence for fair comparison. As illustrated in the table, our method incurs only a negligible computational cost compared to the baseline ESAM while achieving substantial performance improvements, thereby demonstrating both the efficiency and effectiveness of our approach.\n\n   | Method   | AP↑      | AP₅₀↑    | AP₂₅↑    | FPS↑     | Memory    | Training Time(GPU hours) |\n   | -------- | -------- | -------- | -------- | -------- | --------- | ------------------------ |\n   | SAI3D    | 28.2     | 47.2     | 67.9     | 0.24     | 8052M     | training free            |\n   | ESAM-E   | 43.4     | 65.4     | 80.9     | 10.6     | 2932M     | 61h                      |\n   | **Ours** | **46.2** | **67.9** | **81.7** | **10.1** | **3016M** | **66h**                  |\n\n**Q2: Discuss the architectural complexity and deployment for the proposed \"LTM + STM\".**\n\n**A2:** We appreciate the reviewer's concern regarding the potential overhead introduced by our dual-memory system. To address this, we highlight the following key aspects:\n\n1. **Maintained Complexity with Negligible Overhead**: Our dual-memory system seamlessly integrates long-term memory (LTM) and short-term memory (STM) into the original end-to-end network, resulting in only negligible increases in training time and inference latency relative to the baseline ESAM. This design fully preserves real-time performance, as evidenced by the detailed comparisons in the table provided in our response to Q1.\n\n2. **Ease of Deployment**: Both LTM and STM rely solely on simple linear projections and scaled dot-product attention mechanisms, introducing no additional dependencies. Furthermore, since the memory allocation of LTM is statically bounded at initialization, it eliminates the need for dynamic memory allocation, thereby facilitating straightforward deployment.\n\n3. **Opportunities for Further Efficiency Gains.**\n\n   - **Model‑architecture simplification**\n\n     From the table below, it is clear that the backbone network accounts for the majority of inference cost. Therefore, we first focus on reducing its computational burden. For example, we can decrease the number of sparse convolution layers to lower compute requirements and increase the voxel size to reduce the total number of voxel features, both of which reduce downstream computation.\n\n     | FASTSAM | Backbone | Decoder | LTM  | STM  | LMI  | Total |\n     | ------- | -------- | ------- | ---- | ---- | ---- | ----- |\n     | 18ms    | 64ms     | 5ms     | 5ms  | 3ms  | 4ms  | 99ms  |\n\n   - **Precision** **quantization**\n\n     The current implementation uses 32‑bit floating‑point arithmetic. Converting parts of the network to 16‑bit or even 8‑bit precision can further decrease memory bandwidth and accelerate inference without substantial accuracy degradation.\n\nBuilding on the optimization schemes outlined above, we applied targeted speed enhancements and achieved a throughput of 14.1 FPS, nearly 1.4 times faster than ESAM. At the same time our approach continues to surpass ESAM in overall performance.\n\n| Method              | AP↑  | AP₅₀↑ | AP₂₅↑ | FPS↑ |\n| ------------------- | ---- | ----- | ----- | ---- |\n| ESAM                | 43.4 | 65.4  | 80.9  | 10.6 |\n| Ours                | 46.2 | 67.9  | 81.7  | 10.1 |\n| Ours+fp16           | 44.8 | 66.3  | 81.2  | 13.6 |\n| Ours+voxel size*1.5 | 44.4 | 65.7  | 81.0  | 14.1 |\n\n**Q3: Threshold of Learning-based Mask Integration (LMI).**\n\n**A3:** In practice, we utilize a fixed threshold across **all** test scenarios, which consistently delivers robust results. To minimize false positives, we deliberately set this threshold at a conservatively high level. Consequently, minor perturbations around this value exert only negligible impacts on overall performance. To empirically substantiate these assertions, we conducted evaluations of our method using thresholds ranging from 0.5 to 0.9. As shown in the table below, the results exhibit virtually identical AP scores across this range, thereby confirming the robustness of our LMI strategy to threshold variations.\n\n| Threshold     | AP↑  | AP₅₀↑ | AP₂₅↑ |\n| ------------- | ---- | ----- | ----- |\n| without LMI   | 45.6 | 66.9  | 81.0  |\n| 0.5           | 46.0 | 67.6  | 81.5  |\n| 0.6           | 46.2 | 67.4  | 81.4  |\n| 0.8           | 46.1 | 67.6  | 81.8  |\n| 0.9           | 46.0 | 67.4  | 81.8  |\n| 0.7 (current) | 46.2 | 67.9  | 81.7  |\n\n**Q4: Differences between the proposed long-term memory mechanism and ESAM’s** **query** **merging.**\n\n**A4:** We highlight two essential differences between our method and query merging in ESAM, one during training and the other at inference.\n\n1. **Training supervision：Direct tracking versus indirect feature alignment.** **Per-object precise supervision.** Our network learns with an explicit one-to-one matching loss. Each object instance is supervised to establish accurate correspondences across frames. **Set-based coarse supervision in ESAM.** Query merging compresses every instance into a single centroid vector. The loss then pulls centroids of the same object together while pushing different objects apart.\n\n   The feature alignment through centroids in query merging of ESAM can be problematic since: **a) Viewpoint and occlusion changes.** Consecutive LiDAR sweeps expose different surfaces of the same object. Forcing identical centroids penalizes legitimate variations. **b) Loss of spatial structure.** Averaging points erases intra-instance geometry, so the network cannot tell stable regions from naturally varying ones. **c) Over-regularization.** Aggressive centroid overlap reduces sensitivity to fine details, which harms accuracy in cluttered scenes.\n\n2. **Inference behaviour: Lifecycle management.** In ESAM, once a query is created, it remains in memory forever, even if it was a false positive. Without any expiry, these spurious entries keep matching in later frames and propagate errors. We introduce a simple but effective policy: a track that stays unmatched for more than T frames is removed from memory. By discarding stale or erroneous objects, we curb cascading false matches and deliver steadier online tracking.\n\nBeyond the gaps above, ESAM relies on hand-crafted rules to fuse appearance and set-level features. Our model instead adopts a learnable fusion module, allowing the network itself to decide the contribution of each cue.\n\n**Q5: Is the threshold in Mask integration (LMI) robust to fragment granularity and could it** **learn this threshold adaptively?**\n\n**A5:** We thank the reviewer for this insightful question.\n\n1. **Robustness to Fragment Granularity.**\n\n   The learned mask integration (LMI) threshold is not highly sensitive to fragment granularity. We use a single fixed threshold across datasets and categories without per‑scenario tuning, and observe consistent performance. This robustness comes from the fusion rule that combines geometric proximity with feature‑similarity scores, which reduces reliance on any one threshold. We further validate our method by sweeping the threshold and reporting per-class metrics over four representative categories, ordered from fine to coarse namely picture, chair, table, bed. Picture captures small planar instances that often break into many tiny fragments, chair contains multiple thin parts and yields medium to high fragmentation, table has a large contiguous surface with fewer thin elements, and bed is a bulky object that produces coarse fragments.  As shown in the table below, our method maintains strong results across different thresholds and fragment granularities, and it consistently outperforms the variant without LMI. These findings indicate that LMI is insensitive to granularity variations.\n\n   | Threshold     | picture↑ | chair↑ | table↑ | bed↑ |\n   | ------------- | -------- | ------ | ------ | ---- |\n   | without LMI   | 43.9     | 63.2   | 52.1   | 45.9 |\n   | 0.6           | 44.2     | 63.9   | 52.0   | 48.5 |\n   | 0.8           | 44.4     | 64.1   | 52.5   | 47.5 |\n   | 0.9           | 43.9     | 64.0   | 52.2   | 47.3 |\n   | 0.7 (current) | 44.4     | 63.6   | 52.9   | 48.4 |\n\n2. **Adaptive Threshold Learning.**\n\n   We appreciate this insightful suggestion and plan to explore two complementary strategies in the future work:\n\n   - **Statistical Separation:** Fit the confidence scores of *matched* versus *unmatched* fragments to two Gaussian distributions, then choose the threshold that maximizes their separation (e.g., via the Bayesian decision boundary).\n\n   - **Network‑Predicted Thresholds:** Extend the model so that, in addition to mask outputs, it predicts a per‑object threshold at inference time. We would train these thresholds end‑to‑end by evaluating merge quality, allowing each instance to adapt its own threshold based on scene context.\n\n\nThese future directions promise to make our mask‑fusion process even more flexible, while our current results already demonstrate strong robustness under fixed‑threshold operation. Thank the reviewers for this insightful idea!"}, {"review_id": "WAtPutFJlR", "reviewer": "Reviewer_iEi7", "summary": "This paper focuses on building a Vision Foundation Models(VFMs), aiming to do 3D segmentation in real-time. However, perception is an inherently dynamic process, rendering temporal understanding a critical yet overlooked dimension within these prevailing query-based pipelines. This deficiency in temporal reasoning can exacerbate issues such as the over-processing. To address this issue, the authors present a novel, tracking-centric framework for online, real-time, and fine-grained 3D instance segmentation. By recasting the task as continuous instance tracking, our approach integrates Long-Term Memory for robust identity propagation, Short-Term Memory for immediate temporal context, and Spatial Consistency Learning to suppress over-segmentation. Extensive experiments on multiple benchmarks demonstrate that our lightweight system achieves state-of-the-art accuracy while maintaining real-time efficiency.", "questions": "Q1: The authors should provide a more detailed running time experiment to illustrate how claimed ‘real-time’ the proposed method are able to reach? \n\nQ2: Could the authors provide more comparative experiments to further validate the stability and performance of the proposed method under other types of datasets?", "limitations": "This paper has no potential negative societal impact.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe authors reconceptualize online 3D segmentation as an instance tracking problem and their core strategy involves utilizing object queries for temporal information prapagation, where long-term instance association promotes the coherence of features and object identities, while short-term instance update enriches instant observations. Furthermore, the paper introduce spatial consistency learning to mitigate the fragmentation problem inherent in VFMs, yielding more comprehensive instance information for enhancing the efficacy of both long-term and short-term temporal learning. Experiment results further suggest their viewpoint.\n\nWeaknesses:\nDespite the article's claim that their method can maintain real-time throughput, balancing model complexity and computational efficiency remains a challenge, especially when deploying on resource-constrained devices. Furthermore, while the paper demonstrates promising performance on datasets such as ScanNet and SceneNN, its effectiveness on other types of datasets or in more complex and dynamic real-world environments is not yet clear. Specifically, under certain special conditions, additional adjustments or optimizations of the existing model may be required to ensure its efficacy.", "was_revised": true, "final_justification": "The rebuttal resolves most major concerns by providing per-module ablations and cross-dataset evaluations. While some module gains are modest and the temporal reasoning claim is partly indirect, these are minor issues compared to the promising empirical performance, broad generalization, and well-supported methodology. Given that the core issues have been satisfactorily addressed and the remaining reservations have limited impact on the overall contribution, I recommend an acceptance for this paper.", "rebuttal": "Thank you for your detailed review and thoughtful comments. We’re delighted by your recognition of our reconceptualization of online 3D segmentation as an instance‑tracking problem and our spatial consistency learning, as well as your positive evaluation of our experimental results.\n\n**Q1: Balancing model complexity with computational efficiency for deployment  and detailed running time.** \n\n**A1:** We apologize for any confusion caused. \n\n1. In Tab. 1 of the manuscript, we **have** already provided a comparison of end-to-end inference runtime. \n\n2. To offer further clarity, the table below presents a more detailed analysis of efficiency and resource requirements. We infer that the \"point cloud reconstruction-based methods\" mentioned by the reviewer refer to offline paradigms, such as SAI3D in our Tab.1 of the paper. Given that these are not online streaming processes, we computed their average FPS across an entire sequence for fair comparison. As illustrated in the table, our method incurs only a negligible computational cost compared to the baseline ESAM while achieving substantial performance improvements, thereby demonstrating both the efficiency and effectiveness of our approach.\n\n   | Method   | AP       | AP₅₀     | AP₂₅     | FPS↑     | Memory    | Training Time(GPU hours) |\n   | -------- | -------- | -------- | -------- | -------- | --------- | ------------------------ |\n   | SAI3D    | 28.2     | 47.2     | 67.9     | 0.24     | 8052M     | training free            |\n   | ESAM-E   | 43.4     | 65.4     | 80.9     | 10.6     | 2932M     | 61h                      |\n   | **Ours** | **46.2** | **67.9** | **81.7** | **10.1** | **3016M** | **66h**                  |\n\n3. **Opportunities for Further Efficiency Gains.**\n\n   - **Model‑architecture simplification**\n\n     From the table below, it is clear that the backbone network accounts for the majority of inference cost. Therefore, we first focus on reducing its computational burden. For example, we can decrease the number of sparse convolution layers to lower compute requirements and increase the voxel size to reduce the total number of voxel features, both of which reduce downstream computation.\n\n     | FASTSAM | Backbone | Decoder | LTM  | STM  | LMI  | Total |\n     | ------- | -------- | ------- | ---- | ---- | ---- | ----- |\n     | 18ms    | 64ms     | 5ms     | 5ms  | 3ms  | 4ms  | 99ms  |\n\n   - **Precision** **quantization**\n\n     The current implementation uses 32‑bit floating‑point arithmetic. Converting parts of the network to 16‑bit or even 8‑bit precision can further decrease memory bandwidth and accelerate inference without substantial accuracy degradation.\n\nBuilding on the optimization schemes outlined above, we applied targeted speed enhancements and achieved a throughput of 14.1 FPS, nearly 1.4 times faster than ESAM. At the same time our approach continues to surpass ESAM in overall performance.\n\n| Method              | AP↑  | AP₅₀↑ | AP₂₅↑ | FPS↑ |\n| ------------------- | ---- | ----- | ----- | ---- |\n| ESAM                | 43.4 | 65.4  | 80.9  | 10.6 |\n| Ours                | 46.2 | 67.9  | 81.7  | 10.1 |\n| Ours+fp16           | 44.8 | 66.3  | 81.2  | 13.6 |\n| Ours+voxel size*1.5 | 44.4 | 65.7  | 81.0  | 14.1 |\n\n**Q2: Effectiveness on other types of datasets.** \n\n**Q2:** We evaluated our model on Matterport3D [1] dataset. Experimental results show that our approach consistently outperforms ESAM. These findings attest to the strong generalization ability of our method.\n\n| Method   | AP↑      | AP₅₀↑    | AP₂₅↑    |\n| -------- | -------- | -------- | -------- |\n| SAI3D    | 21.5     | 38.3     | 59.1     |\n| ESAM     | 27.6     | 49.1     | 69.9     |\n| **Ours** | **30.2** | **52.1** | **72.8** |\n\n[1] Matterport3D: Learning from RGB-D Data in Indoor Environments"}, {"review_id": "TdkX5NzeS3", "reviewer": "Reviewer_jX3W", "summary": "The paper  presents a novel framework for real-time 3D instance segmentation by reframing the task as a continuous instance tracking problem. In the approach, three components are mentioned. 1. Long-Term Memory (LTM) that Maintains instance identities across frames. 2.Short-Term Memory (STM) that Enhances current frame’s embeddings using distance-aware cross-frame attention, emphasizing nearby instances to avoid noise. 3.Spatial Consistency Learning (SCL) the result achieved SOTA on achieves new state-of-the-art 90 results on ScanNet200, ScanNet, SceneNN, and 3RScan.", "questions": "While the authors provide thorough ablations, most components don't seem to significantly impact the results. Is there also a similar ablation for the cross-dataset setting in Table 3, such as models trained on ScanNet200 and evaluated on other datasets?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strength\n\nThe task is well-motivated, and the proposed design provides meaningful improvements over existing benchmarks. The framework demonstrates strong generalization to unseen datasets, supported by thorough ablation studies and efficient performance with higher FPS.\n\n\n### Weakness\n\nWhile the authors provide thorough ablation studies, many of the individual components yield only modest improvements, raising questions about the overall contribution of each module.\n\nAlthough terms like \"long-term\" and \"short-term\" emphasize the temporal reasoning aspect of the method, it's difficult to conclude from the results alone that the performance gains truly stem from improved temporal understanding. The improvements could potentially be attributed to added complexity or auxiliary modules, rather than genuine temporal modeling.\n\nFig 1 is not that easy to understanding i my opnion.", "was_revised": true, "final_justification": "Thank you for the detailed response, it solves most of my concern and I have updated my score accordingly.", "rebuttal": "Thank you for your detailed review and thoughtful comments. We greatly appreciate your positive evaluation of our task objectives, performance, generalizability, and efficiency.\n\n**Q1: Improvements of individual components.**\n\n**A1:** Thank you for the focus on component-level contributions.\n\n- **Summary.** Tab. 4 shows that our complete system raises the baseline from 41.6 AP to 46.2 AP, an absolute gain of 4.6 points. This represents roughly an 11% relative increase. \n\n- **Per‑module contributions.** \n\n  - **Architectural enhancements.**\n\n    1. **LTM:** **+2.5 AP** (41.6 → 44.1), with **AP₅₀ +2.9** and **AP₂₅ +2.0**.\n\n    2. **STM: +1.3 AP** (41.6 → 42.9), with **AP₅₀ +0.9** and **AP₂₅ +1.3**.\n\n    3. **LTM +STM**: **+3.2 AP** (41.6 → 44.8), with **AP₅₀ +3.8** and **AP₂₅ +2.3**.\n\n       These numbers indicate that both memories address distinct failure modes and that their combination yields a clear net improvement on a strong baseline. \n\n  - **Instance** **Consistency** **Mask Supervision (ICMS).**\n\n    Adding ICMS on top of LTM+STM yields **+0.8** **AP** (44.8 → 45.6) versus LTM+STM. As a supervision‑only change, ICMS does not add inference‑time computation, so this accuracy gain comes effectively at no runtime cost. \n\n  - **Learning‑Based Mask Integration (LMI).**\n\n    Adding LMI on top of LTM+STM yields **+0.7 AP** (44.8 → 45.5) versus LTM+STM. This module is model‑agnostic by design, and here it produces a reproducible boost without altering the backbone architecture. In fact, when integrated into ESAM, it delivers an additional +0.9 AP improvement.\n\n- **Cumulative and complementary gains of ICMS and LMI.** With the baseline of 44.8 AP, adding ICMS alone raises the score by 0.8 to 45.6 AP, while adding LMI alone raises it by 0.7 to 45.5 AP. When both modules are enabled, the score climbs by 1.4 to 46.2 AP, essentially the sum of their individual gains. The almost perfect additivity shows that ICMS and LMI complement each other, where each tackles a different source of error so neither one diminishes the impact of the other. Their benefits therefore accumulate, yielding a clear and meaningful improvement over the baseline.\n\n**Q2: Improvements of temporal reasoning instead of model complexity.**\n\n**A2:** We thank the reviewer for raising this point, it motivated us to include the ID-switch analysis to disentangle temporal reasoning from added capacity.\n\n1. Regarding model complexity, our approach adds only a modest number of extra parameters. As the table below illustrates, inference latency and GPU memory usage stay almost identical to the baseline. These findings confirm that our model size has not increased substantially.\n\n   | Method   | AP↑      | AP₅₀↑    | AP₂₅↑    | FPS↑     | Memory    | Training Time(GPU hours) |\n   | -------- | -------- | -------- | -------- | -------- | --------- | ------------------------ |\n   | ESAM-E   | 43.4     | 65.4     | 80.9     | 10.6     | 2932M     | 61h                      |\n   | **Ours** | **46.2** | **67.9** | **81.7** | **10.1** | **3016M** | **66h**                  |\n\n2. We agree that labels like long‑term and short‑term emphasize temporal reasoning, yet accuracy gains alone do not prove stronger temporal understanding. To separate temporal effects from added capacity, we evaluate tracking consistency with **ID‑switch**. ID‑switch is the CLEAR MOT [1] metric that counts how often a ground‑truth trajectory matched to tracker ID A at time t becomes matched to a different tracker ID at time t+1 under optimal one‑to‑one matching, so fewer ID‑switches imply more stable cross‑frame identity. In our results, ID‑switch drops markedly, which is direct evidence that the gains come from enhanced temporal modeling rather than auxiliary complexity.\n\n   | **Method** | **ID-Switch↓** |\n   | ---------- | -------------- |\n   | ESAM       | 1899           |\n   | **Ours**   | **1707**       |\n\n[1] Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics\n\n**Q3: Details of Fig. 1.**\n\n**A3:** We apologize for any confusion caused by Fig. 1. We will improve this illustration in the revision. \n\n1. **Temporal Learning.** Figure 1 presents a comparison of ESAM and our method. In ESAM (top), the single‑frame query $Q_t$ is sent directly to the decoder and produces per‑frame predictions without any cross‑frame memory. In our pipeline (bottom row), each frame first passes through the Short‑Term Memory (STM) block before decoding, which brings in information from the previous frame. \n2. **Tracking and Matching.** Our method also employs explicit one-to-one assignments for robust tracking.  Red dotted links indicate one‑to‑one assignments to ground truth, and the small GT panel on the right collects these per‑frame pairings that provide supervision. The Long‑Term Memory (LTM) panel depicts a memory bank that persists across $t$, $t+1$, and $t+2$. Objects that are matched in the current frame are updated, as marked by the dotted “Updating” annotation, while unmatched objects are carried forward. Consistent colors denote the same object across time. Together, STM refines current queries using immediate temporal context and LTM preserves identity through time, providing explicit temporal modeling that is absent in ESAM.\n\n**Q4: Impact of each module and ablation for cross-dataset setting.**\n\n**A4:** Thank you for this insightful question.\n\n1. **Impact of each module.** For the performance gains contributed by each module, please refer to our response to **Q1**.\n\n2. **Ablation for cross-dataset setting.**  We have already reported an overall ScanNet200→SceneNN transfer in the manuscript . The sustained performance arises from tracking that models feature‑level correspondence and identity consistency, which are transferable across datasets. \n\n   To further demonstrate each module’s generality, we added a cross‑dataset evaluation in which models are trained on ScanNet200 and evaluated on SceneNN without any fine‑tuning.  We keep the backbone, training schedule, and inference protocol unchanged to ensure a fair, apples‑to‑apples comparison. Across this ScanNet200→SceneNN transfer, each module continues to yield consistent improvements over the corresponding ablated variant. We will include the full table (with the same metrics as Tab. 3) in the revision. These results support that the gains are not dataset‑specific and that the proposed components improve robustness under distribution shift.\n\n   | Setting | LTM  | STM  | LMI  | ICMS | AP↑  | AP₅₀↑ | AP₂₅↑ |\n   | ------- | ---- | ---- | ---- | ---- | ---- | ----- | ----- |\n   | ①       | –    | –    | –    | –    | 24.3 | 44.8  | 63.9  |\n   | ②       | ✓    | –    | –    | –    | 27.0 | 49.2  | 68.4  |\n   | ③       | –    | ✓    | –    | –    | 25.6 | 46.7  | 66.9  |\n   | ④       | ✓    | ✓    | –    | –    | 27.8 | 51.1  | 70.2  |\n   | ⑤       | ✓    | ✓    | –    | ✓    | 28.4 | 52.1  | 70.8  |\n   | ⑥       | ✓    | ✓    | ✓    | –    | 28.8 | 52.4  | 71.0  |\n   | ⑦       | ✓    | ✓    | ✓    | ✓    | 29.7 | 53.6  | 71.9  |"}, {"review_id": "ArAcG87t8e", "reviewer": "Reviewer_XxuF", "summary": "This work aims to solve the online 3D instance segmentation problem. A main problem of this problem is to merge inter-frame 3D masks if they belong to the same object. To solve this problem, this paper proposes to model the instance merging problem as object tracking. It introduces long-term and short-term memory for object tracking. The results on several benchmarks show this method achieves stateof-the-art performance while being efficient.", "questions": "My major concern about this work is the poor writing quality, which makes me really hard to understand what is the main contribution of the proposed method. I think the order of approach section should be totally re-structured to improve the readability. Currently, this paper has not met with the standard of NeurIPS.\n\nFor rebuttal, I hope the authors can solve the questions I raised in Weakness part.", "limitations": "yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n1. The online 3D segmentation problem is meaningful. This method achieves SOTA performance with >10 FPS, which has good application value.\n2. Modeling the instance merging problem as tracking makes sense. This idea achieves better performance than the query-based similarity computation in EmbodiedSAM.\n\n# Weakness\n1. The writing is bad. First, no preliminary is provided. The authors do not explain what is online instance segmentation and why it is important to merge inter-frame instances. Second, the order of section 3.2/3.3/3.4 is totally different from the order in Figure2. Third, the approach section is more like the combination of several operations, while the motivation is unclear. Besides, it seems this method is quite similar to ESAM. So the authors should clarify which modules are proposed in ESAM, and which modules are proposed by this paper.\n2. The proposed method is slightly slower than ESAM. Is this method has the potential to be applied on edge device with real-time inference speed? Is there any space to improve the efficiency of this method?", "was_revised": true, "final_justification": "Thanks for the authors' rebuttal. Overall I think this paper studies a valuable problem and achieves satisfactory performance. My main concern is about the writing quality and theoretical comparison with previous method like EmbodiedSAM, as well as clarification on the motivation of each proposed module.\n\nAccording to the rebuttal, I suggest to add A4 and A5 in a revised version to make this paper more readable. I will raise my score to borderline accept.", "rebuttal": "Thank you for your detailed review and thoughtful comments. We’re grateful that you endorse our approach of modeling instance merging as a tracking task and acknowledge our method’s performance and efficiency.\n\n**Q1: Preliminary of online 3D segmentation.**\n\n**A1:** We study streaming 3D semantic segmentation where each time step provides an RGB image with its aligned LiDAR point cloud. We consider two regimes.\n\n**Offline 3D segmentation.** The model receives the entire spatio-temporal sequence before inference and predicts segmentations for all frames in one pass. It can exploit global temporal context and is not bound by real-time constraints on latency or memory.\n\n**Online 3D segmentation.** The model processes a stream. At each step it sees only the current image and point cloud and may carry a compact state from previous steps. It must output labels for the current frame immediately. The procedure is causal and must meet strict per frame latency and memory budgets.\n\n**Q2: Importance of merging inter‑frame instances.**\n\n**A2:** Thanks for the suggestion, we clarified the importance below.\n\n- **Merging inter-frame instances in LTM.** Enforcing consistent object IDs across frames is essential to reconstruct a complete scene segmentation. Without ID continuity, per‑frame predictions cannot be merged into a coherent multi‑frame output.\n- **Merging inter-frame instance features in STM.** Aggregating feature representations over recent frames enriches each object’s descriptor. Historical context improves segmentation accuracy by reinforcing correct object boundaries and reducing noise.\n\n**Q3: Section order.**\n\n**A3:** Thank you for pointing this out. We agree that aligning the section order with the pipeline shown in Fig.2 (i.e., the inference flow) will improve readability.\n\n- **Planned revision.** In the revised manuscript, we will re‑sequence Sections 3.2–3.4 to mirror the left‑to‑right flow of Figure 2 and add explicit step labels in both the subsection titles and the figure to ensure one‑to‑one correspondence. We will also add forward references to Figure 2 at the start of each subsection.\n- **Rationale for the current organization.** Our present ordering follows how the system was actually constructed: we first introduce the tracking backbone (LTM), then present STM, Learning‑Based Mask Integration, and Instance Consistency Mask Supervision as targeted extensions built on that tracking framework. To make this design logic transparent, we will add a short “Motivation & Roadmap” paragraph at the beginning of Sec. 3 explaining this build‑up while making clear that the exposition now follows the inference pipeline depicted in Fig. 2.\n\n**Q4: Motivation of each module.**\n\n**A4:** Thanks to the reviewer for asking this.\n\n- **LTM as core tracking motivation.** We reformulate online segmentation as a tracking problem, using long‑term ID associations to link object proposals over time.\n- **STM for temporal enhancement.** Tracking is inherently a temporal modeling task. STM injects recent frame features into the current frame to improve both tracking stability and segmentation quality.\n- **LMI and ICMS for enhanced segmentation.** In online 3D segmentation, the matching process is highly dependent on each frame’s segmentation outputs, so redundant or erroneous segments can trigger a cascade of errors. To address this, we integrate Learning‑based Mask Integration (LMI) and refine single‑frame masks with Instance Consistency Mask Supervision (ICMS), which reduces over‑segmentation and delivers cleaner, less noisy 3D segmentation results to the tracker.\n\n**Q5:  Difference from ESAM and main contributions.**\n\n**A5:** Our approach departs from ESAM in how it learns identity over time, how it manages memory, and how it fuses and consolidates segmentations. The central idea is explicit identity supervision with stable lifecycle control and learnable fusion. This design yields stronger temporal reasoning while keeping inference nearly unchanged.\n\nShared Components. We share only the backbone network and the decoder head with ESAM. Importantly, ESAM's backbone comes from the prior Online3D [1] work rather than being their original design.\n\n**Core idea.** We recast online 3D segmentation as continuous instance tracking, where each VFM‑derived mask becomes a track query. This tracking view enables stronger temporal reasoning, robust memory management, and principled handling of fragmented masks.\n\n**Key differences and contributions.**\n\n1. **Different training supervision: direct tracking (ours) versus indirect feature alignment (ESAM).** We apply an explicit one-to-one matching mechanism/loss. For each object instance, this loss supervises the network to produce precise correspondences across frames. In contrast, ESAM's set-based supervision collapses each instance's features into a single centroid vector. It then encourages centroids of the same object across frames to align while repelling all others. (see **Xyr3-Q4** for more discussion)\n2. **Different lifecycle management.** Once ESAM instantiates an instance query, even if it represents a false detection, it stays in memory indefinitely. This lack of an expiration mechanism lets spurious detections persist and spread matching errors to later frames. Our method incorporates a simple yet effective lifecycle policy. Any tracked object that goes unmatched for more than T frames gets automatically removed from long-term memory. This approach stops stale or erroneous entries from accumulating, which cuts down on cascading false matches and boosts overall tracking stability in continuous real-time settings.\n3. **Learnable fusion for inter-frame object matching.**  During inter-frame object matching, ESAM relies on manually designed fusion rules to integrate appearance features with geometric information. In contrast, our approach uses a learnable fusion mechanism that enables the model to adaptively weight each source of information.\n4. **Novel instance-Level Temporal Information.** Unlike the original ESAM, our Short-Term Memory module refines each instance's embedding through distance-aware cross-frame attention. This process injects fine-grained temporal context while suppressing background noise.\n5. **Novel Learnable Mask Integration Module.** We introduce a learnable LMI to address within-frame fragmentation during inference. It estimates affinities among fragments, clusters and merges high-affinity groups, and then re-pools features to restore coherent instances. LMI operates only at inference with minimal overhead. During training, we leave fragments unmerged to serve as implicit augmentation.\n6. **Different Training Strategy.** We propose Instance Consistency Mask Supervision alongside a dual-branch decoder that separates two objectives. In the selection branch, self-attention remains enabled, and a one-to-one loss preserves the optimal fragment. In the consistency branch, self-attention stays disabled, and one-to-many supervision stabilizes fragmented queries. Both branches activate only during training, so inference costs stay the same. As a result, the model gains robustness to fragmentation while keeping precise assignments.\n\n**Why this matters.** Together, LTM and STM reduce drift and false matches by combining long‑horizon memory with fine‑grained temporal refinement. LMI fixes within‑frame fragmentation when it actually hurts outputs, and ICMS improves mask quality without runtime penalty. Ablations in the paper show that each component provides a meaningful, non‑redundant gain and that the improvements accumulate, leading to a substantial overall lift on online 3D segmentation.\n\nWe will integrate these clarifications and reorganize the manuscript as needed.\n\n[1] Memory-based Adapters for Online 3D Scene Perception, CVPR 2024\n\n**Q6: Potential to be applied to edge devices and improve efficiency.**\n\n**A6:** We appreciate this insightful question.\n\n1. **Potential for Edge‑Device Deployment.**\n\n   Our approach relies exclusively on widely supported operators such as attention layers and convolutions. Its end‑to‑end inference latency remains close to that of ESAM, indicating strong potential for real‑time execution on resource‑constrained edge devices.\n\n2. **Opportunities for Further Efficiency Gains.**\n\n   - **Model‑architecture simplification**\n\n     From the table below, it is clear that the backbone network accounts for the majority of inference cost. Therefore, we first focus on reducing its computational burden. For example, we can decrease the number of sparse convolution layers to lower compute requirements and increase the voxel size to reduce the total number of voxel features, both of which reduce downstream computation.\n\n     | FASTSAM | Backbone | Decoder | LTM  | STM  | LMI  | Total |\n     | ------- | -------- | ------- | ---- | ---- | ---- | ----- |\n     | 18ms    | 64ms     | 5ms     | 5ms  | 3ms  | 4ms  | 99ms  |\n\n   - **Precision** **quantization**\n\n     The current implementation uses 32‑bit floating‑point arithmetic. Converting parts of the network to 16‑bit or even 8‑bit precision can further decrease memory bandwidth and accelerate inference without substantial accuracy degradation.\n\nBuilding on the optimization schemes outlined above, we applied targeted speed enhancements and achieved a throughput of 14.1 FPS, nearly 1.4 times faster than ESAM. At the same time our approach continues to surpass ESAM in overall performance.\n\n| Method              | AP↑  | AP₅₀↑ | AP₂₅↑ | FPS↑ |\n| ------------------- | ---- | ----- | ----- | ---- |\n| ESAM                | 43.4 | 65.4  | 80.9  | 10.6 |\n| Ours                | 46.2 | 67.9  | 81.7  | 10.1 |\n| Ours+fp16           | 44.8 | 66.3  | 81.2  | 13.6 |\n| Ours+voxel size*1.5 | 44.4 | 65.7  | 81.0  | 14.1 |"}]

Author rebuttal:

Meta-review: All reviewers recommend accepting this paper, congratulations! 

The paper addresses online 3D segmentation by formulating it as an instance tracking task. It leveraged both long-term and short-term memory mechanisms and spatial consistency learning. This design yields strong performance on ScanNet, SceneNN, and related benchmarks, with thorough ablations and high FPS demonstrating promising application potential.

The main concerns raised were around architectural complexity and its derived consequence on deployment. Although a simpler method is more preferable, this does not prevent the work meeting the criteria for acceptance. As noted by the reviewers, please clarify some writing parts. 

Overall, the contributions are meaningful and timely. I recommend acceptance, with revisions to improve clarity, efficiency discussion, and positioning relative to prior work.
Decision: Accept (poster)