Title: Online Segment Any 3D Thing as Instance Tracking
Venue: NeurIPS 2025
Accepted: yes
Keywords: Online 3D Segmentation

Embodied Intelligence
Abstract: Online, real-time, and fine-grained 3D segmentation constitutes a fundamental capability for embodied intelligent agents to perceive and comprehend their operational environments. Recent advancements employ predefined object queries to aggregate semantic information from Vision Foundation Models (VFMs) outputs that are lifted into 3D point clouds, facilitating spatial information propagation through inter-query interactions.
Nevertheless, perception, whether human or robotic, is an inherently dynamic process, rendering temporal understanding a critical yet overlooked dimension within these prevailing query-based pipelines. This deficiency in temporal reasoning can exacerbate issues such as the over-segmentation commonly produced by VFMs, necessitating more handcrafted post-processing. Therefore, to further unlock the temporal environmental perception capabilities of embodied agents, our work reconceptualizes online 3D segmentation as an instance tracking problem (AutoSeg3D). Our core strategy involves utilizing object queries for temporal information propagation, where long-term instance association promotes the coherence of features and object identities, while short-term instance update enriches 
instant observations. Given that viewpoint variations in embodied robotics often lead to partial object visibility across frames, this mechanism aids the model in developing a holistic object understanding beyond incomplete instantaneous views. Furthermore, we introduce spatial consistency learning to mitigate the fragmentation problem inherent in VFMs, yielding more comprehensive instance information for enhancing the efficacy of both long-term and short-term temporal learning. The temporal information exchange and consistency learning facilitated by these sparse object queries not only enhance spatial comprehension but also circumvent the computational burden associated with dense temporal point cloud interactions. Our method establishes a new state-of-the-art, surpassing ESAM by 2.8 AP on ScanNet200 and delivering consistent gains on ScanNet, SceneNN, and 3RScan datasets, corroborating that identity-aware temporal reasoning is a crucial, previously underemphasized component for robust 3D segmentation in real-time embodied intelligence. Code is at https://github.com/AutoLab-SAI-SJTU/AutoSeg3D.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to solve the online 3D instance segmentation problem. A main problem of this problem is to merge inter-frame 3D masks if they belong to the same object. To solve this problem, this paper proposes to model the instance merging problem as object tracking. It introduces long-term and short-term memory for object tracking. The results on several benchmarks show this method achieves stateof-the-art performance while being efficient.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n1. The online 3D segmentation problem is meaningful. This method achieves SOTA performance with >10 FPS, which has good application value.\n2. Modeling the instance merging problem as tracking makes sense. This idea achieves better performance than the query-based similarity computation in EmbodiedSAM.\n\n# Weakness\n1. The writing is bad. First, no preliminary is provided. The authors do not explain what is online instance segmentation and why it is important to merge inter-frame instances. Second, the order of section 3.2/3.3/3.4 is totally different from the order in Figure2. Third, the approach section is more like the combination of several operations, while the motivation is unclear. Besides, it seems this method is quite similar to ESAM. So the authors should clarify which modules are proposed in ESAM, and which modules are proposed by this paper.\n2. The proposed method is slightly slower than ESAM. Is this method has the potential to be applied on edge device with real-time inference speed? Is there any space to improve the efficiency of this method?\n\nQUESTIONS:\nMy major concern about this work is the poor writing quality, which makes me really hard to understand what is the main contribution of the proposed method. I think the order of approach section should be totally re-structured to improve the readability. Currently, this paper has not met with the standard of NeurIPS.\n\nFor rebuttal, I hope the authors can solve the questions I raised in Weakness part.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
The paper  presents a novel framework for real-time 3D instance segmentation by reframing the task as a continuous instance tracking problem. In the approach, three components are mentioned. 1. Long-Term Memory (LTM) that Maintains instance identities across frames. 2.Short-Term Memory (STM) that Enhances current frame’s embeddings using distance-aware cross-frame attention, emphasizing nearby instances to avoid noise. 3.Spatial Consistency Learning (SCL) the result achieved SOTA on achieves new state-of-the-art 90 results on ScanNet200, ScanNet, SceneNN, and 3RScan.

STRENGTHS AND WEAKNESSES:
### Strength

The task is well-motivated, and the proposed design provides meaningful improvements over existing benchmarks. The framework demonstrates strong generalization to unseen datasets, supported by thorough ablation studies and efficient performance with higher FPS.


### Weakness

While the authors provide thorough ablation studies, many of the individual components yield only modest improvements, raising questions about the overall contribution of each module.

Although terms like "long-term" and "short-term" emphasize the temporal reasoning aspect of the method, it's difficult to conclude from the results alone that the performance gains truly stem from improved temporal understanding. The improvements could potentially be attributed to added complexity or auxiliary modules, rather than genuine temporal modeling.

Fig 1 is not that easy to understanding i my opnion.

QUESTIONS:
While the authors provide thorough ablations, most components don't seem to significantly impact the results. Is there also a similar ablation for the cross-dataset setting in Table 3, such as models trained on ScanNet200 and evaluated on other datasets?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on building a Vision Foundation Models(VFMs), aiming to do 3D segmentation in real-time. However, perception is an inherently dynamic process, rendering temporal understanding a critical yet overlooked dimension within these prevailing query-based pipelines. This deficiency in temporal reasoning can exacerbate issues such as the over-processing. To address this issue, the authors present a novel, tracking-centric framework for online, real-time, and fine-grained 3D instance segmentation. By recasting the task as continuous instance tracking, our approach integrates Long-Term Memory for robust identity propagation, Short-Term Memory for immediate temporal context, and Spatial Consistency Learning to suppress over-segmentation. Extensive experiments on multiple benchmarks demonstrate that our lightweight system achieves state-of-the-art accuracy while maintaining real-time efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe authors reconceptualize online 3D segmentation as an instance tracking problem and their core strategy involves utilizing object queries for temporal information prapagation, where long-term instance association promotes the coherence of features and object identities, while short-term instance update enriches instant observations. Furthermore, the paper introduce spatial consistency learning to mitigate the fragmentation problem inherent in VFMs, yielding more comprehensive instance information for enhancing the efficacy of both long-term and short-term temporal learning. Experiment results further suggest their viewpoint.\n\nWeaknesses:\nDespite the article's claim that their method can maintain real-time throughput, balancing model complexity and computational efficiency remains a challenge, especially when deploying on resource-constrained devices. Furthermore, while the paper demonstrates promising performance on datasets such as ScanNet and SceneNN, its effectiveness on other types of datasets or in more complex and dynamic real-world environments is not yet clear. Specifically, under certain special conditions, additional adjustments or optimizations of the existing model may be required to ensure its efficacy.\n\nQUESTIONS:\nQ1: The authors should provide a more detailed running time experiment to illustrate how claimed ‘real-time’ the proposed method are able to reach? \n\nQ2: Could the authors provide more comparative experiments to further validate the stability and performance of the proposed method under other types of datasets?\n\nLIMITATIONS:\nThis paper has no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reframes online 3D segmentation as an instance tracking task to enable identity-aware temporal reasoning for embodied agents. By leveraging object queries for long- and short-term temporal association and introducing spatial consistency learning, the method addresses over-segmentation and enhances holistic object understanding across frames.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is clearly written and easy to follow, with good organization of methodology and experiments.\n2. Tackling instance-level 3D segmentation in the context of embodied AI is both meaningful and timely, especially in light of recent progress in open-vocabulary segmentation.\n3. The proposed method demonstrates competitive performance across benchmarks, with thorough ablations validating each module’s contribution.\n\nWeaknesses:\n1. In embodied scenarios, real-time inference is crucial for downstream decision-making and interaction. However, the paper lacks a direct runtime comparison with prior approaches such as ESAM or point cloud reconstruction-based methods, making it difficult to assess its practical efficiency in deployment settings.\n2. The dual-memory system (LTM + STM) introduces architectural complexity that could make training and inference more difficult, especially when deployed in real-time systems.\n3. Although the Learning-based Mask Integration (LMI) is supervised during training, the actual merging still relies on a manually chosen threshold for clustering, which may be brittle across scenarios.\n\nQUESTIONS:\n1. Has the method been profiled against ESAM in terms of inference latency and memory usage under a standard embodied setup?\n\n2. What are the conceptual and implementation-level differences between the proposed long-term memory mechanism and ESAM’s query merging?\n\n3. How sensitive is the learned mask integration (LMI) threshold to variation in fragment granularity? Is it feasible to learn this threshold adaptively rather than using fixed heuristics?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}