Title: OnlineSplatter: Pose-Free Online 3D Reconstruction for Free-Moving Objects
Venue: NeurIPS 2025
Accepted: yes
Keywords: pose-free

online reconstruction

object understanding

3D reconstruction
Abstract: Free-moving object reconstruction from monocular video remains challenging, particularly without reliable pose or depth cues and under arbitrary object motion. We introduce OnlineSplatter, a novel online feed-forward framework generating high-quality, object-centric 3D Gaussians directly from RGB frames without requiring camera pose, depth priors, or bundle optimization. Our approach anchors reconstruction using the first frame and progressively refines the object representation through a dense Gaussian primitive field, maintaining constant computational cost regardless of video sequence length. Our core contribution is a dual-key memory module combining latent appearance-geometry keys with explicit directional keys, robustly fusing current frame features with temporally aggregated object states. This design enables effective handling of free-moving objects via spatial-guided memory readout and an efficient sparsification mechanism, ensuring comprehensive yet compact object coverage. Evaluations on real-world datasets demonstrate that OnlineSplatter significantly outperforms state-of-the-art pose-free reconstruction baselines, consistently improving with more observations while maintaining constant memory and runtime.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a feed-forward reconstruction method in an online manner. This paper focuses on the reconstruction of free-moving objects. It introduces a dual-key memory model to fuse new features with the current state (object representation) via the Gaussian field. The authors claimed that such a spatial-guided memory readout keeps both the quality of object reconstruction and the computational cost. The authors also conducted a lot of experiments. However, as I asked in the following sections, I believe all of the experiments are about objects with rigid motions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The authors introduced an online reconstruction method to reconstruct the free-moving objects.\n2. The authors proposed a spatial-guided memory readout to achieve both reconstruction quality and computational efficiency.\n3. The authors conducted extensive ablation studies.\n\n**Weaknesses**\n1. What is the difference between the Cut3r[1] and the proposed method? The Cut3r model also learn the state information. I will consider increasing my ratings if the authors can provide a reasonable answer.\n2. This paper is limited to rigid motion. I think models like Cut3r and VGGT[2] can already tackle such tasks. So the problem settings and the motivation for this task are not obvious to me.\n3. In L262, I highly suggest that the authors make their target clearer. I believe dynamic object reconstruction includes rigid motion and non-rigid motion. The authors should clarify which type their work belongs to.\n4. In the experiments, I do believe that the quantitative experiments are not enough. I highly suggest that the authors conduct more comparisons with the current SOTA methods like Monst3r[3], Cut3r, and VGGT. \n\n[1] Wang, Qianqian, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. \"Continuous 3D Perception Model with Persistent State.\" arXiv preprint arXiv:2501.12387 (2025).\n\n[2] Wang, Jianyuan, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. \"Vggt: Visual geometry grounded transformer.\" In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5294-5306. 2025.\n\n[3] Zhang, Junyi, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. \"Monst3r: A simple approach for estimating geometry in the presence of motion.\" arXiv preprint arXiv:2410.03825 (2024).\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nPlease refer to the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel incremental and feed-forward pipeline for reconstructing freely moving objects from a sequence of images, with the final output represented as a 3D Gaussian Splatting (3DGS). The method is designed to address two central challenges: (1) performing reconstruction from unposed objects without depth priors, and (2) maintaining constant computational cost regardless of video length, enabling online applicability. While the first challenge has been explored in prior work, this paper contributes a practical and efficient solution to the second. Additionally, enabling feed-forward generation of 3DGS in an online setting is both technically demanding and impactful. Experiments on real-world datasets validate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. This paper propose a novel feed-forward reconstruction pipeline characterized by a concise and effective design.\n\n2. The introduced dual-key memory module demonstrates strong performance and effectively addresses practical challenge in incremental reconstruction.\n\n3. The methodology is presented with clear and rigorous exposition, and the quantitative results convincingly validate the effectiveness of the approach.\n\nWeeknesses\n\n1. As stated in L30, the authors claim that prior methods struggle to capture instance-specific geometric details, limiting reconstruction accuracy. However, the proposed method does not appear to offer a clear advantage in capturing fine-grained geometry—apart from incorporating certain depth-based losses. This is particularly apparent when compared to 3DGS-based methods that benefit from explicit geometric initialization. Moreover, the authors do not report any quantitative metrics related to geometric accuracy (e.g., depth error, chamfer distance). It is therefore unclear whether the proposed method achieves better geometric fidelity or how it captures fine-grained geometry. If such improvement exists, further justification is needed. I would also encourage the authors to report geometric accuracy metrics to support their claim.\n\n2. After masking out the background, the task essentially becomes pose-free object reconstruction. In this case, it would be more appropriate to compare against other pose-free baselines, such as COLMAP-Free 3D Gaussian Splatting. Although these methods are optimization-based, for 3D asset generation, reconstruction quality and geometric accuracy is often more critical than reconstruction efficiency. Such comparisons would help position the contribution more clearly.\n\n3. The authors have not released the code, and the visualizations in the experiments are relatively limited. It is also unclear whether the test dataset includes objects that are entirely unseen in train dataset. The qualitative example of the \"blue kettle\" is not particularly convincing in terms of generalization. I would strongly encourage the authors to provide more diverse and challenging visual examples to support the claimed generalizability.\n\nQUESTIONS:\nPlease refer to the Weaknesses. \n\nIn particular, including videos that showcase the 3D Gaussian Splatting reconstruction results along with their corresponding input video sequences would provide more convincing evidence of the method’s performance. I would be happy to raise my score if the above concerns are adequately addressed during the rebuttal.\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a feed-forward method for object tracking and reconstruction in an online manner. The method operates only on RB frames and claims to be real-time. Moreover, the proposed method doesn't require any camera poses and references everything w.r.t. to the canonical representation in the first frame. Experimental validation is sufficient and shows new state-of-the-art performance on several datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The proposed method is novel, and the idea of using a feed-forward model for splatting a dynamic object reconstruction is interesting.\n* The whole pipeline seems to be well-engineered, e.g. runs in real-time, operates in an online manner, and requires only monocular RGB frames.\n* The method achieves state-of-the-art results on GSO and HO3D datasets, both in early, mid and late stage of object tracking. The improvement over other methods is significant. \n* The introduced dual-key memory module is an important contribution, together with the memory sparsification mechanism that drops tokens that are least useful.\n\nWeaknesses:\n* The method assumes that object masks as already known. This requires some user input and may limit some applications since it requires some user clicks and the video object segmentation model XMem.\n* Training still requires ground-truth poses, and overall the training is very supervised. This could be a problem for good generalization capabilities of this method.\n* The paper should also compare to other methods, such as [8].\n* Eq. (9) is confusing. Why is there a dot sign followed by a minus sign? Are brackets missing there? \n* There are many typos, missing letters, etc. The paper would benefit from proof-reading.\n\nQUESTIONS:\n1. Would a comparison to [8] be possible?\n2. Another ablation on the used VOS module would be interesting.\n3. Is it possible to extend the training procedure to avoid requiring ground truth camera poses? \n4. The method doesn't use any global optimisation, but it can benefit from it, e.g. 3DGS optimization on all frames. How much would this improve the results?\n\nLIMITATIONS:\nYes, the limitations are discussed in the supplementary materials.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}