OnlineSplatter: Pose-Free Online 3D Reconstruction for Free-Moving Objects
Authors: Mark He Huang
Lin Geng Foo
Christian Theobalt
Ying Sun
De Wen Soh
Venue: NeurIPS 2025 spotlight ()
Keywords: pose-free
online reconstruction
object understanding
3D reconstruction
Primary area: deep_learning
TL;DR: OnlineSplatter enables pose-free 3D reconstruction of free-moving objects from monocular video by fusing frame-wise features into a compact Gaussian field via a dual-key memory, with constant-time and memory efficiency.

Abstract: Free-moving object reconstruction from monocular video remains challenging, particularly without reliable pose or depth cues and under arbitrary object motion. We introduce OnlineSplatter, a novel online feed-forward framework generating high-quality, object-centric 3D Gaussians directly from RGB frames without requiring camera pose, depth priors, or bundle optimization. Our approach anchors reconstruction using the first frame and progressively refines the object representation through a dense Gaussian primitive field, maintaining constant computational cost regardless of video sequence length. Our core contribution is a dual-key memory module combining latent appearance-geometry keys with explicit directional keys, robustly fusing current frame features with temporally aggregated object states. This design enables effective handling of free-moving objects via spatial-guided memory readout and an efficient sparsification mechanism, ensuring comprehensive yet compact object coverage. Evaluations on real-world datasets demonstrate that OnlineSplatter significantly outperforms state-of-the-art pose-free reconstruction baselines, consistently improving with more observations while maintaining constant memory and runtime.

OnlineSplatter: Pose-Free Online 3D Reconstruction for Free-Moving Objects
Mark He Huanglo 1.3 Lin Geng Foo?, Christian Theobalt', Ying Sun', De Wen Soh
Singaapoee University of Technology and Design, Singapore Max Planck Institute for Informatics, Saarland Informatics Campus, Germany 3 Institute for Infocomm Research (PRR) Centre for Frontier Al Research, A * Singapore he www.huunnmmaali..cmmmmmm sutd .ut..edu..g.........eeduue
30 Coac: Doundera
8:11
1-2
1:4
1-5
ObteciManory
Tranofornee
Pass
OnlineSplatter
Onèra Recorafruction
Figure I: Outline of proposed OnlineSplatter. From the incoming stream of pose-free RGB frames, OnlineSplatter "splats" the observations into a canonical cloud of 3D Gaussians anchored to the first frame. Every new frame triggers a single, O(1) memory-and-Hn improves the reconstruction of the object. The system copes seamlessly with freely moving objects and requires neither pre-computed poses nor depth maps. The result is continually improving 3D representation suitable for real-time applications.
update that immediately
Abstract
Free-moving object reconstruction from monocular video remains challenging, par- ticularly without reliable pose or depth cues and under arbitrary object motion. We introduce OnlineSplatter. a novel online feed-forward framework generating high- quality, object-centric 3D Gaussians directly from RGB frames without requiring camera pose, depth priors, or bundle optimization. Our approach anchors recon- struction using the first frame and progressively refines the object representation through dense Gaussian primitive field, maintaining constant computational cost regardless of video sequence length. Our core contribution is dual-key memory module combining latent appearance-geomeery keys with explicit directional keys, robustly fusing current frame features with temporally aggregated object states. This design enables effective handling of free-moving objects via spatial-guidde memory readout and an efficient sparsification mechanism, ensuring comprehen- sive yet compact object coverage. Evaluations real-world datasets demonstrate that OnlineSplatter significantly outperforms state-of- the-art art pose-free reconstruc- tion baselines, consistently improving with more observations while maintaining constant memory and runtime. Project page: markhh. com/ inliinnnnnpppttrr
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Real-time 3D reconstruction of freely moving objects from monocular video remains a fundamental challenge in computer vision, with far-reaching implications for robotics, augmented reality, and Although recent work achieves impressive results on static scenes, real-world deployments are often more demanding: objects move firely, undergoing arbitrary rotations and translations while a moving camera observes them. This dynamic setting violates the static-scene assumptions, reliable pose or depth cues, that underpin most existing methods. The challenge is especially acute in online setting, where systems must update their understanding of objects as every new frame arrives, a capability essential for autonomous robots and AR devices operating in unpredictable environments.
interactive 3D applications [1 33 45
53
Recent advances in 3D object reconstruction follow two main paradigms. The first leverages diffusion-
based generative models and large-scale object-level priors [1 23 36 22 21
25 38 to synthesize
plausible 3D assets from single or a few images. While generating visually compelling geometry, these methods rely on their learned priors to hallucinate unseen parts of the object. Although beneficial for 3D asset generation, this approach limits their applicability in real time perception. The second paradigm comprises pointmap- boasteap-based regress pixel-aligned pointmaps from unposed images. Although effective for stationary scenes and accurate surface recovery, they falter when confronted with objects undergoing unrestricted motion.
methods [42 g 24 48
39
2 [1] that directly
common thread among existing approaches is their reliance on camera poses, depth information, or global optimization [20 9 3 44 Such limitations not only restrict their applicability in online settings but also hinder their performance environments where additional sensor data is unavailable.
Motivated by these challenges, we propose Online a feed-forward framework for online reconstruction of freely moving objects. Anchoring reconstruction in the canonical coordinate system defined by the first frame, our network predicts a dense field of per-pixel Gaussian primitives and refines the object model causally as new RGB frames arrive (Fig. DT. Moreover, control memory growth as observations accumulate, we propose an attention based memory module that fuses incoming frame features with a compact latent state, eliminating the overhead of bundle adjustment or additional data processing. Central to this design is dual-key memory retrieval strategy that combines appearance- geometry features with explicit spatial cues, providing robust temporal fusion and comprehensive spatial coverage.
Our contributions are: (i) novel feed-forward framework for object-centric online 3D reconstruction that operates on monocular RGB streams in
eliminating the need for camera poses, depth priors, or global optimization while maintaining constant computational complexity regardless of sequence length; (ii) dual-key memory module that pairs latent features with spatial cues for efficient sratial-tempore fusion and principled sparsification; (iii) extensive experiments on GSO 7 and HO3D [10] demonstrate state-of- offte-of-th-art the--att performance on freely moving objects, paving the way for practical online reconstruction in dynamic environments.
Related Works
Pose-free 3D Reconstruction. Eliminating camera pose as input remains a key challenge in 3D reconstruction. Classical pipelines such as COLMAP (30) recover structure and poses via incremental SfM with MVS. Recent methods like 3D Gaussian Splatting [16], when combined with joint- pose optimization 14 17 can handle unposed images but still require static scenes. These aptimization-hasse approaches fail to reconstruct moving objects due to their reliance on cross-frame correspondences, even when object masks are available. In contrast, our approach processes each frame causally in feed-forward manner, eliminating the need for cross-fram optimization while handling freely moving objects. Recent feed-forward methods like DUSt3R 42 predict aligned pointmaps from pairs of unposed images within shared coordinate space and merge them via global aligmment. While this avoids explicit optimization, DUSt3R and its variants (e.g., Spann3R [39). VGGT [40], etc. 8 (1)) still assume static scenes and treat moving objects as outliers due to low cross-frame consistency. Their implicit reliance on large background surfaces further limits their applicability in dynamic, online reconstruction environments. Our method differs by focusing solely on object reconstruction without requiring background information, making suitable for online reconstruction of freely moving objects.

---

Optimization-base based Object Reconstruction. Optimization sriimization-based based methods typically require global bundle adjustment across all frames, which poses fundamental limitations for real-time applications. Early approaches such as BARF 200 employ coarse-to-fine registration strategy to jointly optimize shape and pose for NeRF using unposed RGB frames. Similarly, Hampali et al. [9] propose a splitring-ande mergnng strategy tailored for in-hand object scanning, enabling reconstruction of freely moving objects. However, both methods rely on global bundle optimization over all frames, rendering them unsuitable for online, causal settings. Fmov [31] leverages a virtual camera model to narrow the camera pose search space during initial joint 3D shape and pose optimization, followed by global optimization across frames. BundleSDF 44] utilizes keypoint matching network and explicit frame selection strategy to optimize an object-centric neural signed distance field in a causal manner. While it can handle freely moving objects, it requires ground truth depth information input during optimization. In contrast, our method employs an attention-basee memory module that combines relative pose and latent features for purely feed-forward reconstruction, eliminating the need for depth input and computationally expensive cross-frame optimization.
Few-view Object Reconstruction. Recent advances in few-view reconstruction focus generating 3D assets from limited views. Earlier methods like FvOR [50] combine learned object pose priors with alternating pose-shape optimization to reconstruct unknown objects from just few images. FORGE [33 proposes a framework that jointly infers relative camera poses and fuses per-view 3D voxel features into neural radiance field, achieving
object reconstruction. However, these methods implicitly assume fully visible, centered objects and struggle with occlusions, sub- optimal observations, dense view sequences. More recent large-scale data-driven approaches [36] generate plausible geometry but typically focus on single- shot reconstruction and do not incorporate temporal observations for incremental refinement. In contrast, our framework continuously updates its representation as new frames arrive, yielding high-fidelity reconstructions that improve over time while remaining real-time and pose-freee
11 54
35
46
38
37
25 56
47
23 [2
3 Method
The goal of our method is to perform an online reconstruction of a freely moving rigid object using monocular RGB images without relying on known camera poses or any object prior (such as depth, shape, or category). The term "online" implies that our approach processes incoming data causally, updating the reconstructed object representation incrementally as new frames become available. Fig. 2 shows an overview of our proposed OnlineSplatter framework. In the following sections, we elaborate on the details of our framework's pipeline (Sec. B.1), the design of our dual-key 3D Object Memory (Sec. 3.2). and the training and inference procedure for our model (Sec. 3.3)
3.1 OnlineSplatter Pipeline
Image Encoding. At each timestep I, we observe an RGB view V., While most reconstruction methods [42) implicitly rely on static scenes and background surfaces for stability, our approach explicitly handles freely moving objects in isolation. To focus the object of interest, we leverage an off-the-shelf video object segmentation model XMem [3] to obtain the object mask M,, and apply it to each frame to remove the background. The masked frame V/ is then encoded into patch features via a hybrid strategy that concatenates two complementary encoders:
fot Concat (V7), Encoder; (V())
where Encoder is frozen DINO backbone [ 28 321, providing strong self self-supervised appearance cues. However, DINO alone lacks the 3D awareness required for accurate reconstruction. We therefore introduce a learnable counterpart, Encoder which adopts the same architecture but is trained mnd-to-end within OnlineSplatter to capture complementary geometric cues. This dual- design enables our model to leverage both rich visual priors and geometry-aware features.
OnlineSplatter Transformer. After encoding the input image at each timestep, these features are input into our proposed OnlineSplatter Transformer, which is designed to process three distinct types of tokenized inputs: reference-vier tokens Tin (encoded from the initial frame Vo), source-view tokens Tnn areet (encoded from the current frame V,), and memory-reudour tokens Tén mem. (readout from the object memory at time t). To differentiate and contextualize these tokens, we introduce
3

---

Object Memory (C gert yelg
to Trainable Weights
D value store Ee/88
value encoder
dual-kos B query
Restercation
readout
Bq F4l
G
readout
deccde talsi
Splanes Transflrmmr
(iD0) Streem
Greft
frome
Ro,
encade
dec006
VOS Module
inpatchiiy
G
Yi fraire
Res
Srel
Mask Bund
2T1
Figure 2: Overview of OnlineSplatter Pipeline. The input to our framework consists of a stream of RGB images {Vi where object masks {M, ) are generated and applied to remove background on-the-fly using an off-the-shelf online video segmentation (OVS) module running alongside our framework. At each timestep OnlineSplatter processes the input frame V, by first patchifying into patch tokens. These tokens are then fed into a transformer-basee architecture, which directly reasons and outputs pixel-aligned 3D Gaussian representations in a canonical space. Central to our method is object memory, implicit module based ross-attention, which is queried and updated at every timestep. This memory enables the incremental reconstruction of the object, consistently refining the object representation ob).t) new observations arrive in fully ferd-forward manner.
corresponding learnable embeddings to each the tokenized features. These embeddings are added with their respective encoded features and positional embeddings prior transformer processing.
Tin Bsew
fview gourb +
Vview {ref, src, mem]
Jsew:
Gaussian Decoding. The three types of token (T T sPa.c: The (em,.)) are processed jointly by our unified OnlineSplatter Transformer that produces the output tokens (Tref, Tare, Tout then decoded into 3D Gaussian parameters through trainable unpatchifier.
Tmem.e)s which
4 4N obj.t = (G2N f: Graf,t G Unpatchify((TTaa mem,ts Tout Tout
(3)
where each Gaussian set K consists of K Gaussian primitives defined as: (er, Pk; Sk, Ck. ) with = Н x W, corresponding to pixels per RGB input V E ##* x3, Note that, instead of predicting 3D primitives only for the newly observed frame each timestep and accumulating predictions over time, our method fetches current understanding of the object from the memory and refines it with the latest observation. Consequently, our approach avoids accumulation any explicit global aggregation operations entirely, which is advantageous for object-centric reconstruction as large observation overlapping is expected over time. Thus, each timestep. the decoded Gaussian (GAN ab collectively form the updated representation of the object.
Rasterization. During training, we employ differentiable rasterizer render images for photometric supervision. Specifically, at each timestep, given the predicted Gaussian representation (G the ground truth poses up to time we render + images {Ro 1iz- In addition, render the frame-level subsets G.Y f.t and G. using the poses of Vo and V,, respectively, producing Ro.: and Re.. These additional renders are critical for training stability, as they encourage each Gaussian subgroup specialize in reconstructing the portion of the object visible in the corresponding input view. This design, in turn, incentivizes the memory tokens to encode features that are most useful for reconstructing complete object representation.
objjt) ) and
3.2 Dual-Key 3D Object Memory
To enable object-centric online reconstruction, we need a memory bank that can store the most useful features of the object and can support producing a progressively better representation of the object at each timestep. Achieving this presents two primary challenges: (1) Naive accumulation of latent features in memory will result in ever growing memory bank, which is computationally demanding

---

and memory inefficient as more frames are observed. (2) Naive accumulation of predictions in the output space will result in redundant overlapping predictions, which requires additional optimization steps to consolidate and simplify the representation.
To address these limitations, we propose novel object-centric memory mechanism, Dual-Key 31> Object Memory, that consists of a key-value memory bank. Each memory entry consists of two keys and value, where the two keys facilitate robust readout of the most relevant memory values, via our dual-key memory readout approach. In this subsection, we first discuss how we encode the memory into the memory bank, followed by how we read from the memory. Lastly, we also introduce our memory sparsification mechanism that helps keep the memory bank compact yet effective.
Memory Encoding Unlike conventional single-key memories, our method leverages two comple- mentary keys-one latent key and one direction key. Intuitively, the latent key facilitates retrieval of information to guide visual-geometrical reasoning, while the direction key provides explicit spatial guidance to improve spatial coverage during memory reading. At every timestep i, we encode the latent key k,d") from our tokenized features using a lightweight learnable key encoder (Encoder") ") to capture fundamental risual-geomelee cues:
( = f. = Encoder (fuc)
(4)
We further leverage a lightweight pre- pre-trained zero-shot 3D orientation estimator [43 denoted as Encoder", to encode our direction key k(D), Specifically, k, is computed based on the R axis-rotation of the object orientation (0c, de, rt::
(D) == (sin Øe cos l.. sin Φe sin 0,, cos Φi) [fr, Φe. 7 = Encoder (V))
k
(5)
where we use the Azimuth (9c) and Polar (Øx) value from the prediction to convert to a unit directional
vector as our directional key k(D). Thus, k,t) and k(()) from Eq. and Eq. 5 are keys used to retrieve the suitable memory values. Note that all keys and values in our object memory are at the token level, thus kg(") E Bexe where is number of patches per view, we also broadcast k. from RIx3 to PR2 such that patches from the same view share the same directional key. Then, after OnlineSplatter produces output tokens at time I, we encode the output tokens to update the value Specifically, trainable value encoder (defined as Encoder takes output tokens Toee1 as input to produce the new value:
vi()) corresponding to the encoded keys in Eq. and Eq. S
(L) = f. = Encoder (Tout, sre,t)
(kk (C) (D) (D))). The stored in-memory dual-key-values as a whole are denoted as (P) E (88 P)x3 and K())) v(()) R(S-P)xC where S represents the maximum size set for the object memory.
Lastly, our dual-key pairs with the encoded value form new entry at time i:
Spatial-Guided Dual-Key Memory Reading Our memory module maintains a large collection of tokens that encode different perspectives of the object's appearance and geometry. However, using all memory tokens for every forward pass would be computationally expensive and potentially noisy. Instead, we need an efficient mechanism to select the most relevant memory features for reconstructing the object at each timestep.
While our latent key, derived from tokenized features through end-to-end training with 3D reasoning objectives, captures both visual and geometric information, relying solely on latent key-based attention may not be optimal for object-centric reconstruction. This is because our memory readout serves two critical purposes: (1) supporting prediction for newly observed regions, and (2) retrieving the most informative features for reconstructing the complete object. To address this dual objective, we introduce a directional key that provides explicit spatial guidance for memory readout.
To perform memory read from our Object Memory (c(P)), v(O)) at timestep t, we treat the current (at time t) encoded latent key as the querying latent query (i.e. gfth <- k,").. ). At the same time, to compute a direction query qi (D), we average the directions between the current view k(2) and the reference view k, (D), which represents an orientation that is likely to already have good coverage:
(D)
k( 41.
9
kg
(D)
+ DD
5

---

At each timestep, we perform two complementary memory reading operations (Orientation-Aligee Read and Orientation- Orientation-Compleeee Complementary Read) retrieve memory features fmem,t for our OnlineS- platter transformer to reason with the reference view and current source view.
Orientation-Aligneel Read emphasizes memory entries that closely match both the latent and directional query keys. The similarity measure (s))))))) for the i-th memory entry is defined as:
(align) 1,t
(L)T (1) (q)
(D)T (D) _
(8)
where Tr is a temperature coefficient to dampen potential inaccuracies produced by Encoder P. We dynamically set 7t 2.5 St, where or E [0, 1] is the confidence value of 43 for observation V/-
Orientation- Orientation-Compleerte Complementary Read retrieves memory entries that closely match the latent key but significantly differ in orientation, thus capturing complementary viewpoints. The complementary similarity score 8
(comp) is computed as:
(comp) = (qi (L)T
(o)kk()) (-qtt
(D)
(D)T (D) 1
k
(9)
The final retrieved features are computed via attention- sums:
N exp(s((nn))
(aaiign)
exp(s ( E, exp(s, (olou) Vis
Immm
(oomp) Vss where i, = 1,.., (S (S.P) P)
Σ, exp(sf(um)
(10)
Overall, our dual-key memory reading approach allows the memory readout to be guided by explicit spatial cues, encouraging spatial coverage on top of the visual-geometrical cues.
Memory Sparsification Mechanism Furthermore, we would like to maintain a bounded memory size while preserving good coverage of diverse viewpoints. To achieve this, inspired by attention based memory design in B 33], we propose a memory sparsification strategy that leverages our dual key design. Specifically, when the memory reaches its capacity S, drop 20% of memory that is deemed least useful by considering two factors: (i) usage (cross-attention contribution), and (ii) spatial coverage (average angular distance other entries). Specifically, for each memory entry i, define the total usage U, by averaging the accumulated cross-attention weight:
U, =
Σ EE
(allggn L.t
(align)
+
(zomp)
(1I)
TT.
where T; is the set of timesteps in which entry participated, and A expla
i.., A denotes
Σ, () the normalized cross-attention weight derived from Eq. (TO) for memory reads. Intuitively, U₁ is larger if an entry has been repeatedly or strongly attended to, meaning that it is useful. Subsequently, we compute spatial coverage for each memory entry. Let k(D) R be the direction key for entry i. We define its coverage measure C₁ as the average angular dot product to all other entries:
N
C.
k
(D)
K/P!
(12)
Lastly, to prune object memory. we sort all entries by C, and select the top 50% of entries as the dense subset, which represent viewpoints in memory that are spatially well covered. From this dense subset, we remove the 40% lowest-ranked entries with respect to usage U.. Formally,
PruneSubset
DenseSubset | U: - Un).
(13)
where Ug is the usage value the 40-th percentile within DenscSubsct. These pruned entries each time collectively constitute 20% of the full memory cap, balancing between retaining unique coverage and discarding underused features.
3.3 Training and Inference
Staged Training. Our OnlineSplatter model trained to optimize two complementary objectives: (1) learning relative object-camera pose relationships and predicting pixel-aligned 3D Gaussian

---

parameters, and (2) effectively utilizing our attention based Object Memory to reason about the object in canonical coordinate space through memory encoding and reading. These objectives present a challenging optimization landscape, as the gradients for the second objective only become meaningful after the first objective reaches certain level of convergence. To address this, we employ a two-stage training strategy: (1) Warm-up Training: We train the core reconstruction components without the Object Memory module. Specifically, we optimize the view encoder (Encoder(), positional and view embeddings (f pos and fesew), OnlineSplatter transformer, and unpatchify decoder in the first stage. (2) Main Training: We include the Object Memory module and train the entire network end-to-end, allowing the model to learn both reconstruction and memory simultancously.
Training Loss. We employ a combination of both photometric and geometrical losses to train model. First, the photometric loss Cphoto minimizes the MSE between the ground truth images and rendered images from predicted 3D Gaussian parameters at ground truth camera poses, where Lphoto is computed on object regions only. We include a background penalty term (Lhg) in Lyhoto to penalize Gaussians' color and opacity outside the object's visual hull. While the photometric loss alone can theoretically supervise feedforward reconstruction tasks 51] geometrical supervision can
often enhance convergence speed and reconstruction quality [88 14 29 Thus, we incorporate geometrical loss Lgeo. which includes a ray alignment component (Cray) to ensure that predicted 3D points lie along their corresponding camera rays, and a depth component (Caepth) to minimize MSE between predicted and ground truth relative depths. The overall training objective is: Lioial Cppoooo + Xgfe where Xg balances the contribution of geometrical supervision. We provide full details in the appendix.
Implementation Details. We train and evaluate our model on 256 x 256 resolution images. We use 8x A A100 GPUs for 250K steps with a batch size of 64 in the Warm-up Training stage and 500K steps with batch size of 16 in the Main Training stage. We sample 3-5 views per object during warm-up and 6- 12 views per object during main training. We provide more details on implementation and hyperparameters in the appendix.
4 Experiments
This section evaluates our approach by outlining the evaluation protocol, describing the datasets for training and testing, comparing against state-of-the-art baselines, and conducting ablation studies to analyze each component's 's impact.
4.1 Experimental settings
To properly evaluate our online object reconstruction framework, we need to assess how well it performs at different stages of observation accumulation. This is crucial because real-world applications often require reliable reconstruction even with limited initial observations. We therefore design stage-wise evaluation protocol that examines performance across three distinct phases: 1) Early Stage (Tenty == {1 < t < 4)); Tests the model's ability to quickly establish an initial object representation with minimal observations: 2) Mid Stage (Tmis := {5 S < 10)): Evaluates how well the model refines its reconstruction
Evaluation Protocol
more views become available; 3) Late Stage (Time = (11 S f S T)): Assesses the model's capability to maintain and improve reconstruction quality with extended observation sequences. For each test sequence of N frames (V, J.V we split the frames into two sets: Input frames (Viqun): A randomly sampled subset of N. frames used for input; Target frames (Vurger:: The remaining d frames reserved for NVS-based evaluation. Specifically, during evaluation, each frame V,, provided to the model at each timestep f E (1,..., (where T = N and we expect the model to output a 3D object representation Ô, at each timestep I and produce novel view images fi.,, for all target viewpoints e E Vingen- We then compare these renders against the corresponding ground-truth frames V,, The stage-wise performance then computed as: Masge Truee ExeTag Evevage M (RR,,,, Vo) where M represents standard image quality metries used for 3D reconstruction [16 51 2: PSNR, SSIM, and LPIPS.
Datasets. We train on 100K objects sampled from Objaverse 55 Unlike conventional few-view or
image-to-3D setups that render from biased polar angles and fixed upright poses, our setting targets real-world freely moving object reconstruction, where objects appear in arbitrary poses relative to the camera. This difference is critical, real-world data often includes partial views with unknown
7

---

GSO
Etrly-Stage
Mid-Stage SSIM1
Late-Stage SSIM
Method
PSNR1
SSIM1
LPIPS]
PSNRT
LPIPS]
PSNRT
LPIPS,
21.358
0.861
0.177
21.919
0.877
0.181
21.737
0.855
0.181
22.365
0.119
23.757
0.862
0.117
23.751
0.873
0.120
NPSaaz
22,986
0.859
0.155
23.050
0.863
0.162
22.949
0.878
0.156
diS
23.331
0.862
0.149
23.206
0.861
0.138
24.141
0.863
0.125
Ours
26.329
0.921
0.084
27.553
0.933
0.066
31.737
0.969
0.075
HO3D
Etrly-Stage
Mid-Stage SSIM1
Late-Stage SSIM1
Method
PSNRT
SSIM1
L.PPSS
PSNRT
I.PPSS
PSNR
I.PIPS.,
FSO
18.488
0.820
0.187
18.552
0.817
0.191
17.683
0.810
0.199
FSOnss disss
18.594
0.837
0.177
19.215
0.848
0.184
19.619
0.843
0.183
dPS
21.063
0.855
0.160
22.803
0.841
0.158
22.134
0.846
0.164
NPSai dist
21.134
0.853
0.162
22.967
0.869
0.165
22.947
0.860
0.163
Ours
23.627
0,910
0.152
25.803
0.912
0.122
27.928
0.952
0.099
Table I: Comparison different baselines on two datasets. Results are shown for early-stage, mid-stage, and late-stage settings. Best results are bolded and second best results are underlined.
object centers, rendering naive pre-processing ineffective. To simulate such conditions, develop custom script (details in the appendix) that generates diverse trajectories with look-at jitter, varying focal lengths, and randomized lighting. Each object receives unique trajectory, ensuring diverse motion patterns for training.
For evaluation, (GSO) 77. rendering 36 frames per object using our training pipeline (each with distinct lighting and motion). Second, we assess generalization real-world monocular videos with occlusions using the HO3D dataset [00] which contains hand-object interaction sequences.
two datasets of unseen objects. First, we test Google Scanned Objects
Baselines. No prior feed-forward model supports pose-free, RGB-only reconstruction in online set- tings. We thus adapt two leading pose-free few-view methods, FreeSplatter (48] and NoPoSplat (51) to our online setting. Both consume uncalibrated RGB images and predict 3D Gaussians.
FreeSplatter-O O [48] is a transformer-basee object-centric method trained on Objaverse. It jointly processes 4 fose-free views per sample. To adapt it online, we introduce two frame selection strategies for each timestep: (1) rand4: randomly selects frames from past observations (FSOranda): (2) dist4: selects 4 frames with the largest feature differences using DINO [ encoder (FSOdist4).
NoPoSplat [51] extends DUSt3R 42 backbone with 3D Gaussian output heads and is fine-tuned on scene-level data with 2-3 input views. We adapt it for object-centric reconstruction by fine-tuning on Objaverse with object mask supervision, and apply the dist strategy select diverse frames, resulting in NPSaist2 and NPSaist3-
4.2 Results
Table compares method against strong baselines both GSO and HO3D datasets across different stages of observation accumulation. GSO offers statistical significance through its diverse set of unseen objects, while HO3D introduces real-world challenges, including complex interactions and occlusions. Across all metrics and stages, OnlineSplatter achieves superior cetformance-impree up to +7.596 PSNR and +0. 106 SSIM on GSO, and +4.981 PSNR and +0.092 SSIM on HO3D in late-stage reconstruction. Two key insights emerge: good early-stage performance and temporal development. Even with fewer than four observations, OnlineSplatter significantly outperforms all baselines. Over time, a clear divergence appears. Baselines using explicit frame selection often exhibit unstable or stagnant performance. In contrast, OnlineSplatter consistently improves with more observations, also shown qualitatively in Fig. 3 where our method delivers notably better visual quality and geometric accuracy from early to late stages. This underscores the strength of our Object Memory mechanism in leveraging temporal cues for progressive reconstruction refinement.

---

Early-Stage(t=4)
Late-Starl Mite-Staee(t=2)) (t=16)
Early-Stage
(tae-ttt((((t1a)
FSOdist4
NPSdiat3
Ours
GT
Figure 3: Qualitative results of different baselines and our method on the GSO (left) and HO3D (right) datasets. We visualize the results at inference timestep and 16, which corresponds to the early-stage and late-stage settings, respectively. Our reconstructed outputs show significantly better visual quality and geometric accuracy as more observations become available.
Variants
Early-Stage Maug
Mid-Stage Marg
Late-Stage MMaug
Variants
Early-Stage
Masa
Mid-Stage
Masa
Late-Stage
Mary
Ours
0,699
0.734
0810
Ours
0.699
0.734
0.810
training
0.545
01582
0.588
Wio key
(Cup)
0.599
0.682
wio direction key
0.699
0.701
0.723
penolty (Cog)
encode from gs 0.541
0.582
0.611
wio depth loss (Ceyes) w/ sequeniaa sampling only w/ random samsplnng only
0.691
0.728
0.805
random pruning 0.697
0.728
0.764
0.645
0.633
0.688
0.697
0.728
0.764
Table 2: Impact of dual-key object memory design. Results are reported on GSO dataset.
Table 3: Impact of training strategy components.
Results reported GSO dataset.
4.3 Ablations and Analysis
In this section, we ablate different components of our method and analyze the results. For better readability, we normalize and average metries of PSNR, SSIM, and LPIPS to [0, 1] and report the Masg value, where a higher value indicates better performance, more details in the appendix.
Impact of Object Memory Design. We conduct ablation studies to validate our dual-key ob- ject memory design Sec. 3.2 summarizing results on the GSO dataset Table Specifically: Dual-key Design: Removing the latent key severely degrades performance all stages due to loss of -eomeriaa cues. Besides, removing the direction key significantly impacts later stages, highlighting its role in spatial coverage. Memory Encoding: Encoding memory from unpatchified

---

Vrtual- Can Giobal Rei nement
depth Sash- Optimization
3002 based Adjustment
Port- baxed Offiine BA
veit Feed Fend-Forward
Mutii L-M
Matt-view Feedi Forward
Online Feed- Feed-Tormmnt Formard
BundludDF
3D05*
COLMAP
Fast3R
InstantMesh
Figure 4: Visual comparison of mesh results between different methods. Methods marked with an asterisk (*) indicate that additional pre- or post-processing steps were applied to generate the visual results. More details in the appendix.
Gaussian parameters rather than direct Transformer token outputs leads to degraded performance, showing that using Transformer tokens for memory encoding better preserves learned representa- tions. Memory Sparsification: Our sparsification strategy based on usage rate and spatial coverage outperforms random pruning, particularly in later stages, showing the efficacy of our pruning criteria.
Impact of Training Strategy. We evaluate our training strategy (discussed in Sec. 3.3 through ablation studies and show results in Table 3 demonstrating: Staged Training: Removing the two- stage training (warm-up followed by main training) by using a single stage only greatly lowers performance, validating our intuition that the memory module requires well-initialized backbone. Loss Components: Removing the ray alignment (Lmy) notably reduces convergence speed and stabil- ity, harming performance. Excluding the visual hull (Lhg) penalty moderately degrades performance, underscoring its role preserving object boundaries. Removing depth term (C slightly impacts performance, it primarily aids convergence during warm-up training. Training Frame Sampling: Our progressive sampling strategy (discussed the appendix) outperforms sequential-onk and random-only alternatives, underscoring the advantage of curriculum learning in training.
Mesh Visual Comparison.
demonstrate our approach's efficacy, we convert our final 3DGS representation into meshes and visually compare it comprehensively with state-of-ther thettrt methods from different paradigms. We evaluate against representative methods: (1) those leveraging ground- (2) offline bundle adjustment techniques performing
truth depth information during optimization [4]
global optimization across frames [31
16 30 (3) diffusion-basee -based approaches using learned 3D
As shown in Fig. despite being provided with object masks, many methods struggle to reconstruct freely moving objects. In contrast, method achieves comparable quality those requiring extensive optimization or additional depth supervision while retaining the benefits a feed-forward online framework.
priors [47], and (4) recent werd-forward pointmap models operating offline [49 40
5 Limitations and Future Work
Our current framework has some limitations that warrant attention. First, the framework outputs 3D Gaussian Splatting (3DGS) representations, which while efficient for object understanding and rendering, may not be directly suitable for certain downstream applications requiring explicit mesh representations. Converting 3DGS meshes robustly remains challenging and is active area of research. Future work could explore hybrid representations that maintain both rendering efficiency and mesh compatibility. Second, the framework's performance may depend the quality of the initial reference view. Poor initial observation, such as heavily occluded or blurry first frames, could impact subsequent reconstruction quality. Lastly, our framework is currently limited to rigid objects. Future work could explore modeling non-rigid objects and integrate it with downstream tasks like robotic manipulation.
Conclusion
We presented OnlineSplatter, novel framework for pose-free online 3D reconstruction of freely moving objects from monocular RGB video. Our dual-key memory module enables efficient temporal feature fusion with constant computational complexity. Extensive evaluation demonstrates superior reconstruction quality without requiring camera poses, depth priors, global optimization. This makes our approach particularly suitable robotics applications in dynamic environments requiring online continuous moving object perception and manipulation.
10

---

References
[1] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[2] David Charatan, Sizhe Lester Li. Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 19457-19467, 2024.
[3] Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an atkinson- shiffrin memory model. In European Conference on Computer Vision, pages 640-658. Springer, 2022.
[4] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte. Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsami, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint Xix:2337.006665 2023.
[5] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.0800 2022.
[6] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lece, 2009.
[7] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages IEEE, 2022.
[8] Bardienus Duisterhof, Lojze Zust, Philippe Weinzaepfel, Vincent Leroy, Yohann Cabon, and Jerome Revaud. Mast3r-sfm; a tully-integrated solution for unconstrained cottuure-foo-mmoiiiio arXiv preprint arXiv:2409. 19152, 2024.
[9] Shreyas Hampali, Tomas Hodan, Luan Tran, Lingni Ma, Cem Keskin, and Vincent Lepetit. In-hand 3d object scanning from an rgb sequence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17079-17088, 2023.
[10] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3196 3206, 2020.
[11] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400, 2023.
[12] Zhisheng Huang, Peng Wang, Jingdong Zhang, Yuan Liu, Xin Li, and Wenping Wang. 3r-gs: Best practice in optimizing camera poses along with 3dgs. arXiv preprint arXiv:2504. 042244 2025.
[13] Hanwen Jiang, Zhenyu Jiang, Kristen Grauman, and Yuke Zhu. Few-view object reconstruction with unknown categories and camera poses. International Conference 3D Vision (3DV), 2024.
[14] Jaewoo Jung, Jisang Han, Honggyu An, Jiwon Kang, Seonghoon Park, and Seungryong Kim. Relaxing accurate initialization constraint for 3d gaussian splatting. arXiv preprint arXiv:2403 094113 2024.
[15] Alexander Kaszynski. fast-simplification: Fast quadratic mesh simplification. https: //t::/bggttttb... com/
pyvista/fast- simpl if ifaiciiiinnniii 2025.
[16] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(44, July 2023.
[17] Shakiba Kheradmand, Daniel Rebain, Gopal Sharma, Weiwei Sun, Yang-Cle Tseng, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. 3d gaussian splatting as markov chain monte carlo. Advances in Neural Information Processing Systems, 37:80965-80986, 2024.
[18] Lei Lan. Tianjia Shao, Zixuan Lu, Yu Zhang, Chenfanfu Jiang, and Yin Yang. 3dgs2: Near accond-order converging 3d gaussian splatting, arXiv preprint 2xXiv:2501.13975 2025.
11

---

[19] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong. Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang. Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling library. 2022.
[20] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pages
5741-5751, 2021.
[21] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-4544: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. In Proceddings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10072-10083, 2024.
[22] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T. Zexiang Xu, and Hao Su. One-2- 3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
[23] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9298 9309,, 2023.
[24] Yuzheng Liu, Siyan Dong, Shuzhe Wang, Yingda Yin, Yanchao Yang. Qingnan Fan, and Baoquan Chen. Slam3r: Real-time dense scene reconstruction from monocular rgb videos. arXiv preprint urXiv:2412.09401, 2024.
[25] Xiaoxiao Long, Yuan- Chan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song- Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9970-9980, 2024.
[26] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint arXiv:1608.00 03983, 2016.
[27] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1x51v:1711.05100,2000 2017.
[28] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov. Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Po- Yao Huang, Hu Xu, Vasu Sharma, Shang-Wer Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. arXiv:2304.0711 2023.
[29] Hamza Pehlivan, Andrea Boscolo Camiletto, Lin Geng Foo, Mare Habermann, and Christian Theobalt. Second-order optimization of gaussian splats with importance sampling. arXiv preprint arXiv:2504.1290 2025.
[30] Johannes Lutz Schönberger and Jan-Michael Frahm. frontuurr-foommmoiionn revisited. Conference on Computer Vision and Pattern Recognition (CVPR),
[31] Haixin Shi, Yinlin Hu, Daniel Koguciuk, Juan-Ting Ting Lin, Mathieu Salzmann, David Ferstl. Free-moving object reconstruction and pose estimation with virtual camera. arXiv preprint arXiv:240S.0m 05858, 2024.
[32] Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luea Wehrstedt, Jianyuan Wang. Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. DINOv3, 2025.
[33] Michael Streeke and Jorg Stuckler. Em-fusion: Dynamic object-level slam with probabilistic data associa- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2019.
[34] Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. NeuralRecon: Real-time coherent 3D reconstruction from monocular vidco. CVPR, 2021.
[35] Stanislaw Szymanowicz, Chrisitian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra fast single-view 3d reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 2024.
12

---

[36] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng. and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1-18. Springer, 2024.
[37] Shitao Tang, Jiacheng Chen, Dilin Wang. Chengzhou Tang. Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan. Mydiffusiou++:: A dense high-resolution muet diffusion model for single or sparse-view 3d object reconstruction. In European Conference on Computer Vision, pages 191. Springer, 2024.
[38] Vikram Voleti, Chun- Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-vier view synthesis and 3d generation from a single image using latent video diffusion. In European Conference on Computer Vision, pages 439 457. Springer, 2024.
[39] Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory. arXiv:2408. 16061, 2024.
arXiv preprint
[40] Jianyuan Wang. Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. arXiv preprint arXiv:2503.1 1165,, 2025.
[41] Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei Eiros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state, In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 105 00510- 2025.
[42] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made casy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20697-20709, 2024.
[43] Zehan Wang, Ziang Zhang, Tianyu Pang, Chao Du, Hengshuang Zhao, and 7hou Zhao. Orient anything: Learning robust object orientation estimation from rendering 3d models. arXiv preprint arXiv:2412. 18605, 2024.
[44] Bowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas Müller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 606-617, 2023.
[45] Bowen Wen, Matthew Trepte, Joseph Aribido, Jan Kautz, Orazio Gallo, and Stan Birchfield. Foundation- stereo: Zero-shot stereo matching. CVPR, 2025.
[46] Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:2412.01506, 2024.
[47] Jiale Xu, Weihan Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh; Efficient mesh generation from single image with sparse-vien view large reconstruction models. arXiv preprint arXiv:2404.07199 2024.
[48] Jiale Xu, Shenghua Gao, and Ying Shan. Freesplatter: Pose-free gaussian splatting for sparse-view 3d reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 25442 25442-25452, October 2025.
[49] Jianing Yang. Alexander Sax, Kevin J Liang. Mikael Henaff, Hao Tang. Ang Cao, Joyce Chai, Franziska Meier, and Mau Feiszli. Fast3r; Towards 3d reconstruction of 1000+ images in one forward pass. arXiv preprint Xiv: 13928 2025.
[50] Zhenpei Yang, Zhile Ren, Miguel Angel Bautista, Zaiwei Zhang, Qi Shan, and Qixing Huang. Fvor: Robust joint shape and pose optimization for few-view object reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2497-2507. 2022.
[51] Botao Ye, Sifei Liu, Haofei Xu, Xueting Li, Mare Pollefeys, Ming- Hssan Yang, and Songyou Peng. No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed images. arXiv preprint arXiv:2410.24207 2024.
[52] Vickie Ye, Ruilong Li, Justin Kerr, Matias Turkulainen, Brent Yi, Zhuoyang Pan, Otto Seiskari, Jianbo Ye, Jeffrey Hu, Mathew Tancik, and Angjoo Kanazawa, gsplat: An open-soure library for gaussian splatting. Journal of Machine Learning Research, 26 2025.
13

---

[53] Junyi Zhang, Charles Herrmann. Junhwa Hur. Varun Jampani. Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Ilsuan Yang. Monst3r: A simple approach for estimating geometry the presence of motion. arXiv preprint arX.v24410.38825, 2024.
[54] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for gaussian splatting. In European Conference on Computer Vision, pages 1-19. Springer, 2024.
[55] Qian-Y Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: modern library for 3D data processing. arXiv:1801. 09847, 2018.
[56] Zixin Zou, Weihao Cheng, Yan-Pei Cao, Shi-Sheng Huang, Ying Shan, and Song- Zhang. Sparse3d: Distilling multiview-consiserr diffusion for object reconstruction from sparse views. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 7900-7908, 2024.
14

---

A Notations and Definitions
Please refer Table for the list of symbols and their definitions used in the paper.
B Additional Details on Training Losses
In Section 3.3 of the main paper, we introduced our losses used for training. Below, we give more details.
Photometric Supervision For photometric supervision, we employ a differentiable rasterizer to render 2D images from the predicted 3D Gaussian parameters (Gobj,) and ground truth camera poses.
The photometric loss Cphoto consists of two components: Lyinto = Lmisked + Abg Ang, where Laoked is the masked MSE loss computed only object regions:
Lenasked
Σ
[[R(()) Vi(p)
(14)
|
PES.
Here, S. denotes the set of pixels belonging the object silhouette time t. R2(p) and Vi(p) respectively refer to the rendered color and ground truth color at pixel p time t. The background penalty term Lng penalizes Gaussians's 's color and opacity outside the object's visual hull:
Dig =
1
Σ gEst
(11c,112 + ao2)
(15)
|9
where S. Gby, is the subset of predicted Gaussians that is outside of the object's visual hull defined by object mask from reference view (Mg) and current view (M1) 0g is the opacity of Gaussian g. and a is weighting factor.
Geometrical Supervision While the photometric loss alone can theoretically supervise feedforward reconstruction tasks [51]. we incorporate geometrical supervision to enhance convergence speed and reconstruction quality. This is motivated by prior work [88 14 29 showing that Gaussian means significantly impact 3DGS convergence. To handle potential missing noisy depth ground truth, we decompose the geometrical loss into two terms: Lgo = Lay + Ca The ray alignment loss Lay ensures that predicted 3D points lie along their corresponding camera rays:
Lay
1
Σ
(1
(1
(16)
SN
PES.
where rp is the normalized predicted ray from camera center to pixel p. The normalized depth loss feepth compares relative depths:
2p
despth
(17)
S
d
PES,
where d, and Zp are predicted and ground truth depths, and and 2 are their respective means. Overall training objectives. The final training objective combines both photometric and geometrical losses: Lioul Lphoto + Xgeeeo where Ag balances the contribution of geometrical supervision.
C Additional Analysis
C.1 Impact of Training Data Scaling and Selection.
While the complete Objaverse [5] dataset contains over 10M objects, we conduct a systematic study of our method's 's scalability under computational constraints by examining two critical factors: (1) the relationship between dataset size and model performance, scaling from 1K 100K objects, and (2) the impact of data curation strategies on reconstruction quality. To evaluate these factors, we compare two sampling approaches: random selection from the full Objaverse dataset versus quality-based selection using aesthetic scores from [46]. All experiments are conducted with identical training tetiggurrtinnssOie training steps and consistent hyperparameters to ensure a fair and controlled comparison.
15

---

Section
Symbol
Definition
V,
RGB input frame timestep
Me
Object mask at timestep t for frame V2
Input and Output
V
Masked RGB frame at timestep / with background removed
Ó
Predicted 3D object representation at timestep
R
Rendered novel view image viewpoint o timestep
T
Reference-view input tokens (from initial frame)
Tin sne,t
Source-view input tokens at timestep t
Tin mem,,
Memory-readout input tokens at timestep t
To
Generic input tokens for view E fref, src, mem]
eiew
Tre
Reference-view output tokens Source-view output tokens timestep
Token Representations
Tas
sne.t
Tout
mem,e
Memory-reade output tokens at timestep
fvew
Encoded features for view E {ref, arc, mem)
remb
Positional embeddings
usew
View- speciipecific embeddings for view (ref, sre, mem)
G4N 2N G
Complete object Gaussian representation at timestep Memory-basee based subset of Gaussians timestep
G
Reference-view subset of Gaussians at timestep
G
Source-view subset of Gaussians timestep
Gaussian Parameters
G
Set K Gaussian primitives
14
Mean position of k-th Gaussian Rotation k-th Gaussian
rk
Scale of k-th Gaussian
Color Spherical harmonies coefficients of k-th Gaussian
Ok
Opacity of k-th Gaussian
k!"))
Latent key at timestep
k(
Direction key timestep Memory value timestep Set of stored directional keys Set stored latent keys stored memory values Latent query timestep Direction query timestep
((4)
(())
(L2)
Memory Module
q
(diggn
Alignment similarity score for entry at timestep / Complementary similarity Temperature coefficient at timestep t Confidence value from orientation estimator Usage measure for memory entry Coverage measure for memory entry Set of timesteps for memory entry
comp)
for entry
T
U
C;
T
A
Normalized cross-attention weight
M
Generic image quality metric (PSNR, SSIM, LPIPS)
Masae
Stage-wise performance metric Normalized average of metrics Set input frames
Maeg
Evaluation Metries
Vinpa
Varget
Set of target frames
Tange
Set of timesteps in stage
N
Number pixels per frame (H W)
Maximum memory size
Dimensions Constants
P
Number of patches per view Feature dimension
Table 4: List of mathematical notations used throughout the paper.
16

---

As shown in Fig. our analysis reveals two key findings. First, our method demonstrates strong scaling behavior, with performance continuing to improve as the dataset size increases 100K objects without showing signs of saturation. This suggests significant potential for further gains with larger datasets. Second, we find that careful data curation through aesthetic quality filtering yields substantial performance improvements, indicating that strategic data selection can be an effective approach for optimizing model performance under limited computational resources.
Objects Randomly Sam plep
Objects Filtered by Quality
Baseline Level
No Convergence
Combined loormalized Metric
0.6
Baseline Level
No Signs of Convergence
Number Objects
Figure 5: Impact of Training Data Quantity and Quality.
C.2 Impact of Ray Alignment Loss in Geometrical Supervision.
While photometric RGB-based loss can effectively supervise 3D Gaussian positions when they are already well-aligned with the ground truth and visible from the rendered pose, it struggles to guide Gaussians that are far from their optimal positions. This limitation becomes particularly problematic in object-centric reconstruction, where significant portion of the region in observed frame consists of plain background regions that do not receive meaningful supervision. In such cases, predicted Gaussians may either float arbitrarily in space or be suppressed by the background penalty term (Log), failing to contribute meaningfully to object reconstruction.
To address this challenge and accelerate convergence, we introduce a ray alignment term Lry that explicitly regularizes the 3D Gaussian positions (as detailed in Sec. [ This term ensures that predicted Gaussians align with their corresponding camera rays in the object region, providing crucial geometric guidance even when photometric supervision is insufficient.
To demonstrate the effectiveness of the ray alignment loss, we conduct a comparative analysis between training with and without Lay, visualized in Fig. 6 The visualization shows two camera views from a training sample at different stages (1K-10K training steps). In each view, blue lines represent ground-trutt trrtt per-pu per-pixel camera rays, while red lines indicate predicted 3D rays from the ground-truth camera center to the predicted Gaussian means. The comparison reveals two key findings:
Without Lmy, many Gaussians drift away from the object region, effectively becoming "dead" primitives that contribute little to reconstruction.
With Lay, Gaussians maintain better alignment with ground-trufe -ruth rays in object regions, and background Gaussians actively migrate toward object regions to participate in reconstruction.
These observations confirm that the ray alignment loss serves as effective regularizer for 3D Gaussian positions and significantly improves convergence speed during training. The quantitative results also reflect this significant improvement (See Table 5).
17

---

Step
Traning Dep
Training Step
Dooaaatee th
With Crwe
Trainig Step 1<
Traning Stop 5K
Training Soep 10c
Figure 6: Visualization of the effect of without (top row) and with (bottom row) ray alignment loss Lra over 1K 10K training steps. The visualization shows both the ground-truth per-pixel camera rays (in blue) and the predicted 3D ray pointing from the ground-truth camera center to the predicted 3D point (in red). The qualitative visualization shows clearly that the ray alignment loss is effective in regularizing 3D gaussians positions and converges quickly.
C.3 Additional Notes on Ablation Study.
In the main paper, we use the averaged metrics (Mavg) ) of PSNR, SSIM, and LPIPS to report results
in ablation studies (Table 2 and Table 3 due to space limitation and better readability. Note that
Mang is computed Mavg normalize PSNR, SSIM, and LPIPS [0, I] by clipping scaling such a higher Mang value indicates
clip PSNR-20
1) SSIM (1
clip(
LrPS 1)) where
0.6
relatively better performance. Here in T we provide the detailed original results for completeness.
Impact of Dual-key Object Memory Design
Farly-Stage SSIM
Mid-Stage SSIM
Late-Stage SSIM
Method
PSNR1
LPIPS.
PSNR
LPIPS
PSNR
LPIPS,.
Ours
26.329
0.921
0.084
27.553
0.933
0.066
31.737
0.969
0.075
w/a latent key
23.260
0.764
0.176
23.947
0.805
0.153
24.212
0.822
0.147
direction key
26.320
0.917
0.084
26.385
0.923
31.147
0.749
0.083
encode from
23.237
0.759
0.179
23.983
0.804
0.154
24.538
0.837
0.138
w/ random pruning
26.261
0.919
0.086
27.089
0.927
0.080
31.443
0.852
0.080
Impact of Training Strategy Components
Farly-Stage SSIM?
Mid-Stage SSIM
Late-Stage SSIMf
Method
PSNRT
LPIPS
PSNR
LPIPS
PSNR
LPIPS..
Ours
26.329
0.921
0.084
27.553
0.933
0.066
31.737
0.969
0.075
staged training
23.257
0.766
0.177
24.016
0.803
0.155
24.169
0.808
0.151
ray loss
23.528
0.788
0.166
24.312
0.823
0.145
25.962
0.905
0.094
w/n bg penalty
25.844
0.896
0.097
26.562
0.937
0.077
31.714
0.949
0.091
w/o depth loss
26. 147
0.917
0.091
28.
0.921
0.150
31.643
0.960
0.077
w/ sequential sampling only
25.268
0.866
0.117
25.929
0.904
0.093
26.182
0.905
0.091
random sampling only
26.249
0.920
0.085
0.923
31.695
0.906
0.119
Table 5: Expanded version of Table and Table in the Main Paper
18

---

D Additional Training and Implementation Details
In Section 3.3 of the main paper, due to space constraints, we provided only the most significant training and implementation details. Here, we provide comprchensive details on our training (Section D.I and D.2) and implementation (Section D.3).
D.1 Progressive Training Frame Sampling.
Most methods that leverage monocular video data rely on implicit temporal continuity i.c., minimal changes between consecutive frames during training [39 9 [1] which substantially simplifies the learning problem. In contrast, our setting involves dynamic motion from both the camera and the target object. with no assumptions made ahout the nature of the interacting agent (e.g., human hand robotic manipulator). As a result, the relative object-camera pose, especially the angular velocity, may vary significantly hetween adjacent frames. To enable the model to learn across a wide range of motion dynamics, we leverage a curriculum-basee progressive frame sampling strategy. Training begins with temporally-closs frames, gradually increasing the frame interval (thereby reducing CO- visibility). and eventually transitions to fully random frame sampling. This staged approach ensures exposure to diverse object motions throughout training. result, trained model generalizes well to both sequential video data and unordered image sets.
D.2 Training Data Rendering Pipeline.
For each object sampled from the Ohjaverse dataset, we generate a unique fly-around sequence using a custom Blender script that introduces controlled randomization across camera motion, optical parameters, and scene illumination. Our pipeline operates as follows:
First, we construct smooth camera trajectory by sampling Ki =4 elevation angles {0x} and K2=8 radius values (ri) within spherical shell [dminn; dmax]. This sampling strategy ensures comprehensive coverage both polar and azimuthal viewing angles. Through linear interpolation of these key points combined with uniformly distributed azimuth angles G, we generate --100 waypoints that exhibit smooth transitions in both vertical and radial dimensions. We then construct a periodic cubic spline through these waypoints and uniformly sample it to obtain the final camera positions. By seeseddin the random number generator for each object, we ensure that every sequence features unique trajectory with object-specific zoom patterns.
During rendering, we employ a Track-To constraint maintain camera focus on object while introducing a small per-frame look-at jütter (#:5 em on each axis) to prevent the object from remaining perfectly centered. To further enhance diversity, we randomly sample focal lengths from (30, 35, 40, 45, 50) mm and configure combination of area and point lights with randomized positions and intensities drawn from broad uniform distributions. This comprehensive approach yields training sequences characterized by diverse motion patterns, controlled scale variations, and rich illumination conditions.
D.3 Implementation Details.
We provide comprehensive implementation details of our OnlineSplatter framework below.
Model Architecture. Our model consists of several key components:
Image Encoders: We use frozen DINO backbone (I as Encoderf with patch size 8 and output dimension 768. The leamable Encoder follows the same architecture but is output dimension is 256. The concatenation of Encoderi and Encoder provides a 1024 dimensional feature vector for each patch token.
OnlineSplatter Transformer: The transformer processes tokenized inputs through 24 layers, each with 16 attention heads and hidden dimension 1024. We use layer normalization and a dropout rate of 0.05.
The object memory maintains maximum of S = 1024 x 20 entries, with each entry containing token-level fcatures of dimension C 1024. The number of patches per view P 1024 is determined by the input resolution 256 256 and patch size 8. The key and value encoders (Encoder and Encoder are implemented as 3-layer MLPs with 1024 hidden units. We use OrientAnything
Memory Module:
[33 as the pre-trained direction key encoder (Encoder ) and keep it frozen.
Model Size: The total number of parameters in the model is around 488M, of which 402M is trainable.
We use the CUDA diffeerentiable rasterizer implemented in the original 3DGS [66 to render the predicted 3D Gaussians. The rendering resolution matches the input resolution. We use near plane 0.1 and far plane 100.0 for rasterization.
Rasterization:
19

---

Training Configuration. We employ two-stage training strategy:
Optimizer: We use Adam W 27 optimizer with learning rate 1e 4, weight decay 0.05, and Cosine Annealing 26 learning rate schedule. The learning rate is warmed up for 2000 steps.
Training Losses: The overall training objective (detailed in Section [ combines photometric and
geometrical losses: Lumal = Lehow + Xg Eye = Lmusked + AogCng + (a(na + da Ca
Warm-up Training Stage:
Steps: Trained for 250K steps with effective hatch size 64. We train the model without the memory module during this stage.
Loss Weights: Ag = 0.3, Ang = 0.3, Ad 0.5
View-specific Embedding: At this stage, we train the reference-view embeddings (Fref") and source-view embeddings (fave)) while keeping the memory-view embeddings not involved. Initialization: All weights initialized randomly using the truncated normal distribution with mean and standard deviation 0.02. Note that the pre-trained DINO encoder weights (i.c., Encoder;) are frozen and not updated, while the learnable encoder (Encoder() is being updated.
Input Sampling: We sample 3 5 views per sequence sample, with a sampling schedule as described in Sec. D.
Main Training Stage:
Steps: Trained for 500K steps with effective batch size 16, incorporating the memory module.
Loss Weights: Ag = 0.3, Ang 0.3, Aa (Casph removed)
View-specific Embedding: To differentiate the kinds of memory read-out tokens, initialize two of memory-view embedding (i.c. (f femb }), one for orientation- aligned memory read-out and the other for prionation-commlemmentay memory read-out.
Initialization: We initialize the transformer weights using the warm-up stage weights and we copy source-view embeddings (fame) weights from the warm-up stage to initialize the memory-viww embeddiings (1 [[feemt: peinb }). While the memory key encoder and value encoder weights are initialized randomly using the truncated normal distribution with mean 0 and standard deviation 0.02. Note that the pre- pre-trained direction key encoder weights (i.e.. Encoder are frozen not updated.
Input Sampling: We sample 12 views per sequence sample, with sampling schedule described in Sec. [.T]
Data Processing and Inference Details.
Image Processing: Input images are resized to 256 256 and normalized using ImageNet statisties (
We apply random augmentations including mirroring with a probability of 0.5.
Camera Normalization: We preprocess all the ground truth poses in sequence to be relative to the reference view, such that the reference view pose the identity matrix.
Memory Sparsification: The memory is pruned when reaching the maximum memory size, removing 20% of entries based on usage and coverage metrics. The temperature coefficient 7ı is dynamically adjusted based on orientation confidence: from the orientation estimator 43
2.5 dt. where E [0, 1] is the inferred confidence
At inference time, we use the off-the-shelf video online segmentation model from Xmem 3 to estimate the object mask based on the initial object mask from the reference view,
Object Masking:
Rendering: At inference time, filter out the predicted Gaussians that are either low in opacity (i.e., og < 0.0001) ) or the Oth-degree spherical harmonics coefficients are close to the background color (set to 1,, 1] for rendering object against white background).
Training Environment: We train model on 8x NVIDIA A100 GPUs with 80GB memory. Our implementation uses Python 30, PyTorch 2.1.2, torchvision 0.16.2, and we leverage xFormers [9 0.0.23 for efficient attention computation.
Evaluation Environment: We run all inference on single L40S GPU with 48GB memory.
E
Additional Discussions
E.1
Additional Notes on Mesh Visual Comparison.
In of the main paper, we convert our final 3D Gaussian Splatting (3DGS) representation into meshes and conduct comprehensive visual comparison with state-of-the-a the-arrt methods across different paradigms. Methods
20

---

marked with an asterisk (+) indicate that additional pre- post-processing steps were applied to generate the visual results. Below, detail the mesh generation process for each method and provide additional analysis.
For Fmov [II, BundleSDF [44), Fast3R 49, and InstantMesh 47 we either utilized their official implementations to generate meshes or obtained results directly from their published papers official websites.
3DGS Since the original 3DGS 16] implementation lacks support for pose-free object reconstruction and joint camera pose optimization, we employed the open-source gsplat [52] library with MCMC [77 random initialization for both 3DGS parameters and camera poses. We incorporated ground-trun object masks to optimize 3DGS parameters.
COLMAP*; Wc attempted reconstruction using the official COLMAP (3 implementation both with and without ground-truth object masks. However, neither approach converged successfully, resulting in failed reconstructions.
Using the official VGGT [40] implementation, we preprocessed the input images by applying ground-truth object masks to remove hackground content. We utilized the Depthmap and Camera Branch outputs, which their paper indicates provide superior performance compared the Pointmap branch. We additionally filtered out redundant hackground predictions to generate the final results.
VGGT*;
Ours#: To convert our OnlineSplatter framework's output into mesh format, we rendered 30 uniformly distributed views of the predicted 3D Gaussians, generating corresponding RGB-D images and masks. These were fused with their camera poses in a TSDF volume,
we extracted the mesh using Open3D [55] The resulting mesh was further simplified using the open-source mesh simplification library [15] Note that we only generate the mesh for a rough visual comparison, our method does not focus mesh generation, we leave it future work.
From the visual comparison, our method demonstrates reconstruction quality comparable to approaches that require extensive optimization or additional depth supervision, while maintaining the advantages of a feed- forward, online framework. While optimization- based methods such COLMAP* and 3DGS* excel reconstructing static scenes, they face significant challenges when applied freely moving objects, even with access to ground-trutt -ruth ohject masks and the ability to perform global optimization across all frames. This observation highlights both the inherent difficulty of reconstructing freely moving ohjects from monocular videos and the promising capabilities of our approach. A promising future direction could be combining our online feed-forward framework with optimization-based refinement achieve higher-quality mesh reconstructions.
E.2 Broader Impacts
Our work on online 3D reconstruction has several societal implications. On the positive side, the technology could democratize 3D content creation by making real-time 3D scanning more accessible the general public, while also improving efficiency in manufacturing through real-time quality control and inspection. The real-time nature our system could benefit assistive technologies and education by enabling interactive 3D visualization and understanding of objects. However, there are potential concerns regarding privacy and intellectual property, the ability to quickly reconstruct 3D models could be misused for unauthorized scanning copying of physical objects. We recommend implementing appropriate usage guidelines and access controls when deploying the technology in sensitive contexts, while encouraging responsible development that considers both privacy and intellectual property rights.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our introduction clearly states the claims made, including contributions made in the paper and important assumptions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in paper.
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does include thcoretical results.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
22

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose comprehensive training and implementation details in the main paper and appendix to facilitate reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clcarly and fully.
(c) the contribution a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code. with sufficient instructions faithfully reproduce the main experimental results, as described supplemental material?
Answer: [No]
Justification: We provide reference implementation where possible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines (httpa://nips... httpp:///nips.ccppubblls cc/public/ htt////ooSuumiiiiioooo
for more details.
While we encourage release of code and data, we understand that this might not possible,
so "No" is an acceptable answer. Pupers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : ://nis///.////////// cc/public/ Quiidess/CoodSuubmieessoooooo... for more details.
23

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide comprehensive training and implementation details in the main paper and appendix to facilitate reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Dnes the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report average performance over multiple runs with different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma enror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment. does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We fully disclose all compute resources used in the experiments in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
24

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code of ethics
Does the research conducted the paper conform. every respect, with NeurIPS Code of Ethics httpp/////rrrpps..pppllltt//////
Question:
Answer: [Yes]
Justification:
We have checked the Code of Ethics, and our research fully conforms to it.
Guidelines:
The answer means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure preserve anonymity (e-g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of our work in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will he foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
25

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All papers code used by us are properly cited.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new
Guidelines:
The NA means that the paper does release
Researchers should communicate the details of the dataset/code/model part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
answer means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent research with human subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
26

---

Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important. original. or non nta-standard component of the core methods in this research? Note that if the L.M is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the rescarch, declaration is not required.
Question:
Answer: [NA]
Justification:
The core method development this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http:///urrppccccoooooeeeeeeee/e for what should or should not be described.
27


Reviews:
[{"review_id": "d4IViNWY9U", "reviewer": "Reviewer_ZqiV", "summary": "The paper proposes a feed-forward method for object tracking and reconstruction in an online manner. The method operates only on RB frames and claims to be real-time. Moreover, the proposed method doesn't require any camera poses and references everything w.r.t. to the canonical representation in the first frame. Experimental validation is sufficient and shows new state-of-the-art performance on several datasets.", "questions": "1. Would a comparison to [8] be possible?\n2. Another ablation on the used VOS module would be interesting.\n3. Is it possible to extend the training procedure to avoid requiring ground truth camera poses? \n4. The method doesn't use any global optimisation, but it can benefit from it, e.g. 3DGS optimization on all frames. How much would this improve the results?", "limitations": "Yes, the limitations are discussed in the supplementary materials.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The proposed method is novel, and the idea of using a feed-forward model for splatting a dynamic object reconstruction is interesting.\n* The whole pipeline seems to be well-engineered, e.g. runs in real-time, operates in an online manner, and requires only monocular RGB frames.\n* The method achieves state-of-the-art results on GSO and HO3D datasets, both in early, mid and late stage of object tracking. The improvement over other methods is significant. \n* The introduced dual-key memory module is an important contribution, together with the memory sparsification mechanism that drops tokens that are least useful.\n\nWeaknesses:\n* The method assumes that object masks as already known. This requires some user input and may limit some applications since it requires some user clicks and the video object segmentation model XMem.\n* Training still requires ground-truth poses, and overall the training is very supervised. This could be a problem for good generalization capabilities of this method.\n* The paper should also compare to other methods, such as [8].\n* Eq. (9) is confusing. Why is there a dot sign followed by a minus sign? Are brackets missing there? \n* There are many typos, missing letters, etc. The paper would benefit from proof-reading.", "was_revised": true, "final_justification": "Most of my concerns are well-addressed, and I hope that those additional experiments will be added to the final paper (including ablating different VOS modules, comparison to [8], etc). I'm willing to raise my score and view paper as worth publishing at NeurIPS.", "rebuttal": "We appreciate reviewer `ZqiV`'s constructive feedback and recognizing our \"**method is novel**\", the idea is \"**interesting**\", pipeline is \"**well-engineered**\". Below, we address reviewer's questions accordingly:\n\n> Q1: The method assumes that object masks as already known. This requires some user input and may limit some applications since it requires some user clicks and the video object segmentation model XMem.\n\nYes, in order for our current method to run, we currently require an initial specification of the object of interest. This is because there may be multiple objects in the video, and so some human input is needed to specify the target object at test time. The current setup is consistent with related works[33] in tracking and reconstruction in online settings.\n\nCrucially, after this initial step, our online pipeline leverages the highly efficient XMem Video Object Segmentation (VOS) module. XMem is well-suited for online applications due to its strong performance, speed, and bounded memory footprint. Therefore, the computational overhead is minimal and primarily confined to the first frame, ensuring our framework remains practical for a wide range of real-time applications.\n\nFurthermore, the input mechanism is flexible (e.g., click, box, or text prompt) and not a hard constraint of the architecture. For applications requiring full automation, our framework can also be readily extended to process all salient foreground objects in parallel by integrating an off-the-shelf object detector.\n\n\n> Q2: Training still requires ground-truth poses, and overall the training is very supervised. This could be a problem for good generalization capabilities of this method.\n\nWe agree with the reviewer that the current training setup requires ground-truth poses, however, we would like to clarify that, firstly, these ground-truth poses are obtained at nearly no cost. As long as we have an object model, either a virtual object model, a real-world scanned model, or a generated one, we can render it at unlimited views and obtain RGB frames with ground-truth poses. Thus, we do not face the issue of data scarcity in terms of camera poses required by our method.\n\nFurthermore, to ensure robust generalization, we took several deliberate steps in our training. We generated our training data using a custom Blender script, which introduces diverse and randomized camera trajectories, lighting, and focal lengths (as detailed in our Supp. D.2). This strategy minimizes distributional bias and exposes the model to a comprehensive range of relative object-camera poses, such that our learned model is not biased towards a specific range of relative position. Our evaluations validates our generalization capabilities on datasets with entirely unseen object categories, including challenging real-world data from HO3D, demonstrate the success of this approach. \n\nHaving said that, the idea of removing the reliance on ground-truth poses in training is an interesting and challenging problem to solve. Currently, the ground-truth poses are essential due to our reliance on using 3DGS differentiable rendering to provide gradient from RGB frames to model parameters. We could consider treating the camera pose as a latent variable and optimize reconstruction losses (photometric/silhouette) after marginalizing over poses proposed by a small amortized pose predictor. We could also leverage feedback from external discriminator (a judging model to compare predicted and ground-truth frames) as a training signal to guide the pose prediction, but we leave this interesting direction as future work.\n\n\n> Q3: The paper should also compare to other methods, such as [8].\n\nWe thank the reviewer for this suggestion. [8] is an inspiring and seminal work that tackles free-moving object reconstruction via multi-stage offline bundle optimization. The primary reason for not including a direct quantitative comparison with [8] is the fundamental difference in our objectives and assumptions. Our work focuses on an online, feed-forward framework designed for continuous perception, whereas [8] is an offline, bundle optimization-based method, assuming all frames are available upfront. Furthermore, lacking of open-source code from [8] also made the comparison extremely challenging.\n\nWhile it is not a fair comparison for either work, below, we try to compare with [8] by taking numerical results directly from their paper, and we report our reconstruction result at the final time step. We evaluate on the same objects as described in [8]'s paper and we calculate the Hausdorff distance following [8]. The results shown that our method achieves better reconstruction quality than [8] in terms of Hausdorff distance, this is consistent with our qualitative results in Figure 4 when compare with qualitative results of [8].\n\n| Method | avg. Hausdorff dist. |\n| :----: | :------------------: |\n|  [8]   |         4.65         |\n|  Ours  |        2.196         |\n\nWe are unable to compare the PSNR/SSIM/LPIPS metrics due to the lack of their evaluation details, we are unable to infer whether their reported results are computed via rendering or texture map. However, we will add a visual comparison with the results presented on [8]'s official website to our Figure 4. (We are unable to provide the link nor the visualization here due to NeurIPS policy)\n\n> Q4: Eq. (9) is confusing. Why is there a dot sign followed by a minus sign? Are brackets missing there?\n\nYes, thanks for the suggestion, adding a pair of bracket would imporove clarity, we will update Eq.(9) as: $s_i^{(\\text{comp})} = (\\mathbf{q}_t^{(L)\\top}\\mathbf{k}_i^{(L)}) \\cdot (-\\mathbf{q}_t^{(D)\\top})\\mathbf{k}_i^{(D)} \\cdot \\tfrac{1}{\\tau_t}$\n\n> Q5: There are many typos, missing letters, etc. The paper would benefit from proof-reading.\n\nThank you, we will thoroughly proof‑read and fix any typos.\n\n> Q6: Would a comparison to [8] be possible?\n\nThank you for the question. Please see our detailed response to Q3.\n\n> Q7: Another ablation on the used VOS module would be interesting.\n\nThis is an insightful suggestion. We selected XMem[3] as our default VOS module due to its well-established performance, efficiency, and bounded memory usage, which aligns perfectly with our online framework's requirements.\n\nFor our experiments, we pre-processed all object masks to ensure a fair and rigorous comparison between our method and the baselines, eliminating segmentation as a variable. However, to address the reviewer's question, we conducted a brief analysis using other state-of-the-art VOS models, such as Cutie[a] and SAM-2[b]. We observed that these models produce highly comparable segmentation results on our test set, with performance variations leading to negligible differences (<0.5%) in our final reconstruction metrics. This suggests our pipeline is not overly sensitive to the choice of a specific high-performing VOS module, thus we selected XMem[3] as our default VOS module due to its smaller model size compared to the other state-of-the-art VOS models.\n\n[a]: Cutie: Putting the Object Back into Video Object Segmentation (CVPR 2024) \n[b]: SAM 2: Segment Anything in Images and Videos (2024)\n\n\n> Q8: Is it possible to extend the training procedure to avoid requiring ground truth camera poses?\n\nThank you for the question. Please see our detailed response to Q2.\n\n> Q9: The method doesn't use any global optimisation, but it can benefit from it, e.g. 3DGS optimization on all frames. How much would this improve the results?\n\nThis is an excellent question regarding the potential benefits of post-process optimization. We agree that incorporating a global optimization on all frames would improve the final reconstruction quality. However, we conducted such experiments where we leverage 3DGS + joint pose optimization and implemented additional losses to encourage pixel alignment (similar to our training loses), we are only able to get an average $+1.26$ PSNR performance gain by optimizing for more than 100,000 iterations or over 2 mins. \n\nWe attribute the underwhelming improvement over prolonged time to a few reasons: 1. 3DGS optimization is sensitive to hyperparameter choice (such as different learning rate and scheduler are required for different parameter groups, especially for joint pose optimization, requires careful tuning), while our feed-forward model already produces good results from large-scale training, added per-instance optimization may struggle to improve further. 2. 3DGS optimization is also hyper-sensitive to imperfect object masks and it is prone to produce floaters around object edges, we have shown this evidence in our Figure 4 of main paper, the floaters around the edges significantly harms the performance. While our feed-forward method can be trained to handle the object edges and imperfections in object masks better by batched training.\n\nHowever, we believe there is still room for exploration in this hybrid direction, by carefully integrating optimization into our online framework while maintaining the efficiency and bounded memory footprint, we leave this as future work."}, {"review_id": "KGGrYXBySd", "reviewer": "Reviewer_bxd2", "summary": "This paper introduces a novel incremental and feed-forward pipeline for reconstructing freely moving objects from a sequence of images, with the final output represented as a 3D Gaussian Splatting (3DGS). The method is designed to address two central challenges: (1) performing reconstruction from unposed objects without depth priors, and (2) maintaining constant computational cost regardless of video length, enabling online applicability. While the first challenge has been explored in prior work, this paper contributes a practical and efficient solution to the second. Additionally, enabling feed-forward generation of 3DGS in an online setting is both technically demanding and impactful. Experiments on real-world datasets validate the effectiveness of the proposed method.", "questions": "Please refer to the Weaknesses. \n\nIn particular, including videos that showcase the 3D Gaussian Splatting reconstruction results along with their corresponding input video sequences would provide more convincing evidence of the method’s performance. I would be happy to raise my score if the above concerns are adequately addressed during the rebuttal.", "limitations": "The authors have addressed the limitations of their work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1. This paper propose a novel feed-forward reconstruction pipeline characterized by a concise and effective design.\n\n2. The introduced dual-key memory module demonstrates strong performance and effectively addresses practical challenge in incremental reconstruction.\n\n3. The methodology is presented with clear and rigorous exposition, and the quantitative results convincingly validate the effectiveness of the approach.\n\nWeeknesses\n\n1. As stated in L30, the authors claim that prior methods struggle to capture instance-specific geometric details, limiting reconstruction accuracy. However, the proposed method does not appear to offer a clear advantage in capturing fine-grained geometry—apart from incorporating certain depth-based losses. This is particularly apparent when compared to 3DGS-based methods that benefit from explicit geometric initialization. Moreover, the authors do not report any quantitative metrics related to geometric accuracy (e.g., depth error, chamfer distance). It is therefore unclear whether the proposed method achieves better geometric fidelity or how it captures fine-grained geometry. If such improvement exists, further justification is needed. I would also encourage the authors to report geometric accuracy metrics to support their claim.\n\n2. After masking out the background, the task essentially becomes pose-free object reconstruction. In this case, it would be more appropriate to compare against other pose-free baselines, such as COLMAP-Free 3D Gaussian Splatting. Although these methods are optimization-based, for 3D asset generation, reconstruction quality and geometric accuracy is often more critical than reconstruction efficiency. Such comparisons would help position the contribution more clearly.\n\n3. The authors have not released the code, and the visualizations in the experiments are relatively limited. It is also unclear whether the test dataset includes objects that are entirely unseen in train dataset. The qualitative example of the \"blue kettle\" is not particularly convincing in terms of generalization. I would strongly encourage the authors to provide more diverse and challenging visual examples to support the claimed generalizability.", "was_revised": true, "final_justification": "1.The authors have thoroughly addressed my concerns in the rebuttal. I believe the paper meets the NeurIPS acceptance bar and have decided to increase my score to accept.\n\n2.The work is complete and demonstrates novelty.", "rebuttal": "We appreciate reviewer `bxd2`'s constructive feedback and recognizing our feed-forward generation of 3DGS in an online setting is \"**both technically demanding and impactful**\", the method is \"**concise and effective**\", our paper is \"**clear and rigorous**\". Below, we address reviewer's questions accordingly:\n\n> Q1: As stated in L30, the authors claim that prior methods struggle to capture instance-specific geometric details, limiting reconstruction accuracy. However, the proposed method does not appear to offer a clear advantage in capturing fine-grained geometry—apart from incorporating certain depth-based losses. This is particularly apparent when compared to 3DGS-based methods that benefit from explicit geometric initialization. Moreover, the authors do not report any quantitative metrics related to geometric accuracy (e.g., depth error, chamfer distance). It is therefore unclear whether the proposed method achieves better geometric fidelity or how it captures fine-grained geometry. If such improvement exists, further justification is needed. I would also encourage the authors to report geometric accuracy metrics to support their claim.\n\nWe thank the reviewer for this question. We would like to clarify that, in Line 30, we specifically refer to the generative type of reconstruction methods, such as TRELLIS[d], LRM[10], and many more, that learn object-level priors via large-scale training, and **use hallucination as a feature**, with the main aim of *generating a plausible 3D asset* even when a large portion of the 3D geometry is not observed. These methods are great for single-image-to-3D asset generation, but if we adapt them to online perception settings, not only does their hallucination feature not help in constructing faithful “instance-specific geometry,” but their predictions also do not benefit from continuous observation (i.e., stream-based video input), which should improve the object reconstruction over time, as we demonstrated in our work.\n\nIn order to reconstruct object faithfully to the instance on the fly while observing it, we leverage recent advances in training feed-forward models using 3DGS parameterization (e.g. pixelSplat[2], NoPoSplat[39]), where our primary training signal teaches the model to produce pixel-aligned Gaussian primitives on the observable object surface. Different from the generative line of works that we mention above, our training objective is implicitly different, where our depth-based and ray-alignment losses supervises the model to predict local geometric details, rather than object-level distributions. Therefore, to reconstruct the full object, we designed the spatial-guided object memory to perform spatial coverage tracking, and the dual-key memory readout strategy to fuse new observations into a single, canonical 3D object representation. All of these designs ensures that our model reconstruct faithfully to the instance, and the continuous observation enables our model to improve the reconstruction over time.\n\nWe appreciate the reviewer’s suggestion to add geometric accuracy metrics. In line with prior 3DGS-based methods [2, 14, 39], we primarily reported reconstruction quality using PSNR, SSIM, and LPIPS. We will, however, add Chamfer Distance (CD) to better reflect geometric accuracy. Below, we present CD results on the HO3D dataset, comparing against the FSO [36] and NPS [39] baselines, where we observe substantial improvements over both.\n\n|    Method     | Early-Stage CD $\\downarrow$ | Mid-Stage CD $\\downarrow$ | Late-Stage CD $\\downarrow$ |\n| :-----------: | :-------------------------: | :-----------------------: | :------------------------: |\n| $FSO_{dist4}$ |            21.57            |           19.62           |           22.91            |\n| $NPS_{dist3}$ |            12.32            |           13.08           |           12.15            |\n|     Ours      |            1.31             |           1.14            |            1.03            |\n\n\n[d] TRELLIS: Structured 3D Latents for Scalable and Versatile 3D Generation (CVPR 2025)\n\n> Q2: After masking out the background, the task essentially becomes pose-free object reconstruction. In this case, it would be more appropriate to compare against other pose-free baselines, such as COLMAP-Free 3D Gaussian Splatting. Although these methods are optimization-based, for 3D asset generation, reconstruction quality and geometric accuracy is often more critical than reconstruction efficiency. Such comparisons would help position the contribution more clearly.\n\nWe appreciate the reviewer's perspective, however, we do not position our paper for 3D asset generation. We agree that for 3D asset generation, efficiency doesn’t matter that much given that existing methods can already produce 3D assets in less than a few seconds. Instead, the value added by our novel online formulation comes from downstream applications that require real-time 3D perception and continuous perception, in such applications, it values efficiency and bounded memory footprint. Therefore, we tackle the challenging balance between compute-memory efficiency and good reconstruction quality.\n\nAdditionally, in the Figure 4 of our main paper, we do show that using COLMAP-Free 3DGS with Markov Chain Monte Carlo (MCMC) does not work out of the box for (free-moving rigid) object reconstruction, where we also added joint pose optimization and employ ground-truth object masks on top of COLMAP-Free 3DGS to perform global optimization using all frames. despite extended optimization (over 6 minutes on a 4090 GPU), the end result could still be very unsatisfactory (with messy floaters & inaccurate geometry).\n\nWhile dedicated offline methods like GaussianObject[c] can achieve higher quality reconstruction, they do so through a complex multi-stage optimization pipeline, including computing visual hull -> optimize a coarse 3DGS model -> perform leave-one-out training -> fine-tune a LoRA for repair -> perform Gaussian repair, as detailed in [c], placing them in a different category from our online approach that can do on-the-fly reconstruction.\n\n[c]: GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splatting (TOG 2024)\n\n\n> Q3: The authors have not released the code, and the visualizations in the experiments are relatively limited. It is also unclear whether the test dataset includes objects that are entirely unseen in train dataset. The qualitative example of the \"blue kettle\" is not particularly convincing in terms of generalization. I would strongly encourage the authors to provide more diverse and challenging visual examples to support the claimed generalizability.\n\nWe thank the reviewer for these important points. We will release our code and pre-trained models upon acceptance to facilitate reproducibility. We can also confirm that our training and evaluation datasets (Objaverse and GSO/HO3D, respectively) are entirely separate, all test objects were **unseen during training**.\n\nWe also agree that more diverse visualizations would strengthen the paper. Unfortunately, NeurIPS policy prevents us from providing additional visual materials or external links during the rebuttal period, we will add more challenging qualitative examples to the technical appendix and a project website to better demonstrate the generalization capabilities of our method.\n\n> Q4: In particular, including videos that showcase the 3D Gaussian Splatting reconstruction results along with their corresponding input video sequences would provide more convincing evidence of the method’s performance. I would be happy to raise my score if the above concerns are adequately addressed during the rebuttal.\n\nWe thank the reviewer for this excellent suggestion. Unfortunately, NeurIPS policy prevents us from providing additional visual materials or external links during the rebuttal period, we fully commit to addressing this concern upon acceptance. Specifically, we will showcase more progressive qualitative results (like our Figure 1) in our technical appendix and provide more video-based demo on our website."}, {"review_id": "66qn9EJnSb", "reviewer": "Reviewer_kmHg", "summary": "The authors proposed a feed-forward reconstruction method in an online manner. This paper focuses on the reconstruction of free-moving objects. It introduces a dual-key memory model to fuse new features with the current state (object representation) via the Gaussian field. The authors claimed that such a spatial-guided memory readout keeps both the quality of object reconstruction and the computational cost. The authors also conducted a lot of experiments. However, as I asked in the following sections, I believe all of the experiments are about objects with rigid motions.", "questions": "Please refer to the weaknesses.", "limitations": "Please refer to the weaknesses.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The authors introduced an online reconstruction method to reconstruct the free-moving objects.\n2. The authors proposed a spatial-guided memory readout to achieve both reconstruction quality and computational efficiency.\n3. The authors conducted extensive ablation studies.\n\n**Weaknesses**\n1. What is the difference between the Cut3r[1] and the proposed method? The Cut3r model also learn the state information. I will consider increasing my ratings if the authors can provide a reasonable answer.\n2. This paper is limited to rigid motion. I think models like Cut3r and VGGT[2] can already tackle such tasks. So the problem settings and the motivation for this task are not obvious to me.\n3. In L262, I highly suggest that the authors make their target clearer. I believe dynamic object reconstruction includes rigid motion and non-rigid motion. The authors should clarify which type their work belongs to.\n4. In the experiments, I do believe that the quantitative experiments are not enough. I highly suggest that the authors conduct more comparisons with the current SOTA methods like Monst3r[3], Cut3r, and VGGT. \n\n[1] Wang, Qianqian, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. \"Continuous 3D Perception Model with Persistent State.\" arXiv preprint arXiv:2501.12387 (2025).\n\n[2] Wang, Jianyuan, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. \"Vggt: Visual geometry grounded transformer.\" In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5294-5306. 2025.\n\n[3] Zhang, Junyi, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. \"Monst3r: A simple approach for estimating geometry in the presence of motion.\" arXiv preprint arXiv:2410.03825 (2024).", "was_revised": true, "final_justification": "My concerns are addressed.", "rebuttal": "We appreciate reviewer `kmHg`'s constructive feedback and recognizing our method achieves \"**both reconstruction quality and computational efficiency**\".\n\n> Q1: What is the difference between the Cut3r[1] and the proposed method? The Cut3r model also learn the state information. I will consider increasing my ratings if the authors can provide a reasonable answer.\n\nWe thank the reviewer for this question regarding the recent work Cut3R[f]. While both our work and Cut3R[f] explore online feed-forward reconstruction, there are several fundamental differences in our approach and objectives.\n\n- Cut3R[f] tackles scene-level dynamic reconstruction, their output is a time-varying point cloud, whereas our method is object-level reconstruction, our output is a coherent object 3D representation. The difference in scope and final objective positions our paper quite differently. \n- The most critical distinction lies in how past information is aggregated. Cut3R[f]'s predictions are accumulative/additive, that is, it predicts new 3D points for each new frame without implicitly nor explicitly aggregating/aligning them with past predictions. That means, static regions in the scene will receive repetitive predicted points, and more importantly, dynamic regions (incl. moving objects) will receive multiple disconnected 3D instances of the same object in the output space. Specifically, if a rigid object is moving across the scene, the predictions of the object from Cut3R[f] will be scattered across the scene, creating multiple disconnected instances of the same object in the output space, and there is no way to aggregate them into a coherent object model. Thus, Cut3R[f] is unable to perform reconstruction for moving objects, which is the setting of our work.\n- Furthermore, due to the accumulative nature of Cut3R[f] in the output space, overlapping regions across views will receive repetitive predictions and they do not provide a principled way to align, nor remove those ever-growing unbounded number of points. Consequently, they also suffer growing memory usage, and significant drifting over time (as they acknowledged in their paper, and can be observed in their qualitative results).\n- In contrast, our method is implicitly aggregative for free-moving rigid objects. As we highlight in L132-L136 of our paper, our spatially-guided memory module is specifically designed to perform spatial coverage tracking, our dual-key memory readout strategy enables our OnlineSplatter transformer to learn to reason and align new observations with the past information at object level, such that we can discard out-dated past predictions. This enables our framework to produce a continuously-improving object representation with a fixed number of Gaussian primitives at every time step, effectively incorporating all past information while maintaining a constant memory footprint. \n- Cut3R[f] and our method both share the vision of incorporating a latent state to support online reconstruction, However, our method is tailored designed for object-level reconstruction. While Cut3R's prediction strategy serves well in understanding the dynamic scene, it does not serve well in our setting where we want to derive a continuously improving, well aligned 3D model for moving objects in online setting.\n- Having said that, if we try to adapt Cut3R[f] into our object-level settings and compare with them (although it is unfair to Cut3R and it's _not rigorous_ due to our fundamental differences in objectives), we extract the object region from per-frame predictions by Cut3R, and evaluate their object-level results in both per-frame mode (consider their per-frame output only) or accumulative mode (consider all past frame output combined). We provide the comparison on HO3D dataset and average across all frames and objects. Results show that our method outperforms Cut3R in both per-frame and accumulative mode.\n\n|        Method        | avg PSNR $\\uparrow$ | avg SSIM $\\uparrow$ | avg LPIPS $\\downarrow$ | avg Chamfer Dist. $\\downarrow$ |\n| :------------------: | :-----------------: | :-----------------: | :--------------------: | :----------------------------: |\n|  Cut3R (per-frame)   |       14.331        |        0.830        |         0.375          |             35.32              |\n| Cut3R (accumulative) |       14.760        |        0.882        |         0.419          |             59.26              |\n|         Ours         |       25.786        |        0.925        |         0.124          |              1.16              |\n\n\n\n> Q2: This paper is limited to rigid motion. I think models like Cut3r and VGGT[2] can already tackle such tasks. So the problem settings and the motivation for this task are not obvious to me.\n\nWe appreciate the opportunity to clarify our problem setting and motivation. \nWhile Cut3R[f] & VGGT[30] are impressive recent works, they do not address the specific challenges of online reconstruction for freely-moving objects.\n\nVGGT[30] is designed for static scenes. When applied to video with a freely moving object, it implicitly treats the object as a transient outlier, causing it to be lost in the final output. We experimented with VGGT on HO3D test clips: feeding the original video (without object masks) to VGGT fails the reconstruction entirely. Moreover, even when we perform preprocessing and postprocessing using ground-truth object masks, VGGT still struggles to reconstruct the object, producing misaligned and redundant predictions (as we show in our Figure 4 of main paper). Furthermore, VGGT is stateless and does not handle online reconstruction or the continuous perception needed for real-time applications.\n\nCut3R[f], as discussed in our response to Q1, handles moving objects by predicting separate point clouds for each frame. Without alignment or aggregation for moving objects, this results in multiple disconnected instances of the same object in the output space. This approach fails to leverage the continuous observation to build a single, coherent 3D object model, making it unusable in our object-level setting.\n\nOur work is motivated by the need for a continually improved 3D representation of a moving object, a realistic problem where an interacting agent (e.g. an autonomous robot) always observe an object partially first, and then interact with it, and gradually build up the complete understanding of the object. The continuous perception at object level is a problem that VGGT & Cut3R do not solve, while we achieve it with highly-efficient $O(1)$ memory-and-time update per time step.\n\n> Q3: In L262, I highly suggest that the authors make their target clearer. I believe dynamic object reconstruction includes rigid motion and non-rigid motion. The authors should clarify which type their work belongs to.\n\nWe agree that the term \"*dynamic object*\" (in L262) can be ambiguous when viewed without context. To clarify, our work focuses on the online reconstruction of *freely-moving* ***rigid*** objects in dynamic environments, (see Line 84). Here, \"*freely-moving*\" refers to the object is undergoing arbitrary rotations and translations while a moving camera observes them, (see Line 22). This eliminates the common assumption often existing in object reconstruction works [cite a few] that the object must be static, centered in frame, and fully visible without occlusion. \nBy alleviating these assumptions, our task becomes much more challenging and our method becomes much more applicable to real-world scenarios, for example, an autonomous robot observing a new object while handling it in a real household.\n\nTo improve clarity, we will revise the term \"dynamic object\" in L262 to \"freely-moving object\" and emphasize the rigid object assumption in other relevant parts of the manuscript to ensure consistency and clarity.\n\n> Q4: In the experiments, I do believe that the quantitative experiments are not enough. I highly suggest that the authors conduct more comparisons with the current SOTA methods like Monst3r[3], Cut3r, and VGGT.\n\nWe thank the reviewer for this suggestion. While MonST3R[e], Cut3R[f], and VGGT[30] are in related topics, they are designed for different tasks and settings than our work. A direct quantitative comparison would be challenging and potentially misleading, as the output and goals are not aligned. Specifically:\n\n- MonST3R[e] & VGGT[30] process a video as a whole in an offline setting, requiring all frames to be available upfront and fed into the model at once. It is not comparable to our online setting where video is fed as a stream, one frame at a time.\n- MonST3R[e] & Cut3R[f] focuses on scene-level dynamic environment modeling, understanding scene dynamics as a whole. They do not perform object-level reconstruction, and not easily adaptable to reconstruct free-moving objects.\n- In particular, Cut3R[f] outputs a per-step dynamic point cloud, but it does not aggregate object predictions into a coherent object model. Reference comparison is provided in Q1, but it is not rigorous due to our fundamental differences in objectives.\n- As a side note, both peer-reviewed papers of VGGT[30] & Cut3R[f] are only officially published on CVPR 2025 (released on June 3, 2025), after NeurIPS 2025 submission deadline. They are very recent, de facto concurrent works.\n\nHowever, we strongly agree that positioning our work relative to these methods is important. We will add a new discussion section in the paper that clearly articulates the differences in problem formulation, technical methodology, and objectives between our method and MonST3R[e], Cut3R[f], and VGGT[30]. We will also include qualitative comparisons where appropriate to visually highlight these distinctions and contextualize our contributions.\n\n\n[e] MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (ICLR 2025)\n\n[f] Cut3R: Continuous 3D Perception Model with Persistent State (CVPR 2025)\n\n[30] VGGT: Visual Geometry Grounded Transformer (CVPR 2025)"}]

Author rebuttal:

Meta-review: All reviewers recommend accepting this paper, congratulations! 

The paper proposes a novel feed-forward pipeline for dynamic object reconstruction with a dual-key memory module and sparsification, enabling real-time online performance from monocular RGB input. The approach is well-designed, achieves strong improvements on GSO and HO3D, and is methodologically distinct from prior work such as VGGT/CUT3R.

Reviewers highlighted the paper’s novelty, clarity, and empirical strength, but also noted limitations: reliance on known masks and supervised training, lack of geometric fidelity metrics, and missing comparisons to pose-free 3DGS baselines. Additional qualitative results, improved visualizations, and code release would further strengthen the work. Although I can see the motivation for questioning around the work on VGGT/ CUT3R, the problem setup and method itself is different enough. In addition, authors provided good rebuttal contents that is fairly convincing. Please include some of these discussion in the final version.

Overall, the rebuttal addressed key concerns and the strengths outweigh the weaknesses. I recommend acceptance, and encourage the authors to incorporate reviewer feedback—particularly on experimental completeness, geometric evaluation, and clarity. Given its novelty and potential applications, this paper is also a good candidate for a highlight presentation.
Decision: Accept (spotlight)