Deep Gaussian from Motion: Exploring 3D Geometric Foundation Models for Gaussian Splatting
Authors: Yu Chen
Rolandos Alexandros Potamias
Evangelos Ververas
Jifei Song
Jiankang Deng
Gim Hee Lee
Venue: NeurIPS 2025 poster ()
Keywords: Gaussian Splatting
Novel View Synthesis
Primary area: deep_learning

Abstract: Neural radiance fields (NeRF) and 3D Gaussian Splatting (3DGS) are popular techniques to reconstruct and render photorealistic images. However, the prerequisite of running Structure-from-Motion (SfM) to get camera poses limits their completeness. Although previous methods can reconstruct a few unposed images, they are not applicable when images are unordered or densely captured. In this work, we propose a method to train 3DGS from unposed images. Our method leverages a pre-trained 3D geometric foundation model as the neural scene representation. Since the accuracy of the predicted pointmaps does not suffice for accurate image registration and high-fidelity image rendering, we propose to mitigate the issue by initializing and fine-tuning the pre-trained model from a seed image. The images are then progressively registered and added to the training buffer, which is used to train the model further. We also propose to refine the camera poses and pointmaps by minimizing a point-to-camera ray consistency loss across multiple views. When evaluated on diverse challenging datasets, our method outperforms state-of-the-art pose-free NeRF/3DGS methods in terms of both camera pose accuracy and novel view synthesis, and even renders higher fidelity images than 3DGS trained with COLMAP poses.

Deep Gaussian from Motion: Exploring 3D Geometric Foundation Models for Gaussian Splatting
Yu Chen' Rolandos Alexandros Potamias
Evangelos Ververas*
Jifei Song"
Jiankang Deng²
Gim Hee Lee'
National University of Singapore
Imperial College London UUniversity of Surrey
LLFE Harrs
MiNNeRE360 Carden
Tarlo-and Trifo-and-Teeppllee Familyy
Pacenes- Fire
Sey Torko cerde--nd---aap-ee pemppe Church
COUMA7
Our
Gurs
COUMA
Ouna
ACED
Sea Banksaadd-empper Hore
MipNell2360 Counter
500
Tankyand-Temple Agnatiis
Figure I: Reconstruction results DeepGIM. Our method reconstructs scenes from unordered image collections without COLMAP poses. Top row: Reconstructed camera poses and point clouds. Bottom row: Rendered novel views compared with COLMAP ACE0.
Abstract
Neural radiance fields (NeRF) and 3D Gaussian Splatting (3DGS) are popular techniques to reconstruct and render photorealistic images. However, the prereq- uisite running Moructurr-from-Motteee completeness. Although previous methods can reconstruct few unposed images, they are not applicable when images are unordered or densely captured. this work, propose method to train 3DGS from unposed images. Our method leverages a pre- traitra 3D geometric foundation model as the neural scene rep- resentation. Since the accuracy of the predicted pointmaps does not suffice for accurate image registration and high-fidelity image rendering, we propose to mit- igate the issue by initializing and fine-tuning the pre-trained model from seed image. The images are then progressively registered and added to the training buffer, which is used train the model further. We also propose to refine the camera poses and pointmaps by minimizing point-to-camera ray consistency loss across multiple views. When evaluated diverse challenging datasets, our method outperforms state-of-the-art pose-free NeRF/3DGS methods in terms of both camera pose accuracy and novel view synthesis, and even renders higher fidelity images than 3DGS trained with COLMAP poses. Our project page is available httpss//aibbuuctsherr.g httpee/hssseeiiiiiiubooeeeieiii
(SIM) to get camera poses limits their
1 Introduction
In recent years, the rapid advancement of 3D reconstruction techniques has enabled diverse applica- tions for digitizing real-world scenes. Travelers and tourists routinely capture photographs and videos of landmarks, which online platforms now process using neural radiance fields (NeRF) 42 or 3D> Gaussian Splatting (3DGS) 26 to generate photorealistic 3D models.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite their fidelity, NeRF and 3DGS pipelines rely heavily on classical Structure- (SIM) methods like COLMAP 47 for precomputing camera poses. This decoupled setup introduces inconsistencies: SfM relies on sparse keypoint reprojection errors, while neural rendering optimizes dense photometric losses. Even minor pose inaccuracies can cause ghosting artifacts, degrading visual quality (cf. Bottom row of Fig.
This limitation raises a fundamental question: Can we train a fully differentiable reconstruction network guided solety by photometric loss? While recent work like CF-3DGS [9] avoids explicit pose estimation, it relies on consecutive-frast unordered images or large camera motions. Thus, unified framework for joint geometry estimation and rendering is critical for robust 3D reconstruction across diverse imputs.
poses and monocular depth priors, limiting generalization to
The 3D geometric foundation model DUSt3R 62 marked a milestone toward end-to-end neural SfM, inspiring pose-free 3DGS methods 49 69
However, these approaches handle only image pairs due to DUS(3R's limitations. Extensions like Spann3R [58] and its variants (VGGT [59] Fast3R [68], and FLARE (73)) fine-tuned DUS:3R on globally aligned pointmaps. thus extending the input of DUSI3R from just image pairs to entire image sequences. However, their geometric accuracy lags behind traditional SIM. Moreover, the GPU memory requirement of these methods grows rapidly when handling hundreds of images(cf. Table [. hindering high-fidelity rendering on consumer hardware. This motivates our work to fine-tune existing pre-trained DUSt3R variants [58 68 59
for high-fidelityy novel view synthesis from videos or unordered image collections.
We propose Deep Gaussian from Motion (DeepGfM), an end-to-end neural reconstruction frame- work that unifies geometry estimation and novel view synthesis for video sequences and unordered image collections. Our approach leverages pre-trained 3D geometric foundation model (e.g., Spann3R or DUS:3R variants) as a scene regressor, extending its output beyond pointmaps to 3D Gaussian primitives enabling direct optimization of scene geometry and appearance via photometric losses. To address the limitations of prior pose-free methods, we introduce a progressive training strategy with three core stages: 1) Initialization: The scene regressor processes images from training buffer, predicting pointmaps and 3D Gaussians in a global coordinate frame. We then compute initial camera poses via RANSAC and PnP. 2) Iterative Refinement: To mitigate pose inaccuracies, we propose point-to toaaterr- ray consistency loss, jointly refining camera poses and pointmaps. Newly registered images are added to the training buffer, and the scene regressor is fine-tuned on this expanded set using photometric losses. This loop continues until no further images can be registered. 3) Final Optimization: Accumulated errors are alleviated by re-optimizing all camera poses with the previous results as initialization, ensuring global consistency, Our framework archilecture-agnostii:: it retains the internal structure of the pre-trained regressor (e.g., DUS(3R's transformer backbone) while adapting its output space for Gaussian primitives. This design bal- ances generalization (inheriting geometric priors from foundation models) and flexibility (supporting diverse DUSt3R variants).
Experiments demonstrate that DeepGIM surpasses both pose-free NeRF/3DGS methods and COLMAP-based 3DGS in pose accuracy and rendering quality. While feed-forward pipelines like VGGT and Fast3R prioritize speed, DeepGfM addresses the reconstruction of dense image capture workflows to enable high-fidelity novel view synthesis without reliance on pose annotations or scale memory-intensin methods.
Our main contributions are summarized as follows:
Pose-free Foundation Model Adaptation. Unlike VGGT/MegaSAM, which rely on pre-computed poses (potentially affected by inaccuracies), our pipeline operates without pose annotations. This is achieved by refining Gaussian geometries dynamically to align photometric appearance with ray-consistent novel view synthesis.
Progressive and Modular Framework Design. The progressive design enables iterative scalability, addressing GPU bottlenecks evident in VGGT-like pipelines. Modularity ensures robustness against scene diversity, allowing refinement of components independent of memory constraints imposed by dense image sets.
Scene-specific Gaussian Prediction. Our method dynamically predicts Gaussian geome- tries for each input scene, adapting to its unique photometric and geometric character- istics for high-quality synthesis a flexibility less evident in feed-forward methods like VGGT/MegaSAM. which process inputs less adaptively.

---

Table
GPU memory usage (GB) of neural scene reconstruction methods with 200 images as input.
Method
CH-DGS 19
PRst3R 668
GPU Memory
OOM
OOM
33.17
40.63
23.07
2
Related Work
Neural Rendering. Neural radiance fields [42 enable rendering from novel viewpoints with encoded frequency features [54]. Many follow-up works try to improve the rendering and training efficiency 34 71 I0 by encoding scenes into sparse voxels, multi-resolution hash tables [4], or three orthogonal axes and planes [0O]. Another branch of NeRF methods focuses on generalizable
NeRF [T 52 (6). alleviating the aliasing 2
registering multiple blocks of NeRF [3]
Different from NeRF, 3D Gaussian Splatting 226 (3DGS) initializes 3D Gaussians from a sparse point cloud and renders scenes by differentiable rasterization, and can achieve real-time rendering performance. Follow-up works include learning the scene geometry implicitly by MLP (37] or GNN [57], fitting the surface via 2D)
and extending NeRF to city-scale scenes [33
55
45 40
65
Gaussians 23 21], alleviating the aliasing issue (72 67 31 50] improving the training efficiency [33
27 35
[4 for large-scale scenes.
Pose-Free Neural Rendering. To alleviate the reliance of SfM poses, NeRFmm (44 jointly optimizes NeRF and camera pose embeddings. BARF [32) proposes joint training of NeRF with imperfect camera poses from coarse- where high-frequencies are progressively activated during training alleviate the gradient inconsistency issue. GARF [ extends BARF with a Gaussian activation, enabling training a resitional-embeddee
less coordinate network. However, these methods can only handle forward-facte cameras or require accurate pose priors converge. Nope-NeRF and CF-3DGS [19] leverage the inter-frame relationship (e.g. relative poses) and monocular depth maps as regularization to train NeRF or 3DGS without precomputed camera poses, but limited to short image sequences. Moreover, Nope-NeRE and CF-3DGS are highly susceptible to failure when camera poses change significantly or images are unordered. InstantSplat [17] leverages the pre-trained DUSt3R (62) to regress dense pointmaps between image pairs, followed by computing camera poses by aligning the pointmaps into a global coordinate frame. However, aligning dense pointmaps is time- and memory-consuming. As result, InstantSplat can only handle very few images. Other related work includes Splatt3R [49] and NoPoSplat [69] to predict 3D Gaussians from pairwise unposed images. Other methods include introducing the SLAM pipeline into the
NeRF 51 74 or 3DGS 25
39
66
for indoor video sequences.
30 reconstructs dense pointmaps from image pairs. By aligning dense points, DUSt3R can obtain globally aligned camera poses and dense points. Though DUSt3R generalizes very well since it is trained on massive diverse datasets, it produces only local points and requires long post- processing time with a heavy GPU memory burden to obtain globally aligned results. Spann3R (58) is the first work that tries to eliminate the post-processing step of DUSt3R by utilizing spatial-memory to decode the global points. Concurrent to our work, improve the reconstruction speed and generality by fine-tuning DUSt3R on globally aligned 3D points. ACEZero (9) reconstructs from unordered image collections via MI.P and achieves faster reconstruction speed than COLMAP while with less accurate camera poses. Orthogonal to existing methods, our work focuses on the reconstruction accuracy in terms of both camera pose and novel view synthesis.
Neural 3D Reconstruction. DUSt3R 662
methods 668 73
3 Method
We first give the preliminaries of our scene regressor network in Sec. 3.1 and 3.2 followed by introducing our network architecture and progressive training strategy Sec. ig shows brief illustration of our network.
3

---

Training Buffer [g]
3D GS 6,
+ VIT Encoder
Rasterization
Pointmaps
Transformer Decoder
Multi-View Pose Rofinwmegtt
(Tinit)
(T.)
(T
Next Training Batch
Learnalle Parameters
Frozen Parameters
Stop Gradient
Initial Pose
Refined Pose
Figure The training pipeline of our proposed method. The method follows the classical incremental SfM reconstruction pipeline with the key difference that the input is no longer an image but pair of images in a progressively updated training buffer. The scene regressor network is trained as follows: 1) Use Spann3R (58) the scene regressor network to predict 3D Gaussians G5 and pointmaps X5 from pair of images. 2) Leverage RANSAC and a PnP solver obtain the initial camera poses based direct 2D-3D correspondences. 3) Refine the coarse camera poses by minimizing the point-to-ray consistency loss between 3D tracks and camera centers. 4) Rasterize the 3D Gaussians with the refined camera poses to render images. A photometrie loss is adopted for back- boock-ropaggting gradients. 5) After each training epoch, we update the training buffer by registering more images.
3.1 Preliminaries
Our network architecture is based on DUSt3R 62 and Spann3R [58]. Given an image pair (I, I2), DUS(3R predicts the corresponding pointmaps (fx+*), {X++)) for each image, where X denotes the pointmap Xj expressed in camera i's coordinate frame.
DUSI3R uses VIT [6] as a shared encoder for both images and two transformer decoders for the reference image i and the target image j, respectively. The two decoders denoted as reference decoder Per and target decoder Dug consist of two projection heads Hyets Hig that map the decoded features into pointmaps:
E V(L (fe (fg ii X (rer(( (1))
DUSt3R reconstructs image pairs in a local coordinate frame. When handling more than two images, DUSt3R a post-processing step align the pairwise dense pointmaps to global coordinate frame, which is time-consuming and can exceed the GPU memory limitation.
Spann3R proposes a feature fusion mechanism to predict pointmaps ((X**), {Xi:)} in global coordinate frame. It computes a fused feature f.¹ the 1-th training epoch from a spatial feature memory. The reference decoder inputs the fused feature for reconstruction, and the target decoder produces features for memory querying- Furthermore, Spann3R uses two additional projection heads to compute the key and query feature for reconstructing the next image pairs, and memory encoder Vinem which encodes the pointmaps from the reference decoder:
HCg (179), pK H (f.),, P. Vinem (X*9).
Although Spann3R can reconstruct out-of cittof-dditribution scenes, it reconstructs images individually and limited very short frames due GPU memory limitations. Moreover, the 3D points predicted by Spann3R in these scenes lack accuracy. We refer readers to 62 58 for more details.
3.2 Neural Scene Representation
Modern 3D scene representations fall into two categories: explicit (e.g., 3D Gaussian primitives (26 and implicit (neural networks 337 46 Our work builds on Spann3R (58), extending it to predict both dense pointmaps and 3D Gaussian primitives. We refer to Spann3R as the scene regressor
network fSCR, which analogs to the scene regressor pose regression or localization networks 6 9.

---

Unlike the sparse scene regressor that takes an image as input, our scene regressor takes an image pair as input and predicts dense pointmaps {X;} and per-pixel 3D Gaussians G {G;} in global coordinate frame:
(N,, Gz: K,, fSCK 1,)
where I, is the reference and I is the target image.
By rasterizing the set of 3D Gaussian primitives G, we pack-propagate gradients to the model using a photometric loss. More specifically, a 3D Gaussian primitive composed of the opacity o, the mean u, the covariance Σ, and the coefficients of the spherical harmonics SH. The covariance is decomposed into a rotation matrix and a scaling matrix $ to ensure the positive semi-definttness:: Σ, RSS R In addition, instead directly predicting the mean for each 3D Gaussian, predict an offset AX and apply to the pointmaps to obtain the mean u = X + AX. To render the color for pixel p. the 3D Gaussians are projected into the image space for alpha blending with the projected 2D Gaussian G
i-1
C - c II (1-))) j=1
where o; is the rendering opacity and is computed by a = o (p)) cj is the per-pixel color that computed from the spherical harmonics SH. In practice, Eq. computed using differentiable rasterizer [66]:
i- R K; u, R. S, SH) R(T, K; (G.))
(5)
where T is the camera extrinsics and K is the intrinsics.
3.3 Progressive Neural Reconstruction
Fine-tuning a pre-trained geometric foundation model such as DUSt3R on unseen scenes is challeng- they need ground-truth 3D data supervision. However, obtaining ground-truit 3D points is difficult and expensive. We circumvent this limitation through a self-supervised progres- sive approach.
ing. This is because
3.3.1 Seed Initialization
Given an unordered image set [I,), we first select seed image for initialization. This differs from incremental SfM, which requires a seed image pair with sufficient matching inliers and a wide baseline for two- view reconstruction. We follow similar criterion search for an initial image that overlaps with as many other images possible for faster convergence. To stabilize the training, use NetVLAD
compute global descriptor for each image. Then we compute similarity scores between image descriptors. We further build a similarity graph. where the node represents an image, the edge represents an image pair, and the edge weight represents the similarity score of that pair. We remove edge if its weight is lower than the threshold 8sim- We then select the node that has the maximum degree as the seed image Isedd Intuitively, node with maximum degree suggests has the most adjacent images, which is beneficial for image registration. After selecting the seed image, we finetune the scene regressor in a self-super manner. Specifically, we set the seed image as the reference frame with identity camera pose T Tred = I. We then compute photometric loss:
Ephoto
Σ I - i|l Σ II R(T K, (G.))
(6)
Note that, during initialization, the seed image serves both the reference and target image the scene regressor, and the camera pose is fixed as an identity matrix.
3.3.2 Progressive Registration
After seed initialization, we progressively increase the training buffer by registering images Ihar = (I&) in training epoch. The image registration stage includes: 1) coarse camera pose estimation step to obtain initial camera poses with the fine-tuned scene regressor. 2) a camera pose refinement step to improve the accuracy of registered cameras. With newly registered images, tune the scene regressor in the current epoch. Upon training convergence, we increase the training buffer by selecting more images. This process is repeated until all images are registered.
further fine-
5

---

Coarse Camera Pose Estimation. Given a registered reference image Let and unregistered target image Lx. we pass them to the scene regressor and obtain the 3D points {X+} in global
coordinate frame. Since we have the coordinates {yt) of each image pixel and their corresponding
3D coordinates [X+)- we can easily find the 2D-3D correspondences {(yk, X1)}. We then use RANSAC and a PnP solver to obtain a coarse camera pose:
Tk Ss = PnP(K, Xx))),
(7)
where Sş is the number of inliers and X = jsCk(T Is). Lee is the reference image and I is the target image we want to register. We add the target image I& into the training buffer only when the inlier number is larger than the inlier threshold Sintie-- After initialization, the seed image is selected
as the reference image. the following training batches, we select the reference image from the registered images that connect to most of the unregistered images.
Multi- View Consistent Pose Refinement. From the previous steps, the camera poses of newly registered images can still be inaccurate. This is because: 1) some regions in images have not been observed by the scene regressor; 2) each target image is registered individually, which lacks multiple view constraints. Although ACEO [9] uses an MLP pose refiner to alleviate this problem during training, we experimentally found that it does not improve pose accuracy with our transformer-basee scene regressor. This is because ACEO uses MLP as the scene coordinate decoder, and each pixel is individually mapped onto the 3D space. ACEO thus enables network training by mixing millions of pixels from mutiple views in training batch, and the multiple-view constraint helps constrain the network training. However, since we use a transformer-base decoder and due to the GPU memory limitation, we can use only a limited number of views in each training batch, which can easily diverge the network training. To solve the aforementioned issue, we propose to further refine the coarse camera poses by minimizing point-to-camera ray consistency loss which involves multiple views:
X,, T arg min Σ pllld;, ViAk -X, t2) R, E%
(8)
i,k
where t. is the camera position for image It, di, is the scaling factor between a 3D point X, and the camera position tg, Vi,. is the ray direction between X; and fμ. This loss function is more robust than the reprojection loss in ACEO (9 since the error in Eq. is bounded 775 And since the scene regressor already provided a coarse but reliable initial estimation, the convergence is fast. During optimization, we fix the camera pose the seed image for the gauge ambiguity and fix the scaling factor between the seed image and its most similar adjacent image for the scale ambiguity. Moreover, for a new training epoch, we fix the camera poses registered in the previous epoch and only optimize the camera poses registered in the current training epoch to improve the optimization efficiency. Fixing camera poses registered from previous epochs is important for our method to reconstruct scenes at larger scale because:
After the previous fine-tuning epochs, the camera poses and pointmaps are accurate enough.
By optimizing only local parameters, the GPU memory footprint is highly reduced. As a comparison, the global alignment step of DUSt3R and InstantSplat will encounter the OOM issue even with an 80GB A100 GPU.
After obtaining the refined camera poses, we fine-tune the scene regressor with a minor modification to Eq. 66 by:
Cyhoco I K, (G.))l..
(9)
3.3.3 Finalizing Neural Scene Reconstruction
We propose a two-stage strategy to improve the final reconstruction quality when all images have been registered or no more images can be added to the training buffer. The first stage to optimize all camera poses using Eq. (8). This is because we incrementally register images and errors accumulate during training. In this stage, we only fix the camera pose of the seed image, and camera poses obtained from all previous training epochs are used as initial values for optimization. Since the initial values are accurate enough, the final optimization converges very fast. To further improve the image rendering quality, we proposed to refine the scene details using explicit 3D Gaussian primitives [26] in second stage. This is because we only used fixed low-resolution images during the training of our scene regressor due to GPU memory limitations. The scene regressor can therefore only represent the coarse scene geometry. In the second stage, we use the same strategy as in (26) for 3D Gaussian densification and pruning during refinement.

---

Experiments
CF-3DGS
ACEO
Spunn3R
Ours
Frm
Fortees
Figure 3: Visualization of camera poses accuracy the LLFF dataset (Zoom in for best view).
Black: pseudo- soauud--round-trutfe camera poses obtained from COLMAP 47 Colored: predicted camera poses.
ACE0
Spann3R
Ours
Family
Francis
Figure 4: Visualization of camera poses accuracy on the Tanks-and- -al---nn--Temllee dataset (Zoom in for best view). Black: pseudo-ground- ground-trume camera poses obtained from COLMAP 47 Colored: predicted camera poses.
Evaluation Datasets. We evaluate our method on the LLFF dataset [41), the Mip-NeRF360 dataset 3 the mannks-andTTepllee dataset [29]. the sequential manks-and-Templee dataset (Seq TnT) [5). and the 7scenes dataset [8]]. The LLFF dataset contains scenes with cameras facing forward. The Mip-NeRF360 dataset contains different scenes where cameras are distributed evenly in 360 degrees in the 3D space. The Tanks-and hemplee dataset is similar to the Mip-NeRF360 dataset in camera poses distribution and scene scales, but with more illumination and appearance changes. Note that the Panks-and-Templer dataset is different from the sequential Tanks-and- Templeeessss dataset preprocessed by Nope-NeRF, where the Seq TnT dataset is extracted from video segments and contains fewer regions than the 360-degree Lanks-and-Temmpe Gaussians to represent and is less challenging. The 7Scenes dataset contains indoor scenes scanned with the Kinect camera.
dataset and thus requires fewer 3D
Table 2: Camera pose accuracy averaged on each dataset. The red orange and yellow colors respectively denote the best, the second best, and the third best results.
LLFF
MipNeRF330
Tanks-and Meakk------Temlees
Seq TnT
Scenes
AR
At
AR
At
AR
At
AR.
At
NeRFmm
64
1.04
0.423
77.14
2.837
61.40
2,083
0.48
1.735
BARF
39
0.45
0.331
76.38
3.371
69.50
1.227
0.44
1.046
Nope-NeRF S
0.45
0.184
77.12
1.659
58.33
1.078
0.04
0.080
DFS
19
1.27
5.087
0.07
0,041
ACE0 9
3.20
0.043
0.83
0.014
9.13
0.170
9.52
0.103
Spann3R
58
20.00
0.485
11.80
0.222
14.93
0.335
13.34
0.592
Ours
0.17
0.005
0.04
0.04
0.04
0.005
0.31
0.002
Implementation Details. We initialize the scene regressor network using the pre-trained Spann3R model [58). During training, use the image resolution of 512 x 512 for all the datasets. We use
7

---

a learning rate of le - 5 to finetune the scene regressor. We use Ssim = 0.3 to reject edges when building the similarity graph. We use DSAC 7 g to compute the camera poses for candidate image registration. Since DSAC supports only a single focal length, we modify it to use different focal lengths for the image x-axis and y-axis. We set the threshold of inlier number Sinlier to 5, 000 and the threshold of reprojection error to be within 6 pixels in DSAC. For the seed initialization, we finetune the scene regressor by 500 iterations. During the incremental training, we finetune the scene regressor by 1,000 iterations the LLFF dataset and 1,500 iterations the Mip-NeRF360 dataset. For the novel view synthesis task, images are downsampled by 4 during training and inference. We use the CUDA rasterizer provided by [39 to enable the gradient computation of camera poses during rasterization.
Camera Pose Accuracy. To evaluate the camera pose accuracy, we use COLMAP poses as ground- truth and compare our method with NeRFmm 64 BARF B2, and Nope-NeRt -eRF-NeRFF 5 which are pose- free NeRF methods. We also compare our method to the free 3DGS methods CF-3DGS [I9]. Besides pose-free NeRF/3DGS baased based methods, we also compare with the neural 3D reconstruction methods ACE0 [9] and Spann3R [58] with their official code. We present the quantitative results in T The unit for rotation error is degree, and the unit for translation error is dimensionless since the absolute scale in ground-truth is unknown. The results show that the camera pose accuracy of our method consistently outperforms all other methods. The pose-free NeRF methods perform very well on the LLFF dataset and the Seq TnT dataset, while failing on the MipNeRF360 dataset and the Tanks-and-Temmp Temples dataset. While CF-3DGS claims to be COLMAP- it performed badly on the LLFF dataset (cf. Fig. 8 and has similar performance on the Seq TnT dataset since it is only designed for handling sequential data. CF-3DGS also failed on the MipNeRF360 and Tanks-and- Templees dataset due to out-of- out-oo-memory (marked by "'). ACEO achieved fairly good results
almost all the datasets, but the accuracy on the LLFF dataset and the Seq TnT dataset is not as good as other methods except Spann3R. We conjecture this is because these two datasets have only forward-facin cameras, which provide fewer multi-view constraints than the Mip dataset. Notably, we found ACEO has large reconstruction error on the 'Francis' scene of the Tanks-and- Temples dataset (ef. the bottom row in Fig. 4 which suggests there is still room to improve ACE0 on robustness.
Table 3: Averaged Novel view synthesis results. COLMAP* and ACEO" denote training 3DGS with pose optimization. The red orange. and yellow colors respectively denote the best, the second best, and the third best results.
LLFF
MipNeRF360
Tanks-and- Temples
Seq TNT
Scenes
SSIM 1
LPIPS
PSNR ? SSIM
LPIPS
PSNR 1
SSIM
PSNR 1
SSIM
LPIPS
NcRFrum
22.84
0.640
0.460
11.23
0.410
0.812
10.31
0.332
0.881
22.50
0.39
0.54
BARF 32
23.97
0.626
0.238
12.92
0.428
0.501
11.55
0.366
0.840
23.42
0.610
0,540
Nope-NeRF
25.09
13.62
0:451
11.93
0.373
0.39
CF-3DGS 19
16.52
0.479
0.437
31.28
0.93
0.09
COI MAP
47
24.64
0,794
0.132
28.08
0.845
0.121
23.06
0.743
0.200
30.64
0.92
0.97
COLMAP
24.77
28.11
0.843
23.17
0.743
0.197
0.07
ACED (
22.74
0.709
0.182
24.31
0.663
0.269
20.93
0.661
0.302
24.32
0.81
0.19
ACPD*
22.71
0,709
0.183
24.32
0.664
0.271
20.67
0.644
0.324
24.49
0.82
0.18
Ouurs
28.19
23.17
0.745
30.96
0.06
We also provide the recovered camera poses in Fig. to show that our method also works on videos (left) and images with unevenly distributed camera poses (right).
Novel View Synthesis. We further evaluate our method on the task of novel view synthesis. which could provide better metrics to validate the accuracy of camera poses since COLMAP poses can contain errors. We provide the metrics of PSNR, SSIM 663, and LPIPS in Table For COLMAP and ACEO, we use the same framework to train 3DGS. The pose-free NeRF methods have low numbers in the MipNeRF360 and Tanks-and- -emm-pes dataset since they failed to estimate reliable camera poses. Surprisingly, we found our method obtains the best results in image rendering quality on almost all datasets and even outperforms COLMAP, showing that our method can compute more accurate camera poses than COLMAP. Though our method is not specifically designed for sequential images (in contrast to CF-3DGS), it still obtains comparable results to CF-3DGS and is even better than CF-3DGS LPIPS. To further highlight the importance of our method, we conducted experiments to show the difficulty of jointly optimizing camera poses with 3DGS. We enable the gradients backpropagation of camera poses during rasterization when training 3DGS with COLMAP and ACEO (denoted COLMAP* and ACEO* respectively). From Tab. 3 we observe COLMAP" is
8

---

X
Tanks Ballroom
7scenes fire
Figure 5: Reconstructed camera poses of DeepGfM on the indoor 7scenes and sequential Lanks-and- temples (Seq Tn'T dataset.
slightly better than COLMAP given highly accurate initial poses; In contrast, ACEO even worse than ACEO when initial poses are inaccurate. We also provide the qualitative results of novel view synthesis in Fig. 6 Fig. and Fig. g More qualitative results are provided in our supplementary.
Full Image
Ground Truth
BARF
COLMAP+3DGS
CF-3DGS
Ours
Figure 6: The qualitative results of novel view synthesis on LLFF forward-facing dataset
Full Image
Giroud Truth
BARF
COLMAP43DOS
CF-3DGS
Ours
Figure 7: The qualitative results of novel view synthesis on LLFF forward-facing dataset [41]
Ablation Study. We ablate the effectiveness of our incremental training step and camera pose refinement step in Table which are averaged on eight scenes the LLFF dataset. We can see that with the refinement step, the camera pose accuracy is improved significantly. Although our proposed method is initialized from the pre-trained Spann3R model, the convergence of the refinement step can still fail in cases where the derived camera poses grossly erroneous. We observe that the camera pose accuracy is improved with our incremental training pipeline (Ourscarne v.s. Spann3R), which provides much better initial values for refinement than simply from Spann3R.
further ablate the effectiveness of finalizing the camera poses in our training pipeline Table The unit for rotation emrrr is degree. We denote method without the finalizing step as Ours We

---

Full Image
Ground Truth
ACE0+3DGS
COLMAP+3DGS
Ours
Figure 8: The qualitative results of novel view synthesis the MipNeRF360 dataset (top two rows) and the Tanks-and- annes--ee-eeeplss dataset (bottom two rows).
Table 4: Ablation study of the camera pose refinement step.
Table Ablation study of the reconstruction finalization step.
Spann3R
[8)
Our%oure
Ourbaao
Scenes
Blcycle
AR
AR
At
At
At
0.096
0.015
0.043
0.877
0.15)
0,219
Avg 20.00
0.485
10.20
0.229
0.17
0.005
0.05
0,805
0.029
0.022
0,828
0
0
0.008
can see that the finalization effectively mitigates the error accumulation in both camera rotations and translations. We also emphasize that the camera pose finalizing step important the neural scene refinement step since 3DGS is sensitive to even small perturbations in camera poses. Moreover, jointly optimizing explicit 3DGS and camera poses during training has limited effect when camera poses are close to ground truth and can even diverge the training [70].
Conclusion
In this paper, we propose DeepGfM to reconstruct neural scenes from unposed images. Our method adopts pre-trained 3D foundation model scene regressor and leverages its learned geometry priors to ease the task of pose-free 3DGS training. Based on the learned geometry, we obtain coarse camera poses by RANSAC and PnP solver and refine them with a point-to-camera ray consistency loss. Our training pipeline incrementally registers the image batch into training buffer and progressively finetunes the model in self- superviseed manner. Our method surpassed saate-of-the-rt art pose-free NeRF/3DGS methods and even outperforms COLMAP 3DGS on multiple datasets.
Limitations. Though our method can produce higher quality reconstruction results in both camera poses and novel view synthesis, it requires more GPU memory and training time than ACEO since we are fine-tuning transformers. addition, our method uses Spann3R as the backbone of the scene regressor. Therefore, the training speed and the convergence rate depend on the performance of Spann3R. Moreover, when reconstructing larger-scale scenes, the spatial memory bank would require more GPU memory, which limits its application to larger scenes. To solve these limitations, future work may include the use of better variants of DUSt3R (e.g., Fast3R (8] and CUT3R (61) to enable reconstruction on larger scenes, adopting LORA
22
38
to improve fine-tuning efficiency.
10

---

This research project is supported by the National Research Foundation (NRF) Singapore, under its IRF-lnvestigatorship Programme (Award ID. NRF-NRF109-00083, Yu Chen is also partially supported by Google PhD) Fellowship.
Acknowledgement.
References
[1] Relja Arandjelovic, Petr Gronát, Akihiko Torii, Tomás Pajdla, and Josef Sivic. Netvlad: CNN architecture for weakly supervised place recognition. In IEEE Conference Computer Vision and Pattern Recognition, pages 5297-5307, 2016.
[2] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In IEEE/CVF International Conference on Computer Vision, pages 5835-5844, 2021. [3] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5460-5469, 2022.
[4] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In JEEE/CVF International Conference on Computer Vision, pages 19640-19648, 2023.
[5] Wenjing Bian, Zirui Wang, Kejie Li, and Jia-Wang Bian. Nope-nerf: Optimising neural radiance field with no pose prior. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver. BC, Canada, June 17-24, 2023, pages 4160-4169, IEEE, 2023.
[6] Eric Brachmann, Tommaso Cavallari, and Victor Adrian Prisacariu. Accelerated coordinate encoding: Learning to relocalize in minutes using RGB and poses. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5044-5053, 2023.
[7] Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. DSAC differentiable RANSAC for camera localization. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2492-2500, 2017.
[8] Eric Brachmann and Carsten Rother. Visual camera re-localization from RGB and RGB-D images using DSAC. IEEE Trans. Patlern Anal. Mach. Intell., 44()):5847-5865, 2022.
[9] Eric Brachmann, Jamie Wynn, Shuai Chen, Tommaso Cavallari, Áron Monszpart, Daniyar Turmukhambetov, and Vietor Adrian Prisacariu. Scene coordinate reconstruction: Posing of image collections via incremental learning of a relocalizer. In Computer Vision - ECCV 2024 18th European Conference, volume 15114, pages 421-440, 2024.
[10] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In Computer Vision ECCV 2022 17th European Conference, volume 13692, pages 333-350, 2022.
[11] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi view stereo. In IEEE/CVF International Conference on Computer Vision, pages 14104-14113, 2021.
[12] Yu Chen and Gim Hee Lee. DBARF: deep bundle-adjusting generalizable neural radiance fields.
In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24-34, 2023.
[13] Yu Chen and Gim Hee Lee. Dreg-nerf: Deep registration for neural radiance fields. In IEEE/CVF
International Conference on Computer Vision, pages 22646-22633 2023.
[14] Yu Chen and Gim Hee Lee. Dogaussian:
Distributed-oriente gaussian splatting for large-scale 3d reconstruction via gaussian consensus. CORR, abs/2405. 1,943, 2024.
[15] Shin-Fang Chng, Sameera Ramasinghe, Jamie Sherrah, and Simon Lucey. GARF: gaussian acti- vated radiance fields for high fidelity reconstruction and pose estimation. CoRR, abs/2204.05735. 2022.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
[17] Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang, Jian Zhang, Xinghao Ding, Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, Zhangyang Wang, and Yue Wang. Instantsplat: Unbounded sparse-view pose-free gaussian splatting in 40 seconds, 2024.
11

---

[18] Sara Fridovich- Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5491-5500, 2022.
[19] Yang Fu, Xiaolong Wang, Sifei Liu, Amey Kulkarni, Jan Kautz, and Alexei A. Efros, Colmap- free 3d gaussian splatting. In IEEE/CVF Conference on Computer Vision and Pattern Recogni- lion, pages 20796-20805, 2024.
[20] Lily Goli, Daniel Rebain, Sara Sabour, Animesh Garg, and Andrea Tagliasacchi. nerf2nerf: Pairwise registration of neural radiance fields. In IEEE International Conference on Robotics and Automation, pages 9354-9361, 2023.
[21] Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and bigh-quality mesh rendering. In IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 5354-5363, 2024.
[22] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, 2022.
[23] Binbin Huang. Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 Conference Papers, page 32, 2024.
[24] Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18344 18347, 2022.
[25] Nikhil Varma Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebas- tian A. -Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense RGB-D SLAM. In IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 21357-21366, 2024.
[26] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4)::39:1-139) 14, 2023.
[27] Bernhard Kerbl, Andreas Meuleman, Georgios Kopanas, Michael Wimmer, Alexandre Lanvin, and George Drettakis. A hierarchical 3d gaussian representation for real-time rendering of very large datasets. ACM Trans. Graph., (3(4):62:1-62:1)1 2024.
[28) Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, 2015.
[29] Arno Knapitsch, Jaesik Park, Qian-Y Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4), 2017.
[30] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In Computer Vision - ECCV 2024 18th European Conference, volume 15130, pages 71-91, 2024.
[31] Zhihao Liang, Qi Zhang, Wenbo Hu, Ying Feng, Lei Zhu, and Kui Jia. Analytic-pplating:: Anti-aliased 3d gaussian splatting via analytic integration. CoRR, abs/2403. 105/2403.11056, 2024.
[32] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: bundle-adjusting neural radiance fields. In IEEE/CVF International Conference on Computer Vision. pages
5721-5731, 2021.
[33] Jiaqi Lin, Zhihao Li, Xiao Tang, Jianzhuang Liu, Shiyong Liu, Jiayue Liu, Yangdi Lu, Xiaofei Wu, Songeen Xu, Youliang Yan, and Wenming Yang. Vastgaussian: Vast 3d gaussians for large scene reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
pages
[34] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In Advances in Neural Information Processing Systems 33, 2020.
[35] Yang Liu, He Guan, Chuanchen Luo, Lue Fan, Junran Peng, and Zhaoxiang Zhang. Citygaussian: Real-time high-quality large-scald scale scene rendering with gaussians. CoRR, abs/2404.01133. 2024.
[36] Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-aware image-based rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7814-7823, 2022.
[37] Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold- gs: Structured 3d gaussians for view-adeptive rendering. IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 20654-20664 2024.
12

---

[38] Ziqi Lu, Heng Yang, Danfei Xu, Boyi Li, Boris Ivanovic, Marco Pavone, and Yue Wang. Lora3d: Low-rank self-calibration of 3d geometric foundation models. CoRR, abs/2412.07746,, 2024. Hidenobu Matsuki, Riku Murai, Paul H. Kelly, and Andrew Davison. Gaussian splatting SLAM. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18039- 18048, 2024.
[39]
[40] Zhenxing Mi and Dan Xu. Switch-nerf: Learning scene decomposition with mixture of experts for large-scale neural radiance fields. In The Eleventh International Conference on Learning Representations, 2023.
[41] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph., (((4)::::1129:]]] 2019.
[42] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Michael Frahm, editors, Computer Vision ECCV2020 16th European Conference, Glasgow, volume 12346, pages 405-421.
[43] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41 10::): 2022.
[44] René Ranfil, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In IEEE/CVF International Conference on Computer Vision, pages 2159-12168, 2021.
[45] Konstantinos Rematas, Andrew Liu, Pratul P. Srinivasan, Jonathan T. Barron, Andrea Tagliasac- chi, Thomas A. Funkhouser, and Vittorio Ferrari. Urban radiance fields. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 12922-12932, 2022.
[46] Kerui Ren, Lihan Jiang, Tao Lu, Mulin Yu, Linning Xu, Zhangkai Ni, and Bo Dai. Octree-gs: Towards consistent real-time rendering with lod-structured 3d gaussians, 2024.
[47] Johannes L. Schönberger and Jan-Michael Frahm. Structure- f-wo-foooootte
revisited. In IEEE Conference on Computer Vision and Pattern Recognition, pages 41 4104-411 13, 2016.
[48] Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and An- drew W. Fitzgibbon. Scene coordinate regression forests for camera relocalization in RGB-D images. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2930-2937, 2013.
[49] Brandon Smart, Chuanxia Zheng, Iro Laina, and Victor Adrian Prisacariu. Splatt3r: Zero-shot gaussian splatting from uncalibrated image pairs. CoRR, abs/2408. 2024.
[50] Xiaowei Song, Jv Zheng, Shiran Yuan, Huan-ang Gao, Jingwei Zhao, Xiang He, Weihao Gu, and Hao Zhao. SA-GS: scale-adaptive gaussian splatting for training-free anti-aliasing. CoRR, abs/2403. 19615, 2024.
[51] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J. Davison. imap: Implicit mapping and positioning in real-time, In IEEE/CVF International Conference on Computer Vision, pages 6209-6218, 2021.
[52] Mohammed Suhail, Carlos Esteves, Leonid Sigal. and Ameesh Makadia. Light field neural rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2259-8269, 2022.
[53] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben P. Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In /EEE/CVF Conference on Computer Vision and Pattern Recognition,
pages 8238-82488 2022.
[54] Matthew Taneik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems, 2020.
[55] Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construc- tion of large-scale nerfs for virtual fly- throughs. In IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 12912-12921, 2022.
[56] Shinji Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Trans. Pattern Anal. Mach. Intell., 13(4):37--300 1991.
[57] Evangelos Ververas, Rolandos Alexandros Potamias, Jifei Song, Jiankang Deng, and Stefanos Zafeiriou. SAGS: structure-awand 3d gaussian splatting. CoRR, abs/2404. 19149, 2024.
13

---

[58] Hengyi Wang and Lourdes Agapito. 3d reconstruction with spatial memory.
CoRR,
abs/2408. 16061, 2024.
[59]
Jianyuan Wang. Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotný. VGGT: visual geometry grounded transformer. CoRR, abs/2503. 1651, 2025.
[60] Qianqian Wang. Zhicheng Wang. Kyle Genova, Pratul P. Srinivasan, Howard Zhou, Jonathan T. Noah Snavely, and Thomas A. Funkhouser. Ibrnet: Learning multi-view image-based rendering. In IEEE Conference on Computer Vision and Pattern
Barron, Ricardo
Recognition, pages 4690-4699, 2021.
[61] Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10510- 2025.
[62] Shuzhe Wang. Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jérôme Revaud. Dust3r: Geometric 3d vision made easy. IEEE/CVF Conference on Computer Vision and Puttern Recognition,, pages 20697-20709, 2024.
[63] Zhou Wang. Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment:
from error visibility to structural similarity. IEEE Trans. Image Process., 13(4):600- 6120 2004.
[64] Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. CoRR, abs/2212.00700 2021.
[65] Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, and Dahua Lin. Grid-guided neural radiance fields for large urban scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8296-8306 2023.
[66] Chi Yan, Delin Qu, Dan Xu, Bin Zhao, Zhigang Wang, Dong Wang, and Xuelong Li. GS-SLAM: dense visual SLAM with 3d gaussian splatting. In IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 19595-19604, 2024.
[67] Zhiwen Yan, Weng Fei Low, Yu Chen, and Gim Hee Lee. Multi-scale 3d gaussian splatting for anti-aliased rendering. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20923-20931 2024.
[68] Jianing Yang. Alexander Sax, Kevin J. Liang, Mikael Henaff, Hao Tang, Ang Cao, Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. CoRR, abs/2501. 139288, 2025.
[69] Botao Ye, Sifei Liu, Haofei Xu, Li Xueting, Mare Pollefeys, Ming-Hsuan Yang, and Peng Songyou. No pose, no problem: Surprisingly simple 3d gaussian splats from sparse unposed images. arXiv preprint 2:xiv:2410.242001 2024.
[70] Vickie Ye, Ruilong Li. Justin Kerr. Matias Turkulainen, Brent Yi, Zhuoyang Pan, Otto Seiskari, Jianbo Ye, Jeffrey Hu, Matthew Tancik, and Angjoo Kanazawa. gsplat: An spen-source library for Gaussian splatting. arXiv preprint arXiv:2409.06777
2024.
[71] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In JÉEE/CVF International Conference on
Computer Vision, pages 5732-5741, 2021.
[72] Zehao Yu. Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19447-19456, 2024.
[73] Shangzhan Zhang. Jianyuan Wang, Yinghao Xu, Nan Xue, Christian Rupprecht, Xiaowei Zhou, Yujun Shen, and Gordon Wetzstein. Flare: Feed-forward geometry, appearance and camera estimation from uncalibrated sparse views, 2025.
[74] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Mare Pollefeys. NICE-SLAM, neural implicit scalable encoding for SLAM. In JEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12776-12786, 2022.
[75] Bingbing Zhuang, Loong-Fah Cheong, and Gim Hee Lee, Baseline desensitizing in translation averaging. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4539-4547,
2018.
14

---

A Implementation
Neural Scene Regressor. We initialize part of our model with the pretrained weights from Spann3R (58) Following Spann3R, we use a ViT-large encoder, two ViT-base decoders, and DPT head 44 for predicting dense pointmaps, We additionally use another DPT head to predict 3D Gaussian primitives. Though Spann3R is trained images with resolution 224 224, we finetune on the reconstructed scene with image resolution 512 x 512 using Adam W [2 optimizer.
Image Registration. We use DSAC to register images and compute coarse camera poses. An image is successfully registered it has at least 5, 000 inliers with a reprojection threshold of 6 px. We use 64 hypotheses and an inlier alpha of 100. To accelerate registration, the dense pointmaps for each image are downsampled by 4. To further speed up the training and reduce memory footprint, we do not use all pointmaps from the newly registered images to refine camera poses. Instead, we pre-build sparse point tracks X = {(y,j. I,), where Yij denotes the j-th pixels observed on image
I, We adopt the Union-Find algorithm to remove duplicate and ambiguous tracks to improve the robustness during refinement. We adopt the Huber loss with a threshold of 0.1 as the robust loss function in Eq. 8
Pseudo Algorithm of Our Training Pipeline. We provide the pscudo algorithm of our incremental training pipeline as described in Sec. [33 in Alg. [ At line 1, V; denotes the set graph nodes, and E, denotes the set of graph edges. At line 4, T denotes the capacity of set. We align predicted camera poses to seeudo-ground-o camera poses and sparse points of COLMAP are normalized at the end of reconstruction. In line 14, we also normalize our predicted camera poses and dense points before refining the neural scene for fair comparison. We experimentally found that this can improve the training stability of 3DGS.
using the Umeyama [56] algorithm. Note that the
Algorithm 1 Incremental Neural Reconstruction Algorithm
Require: a set of (unordered) images {1,), maximum iteration per epoch itermax
Ensure: Camera poses (T,), }. 3D Gaussian primitives {9i}
1: Construct a similarity graph Gsim = (V, VV,&E) Eg)
2: Initialization from a seed image I (cf. Sec. 3.3.1
3: Registered image set (Ireg Treg) = { (Ieeed, T weed))
4: while Inggl < | do
5: Register a new image batch (Inew: Tnew) by Eq.
6: Refine newly registered camera poses by Eq. (
7: Update training buffer using (Inew; Tnew)
8: iter 0
9:
while iter < iter do
Finetune scene regressor fSCR using Eq. (9)
11:
iter := iter +
12: Ing : Treg + Inew, Trg: : Treg + Tnow
13: Finalize camera poses Trey = (T,) using 8
14: Normalize camera poses TTromm = Normalized Trag)
15: Finalize neural scene {Gi} (cf. Sec. 3.3.3)
B Additional Qualitative Results
We present more qualitative results in this Section. More reconstruction results of camera pose and pointmaps are included in Fig. 17
Ablation of Pose Refinement. We present the visual comparison of our method with (Ours,elme) and without (OurScoarse) the refinement step Fig. 9 As shown Fig. 9 camera poses are aligned closer to the ground truth after camera pose refinement. Compared to the camera poses obtained from Spann3R of the first row in Fig. and the third row in Fig. [O] the coarse camera poses are closer to the ground truth, which is coherent with the quantitative results provided in Table
15

---

Table 6: Ablation study of camera pose accuracy.
Spann3R
OurSouse
QurSycine
Scenes
AR
åt
AR
Át
AR
Åt
Fern
39.03
0.767
01.30
0.125
0.26
0.005
Flower
11.91
0.285
16.53
0.609
0.52
0.011
Fortress
08.31
06.07
Horns
06.98
0.349
14.23
0.145
0.03
0.001
Leaves
44.09
0.801
18.24
0.187
0.22
0.006
Orchids
09.77
0.256
07.22
0.255
0.006
Room
07.48
0,513
10.22
0.180
0.013
0.001
Trex
32.39
0.758
07.76
0.210
0.010
Avg
20.00
10.20
0.229
0.17
0.005
Forward-faete facing
Top Top-down
Forward-faceh facing
Top-down
Forward--aaing
Top-down
8
Vp
Ma
RM0
Spann3R
Ours (coarse)
Ours (refined)
Figure 9: Ablation of camera poses refinement on the LLFF dataset (Zoom in for best view). Top and bottom respectively the camera poses of the 'Fern' and 'Room' scenes.
visual comparison of camera poses the LLFF dataset in Fig. [0] The results on the MipNeRF360 dataset and more results on the Tanks-and- Temples dataset are respectively provided in Fig. II and Fig. 12] We can observe that CF-3DGS [19 failed to produce faithfully camera poses on the LLFF dataset, which has been analyzed in the main paper. performs very well the M6pNeRI330 dataset, struggles
More Results of Camera Poses. We provide
While ACEO
the challenging Siiis-anddTeemplee dataset (cf. The 'Train' scene in Fig. T2] Moreover, we find that ACEO performs poorly on the LLFF dataset. This may be due to the MLP decoder of ACEO maps individual pixels to 3D space, while it requires well-distributed training views to constrain the network. However, the LLFF dataset contains only forward- facing cameras, which do provide strong constraints from different view directions and therefore degenerate the training of ACEO.
9
We further provide the camera poses of method the indoor 7scenes dataset in Fig. 13 and the camera poses of our method on the sequential aanksannd-tmppes dataset in Fig. 14 show that our method can work on diverse scenes and video sequences.
Table Quantitative results of camera pose accuracy on Mip.Ne dataset.
ACE0 9
Spann3R (88
Ours
Scenes
At
AR
AR
Bicycle
Counter
0.83
0.014
11.16
0.226
0.029
0.002
Garden
0.82
0.012
15.65
0.279
0.028
0.002
0.83 0.014
11.8
0.222
0.04
0.004
More Qualitative Results of Novel View Synthesis. We present more qualitative results on novel view synthesis task in Fig. 15 and Fig. 16 The quantitative camera pose accuracy only reveals how close the predicted camera poses are to the COLMAP poses. However, cannot distinguish which is more accurate since COLMAP poses only pseudo-ground-truth and it produce
16

---

Table 8: Quantitative results of camera pose accuracy Tanks-and-Temples dataset.
ACEO
Spann3R
Ours
Scenes
AR
At
AR
At
AR
At
Family
6,41
0.088
16.98
0.378
0.036
0.003
Francis
21.63
0.351
14.19
0.361
0.030
0.002
Ignatius
2.96
0.043
11.23
0.313
0.028
0.002
Train
5.52
0.196
17.33
0.286
0.065
0.011
Avg
9.13
0.170
14.93
0.335
0.04
0.005
Table 9: Quantitative results of camera pose accuracy 7scenes dataset.
ACE0
Spann3R
Ours
Scenes
AR.
At
AR.
At
AR.
At
chess
0.590
0.016
6.468
0.300
0.082
0.010
fire
0.680
0.016
8.662
0.346
0.116
0.004
heads
1.141
0.038
9.258
0.464
0,420
pumpkin
0.616
0.022
7.731
0.551
0.141
0.012
Avg
0.757
0.023
8.033
0.415
0.440
0.010
CF-3DGS
ACE0
Spann3R
Ours
Room
Ordiids
Figure 10: More qualitative comparisons of camera poses accuracy the LLFF dataset (Zoom in for best view). Black: neeudo-groun--rrut camera poses obtained from COLMAP 47 Colored: predicted camera poses.
17

---

ACEO
Spann3R
Ours
Bicycle
Kitchen
Bicyele
Kitchen
Figure More qualitative comparisons of camera poses accuracy on the MipNeRF360 dataset (7oom in for best view). Black: seeudo-rouun-trutte camera poses obtained from COLMAP 47 Colored: predicted camera poses.
ACE0
Spann3R
Ours
Ignatius
Thain
Figure 12: More qualitative comparisons of camera poses accuracy on the Tanks-and-Temples dataset (Zoom in for best view). Black: pseudo-ground- camera poses obtained from Colored: predicted camera poses.
COLMAP 47
7scenes chess
7scenes fire
7scenes heads
7scenes pumpkin
Figure 13: Visualizations of our camera poses the 7scenes dataset.
18

---

Ballroom
Barn
Church
Family
Figure 14: Visualizations of our camera poses the Sequential Banks-and-Temples Temples dataset.
erroneous camera poses. Nonetheless, the results of novel view synthesis provide better metrics to show which one better when two camera poses are close. In Fig. 15 and Fig. 16] we can observe that method can render finer details when zoom into the same areas. The visual comparison also provides coherent support to the quantitative results of novel view synthesis in Table 15 and Table 16
FIRE
FIRE
FIRE
FIRE
Full Image
Ground Truth
ACE0+3DGS
COLMAP:3DGS
Ours
Figure 15: More qualitative comparisons of novel view synthesis the 7Scenes dataset. From top to bottom are scenes of fire, heads, pumpkin, and chess.
C More Discussion
Differences with (neural) incremental SIM methods. Our incremental training pipeline is similar to the classical incremental SfM method 47 but differs follows:
Seed Initialization. Incremental SfM initializes from an image pair, where the camera poses the image pair are fixed after initialization to fix the gauge freedom. However, our method initializes from pretrained model and only seed image.
19

---

M
10000000
Full Image
Ground Truth
ACDO-3DO 3DGS
COLMAP- 3DGS
Ours
Figure 16:
More qualitative comparisons of novel view synthesis the sequential Tanks-and- Temples dataset. From top to bottom are the scenes of the ballroom, barn, horse, and church.
Objective Function. We fine-tune our model using a rendering loss while scenes are optimized by the reprojection error in SfM.
Scene Sparsity. Our model predicts dense scene geometries, while SfM outputs sparse scene structures.
Our incremental training pipeline also shares some similarities with recent learning-based SIM method ACEO 19], with several key differences: 1) ACEO only predicts sparse pointmaps, while our method predicts dense pointmaps and 3D Gaussian primitives. as the neural scene representation, while we use transformers as the scene representation. 3) The training batch ACEO composed of pixels from multiple views, while our method takes as input image pairs in the training batch. Moreover, fine-tuning a pre-trained foundation model such as DUSt3R not easy. This is because these 3D foundation models are supervised by ground-truth 3D> points that are difficult to obtain in unseen scenes.
ACEO uses 2D CNN and MLP
On the ILFF,, 7Scenes and Sequential Tanks-and- Templees, our method converges in two epochs, which takes about 25 minutes for each scene; On the MipNeRF360 dataset and the Tanks-and Temples dataset, our method converges in 5 15 epochs, which takes about 2 hours for each scene. We evaluate the model during training and save intermediate results to disk for every 1, 000 iteration. The evaluation time is also included the training step. Our method takes about 21 GB with batch size of during training on an NVIDIA 4090 GPU.
Run Time and Memory Footprint Our pipeline converges faster than COLMAP
D Per-Scene Breakdown
We provide per-scene quantitative results for both camera pose accuracy in Table TO
Table 7 Table 8
Table 9 and Table T8, as well as novel view synthesis Table [T] Table 14 Table T5]
Table T6
Table 17
Table 19 and Table 20 Note that, for the MipNeRF360, Tanks-and-Templess and 7Scenes
20

---

Figure 17: More visual reconstruction results on real-world datasets.
datasets, NeRFmm, BARF, and Nope-NeRF failed to obtain reasonable camera poses and clear novel views, thus, we did not present the per-scene quantitative results in tables for readability.
Table 10: Quantitative results of camera pose accuracy LLFF dataset.
Scenes
Fimm
Nope-NeRI
BARF 32
DBARK
ACE9 0
CF -INGS 19)
Spann33 X
Ours
ARR
2.R
2.R
A1
At.
Fern
1,82
0.706
0.99
0.11
0.39
0.064
2.81
9.254
0.767
0.005
Flower
0.42
0.086
0..10
0.035
0.25
0.224
1.39
0.318
6.73
0.031
0.24
2.586
11.91
0.285
0.52
0.011
Fortress
0.74
0.233
0.30
0.081
0.48
0.364
0.59
0.220
0.98
0.010
1.28
8.592
8.31
0.152
0.04
0.002
Homs
0.30
Leaves
4.63
0.036
7.350
Orchids
2.03
0.686
0.38
0.203
0.63
0.404
1.16
0.573
1.19
0.093
1.45
2.772
9.77
0.256
0.24
0.006
Room
L66
0.670
0.94
0.244
0.32
0.270
0.53
0.360
1.06
0.041
1.36
3.336
7.48
0.513
0.03
0.001
Trex
0.78
0.542
0.32
0.219
0.11
0.720
1,06
0.163
1.73
0.059
1.56
4,431
32.39
0.758
0.03
0.010
Aog
Table 11: Quantitative results of novel view synthesis LLFF dataset.
Sunm
L/11%
PSN11
195-8
PENN
15190
15581
PS36
551011
19N8 LAFS
23:79
1.540
8:320
38.17
11293
11:64
28.17
0.9:2
0136
35.91
0.43
04
1.501
11:12
0.155
13:57
5.40
8
5
6130
0.333
35.73
0.31
1.38
13.99
0.612
0.56
30/0
0.33
11:3
31
1331
61131
300
1180
0.03
835
01:81
09.33
18181
61%6
0170
1418
1.1
1188
0117
2.70
8.300
2382
03.39
6222
0434
24.24
0.80
1148
23.52
1.90
0/84
Aeg
166
819
08
6191
74.0
3716
18:97
647
049
76.1
0.71
1110
1.30
1020
8119
E Social Impacts
Since is the era of large foundation models, popular for more and more works that dive into training generalizable geometric foundation model and then use it to reconstruct scenes in a fast
21

---

Table 12: Quantitative results of novel view synthesis on MipNeRF360 dataset.
Bonal
11.38
0.388
0.300
01.34
02033
08.33
16.34
0.301
11.33
0803
02833
14.45
8.55
0.833
34.31
10091
00097
36.38
0203
9,45
Rev
23:25
6.300
0333
2148
23.11
5.90
1338
21.08
01553
0.65
11:09
0352
03:29
21.23
0.312
0.50
keen
04
1.111
817
0700
0991
01095
81
8888
0.1%6
11.09
01083
0.75
8.30
6.20
0.35
0001
8:00
11:14
(16
8:11
23.00
0.300
6.53
53.00
0/25
0.333
22.35
841
0.36
22.88
016
0.44
6414
0.390
23.60
066.8
0397
2500
0401
0,37
13.
6881
0:11
71445
0.88
09135
13.35
689
6:16
02.10
07555
0.153
0311
6:19
09.15
06006
01197
1358
0.10
Table 13: Quantitative results of novel view synthesis on Tanks- -andks-and--a Templess dataset.
Scame
30115
1.999
Diaaly
$3.75
3.714
18:10
14:08
0.734
1:19
22:45
3.744
20.11
01681
01:31
11.81
0,242
21.63
63342
3.341
State
$2.38
04830
0.43
23.81
8.58
0.15
12.03
0825
1-15
23.00
1324
3113
12.34
0.33
0.19
23
3.85
11145
23.55
0331
8144
30.55
01645
81348
28.45
8419
0942
0220
8808
0607
014
0.881
7183
1314
22.34
23.99
8338
0.330
Table 14: Quantitative results of novel view synthesis with camera pose optimization on LLFF dataset.
COLMAP' 3DGS
COLMAP
3DDDS
opt LPEPS
ACE9 opt
FSNR
SSIM
[.PIPS
PSNRR
550M
PSNRR 1
5SIM
LPIES
PSNR.
SSIM
LPEPS
PSNNR
SSIM
1.PIPS
Fer
23.65
0,794
0.335
23.58
0.791
1.133
19.33
0,585
0.295
19:39
0.555
0.250
23:27
0,257
0.314
Flowzr
26.91
0.820
0.095
26.92
0.830
0,005
25.84
0.787
0.109
25.71
0.786
0.110
25.81
0.312
0.108
Reerras
29:93
0.380
0.073
29.92
0.580
0.077
29.09
0.848
0.855
29.21
0.849
0.055
20.31
0.368
0.084
Home
20,02
0,301
0.321
23.72
0.854
0.140
24.99
0.821
0.137
26.93
0.819
0.142
26,07
0,381
B.003
Lemes
17:91
0.391
0:305
18:06
0.201
17.09
0.522
0.242
17:10
0.319
0.244
16.45
0.3.5
0.3.0
18.98
0,612
0.159
15.15
0.417
0.347
Room
0.315
29.76
25.89
0.894
0.073
22.25
24.66
0.794
0.153
24.77
0.134
22.74
22.71
0.709
0.123
Table 15: Quantitative results of novel view synthesis on MipNeRF360 dataset.
Scomes
COLMAP 3DGS
COLMAP
33DDSS
opt
ACE8 30MGS
ACBI
-3DGS pase
PSNR. 7
SSIM
LPIPS
PSSSR
553M
LPSPS 1
|PSNR 1
SSIM 1
LPEPS
PSNRR1
SSIM 1
LPSPS
PSNNR "
SSIM
1.IISS
Blaycle
23:23
0.655
0.282
23.19
0.648
0.291
21.08
0.413
0.523
21.02
0.410
0.537
23.86
0.725
0.181
Couneer
29.26
0.916
0.081
29.33
0.915
0.091
26.34
0.824
0.157
25.38
0.826
0.152
29.34
0.917
0.080
27:46
0.361
0/079
27:48
0.861
0.79
0.883
0.287
33:97
0.990
0.283
0.363
0,077
Kinche
0,947
0.082
32.43
27.03
26.89
0.300
0.130
32.12
0,043
Aug
0.121
24.31
0.644
Table 16: Quantitative results of novel view synthesis on Tanks-and- Tem-les dataset.
COLMAP-3DGS
COLMMAP3DDUS upt
ACE0+3DGS
ACE9+3DGS opt
Ours
PSNR
1.PPPS
PSNR
5SSM
1.POPS
PSNN
SSIM
[.PIPS..
Family
23.04
0.147
0.713
20.45
0.259
19.32
0.591
0.352
25.11
0.775
0.144
Francis
26.42
26.38
0.546
24.08
0.796
0.214
24.18
0.795
0.275
26.70
0.545
0.182
Igunii
22.07
0.671
0.228
21.50
0.650
0.320
30.5
0.615
0.273
36.02
0.614
0.271
21.95
0.669
01250
Teain
20.72
0.685
0.235
30.74
0.686
0.235
18.29
0.580
0.400
18.34
0.57)
0.398
20.03
0.689
01230
Avg
23.06
0.743
0.200
23.17
0.743
0.997
20.95
0.661
0.302
20.67
0.644
0.324
23.17
0.745
0.197
Table 17: Quantitative results of novel view synthesis on 7scenes dataset.
Scemos
COLMAP-3IGS
COIMAAP.IDGSSppee..
ACRIANDGS
ACRIANDGSSpoon up1
Ours
PSNR
SSIM
LPBPS
PSNI
SSIM
1P1P5
PSNK
SSIM 1
1
PSNK 7.
SSIM 1
LIPPS
PSNN 7
SSIM 1
L.MIPS
chess
26.13
0.872
0.151
26.13
0.575
0.151
25.48
0.855
0.165
25.52
0.858
0.150
26.00
01872
0.154
fre
2602
0,805
0.195
26.17
0.507
0.187
25.11
0.714
0.301
25.38
0.765
01210
25.67
0.800
0.193
hrads
2503
0.643
0.225
3536
0.540
0.230
0.830
0.288
34.12
01861
01155
21.68
0.854
0:111
pampkin
1662
0.864
0219
1663
03964
0.20
0.855
0212
36.8%
0.660
0883
0211
Aug
0.566
0197
35.30
0.99
0,8%
029
35311
0.86
0.194
22

---

Table 18: Quantitative results of camera pose accuracy the sequential Tanks-and-Teni Temples
dataset.
BARF
NeRFmm
Nope-NeRF
CF-3DGS
ACE0
Ours
Scenes
AR
AR
At
AR
AR
At
AR
AR
At
Ballroom
0.228
0.531
0.177
0.449
0.018
0.041
0.024
0.037
8.047
0.018
1.303
0.002
Barn
0.265
0.314
0.494
1.629
0.032
0.046
0.034
0.034
10.488
0.035
0.161
0.001
Church
0.038
0.114
0.127
0.626
0.008
0.034
0.018
0.008
4.316
0.023
0.136
0.004
Family
0.591
1.371
2.743
0.015
0.047
0,024
0.022
1.589
0.111
0.002
Francis
0.558
1.321
1.647
0.009
0.057
0.154
18,370
0.429
0.002
Horse
0.394
1.333
0.434
1.349
0.017
0.179
0.057
0.112
2.439
0.023
0.216
0.007
Ignatius
0.324
0.736
0.379
1.302
0.005
0.026
0.057
0.122
2.434
0.025
0.133
0.002
Museum
1.128
3.442
1.051
4.134
0.202
0.207
0.215
0.052
28.510
0.648
0.227
0.010
Avg
0.441
1.046
0.477
1.735
0.038
0.080
0.069
0.041
9.524
0.103
0.313
0.002
Table 19: Quantitative results of novel view synthesis the sequential Tanks-and- dataset.
BARY
NilFien
Nape-helt
CI-BIG
ACEIOOOGS
EOEMAP-IDG
20111
LFBSS
POURT
S0:4
199%
SSGM
18FS
POAKT
19995
201R
70AR
5004
LIT7S
6004
$1/66
313
1.M1
1.37
35.33
9.3
13.23
12.38
1.91
33.13
1.90
1:91
Clura
25.17
0/5
1.55
11.64
3.50
3.56
25.11
1.15
0.30
33.23
8:33
1.11
26.47
1.90
8.90
34.4
8.43
8.36
23.00
8.55
0.36
11
(1
8:41
811
1:11
0.22
1.4
9.36
8.90
Navee
33.56
811
32.31
1.4
1.5
34.71
3.1
8.3
36.91
8:11
1111
30.00
1.55
8:1
1.6
8:3
13.1
8.84
0.0
heed-forward manner. Regardless of its efficiency in inference, the accuracy of the reconstruction results is still not well solved. We believe that many variants of DUSt3R will benefit from our work in improving their reconstruction accuracy.
Table 20: Quantitative results of novel view synthesis with camera pose optimization the sequential Tanks-and-Tena Temples dataset.
COLMAP
pose opt
3-GGS
posc opt
1PPIPS
IPPISS
LPEPS
PSNR
Ballmaon
34.31
0.36
0102
14.42
0..97
01022
22.28
0.50
0.17
35.24
0.86
0.11
35.15
03:97
0.01
Bam
32.05
0.95
11.04
32.20
0.95
0.04
25.88
0.87
0.15
25.82
0.87
0.14
31.81
0.95
0.04
Cturch
50114
0.951
1106
20.55
(191
(105
28.47
0230
11.9
28:32
0190)
0109)
-9830
0.993
0.26
Fanily
28.74
0.93
0.07
28.54
0.93
0.07
23.31
0.85
0.15
23.40
0.85
0.15
27.99
0.92
0.06
Franiis
32.00
0.92
0.11
32.24
0U92
0.11
27.14
0.83
0.24
26.72
0.83
0.24
31.76
0.92
0.11
Home:
0.81
0.81
0.92
0.92
0/66
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Refer to the abstract and the last paragraph of Sec. [oor our contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Sec. 5 in the conclusion section for the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Implementation details are included in Sec. in the main paper. Additional implementation details are included in Sec. A the supplementary. Pseudo algorithms are included in Sec. in the supplementary for better understanding and reproducing the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release related code and checkpoints upon the acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iips.cc/pill wttpcccppppcccceeeeeeeeee..c.ett httpe//gggeeee/eeeeubbissssssss /Cc////oddiiisssseeeeeee ssiiiiiiccyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Training and testing details are included in Sec. 4in the main paper. Additional details are included in Sec. i the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We follow related works for the setting of error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
26

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources and running time are included in Sec. in the main paper and Sec. Cin the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We strictly follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the social impacts of our work in Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper has such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We included the urls and cited the papers for code and datasets used in this paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
28

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We use and cite existing datasets in this work. Other assets including related code/models will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
29

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
30


Reviews:
[{"review_id": "PvUB01RxZb", "reviewer": "Reviewer_JX9E", "summary": "The paper proposes to fine-tune a 3D foundation model (Spann3r) to directly predict 3DGS parameters via incremental registration of the scene. By mimicking the incremental mapping step in Sfm, this method tries to register images starting from a seed image and progressively increasing its training buffer. At each step, it adds an image to the buffer, predicts 3DGS parameters, refines camera poses and then applies a photometric loss on the predicted GS parameters. The authors show improved camera pose estimation and NVS results on multiple datasets compared to prior art.", "questions": "- Since the method is agnostic to the Dust3r variant used, it would be good to see how Dust3r faired in comparison to DeepGfm for pose estimation and nvs. \n- How would the model scale with respect to VRAM and processing time with increasing in #images?\n- L226-228: Are the photometric losses also applied on the low resolution ground-truth?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths  \n- Quality improvement over Spann3r is impressive. Spann3r seems to fail on many sequences whereas DeepGfm seems to work much better thanks to the incremental camera optimization strategy. \n- The novel view synthesis seems to be at par with COLMAP trained 3DGS while the camera pose estimation seems strong as well. \n\nWeaknesses  \n- Choice of pretrained model; It seems like Spann3r is a worse pretrained model than Dust3r/Mast3r/Mast3r-Sfm discarding the OOM issue for large sequence lengths. Its possible that a lot of the finetuning goes in just getting camera poses to look reasonable from the poor performance of Spann3r. Moreover, Spann3r is meant for online scene reconstruction whereas this paper uses it as part of a two-stage finetuning strategy. Spann3r's memory retrieval architecture is suited for sequential data so using unordered collections might hurt more than expected. The motivation to use Spann3r background is weak.\n- Splat prediction head training: Since this model is finetuned per scene, what is the benefit of the splat prediction head? Isn't it equivalent to optimizing for 3DGS parameters directly as in vanilla 3DGS? What cost/speed/quality benefit does the head provide?\n- This method can be treated like a more efficient global alignment step compared to Dust3r's. Consequently, it would be good to compare results with Mast3r-Sfm on camera pose estimation.\n- Dust3r comparisons missing from Table 2\n- Missing NoPo-Splat, Splatt3r and MV-Dust3r+ comparison for splat quality.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback highlighting the strengths and weaknesses of our work. We appreciate your recognition of DeepGfM's improved quality over Spann3r and its strong performance in novel view synthesis and camera pose estimation. Below, we address your concerns point by point.\n\n**1. Choice of Pretrained Model (Spann3r)**\n\nWe acknowledge the reviewer’s concern about the choice of Spann3r as the pretrained model, especially given its weaker performance compared to Dust3r, Mast3r/Mast3r-SFM. However, our decision to use Spann3r stemmed from several considerations:\n\n**(1) Balancing Memory Constraints with Scalability:**\nSpann3r enables scalability and model initialization for large-scale image sets due to its lightweight memory retrieval architecture. While Dust3r and Mast3r provide stronger pretrained baselines, their memory limitations (outlined in their respective publications) often result in out-of-memory (OOM) issues for dense or extensive image sequences. Spann3r’s architecture allows foundational initialization for unordered collections without the prohibitive memory costs of Dust3r/Mast3r.\n\n**(2) Two-Stage Fine-Tuning Strategy:**\nBy designing DeepGfM’s incremental camera optimization over Spann3r’s pretrained framework, our method significantly improves camera pose estimation even when initialized with weaker models like Spann3r. This setup reduces dependency on the pretrained model’s inherent accuracy and allows DeepGfM to overcome challenging sequences effectively.\n\n**(3) Online Scene Reconstruction Background:**\nWe agree that Spann3r’s retrieval structure is indeed tailored for sequential rather than unordered data collections. However, our two-stage fine-tuning specifically compensates for unordered collections by progressively aligning photometric and geometric features during camera pose refinement. As demonstrated in the manuscript results (Figures 4-5), this approach consistently improves Spann3r’s outputs, capitalizing on its resource-efficient initialization while mitigating its design limitations.\n\n**2. Splat Prediction Head Training**\n\nWe appreciate the question regarding the benefit of introducing a splat prediction head in our pipeline and its equivalence with optimizing vanilla 3DGS parameters. The splat prediction head offers several advantages:\n\n**(1) Scene-Specific Fine-Tuning:**\nUnlike generic 3DGS parameter optimization, the splat prediction head is dynamically fine-tuned per input scene to capture Gaussian parameters specific to the photometric and geometric properties of the given set of images. This adaptive mechanism enhances splat quality and ray-consistent synthesis at the scene level, which would be harder to achieve with static optimization of global 3DGS parameters.\n\n**(2) Cost and Quality Efficiency:**\nThe splat prediction head reduces computational overhead associated with full geometric optimization by focusing on localized Gaussian prediction within the progressive framework. This modular refinement results in noticeable improvements in novel view synthesis (as highlighted by qualitative and quantitative results in Tables 3-4), with reduced memory demands compared to full-scale 3DGS optimization.\n\n**(3) Fidelity Improvements:**\nCompared to vanilla 3DGS, the splat prediction head provides greater visual coherence and depth consistency in regions with high occlusion or sparsity. This benefit is particularly evident in challenging scenes, as demonstrated in comparisons to Spann3r and COLMAP-trained 3DGS.\n\n**3. Dust3r comparisons missing from Table 2**\n\nWe acknowledge the absence of direct Dust3r results in Table 2. Since Dust3r has the OOM issue on a 24GB GPU with hundreds of images as input, we only report the results on the LLFF dataset (each scene contains only 20-50+ images): $\\Delta \\mathbf{R}: 0.79$, $\\Delta \\mathbf{t}: 0.173$. We will include these numbers in the revised manuscript.\nPreliminary evaluations indicate that DeepGfM demonstrates competitive pose estimation accuracy and improved scalability compared to Dust3r for large dense image sets.\n\n**4. Missing Comparisons for Splat Quality**\n\nWe appreciate the reviewer’s suggestion to include comparisons with NoPo-Splat, Splatt3r, and MV-Dust3r+ for assessing splat quality. Since these methods are designed for sparse view rendering, we provide a more thorough comparison to SOTA methods on sparse view settings (2-views and 8-views). We conducted the experiments on the RealEstate10K dataset and the DL3DV dataset.  For these datasets, we follow the protocol in Flare and NopoSplat, which samples 2/8 views for the testing video sequence as input. \nNote that Splatt3R and CopoNeRF only support inputs with 2-views, we did not provide the results on the 8-view setting.\n\n- **Comparison on two-view settings**\n| Method             | Dataset         | PSNR ↑   | SSIM ↑   | LPIPS ↓   |\n|------------------ |----------------|------------|-----------|-----------|\n| DUSt3R          | DL3DV10k       | 13.47 |   0.390 |   0.590  |\n| MASt3R          | DL3DV10k       | 13.88 |   0.405 |   0.513  |\n| Splatt3R          | DL3DV10k       | 15.23 |   0.432 |   0.491 |\n| CoPoNeRF      | DL3DV10k       | 16.06  |  0.472 |  0.474 |\n| Flare                | DL3DV10k       | 23.04 |  0.725  |  0.182  |\n| NoPo-Splat     | DL3DV10k       | 24.19 |  0.745  |  0.173  |\n| MV-DUSt3R+  | DL3DV10k       | 19.95 |  0.698 |   0.195 |\n| DeepGfM        | DL3DV10k       |  20.33 |  0.702 |  0.190 |\n\n| Method             | Dataset         | PSNR ↑   | SSIM ↑   | LPIPS ↓   |\n|------------------ |----------------|----------|----------|-----------|\n| DUSt3R          | RealEstate10K  | 15.382 |   0.447 |   0.432  |\n| MASt3R          | RealEstate10K  | 14.907 |   0.431 |   0.452  |\n| Splatt3R          | RealEstate10K  | 15.113 |   0.492 |   0.442  |\n| CoPoNeRF      | RealEstate10K  | 19.843 |   0.652 |   0.360  |\n| Flare                | RealEstate10K  |  23.765 |  0.801 |   0.191  |\n| NoPo-Splat     | RealEstate10K  |  25.033 |  0.838 |   0.160  |\n| MV-DUSt3R+  | RealEstate10K  |  20.313 |  0.722 |   0.289  |\n| DeepGfM        | RealEstate10K  |  21.715 |  0.753 |   0.229  |\n\n- **Comparison on eight-view settings**\n| Method             | Dataset         | PSNR ↑   | SSIM ↑   | LPIPS ↓   |\n|------------------ |----------------|------------|-----------|-----------|\n| DUSt3R          | DL3DV10k       | 17.37 |   0.601 |   0.390  |\n| MASt3R          | DL3DV10k       | 18.41 |   0.675 |   0.427  |\n| Splatt3R          | DL3DV10k       |  -   |   -  |   -   |\n| CoPoNeRF      | DL3DV10k       |  -   |   -  |   -   |\n| Flare                | DL3DV10k       | 23.33 |  0.746  |  0.237  |\n| NoPo-Splat     | DL3DV10k       | 22.77 |  0.722  |  0.240  |\n| MV-DUSt3R+  | DL3DV10k       | 21.33 |  0.707 |   0.188 |\n| DeepGfM        | DL3DV10k       |  22.90 |  0.728 |  0.176 |\n\n| Method             | Dataset         | PSNR ↑   | SSIM ↑   | LPIPS ↓   |\n|------------------ |----------------|----------|----------|-----------|\n| DUSt3R          | RealEstate10K  | 19.373 |   0.695 |   0.305  |\n| MASt3R          | RealEstate10K  | 19.877 |   0.705 |   0.362  |\n| Splatt3R          | RealEstate10K  |       -    |      -     |      -       |\n| CoPoNeRF      | RealEstate10K  |       -    |      -     |      -       |\n| Flare                | RealEstate10K  |  23.813 |  0.822 |   0.188  |\n| NoPo-Splat     | RealEstate10K  |  23.145 |  0.809 |   0.195  |\n| MV-DUSt3R+  | RealEstate10K  |  22.743 |  0.798 |   0.213  |\n| DeepGfM        | RealEstate10K  |  23.370 |  0.815 |   0.190  |\n\n- **Experiments summarization on sparse-view settings**\n\n**(1) Comparison on Two-View Settings.** Though DeepGfM is not specifically designed for sparse-view settings, it demonstrates competitive performance, surpassing several sparse-view methods.\n\n- **DL3DV10k Dataset:**\nDeepGfM achieves a PSNR of 20.33, significantly outperforming DUSt3R, MASt3R, and Splatt3R.\nSSIM (0.702) and LPIPS (0.190) are also competitive against MV-DUSt3R+, while slightly behind Flare and NoPo-Splat, which are optimized for sparse-view tasks.\n\n- **RealEstate10K Dataset:**\nDeepGfM records PSNR of 21.715, ranking higher than DUSt3R, MASt3R, and Splatt3R, while trailing Flare and NoPo-Splat.\nSSIM (0.753) is higher than MV-DUSt3R+ and comparable to NoPo-Splat.\nLPIPS (0.229) is better than both DUSt3R and MASt3R, showing robustness in perceptual quality.\n\n**(2) Comparison on Eight-View Settings.** DeepGfM shows strong capabilities in handling sparse-view tasks even with more input views.\n\n- **DL3DV10k Dataset:**\nWith a PSNR of 22.90, DeepGfM outperforms MV-DUSt3R+ while approaching Flare and NoPo-Splat.\nSSIM (0.728) and LPIPS (0.176) are the best among all methods, highlighting DeepGfM’s ability to preserve structural and perceptual quality even with sparse-view conditions.\n\n- **RealEstate10K Dataset:**\nDeepGfM achieves a PSNR of 23.370, outperforming MV-DUSt3R+ and NoPo-Splat, and remaining competitive with Flare (23.813).\nSSIM (0.815) and LPIPS (0.190) are also highly competitive, showing improvements over MV-DUSt3R+ and comparable values with Flare.\n\n- **Highlight:**\nIt is important to note that these models are optimized for sparse-view settings, whereas DeepGfM addresses dense-view workflows with hundreds of images. These results underline that, despite being designed for dense-view settings, DeepGfM performs competitively in sparse-view tasks, demonstrating robust generalization and adaptability across both two-view and eight-view configurations. This flexibility is beneficial for tasks requiring high-quality synthesis in diverse input-view scenarios.\n\nWe hope this addresses the reviewer’s concern, and we will include explicit clarifications regarding sparse-view vs. dense-view settings within the revised manuscript for better context."}, {"review_id": "GDoWq3NKpW", "reviewer": "Reviewer_wnHi", "summary": "The paper introduces an approach for jointly learning camera poses along with a 3DGS representation. It jointly optimizes camera poses and scene representation through a three-stage pipeline involving a fine-tuned Spann3R transformer (as a scene predictor), a parameter-free pose predictor, and a renderer. The predicted 3D Gaussians and point maps are refined using a photometric loss in an incremental manner. The approach outperforms existing methods on a variety of datasets.", "questions": "- For baseline comparisons, are the optimization settings changed from the original ones provided with code?\n\n- Are 3DGS parameters here the same as original approach?\n\n- Are the COLMAP poses estimated again or obtained with the datasets like LLFF. MipNerF360? (when available)", "limitations": "The paper needs a more extensive evaluation of it's and baselines limitations in terms of total time for optimization and inference as well as edge cases where the methods fail.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is mostly well written and is easy to follow.\n\n- The paper tackles an important problem of pose-free 3DGS optimization from a random collection of images of a scene. Using similarity graph for iterative pose refinement while utilizing photometric loss is a clever idea.\n\n- The proposed approach outperforms SOTA methods in terms of camera pose estimation and novel view synthesis quality.\n\nWeaknesses:\n- Not all SOTA approaches tackling similar problems considered like COGS [1] and VGGT [2] (which is just referenced once in Table 1).\n\n- The LLFF results look uncharacteristically worse both visually and quantitatively in baseline approaches. The settings and any deviations from original implementation should be clearly mentioned.\n\n- While the memory is discussed with Table 1, the total run-time is not clearly evaluated with other approaches. There is an incomplete explanation in the supplementary. This needs to be more thorough. How much slower is COLMAP? COLMAP + 3DGS is probably less resource intensive and comparable in speed and the metrics performance is also very close. It would also help to show failure cases of other approaches where the proposed approach works.\n\n- The novelty is limited since a lot of the design components like Spann3R are based on previous appraoches.\n\n[1] A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose\n\n[2] VGGT: Visual Geometry Grounded Transformer", "was_revised": true, "final_justification": "The authors addressed most of my concerns during the rebuttal and discussion phase. In addition, they also provided a good plan to incorporate the feedback and improve presentation. Therefore, I'm leaning towards acceptance.", "rebuttal": "Thank you for your thoughtful review and valuable suggestions. We address your comments point-by-point as follows:\n\n**1. State-of-the-art Approaches Not Covered (COGS, Limited Inclusion of VGGT)**\n\nWe appreciate the suggestion to compare with additional state-of-the-art methods. However, we wish to clarify the reasons for not including VGGT and COGS in our evaluation:\n- (1) **VGGT**: The released model is limited to 3D point and camera pose reconstruction, and the finetuned model for the novel view synthesis task was not made available. Additionally, VGGT requires substantial memory (hundreds of input images), which exceeds the capacity of our experiment settings conducted on a 24GB GPU. Thus, a direct comparison was not feasible within our experimental constraints.\n- (2) **COGS**: This method is designed specifically for sparse view settings (at most 12 views, as outlined in the original paper), while our framework scales to hundreds of images without performance degradation. Comparing our approach to COGS would introduce an imbalance in the evaluation, as our method was designed for dense and highly complex datasets.\n\nIt is important to emphasize that we have already compared with multiple well-established state-of-the-art methods, such as NeRF, BARF, NeRFmm, Nope-NeRF, DBARF, COLMAP+3DGS, and ACE0+3DGS. These comparisons were performed under identical experimental settings, which we believe are sufficient to demonstrate the superiority of our framework over existing approaches.\n\n**2. LLFF Results Appear Worse**\n\nThank you for raising this concern. We would like to clarify that, in our experiments, we strictly adhered to the standard settings and pipeline provided in the official repositories for all baseline methods, including CF-3DGS, COLMAP+3DGS, ACE0+3DGS, and others. No deviations or modifications were made to the training parameters or implementations.\n\nAdditionally, the LLFF results presented in Table 3 (main paper) and Table 11 (supplementary) align closely with the averaged and per-scene breakdown results reported in previous works such as NeRF (Table 5), BARF (Table 3), NeRFmm (Table 1), Nope-NeRF (Table 9), and DBARF (Table 1). These consistency checks confirm the fidelity of our evaluation methodology. Furthermore, all experiments follow the default settings and parameters established in the official 3DGS repository.\n\nWe believe this concern stems from a misconception, as our results objectively mirror the behavior demonstrated by baseline methods in prior papers. If the reviewer would like additional clarification or supplementary evidence, we would be happy to provide detailed breakdowns across more metrics or include additional visualizations for the LLFF dataset.\n\n**3. GPU Memory and Runtime Evaluation**\n\nYour feedback highlights an essential gap in our analysis. While GPU memory reduction is a primary focus of our work (Table 1), runtime metrics are equally critical for practical applications. We will evaluate total run-time for both individual substeps (COLMAP pose estimation and subsequent rendering for COLMAP+3DGS) in the revised manuscript.\n\nIn terms of runtime, our progressive training strategy optimizes complex neural architectures over multi-frame inputs, which indeed demands additional computational time compared to COLMAP. Nevertheless, our focus is on scaling high-fidelity reconstructions to larger datasets, for which existing pose-free COLMAP/3DGS-based methods' preprocessing remains insufficient. Our method demonstrates significant advantages over CF-3DGS, which is also a progressive method but requires days for training on datasets with hundreds of images and still suffers from out-of-memory issues. In contrast, our approach efficiently scales to hundreds of input images while remaining well within the memory constraints of a 24GB GPU. This efficiency is a key strength of our method and highlights its suitability for large-scale and high-resolution datasets.\n\nDetailed runtime comparisons with failure cases for other approaches (where our method succeeds) will be included in the revisions.\n\n**4. Perceived Limited Novelty**\n\nWe respectfully disagree with the notion that our method lacks sufficient novelty. Our method introduces several key innovations and contributions:\n- **Novel Training Pipeline:** This is the first approach capable of finetuning an existing feed-forward 3D foundation model for high-fidelity novel view synthesis, a task that prior feed-forward methods could not achieve for dense inputs and fine-level reconstruction details.\n- **General Framework Design:** While Spann3R is selected as a backbone for our experiments, it is not integral to the pipeline itself and can be replaced with other foundation models. This flexibility highlights the robustness and generalizability of our method, which enables a unified path for scaling different backbone approaches for novel view synthesis tasks.\n- **Iterative Refinement Pipeline:** Our iterative refinement mechanism addresses geometric inaccuracies present in foundation models and introduces point-to-ray consistency alongside fine-tuned photometric losses. Such refinement has not been previously explored in pose-free pipelines, further demonstrating the uniqueness of our approach.\n\nWe sincerely appreciate your insights and constructive feedback. We are committed to addressing all concerns comprehensively in the revised manuscript."}, {"review_id": "JKVCdvYXkB", "reviewer": "Reviewer_hmgh", "summary": "The paper introduces Deep Gaussian from Motion (DeepGfM), a framework that learns a high-fidelity 3-D Gaussian Splatting (3DGS) scene representation directly from an unordered collection of unposed images by combining Spann3r (finetuned to predict Gaussians instead of plain pointmaps) and a Progressive Neural Reconstruction pipeline hinted by classical SfM.", "questions": "The motivation behind this work is mixed; Not sure how should readers compare DeepGfM to more recent advances like VGGT[2] and MegaSAM[1] which also aims to use foundation model to resolve unposed dense image sets' reconstruction, but does not need a separate (potential hacky) progressive framework. I get such separate framework might help large-scale reconstruction, but, as mentioned by the authors, cannot extend to larger image sets yet due to the memory issues. \n\nIn this case, could you clarify your problem statement and novelty more? \n\n[1] MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos, CVPR 2025\n\n[2] VGGT: Visual Geometry Grounded Transformer, CVPR 2025", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n- Experimental results on datasets from various domain show that the proposed method is able to achieve competitive results in both novel view synthesis and camera pose estimation in comparison to other pose-free NeRF/GS baselines.\n\nWeakness\n- While combining geometric foundation models with SfM is novel, most components (Spann3R backbone, PnP, Gaussian rasterizer) are reused (PnP is not new to Foundation model pipelines already). \n- More details are needed to clarify how the spann3r is finetuned to predict Gaussians and fit to each scene on the fly.\n- Overall a fragile systems with many separate steps by design with the Progressive Neural Reconstruction pipeline", "was_revised": true, "final_justification": "Please refer to rebuttal discussion reply.", "rebuttal": "Thank you for your thorough review and valuable insights. We address each weakness raised point-by-point below:\n\n**1. Reusing Existing Components (Spann3R, PnP, Gaussian Rasterizer)**\n\nWe acknowledge that some components of our approach, such as Spann3R and PnP, are not entirely novel. However, the novelty of our method lies in the unique combination of these components within a unified pipeline and its ability to finetune a feed-forward 3D foundation model for high-fidelity, pose-free novel view synthesis—a capability that prior methods have not achieved.\n\nSpecifically, our contributions include:\n- **Integration of Spann3R with Scene-specific Fine-tuning:** While Spann3R itself is pre-trained, our method enables it to predict Gaussian-based geometry and photometric texture for each specific scene efficiently. This is achieved by introducing a tailored refinement mechanism geared toward high-quality view synthesis tasks, distinct from Spann3R's typical sparse reconstruction use case.\n- **Point-to-Ray Iterative Refinement:** We designed an iterative refinement loop incorporating ray-consistent optimization and photometric adaptations to correct the inherent inaccuracies and biases often present in geometric foundation models during runtime. This separates our framework from standard pipeline reuse.\n- **Unified Progressive Pipeline:** The combination of these components forms a cohesive pipeline that addresses foundational challenges in view synthesis, such as adapting to scene-specific conditions and maintaining scalability with high-resolution inputs, which remains a challenge for existing foundation pipeline methods.\n\nWhile PnP and Gaussian rasterization are standard operations, their combined use with foundation model fine-tuning and our iterative, scene-adaptive approach is novel in the context of high-fidelity novel view synthesis.\n\n**2. Details on Spann3R Fine-tuning**\n\nWe appreciate the request for additional clarification on how Spann3R is finetuned within our pipeline. Below, we summarize the process:\n- **Geometry Initialization:** Spann3R predictions act as coarse geometric foundations and texture priors for our pipeline, providing initial states for Gaussian point clouds and precomputed rasterizations.\n- **Scene-specific Gaussian Prediction:** Spann3R's outputs are refined dynamically using extensive fine-tuning (with photometric losses and gradient consistency) to predict high-fidelity, scene-specific Gaussian distributions. This step enables scene-adaptive representations without requiring explicit pose annotations.\n- **Photometric and Geometric Consistency:** During each stage of refinement, Spann3R adapts moment-based Gaussian parameters by minimizing multi-view photometric errors and ensuring ray-aligned measurements across input images (addressing alignment inaccuracies in pre-trained foundation models).\n- **On-the-fly Optimization:** The system iteratively updates Spann3R predictions for each incoming scene, producing output consistent with progressive geometric improvements for novel views. Gaussian scene distributions evolve naturally alongside these refinements, resulting in high-quality, view-consistent reconstructions.\n\nWe will expand this section in the revised manuscript and provide illustrative breakdowns of Spann3R’s fine-tuning process across different stages of the pipeline for greater clarity.\n\n**3. Fragility of the Progressive Neural Reconstruction Pipeline**\n\nWe appreciate the concern regarding the pipeline’s perceived fragility due to its multi-step design. We acknowledge that complex pipelines often face challenges related to robustness and ease of deployment. However, we believe the following points address this concern:\n- **Step Modularity:** While our pipeline leverages multiple distinct steps, each module (geometry foundation, Gaussian prediction, photometric refinement) is designed to operate independently, allowing for flexibility during implementation and scaling. For example, Spann3R can easily be replaced with alternative backbones without impacting the pipeline's overall functionality.\n- **Ease of Fine-tuning:** Unlike traditional SfM methods, which rely on pose annotations and explicit camera predictions, our pipeline operates pose-free, reducing dependency on components prone to error. This makes the system less susceptible to breakdowns caused by pose inaccuracies or upstream variability.\n- **Empirical Robustness:** Our results across diverse datasets (LLFF, Tanks-and-Temples, 7Scenes, etc) demonstrate stable synthesis quality despite inherent challenges (e.g., dense inputs, viewpoint diversity, appearance changes). This underscores the resilience of our design even under variable and complex input configurations.\n\nWhile we understand the concern regarding multi-step pipelines, we argue that our modular design ensures flexibility and robustness during training and deployment.\n\n**4.Comparison to VGGT and MegaSAM**\nWe appreciate the insightful comparison drawn between DeepGfM and VGGT/MegaSAM and would like to elaborate on the differences and complementary nature of these approaches.\n\n**(1) Feed-forward Nature of VGGT and MegaSAM**\n\nBoth VGGT and MegaSAM operate in feed-forward pipelines where foundation model predictions are applied directly to dense image reconstruction workflows. This design inherently offers computational speed and efficiency but comes with certain limitations:\n\n- **i. GPU Memory Constraints:** Feed-forward methods often encounter severe GPU memory limitations during dense scene handling, constraining their scalability for large-scale datasets or high-resolution inputs. In our approach, the modular nature of the progressive pipeline allows decomposition of scene data into iterative levels of detail, effectively reducing the memory burden. This design better accommodates larger-scale reconstructions.\n\n- **ii. Pose Accuracy for High-Fidelity Synthesis:** Foundation model-derived camera poses, as utilized by VGGT and MegaSAM, often lack the precision required for high-quality novel view synthesis. While VGGT and MegaSAM aim to improve reconstruction alignment, their reliance on pose estimation can introduce inaccuracies in high-fidelity outputs under challenging dense image set scenarios. DeepGfM circumvents this issue by adopting a pose-free progressive refinement approach, leveraging Gaussian geometry prediction to achieve superior ray-consistent novel views without explicit reliance on pose annotations.\n\n- **iii. Scene-specific Adaptation:** VGGT and MegaSAM pipelines are not explicitly optimized for fine-tuning to adapt to specific scenes. In contrast, DeepGfM integrates scene-specific geometry prediction (via Gaussian rasterization and photometric corrections), producing fine-grained outputs tailored to the input set's particular conditions. This customization empowers a more detailed and view-consistent synthesis compared to feed-forward methods.\n\n**(2) Design Focus of MegaSAM:**\n\nMegaSAM, while valuable, is specifically suited for dynamic scene analysis and reconstruction (e.g., leveraging Droid-SLAM for motion-based scenarios). Our approach, instead, targets dense and static scene reconstruction workflows that demand ultra-high fidelity outputs without pose reliance. These complementary focuses highlight the distinct goals of MegaSAM and DeepGfM, with each addressing different application spaces.\n\n**(3) Clarification of Problem Statement and Novel Contributions**\n\nThe problem DeepGfM addresses is the reconstruction of dense image capture workflows to enable high-fidelity novel view synthesis without reliance on pose annotations or large-scale memory-intensive methods. While feed-forward pipelines like VGGT and MegaSAM prioritize speed, DeepGfM emphasizes depth and quality of reconstruction, introducing the following key novelties:\n\n- **Pose-free Foundation Model Adaptation:** Unlike VGGT/MegaSAM, which rely on pre-computed poses (potentially affected by inaccuracies), our pipeline operates without pose annotations. This is achieved by refining Gaussian geometries dynamically to align photometric appearance with ray-consistent novel view synthesis.\n\n- **Progressive and Modular Framework Design:** The progressive design enables iterative scalability, addressing GPU bottlenecks evident in VGGT-like pipelines. Modularity ensures robustness against scene diversity, allowing refinement of components independent of memory constraints imposed by dense image sets.\n\n- **Scene-specific Gaussian Prediction:** Our method dynamically predicts Gaussian geometries for each input scene, adapting to its unique photometric and geometric characteristics for high-quality synthesis -- a flexibility less evident in feed-forward methods like VGGT/MegaSAM, which process inputs less adaptively.\n\nWe emphasize that these methods complement our work, as MegaSAM excels in dynamic scene reconstruction, and VGGT provides strong speed-oriented solutions for dense workflows. In promoting each other, they strengthen the broader research direction of foundation model-driven image set reconstruction. We will revise our manuscript to clarify this relationship and explicitly highlight how DeepGfM complements these methods and contributes novel alternatives to high-fidelity workflows.\n\nWe are grateful for your constructive feedback and suggestions. Each point raised will be carefully addressed in our revised manuscript with additional details and clarifications to further substantiate the strengths and novelty of our approach."}, {"review_id": "NhKVln2UXD", "reviewer": "Reviewer_rL7u", "summary": "This paper presents a per-scene optimization method for reconstructing 3dgs from unordered and unposed densely captured based on predicted pointmaps from Spann3R. To address the inaccuracy of predicted pointmaps, the authors propose three stages: seed-based initialization, progressive image registration and camera pose re-optimization. And they use the point-to-ray consistency loss to refine camera poses and pointmaps.\n\nThey compare the method to the pose-free NeRF/3dgs and COLMAP/ACE0-based 3dgs on the LLFF, MipNeRF360, Tanks-and-Temples dataset for both camera pose accuracy and novel view synthesis.", "questions": "Refer to [Major Weaknesses] 1) 2) 3) 4). \n\nFor now, I am not yet convinced by this paper due to the mentioned points around evaluation and leaning more towards reject. However, I am open to reconsider after the rebuttal.", "limitations": "No, refer to [Minor Weaknesses] 1) 2)", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "[Strengths]\n\n1) This paper is well-written.\n\n2) Combining 3dgs with *3R family is promising. Table 1 demonstrates its lower GPU memory cost than DUSt3R (with global alignment stage), Fast3R, VGGT (with full attention), which benefits from the progressiveimage registration stage.\n\n3) The point-to-ray consistency loss is novel and more convinced than the reprojection loss proposed in ACE0.\n\n4) The experiments on camera pose estimation and NVS demonstrate that the method achieves comparable performance to the well-established COLMAP-based 3dgs pipeline.\n\n[Major Weaknesses]\n\n1) Why COLMAP/GLOMAP shouldn't be used?\n\n    - Minor improvement:\n    This paper is under dense-view setting, which is well-established in COLMAP-based 3dgs pipeline. It is necessary to provide a reason why COLMAP shouldn't be used since it already provides sufficiently great performance in static scenes. Also, GLOMAP, an advanced version of COLMAP, provides faster and more accurate pipeline than COLMAP. According to the results shown in paper, performance gain seems marginal from COLMAP + 3dgs on novel view synthesis. \n\n     - Missing runtime comparison:\n    The runtime of this method compared with COLMAP/GLOMAP+ 3dgs and CUT3R + 3dgs did not be reported, which may provide the necessity for using this method rather than COLMAP/GLOMAP/CUT3R-based.\n\n2) Lack of evaluations on different *3R backbones.\n\n    The authors claim that the framework is architecture-agnostic, but this point is not validated in the paper. It remains unclear how its generalizability and performance would be affected if it builds upon other *3R backbones like Fast3R, VGGT, CUT3R, or AnySplat (a feed-forward 3dgs model built upon VGGT).\n\n3) Why Spann3R rather than CUT3R?\n\n    This paper misses an important comparison: CUT3R (CVPR 25) is more powerful than Spann3R in reconstruction accuracy, GPU memory and time costs. One of the contributions of this method is the lower GPU memory cost, as shown in Table 1. However, for 250 images, CUT3R only requires 17GB of memory with a runtime of 20fps. CUT3R also circumvents the global alignment operation, similar to Spann3R. Unlike DUT3R or Spann3R relying on PnP for camera pose estimation, CUT3R directly infers the camera pose and point maps in global coordinates, enabling the network to converge faster during training.\n\n4) Lack of evaluation on DL3DV:\n\n    DL3DV is a widely used multi-view dataset for evaluating novel view synthesis. Its diversity and large scale ensure more convincing evaluation results.\n\n[Minor Weaknesses]\n\n1) Missing comparison with *3R-based 3dgs works: Flare, Anysplat on novel view synthesis tasks.\n2) Missing evaluation on sparse views:\nHow about the performance for sparse-view reconstruction? Will it degrade as the number of views decreases?", "was_revised": true, "final_justification": "The authors provide a comprehensive rebuttal and reply to most of my questions, which is very much appreciated.\n\nI also appreciate the responsiveness of the authors and verify major weakness 4) and minor weakness 1) 2). I will update my rating to Borderline accept.\n\nThere seems to be a slight misunderstanding regarding Major Weakness 2). My point was to suggest evaluating DeepGfM with alternative *3R backbones in place of Spann3r to better assess the method’s generalizability.\n\nSome additional suggestions: In the title and early parts of this paper, I initially thought this was a feed-forward 3DGS reconstruction method. This could be somewhat misleading. I encourage that the authors clarify in the final version, particularly in the title, abstract and introduction that the proposed method is based on per-scene optimization, to avoid unnecessary confusion for readers.", "rebuttal": "Thank you for your detailed and thoughtful feedback. We greatly value your recognition of the strengths of our work, including **its writing quality**, **novel point-to-ray consistency loss**, and **promising experimental results**. Below, we address the concerns you raised and provide clarifications based on additional analysis and updated results.\n\n**1. Why COLMAP/GLOMAP shouldn't be used?**\n\nWe would like to clarify that our goal is not to replace COLMAP/GLOMAP pipelines entirely. Instead, our motivation is to leverage and fine-tune existing 3D foundation models for high-fidelity novel view synthesis in real-world applications. While COLMAP and GLOMAP are robust and fast, they are not sufficiently accurate for certain rendering tasks, particularly where high-quality novel view synthesis is crucial.\n\nOur method is designed to bridge this gap by enabling 3D foundation models to achieve:\n\n(1) **Comparable Pose Accuracy:** Using our progressive registration and point-to-ray consistency loss, 3D foundation models can reach pose accuracy levels close to those of COLMAP/GLOMAP.\n(2) **Improved Rendering Quality:** Our framework enhances the rendering fidelity of 3D foundation models, making them suitable for applications where visual quality is essential.\n\nBy combining the robustness and speed of existing 3D foundation models with the accuracy and rendering quality demonstrated in COLMAP/GLOMAP-based pipelines, our approach significantly improves their versatility for real-world applications. This hybrid approach focuses on optimizing foundational systems to work efficiently without discarding their inherent strengths.\n\n**2. Lack of Evaluations on Different 3R Backbones** and **Evaluation on DL3DV dataset**\n\nWe provided the results of different 3R backbones (DUSt3R, MASt3R, Splatt3R, MV-DUSt3R+, Flare, etc.) and our method on the DL3DV dataset to **Reviewer JX9E**. For convenience, we copy and paste the results and analysis here for your reference.\n\nWe provide a more thorough comparison to SOTA methods on sparse view settings (2-views and 8-views). We conducted the experiments on the RealEstate10K dataset and the DL3DV dataset.  For these datasets, we follow the protocol in Flare and NopoSplat, which samples 2/8 views for the testing video sequence as input. \nNote that Splatt3R and CopoNeRF only support inputs with 2-views, we did not provide the results on the 8-view setting.\n\n- **Comparison on two-view settings**\n| Method             | Dataset         | PSNR ↑   | SSIM ↑   | LPIPS ↓   |\n|------------------ |----------------|------------|-----------|-----------|\n| DUSt3R          | DL3DV10k       | 13.47 |   0.390 |   0.590  |\n| MASt3R          | DL3DV10k       | 13.88 |   0.405 |   0.513  |\n| Splatt3R          | DL3DV10k       | 15.23 |   0.432 |   0.491 |\n| CoPoNeRF      | DL3DV10k       | 16.06  |  0.472 |  0.474 |\n| Flare                | DL3DV10k       | 23.04 |  0.725  |  0.182  |\n| NoPo-Splat     | DL3DV10k       | 24.19 |  0.745  |  0.173  |\n| MV-DUSt3R+  | DL3DV10k       | 19.95 |  0.698 |   0.195 |\n| DeepGfM        | DL3DV10k       |  20.33 |  0.702 |  0.190 |\n\n| Method             | Dataset         | PSNR ↑   | SSIM ↑   | LPIPS ↓   |\n|------------------ |----------------|----------|----------|-----------|\n| DUSt3R          | RealEstate10K  | 15.382 |   0.447 |   0.432  |\n| MASt3R          | RealEstate10K  | 14.907 |   0.431 |   0.452  |\n| Splatt3R          | RealEstate10K  | 15.113 |   0.492 |   0.442  |\n| CoPoNeRF      | RealEstate10K  | 19.843 |   0.652 |   0.360  |\n| Flare                | RealEstate10K  |  23.765 |  0.801 |   0.191  |\n| NoPo-Splat     | RealEstate10K  |  25.033 |  0.838 |   0.160  |\n| MV-DUSt3R+  | RealEstate10K  |  20.313 |  0.722 |   0.289  |\n| DeepGfM        | RealEstate10K  |  21.715 |  0.753 |   0.229  |\n\n- **Comparison on eight-view settings**\n| Method             | Dataset         | PSNR ↑   | SSIM ↑   | LPIPS ↓   |\n|------------------ |----------------|------------|-----------|-----------|\n| DUSt3R          | DL3DV10k       | 17.37 |   0.601 |   0.390  |\n| MASt3R          | DL3DV10k       | 18.41 |   0.675 |   0.427  |\n| Splatt3R          | DL3DV10k       |  -   |   -  |   -   |\n| CoPoNeRF      | DL3DV10k       |  -   |   -  |   -   |\n| Flare                | DL3DV10k       | 23.33 |  0.746  |  0.237  |\n| NoPo-Splat     | DL3DV10k       | 22.77 |  0.722  |  0.240  |\n| MV-DUSt3R+  | DL3DV10k       | 21.33 |  0.707 |   0.188 |\n| DeepGfM        | DL3DV10k       |  22.90 |  0.728 |  0.176 |\n\n| Method             | Dataset         | PSNR ↑   | SSIM ↑   | LPIPS ↓   |\n|------------------ |----------------|----------|----------|-----------|\n| DUSt3R          | RealEstate10K  | 19.373 |   0.695 |   0.305  |\n| MASt3R          | RealEstate10K  | 19.877 |   0.705 |   0.362  |\n| Splatt3R          | RealEstate10K  |       -    |      -     |      -       |\n| CoPoNeRF      | RealEstate10K  |       -    |      -     |      -       |\n| Flare                | RealEstate10K  |  23.813 |  0.822 |   0.188  |\n| NoPo-Splat     | RealEstate10K  |  23.145 |  0.809 |   0.195  |\n| MV-DUSt3R+  | RealEstate10K  |  22.743 |  0.798 |   0.213  |\n| DeepGfM        | RealEstate10K  |  23.370 |  0.815 |   0.190  |\n\n- **Comparison with AnySplat**\n\nWe would like to emphasize that **the AnySplat paper was published on arXiv after the paper submission deadline (29 May 2025) for the conference**. As AnySplat was released later than our submission, it is **not reasonable to expect a paper to compare its methods to those published without peer review and after the submission deadline**.\n\nImportantly, scientific evaluations aim to compare methodologies that are accessible at the time of research development and submission. While AnySplat introduces interesting advancements, it was not publicly available during the preparation of our work, and therefore, could not be included in our study or baseline comparisons.\n\nThat said, we acknowledge the relevance of AnySplat for future explorations, and we are excited about the possibility of extending our work to compare and integrate complementary ideas introduced by such methods. The addition of results compared to AnySplat would further strengthen the impact of our proposed framework, but this remains work for future research efforts.\n\n- **Experiments summarization on sparse-view settings**\n\n**(1) Comparison on Two-View Settings**\nThough DeepGfM is not specifically designed for sparse-view settings, it demonstrates competitive performance, surpassing several sparse-view methods.\n\n- **DL3DV10k Dataset:**\nDeepGfM achieves a PSNR of 20.33, significantly outperforming DUSt3R, MASt3R, and Splatt3R.\nSSIM (0.702) and LPIPS (0.190) are also competitive against MV-DUSt3R+, while slightly behind Flare and NoPo-Splat, which are optimized for sparse-view tasks.\n\n- **RealEstate10K Dataset:**\nDeepGfM records PSNR of 21.715, ranking higher than DUSt3R, MASt3R, and Splatt3R, while trailing Flare and NoPo-Splat.\nSSIM (0.753) is higher than MV-DUSt3R+ and comparable to NoPo-Splat.\nLPIPS (0.229) is better than both DUSt3R and MASt3R, showing robustness in perceptual quality.\n\n**(2) Comparison on Eight-View Settings**\n\nDeepGfM shows strong capabilities in handling sparse-view tasks even with more input views.\n\n- **DL3DV10k Dataset:**\nWith a PSNR of 22.90, DeepGfM outperforms MV-DUSt3R+ while approaching Flare and NoPo-Splat.\nSSIM (0.728) and LPIPS (0.176) are the best among all methods, highlighting DeepGfM’s ability to preserve structural and perceptual quality even with sparse-view conditions.\n\n- **RealEstate10K Dataset:**\nDeepGfM achieves a PSNR of 23.370, outperforming MV-DUSt3R+ and NoPo-Splat, and remaining competitive with Flare (23.813).\nSSIM (0.815) and LPIPS (0.190) are also highly competitive, showing improvements over MV-DUSt3R+ and comparable values with Flare.\n\n- **Highlight:**\nIt is important to note that these models are optimized for sparse-view settings, whereas DeepGfM addresses dense-view workflows with hundreds of images. These results underline that, despite being designed for dense-view settings, DeepGfM performs competitively in sparse-view tasks, demonstrating robust generalization and adaptability across both two-view and eight-view configurations. This flexibility is beneficial for tasks requiring high-quality synthesis in diverse input-view scenarios.\n\n**3. Why Spann3R rather than CUT3R?**\n\nWe fully agree that CUT3R is a strong alternative to Spann3R and a more competitive backbone. However, it is important to highlight that the CUT3R code was publicly released only two months before our paper submission deadline. Due to this timeline constraint, we were unable to integrate CUT3R into our codebase and perform further validation experiments within the limited preparation period.\n\nThat said, CUT3R offers several advantages as a backbone:\n\n- **Training Time Reduction:** CUT3R is designed to converge faster, reducing the overall training time compared to Spann3R.\n- **Handling Sparse Inputs:** CUT3R performs better in settings with fewer input images due to its efficient coordinate prediction capabilities.\n\nWe are confident that substituting Spann3R with CUT3R as the backbone would yield even better results, especially in reducing training time and improving performance with a lower number of input images. This strengthens the robustness and generalizability of our framework.\n\nEven though we did not have sufficient time to integrate CUT3R during this study, we consider future explorations with CUT3R to be an exciting direction that could further validate the flexibility and scalability of our method.\n\nWe sincerely thank you for pointing out this potential improvement and for your constructive feedback overall. We firmly believe our approach remains strong irrespective of the backbone used, and CUT3R would likely offer additional benefits that augment the results presented in this study."}]

Author rebuttal:

Meta-review: The reviewers appreciated the presentation of the paper, strong improvements over the (Spann3R) baseline, and the novelty of the overall pipeline and proposed components. Initial concerns about the choice of the base model, benefits over COLMAP+3DGS, and the novelty of the pipeline were all addressed/clarified during the rebuttal. Additional experimental results resolved requests for runtime comparisons and comparisons to sparse-view backends. After the rebuttal and discussion, reviewers unanimously appreciated the novelty of the paper. The AC agrees with this assessment and recommends acceptance.
Decision: Accept (poster)