Title: Deep Gaussian from Motion: Exploring 3D Geometric Foundation Models for Gaussian Splatting
Venue: NeurIPS 2025
Accepted: yes
Keywords: Gaussian Splatting

Novel View Synthesis
Abstract: Neural radiance fields (NeRF) and 3D Gaussian Splatting (3DGS) are popular techniques to reconstruct and render photorealistic images. However, the prerequisite of running Structure-from-Motion (SfM) to get camera poses limits their completeness. Although previous methods can reconstruct a few unposed images, they are not applicable when images are unordered or densely captured. In this work, we propose a method to train 3DGS from unposed images. Our method leverages a pre-trained 3D geometric foundation model as the neural scene representation. Since the accuracy of the predicted pointmaps does not suffice for accurate image registration and high-fidelity image rendering, we propose to mitigate the issue by initializing and fine-tuning the pre-trained model from a seed image. The images are then progressively registered and added to the training buffer, which is used to train the model further. We also propose to refine the camera poses and pointmaps by minimizing a point-to-camera ray consistency loss across multiple views. When evaluated on diverse challenging datasets, our method outperforms state-of-the-art pose-free NeRF/3DGS methods in terms of both camera pose accuracy and novel view synthesis, and even renders higher fidelity images than 3DGS trained with COLMAP poses.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a per-scene optimization method for reconstructing 3dgs from unordered and unposed densely captured based on predicted pointmaps from Spann3R. To address the inaccuracy of predicted pointmaps, the authors propose three stages: seed-based initialization, progressive image registration and camera pose re-optimization. And they use the point-to-ray consistency loss to refine camera poses and pointmaps.\n\nThey compare the method to the pose-free NeRF/3dgs and COLMAP/ACE0-based 3dgs on the LLFF, MipNeRF360, Tanks-and-Temples dataset for both camera pose accuracy and novel view synthesis.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n\n1) This paper is well-written.\n\n2) Combining 3dgs with *3R family is promising. Table 1 demonstrates its lower GPU memory cost than DUSt3R (with global alignment stage), Fast3R, VGGT (with full attention), which benefits from the progressiveimage registration stage.\n\n3) The point-to-ray consistency loss is novel and more convinced than the reprojection loss proposed in ACE0.\n\n4) The experiments on camera pose estimation and NVS demonstrate that the method achieves comparable performance to the well-established COLMAP-based 3dgs pipeline.\n\n[Major Weaknesses]\n\n1) Why COLMAP/GLOMAP shouldn't be used?\n\n    - Minor improvement:\n    This paper is under dense-view setting, which is well-established in COLMAP-based 3dgs pipeline. It is necessary to provide a reason why COLMAP shouldn't be used since it already provides sufficiently great performance in static scenes. Also, GLOMAP, an advanced version of COLMAP, provides faster and more accurate pipeline than COLMAP. According to the results shown in paper, performance gain seems marginal from COLMAP + 3dgs on novel view synthesis. \n\n     - Missing runtime comparison:\n    The runtime of this method compared with COLMAP/GLOMAP+ 3dgs and CUT3R + 3dgs did not be reported, which may provide the necessity for using this method rather than COLMAP/GLOMAP/CUT3R-based.\n\n2) Lack of evaluations on different *3R backbones.\n\n    The authors claim that the framework is architecture-agnostic, but this point is not validated in the paper. It remains unclear how its generalizability and performance would be affected if it builds upon other *3R backbones like Fast3R, VGGT, CUT3R, or AnySplat (a feed-forward 3dgs model built upon VGGT).\n\n3) Why Spann3R rather than CUT3R?\n\n    This paper misses an important comparison: CUT3R (CVPR 25) is more powerful than Spann3R in reconstruction accuracy, GPU memory and time costs. One of the contributions of this method is the lower GPU memory cost, as shown in Table 1. However, for 250 images, CUT3R only requires 17GB of memory with a runtime of 20fps. CUT3R also circumvents the global alignment operation, similar to Spann3R. Unlike DUT3R or Spann3R relying on PnP for camera pose estimation, CUT3R directly infers the camera pose and point maps in global coordinates, enabling the network to converge faster during training.\n\n4) Lack of evaluation on DL3DV:\n\n    DL3DV is a widely used multi-view dataset for evaluating novel view synthesis. Its diversity and large scale ensure more convincing evaluation results.\n\n[Minor Weaknesses]\n\n1) Missing comparison with *3R-based 3dgs works: Flare, Anysplat on novel view synthesis tasks.\n2) Missing evaluation on sparse views:\nHow about the performance for sparse-view reconstruction? Will it degrade as the number of views decreases?\n\nQUESTIONS:\nRefer to [Major Weaknesses] 1) 2) 3) 4). \n\nFor now, I am not yet convinced by this paper due to the mentioned points around evaluation and leaning more towards reject. However, I am open to reconsider after the rebuttal.\n\nLIMITATIONS:\nNo, refer to [Minor Weaknesses] 1) 2)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Deep Gaussian from Motion (DeepGfM), a framework that learns a high-fidelity 3-D Gaussian Splatting (3DGS) scene representation directly from an unordered collection of unposed images by combining Spann3r (finetuned to predict Gaussians instead of plain pointmaps) and a Progressive Neural Reconstruction pipeline hinted by classical SfM.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- Experimental results on datasets from various domain show that the proposed method is able to achieve competitive results in both novel view synthesis and camera pose estimation in comparison to other pose-free NeRF/GS baselines.\n\nWeakness\n- While combining geometric foundation models with SfM is novel, most components (Spann3R backbone, PnP, Gaussian rasterizer) are reused (PnP is not new to Foundation model pipelines already). \n- More details are needed to clarify how the spann3r is finetuned to predict Gaussians and fit to each scene on the fly.\n- Overall a fragile systems with many separate steps by design with the Progressive Neural Reconstruction pipeline\n\nQUESTIONS:\nThe motivation behind this work is mixed; Not sure how should readers compare DeepGfM to more recent advances like VGGT[2] and MegaSAM[1] which also aims to use foundation model to resolve unposed dense image sets' reconstruction, but does not need a separate (potential hacky) progressive framework. I get such separate framework might help large-scale reconstruction, but, as mentioned by the authors, cannot extend to larger image sets yet due to the memory issues. \n\nIn this case, could you clarify your problem statement and novelty more? \n\n[1] MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos, CVPR 2025\n\n[2] VGGT: Visual Geometry Grounded Transformer, CVPR 2025\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an approach for jointly learning camera poses along with a 3DGS representation. It jointly optimizes camera poses and scene representation through a three-stage pipeline involving a fine-tuned Spann3R transformer (as a scene predictor), a parameter-free pose predictor, and a renderer. The predicted 3D Gaussians and point maps are refined using a photometric loss in an incremental manner. The approach outperforms existing methods on a variety of datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is mostly well written and is easy to follow.\n\n- The paper tackles an important problem of pose-free 3DGS optimization from a random collection of images of a scene. Using similarity graph for iterative pose refinement while utilizing photometric loss is a clever idea.\n\n- The proposed approach outperforms SOTA methods in terms of camera pose estimation and novel view synthesis quality.\n\nWeaknesses:\n- Not all SOTA approaches tackling similar problems considered like COGS [1] and VGGT [2] (which is just referenced once in Table 1).\n\n- The LLFF results look uncharacteristically worse both visually and quantitatively in baseline approaches. The settings and any deviations from original implementation should be clearly mentioned.\n\n- While the memory is discussed with Table 1, the total run-time is not clearly evaluated with other approaches. There is an incomplete explanation in the supplementary. This needs to be more thorough. How much slower is COLMAP? COLMAP + 3DGS is probably less resource intensive and comparable in speed and the metrics performance is also very close. It would also help to show failure cases of other approaches where the proposed approach works.\n\n- The novelty is limited since a lot of the design components like Spann3R are based on previous appraoches.\n\n[1] A Construct-Optimize Approach to Sparse View Synthesis without Camera Pose\n\n[2] VGGT: Visual Geometry Grounded Transformer\n\nQUESTIONS:\n- For baseline comparisons, are the optimization settings changed from the original ones provided with code?\n\n- Are 3DGS parameters here the same as original approach?\n\n- Are the COLMAP poses estimated again or obtained with the datasets like LLFF. MipNerF360? (when available)\n\nLIMITATIONS:\nThe paper needs a more extensive evaluation of it's and baselines limitations in terms of total time for optimization and inference as well as edge cases where the methods fail.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to fine-tune a 3D foundation model (Spann3r) to directly predict 3DGS parameters via incremental registration of the scene. By mimicking the incremental mapping step in Sfm, this method tries to register images starting from a seed image and progressively increasing its training buffer. At each step, it adds an image to the buffer, predicts 3DGS parameters, refines camera poses and then applies a photometric loss on the predicted GS parameters. The authors show improved camera pose estimation and NVS results on multiple datasets compared to prior art.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths  \n- Quality improvement over Spann3r is impressive. Spann3r seems to fail on many sequences whereas DeepGfm seems to work much better thanks to the incremental camera optimization strategy. \n- The novel view synthesis seems to be at par with COLMAP trained 3DGS while the camera pose estimation seems strong as well. \n\nWeaknesses  \n- Choice of pretrained model; It seems like Spann3r is a worse pretrained model than Dust3r/Mast3r/Mast3r-Sfm discarding the OOM issue for large sequence lengths. Its possible that a lot of the finetuning goes in just getting camera poses to look reasonable from the poor performance of Spann3r. Moreover, Spann3r is meant for online scene reconstruction whereas this paper uses it as part of a two-stage finetuning strategy. Spann3r's memory retrieval architecture is suited for sequential data so using unordered collections might hurt more than expected. The motivation to use Spann3r background is weak.\n- Splat prediction head training: Since this model is finetuned per scene, what is the benefit of the splat prediction head? Isn't it equivalent to optimizing for 3DGS parameters directly as in vanilla 3DGS? What cost/speed/quality benefit does the head provide?\n- This method can be treated like a more efficient global alignment step compared to Dust3r's. Consequently, it would be good to compare results with Mast3r-Sfm on camera pose estimation.\n- Dust3r comparisons missing from Table 2\n- Missing NoPo-Splat, Splatt3r and MV-Dust3r+ comparison for splat quality.\n\nQUESTIONS:\n- Since the method is agnostic to the Dust3r variant used, it would be good to see how Dust3r faired in comparison to DeepGfm for pose estimation and nvs. \n- How would the model scale with respect to VRAM and processing time with increasing in #images?\n- L226-228: Are the photometric losses also applied on the low resolution ground-truth?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}