PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting

Authors: Sunghwan Hong, Jaewoo Jung, Heeseong Shin, Jisang Han, Jiaolong Yang, Chong Luo, Seungryong Kim
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Generalized Pose-Free Novel View Synthesis, 3D Reconstruction

Abstract:
We consider the problem of novel view synthesis from unposed images in a single feed-forward. Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS, where we further extend it to offer a practical solution that relaxes common assumptions such as dense image views, accurate camera poses, and substantial image overlaps. We achieve this through identifying and addressing unique challenges arising from the use of pixel-aligned 3DGS: misaligned 3D Gaussians across different views induce noisy or sparse gradients that destabilize training and hinder convergence, especially when above assumptions are not met. To mitigate this, we employ pre-trained monocular depth estimation and visual correspondence models to achieve coarse alignments of 3D Gaussians. We then introduce lightweight, learnable modules to refine depth and pose estimates from the coarse alignments, improving the quality of 3D reconstruction and novel view synthesis. Furthermore, the refined estimates are leveraged to estimate geometry confidence scores, which assess the reliability of 3D Gaussian centers and condition the prediction of Gaussian parameters accordingly. Extensive evaluations on large-scale real-world datasets demonstrate that PF3plat sets a new state-of-the-art across all benchmarks, supported by comprehensive ablation studies validating our design choices. We will make the code and weights publicly available.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729661117469

thread_type:
official_review

<thread_object>
creator: reviewer_hhva

<content>
summary:
Given G.T. camera intrinsics, the authors first leverage the existing work to obtain the coarse camera poses and depth. Then to refine the estimation, the authors design a module to learn the depth offset estimation with the help of an existing depth estimation network. Furthermore, the camera pose refinement is conducted in another module. The idea of feedforward pose estimation is interesting, but there is still a gap between the performance of the proposed method and some per-scene optimization methods. Since I did not see the authors report any inference time result and I believe some static scene pose-free per-scene optimization methods (CF-3DGS, ..) are very fast and accurate, I expect the authors to provide more comparisons. There are still some questions and limitations raised below. I will consider improving the grade upon the feedback from the authors.

strengths:
1. The authors proposed a new pose-free feed-forward method in camera pose estimation.
2. The authors conducted enough ablation studies to present the contribution of each component of their method.

weaknesses:
1. The biggest limitation of this work is the requirement of G.T. camera intrinsic.
2. The performance on RealEstate-10K seems to be SOTA, however, the performance on ACID is not. Does it mean such a method does not generalize well to the outdoor scenes?
3. So I expect more comparisons on DL3DV or more public datasets (like DAVIS[1], iPhone[2]) to prove the effectiveness of the proposed method. 


[1] Pont-Tuset, Jordi, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. "The 2017 davis challenge on video object segmentation." arXiv preprint arXiv:1704.00675 (2017).

[2] Gao, Hang, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. "Monocular dynamic view synthesis: A reality check." Advances in Neural Information Processing Systems 35 (2022): 33768-33780.

questions:
1. Can the authors provide more insights on how you obtain the coarse camera parameters? I guess the authors directly implemented the existing work to obtain those things.
2. the authors trained different checkpoints on different datasets in the implementation details. Does it mean the 'feed-forward' claimed by the authors is actually dataset-specific? If so, I think it is a big limitation of the proposed method lacking the generalizability to different scenes.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730574596021

thread_type:
official_review

<thread_object>
creator: reviewer_ridd

<content>
summary:
This paper focuses on the pose-free feed-forward novel view synthesis task. It leverages a pre-trained monocular depth estimation model and a visual correspondence model to generate coarse pose and depth estimates, enhancing the stability of the 3DGS training process. Subsequently, a lightweight refinement model is used to further improve the depth and pose estimations. Extensive experiments have been conducted to demonstrate the effectiveness of the proposed method.

strengths:
S1 - This paper addresses a meaningful task with significant potential for real-world applications.

S2 - This paper leverages two robust pre-trained models to generate initial pose and shape estimates, which significantly enhance the model's performance.

S3 - The paper is well-written, and the experiments are logically sound.

weaknesses:
W1 - The performance of this method appears to be highly dependent on the coarse pose and depth results provided by the pre-trained model.

W2 - The paper lacks qualitative results for the pose estimation, which would provide a clearer assessment of the model's performance in this area.

questions:
Q1 - How would the results differ if alternative coarse prediction networks, such as Dust3r[1], Mast3r[2], or others, were used?

Q2 - Qualitative results for the pose estimation task.

[1]Wang S, Leroy V, Cabon Y, et al. Dust3r: Geometric 3d vision made easy[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 20697-20709.
[2]Leroy V, Cabon Y, Revaud J. Grounding Image Matching in 3D with MASt3R[J]. arXiv preprint arXiv:2406.09756, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659967253

thread_type:
official_review

<thread_object>
creator: reviewer_943i

<content>
summary:
This paper introduces a method to tackle the challenging problem of novel view synthesis (NVS) from unposed images in a feed-forward manner. They identify the issue in previous pixel-aligned 3DGS methods, where the predicted 3D Gaussians from different views have the problem of misalignment, leading to noisy gradient flows and poor NVS results. They propose a method where they don’t need to rely on the poses obtained from off-the-shelf tools. Instead, they leverage pre-trained monocular depth estimation and visual correspondence models to obtain coarse alignment, with further refinement of the depth map and poses. Results show that among the pose free methods, they can perform decently better results for NVS tasks. However, for pose estimation, it is still worse than general methods like Mast3R.

strengths:
1. The authors tackle the problem of 3D Gaussian splatting from unposed sparse images, which is an interesting and important topic
2. The authors apply the recent state-of-the-art depth estimation and pose estimation methods for coarse pose alignment, and further introduces pose and depth refinements, to some extend improving the final performance.
3. The paper is overall well-written and easy to follow in most parts.

weaknesses:
**The method section is overall clear, but missing some details and discussions**

*Unclear descriptions in camera pose refinement in Sec 3.2.3*  
1. It is quite unclear to me how exactly it is done. From your writing, it makes me feel that you first will get a newly computed camera poses $\hat{P_{ij}}$ similarly as done in the coarse step, but with the refined depth. This $\hat{P}_{ij}$ is already the refined poses, right? However, you further have another refinement step as shown in Eq. (2). What are the rationale before?
2. what is the T_pose network? What is the E_pos in eq. (2)?

*Cost volume*  
In section 3.2.4, for the “cost volume construction and aggregation”, is that any different to the MVSplat paper? Can you justify the differences?

*2D-3D Consistency loss*  
Line 291-292, you said that you improve the robustness of the model in regions with low texture or significant viewpoint changes. However, the correspondences from the feature matching methods like LightGlue do not provide many correspondences in those low-texture regions. I guess you cannot claim the robustness there? 

*Unclear implementation details*  
What is the frame distance from 45 to 75? Do you mean you sample one frame every 45/75 frames in the video sequence? You might want to make this point clearer. And why for DL3DV, you only sample every 5 or 10 frames, way smaller than ACID or RE10LK?
Also, you mention that you train for 40000 iterations on a single A6000 GPU, is it the same for all three datasets? If true, I think it might not make too much sense? As you mentioned in line 351-354, RE10K has ~21K videos from training, while you only use 2K scenes for the training of DL3DV?



**The experimental section is convincing in general, but lacks some important experiments / baselines and explanation.**

1. For pose estimation comparison, the sota method right now is RoMa [1], I think it is fair to ask to compare to it.
2. I wonder why your method is still lacking behind Mast3R on the pose estimation for both RE10K and ACID, even if Mast3R is is not even trained at all on those dataset, while your method was trained on those dataset separately.
3. Why for novel view synthesis in Table 1, you don’t show the comparison to the recent pose-required methods pixelSplat?
4. Based on the experiments you show, your pose estimation is worse than Mast3R in almost all cases, and the NVS results are also worse than MVSplat in all scenarios (even on DL3DV in Table 3, where MVSplat was not trained on). One reasonable baseline to me is, get camera poses from Mast3R, and then run MVSplat directly. I wonder how your method compares to such a simple baseline?
5. In your ablation study table 4, what is the point of adding V, I-I, I-II, I-V, if they are just all N.A.? That is really weird to me. You can just describe them in texts.


**Writing**  
Sec 3.2.1: You use two paragraphs to motivate by mentioning the limitations of the previous methods. The real content for the coarse alignment is really just the third paragraph between line 186-191. The motivations part is actually unrelated to the coarse alignment but more on why your method is needed, so you should just put them in the introduction instead.

[1] Edstedt et al.: RoMa: Robust dense feature matching, CVPR 2024

questions:
As I already discussed in the weakness section, it would be very imporatnt if you can justify those points in the experiments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692103135

thread_type:
official_review

<thread_object>
creator: reviewer_tpsu

<content>
summary:
This paper introduces PF3plat, a novel framework designed for novel view synthesis from unposed images in a single feed-forward pass. PF3plat leverages pre-trained monocular depth estimation and visual correspondence models to achieve an initial coarse alignment of 3D Gaussians. Subsequently, PF3plat incorporates refinement modules and geometry-aware scoring functions to further refine the depth and pose estimates derived from the coarse alignment to enhance the quality of view synthesis.

strengths:
1. The task of novel view synthesis from unposed images in a single feed-forward pass is highly practical.
2. The paper demonstrates state-of-the-art results on Re10k and ACID, showcasing the effectiveness of the proposed method. 
3.  The refinement modules designed in the paper have significantly improved the effectiveness.

weaknesses:
1. PF3plat leverages a robust solver for pose estimation between each pair of cameras; thus, increasing the number of viewpoints significantly extends the feed-forward pass time.
2. PF3plat relies on the coarse alignment of 3D Gaussians, and a small overlap may affect the quality of the correspondence model.

questions:
1. Why does using the pixel-wise depth offset estimation model promote consistency across views? (line 211)
2. How about the performance of PF3plat in dynamic scenes?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732201656323

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We are grateful that the reviewers recognize the strengths of our work, including its well-written presentation (943i, RiDd), focus on an important topic, and proposal of a highly practical solution (TPSu, 943i, RiDd). Additionally, the reviewers acknowledged that our proposed modules are thoroughly ablated and proved to be highly effective, achieving state-of-the-art performance in all view synthesis benchmarks (TPSu, 943i, RiDd, HHva).

In our revised PDF, we have included additional figures detailing our architecture and qualitative comparisons for pose estimation. Furthermore, in this rebuttal, we have addressed key points raised by the reviewers, including:

- Inference time (TPSu),
- Coarse alignment (TPSu, RiDd),
- Extension to dynamic scenes (TPSu, HHva),
- Additional qualitative results (RiDd), and
- Cross-dataset generalization and further evaluation on DL3DV (HHva).

For reviewer 943i:
- we have clarified the missing details in the method section
- provided results of RoMa and PixelSplat as additional references, and 
- justified why pose-required and matching-based methods should not be directly compared to pose-free methods like ours 

We sincerely thank the reviewers for their constructive feedback and thorough evaluation. We hope that the revisions and clarifications provided in our rebuttal address all raised concerns, ensuring that our contributions are well-positioned within the scope of pose-free novel view synthesis.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733150791973

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,
 
We greatly appreciate your time and effort in reviewing our manuscript. As the author-reviewer discussion period is concluding, we would like to provide a final response that may assist in further considerations.
 
PF3plat, in which its main contribution lies in addressing the task of pose-free **novel view synthesis** in a single feed-forward pass using **3D Gaussian Splatting**, first adopts a novel approach that employs **coarse alignment** to tackle the unique challenges arising from the use of pixel-aligned 3D Gaussian Splatting. Specifically, misaligned 3D Gaussians across different views can induce noisy or sparse gradients, which destabilize training and hinder convergence, especially when common assumptions such as ground-truth camera poses, dense views, and substantial image overlaps are not met. We then
introduce **lightweight refinement modules and geometry-aware scoring functions**, which
not only enhance the reconstruction and view synthesis quality, but also prevent catastrophic forgetting issues typically associated with direct fine-tuning of coarse-alignment module, e.g., monocular depth. With this model, we evaluated on real-world large-scale datasets including **RealEstate10K, ACID, and DL3DV**, achieving state-of-the-art performance on all of them.
 
Regarding the remaining concerns that some reviewers have yet to respond to, we emphasized that while pose estimation is of prime importance ( yet our method achieves state-of-the-art or comparable results without relying on GT depth or correspondences, which previous works leverage for directly addressing pose estimation), our primary objective is **novel view synthesis**  (Reviewer HHva),  as clearly stated in the title, introduction, and contributions (L88).  It is also important to note that on top of Dust3R and Mast3R, methods exclusively for 3D reconstruction,  additional methods, such as NeRF or 3D Gaussian Splatting, is required to train a radiance field for novel view synthesis. This approach: 1) **is not a feature of VGGSfM or Dust3R**; 2) **requires a long optimization time for each scene**; and 3) **depends on GT camera poses, tracking, intrinsics, and depth for training**. Finally, our method can infer much faster than InstantSplat (Reviewer 943i), with InstantSplat taking 53 seconds and ours 0.390 seconds, highlighting the practicality and advantage of our single feed-forward approach, not to mention the **generalizability advantages that feed-forward approaches offer over optimization-based methods**. In terms of performance, we also show that incorporating test-time optimization enables our method to surpass InstantSplat as well as with significantly less time consumption.
 
We believe that our submission has sufficiently demonstrated its effectiveness through extensive experiments on real-world large-scale datasets for novel view synthesis, outperforming existing generalized pose-free NVS methods. We sincerely hope that the reviewers consider these for the subsequent discussion period.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper receives borderline final ratings of 6,5,6,5. The AC looked through the reviews, the rebuttal and the discussions between the reviewers and authors, and finally decided to reject the paper due to the remaining concerns raised by the two reviewers who gave negative ratings are valid. Specifically, there is a limited improvement over other simple baselines like instantsplat. The requirement of the depth refinement step is questionable, and the core algorithm is MVSplat with the replacement of estimated camera poses to replace the ground truth. Furthermore, the proposed method requires known camera intrinsics. In contrast, Vggsfm and Dust3R do not have this requirement. The results are also benchmarked on one public dataset, which is deemed insufficient. The two reviewers decided to give borderline rejects even after rebuttal and discussions. On the other hand, the two reviewers who are positive did not give very strong reasons to justify the acceptance of this work. The comments are this is a meaningful task, paper is well-written, highly practical, etc.