PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting

## Abstract

Abstract We consider the problem of novel view synthesis from unposed images in a single feed-forward. Our framework capitalizes on fast speed, scalability, and high-quality 3D reconstruction and view synthesis capabilities of 3DGS, where we further extend it to offer a practical solution that relaxes common assumptions such as dense image views, accurate camera poses, and substantial image overlaps. We achieve this through identifying and addressing unique challenges arising from the use of pixel-aligned 3DGS: misaligned 3D Gaussians across different views induce noisy or sparse gradients that destabilize training and hinder convergence, especially when above assumptions are not met. To mitigate this, we employ pre-trained monocular depth estimation and visual correspondence models to achieve coarse alignments of 3D Gaussians. We then introduce lightweight, learnable modules to refine depth and pose estimates from the coarse alignments, improving the quality of 3D reconstruction and novel view synthesis. Furthermore, the refined estimates are leveraged to estimate geometry confidence scores, which assess the reliability of 3D Gaussian centers and condition the prediction of Gaussian parameters accordingly. Extensive evaluations on large-scale real-world datasets demonstrate that PF3plat sets a new state-of-the-art across all benchmarks, supported by comprehensive ablation studies validating our design choices. Our code will be made available at: https://cvlab-kaist.github.io/PF3plat/

## 1 Introduction

In recent years, 3D reconstruction and novel view synthesis have garnered significant attention, particularly with the emergence of Neural Radiance Fields (NeRF) (Mildenhall et al., ) and 3D Gaussian Splatting (3DGS) (Kerbl et al., ). These advancements have enabled the high-quality 3D reconstruction and novel view synthesis. However, many existing methods rely on stringent assumptions, such as dense image views (Yu et al., ; Barron et al., ; ), accurate camera poses (Kim et al., ; Kwak et al., ; Zhu et al., ; Charatan et al., ; Chen et al., ), and substantial image overlaps (Yu et al., ; Johari et al., ), which limit their practical applicability. In real-world scenarios, casually captured images contain sparse and distant viewpoints, and lack precise camera poses, making it impractical to assume densely captured views with accurate camera poses. Ideally, a practical novel view synthesis solution should operate quickly and effectively with as few as two images, even under significant viewpoint changes.

To address some of these limitations, recent efforts (Yu et al., ; Johari et al., ; Chen et al., ; Yang et al., ) have introduced generalized view synthesis frameworks capable of performing single feed-forward novel view synthesis from sparse images with minimal overlaps (Du et al., ; Xu et al., ). Among these methods, particularly those utilizing 3DGS (Charatan et al., ; Chen et al., ), have demonstrated remarkable rendering speed and efficiency, alongside impressive reconstruction and view synthesis quality, highlighting the potential of 3D Gaussian-based representations. However, they still depend on accurate camera poses, which are challenging to acquire in sparse settings, thereby restricting their practical use.

More recently, pose-free generalized view synthesis frameworks (Chen & Lee, ; Fan et al., ; Jiang et al., ; Smith et al., ; Hong et al., ) have been introduced to decouple 3D reconstruction and novel view synthesis from camera poses. Given a set of unposed images, these frameworks aim to jointly learn radiance fields and 3D geometry without relying on additional data, such as ground-truth camera pose. The learned radiance fields and geometry can then be inferred through trained neural networks, enabling single feed-forward inference. While these pioneering efforts enhance practicality, their performance remains unsatisfactory and their slow rendering speeds (Chen & Lee, ; Fan et al., ; Jiang et al., ; Smith et al., ) remain unresolved. In an attempt to boost performance and robustness under extreme scenarios when wide-baseline images are given, Hong et al. () developed a unified framework that jointly estimates camera poses, correspondences, and radiance fields using additional data, such as ground truth poses, for supervision. However, this approach still inherits the inherent limitations of NeRF, including intensive memory consumption and slow rendering speeds, making it impractical for real-world applications.

In this work, we propose PF3plat (Pose-Free Feed-Forward 3D Gaussian Splatting), a novel framework for fast and photorealistic novel view synthesis from unposed images in a single feed-forward pass. Our approach leverages the efficiency and high-quality reconstruction capabilities of pixel-aligned 3DGS (Charatan et al., ; Szymanowicz et al., ), while relaxing common assumptions such as dense image views, accurate camera poses, scene-specific optimization and substantial image overlaps. However, a primary challenge in using pixel-aligned 3DGS is its dependency on precise depth and camera pose estimates for accurate localization of 3D Gaussian centers. Inaccuracies in these estimates can cause misalignments, leading to noisy or sparse gradients that destabilize training and hinder convergence, especially when the above assumptions are relaxed or scene-specific optimizations to rectify errors cannot be applied during multi-scene training.

To mitigate these issues, we find that leveraging pre-trained monocular depth estimation (Piccinelli et al., ) and visual correspondence (Lindenberger et al., ) models to achieve a coarse alignment of 3D Gaussians is highly effective, thereby promoting a stable learning process. Subsequently, we introduce learnable modules designed to refine the depth and pose estimates from the coarse alignment to enhance the quality of 3D reconstruction and view synthesis. These modules are geometry-aware and lightweight, since we leverage features from the depth network and avoid direct fine-tuning. These refined depth and pose estimates are then used to implement geometry-aware confidence scores to assess the reliability of 3D Gaussian centers, conditioning the prediction of Gaussian parameters such as opacity, covariance, and color.

Our extensive evaluations on large-scale real-world indoor and outdoor datasets (Liu et al., ; Zhou et al., ; Ling et al., ) demonstrate that PF3plat sets a new state-of-the-art across all benchmarks. Comprehensive ablation studies validate our design choices, confirming that our framework provides a fast and high-performance solution for pose-free generalizable novel view synthesis. We summarize our contributions below:

- •
We address the challenging task of pose-free, feed-forward novel view synthesis using 3DGS, relaxing many common assumptions of existing methods to improve practicality. Our approach works well with wide-baseline images and requires no additional data beyond RGB image collections for both training and inference, highlighting its practicality.
- •
To address the unique challenges arising from the misalignment of 3D Gaussians that destabilizes the learning process, we devise an approach to provide coarse alignment. We then introduce lightweight refinement modules and geometry-aware scoring functions, which not only enhance the reconstruction and view synthesis quality, but also prevent catastrophic forgetting issues typically associated with direct fine-tuning.
- •
Our framework presents an effective approach to enable fast and high-performance 3D reconstruction and novel view synthesis from sparse and unposed images. We have shown that our method sets a new state-of-the-art across all benchmarks.

## 2 Related Work

##### Generalizable Scene Reconstruction and View Synthesis from Unposed Imagery.

Several innovative efforts have addressed the joint learning of camera pose and radiance fields within NeRF-based frameworks. Starting with BARF (Lin et al., ), subsequent research (Jeong et al., ; Wang et al., ; Bian et al., ; Truong et al., ) has expanded upon this foundation. Notably, the use of 3D Gaussians as dynamic scene representations has led to significant advancements: Fu et al. () progressively enlarges 3D Gaussians by learning transformations between consecutive frames, SplaTAM(Keetha et al., ) utilizes RGB-D sequences and silhouette masks to jointly update Gaussian parameters and camera poses, and InstantSplat (Fan et al., ) optimizes 3D Gaussians rapidly for scene reconstruction and view synthesis. Among these, methods like DBARF (Chen & Lee, ), FlowCAM (Smith et al., ), CoPoNeRF (Hong et al., ), and GGRt (Li et al., ) aim to determine camera pose and radiance fields in a single feed-forward pass. Concurrently, Splatt3R (Smart et al., ) builds on pre-trained 3D reconstruction models (Leroy et al., ) to relax some assumptions; however, it still relies on additional data such as ground truth depth and pose. In contrast, our method reconstructs 3D scenes and synthesizes novel views from unposed images with minimal overlaps in a single feed-forward pass, eliminating the need for scene-specific optimization and extensive pose data.

##### Monocular Depth Estimation and Correspondence Estimation.

Monocular depth estimation and visual correspondence estimation are fundamental computer vision tasks with extensive research spanning decades. Recent advancements (Yin et al., ; Piccinelli et al., ; Ke et al., ; Yang et al., ) in monocular depth estimation have matured these models, significantly enhancing various 3D vision applications. Similarly, visual correspondence estimation has advanced significantly since the emergence of deep neural networks. Conventionally, the task is evaluated primarily in camera pose estimation, where classical methods follow a pipeline that includes keypoint detection and descriptor extraction (Lowe, ; Bay et al., ; Tola et al., ; Salti et al., ; Rusu et al., ), tentative matching and outlier filtering (Fischler & Bolles, ), and camera pose estimation using solvers (Gower, ; Nistér, ; Hartley, ; Hesch & Roumeliotis, ). More recently, deep learning-based approaches have optimized each stage, outperforming traditional methods in tasks such as 2D descriptor extraction (Yi et al., ; DeTone et al., ), 3D descriptor extraction (Yew & Lee, ; Choy et al., ), sparse and dense matching (Hong & Kim, ; Cho et al., ; ; Hong et al., ; ; ; Sun et al., ; Edstedt et al., ), and outlier filtering (Barath et al., ; Wei et al., ). In this work, we propose a lightweight plug-and-play depth and pose refinement modules to enhance the quality of view synthesis.

## 3 Method

Figure: Figure 1: Overall architecture and loss of the proposed method. (a) Given a set of unposed images, our method aligns the 3D Gaussians using a coarse-to-fine strategy. (b) In addition to photometric loss, we enforce 3D Gaussian consistency by ensuring they are placed on the same object surface through 2D-3D and 3D-3D consistency losses.
Refer to caption: x1.png

### 3.1 Problem Formulation

Our objective is to reconstruct a 3D scene from a set of $N$ unposed images $\{I_{i}\}_{i=1}^{N}$ with $I_{i}\in\mathbb{R}^{H\times W\times 3}$ and synthesize photo-realistic images $\hat{I}_{t}$ from novel viewpoints in a single feed-forward pass. To achieve this, we output the depth maps $\mathcal{D}_{i}\in\mathbb{R}^{H\times W}$ for each image $I_{i}$, along with their corresponding camera poses $\mathcal{P}_{i}\in\mathbb{R}^{3\times 4}$, consisting of a rotation matrix $R_{i}\in\mathbb{R}^{3\times 3}$ and a translation vector $t_{i}\in\mathbb{R}^{3\times 1}$. Additionally, we compute a set of pixel-aligned 3D Gaussians denoted as $\mathcal{G}=\{\mu_{i},\sigma_{i},\Sigma_{i},c_{i}\}_{i=1}^{N}$. Here, $\mu_{i}(p)\in\mathbb{R}^{3}$ indicates the 3D Gaussian center derived from the depth $\mathcal{D}_{i}(p)$, camera pose $\mathcal{P}_{i}$, and camera intrinsic $K_{i}$, where $p\in\mathbb{R}^{H\times W}$ represents each pixel. The opacity is represented by $\sigma_{i}(p)\in[0,1)$, $\Sigma_{i}(p)\in\mathbb{R}^{3\times 3}$ is the covariance matrix, and the color is encoded using spherical harmonics $c_{i}(p)\in\mathbb{R}^{3(L+1)}$, where $L$ is the order of the spherical harmonics. Note that, for both training and inference, we do not use any ground truth camera poses. Instead, we let the network predict the camera poses for all input images while also learning the varying scale factors across different scenes.

### 3.2 PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting

#### 3.2.1 Coarse Alignment of 3D Gaussians

Inspired by recent advancements (Charatan et al., ; Chen et al., ) that highlight the advantages of pixel-aligned 3D Gaussians, such as speed, efficiency, and high-quality reconstruction and view synthesis, we extend these benefits to more challenging scenarios, specifically in the context of pose-free feed-forward view synthesis. To this end, we adopt pixel-aligned 3D Gaussians as our scene representation. However, this representation also poses certain challenges. Unlike previous methods for generalized novel view synthesis that utilize implicit representations (Chen & Lee, ; Smith et al., ; Hong et al., ) and benefit from the interpolation capabilities of neural networks, our approach is challenged by the explicit nature of this representation. Specifically, our method directly localizes 3D Gaussian centers using depth and camera pose estimates (Charatan et al., ), making it highly sensitive to inaccuracies in these estimates, which cannot be easily compensated.

Such misalignments can cause severe performance degradation and disrupt the learning process by producing sparse and noisy gradients. This issue is particularly exacerbated when wide-baseline images are given as input or the absence of ground-truth pose prevents alignments of 3D Gaussians. Without effectively addressing these challenges, we find the problem becomes nearly intractable. These issues can be mitigated by employing iterative scene-specific optimization steps or by assuming ground-truth camera poses to guide 3D Gaussians toward object surfaces. However, these solutions are incompatible with our goal of achieving a single feed-forward process from unposed images. Therefore, overcoming these limitations requires a novel strategy that can handle depth and pose ambiguities while maintaining efficiency in a feed-forward manner.

Figure: Figure 2: Proposed refinement and confidence estimation modules. In our Fine Alignment module, we refine depth and pose to improve 3D reconstruction and view synthesis quality, alongside estimating confidence to assess the reliability of predicted 3D Gaussian centers.
Refer to caption: x2.png

To mitigate the challenges mentioned above, we find it necessary to provide coarse alignment of 3D Gaussians, where quantitative results can be found in Tab. 4. To this end, we employ off-the-shelf models (Piccinelli et al., ; Lindenberger et al., ) to estimate initial depths $\mathcal{D}_{i}$ and camera poses $\mathcal{P}_{i}$ for our images $I_{i}$. Specifically, given depth maps $\mathcal{D}_{i}$ and sets of correspondences $\mathcal{M}_{ij}$ and their confidence values $\mathcal{C}_{ij}$ acquired from each pairwise combinations of images, e.g., $(I_{i},I_{j})$, where $i,j$ refer to image indices, we use a robust solver (Fischler & Bolles, ; Li et al., ) to estimate the relative poses $\mathcal{P}_{ij}$ between image pair.
Integrating these components, we provide the necessary coarse alignment to promote stabilizing the learning process and serve as a strong foundation for further enhancements.

#### 3.2.2 Multi-View Consistent Depth Estimation

While pre-trained monocular depth models (Wang et al., ; Leroy et al., ; Piccinelli et al., ; Yin et al., ) can offer powerful 3D geometry priors, inherent limitations of these models, namely, inconsistent scales among predictions, still remain unaddressed. This requires further adjustments to ensure multiview consistency across predictions. To overcome this challenge, we aim to refine the predicted depths and camera poses obtained from coarse alignment in a fully learnable and differentiable manner.

Our refinement module includes a pixel-wise depth offset estimation that uses the feature maps $F_{i}$ from the depth network (Piccinelli et al., ) as the sole input and processes them through a series of self-attention operations, making it lightweight and geometry-aware (Xu et al., ). The process is defined as:

$$ $\begin{split}&\Delta\delta_{i}=\phi_{\mathrm{mlp}}(\mathcal{T}_{\mathrm{depth} }(F_{i})),\\ &\hat{\mathcal{D}}_{i}=\mathcal{D}_{i}+\Delta\delta_{i},\\ \end{split}$ (1) $$

where $\phi_{\mathrm{mlp}}(\cdot)$ is a linear projection, $\mathcal{T}_{\mathrm{depth}}$ is a deep Transformer architecture and $\Delta\delta$ is the pixel-wise depth offset. This extension promotes consistency across views and enhances performance without relying on explicit cross-attention. Instead, it leverages supervision signals derived from pixel-aligned 3D Gaussians that connect the information across views, and leverage them for the novel view synthesis task (Zhou et al., ) and our loss functions, which are detailed in Section 3.3. Additionally, we avoid fine-tuning the entire depth network, thereby reducing computational costs and mitigating the risk of catastrophic forgetting.

#### 3.2.3 Camera Pose Refinement

In this module, we further refine camera poses to enhance reconstruction and view synthesis quality. Initially, we replace the estimated relative poses $\mathcal{P}_{ij}$ with newly computed camera poses $\hat{\mathcal{P}}_{ij}$ derived from following the similar process in coarse alignment and using the previously obtained refined depths $\hat{\mathcal{D}}_{i}$. We then introduce a learnable camera pose refinement module that estimates rotation and translation offsets. To streamline this process, we first utilize a fully differentiable transformation synchronization operation that takes $\hat{\mathcal{P}}_{ij}$ and $\mathcal{C}_{ij}$ as inputs. Using power iterations (El Banani et al., ), this operation efficiently recovers the absolute poses $\hat{\mathcal{P}}_{i}$ prior to the refinement module.

Next, we convert the absolute poses $\hat{\mathcal{P}}_{i}$ into Plücker coordinates (Sitzmann et al., ), defined as $r=(\mathrm{d},\mathrm{o}\times\mathrm{d})\in\mathbb{R}^{6}$, where $\mathrm{d}$ represents the camera direction and $\mathrm{o}$ denotes the camera origin. These coordinates, along with the feature maps $F_{i}\in\mathbb{R}^{h\times w\times d}$ and a pose token $\mathcal{P}_{\mathrm{CLS}}\in\mathbb{R}^{d}$, are input into a series of self- and cross-attention layers. In our approach, we designate $\hat{\mathcal{P}}_{1}$ as the reference world space and update only the other pose estimates. The resulting pose token is then transformed into 6D rotations (Zhou et al., ) and translations, which are added to the initial camera poses to estimate the rotation and translation offsets. Formally, these are defined as:

$$ $\begin{split}&\hat{\mathcal{P}}_{\mathrm{CLS}}=\mathcal{T}_{\mathrm{pose}}([{F _{i}},\mathcal{P}_{\mathrm{CLS}},r]+E_{\mathrm{pos}}),\\ &\Delta R_{i},\Delta t_{i}=\phi_{\mathrm{rot}}(\hat{\mathcal{P}}_{\mathrm{CLS} }),\phi_{\mathrm{trans}}(\hat{\mathcal{P}}_{\mathrm{CLS}}),\\ &\hat{R}_{i},\hat{t}_{i}\Leftarrow\hat{R}_{i}+\Delta R_{i},\hat{t}+\Delta t, \end{split}$ (2) $$

where $\Delta R,\Delta t$ are the pose offsets, and $E_{\mathrm{pos}})$ is positional embedding.

#### 3.2.4 3D Gaussian Paramter Predictions

##### Cost Volume Construction and Aggregation.

Using the refined pose and the monocular depth estimates, $\hat{\mathcal{P}}_{i}$ and $\hat{\mathcal{D}}_{i}$, we assess the quality of the predictions to obtain confidence scores, to assist predicting 3D Gaussian parameters. To achieve this, we construct both a conventional multi-view stereo cost volume and a guidance cost volume derived from ${\mathcal{D}}_{i}$.

Specifically, given the current pose estimates $\hat{\mathcal{P}}_{i}$, we build a multi-view stereo cost volume $\mathcal{C}_{i}^{\mathrm{multi}}\in\mathbb{R}^{h\times w\times K}$ following the plane-sweeping approach (Yao et al., ; Chen et al., ). For each of the $K$ depth candidates within specified near and far ranges along the epipolar lines, we compute matching scores using cosine similarity (Cho et al., ; ; Hong et al., ). Subsequently, to guide the depth localization along the epipolar lines, we construct a guidance cost volume $\mathcal{C}_{i}^{\mathrm{guide}}$ (Li et al., ), where each spatial location is represented by a one-hot vector indicating the depth candidate closest to the monocular depth estimate. The constructed $\mathcal{C}_{i}^{\mathrm{multi}}$ and $\mathcal{C}_{i}^{\mathrm{guide}}$ undergo cross-attention to update the multi-view cost volume $\hat{\mathcal{C}}_{i}^{\mathrm{multi}}$.

##### Geometry-aware Confidence Estimation.

From the updated multi-view cost volume $\hat{\mathcal{C}}_{i}^{\mathrm{multi}}$, we apply a softmax function (Xu et al., ; Hong et al., ) along the $K$ dimension to obtain a matching distribution. We then extract the maximum value from this distribution to derive a confidence score, $S_{\mathrm{geo}}$, which assesses the quality of the estimated camera pose and depth. Formally, this is defined as:

$$ $S_{i}^{\mathrm{geo}}=\max_{k\in{\{1,2,...,K\}}}\mathrm{softmax}(\hat{\mathcal{ C}}_{i}^{\mathrm{multi}})(k).$ (3) $$

These confidence scores assess the reliability of the predicted 3D Gaussian centers, where high confidence indicates accurate localization and low confidence suggests potential inaccuracies due to noise or misalignment. To condition the prediction of Gaussian parameters, opacity, covariance, and color, we incorporate $S_{\mathrm{geo}}$ as additional input.

##### 3D Gaussian Parameters.

Finally, using the inputs $[I_{i},\hat{\mathcal{D}}_{i},F_{i},S_{i}^{\mathrm{geo}}]$, we compute the opacity $\sigma_{i}$ through small convolutional layers, derive the covariances from the estimated rotations and scales, and obtain the color from the estimated spherical harmonic (SH) coefficients. A key idea of our approach is that $S^{\mathrm{geo}}$ enables supervision signals to flow from the Gaussian parameters back to the depth and pose estimates. This feedback loop enhances the accuracy of both depth and pose estimations, resulting in more consistent and reliable 3D reconstructions.

### 3.3 Loss Function

##### Reconstruction Loss.

By making both depth and camera pose refinement modules learnable and differentiable, our network leverages supervision signals from the pixel-aligned 3D Gaussians and the novel view synthesis task (Zhou et al., ) to refine these estimates during training. Specifically, we combine photometric loss, defined as the L2 loss between the rendered and target images, as well as SSIM (Wang et al., ) loss $\mathcal{L}_{\text{SSIM}}$ and LPIPS (Zhang et al., ) loss $\mathcal{L}_{\text{LPIPS}}$ to form our reconstruction loss $\mathcal{L}_{\text{img}}$.

##### 2D-3D Consistency Loss.

We identify that provided good coarse alignments, RGB loss is sufficient, but with larger baselines, the training process starts to destabilize.
Moreover, one remaining issue with learning solely from the photometric loss is that the gradients are mainly derived from pixel intensity differences, which suffer in textureless regions. To remedy these, we enforce that corresponding points in the set of images $\{I\}_{i=1}^{N}$ lie on the same object surface, drawing from principles of multi-view geometry Hartley & Zisserman ().

Formally, using the estimated depths $\hat{D}$, the camera poses $\hat{\mathcal{P}}$, and the correspondence sets $\mathcal{M}$, we can define a geometric consistency loss that penalizes deviations from the multi-view geometric constraints. For each correspondence $(p,q)\in\mathcal{M}_{ij}$ between images $I_{i}$ and $I_{j}$, we compute the 3D point from the pixel $p$ and its estimated depth $\hat{D}_{i}(p)$ using the camera intrinsics. We then transform this to the coordinate frame of $I_{j}$ using the relative pose $\hat{\mathcal{P}}_{ij}$ and project it back onto the image plane to obtain the predicted correspondence $\tilde{p}$. This is defined as $\mathcal{L}_{\mathrm{2D-3D}}=\sum_{(p,q)\in\mathcal{M}}\varphi(\tilde{p},q),$ where $\varphi(\cdot)$ denotes huber loss. By integrating this loss into the learning pipeline, we improve the robustness of the model in regions with low texture or significant viewpoint changes, ensuring that the estimated pose and depth are consistent with the underlying 3D structure of the scene.

##### 3D-3D Consistency Loss.

While the multi-view consistent surface loss directly connects each pair of corresponding Gaussians and their centers, guiding the model towards the object’s surface, we find that relying solely on this loss can lead to suboptimal convergence, especially in regions with sparse correspondences. To further stabilize and enhance the learning process, we introduce an additional regularization term that minimizes the discrepancies among the centers of the corresponding Gaussians.

Intuitively, this differs from $\mathcal{L}_{\mathrm{2D-3D}}$ in that, unlike the previous function, which considers the alignment of Gaussian centers from only one side when dealing with pairwise correspondences, the regularization term symmetrically enforces consistency from both sides. Specifically, while the multi-view consistent surface loss projects the Gaussian center from one view to another using the estimated depth and camera pose, e.g., from source to target, the regularization term jointly minimizes the distances between all corresponding Gaussian centers across multiple views. By considering both directions in pairwise correspondences, this approach promotes a more coherent and robust estimation of the object’s surface, reducing the influence of outliers and improving convergence during training. This additional regularization can be formally defined as: $\mathcal{L}_{\mathrm{3D-3D}}=\sum_{(p,q)\in\mathcal{M}}||\mu_{i}(p)-\mu_{j}(q)
||_{2}.$

##### Final Objective Function.

Combining the three loss functions, we define our final objective function: $\mathcal{L}_{\mathrm{img}}+\mathcal{L}_{\mathrm{2D-3D}}+\lambda_{\mathrm{3D-3D
}}\mathcal{L}_{\mathrm{3D-3D}},$ where we set $\lambda_{\mathrm{3D-3D}}=0.05$

## 4 Experiments

### 4.1 Implementation Details

In this work, we assume intrinsic parameters are given, as they are generally available from modern devices (Arnold et al., ). We compute attentions using Flash Attention (Dao et al., ), and for the Gaussian rasterizer, we follow the method described in (Charatan et al., ). Our model is trained on a single NVIDIA A6000 GPU for 40,000 iterations using the Adam optimizer (Kingma, ), with a learning rate set to $2\times 10^{-4}$ and a batch size of 5, which takes approximately two days. For training on the RealEstate10K and ACID datasets, we gradually increase the number of frames between $I_{1}$ and $I_{2}$ as training progresses, initially setting the frame distance to 45 and gradually increasing it to 75. For the DL3DV dataset, we start with a frame distance of 5 and increase it to 10. The target view is randomly sampled within this range. We train under the assumption of $N=2$ and render $\hat{I}_{t}$. Our code is implemented using PyTorch (Paszke et al., ). Additional implementation details can be found in the supplementary material. The code and pretrained weights will be made publicly available.

**Table 1: Novel View Synthesis Performance on RealEstate-10K and ACID. Gray entries indicate methods that use ground truth camera poses during evaluation and are not directly comparable.**
| RealEstate-10K |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pose-Free | Method | Avg | Small | Medium | Large |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR | LPIPS | SSIM | MSE | PSNR | LPIPS | SSIM | MSE | PSNR | LPIPS | SSIM | MSE | PSNR | LPIPS | SSIM | MSE |  |  |
| ✗ | PixelNeRF | 14.438 | 0.577 | 0.467 | 0.047 | 13.126 | 0.639 | 0.466 | 0.058 | 13.999 | 0.582 | 0.462 | 0.042 | 15.448 | 0.479 | 0.470 | 0.031 |
| Du et al. | 21.833 | 0.294 | 0.736 | 0.011 | 18.733 | 0.378 | 0.661 | 0.018 | 22.552 | 0.263 | 0.764 | 0.008 | 26.199 | 0.182 | 0.836 | 0.004 |  |
| MVSplat | 25.054 | 0.157 | 0.827 | 0.008 | 21.029 | 0.226 | 0.747 | 0.013 | 26.369 | 0.116 | 0.874 | 0.004 | 30.516 | 0.074 | 0.926 | 0.002 |  |
| ✓ | DBARF | 14.789 | 0.490 | 0.570 | 0.033 | 13.453 | 0.563 | 0.522 | 0.045 | 15.201 | 0.487 | 0.560 | 0.030 | 16.615 | 0.380 | 0.648 | 0.022 |
| FlowCAM | 18.242 | 0.597 | 0.455 | 0.023 | 15.435 | 0.528 | 0.570 | 0.034 | 18.481 | 0.592 | 0.441 | 0.18 | 22.418 | 0.707 | 0.287 | 0.009 |  |
| CoPoNeRF | 19.536 | 0.398 | 0.638 | 0.016 | 17.153 | 0.459 | 0.577 | 0.025 | 19.965 | 0.343 | 0.645 | 0.013 | 22.542 | 0.250 | 0.724 | 0.008 |  |
| Ours | 22.347 | 0.205 | 0.763 | 0.010 | 18.904 | 0.276 | 0.683 | 0.017 | 23.366 | 0.173 | 0.793 | 0.007 | 27.064 | 0.115 | 0.870 | 0.003 |  |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

### 4.2 Experimental Setting

##### Datasets.

We train and evaluate our method on three large-scale datasets: RealEstate10K (Zhou et al., ), a collection of both indoor and outdoor scenes; ACID (Liu et al., ), a dataset focusing on outdoor coastal scenes; and DL3DV (Ling et al., ), which includes diverse real-world indoor and outdoor environments. For RealEstate10K, due to some unavailable videos on YouTube, we use a subset of the full dataset, comprising a training set of 21,618 scenes and a test set of 7,200 scenes. For ACID, we train on 10,935 scenes and evaluate on 1,893 scenes. Lastly, for DL3DV, which features longer video sequences across 10,510 different scenes, we use 2,000 scenes for training and a standard benchmark set of 140 scenes for testing (Ling et al., ).

##### Baselines.

Following Hong et al. (), we evaluate our method on two tasks: novel-view synthesis and camera pose estimation. For novel view synthesis, we compare our approach against established generalized NeRF and 3DGS variants, including PixelNeRF (Yu et al., ), Du et al. (), PixelSplat (Charatan et al., ), and MVSPlat (Chen et al., ). It is important to note that these methods assume ground-truth (GT) camera poses during inference, so we include them only for reference. Our primary comparisons focus on existing pose-free generalized novel view synthesis methods, such as DBARF (Chen & Lee, ), FlowCAM (Smith et al., ), and CoPoNeRF (Hong et al., ).

For camera pose estimation, we first evaluate against correspondence-based pose estimation methods (e.g., COLMAP), including SP+SG (DeTone et al., ; Sarlin et al., ), PDC-Net+(Truong et al., ), DUSt3R(Wang et al., ), and MASt3R (Leroy et al., ). Additionally, we compare with direct pose regression methods such as 8ViT (Rockwell et al., ), RelPose (Zhang et al., ), and MicKey (Barroso-Laguna et al., ). Our main comparisons, however, are with existing pose-free approaches, including DBARF (Chen & Lee, ), FlowCAM (Smith et al., ), and CoPoNeRF (Hong et al., ).

**Table 2: Pose Estimation Performance on RealEstate-10K and ACID. Gray entries indicate methods not trained on the same dataset due to the absence of ground truth data (e.g., depth and correspondence), making them incomparable.**
| RealEstate-10K |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Task | Method | Avg | Small | Medium | Large |  |  |  |  |  |  |  |  |  |  |  |  |
| Rotation | Translation | Rotation | Translation | Rotation | Translation | Rotation | Translation |  |  |  |  |  |  |  |  |  |  |
| Avg(°$\downarrow$) | Med(°$\downarrow$) | Avg(°$\downarrow$) | Med(°$\downarrow$) | Avg(°$\downarrow$) | Med(°$\downarrow$) | Avg(°$\downarrow$) | Med(°$\downarrow$) | Avg(°$\downarrow$) | Med(°$\downarrow$) | Avg(°$\downarrow$) | Med(°$\downarrow$) | Avg(°$\downarrow$) | Med(°$\downarrow$) | Avg(°$\downarrow$) | Med(°$\downarrow$) |  |  |
| SfM | SP+SG | 5.605 | 1.301 | 14.89 | 5.058 | 9.793 | 2.270 | 12.55 | 4.638 | 1.789 | 0.969 | 9.295 | 3.279 | 1.416 | 0.847 | 21.42 | 7.190 |
| PDC-Net+ | 2.189 | 0.751 | 10.10 | 3.243 | 3.460 | 1.128 | 6.913 | 2.752 | 1.038 | 0.607 | 6.667 | 2.262 | 0.981 | 0.533 | 16.57 | 5.447 |  |
| DUSt3R | 2.527 | 0.814 | 17.45 | 4.131 | 3.856 | 1.157 | 12.23 | 2.899 | 1.650 | 0.733 | 14.00 | 3.650 | 0.957 | 0.476 | 27.30 | 10.27 |  |
| MASt3R | 2.555 | 0.751 | 9.775 | 2.830 | 4.240 | 1.283 | 8.050 | 2.515 | 1.037 | 0.573 | 6.904 | 2.418 | 0.791 | 0.418 | 13.963 | 3.925 |  |
| Pose Estimation | 8ViT | 12.59 | 6.881 | 90.12 | 88.65 | 12.60 | 6.860 | 91.46 | 91.50 | 12.17 | 6.552 | 82.48 | 82.92 | 12.77 | 7.214 | 91.85 | 88.92 |
| RelPose | 8.285 | 3.845 | - | - | 12.10 | 4.803 | - | - | 4.942 | 3.476 | - | - | 4.217 | 2.447 | - | - |  |
| Pose-Free<br>View Synthesis | DBARF | 11.14 | 5.385 | 93.30 | 102.5 | 17.52 | 13.22 | 126.3 | 140.4 | 7.254 | 4.379 | 79.40 | 75.41 | 3.455 | 1.937 | 50.09 | 33.96 |
| FlowCAM | 7.426 | 4.051 | 50.66 | 46.28 | 11.88 | 6.778 | 87.12 | 58.25 | 4.154 | 3.346 | 42.29 | 41.59 | 2.349 | 1.524 | 34.47 | 27.79 |  |
| CoPoNeRF | 3.610 | 1.759 | 12.77 | 7.534 | 5.471 | 2.551 | 11.86 | 5.344 | 2.183 | 1.485 | 10.19 | 5.749 | 1.529 | 0.991 | 15.544 | 7.907 |  |
| Ours | 1.965 | 0.751 | 10.113 | 4.785 | 2.561 | 1.031 | 7.349 | 4.122 | 1.536 | 1.536 | 9.332 | 4.525 | 1.278 | 0.550 | 14.753 | 6.127 |  |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

##### Evaluation Protocol

For the evaluation on RealEstate-10K and ACID, we follow the protocol outlined by Hong et al. (), where evaluation is conducted using unposed triplet images $(I_{1},I_{2},I_{t})$. The test set is divided into three groups, small, middle, and large, based on the extent of overlap between $I_{1}$ and $I_{2}$. This allows the model’s performance to be assessed under varying levels of difficulty, reflecting different real-world scenarios. For the relatively new DL3DV dataset, we introduce a new evaluation protocol. For each scene, we select two context images, $I_{1}$ and $I_{2}$, by skipping frames at intervals of 5 and 10, creating two groups per scene, each representing small and large overlap cases. We then randomly select three target images from the sequence between the context images, similar to the above protocol.

For evaluation metrics, we use standard image quality measures, PSNR, SSIM, LPIPS, and MSE, for novel view synthesis. For camera pose estimation, we compute the geodesic rotation error and angular difference in translation, as commonly done in classical methods (Nistér, ; Melekhov et al., ). These errors are measured by comparing the ground truth relative pose $P^{\mathrm{GT}}_{12}$ and our estimated pose $\hat{\mathcal{P}}_{12}$. Our statistical analysis reports both the average and median errors, with the median providing robustness against outliers.

**Table 3: Novel View Synthesis and Pose Estimation Performance on DL3DV. We include MVSPlat for reference only.**
| DL3DV |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pose-Free | Method | Small | Large |  |  |  |  |  |  |  |  |  |  |  |  |
| PSNR | LPIPS | SSIM | Rotation | Translation | PSNR | LPIPS | SSIM | Rotation | Translation |  |  |  |  |  |  |
| Avg. | Med. | Avg. | Med. | Avg. | Med. | Avg. | Med. |  |  |  |  |  |  |  |  |
| ✗ | MVSPlat | 20.849 | 0.230 | 0.680 | - | - | - | - | 24.211 | 0.147 | 0.796 | - | - | - | - |
| ✓ | CoPoNeRF | 15.509 | 0.563 | 0.396 | 13.121 | 6.721 | 44.645 | 30.269 | 17.586 | 0.467 | 0.469 | 5.609 | 2.905 | 17.974 | 12.445 |
| Ours | 19.355 | 0.280 | 0.611 | 4.698 | 2.475 | 10.692 | 6.869 | 22.105 | 0.211 | 0.706 | 3.353 | 1.604 | 9.407 | 6.334 |  |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

### 4.3 Experimental Results

##### RealEstate-10K & ACID.

Tab.1 summarizes the performance for the novel view synthesis task, while Tab.2 reports the results for pose estimation. From the results in Tab. 1, our method significantly outperforms previous pose-free generalizable methods (Chen & Lee, ; Smith et al., ; Hong et al., ), setting a new state-of-the-art across these benchmarks. Furthermore, compared to the previous state-of-the-art, CoPoNeRF, our method achieves a 2.8 dB improvement in PSNR, demonstrating superior reconstruction quality and robustness. Additionally, our approach demonstrates superior pose estimation performance on RealEstate-10K; however, we observe that Hong et al. () achieves lower pose errors on the ACID dataset. This discrepancy may be attributed to the larger scale of scenes, such as coastal landscapes and sky views, which complicates our refinement process and poses challenges for our depth network in estimating the metric depth of the scene. Nonetheless, this limitation is mitigated by our method’s superior novel view image quality and the fact that Hong et al. () utilizes ground-truth poses for supervision, providing robust guidance for large-scale environments. Moreover, the ACID dataset includes numerous dynamic scenes, which are beyond the scope of our current focus. Consequently, our method may be less effective in estimating poses for dynamic scenes compared to other approaches.

##### DL3DV.

While RealEstate-10K and ACID encompass a variety of indoor and outdoor scenes, RealEstate-10K predominantly includes indoor environments, whereas ACID features numerous dynamic scenes. To comprehensively evaluate our method across a broader spectrum of real-world scenarios, we further assess it on the recently released DL3DV dataset (Ling et al., ). The results are summarized in Table 3. From these results, we observe that our method outperforms CoPoNeRF (Hong et al., ) by over 4 dB in large-overlap scenarios and by 3.8 dB in small-overlap scenarios, highlighting the superior accuracy and robustness of our approach in handling diverse and complex environments. This highlights the effectiveness of our method in managing varied scene and object types, reinforcing its applicability for practical novel view synthesis tasks.

### 4.4 Ablation Study

In this ablation study, we aim to investigate the effectiveness of each component of our method. We first define a baseline model, which combines our depth and pose estimates from coarse alignments with MVSplat (Chen et al., ) for 3D Gaussian parameter prediction. We also explore both full fine-tuning and partial fine-tuning strategies for the depth network. Additionally, we report the results of ablation studies on our loss functions. The results are summarized in Tab. 4.

**Table 4: Component ablations on RealEstate10K.**
|  | Components | Avg |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR | SSIM | LPIPS | Rotation | Translation |  |  |  |
|  | Avg. | Med. | Avg. | Med. |  |  |  |  |
| (0) | Baseline | 20.140 | 0.694 | 0.281 | 2.776 | 0.630 | 10.043 | 3.264 |
| (I) | PFSplat | 22.347 | 0.763 | 0.205 | 1.965 | 0.751 | 10.113 | 4.785 |
| (II) | - Depth Refinement | 21.963 | 0.759 | 0.208 | 2.240 | 1.134 | 9.701 | 3.044 |
| (III) | - Pose Refinement | 21.519 | 0.737 | 0.222 | 2.349 | 1.175 | 12.123 | 6.347 |
| (IV) | - Geometry Confidence | 21.239 | 0.737 | 0.223 | 2.303 | 0.922 | 9.179 | 3.533 |
| (V) | - Corres. Network | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
| (VI) | - Mono. Depth Network | 16.132 | 0.511 | 0.405 | 6.990 | 5.329 | 21.328 | 14.432 |
| (I-I) | Full F.T. Depth Network | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
| (I-II) | Scale/Shift Tuning Depth Network | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
| (I-III) | - Tri. Consis. Loss | 18.832 | 0.6247 | 0.418 | 5.787 | 2.105 | 17.117 | 9.422 |
| (I-IV) | - Regularization Loss | 20.981 | 0.722 | 0.243 | 4.698 | 1.908 | 11.172 | 8.384 |
| (I-V) | (I-IV) - Tri. Consis. loss | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
|  |  |  |  |  |  |  |  |  |

From the results, we find that our method improves by a large margin when comparing (0) with (I). This improvement is further supported by the comparisons from (I) to (IV) and from (I-III) to (I-IV), which show performance degradation as each component is removed. We also demonstrate that without pre-trained weights for the depth and correspondence networks, the training either fails or achieves significantly lower performance. Similar observations are made in (I-I), (I-II), and (I-V), where we identify that directly tuning the depth network or training only with photometric losses leads to failure in the training process. The former issue may arise from overfitting, a common problem when directly manipulating foundation models. With only the photometric loss, we observe that after certain iterations, as the baseline becomes wider, the training loss quickly diverges.

Figure: Figure 3: Qualitative results on RealEstate10K and ACID dataset. Given two context views (a) and (b), we compare novel view rendering results with other methods. We provide more qualitative results on RealEstate10K, ACID and DL3DV in supplementary material.
Refer to caption: x3.png

**Table 5: More Analysis and Results.**
| Method | PSNR | SSIM | LPIPS | Rot. (°$\downarrow$) | Trans. (°$\downarrow$) | Time (s) |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Avg. | Med. | Avg. | Med. |  |  |  |  |  |
| InstantSplat | 23.079 | 0.777 | 0.182 | 2.693 | 0.882 | 11.866 | 3.094 | 53 |
| Splatt3R | 15.636 | 0.502 | 0.360 | 1.312 | 0.521 | 8.715 | 1.891 | 20 |
| Ours | 22.347 | 0.763 | 0.205 | 1.965 | 0.751 | 10.113 | 4.785 | 0.390 |
| Ours + TTO | 23.132 | 0.779 | 0.202 | 1.965 | 0.814 | 9.996 | 4.701 | 13 |

### 4.5 Analysis and More Results

##### Comparison to DUSt3R extensions.

In this study, we compare our results with those of InstantSplat (Fan et al., ) and Splatt3R (Smart et al., ). Although these are preprints, we include this comparison for completeness, as their tasks and methods are closely related to ours. The results are summarized in Tab. 5(a). We find that InstantSplat achieves superior performance in novel view synthesis compared to our method; however, when we adopt a similar test-time optimization (TTO) strategy, our approach requires significantly less optimization time to achieve comparable or better results. Additionally, our method infers substantially faster without TTO, demonstrating its high practicality for real-time applications. Splatt3R, a concurrent work built on a similar concept to ours, requires ground-truth depth and pose for training, which are not available in the RealEstate10K dataset. Due to scale discrepancies, their performance is significantly lower, and without a mechanism to handle the scale differences, an issue not addressed in the original paper, the results are not directly comparable to ours.

##### Inference speed comparisons.

We conduct a comprehensive inference speed comparison between our method and competing approaches using varying numbers of input images, specifically $N\in{2,6,12}$. For each scenario, we evaluate the time required to render 1, 3, and 5 views. The results, summarized in Tab. 5(b), show that our approach is generally faster than existing methods. However, for $N=12$, our inference speed is slower than that of DBARF, as our method involves estimating camera poses via a robust solver for every pairwise combination. Despite this overhead, our approach gains a significant advantage as the number of rendered views increases, due to the efficient rendering capabilities of 3DGS once the scene has been fully reconstructed. Finally, in order to provide a detailed breakdown, we measured three values for our method: overall inference time, UniDepth processing time, and decoder time, isolating the contributions of each component to the total runtime. Given 2, 6 and 12 views and to render a single target view, it takes 0.251, 0.832 and 1.535 seconds for UniDepth inference, while it takes approximately consistent 0.00247 seconds for rendering.

##### Extending to N-views.

In practical scenarios, more than two views ($N>2$) are commonly used. Therefore, we demonstrate that our method can process multiple views and render $I_{t}$. We input $N$ views into our network to obtain $\hat{\mathcal{P}}_{i}$, $\hat{\mathcal{D}}_{i}$, and $(\mu_{i},\sigma_{i},\Sigma_{i},c_{i})$. Following a similar approach to Chen & Lee (), we select the top-$k$ nearby views using $\hat{\mathcal{P}}_{i}$ and render $\hat{I}_{t}$ to compare with the ground truth target view image. For this evaluation, we compare our method with those of Chen & Lee () and Smith et al. (), since the method by Hong et al. () can only take two input views. We also report the Absolute Trajectory Error (ATE). The results are summarized in Tab. 5(c). From these results, we find that our method achieves significantly better performance than the others, highlighting our capability to extend to multiple $N$ views.

##### Cross-Dataset Evaluation.

To demonstrate the generalization capability of our method, we conduct a cross-dataset evaluation and compare against Hong et al. (). Specifically, we evaluate performance on RealEstate10K and DL3DV, using each dataset for training in a cross-dataset setting. The results, summarized in Tab. 5(d), show that our method achieves a PSNR of over 20 dB for both datasets, significantly outperforming Hong et al. (). This indicates that, even under out-of-distribution conditions, our method produces high-quality renderings, highlighting its robustness and effectiveness in zero-shot capability.

## 5 Conclusion

In this paper, we have introduced learning-based framework that tackles pose-free novel view synthesis with 3DGS, enabling efficient, fast and photorealistic view synthesis from unposed images. The proposed framework, PFSplat, is built on the base model comprising of foundation models to overcome inherent limitations of 3DGS. While the devised base model already surpasses the existing methods, we have also devised modules to address the limitations of the base model, enhancing the overall performance. This method is capable of training and inference solely from unposed images, even in scenarios where only a handful of images with minimal overlaps are given. We have shown that our approach surpasses all existing methods on real-world large-scale datasets, establishing new state-of-the-art performance.

## Appendix A Appendix

### A.1 Training Details

We train MVSplat (Chen et al., ) and CoPoNeRF (Hong et al., ) using our data loaders, similarly increasing the distance between context views during
training, as explained in Sec 4.1. Specifically,
we train MVSplat for 200,000 iterations
using a batch size of 8 on a single A6000 GPU. All the hyperparameters are set to the authors’ default setting.
For CoPoNeRF,
we train it for 50,000 iterations using 8 A6000 GPUs with effective batch size of 64, following the authors’ original implementations and hyperparameters. Finally, for InstantSplat (Fan et al., ), we train and evaluate on a single A6000 GPU with a batch size of 1 by following the official code(^1^11https://github.com/NVlabs/InstantSplat), and the hyperparameters were set according to the default settings provided by the authors.

### A.2 More Qualitative Results

Fig. 4, Fig. 5 and Fig. 6 present more novel view rendering results of different methods. On these datasets, our method yields outcomes that are sharper and more geometrically accurate.

### A.3 Limitations and Future Work

As our model currently lacks a mechanism to handle dynamic scenes, it is unable to accurately capture scene dynamics or perform view extrapolation. Additionally, our model’s performance is contingent on the quality of the coarse alignments, which rely on the accuracy of the depth and correspondence models. In cases where either of these models fails, our approach may not function optimally. However, because our refinement modules are lightweight, simple, and model-agnostic, incorporating more advanced methods for coarse alignment could further enhance performance.

For future work, we plan to train our model on diverse large-scale datasets. Since our approach relies exclusively on supervision signals from RGB images, it is straightforward to scale up the training data. We also aim to extend our method to handle 4D objects, ultimately enabling the modeling of 4D scenes, which would be beneficial for applications such as egocentric vision and robotics.

Figure: Figure 4: Qualitative results on RealEstate-10K dataset. Given two context views (a) and (b), we compare novel view rendering results.
Refer to caption: x4.png

Figure: Figure 5: Qualitative results on ACID dataset. Given two context views (a) and (b), we compare novel view rendering results.
Refer to caption: x5.png

Figure: Figure 6: Qualitative results on DL3DV dataset. Given two context views (a) and (b), we compare novel view rendering results.
Refer to caption: x6.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why does using the pixel-wise depth offset estimation model promote consistency across views? (line 211)
2. How about the performance of PF3plat in dynamic scenes?
soundness: 3
strengths: 1. The task of novel view synthesis from unposed images in a single feed-forward pass is highly practical.
2. The paper demonstrates state-of-the-art results on Re10k and ACID, showcasing the effectiveness of the proposed method. 
3.  The refinement modules designed in the paper have significantly improved the effectiveness.
summary: This paper introduces PF3plat, a novel framework designed for novel view synthesis from unposed images in a single feed-forward pass. PF3plat leverages pre-trained monocular depth estimation and visual correspondence models to achieve an initial coarse alignment of 3D Gaussians. Subsequently, PF3plat incorporates refinement modules and geometry-aware scoring functions to further refine the depth and pose estimates derived from the coarse alignment to enhance the quality of view synthesis.
weaknesses: 1. PF3plat leverages a robust solver for pose estimation between each pair of cameras; thus, increasing the number of viewpoints significantly extends the feed-forward pass time.
2. PF3plat relies on the coarse alignment of 3D Gaussians, and a small overlap may affect the quality of the correspondence model.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: As I already discussed in the weakness section, it would be very imporatnt if you can justify those points in the experiments.
soundness: 2
strengths: 1. The authors tackle the problem of 3D Gaussian splatting from unposed sparse images, which is an interesting and important topic
2. The authors apply the recent state-of-the-art depth estimation and pose estimation methods for coarse pose alignment, and further introduces pose and depth refinements, to some extend improving the final performance.
3. The paper is overall well-written and easy to follow in most parts.
summary: This paper introduces a method to tackle the challenging problem of novel view synthesis (NVS) from unposed images in a feed-forward manner. They identify the issue in previous pixel-aligned 3DGS methods, where the predicted 3D Gaussians from different views have the problem of misalignment, leading to noisy gradient flows and poor NVS results. They propose a method where they don’t need to rely on the poses obtained from off-the-shelf tools. Instead, they leverage pre-trained monocular depth estimation and visual correspondence models to obtain coarse alignment, with further refinement of the depth map and poses. Results show that among the pose free methods, they can perform decently better results for NVS tasks. However, for pose estimation, it is still worse than general methods like Mast3R.
weaknesses: **The method section is overall clear, but missing some details and discussions**

*Unclear descriptions in camera pose refinement in Sec 3.2.3*  
1. It is quite unclear to me how exactly it is done. From your writing, it makes me feel that you first will get a newly computed camera poses $\hat{P_{ij}}$ similarly as done in the coarse step, but with the refined depth. This $\hat{P}_{ij}$ is already the refined poses, right? However, you further have another refinement step as shown in Eq. (2). What are the rationale before?
2. what is the T_pose network? What is the E_pos in eq. (2)?

*Cost volume*  
In section 3.2.4, for the “cost volume construction and aggregation”, is that any different to the MVSplat paper? Can you justify the differences?

*2D-3D Consistency loss*  
Line 291-292, you said that you improve the robustness of the model in regions with low texture or significant viewpoint changes. However, the correspondences from the feature matching methods like LightGlue do not provide many correspondences in those low-texture regions. I guess you cannot claim the robustness there? 

*Unclear implementation details*  
What is the frame distance from 45 to 75? Do you mean you sample one frame every 45/75 frames in the video sequence? You might want to make this point clearer. And why for DL3DV, you only sample every 5 or 10 frames, way smaller than ACID or RE10LK?
Also, you mention that you train for 40000 iterations on a single A6000 GPU, is it the same for all three datasets? If true, I think it might not make too much sense? As you mentioned in line 351-354, RE10K has ~21K videos from training, while you only use 2K scenes for the training of DL3DV?



**The experimental section is convincing in general, but lacks some important experiments / baselines and explanation.**

1. For pose estimation comparison, the sota method right now is RoMa [1], I think it is fair to ask to compare to it.
2. I wonder why your method is still lacking behind Mast3R on the pose estimation for both RE10K and ACID, even if Mast3R is is not even trained at all on those dataset, while your method was trained on those dataset separately.
3. Why for novel view synthesis in Table 1, you don’t show the comparison to the recent pose-required methods pixelSplat?
4. Based on the experiments you show, your pose estimation is worse than Mast3R in almost all cases, and the NVS results are also worse than MVSplat in all scenarios (even on DL3DV in Table 3, where MVSplat was not trained on). One reasonable baseline to me is, get camera poses from Mast3R, and then run MVSplat directly. I wonder how your method compares to such a simple baseline?
5. In your ablation study table 4, what is the point of adding V, I-I, I-II, I-V, if they are just all N.A.? That is really weird to me. You can just describe them in texts.


**Writing**  
Sec 3.2.1: You use two paragraphs to motivate by mentioning the limitations of the previous methods. The real content for the coarse alignment is really just the third paragraph between line 186-191. The motivations part is actually unrelated to the coarse alignment but more on why your method is needed, so you should just put them in the introduction instead.

[1] Edstedt et al.: RoMa: Robust dense feature matching, CVPR 2024

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Q1 - How would the results differ if alternative coarse prediction networks, such as Dust3r[1], Mast3r[2], or others, were used?

Q2 - Qualitative results for the pose estimation task.

[1]Wang S, Leroy V, Cabon Y, et al. Dust3r: Geometric 3d vision made easy[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 20697-20709.
[2]Leroy V, Cabon Y, Revaud J. Grounding Image Matching in 3D with MASt3R[J]. arXiv preprint arXiv:2406.09756, 2024.
soundness: 3
strengths: S1 - This paper addresses a meaningful task with significant potential for real-world applications.

S2 - This paper leverages two robust pre-trained models to generate initial pose and shape estimates, which significantly enhance the model's performance.

S3 - The paper is well-written, and the experiments are logically sound.
summary: This paper focuses on the pose-free feed-forward novel view synthesis task. It leverages a pre-trained monocular depth estimation model and a visual correspondence model to generate coarse pose and depth estimates, enhancing the stability of the 3DGS training process. Subsequently, a lightweight refinement model is used to further improve the depth and pose estimations. Extensive experiments have been conducted to demonstrate the effectiveness of the proposed method.
weaknesses: W1 - The performance of this method appears to be highly dependent on the coarse pose and depth results provided by the pre-trained model.

W2 - The paper lacks qualitative results for the pose estimation, which would provide a clearer assessment of the model's performance in this area.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Can the authors provide more insights on how you obtain the coarse camera parameters? I guess the authors directly implemented the existing work to obtain those things.
2. the authors trained different checkpoints on different datasets in the implementation details. Does it mean the 'feed-forward' claimed by the authors is actually dataset-specific? If so, I think it is a big limitation of the proposed method lacking the generalizability to different scenes.
soundness: 2
strengths: 1. The authors proposed a new pose-free feed-forward method in camera pose estimation.
2. The authors conducted enough ablation studies to present the contribution of each component of their method.
summary: Given G.T. camera intrinsics, the authors first leverage the existing work to obtain the coarse camera poses and depth. Then to refine the estimation, the authors design a module to learn the depth offset estimation with the help of an existing depth estimation network. Furthermore, the camera pose refinement is conducted in another module. The idea of feedforward pose estimation is interesting, but there is still a gap between the performance of the proposed method and some per-scene optimization methods. Since I did not see the authors report any inference time result and I believe some static scene pose-free per-scene optimization methods (CF-3DGS, ..) are very fast and accurate, I expect the authors to provide more comparisons. There are still some questions and limitations raised below. I will consider improving the grade upon the feedback from the authors.
weaknesses: 1. The biggest limitation of this work is the requirement of G.T. camera intrinsic.
2. The performance on RealEstate-10K seems to be SOTA, however, the performance on ACID is not. Does it mean such a method does not generalize well to the outdoor scenes?
3. So I expect more comparisons on DL3DV or more public datasets (like DAVIS[1], iPhone[2]) to prove the effectiveness of the proposed method. 


[1] Pont-Tuset, Jordi, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. "The 2017 davis challenge on video object segmentation." arXiv preprint arXiv:1704.00675 (2017).

[2] Gao, Hang, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. "Monocular dynamic view synthesis: A reality check." Advances in Neural Information Processing Systems 35 (2022): 33768-33780.

Meta-review: This paper receives borderline final ratings of 6,5,6,5. The AC looked through the reviews, the rebuttal and the discussions between the reviewers and authors, and finally decided to reject the paper due to the remaining concerns raised by the two reviewers who gave negative ratings are valid. Specifically, there is a limited improvement over other simple baselines like instantsplat. The requirement of the depth refinement step is questionable, and the core algorithm is MVSplat with the replacement of estimated camera poses to replace the ground truth. Furthermore, the proposed method requires known camera intrinsics. In contrast, Vggsfm and Dust3R do not have this requirement. The results are also benchmarked on one public dataset, which is deemed insufficient. The two reviewers decided to give borderline rejects even after rebuttal and discussions. On the other hand, the two reviewers who are positive did not give very strong reasons to justify the acceptance of this work. The comments are this is a meaningful task, paper is well-written, highly practical, etc.

Decision: Reject