No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images

## Abstract

Abstract We introduce NoPoSplat, a feed-forward model capable of reconstructing 3D scenes parameterized by 3D Gaussians from unposed sparse multi-view images. Our model, trained exclusively with photometric loss, achieves real-time 3D Gaussian reconstruction during inference. To eliminate the need for accurate pose input during reconstruction, we anchor one input view’s local camera coordinates as the canonical space and train the network to predict Gaussian primitives for all views within this space. This approach obviates the need to transform Gaussian primitives from local coordinates into a global coordinate system, thus avoiding errors associated with per-frame Gaussians and pose estimation. To resolve scale ambiguity, we design and compare various intrinsic embedding methods, ultimately opting to convert camera intrinsics into a token embedding and concatenate it with image tokens as input to the model, enabling accurate scene scale prediction. We utilize the reconstructed 3D Gaussians for novel view synthesis and pose estimation tasks and propose a two-stage coarse-to-fine pipeline for accurate pose estimation.
Experimental results demonstrate that our pose-free approach can achieve superior novel view synthesis quality compared to pose-required methods, particularly in scenarios with limited input image overlap.
For pose estimation, our method, trained without ground truth depth or explicit matching loss, significantly outperforms the state-of-the-art methods with substantial improvements.
This work makes significant advances in pose-free generalizable 3D reconstruction and demonstrates its applicability to real-world scenarios.
Code and trained models are available on our project page .

## 1 Introduction

We address the problem of reconstructing a 3D scene parameterized by 3D Gaussians from unposed sparse-view images (as few as two) using a feed-forward network.
While current SOTA generalizable 3D reconstruction methods (Charatan et al., ; Chen et al., ), which aim to predict 3D radiance fields using feed-forward networks, can achieve photorealistic results without per-scene optimization, they require accurate camera poses of input views as input to the network.
These poses are typically obtained from dense videos using structure-from-motion (SfM) methods, such as COLMAP (Schonberger & Frahm, ).
This requirement is impractical for real-world applications, as these methods necessitate poses from dense videos even if only two frames are used for 3D reconstruction.
Furthermore, relying on off-the-shelf pose estimation methods increases inference time and can fail in textureless areas or images without sufficient overlap.

Recent methods (Chen & Lee, ; Smith et al., ; Hong et al., ) aim to address this challenge by integrating pose estimation and 3D scene reconstruction into a single pipeline.
However, the quality of novel view renderings from these methods lags behind SOTA approaches that rely on known camera poses (Chen et al., ). The performance gap stems from their sequential process of alternating between pose estimation and scene reconstruction. Errors in pose estimation degrade the reconstruction, which in turn leads to further inaccuracies in pose estimation, creating a compounding effect.
In this work, we demonstrate that reconstructing the scene entirely without relying on camera poses is feasible, thereby eliminating the need for pose estimation.
We achieve this by directly predicting scene Gaussians in a canonical space, inspired by the success of the recent 3D point cloud reconstruction methods (Wang et al., ; Leroy et al., ).
However, unlike DUSt3R, we show that the generalizable reconstruction network can be trained with photometric loss only without ground truth depth information and thus can leverage more widely available video data (Zhou et al., ; Liu et al., ; Ling et al., ).

Figure: Figure 2: Comparison with pose-required sparse-view 3D Gaussian splatting pipeline. Previous methods first generate Gaussians in each local camera coordinate system and then transform them into a world coordinate system using camera poses. In contrast, NoPoSplat directly outputs 3D Gaussians of all views in a canonical space, facilitating a more coherent fusion of multi-view geometric content (see Sec. 3.3 for details). Furthermore, our backbone does not incorporate geometry priors that rely on substantial image overlap (such as epipolar geometry in pixelSplat (Charatan et al., ) or cost volume in MVSplat (Chen et al., )). Consequently, NoPoSplat demonstrates better geometric detail preservation when the overlap between input views is limited.
Refer to caption: x2.png

Specifically, as shown in Fig. 2 (b), we anchor the local camera coordinate of the first view as the canonical space and predict the Gaussian primitives (Kerbl et al., ) for all input views within this space. Consequently, the output Gaussians will be aligned to this canonical space. This contrasts with previous methods (Charatan et al., ; Chen et al., ), as illustrated in Fig. 2 (a), where Gaussian primitives are first predicted in each local coordinate system and then transformed to the world coordinate using the camera pose and fused together.
Compared to the transform-then-fuse pipeline, we require the network to learn the fusion of different views directly within the canonical space, thereby eliminating misalignments introduced by explicit transformations (see Fig. 5).

Although the proposed pose-free pipeline is simple and promising, we observe significant scale misalignment in the rendered novel views compared to the ground truth target views (see Fig. 8), *i.e*., the scene scale ambiguity issue. Upon analyzing the image projection process, we find that the camera’s focal length is critical to resolving this scale ambiguity. This is because the model reconstructs the scene solely based on the image appearance, which is influenced by the focal length. Without incorporating this information, the model struggles to recover the scene at the correct scale.
To address this issue, we design and compare three different methods for embedding camera intrinsics and find that simply converting the intrinsic parameters into a feature token and concatenating it with the input image tokens enables the network to predict the scene of a more reasonable scale, yielding the best performance.

Once the 3D Gaussians are reconstructed in the canonical space, we leverage it for both novel view synthesis (NVS) and pose estimation. For pose estimation, we introduce a two-stage pipeline: first, we obtain an initial pose estimate by applying the PnP algorithm (Hartley & Zisserman, ) to the Gaussian centers. This rough estimate is then refined by rendering the scene at the estimated pose and optimizing the alignment with the input view using photometric loss.

Extensive experimental results demonstrate that our method performs impressively in both NVS and pose estimation tasks. For NVS, we show for the first time that, when trained on the same dataset under the same settings, a pose-free method can outperform pose-dependent methods, especially when the overlap between the two input images is small.
In terms of pose estimation, our method significantly outperforms prior SOTA across multiple benchmarks.
Additionally, NoPoSplat generalizes well to out-of-distribution data. Since our method does not require camera poses for input images, it can be applied to user-provided images to reconstruct the underlying 3D scene and render novel views. To illustrate this, we apply our model to sparse image pairs captured with mobile phones, as well as to sparse frames extracted from videos generated by Sora (OpenAI, ).

The main contributions of this work are:

- •
We propose NoPoSplat, a feed-forward network that reconstructs 3D scenes parameterized by 3D Gaussians from unposed sparse-view inputs, and demonstrate that it can be trained using photometric loss alone.
- •
We investigate the scale ambiguity issue of the reconstructed Gaussians, and solve this problem by introducing a camera intrinsic token embedding.
- •
We design a two-stage pipeline that estimates accurate relative camera poses using the reconstructed Gaussian field.
- •
Experimental results show that our method achieves remarkable performance in both NVS and pose estimation tasks, and can be applied to in-the-wild data.

## 2 Related Work

Generalizable 3D Reconstruction and View Synthesis.
NeRF (Mildenhall et al., ) and 3D Gaussian Splatting (3DGS) (Kerbl et al., ) have significantly advanced 3D reconstruction and novel view synthesis. However, these methods typically require dense posed images (*e.g*., hundreds) as input and minutes to hours of per-scene optimization, even with improved techniques (Chen et al., ; Fridovich-Keil et al., ; Müller et al., ).
This limits their practical applications.
To address these limitations, recent approaches focus on generalizable 3D reconstruction and novel view synthesis from sparse inputs (Yu et al., ; Wang et al., ; Xu et al., ; Johari et al., ; Du et al., ; Charatan et al., ; Chen et al., ).
They typically incorporate task-specific backbones that leverage geometric information to enhance scene reconstruction. For instance, MVSNeRF (Chen et al., ) and MuRF (Xu et al., ) build cost volumes to aggregate multi-view information, while pixelSplat (Charatan et al., ) employs epipolar geometry for improved depth estimation.
However, these geometric operations often require camera pose input and sufficient camera pose overlap among input views.
In contrast, our network is based solely on a vision transformer (Dosovitskiy et al., ) without any geometric priors, making it pose-free and more effective in handling scenes with large camera baselines.

Pose-Free 3D Scene Reconstruction.
Classical NeRF or 3DGS-based methods require accurate camera poses of input images, typically obtained through Structure from Motion (SfM) methods like COLMAP (Schonberger & Frahm, ), which complicates the overall process.
Recent works (Wang et al., ; Lin et al., ; Chng et al., ) jointly optimize camera poses and neural scene representations, but they still require rough pose initialization or are limited to small motions. Others (Bian et al., ; Fu et al., ) adopt incremental approaches from (Zhu et al., ; ; Matsuki et al., ), but they only allow image/video sequences as input.

Moreover, for generalizable sparse-view methods, requiring camera poses during inference presents significant challenges, as these poses are often unavailable in real-world applications during testing. Although two-view pose estimation methods (Wang et al., ; Edstedt et al., ) can be used, they are prone to failure in textureless regions or when images lack sufficient overlap.
Some recent pose-free novel view synthesis methods (Chen & Lee, ; Smith et al., ; Hong et al., ) attempt to address this but typically break the task into two stages: first estimate camera poses, then construct the scene representation. This two-stage process still lags behind pose-required methods because the initial pose estimation introduces noise, degrading reconstruction quality.
In contrast, our method completely eliminates camera poses by directly predicting 3D Gaussians in a canonical space, avoiding potential noise in pose estimation and achieving better scene reconstruction.

One concurrent work, Splatt3R (Smart et al., )(^1^11Arxiv submitted on August 25, 2024), also predicts Gaussians in a global coordinate system but relies on the frozen MASt3R (Leroy et al., ) model for Gaussian centers.
This is unsuitable for novel view synthesis, as MASt3R struggles to merge scene content from different views smoothly.
Moreover, in contrast to us, Splatt3R requires ground truth depth during training, so it cannot utilize large-scale video data without depths or ground truth metric camera poses.

## 3 Method

### 3.1 Problem Formulation

Our method takes as input sparse unposed multi-view images and corresponding camera intrinsic parameters $\left\{\bm{I}^{v},\bm{k}^{v}\right\}_{v=1}^{V}$, where $V$ is the number of input views, and learn a feed-forward network $f_{\bm{\theta}}$ with learnable parameters $\theta$.
The network maps the input unposed images to 3D Gaussians in a canonical 3D space, representing the underlying scene geometry and appearance. Formally, we aim to learn the following mapping:

$$ $f_{\bm{\theta}}:\left\{\left(\bm{I}^{v},\bm{k}^{v}\right)\right\}_{v=1}^{V} \mapsto\left\{\cup\left(\bm{\mu}_{j}^{v},\alpha_{j}^{v},\bm{r}_{j}^{v},\bm{s}_ {j}^{v},\bm{c}_{j}^{v}\right)\right\}_{j=1,\dots,H\times W}^{v=1,\dots,V},$ (1) $$

where the right side represents the Gaussian parameters (Kerbl et al., ). Specifically, we have the center position $\bm{\mu}\in\mathbb{R}^{3}$ , opacity $\alpha\in\mathbb{R}$,
rotation factor in quaternion $\bm{r}\in\mathbb{R}^{4}$,
scale $\bm{s}\in\mathbb{R}^{3}$, and spherical harmonics (SH) $\bm{c}\in\mathbb{R}^{k}$ with $k$ degrees of freedom.
Note that, in line with common practice in pose-free scene reconstruction methods (Hong et al., ; Fu et al., ; Smith et al., ; Chen & Lee, ), we assume having camera intrinsic parameters $\bm{k}$ as input, as they are generally available from modern devices (Arnold et al., ).

By training on large-scale datasets, our method can generalize to novel scenes without any optimization.
The resulting 3D Gaussians in the canonical space directly enable two tasks: a) novel view synthesis given the target camera transformation relative to the first input view, and b) relative pose estimation among different input views.
Next, we will introduce our overall pipeline.

Figure: Figure 3: Overview of NoPoSplat. We directly predict Gaussians in a canonical space from a feed-forward network to represent the underlying 3D scene from the unposed sparse images. For simplicity, we use a two-view setup as an example, and the RGB shortcut is omitted from the figure.
Refer to caption: x3.png

### 3.2 Pipeline

Our method, illustrated in Fig. 3, comprises three main components: an encoder, a decoder, and Gaussian parameter prediction heads. Both encoder and decoder utilize pure Vision Transformer (ViT) structures, without injecting any geometric priors (e.g. epipolar constraints employed in pixelSplat (Charatan et al., ), or cost volumes in MVSplat (Chen et al., )). Interestingly, we demonstrate in Sec. 4 that such a simple ViT network shows competitive or superior performance over those dedicated backbones incorporating these geometric priors, especially in scenarios with limited content overlap between input views. This advantage stems from the fact that such geometric priors typically necessitate substantial overlap between input cameras to be effective.

ViT Encoder and Decoder.
The RGB images are patchified and flattened into sequences of image tokens, and then concatenated with an intrinsic token (details in Sec. 3.4). The concatenated tokens from each view are then fed into a ViT (Dosovitskiy et al., ) encoder separately. The encoder shares the same weights for different views.
Next, the output features from the encoder are fed into a ViT decoder module, where features from each view interact with those from all other views through cross-attention layers in each attention block, facilitating multi-view information integration.

Gaussian Parameter Prediction Heads.
To predict the Gaussian parameters, we employ two prediction heads based on the DPT architecture (Ranftl et al., ). The first head focuses on predicting the Gaussian center positions and utilizes features extracted exclusively from the transformer decoder.
The second head predicts the remaining Gaussian parameters and, in addition to the ViT decoder features, also takes the RGB image as input.
Such RGB image shortcut ensures the direct flow of texture information, which is crucial for capturing fine texture details in 3D reconstruction.
This approach compensates for the high-level features output by the ViT decoder, downsampled by a factor of 16, which are predominantly semantic and lack detailed structural information.
With these prediction heads in place, we now analyze how our method differs from previous approaches in terms of the output Gaussian space and the advantages this brings.

### 3.3 Analysis of the Output Gaussian Space

While our method shares a similar spirit with previous works (Charatan et al., ; Zheng et al., ; Szymanowicz et al., ) in predicting pixelwise Gaussians for input images, we differ in the output Gaussian space. In this section, we first discuss the local-to-global Gaussian space in prior methods and its inherent limitations, and introduce our canonical Gaussian space.

Baseline: Local-to-Global Gaussian Space.
Previous methods first predict the corresponding depth of each pixel, then lift the predicted Gaussian parameters to a Gaussian primitive in the local coordinate system of each individual frame using the predicted depth and the camera intrinsics.
These local Gaussians are then transformed into a world coordinate system using the given camera poses $[\bm{R}^{v}\mid\bm{t}^{v}]$ for each input view. Finally, all transformed Gaussians are fused to represent the underlying scene.

However, this strategy has two main issues:
a) Transforming Gaussians from local to world coordinates requires accurate camera poses, which are difficult to obtain in real-world scenarios with sparse input views.
b) The transform-then-fuse method struggles to combine 3D Gaussians from different views into a cohesive global representation, especially when the overlap among input views is small, or when generalizing to out-of-distribution data (as shown in Fig. 5 and Fig. 6).

Proposed: Canonical Gaussian Space.
In contrast, we directly output Gaussians of different views in a canonical coordinate system.
Specifically, we anchor the first input view as the global reference coordinate system.
Therefore, the camera pose for the first input view is $[\bm{U}\mid\bm{0}]$, where $\bm{U}$ is the unit/identity matrix for rotation, and $\bm{0}$ is the zero translation vector.
The network outputs Gaussians under this canonical space for all input views. Formally, for each input view, we predict the set $\left\{\bm{\mu}^{v\rightarrow 1}_{j},\bm{r}^{v\rightarrow 1}_{j},\bm{c}^{v
\rightarrow 1}_{j},\alpha_{j},\bm{s}_{j}\right\}$, where the superscript $v\rightarrow 1$ denotes that the Gaussian parameters corresponding to pixel $\bm{p}_{j}$ in view $v$, are under the local camera coordinate system of view $1$.

Predicting directly under the canonical space offers several benefits.
First, the network learns to fuse different views directly within the canonical space, eliminating the need for camera poses.
Second, bypassing the transform-then-fuse step results in a cohesive global representation, which further unlocks the application of pose estimation for input unposed views.

### 3.4 Camera Intrinsics Embedding

As discussed in Eq. 1, our network inputs also include the camera intrinsics $\bm{k}$ of each input view.
This is required to resolve the inherent scale misalignment and provide essential geometric information that improves 3D reconstruction quality (*cf*. “No Intrinsics” in Fig. 8 and Tab. 5).
We introduce three different encoding strategies for injecting camera intrinsics into our model.

Global Intrinsic Embedding - Addition.
A straightforward strategy is to feed camera intrinsics $\bm{k}=[f_{x},f_{y},c_{x},c_{y}]$ into a linear layer to obtain a global feature.
This feature is broadcast and added to the RGB image features after the patch embedding layer of the ViT.

Global Intrinsic Embedding - Concat.
After obtaining the global feature, we instead treat it as an additional intrinsic token, and concatenate it with all image tokens.

Dense Intrinsic Embedding.
For each pixel $\bm{p_{j}}$ in the input view, we can obtain the camera ray direction as $\bm{K}^{-1}\bm{p_{j}}$, where $\bm{K}$ is the matrix form of $\bm{k}$. These per-pixel camera rays are then converted using spherical harmonics to higher-dimension features and concatenated with the RGB image as the network input.
Note that the pixel-wise embedding can be viewed as a simplification of the widely-used Plücker ray embedding (Xu et al., ; Tang et al., ), as it does not require camera extrinsic information.

By default, we utilize the intrinsic token option since it is not only an elegant way to inject camera intrinsic information into the network, but also yields the best performance. (see Tab. 5).

### 3.5 Training and Inference

Training.
The predicted 3D Gaussians are used for rendering images at novel viewpoint.
Our network is end-to-end trained using ground truth target RGB images as supervision.
Following Chen et al. (), we also use a linear combination of MSE and LPIPS (Zhang et al., ) loss with weights of 1 and 0.05, respectively.

Relative Pose Estimation.
As mentioned in Sec. 3.3, since our 3D Gaussians are in the canonical space, they can be directly used for relative pose estimation.
To facilitate efficient pose estimation, we propose a two-step approach.
First, we estimate the initial related camera pose of the input two views using the PnP algorithm (Hartley & Zisserman, ) with RANSAC (Fischler & Bolles, ), given the Gaussian centers of the output Gaussians in world coordinates. This step is highly efficient and done in milliseconds.
Next, while keeping Gaussian parameters frozen, we refine the initial pose from the first step by optimizing the same photometric losses used for model training, along with the structural part of the SSIM loss (Wang et al., ).
During the optimization, we calculate the camera Jacobian to reduce the computational overhead associated with automatic differentiation and decrease optimization time as in Matsuki et al. ().

Evaluation-Time Pose Alignment.
Given unposed image pairs, our method learns to reconstruct a plausible 3D scene that aligns with the given inputs.
However, 3D scene reconstruction with just two input views is inherently ambiguous as many different scenes can produce the same two images.
As a result, though the scene generated by our method successfully explains the input views, it might not be exactly the same as the ground truth scene in the validation dataset.
Thus, to fairly compare with other baselines, especially ones that utilize ground truth camera poses(Du et al., ; Charatan et al., ), we follow previous pose-free works (Wang et al., ; Fan et al., ) and optimize the camera pose for the target view.
Specifically, for each evaluation sample, we first reconstruct 3D Gaussians using the proposed method. We then freeze the Gaussians and optimize the target camera pose such that the rendered image from the target view closely matches the ground truth image.
It is important to note that this optimization is solely for evaluation purposes and is not required when applying our method in real-world scenarios (*e.g*., Fig. 7).

## 4 Experiments

Datasets.
To evaluate novel view synthesis, we follow the setting in (Charatan et al., ; Chen et al., ) and train and evaluate our method on RealEstate10k (RE10K) (Zhou et al., ) and ACID (Liu et al., ) datasets separately.
RE10K primarily contains indoor real estate videos, while ACID features nature scenes captured by aerial drones. Both include camera poses calculated using COLMAP (Schonberger & Frahm, ). We adhere to the official train-test split as in previous works (Charatan et al., ; Hong et al., ).
To further scale up our model (denoted as Ours*), we also combine RE10K with DL3DV (Ling et al., ), which is an outdoor dataset containing 10K videos, which includes a wider variety of camera motion patterns than RE10K.

To assess the method’s capability in handling input images with varying camera overlaps, we generate input pairs for evaluation that are categorized based on the ratio of image overlap:
small (0.05% - 0.3%), medium (0.3% - 0.55%), and large (0.55% - 0.8%), determined using SOTA dense feature matching method, RoMA (Edstedt et al., ). More details are provided in the appendix.

Furthermore, for zero-shot generalization, we also test on
DTU (Jensen et al., ) (object-centric scenes), ScanNet (Dai et al., ) and ScanNet++ (Yeshwanth et al., ) (indoor scenes with different camera motion and scene types from the RE10K).
We also demonstrate our approach on in-the-wild mobile phone capture, and images generated by a text-to-video model (OpenAI, ).

Evaluation Metrics.
We evaluate novel view synthesis with the commonly used metrics: PSNR, SSIM, and LPIPS (Zhang et al., ).
For pose estimation, we report the area under the cumulative pose error curve (AUC) with thresholds of 5^∘, 10^∘, 20^∘ (Sarlin et al., ; Edstedt et al., ).

Baselines.
We compare against SOTA sparse-view generalizable methods on novel view synthesis:
1) *Pose-required*: pixelNeRF (Yu et al., ), AttnRend (Du et al., ), pixelSplat (Charatan et al., ), and MVSplat (Chen et al., );
2) *Pose-free*: DUSt3R (Wang et al., ), MASt3R (Leroy et al., ), Splatt3R (Smart et al., ), CoPoNeRF (Hong et al., ), and RoMa (Edstedt et al., ).
For relative pose estimation, we also compare against methods in 2).

Table: Table 1: Novel view synthesis performance comparison on the RealEstate10k (Zhou et al., ) dataset. Our method largely outperforms previous pose-free methods on all overlap settings, and even outperforms SOTA pose-required methods, especially when the overlap is small.

Table: Table 2: Novel view synthesis performance comparison on the ACID (Liu et al., ) dataset.

Implementation details.
We use PyTorch, and the encoder is a vanilla ViT-large model with a patch size of 16, and the decoder is ViT-base.
We initialize the encoder/decoder and Gaussian center head with the weights from MASt3R, while the remaining layers are initialized randomly.
Note that, as shown in the appendix, our method can also be trained with only RGB supervision–without pre-trained weight from MASt3R–and still achieve similar performance.
We train models at two different resolutions, $256\times 256$ and $512\times 512$. For a fair comparison with baseline models, we report all quantitative results and baseline comparisons under $256\times 256$. However, qualitative results for the $512\times 512$ model are presented in the supplementary video and Fig. 7. Additional details on model weight initialization and training resolution can be found in the appendix.

### 4.1 Experimental Results and Analysis

Figure: Figure 4: Qualitative comparison on RE10K (top three rows) and ACID (bottom row). Compared to baselines, we obtain: 1) more coherent fusion from input views, 2) superior reconstruction from limited image overlap, 3) enhanced geometry reconstruction in non-overlapping regions.

Novel View Synthesis.
As demonstrated in Tab. 1, Tab. 2, and Fig. 4, NoPoSplat significantly outperforms all SOTA pose-free approaches.
Note that DUSt3R (and MASt3R) struggle to fuse input views coherently due to their reliance on per-pixel depth loss, a limitation Splatt3R also inherits from its frozen MASt3R module.
On the other hand, we achieve competitive performance over SOTA pose-required methods (Charatan et al., ; Chen et al., ), and even outperform them when the overlap between input images is small, as shown in Fig.4 first row (the left side of Fig. 4 displays the overlap ratio).
This clearly shows the advantages of our model’s 3D Gaussians prediction in the canonical space over baseline methods’ transform-then-fuse strategy, as discussed in Sec. 3.3.

Relative Pose Estimation.

Table: Table 3: Pose estimation performance in AUC with various thresholds on RE10k, ACID, and ScanNet-1500 (Dai et al., ; Sarlin et al., ). Our method achieves the best results across all datasets. Notably, our model is not trained on ACID or ScanNet. Furthermore, our method does not require an explicit matching loss during training, meaning no ground truth depth is necessary.

The proposed method can be applied to pose estimation between input views on three diverse datasets.
Our method is trained either on RE10K (denoted as Ours) or a combination of RE10K and DL3DV (denoted as Ours*).
Tab. 3 shows that the performance consistently improves when scaling up training with DL3DV involved.
This can be attributed to the greater variety of camera motions in DL3DV over in RE10K.
It is worth noting that our method even shows superior zero-shot performance on ACID and ScanNet-1500, even better than the SOTA method RoMa that is trained on ScanNet. This indicates not only the efficacy of our pose estimation approach, but also the quality of our output 3D geometry. The next part verifies this point.

Geometry Reconstruction.

Figure: Figure 5: Comparisons of 3D Gaussian and rendered results. The red and green indicate input and target camera views, and the rendered image and depths are shown on the right side. The magenta and blue arrows correspond to the distorted or misalignment regions in baseline 3DGS. The results show that even without camera poses as input, our method produces higher-quality 3D Gaussians resulting in better color/depth rendering over baselines.
Refer to caption: x4.png

Our method also outputs noticeably better 3D Gaussians and depths over SOTA pose-required methods, as shown in Fig. 5.
Looking closely, MVSplat not only suffers from the misalignment in the intersection regions of two input images (indicated by blue arrows), but also distortions or incorrect geometry in regions without sufficient overlap (indicated by magenta arrows).
These issues are largely due to the noises introduced in their transform-then-fuse pipeline.
Our method directly predicts Gaussians in the canonical space, which faithfully solves this problem.

Figure: (a) Cross-Dataset Generalize: RE10K $\rightarrow$ DTU

Table: Table 4: Out-of-distribution performance comparison. Our method shows superior performance when zero-shot evaluation on DTU and ScanNet++ using the model solely trained on RE10k.

Cross-Dataset Generalization.
We also evaluate the zero-shot performance of the model, where we train exclusively on RealEstate10k and directly apply it to ScanNet++ (Yeshwanth et al., ) and DTU (Jensen et al., ) datasets. The results in Fig. 6 and Tab. 4 indicate that NoPoSplat demonstrates superior performance on out-of-distribution data compared to SOTA pose-required methods. This advantage arises primarily from our minimal geometric priors in the network structure, allowing it to adapt effectively to various types of scenes. Notably, our method outperforms Splatt3R even on ScanNet++, where Splatt3R was trained.

[Uncaptioned image]: x5.png

Model Efficiency.
As shown on the right, our method can predict 3D Gaussians from two $256\times 256$ input images in 0.015 seconds (66 fps), which is around $5\times$ and $2\times$ faster than pixelSplat and MVSplat, on the same RTX 4090 GPU.
This further shows the benefits of using a standard ViT without incorporating additional geometric operations.

Figure: Figure 7: In-the-wild Data. We present the results of applying our method to in-the-wild data, including real-world photos taken with mobile phones and multi-view images extracted from videos generated by the Sora text-to-video model. See the video supplementary for more results.
Refer to caption: x6.png

Apply to In-the-Wild Unposed Images.
One of the most important advantages of our method is that it can directly generalize to in-the-wild unposed images.
we test on two types of data: images casually taken with mobile phones, and image frames extracted from videos generated by Sora (OpenAI, ).
Results in Fig. 7 show that our method can be potentially applied for text/image to 3D scene generations, *e.g*., one can first generate sparse scene-level multi-view images using text/image to multi-image/video models (Gao et al., ; OpenAI, ), then feed those extracted unposed images to our model and obtain 3D models.

### 4.2 Ablation Studies

Ablation on Output Gaussian Space.
To demonstrate the effectiveness of our canonical Gaussian prediction, we compare it with the transform-then-fuse pipeline commonly used by pose-required methods (Charatan et al., ; Chen et al., ).
Specifically, it first predicts Gaussians in each local camera coordinate system which was transformed to the world coordinate using camera poses.
For fair comparisons, both methods use the same backbone and head but differ in the prediction of Gaussian space.
The results in row (f) of Tab. 5 show that our pose-free canonical space prediction method outperforms such pose-required strategy. Fig. 8 illustrates that the transform-then-fuse strategy leads to the ghosting artifacts in the rendering, because it struggles to align two separate Gaussians of two input views when transformed to a global coordinate.

[width=1.0]figures/svg-inkscape/ablation_svg-tex.pdf_tex

Figure: Figure 8: Ablations. No intrinsic results in blurriness due to scale misalignment. Without the RGB image shortcut, the rendered images are blurry in the texture-rich areas. Using the transform-then-fuse strategy causes ghosting problem.

| Num | Design | PSNR | SSIM | LPIPS |
| --- | --- | --- | --- | --- |
| (a) | Ours | 25.033 | 0.838 | 0.160 |
| (b) | No intrinsic emb. | 23.543 | 0.780 | 0.186 |
| (c) | w/ dense emb. | 24.809 | 0.832 | 0.166 |
| (d) | w/ global emb. - add. | 24.952 | 0.836 | 0.161 |
| (e) | No RGB shortcut | 24.279 | 0.810 | 0.183 |
| (f) | Transform-then-fuse | 24.632 | 0.834 | 0.167 |
| (g) | 3 input views | 26.619 | 0.872 | 0.127 |

Table: Table 5: Ablations. intrinsic embeddings are vital for performance and using intrinsic tokens performs the best. Adding the RGB image shortcut also improves the quality of rendered images. Our method achieves better performance compared with the pose-required per-local-view Gaussian field prediction method.

Ablation on Camera Intrinsic Embedding.
Here we study three intrinsic encodings described in Sec. 3.4 as well as inputting no intrinsic information. First, we can see in Tab. 5 row (b) and Fig. 8 that no intrinsic encodings lead to blurry results as the scale ambiguity confuses the learning process of the model.
Second, we notice that the intrinsic token constantly performs the best marginally among these three proposed intrinsic encodings, so, we use it as our default intrinsic encoding.

Importance of RGB Shortcut.
As discussed in Sec. 3.2, in addition to the low-res ViT features, we also input RGB images into the Gaussian parameter prediction head.
As shown in Fig. 8, when there is no RGB shortcut, the rendered images are blurry in the texture-rich areas, see the quilt in row 1 and chair in row 3.

Extend to 3 Input Views.
For fair comparisons with baselines, we primarily conduct experiments using two input view settings. Here we present results using three input views by adding an additional view between the two original input views. As shown in row (g) of Tab. 5, the performance significantly improves with the inclusion of the additional view.

## 5 Conclusion

This paper introduces a simple yet effective pose-free method for generalizable sparse-view 3D reconstruction. By predicting 3D Gaussians directly in a canonical space from any given unposed multi-view images, we demonstrate superior performance in novel view synthesis and relative pose estimation.
While our method currently applies only to static scenes, extending our pipeline to dynamic scenarios presents an interesting direction for future work.

## Appendix A More Implementation Details

More Training Details.
We first describe the process for training the $256\times 256$ model, which serves as the basis for all baseline comparisons. Following Chen et al. (), when training on RealEstate10K (Zhou et al., ) and ACID (Liu et al., ) separately, the model is trained on $2.4\times 10^{6}$ input image pairs, randomly sampled from training video clips to ensure diverse input coverage.
We employ the AdamW optimizer (Loshchilov & Hutter, ), setting the initial learning rate for the backbone to $2\times 10^{-5}$ and other parameters to $2\times 10^{-4}$.
The model is trained using 8 GH200 GPUs with a total batch size of 128, which takes approximately 6 hours to complete.
When trained on the combined RealEstate10k and DL3DV (Ling et al., ) datasets, we sample image pairs evenly from both datasets and double the training steps.

We then train the $512\times 512$ model using the pre-trained weights of the $256\times 256$ model. The $512\times 512$ model is also trained on the combined RealEstate10K and DL3DV datasets, following the same training procedure. Since no fair baseline model is available for comparison, we focus primarily on the qualitative results of the $512\times 512$ model. For instance, the results presented in Fig. 7 are obtained using the $512\times 512$ model.

Evaluation Set Generation.
For the evaluation of the novel view synthesis task, we randomly select one image pair from each video sequence in the official test splits of RealEstate10K (Zhou et al., ) and ACID (Liu et al., ). We then randomly sample three frames between the input views as target views. For the pose estimation task, we use the same input image pair as in the novel view synthesis task, and employ the COLMAP pose as the ground truth.

To determine the overlap ratio of image pairs, we employ RoMa (Edstedt et al., ), a state-of-the-art dense feature matching method. The process for calculating the overlap ratio for an image pair $\left\{\bm{I}^{1},\bm{I}^{2}\right\}$ is as follows:

- 1.
Obtain dense image matching from $\bm{I}^{1}$ to $\bm{I}^{2}$ and vice versa.
- 2.
Consider matching scores above 0.005 as valid.
- 3.
Calculate the overlap ratio $r_{i\rightarrow j}$ for image $\bm{I}^{i}$ to image $\bm{I}^{j}$ as:
$r_{i\rightarrow j}=\frac{\text{Number of valid matched pixels}}{\text{Total
number of pixels}}$
- 4.
Compute $r_{1\rightarrow 2}$ and $r_{2\rightarrow 1}$.
- 5.
Define the final overlap ratio as:
$r_{\text{overlap}}=\min(r_{1\rightarrow 2},r_{2\rightarrow 1})$

The resulting evaluation set and the code used for its generation will be made publicly available to facilitate further research in this area.

Statistics on the Evaluation Set.
The number of scenes in RealEstate10K (Zhou et al., ) for each overlap category is as follows: 1403 for small, 2568 for medium, and 1630 for large overlaps. In the ACID dataset (Liu et al., ), the number of scenes for each overlap category is: 249 for small, 644 for medium, and 448 for large overlaps.

Baseline Setup for Pose Estimation Tasks.
For a fair comparison with RoMa (Edstedt et al., ), we first resize and center-crop the input images to match the dimensions used in our model ($256\times 256$), then resize them to a coarse resolution (560 × 560) and an upsampled resolution (864 × 864) to fit RoMa’s requirements. However, since DUSt3R (Wang et al., ) and MASt3R (Leroy et al., ) are not trained with square image inputs, simply setting the input resolution to $256\times 256$ leads to poor pose estimation performance. Therefore, we resize and center-crop their input images to $512\times 256$, as officially supported by their models. Consequently, the image content visible to DUSt3R and MASt3R is greater than that seen by RoMa and our model.

Details on Pose Estimation.
After obtaining the coarse camera pose, we apply photometric loss-based optimization to refine it. This optimization is performed for 200 steps per input image pair with a learning rate of $5\times 10^{-3}$, taking approximately 2 seconds. Notably, our pose estimation can also operate without the optimization refinement process, resulting in an acceptable performance degradation but decreasing inference time. As shown in Tab. 7, despite the performance drop, our method still performs comparably to RoMa, which is trained on ScanNet, even in a zero-shot setting.

Details on Applying to In-the-Wild Data.
Our method requires camera intrinsic information as input for rendering novel views. When testing our model on out-of-distribution datasets, such as Tanks and Temples (Knapitsch et al., ), we simply use the ground truth intrinsic data provided by the dataset. When tested on photos taken with mobile phones, we extract the focal length from their EXIF metadata. For data generated by SORA (OpenAI, ), we set the focal length to $(H+W)/2$. We found that this heuristic setting works quite well, as our model is relatively robust to focal length variations after training.

## Appendix B More Experimental Analysis

Ablations on Backbone Initialization.

Table: Table 6: Ablation on different weight initialization.

We initialize our backbone network with MASt3R (Leroy et al., ) pre-trained weights in our main experiences. MASt3R is trained on datasets with ground truth depth annotation, but notably, its training data has no overlap with the training and evaluation datasets used in our experiences.
Here, we also explore the effect of different backbone initialization methods. We compare the performance of our method with DUSt3R (Wang et al., ) and CroCo V2 (Weinzaepfel et al., ) pre-trained weights separately.
DUSt3R is also trained with depth supervision but without feature matching loss compared with MASt3R. CroCoV2 is pre-trained with pure 2D image pairs with reconstruction loss (He et al., ). Note that for CroCoV2 initialization, we warm up the training by adding point cloud distillation loss from the DUSt3R model for 1,000 steps, this aims to tell the network the goal is to predict the Gaussians in the canonical space, otherwise, the learning target is too hard for the network to understand as we only train our network with photometric loss.
The results in Tab. 6 demonstrate that our method, even when initialized without depth-supervised pre-trained weights, performs similarly to models initialized with DUSt3R and MASt3R. Moreover, it consistently outperforms previous state-of-the-art pose-required methods (pixelSplat(Charatan et al., ) and MVSplat (Chen et al., )) on scenes with small overlap, while offering comparable performance on scenes with large overlap.
This indicates that our canonical Gaussian space representation can be learned without depth pre-trained weights.
However, we also find that using MASt3R pre-trained weights performs slightly better than others. This improvement may brought by the feature matching loss used in MASt3R, and we will leave this as a further exploration direction.

Ablations on the Effectiveness of Two-Stage Pose Estimation.

Table: Table 7: Ablation on the pose estimation method.

As shown in Tab. 7, relying solely on PnP-RANSAC results in inaccurate pose estimates. However, if we skip the coarse camera pose estimation stage and only use photometric loss-based optimization starting from $[\bm{I}\mid\bm{0}]$, the performance significantly degrades when using the same number of optimization steps (200 steps). This is because optimizing from an initial pose far from the target is more challenging.

Evaluate on the Evaluation Set of pixelSplat.

**Table 8: Performance comparison on the evaluation set of pixelSplat.**
|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| --- | --- | --- | --- |
| pixelNeRF | 20.43 | 0.589 | 0.55 |
| GPNR | 24.11 | 0.793 | 0.255 |
| AttnRend | 24.78 | 0.82 | 0.213 |
| pixelSplat | 26.09 | 0.863 | 0.136 |
| MVSplat | 26.39 | 0.869 | 0.128 |
| Ours | 26.786 | 0.878 | 0.124 |

We also present results based on the evaluation set used by pixelSplat (Charatan et al., ) and MVSplat (Chen et al., ). The results are shown in Tab. 8. However, we do not prioritize this evaluation set as it is relatively simple, with most input pairs exhibiting significant overlap, making it less suitable for more advanced future research.

Details on Extension to 3 Input Views.
Our method can be extended to an arbitrary number of input views. As shown in Tab. 5, using three input views improves the results. Specifically, we add the middle frame between the original two input frames as an additional view. For three-input settings, when generating the Gaussians of each view, we concatenate the feature tokens from all other views and perform cross-attention with these concatenated tokens in the ViT decoder stage, keeping all other operations the same as with two input views. Fig. 9 provides a qualitative comparison between using two and three input views.

Table: Table 9: Performance of retrained Splatt3R (Smart et al., ) model.

Addtional Comparison with Splatt3R.
In Tab.1 of the main paper, we compare our method with the official model provided by the authors of Splatt3R(Smart et al., ), which freezes the MASt3R model and is then trained on the ScanNet++ (Yeshwanth et al., ) dataset.
Here, we also train it on the RealEstate10K (Zhou et al., ) dataset at the same resolution as other baselines ($256\times 256$). However, when attempting to retrain it using the official code provided by the authors on RealEstate10k, we find that the training fails because the original Splatt3R can only be trained on datasets with metric pose ground truth. This limitation arises because Splatt3R relies on a fixed pre-trained MASt3R (Leroy et al., ) model and its capability for metric depth prediction. As MASt3R is trained on ScanNet++ (Yeshwanth et al., ) using ground truth metric depth information, the poses in ScanNet++ are also metric. Consequently, the original Splatt3r cannot be trained on video datasets without metric pose information. We identify two main issues:

- 1.
The camera poses provided by RealEstate10K are up-to-scale, as they are estimated using COLMAP (Schonberger & Frahm, ). This results in a scale misalignment between the ground truth pose and the scale of the estimated Gaussian field. Although Splatt3r predicts additional point offsets based on the point cloud estimated by MASt3R as the final Gaussian center, these offsets are typically small and unable to resolve the misalignment problem. This issue persists even in datasets with metric camera poses, as the point cloud provided by MASt3R is imperfect and not fully aligned with the ground truth pose scale. The training fails if the provided COLMAP target camera poses are used to render target views for the 3D Gaussians. To address this, we first estimate the camera poses of two input views using the point cloud from MASt3R, which is scale-consistent with the Gaussians. We then align the scale of the COLMAP target poses with the scale of the MASt3R point cloud by adjusting the scale of the input view poses to match the MASt3R-estimated ones.
- 2.
We also find that the intrinsic parameters estimated by MASt3R (Leroy et al., ) do not align well with the ground truth intrinsic parameters. Using the ground truth intrinsic during training also causes failure. As a result, we opt to use the intrinsics estimated by MASt3R.

Additionally, we ignore the loss mask used in the original Splatt3R as it requires ground truth depth to generate the mask, but the RealEstate10K dataset lacks ground truth depth. The mask is unnecessary for training on video datasets like RealEstate10K, since during training, we use intermediate frames between the two input frames as targets, and most of the image content in the target frames is well covered by the input frames.
For a fair comparison during evaluation, we apply the same evaluation-time pose alignment technique used in our method. The results are presented in Tab. 9. Although retraining the model on the RealEstate10K dataset yields improved performance, it still significantly lags behind our approach. This performance gap can be attributed to the misalignment issues inherent in the fixed MASt3R model and the persistent scale ambiguity problem.

Figure: Figure 9: Qualitative comparison on different numbers of input views. The model with 2 input views utilizes only the top and bottom reference images, whereas the 3-view model incorporates an additional intermediate view. Adding this extra reference view improves the quality of rendered novel views significantly, as it captures finer spatial details and reduces occlusion effects.

## Appendix C Limitations

Our approach, like previous pose-free methods (Hong et al., ; Fu et al., ), assumes known camera intrinsics. Although heuristically set intrinsic parameters prove effective for in-the-wild images, relaxing this requirement would enhance the robustness of real-world applications.
Additionally, as our feedforward model is non-generative, it lacks the ability to reconstruct unseen regions of a scene with detailed geometry and texture, as demonstrated in the results of the 2-view model shown in Fig. 9. This limitation could potentially be mitigated by incorporating additional input views, which would enhance scene coverage.
Finally, the current training data (limited to RealEstate10K, ACID, and DL3DV) constrains the model’s generalizability to diverse in-the-wild scenarios. Future work could explore training our model on large-scale, diverse indoor and outdoor datasets, leveraging our method’s independence from ground-truth depth information during training.

## Appendix D More Visual Comparisons

We present an additional comparison with previous SOTA pose-dependent and pose-free methods across various levels of image overlap: Fig.10 for small input image overlap, Fig.11 for medium input image overlap, and Fig.12 for large input image overlap. Furthermore, we provide additional comparisons on the ACID dataset (Liu et al., ) in Fig.13.

Figure: Figure 10: More comparisons of the RealEstate10K dataset with small overlap of input images.

Figure: Figure 11: More comparisons of the RealEstate10K dataset with medium overlap of input images.

Figure: Figure 12: More comparisons of the RealEstate10K dataset with large overlap of input images.

Figure: Figure 13: More comparisons on the ACID dataset.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Add quantitative results for geometry reconstruction?
2. Clarifying questions (not affecting the paper rating):
    1. It seems like the main difference between this and Splatter is using / not using depth supervision and metric poses, which leads to an "impression" that using depth supervision would limit the dataset choices and hence constrain the GS prediction performance?
    2. L838-840: "Using the ground truth intrinsic during training also causes failure". I assume this is about training Splatter failed using the ground truth intrinsics?
    3. What is the training hardware and training time?
    4. The proposed method also loaded pre-trained Master weights, are they frozen or fine-tuned?
3. Related work: maybe worth adding another concurrent work: ReconX (not affecting the paper rating)
soundness: 4
strengths: * The paper is very well written and easy to follow.
* Good reconstruction and NVS results while no need for depth, loss masks, and metric poses comparing with prior work.
* Good generalization capability on in-the-wild data.
summary: This submission proposes a sparse Gaussian splats prediction given very sparse unposed images. Comparing with prior works such as Master and Splatter, the method achieves good results without using supervision from depth, loss masks, or metric poses.

As this is a strong submission, the only weakness I could think of is that the paper should include quantitative comparisons on geometry reconstruction results with baselines such as Master and Splatter.

In short, my initial recommandation is **accept**, and I am willing to increase it to strong accept if the quantitative comparisons on geometry reconstruction are provided.
weaknesses: It would be great to include quantitative results for geometry reconstruction, comparing it with Master and Splatter. Currenly only qualitative results are provided in Fig. 5.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - Contribution no. 4 is not a contribution. 
- Multi-view (N>2): Are there any qualitative results for input images more than 2? What would the upper limit on this be given compute constraints? How does the cross-attention in the ViT decoder work for this case since L215 mentions "features from each view interact with those from all other views.."? Also, adding qualitative and quantiative results for N>2 and performance tradeoff cuves would strengthen this method.
- L274: "By default, we utilize the intrinsic token option". This needs to be "Global Intrinsic Embedding - Concat option".
- Table 1, Fig4: Why is Dust3R and Mast3r part of the NVS comparison since they were never trained for novel view synthesis? What was the evaluation protocol for these methods? 
- In Fig4, its possible that the Gaussian structure masks the actual geometry of the scene when comparing to pointcloud outputs of Dust3r. To show geometry, it would be better to either show 3D points or depth maps. To show NVS, it would be better to remove Dust3r output since it is not a fair comparison.
- Discrepancy in metrics:
  * Both MVSplat and pixelSplat report higher PSNR numbers for RealEstate10k and ACID datasets. Since the same train-test split was used, why is there a huge discrepancy in the 'Average' column of the metrics?
  * Table 4: why might be the Splatt3r numbers lower than those reported in the original paper?
- Mentioning the training time and compute requirements would be useful additions to the paper.

Appendix
- Tab. 6 ablation does not provide the complete picture. The NVS metrics are dependent on the accuracy of the 3DGS field. Since the 3DGS parameter head is randomly initialized irrespective of the backbone, its contribution due to backbone initialization might not be as pronounced. A more accurate ablation would be on pose estimation given pure 3D points. 
- L773, L775 typo. "experiences" -> experiments
soundness: 3
strengths: The proposed method is simple and easy to understand. More specifically, the highlighted advantages are:
- A simple VIT approach is competitive to pretrained ViT backbones from Dust3r, and Mast3r when overlap is minimal.
- The method does not require any ground-truth depth (explicit matching loss) during training thus allowing it to scale further. This is an important advantage since simple architectures w/ large datasets is a recipe for improving generalizability.
- Using pretrained Mast3r weights with pure photometric losses on more datasets is a good strategy to scale up training and improve model performance on nvs and relative pose estimation.
summary: The paper proposes a fast feed-forward model for sparse-view 3D reconstruction that is generalizable across scenes. Differently from recent work, the predictions (3DGS parameters) are performed in a common coordinate space directly without requiring any pose info. They perform evaluations on the RealEstate10k and ACID datasets and show improved performance over prior works. Their method also shows zero-shot generalizability to other datasets and in-the-wild scenarios. They also show relative pose estimation using a two-step approach: first with pnp and then with photometric optimization. The performance is benchmarked on similar datasets as well.
weaknesses: The canonical space motivation: The central argument of the paper is that predicting the scene structure in a common coordinate space improves scene geometry and multi-view cohesion compared to local prediction->global fused prediction approach. Although I agree with this statement, the paper does not exploit this advantage:
  * Given two views (which is the only input setting shown in the paper), the method still predicts per-pixel per-view 3DGS fields in a common coordinate (similar to Splatt3r). This is similar to Dust3r pointcloud variants where the pointclouds are predicted in a common coordinate space. I fail to see how this part is different from Dust3r-style prediction. Can you explain how this approach differs from those in the 2-view setting, if at all?
  * The main advantage seems to come from finetuning the Mast3r backbone on more data using photometric loss alone. By directly predicting 3DGS fields, this approach can improve overall model performance. But this orthogonal to the motivation of the method. It would be good to highlight this as one of the main features of this method

This approach is quite similar to Splatt3r although it is concurrent work. The advantage of this approach is that it does not need ground-truth depth during training which is important.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: Please first see comments above. 

**It is necessary to discuss prior works in an appropriate way.** I will give a reject first but I can raise the score to 5 if the author can differentiate their work from the missing related works and rephrase their contribution. 

**Besides, I strongly require the authors to perform experiment using the same training dataset with previous works.** For example, if you want to compare with MVSplat, you should only train on RealEstate10K+ACID. This is the correct way of doing scientific research. Otherwise, we cannot tell whether the performance gain comes from better model design or using larger and high-quality data (and also your MAST3R weight initialization should be ablated). The additionally used datasets in this paper are all from available resources and there is no need for data curation. It is different from LLM research where data collection and curation and be their contribution, and, thus, it is acceptable to directly compare LLMs trained using different data.

I can raise the score to 6 or higher scores if the authors performs the asked three experiments (weakness 3) with promising results.
soundness: 3
strengths: - The paper proposes a simple and straightforward method.
- The paper is well-written and is easy to follow.
- The method shows a good performance, which is even better than previous pose-aware methods, e.g. MVSplat.
summary: This paper proposes a feed-forward generalizable GS method from sparse and unposed images. It encodes images (with known intrinsics) into feature tokens and use a decoder with cross-attention layers for cross-view feature fusion. Then the gaussians are predicted in a canonical space. The proposed method is evaluated on RealEstate10K/ACID and tested generalization on DTU.
weaknesses: 1. This paper is technically sound. However, basic literatures in this field are not discussed and the claimed novelty have already been studied in prior works. I don't understand why the authors ignore these works when the content and techniques are closely related. Please see details below.
- PF-LRM [1] and LEAP [2] are the two earliest works for pose-free reconstruction, but none of them are mentioned. Almost all claimed novelty of the submssion, e.g. canonicalization, PnP pose estimation, using cross-attention for cross-view feature fusion, have already been studied in these two works (also true for DUST3R). The only difference is whether using 3DGS or NeRF as the 3D representation (which is actually also studied in GS-LRM [3] and GRM [4], see below). However, I don't think this is a solid contribution for a publication.
- Besides, the architecture of the proposed work is almost the same with GS-LRM [3] with only two differences of design choice: (1) whether using a ViT encoder or directly tokenizing the images; (2) whether using cross-attention or using concatenation and self-attention for feature fusion. This is also true for GRM [4].

The paper is a direct combination of these previous works (pose-free feed-forward NVS from sparse-views + feed-forward 3DGS prediction).

- Moreover, methods that use latent 3D representations [5,6,7] (SRT is the first pose-free NVS method), reconstruction using sparse-view pose estimation methods [8,9,10,11] and more scene-specific/optimization-based pose-free methods [12] should be discussed.

[1] Wang, Peng et al. “PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction.” ICLR 2024.
[2] Jiang, Hanwen et al. “LEAP: Liberate Sparse-view 3D Modeling from Camera Poses.” ICLR 2024.
[3] Zhang, Kai et al. “GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting.” ECCV 2024.
[4] Xu, Yinghao et al. “GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation.” ECCV 2024.
[5] Sajjadi, Mehdi SM, et al. "Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations." CVPR 2022.
[6] Kani, Bharath Raj Nagoor, et al. "UpFusion: Novel View Diffusion from Unposed Sparse View Observations." ECCV 2024.
[7] Xu, Chao, et al. "Sparp: Fast 3d object reconstruction and pose estimation from sparse views." ECCV 2024.
[8] Zhang, Jason Y., Deva Ramanan, and Shubham Tulsiani. "Relpose: Predicting probabilistic relative rotation for single objects in the wild." ECCV 2022.
[9] Zhang, Jason Y., et al. "Cameras as rays: Pose estimation via ray diffusion." ICLR 2024.
[10] Jiang, Hanwen, et al. "Few-view object reconstruction with unknown categories and camera poses." 3DV 2024.
[11] Lin, Amy, et al. "Relpose++: Recovering 6d poses from sparse-view observations." 3DV 2024.
[12] Truong, Prune, et al. "Sparf: Neural radiance fields from sparse and noisy poses." CVPR 2023.

2. Moreover, there some other weaknesses as follows.
- Inappropriate baselines. DUST3R is not designed for NVS but for reconstruction with point cloud. Evaluating NVS performance on DUST3R is misleading. This is also the same for MAST3R/Splatt3R. If you want to compare with these works, please compare the geometry quality, i.e. point cloud error, rather than NVS performance. Moreover, DUST3R and its following works don't require camera intrinsics.
- Unfair comparison. The proposed method is trained on a joint set of RealEstate10K, DL3DV and ACID, while all the baselines are trained on a different training dataset. Moreover, the model uses the weight initialization of models trained on large dataset (MAST3R/DUST3R/Croco).

3. As there are not a lot directly comparable works on scene-level, I require the authors to include additional five experiment results.
- On object-level, train the proposed model using the same data with the previously mentioned pose-free works and compare their performance. This also verifies whether MAST3R weight initialization generalize to object-level data.
- On scene-level, train with only RealEstate10K+ACID for comparing with MVSplat.
- On scene-level, train a variant of the proposed model, which is also conditioned on the camera poses of inputs (using the plucker ray representation rather than 6D pose representation). If the performance gap is small enough, the model has a strong capability of correlating the two images with the missing poses.
- On scene-level, ablate the weight initialization by training with no weight initialization (from MAST3R/DUST3R/Croco) using the current training set (ACID+RealEstate10K+DL3DV). This experiment ablates whether the pose-free inference capability comes from initialization or your model learning process. Please include results for both pose-free and pose-conditioned variants of your model.

4. Besides, the authors should notify that during training, the method still requires camera poses for rendering. 

5. Camera intrinsics are still needed during inference. The proposed way of merging camera intrinsic information is the same as LRM/PF-LRM, but no discussion.

6. The method is only tested with two views, while the author claim "sparse-view". The authors should perform experiments with more input views, e.g. up to 6-8 views, to support the claim. Otherwise, you should rename the paper as "unposed two images".

Meta-review: This paper introduces a method for reconstructing 3DGS from sparse and unposed images using only photometric constraints for training. The technique is then used for NVS and pose estimation.

The main strengths of this paper are as below:
- The paper is clearly written and easy to follow.
- The paper is innovative, provides a detailed discussion of previous work, and is well-motivated.
- The paper has presented how to train the 3DGS Reconstruction Network only with photometric constraints and without the geometric ground truth, which enables the utilization of a wider range of datasets and improves the reconstruction performance.
- The paper provides comprehensive experiments that fully demonstrate the sophistication of the proposed method.

Needing more experiments and analyses were raised by the reviewers. Please revise the paper according to the discussions before submitting the final version.

Decision: Accept (Oral)