No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images
Authors: Botao Ye
Sifei Liu
Haofei Xu
Xueting Li
Marc Pollefeys
Ming-Hsuan Yang
Songyou Peng
Venue: ICLR 2025 Oral (2025)
Keywords: 3D Gaussian Splatting
Pose Free
Pose Estimation
Novel View Synthesis
3D Reconstruction
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: NoPoSplat is a novel feed-forward model that reconstructs scenes from unposed images by predicting Gaussians in a canonical space, demonstrating superior performance in both novel view synthesis and pose estimation.

Abstract: We introduce NoPoSplat, a feed-forward model capable of reconstructing 3D scenes parameterized by 3D Gaussians from unposed sparse multi-view images. Our model, trained exclusively with photometric loss, achieves real-time 3D Gaussian reconstruction during inference. To eliminate the need for accurate pose input during reconstruction, we anchor one input view's local camera coordinates as the canonical space and train the network to predict Gaussian primitives for all views within this space. This approach obviates the need to transform Gaussian primitives from local coordinates into a global coordinate system, thus avoiding errors associated with per-frame Gaussians and pose estimation. To resolve scale ambiguity, we design and compare various intrinsic embedding methods, ultimately opting to convert camera intrinsics into a token embedding and concatenate it with image tokens as input to the model, enabling accurate scene scale prediction. We utilize the reconstructed 3D Gaussians for novel view synthesis and pose estimation tasks and propose a two-stage coarse-to-fine pipeline for accurate pose estimation. Experimental results demonstrate that our pose-free approach can achieve superior novel view synthesis quality compared to pose-required methods, particularly in scenarios with limited input image overlap. For pose estimation, our method, trained without ground truth depth or explicit matching loss, significantly outperforms the state-of-the-art methods with substantial improvements. This work makes significant advances in pose-free generalizable 3D reconstruction and demonstrates its applicability to real-world scenarios. Code and trained models are available at https://noposplat.github.io/.

Published as a conference paper at ICLR 2025
No POSE, No PROBLEM: SURPRISINGLY SIMPLE 3D GAUSSIAN SPLATS FROM SPARSE UNPOSED IMAGES
Botao Ye!.2
Sifei Liu
Haofei Xu'
Xueting Li
Mare Pollefeys 1.4
Ming-Hsuan Yang
Songyou Peng"
ETH Zurich
ETH AI Center
NNVDII
*Microsoft
UC Merced
ht httpss://nposppll github.
Dhie
NoPoSplat
Downstream
Tasks
Unposed Images
3D Gauasians in Canonical Space
Pose Estimation
Novel View Synthesis
NoPoSplat. Given sparse unposed images, our method reconstructs 3D Gaussians of different views in canonical space using a feed-forward network. The resulting 3D Gaussians can be utilized for accurate relative camera pose estimation and high-quality novel view synthesis. The input images for illustration are extracted from a Sora- cora-generated video.
Figure 1:
ABSTRACT
We introduce NoPoSplat, a feed-forward model capable of reconstructing 3D scenes pa- rameterized by 3D Gaussians from unposed sparse multi-view images. Our model, trained exclusively with photometric loss, achieves real-time 3D Gaussian reconstruction during inference. To eliminate the need for accurate pose input during reconstruction, we anchor one input view's 's local camera coordinates as the canonical space and train the network to predict Gaussian primitives for all views within this space. This approach obviates the need to transform Gaussian primitives from local coordinates into global coordinate sys- tem, thus avoiding errors associated with per-frame Gaussians and pose estimation. To resolve scale ambiguity, ultimately opting to convert camera intrinsics into token embedding and concatenate it with image tokens input to the model, enabling accurate scene scale prediction. We uti- lize the reconstructed 3D Gaussians for novel view synthesis and pose estimation tasks and propose two-stage coarse-to-fine pipeline for accurate pose estimation. Experimental re- sults demonstrate that our pose-free approach can achieve superior novel view synthesis quality compared to pose-required methods, particularly in scenarios with limited input image overlap. For pose estimation, our method, trained without ground truth depth or explicit matching loss, significantly outperforms the state-of-the-arr art methods with sub- stantial improvements. This work makes significant advances in pose-free generalizable 3D reconstruction and demonstrates its applicability to real-world scenarios. Code and trained models are available our project page.
design and compare various intrinsic embedding methods,
INTRODUCTION
We address the problem of reconstructing 3D scene parameterized by 3D Gaussians from unposed sparse-view images (as few as two) using a feed-forward network. While current SOTA generaliz- able 3D reconstruction methods (Charatan et al., 2024; Chen et al., 2024), which aim to predict 3D radiance fields using feed-forward networks, can achieve photorealistic results without per-scene optimization, they require accurate camera poses of input views as input to the network. These
Songyyu Peng is currently at Google DeepMind, with this work mainly done ETH Zarich.

---

Published as a conference paper at ICLR 2025
with geometry prior
Geometry-free
VIT
WarD
(5) Previous pase nose-reeuired local Gaussian prediction
(b) Our canonical space Gaussian prediction
Figure 2: Comparison with pose-required sparse-view 3D Gaussian splatting pipeline. Previ- ous methods first generate Gaussians each local camera coordinate system and then transform them into world coordinate system using camera poses. In contrast, NoPoSplat directly outputs 3D Gaussians of all views in a canonical space, facilitating a more coherent fusion of multi-view geometric content (see Sec. 3.3 for details). Furthermore, our backbone does not incorporate geom- etry priors that rely on substantial image overlap (such as epipolar geometry in pixelSplat (Charatan et al., 2024) cost volume in MVSplat (Chen et al., 2024)). Consequently, NoPoSplat demonstrates better geometric detail preservation when the overlap between input views is limited.
poses are typically obtained from dense videos using structure mrructue-from--oe motoon (SfM) methods, such as COLMAP (Schonberger & Frahm, 2016). This requirement is impractical for real-world applica- tions, as these methods necessitate poses from dense videos even if only two frames are used for 3D reconstruction. Furthermore, relying on off-the-shelf pose estimation methods increases inference time and can fail in textureless areas or images without sufficient overlap.
Recent methods (Chen & Lee, 2023; Smith et al., 2023; Hong et al., 2024a) aim to address this challenge by integrating pose estimation and 3D scene reconstruction into single pipeline. How- ever, the quality of novel view renderings from these methods lags behind SOTA approaches that rely on known camera poses (Chen et al., 2024). The performance gap stems from their sequential process of alternating between pose estimation and scene reconstruction. Errors in pose estimation degrade the reconstruction, which in turn leads to further inaccuracies in pose estimation, creating a compounding effect. In this work, we demonstrate that reconstructing the scene entirely without relying on camera poses is feasible, thereby eliminating the need for pose estimation. We achieve this by directly predicting scene Gaussians in canonical space, inspired by the success of the recent 3D point cloud reconstruction methods (Wang et al., 2024b; Leroy et al., 2024). However, unlike DUSt3R, we show that the generalizable reconstruction network can be trained with photometric loss only without ground truth depth information and thus can leverage more widely available video data (Zhou et al., 2018; Liu et al., 2021; Ling et al., 2024).
Specifically, as shown in Fig. (b), we anchor the local camera coordinate of the first view as the canonical space and predict the Gaussian primitives (Kerbl et al., 2023) for all input views within this space. Consequently, the output Gaussians will be aligned to this canonical space. This contrasts with previous methods (Charatan et al., 2024; Chen et al., 2024), as illustrated in Fig. (a), where Gaussian primitives are first predicted in each local coordinate system and then transformed to the world coordinate using the camera pose and fused together. Compared to the transform-ther then-fuse pipeline, we require the network to learn the fusion of different views directly within the canonical space, thereby eliminating misalignments introduced by explicit transformations (see Fig. 5).
Although the proposed pose-free pipeline is simple and promising, we observe significant scale misalignment in the rendered novel views compared to the ground truth (see Fig. 8), i.e., the scene scale ambiguity issue. Upon analyzing the image projection process, we find that the camera's focal length is critical resolving this scale ambiguity. This because the model reconstructs the scene solely based on the image appearance, which is influenced by the focal length. Without incorporating it, the model struggles to recover the scene at the correct scale. To address this issue, we design and compare three different methods for embedding camera intrinsics and find that simply converting the intrinsic parameters into feature token and concatenating it with the input image tokens enables the network to predict the scene of more reasonable scale, yielding the best performance.
Once the 3D Gaussians are reconstructed in the canonical space, we leverage for both novel view synthesis (NVS) and pose estimation. For pose estimation, we introduce two-stage pipeline: first, we obtain an initial pose estimate by applying the PnP algorithm (Hartley & Zisserman, 2003) to the
2

---

Published as a conference paper at ICLR 2025
Gaussian centers. This rough estimate is then refined by rendering the scene at the estimated pose and optimizing the alignment with the imput view using photometric loss.
Extensive experimental results demonstrate that method performs impressively in both NVS and pose estimation tasks. For NVS, we show for the first time that, when trained on the same dataset un- der the same settings, pose-free method can outperform pose-dependent methods, especially when the overlap between the two input images is small. In terms of pose estimation, our method signif- icantly outperforms prior SOTA across multiple benchmarks. Additionally, NoPoSplat generalizes well to out-of-distribution data. Since our method does not require camera poses for input images, it can be applied to user-provided images reconstruct the underlying 3D scene and render novel views. To illustrate this, we apply our model to sparse image pairs captured with mobile phones, as well as to sparse frames extracted from videos generated by Sora (OpenAI, 2024).
The main contributions of this work are:
We propose NoPoSplat, feed-forward network that reconstructs 3D scenes parameterized by 3D Gaussians from unposed sparse-vies view inputs, and demonstrate that it can be trained using photometric loss alone.
We investigate the scale ambiguity issue of the reconstructed Gaussians, and solve this problem by introducing a camera intrinsic token embedding.
We design a two-stage pipeline that estimates accurate relative camera poses using the recon- structed Gaussian field.
2 RELATED WORK
Generalizable 3D Reconstruction and View Synthesis. NeRF (Mildenhall et al., 2020) and 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) have significantly advanced 3D reconstruction and novel view synthesis. However, these methods typically require dense posed images (e.g.. hundreds) as input and minutes to hours of per-scene optimization. even with improved techniques (Chen et al., 2022; Fridovich- Keil al., 2023; Müller al., 2022). This limits their practical applications. To ad- dress these limitations, recent approaches focus on generalizable 3D reconstruction and novel view synthesis from sparse inputs (Yu al., 2021; Wang al., 2021a; eu al., 2024b; Johari et al., 2022; Charatan et al., 2024; Chen et al., 2024). They typically incorporate task-specific backbones that leverage geometric information to enhance scene reconstruction. For instance, MVSNeRF (Chen et al., 2021) and MuRF (Xu et al., 2024b) build cost volumes to aggregate multi-view information, while pixelSplat (Charatan et al., 2024) employs epipolar geometry for improved depth estimation. However, these geometric operations often require camera pose input and sufficient camera pose overlap among input views. In contrast, our network is based solely on a vision transformer (Doso- vitskiy et al., 2021) without any geometric priors, making it pose-free and more effective in handling scenes with large camera baselines. Some recent approachs (Xu et al., 2024c; Tang et al., 2024; Zhang al., 2025) also employ a simple backbone, but still require the camera pose as input.
Pose-Free 3D Scene Reconstruction. Classical NeRF or 3DGS-based methods require accurate camera poses of input images, typically obtained through Structure from Motion (SfM) methods like COLMAP (Schonberger & Frahm, 2016), which complicates the overall process. Recent per- scene optimization works (Wang et al., 2021b; Lin et al., 2021; Chng et al., 2022; Truong et al., 2023) jointly optimize camera poses and neural scene representations, but they still require rough pose initialization or are limited to small motions. Others (Bian al., 2023; Fu et al., 2024) adopt incremental approaches from (Zhu et al., 2022: 2024; Matsuki et al., 2024), but they only allow image/video sequences input. Moreover, for generalizable sparse-view methods, requiring cam- era poses during inference presents significant challenges, as these poses are often unavailable in real-world applications during testing. Although sparse-view pose estimation methods (Wang al., 2024b; Zhang et al., 2022: Lin et al., 2023; Zhang et al., 2024; Jiang et al., 2024a) can be used, they are prone to failure in textureless regions or when images lack sufficient overlap. Additionally, the initial pose estimation stage introduces noise into the reconstruction process. Some recent general- izable pose-free novel view synthesis methods (Chen Lee, 2023; Smith et al., 2023; Hong et al., 2024a; Xu et al., 2024a) attempt to address this but typically break the task into two stages: first esti- mate camera poses, then construct the scene representation. This two-stage process still lags behind pose-required methods because the initial pose estimation introduces noise, degrading reconstruc- tion quality. Some methods (Sajjadi et al., 2022; Nagoor Kani et al., 2024) transform images into latent representations. However, these methods exhibit poor performance and cannot be used for
3

---

Published as a conference paper at ICLR 2025
Cansecoes Canierr gt"
VIT Encoder
VIT Decoder
Gaussian
g! pllnl
Direct
Splanting
weighs
Crossaatstiie
Gaussian Coster Head
ViT Encoder
ViI Decoder
Jr
Novel View
313 Grinsiars
Canonical Space
Conorimation
Intriniio
Figure 3: Overview of NoPoSplat. We directly predict Gaussians in canonical space from feed- forward network to represent the underlying 3D scene from the unposed sparse images. For simplicity, we use a two-view setup as example, and the RGB shortcut is omitted from the figure.
pose estimation, limiting their applications. LEAP (Jiang et al., 2024b) and PF-LRM (Wang al., 2024a) share a similar approach of lifting multi-view unposed images to a NeRF representation. However, their low rendering efficiency and resolution constraints hinder training effectiveness and limit scalability for complex scene-level reconstruction. In contrast, our method completely elim- inates camera poses by directly predicting 3D Gaussians canonical space, avoiding potential noise in pose estimation and achieving better scene reconstruction.
One concurrent work, Splatt3R (Smart al., 2024). also predicts Gaussians global coordinate system but relies on the frozen MASt3R (Leroy et al., 2024) model for Gaussian centers. This is unsuitable for novel view synthesis. as MASt3R struggles views smoothly. Moreover, in contrast to us, Splan13 requires ground truth depth during training, so it cannot utilize large-scale video data without depths or ground truth metric camera poses. An- other concurrent work, ReconX (Liu et al., 2024), also reconstructs 3D Gaussians from unposed images. However, this approach relies on per-scene optimization and multi-stage inference, making it significantly less efficient compared to feed-forward pipeline.
merge scene content from different
3 METHOD
3.1 PROBLEM FORMULATION
Our method takes as input sparse unposed multi-view images and corresponding camera intrinsic parameters {I", k= ) where V is the number of input views, and learn feed-forward network fe with learnable parameters 6. The network maps the input unposed images to 3D Gaussians in canonical 3D space, representing the underlying scene geometry and appearance. Formally, we aim to learn the following mapping:
So ((",,>))
{U ry. 8/, cj) ( HxW HxW
where the right side represents the Gaussian parameters (Kerbl et al., 2023). Specifically, we have the center position μ 3 opacity E E. rotation factor in quaternion r R2 scale R& and spherical harmonics (SH) e E I* with & degrees of freedom. Note that, in line with common practice pose-free scene reconstruction methods (Hong al., 2024a; Fu et al., 2024; Smith al., 2023; Chen & Lee, 2023), we assume having camera intrinsic parameters k as input, as they are generally available from modern devices (Arnold et al., 2022).
By training on large-scale datasets, our method can generalize novel scenes without any opti- mization. The resulting 3D Gaussians in the canonical space directly enable two tasks: a) novel view synthesis given the target camera transformation relative to the first input view, and b) relative pose estimation among different input views. Next, we will introduce our overall pipeline.
3.2 PIPELINE
Our method, illustrated in Fig. 3, comprises three main components: an encoder, a decoder, and Gaussian parameter prediction heads. Both encoder and decoder utilize pure Vision Transformer (ViT) structures, without injecting any geometric priors (e.g. epipolar constraints employed in pix- elSplat (Charatan et al., 2024), cost volumes in MVSplat (Chen et al., 2024)). Interestingly, demonstrate in Sec. 4 that such a simple ViT network shows competitive or superior performance those dedicated backbones incorporating these geometric priors, especially in scenarios with
4

---

Published as a conference paper at ICLR 2025
limited content overlap between input views. This advantage stems from the fact that such geometric priors typically necessitate substantial overlap between imput cameras to be effective.
VIT Encoder and Decoder. The RGB images are patchified and flattened into sequences of image tokens, and then concatenated with an intrinsic token (details in Sec. 3.4). The concatenated tokens from each view are then fed into a ViT (Dosovitskiy et al., 2021) encoder separately. The encoder shares the same weights for different views. Next, the output features from the encoder are fed into a ViT decoder module, where features from each view interact with those from all other views through cross-attention layers in each attention block, facilitating multi-view information integration.
Gaussian Parameter Prediction Heads. To predict the Gaussian parameters, we employ two pre- diction heads based on the DPT architecture (Ranftl et al., 2021). The first head focuses on predict- ing the Gaussian center positions and utilizes features extracted exclusively from the transformer decoder. The second head predicts the remaining Gaussian parameters and, in addition the ViT decoder features, also takes the RGB image as input. Such RGB image shorteut ensures the direct flow of texture information, which is crucial for capturing fine texture details in 3D reconstruction. This approach compensates for the high-level features output by the ViT decoder, downsampled by a factor of 16, which are predominantly semantic and lack detailed structural information. With these prediction heads in place, we now analyze how our method differs from previous approaches in terms of the output Gaussian space and the advantages this brings.
3.3 ANALYSIS OF THE OUTPUT GAUSSIAN SPACE
While our method shares a similar spirit with previous works (Charatan et al., 2024; Zheng et al., 2024: Szymanowicz et al., 2024) in predicting pixelwise Gaussians for input images, we differ in the output Gaussian space. In this section, we first discuss the local-to-global Gaussian space in prior methods and its inherent limitations, and introduce our canonical Gaussian space.
Baseline: Local-to-Global Gaussian Space. Previous methods first predict the corresponding depth of each pixel, then lift the predicted Gaussian parameters to a Gaussian primitive in the lo- cal coordinate system of each individual frame using the predicted depth and the camera intrinsics. These local Gaussians are then transformed into a world coordinate system using the given camera poses R t²] for each input view. Finally, all transformed Gaussians are fused to represent the underlying scene.
However, this strategy has two main issues: a) Transforming Gaussians from local to world coordi- nates requires accurate camera poses, which are difficult to obtain in real-world scenarios with sparse input views. b) The transform-then-fuss. method struggles to combine 3D Gaussians from different views into a cohesive global representation, especially when the overlap among input views is small, or when generalizing to -itt-f-distributiinn data (as shown in Fig. 5 and Fig. 6).
Proposed: Canonical Gaussian Space. In contrast, we directly output Gaussians of different views in canonical coordinate system. Specifically, coordinate system. Therefore, the camera pose for the first input view is IU O]. where U is the unit/identity matrix for rotation, and 0 is the zero translation vector. The network outputs Gaussians under this canonical space for all input views. Formally, for each input view, we predict the set {1=) -1 1- cy O.j: $3 }. where the superscript e -> denotes that the Gaussian parameters corresponding to pixel Pj in view v, are under the local camera coordinate system of view 1.
anchor the first input view as the global reference
Predicting directly under the canonical space offers several benefits. First, the network learns to fuse different views directly within the canonical space, eliminating the need for camera poses. Second, bypassing the transform-then-fe unlocks the application of pose estimation for input unposed views.
step results a cohesive global representation, which further
3.4 CAMERA INTRINSICS EMBEDDING
As discussed in Eq. 1, our network inputs also include the camera intrinsics k of each input view. This is required to resolve the inherent scale misalignment and provide essential geometric infor- mation that improves 3D reconstruction quality (cf. "No Intrinsics" in Fig. 8 and Tab. 5). Although intrinsic embedding has been employed in previous NeRF-basedd methods, both pose-free (Wang et al., 2024a) and pose-required (Hong et al., 2024b), its necessity for generalizable pose-free free Gaus- sian Splatting prediction and optimal embedding approaches remain unexplored. Therefore, we introduce three different encoding strategies for injecting camera intrinsics into model.
5

---

Published as a conference paper at ICLR 2025
Global Intrinsic Embedding Addition. A straightforward strategy is to feed camera intrinsics = [fa= fy: Cz, Cy] into a linear layer to obtain a global feature. This feature is broadcast and added to the RGB image features after the patch embedding layer of the ViT.
Global Intrinsic Embedding Concat. After obtaining the global feature, we instead treat it as an additional intrinsic token, and concatenate it with all image tokens.
Dense Intrinsic Embedding. For each pixel pj in the input view, we can obtain the camera ray di- rection as Ppj where K is the matrix form of k. These per-pixel camera rays are then converted using spherical harmonics to higher-dimension features and concatenated with the RGB image as the network input. Note that the pixel-wise embedding can be viewed as a simplification of the widely-used Plücker ray embedding (Xu et al., 2023; Tang et al., 2024), as does not require cam- era extrinsic information.
By default, we adopt the "global intrinsic embedding concat" option since it is not only an elegant way to inject camera intrinsic into the network but also yields the best performance (see Tab. 5).
3.5 TRAINING AND INFERENCE
Training. The predicted 3D Gaussians are rendered at novel viewpoints using their corresponding ground-truth camera poses. Our network is end-to-end trained using ground truth target RGB im- ages as supervision. Following Chen et al. (2024), we also use a linear combination of MSE and LPIPS (Zhang et al., 2018) loss with weights of and 0.05, respectively.
Relative Pose Estimation. As mentioned in Sec. 3.3, since our 3D Gaussians are in the canonical space, they can be directly used for relative pose estimation. To facilitate efficient pose estimation, we propose a two-step approach. First, we estimate the initial related camera pose of the input two views using the PnP algorithm (Hartley & Zisserman, 2003) with RANSAC (Fischler & Bolles, 1981), given the Gaussian centers of the output Gaussians world coordinates. This step is highly efficient and done in milliseconds. Next, while keeping Gaussian parameters frozen, refine the initial pose from the first step by optimizing the same photometric losses used for model training, along with the structural part of the SSIM loss (Wang et al., 2004). During the optimization, we calculate the camera Jacobian to reduce the computational overhead associated with automatic dif- ferentiation and decrease optimization time as in Matsuki al. (2024).
Svaluation-Time Pose Alignment. Given unposed image pairs, our method learns to reconstruct a plausible 3D scene that aligns with the given inputs. However, 3D scene reconstruction with just two input views is inherently ambiguous as many different scenes can produce the same two images. As a result, though the scene generated by our method successfully explains the input views, it might not be exactly the same as the ground truth scene in the validation dataset. Thus, to fairly compare with other baselines, especially ones that utilize ground truth camera poses(Du et al., 2023; Charatan et al., 2024). we follow previous pose-free works (Wang et al., 2021b; Fan et al., 2024) and optimize the camera pose for the target view. Specifically, for each evaluation sample, we first reconstruct 3D Gaussians using the proposed method. We then freeze the Gaussians and optimize the target camera pose such that the rendered image from the target view closely matches the ground truth image. It is important to note that this optimization is solely for evaluation purposes and is not required when applying our method in real-world scenarios (e.g., Fig. 7).
EXPERIMENTS
Datasets. To evaluate novel view synthesis, we follow the setting in (Charatan et al., 2024; Chen et al., 2024) and train and evaluate our method on RealEstate10k (RE10K) (Zhou et al., 2018) and ACID (Liu et al., 2021) datasets separately. RE10K primarily contains indoor real estate videos, while ACID features nature scenes captured by aerial drones. Both include camera poses calculated using COLMAP (Schonberger & Frahm, 2016). We adhere to the official train-test split as in pre- vious works (Charatan et al., 2024; Hong al., 2024a). To further scale up our model (denoted as Ours*), ) we also combine REI0K with DL3DV (Ling et al., 2024), which is an outdoor dataset containing 10K videos, which includes wider variety of camera motion patterns than RE10K.
To assess the method's ' capability in handling input images with varying camera overlaps, we gen- erate input pairs for evaluation that are categorized based on the ratio of image overlap: small (5% 30%), medium (30%- 55%), and large (55%- 80%), determined using SOTA dense feature match- ing method, RoMA (Edstedt et al., 2024). More details are provided in the appendix.

---

Published as a conference paper at ICLR 2025
Table I: Novel view synthesis performance comparison on the RealEstate10k (Zhou et al., 2018) dataset. Our method largely outperforms previous pose-free methods on all overlap settings, and even outperforms SOTA pose-required methods, especially when the overlap is small.
Small
Medium
Large
Average
Method
PSNR
SSIM
LPIPS.
PSNR
SSIM
LPIPS|
PSNR*
SSIM
LPIPS.
PSNR
SSIM
LPIPS!
Required
Pose-
pixelSplat
PixeINeRF
AttnRend
20.263
18.417
19.151
0.663
0.601
0.717
0.368
0.266
0.526
23.711
22.532
19.930
0.809
0.763
0.632
0.269
0,480
0.181
27.151
25.897
20.869
0.879
0.845
0.639
0.122
0.186
0.458
23,848
22.064
19.824
0.806
0.626
0.762
0.185
0.485
0.269
MVSplat
20.353
0.724
0.250
23.778
0.812
0.173
27.408
0.884
0.116
23.977
0.811
0.176
Pose- Free
Splat33
14.352
0.475
0.472
15.529
0.502
0.425
15.817
0.483
0.421
15.318
0.490
0,436
CoPuNeRF
17.393
0.585
0.462
18.813
0.616
0.392
20.464
0.652
0.318
18.938
0.619
0.388
22.514
0.784
0.210
24.899
0.839
0.160
27.411
0.883
0.119
25.033
0.838
0.160
Table 2: Novel view synthesis performance comparison on the ACID (Liu et al., 2021) dataset.
Small
Medium
Large
Average
Method
PSNR
SSIM
LPIPS)
PSNR
SSIM
LPIPS)
PSNR1
SSIM
LPIPS.
PSNR
SSIM
LPIPS|
Pose- Required
pixeINeRF
AunRend
19.376
20.942
0.535
0.616
0.398
0.564
20.339
24.004
0.720
0.561
0.301
0.537
27.117
20.826
0.808
0.576
0.509
0.207
24.475
20.323
0.730
0.56]
0.287
0.533
pixelSplat
22.053
0.654
0.285
25,460
0.776
0.198
28.426
0.853
0.140
25.819
0.779
0.195
MVSplat
21.392
0.639
0.290
25.
0.770
0.199
28.388
0.852
0.139
25.512
0.773
0.196
Pose-Free Ours
CoRF
23.087
18.651
0.551
0.685
0.258
0.485
25.624
20.654
0.595
0.777
0.193
0.418
28.043
22.654
0.652
0.841
0.343
0.144
25.961
20.950
0.606
0.781
0.189
0.406
Furthermore, for zero-shot generalization, we also test on DTU (Jensen al., 2014) (object-centric scenes), ScanNet (Dai et al., 2017) and ScanNet++ (Yeshwanth et al., 2023) (indoor scenes with different camera motion and scene types from the RE10K). We also demonstrate our approach on in-the-wild mobile phone capture, and images generated by text-to-video model (OpenAI, 2024).
Evaluation Metrics. We evaluate novel view synthesis with the commonly used metrics: PSNR, SSIM, and LPIPS (Zhang et al., 2018). For pose estimation, we report the area under the cumulative pose error curve (AUC) with thresholds of 5 10 20 (Sarlin et al., 2020; Edstedt al., 2024).
Baselines. We compare against SOTA sparse-view generalizable methods on novel view synthesis: 1) Pose-required: pixelNeRF (Yu et al., 2021), AttnRend (Du et al., 2023), pixelSplat (Charatan et al., 2024), and MVSplat (Chen et al., 2024); 2) Pose-free: DUSI3R (Wang et al., 2024b), MAS13R (Leroy et al., 2024), Splatt3R (Smart al., 2024), CoPoNeRF (Hong et al., 2024a), and RoMa (Edstedt et al., 2024). For relative pose estimation, we also compare against methods in 2).
Implementation details. We use PyTorch, and the encoder is a vanilla ViT-large model with a patch size of 16, and the decoder is ViT-base. We initialize the encoder/decoder and Gaussian center head with the weights from MASt3R, while the remaining layers are initialized randomly. Note that, shown in Tab. 7, our method can also be trained with only RGB wuphrot pre-trained weight from MASt3R-and still achieve similar performance. We train models at two different resolutions, 256 256 and 512 x 512. For fair comparison with bascline models, report all quantitative results and baseline comparisons under 256 x 256. However, qualitative results for the 512 512 model are presented in the supplementary video and Fig. 7. Additional details on model weight initialization and training resolution can be found in the appendix.
4.1 EXPERIMENTAL RESULTS AND ANALYSIS
Novel View Synthesis. As demonstrated in Tab. 1, Tab. 2, and Fig. 4, NoPoSplat significantly outperforms all SOTA pose-free approaches. Note that DUS:3R (and MAS(3 struggle to fuse input views coherently due to their reliance on per-pixel depth loss, limitation Splatt3R also inherits from its frozen MASt3R module. On the other hand, we achieve competitive performance over SOTA pose-required methods (Charatan et al., 2024; Chen et al., 2024), and even outperform them when the overlap between input images is small, as shown in Fig.4 4 first row (the left side of Fig. displays the overlap ratio). This clearly shows the advantages of our model's 3D Gaussians prediction in the canonical space over baseline methods transform- -heen-fu--ee---eeeeee strategy, as discussed in Sec. 3.3.
Relative Pose Estimation. The proposed method can be applied to pose estimation between input views on three diverse datasets. Our method is trained either on RE10K (denoted as Ours) or a combination of RE10K and DL3DV (denoted as Ours*). Tab. 3 shows that the performance consistently improves when scaling up training with DL3DV involved. This can be attributed to the greater variety of camera motions in DL3DV over in RE10K. It is worth noting that method
7

---

Published as a conference paper at ICLR 2025
Ref.
AttaRend
pixelSplat
MVSplat
CoPoNeRF
Splatr3R
Ours
GT
Small
Medium
Large
Small
Figure 4: Qualitative comparison on RE10K (top three rows) and ACID (bottom row). Com- pared to baselines, we obtain: 1) more coherent fusion from input views, 2) superior reconstruction from limited image overlap. 3) enhanced geometry reconstruction in non-overlapping regions.
Table 3: Pose estimation performance AUC with various thresholds on RE10K, ACID, and ScanNet-1500 (Dai et al., 2017; Sarlin et al., 2020). Our method achieves the best results across all datasets. Notably, our model not trained on ACID or ScanNet. Furthermore, our method does not require explicit matching loss during training, meaning ground truth depth is necessary.
RE10k
ACID
ScunNet-1500
Method
5
10a
20"
5
10
20"
5a
10
20"
CoPoNeRF
0.161
0.362
0.575
0.078
0.216
0.398
DUS13R
0.301
0.495
0.657
0.166
0,304
0.437
0.221
0.437
0.636
MASER
0.561
0.396
0.541
0.359
RoMa
0.546
0.698
0.797
0.463
0.588
0.689
0.270
0.492
0.673
Ours (RE1OK)
0.672
0.792
0.869
0.454
0.591
0.709
0.264
0.473
0.655
Ours (REDOR+DL3DV)
0.806
0.486
0.617
0.318
0.538
Table -fittf-dist performance comparison. Our method shows superior performance when zero-shot evaluation on DTU and ScanNet++ using the model solely trained on RE10k.
Training
DTU
ScanNet++
PSNR 1
SSIM f
LPIPS
PSNR
SSIM 1
LPIPS
ScanNet++
Splatt3R
11.628
0.325
0.499
13.270
0.507
0.445
pixelSplat
11.551
0.321
0.633
18.434
0.719
0,277
RcalEstate10K
MVSplat
13.929
0.474
0.385
17.125
0.686
0.297
Ours
17.899
0.629
0.279
22.136
0.798
0,232
even shows superior zero-shot performance on ACID and ScanNet-150k even better than the SOTA method RoMa that is trained ScanNet. This indicates not only the efficacy of pose estimation approach, but also the quality our output 3D geometry. The next part verifies this point.
Geometry Reconstruction. Our method also outputs noticeably better 3D Gaussians and depths over SOTA pose-required methods, as shown in Fig. 5. Looking closely, MVSplat not only suffers from the misalignment in the intersection regions of two input images (indicated by blue arrows), but also distortions or incorrect geometry in regions without sufficient overlap (indicated by magenta arrows). These issues are largely due the noises introduced in their transform-then-fiee pipeline. Our method directly predicts Gaussians in the canonical space, which faithfully solves this problem.
Datase Generalization. We also evaluate the zero-shot performance of the model, where train exclusively on RealEstate 10k and directly apply it to ScanNet++ (Yeshwanth et al., 2023) and DTU (Jensen al., 2014) datasets. The results in Fig. and Tab. indicate that NoPoSplat demon-
8

---

Published as a conference paper at ICLR 2025
Input
Ours (pose-free)
nvspplatipse--rquiieel
pixelSplat requird)
Figure 5: Comparisons of 3D Gaussian and rendered results. The red and green indicate input and target camera views, and the rendered image and depths are shown on the right side. The magenta and blue arrows correspond the distorted misalignment regions in baseline 3DGS.
The results show that even without camera poses as input, our method produces higher-quality 3D
Gaussians resulting better color/depth rendering over baselines.
Ref. pixelSplat
MVSplat
Ours
Ref. pixelSplat
MVSplat
Ours
GT
(a) Generalize: RE10K DTU
Generalize: RE10K ScanNet++
Figure 6: Cross-dataset generalization. Our model can better zero-shot transfer to out-of- distribution data than SOTA pose-required methods.
strates superior performance on out-of--istribuue data compared to SOTA pose-required methods. This advantage arises primarily from our minimal geometric priors in the network structure, allow- ing it to adapt effectively to various types of scenes. Notably, our method outperforms Splatt3R even on ScanNet++, where Splatt3R was trained.
Model Efficiency. As shown on the right, our method can predict 3D
0.08
Gaussians from two 256 256 input images in 0.015 seconds (66 fps), which is around 5x and 2 faster than pixelSplat and MVSplat, on the same RTX 4090 GPU. This further shows the benefits of using a standard VIT without incorporating additional geometric operations.
Time (oe)
3.05
6002
pisc1Splal MVSpkt Ours
Apply to In-the-Witd Wild Unposed Images. One of the most important advantages of method is that it can directly generalize to in-the-wild unposed images. we test on two types data: images casually taken with mobile phones, and image frames extracted from videos generated by Sora (Ope- nAL, 2024). Results in Fig. 7 show that our method can be potentially applied for text/image to 3D scene generations, e.g., to multi-image/vidde models (Gao et al., 2024; OpenAI, 2024), then feed those extracted unposed images our model and obtain 3D models.
first generate sparse scene-level multi-view images using text/image
4.2 ABLATION STUDIES
Ablation on Output Gaussian Space. To demonstrate the effectiveness of our canonical Gaussian prediction, we compare it with the transform-then-fiee pipeline commonly used by pose-required methods (Charatan et al., 2024; Chen et al., 2024). Specifically, it first predicts Gaussians in each transformed to the world coordinate using camera poses.
local camera coordinate system which For fair comparisons, both methods use the same backbone and head but differ in the prediction of Gaussian space. The results in row (f) of Tab. show that pose-free canonical space predic-

---

Published as a conference paper at ICLR 2025
Unposed
3DGS (H) Input Images taken with mobile phones
Novel Views
Unposed Inpues
JUXIS
Novel Views
(b) Input Images generated by SORA
Figure 7: In-the-wild Data. We present the results of applying our method to in-the-wild data, including real-world photos taken with mobile phones and multi-view images extracted from videos generated by the Sora texi-to-video model. See the video supplementary for more results.
tion method outperforms such pose-required strategy. Fig. 8 illustrates that the transform teeerrmtthee--eeee fhen-fuse strategy leads to the ghosting artifacts in the rendering, because struggles to align two separate Gaussians of two input views when transformed global coordinate.
Intrinsic
Short
Trans-then-fuse
Num
Design
PSNK
SSIM
LPIPS
(6)
Ours
25.033
0.538
0.160
intrinsic
(6)
w/ dome
0.166
w/ global
0.836
No
Transform then fuse
24.622
0.834
0,167
26.699
0.872
0.127
Table 5: Ablations. intrin- sic embeddings are vital for performance and using intrin- sic tokens performs the best. Adding the RGB image short- cut also improves the qual- ity of rendered images. Our method achieves better perfor- mance compared with the pose- required per-local-view Gaus- sian field prediction method.
Figure 8: Ablations. No intrinsic results in blurriness due to scale misalignment. Without the RGB image shortcut, the ren- dered images are blurry in the texture-rich areas. Using the transform-then-fusee strategy causes ghosting problem.
Ablation on Camera Intrinsic Embedding. Here study three intrinsic encodings described in Sec. 3.4 as well as inputting no intrinsic information. First, we can see in Tab. 5 row (b) and Fig. that intrinsic encodings lead to blurry results as the scale ambiguity confuses the learning process of the model. Second, we notice that the intrinsic token constantly performs the best marginally among these three proposed intrinsic encodings, so, we
our default intrinsic encoding.
Importance of RGB Shortcut. As discussed in Sec. 3.2, in addition to the low-res ViT features, also input RGB images into the Gaussian parameter prediction head. As shown in Fig. 8, when there is no RGB shortcut, the rendered images are blurry in the texture-rich areas, see the quilt in row 1 and chair in row
Extend to More Input Views. For fair comparisons with baselines, we primarily conduct experi- ments using two input view settings. Here we present results using three input views by adding an additional view between the two original input views. As shown in row (g) of Tab. 5 and Fig. 13, performance significantly improves with the inclusion of the additional view. Results with more input views can be found in Fig. 10.
CONCLUSION
This paper introduces simple yet effective pose-free method for generalizable sparse-view 3D reconstruction. By predicting 3D Gaussians directly in a canonical space from any given unposed multi-view images, we demonstrate superior performance in novel view synthesis and relative pose estimation. While our method currently applies only to static scenes, extending our pipeline to dynamic scenarios presents interesting direction for future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported as part of the Swiss AI Initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID a03 on Alps. Botao Ye was partially supported by the ETH AI Center. Ming-Hsuun Hsuan Yang was supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOL/IBC) contract number 140D0423C0074. We also thank David Charatan for helping us evaluate the performance of some of the previous methods.
REFERENCES
Eduardo Arnold, Jamie Wynn, Sara Vicente, Guillermo Garcia-Hemanndo, Aron Monszpart, Vic- tor Prisacariu, Daniyar Turmukhambetov. and Eric Brachmann. Map-free visual relocalization: Metric pose relative single image. In ECCV, 2022.
Wenjing Bian, Zirui Wang, Kejie Li, Jia-Wang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. In CVPR, 2023.
David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaus- sian splats from image pairs for scalable generalizable 3d reconstruction. CVPR, 2024. 2, 3, 4, 5, 6, 7, 9, 18
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang. Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, 2021.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022. 3
Yu Chen and Gim Hee Lee. Dbarf: Deep bundle-adjusting generalizable neural radiance fields. In CVPR, 2023. 3,4
Yuedong Chen, Haofci Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat- Jen Cham, and Jianfei Cai. Mysplat: Efficient 3d gaussian splatting from sparse multi-view images. arXiv preprint
24t.:24003.46227 2024. 1, 4, 6, 7, 9, 15, 18
Shin-Fang Chng, Sameera Ramasinghe, Jamie Sherrah, and Simon Lucey. Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation. In ECCV, 2022.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießnnrr Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. 7,8
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021. 5.
Yilun Du, Cameron Smith, Ayush Tewari, and Vincent Sitzmann. Learning to render novel views from wide-baseline stereo pairs. CVPR, 2023, 6,
Johan Edstedt, Qiyu Sun, Georg Bökman, Mårten Wadenbäck, and Michael Felsberg, Roma: Robust
dense feature matching. In CVPR, 2024. 6, 7, 15
Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang, Jian Zhang, Xinghao Ding, Danfei Xu, Boris Ivanovic, Marco Pavone, Georgios Pavlakos, et al. Instantsplat: Unbounded sparse-view pose-free gaussian splatting 40 seconds. arXiv preprint arXiv:2403.200 2024. 6
Martin Fischler and Robert Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981. 6
Sara Beclovich-Keill Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.

---

Published as a conference paper at ICLR 2025
Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A Efros, and Xiaolong Wang. Colmap-free 3d gaussian splatting. In CVPR, 2024. 3, 3,4, 20
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin Brruala,, Pratul Srinivasan, Jonathan Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-viww diffusion models. In NeurlPS, 2024. 9
Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge university press, 2003. 2, 6
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 17
Sunghwan Hong, Jacwoo Jung, Heescong Shin, Jiaolong Yang, Seungryong Kim, and Chong Luo. Unifying correspondence pose and nerf for generalized pose-free novel view synthesis. In CVPR, 2024a. 2, 3, 4, 6, 7, 20
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image 3d. In ICLR, 2024b.
Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi- view stereopsis evaluation. In CVPR, 2014. 7,8
Hanwen Jiang, Zhenyu Jiang, Kristen Grauman, and Yuke Zhu. Few-view object reconstruction with unknown categories and camera poses. In 3DV, 2024a. 3
Hanwen Jiang, Zhenyu Jiang, Yue Zhao, and Qixing Huang. Leap: Liberate sparse-view 3d model- ing from camera poses. In ICLR, 2024b. 4, 16
Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In CVPR, 2022.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM TOG, 2023. 2,3,4
Arno Knapitsch, Jaesik Park, Qian-Y Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM TOG, 2017. 16
Vincent Leroy, Yohann Cabon, and Jérôme Revaud, Grounding image matching in 3d with mast3r. arXiv preprint arXiv:2406.09556, 2024. 2, 4, 7, 15, 17, 19, 20
Amy Lin, Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose++: Recovering 6d poses from sparse-view observations. arXiv preprint arXiv:2305.0492 2023.
Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey, Barf: Bundle-adjusting neural radiance fields. ICCV, 2021. 3
Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D13dv-10k: A large-scale scene dataset for deep learning-basee 3d vision. In CVPR, 2024. 2, 6, 15
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In ICCV, 2021. 2, 6, 7, 15, 20
Fangfu Liu, Wenqiang Sun, Hanyang Wang, Yikai Wang, Haowen Sun, Junliang Ye, Jun Zhang, and Yueqi Duan. Reconx: Reconstruct any scene from sparse views with video diffusion model. arXiv preprint vrXiv:2408.1666) 2024. 4
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018. 15
Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew J Davison. Gaussian splatting slam. In CVPR, 2024. 3, 6
12

---

Published as a conference paper at ICLR 2025
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, (CCCV,000 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphies prim- itives with multiresolution hash encoding. ACM TOG, 2022.
Bharath Raj Nagoor Kani, Hsin- Ying Lee, Sergey Tulyakov, and Shubham Tulsiani. Upfusion: Novel view diffusion from unposed sparse view observations. In ECCV, 2024.
OpenAL. Creating video from text, February 2024. URL https: com/sora. 3, 9, 16
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 20233 17
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction.
In CVPR, 2021. 5
Mchdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation trans- former: Geometry-free novel view synthesis through set-latent representations. In CVPR, 2022. 3
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue:
Learning feature matching with graph neural networks. In CVPR, 2020. 7,8
Johannes L Schonberger and Jan-Michael Frahm. ftructur-from-mmotte revisited. In CVPR, 2016.
2, 3, 6, 19
Brandon Smart, Chuanxia Zheng, Iro Laina, and Victor Adrian Prisacariu. Splatt3r: Zero-shot gaussian splatting from uncalibrated image pairs, arXiv preprint arXiv:2408 13912, 2024. 4, 7, 19,20 20
Cameron Smith, Yilun Du Ayush Tewari, and Vincent Sitzmann. Flowcam: training general- izable 3d radiance fields without camera poses via pixel-aligned scene flow. arXiv preprint 1xXi:23306.00/80 2023. 3,4
Stanislaw Szymanowicz. Chrisitian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In CVPR, 2024. 5
Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang. Gang Zeng, and Ziwei Liu. Lgm: Large view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054, 2024. 6
Prune Truong, Marie-Julis Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In CVPR, 2023.
Peng Wang. Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexi- ang Xu, and Kai Zhang. Pf-Irm: Pose-free large reconstruction model for joint pose and shape prediction. In ICLR, 2024a. 4,5
Qianqian Wang, Zhicheng Wang. Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi- view image-based rendering. In CVPR, 2021a. 3
Shuzhe Wang. Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. CVPR, 2024b. 2, 3, 7, 15, 17
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004. 6
Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2 2021b. 3,
13

---

Published as a conference paper at ICLR 2025
Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. Croco v2: Improved vies completion pre-training for stereo matching and optical flow. In ICCV, 2023. 17
Chao Xu, Ang Li, Linghao Chen, Yulin Liu, Ruoxi Shi, Hao Su, and Minghua Liu. Sparp: Fast 3d
object reconstruction and pose estimation from sparse views. ECCV, 2024a. 3
Haofei Xu, Anpei Chen, Yuedong Chen, Christos Sakaridis, Yulun Zhang, Mare Pollefeys, Andreas Geiger, and Fisher Yu. Murf: Multi-baseline radiance fields. In CVPR, 2024b.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:231 1.09217, 2023.
Yinghao Xu, Zifan Shi. Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint arXiv:2403. /4621, 2024c. 3
Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai.
Scannet++: high-
fidelity dataset of 3d indoor scenes. In ICCV, 2023. 7, 8, 19
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In CVPR, 2021. 3,7
Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose: Predicting probabilistic relative rotation for single objects in the wild. ECCV, 2022.
Jason Y Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang, Deva Ramanan, and Shubham Tul- siani. Cameras rays: Pose estimation via ray diffusion. arXiv preprint arXiv:2402.148877 2024.
Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Imm: Large reconstruction model for 3d gaussian splatting. In ECCV, 2025.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018. 6.
Shunyuan Zheng, Boyao Zhou, Ruizhi Shao, Boning Liu, Shengping Zhang, Liqiang Nie, and Yebin Liu. gasssaausii:: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. In CVPR, 2024. 5
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images. ACM TOG, 2018. 2, 6, 7, 15, 19
Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Os- wald, and Mare Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In CVPR, 2022. 3
Zihan Zhu, Songyou Peng, Viktor Larsson, Zhaopeng Cui, Martin R Oswald, Andreas Geiger, and Mare Pollefeys. Nicer-slam: Neural implicit scene encoding for rgb slam. In 3DV, 2024. 3
14

---

Published as a conference paper at ICLR 2025
A MORE IMPLEMENTATION DETAILS
More Training Details. We first describe the process for training the 256 x 256 model, which serves as the basis for all baseline comparisons. Following Chen et al. (2024), when training on RealEstate (Zhou et al., 2018) and ACID (Liu et al., 2021) separately, the model is trained on 2.4 10 input image pairs, randomly sampled from training video clips to ensure diverse input coverage. We employ the A optimizer (Loshchilov & Hutter, 2018), setting the initial learning rate for the backbone to x 10-5 and other parameters to 2 x 10 When trained on the combined RealEstate 10k and DL3DV (Ling et al., 2024) datasets, sample image pairs evenly from both datasets and double the training steps.
We then train the 512 512 model using the pre-trained weights of the 256 256 model. The 512 x 512 model also trained on the combined RealEstate and DL3DV datasets, following the same training procedure. Since no fair baseline model is available for comparison, focus primarily on the qualitative results of the 512 x 512 model. For instance, the results presented in Fig. 7 are obtained using the 512 x 512 model.
For the 256 x 256 version of the model, training was conducted on 8 NVIDIA GH200 GPUs (each with >80 GB memory) for approximately 6 hours. We also experimented with training our model on single A6000 GPU (48 GB memory). While this setup required more time (approximately 90 hours), it achieved comparable performance (PSNR on REIOK: 25. 25.018 with A6000 vs. 25.033 with GH200). For the 512 x 512 version, training was performed on 16 NVIDIA GH200 GPUs and required approximately one day.
Evaluation Set Generation. For the evaluation of the novel view synthesis task, we randomly select one image pair from each video sequence the official test splits of RealEstate 10K (Zhou et al., 2018) and ACID (Liu et al., 2021). We then randomly sample three frames between the input views as target views. For the pose estimation task, we use the same input image pair as in the novel view synthesis task, and employ the COLMAP pose the ground truth.
To determine the overlap ratio of image pairs, we employ RoMa (Edstedt et al., 2024), a state-of- the-art dense feature matching method. The process for calculating the overlap ratio for an image pair [T' 12] } is as follows:
1. Obtain dense image matching from I to 12 and vice versa.
2. Consider matching scores above 0.005 as valid.
3. Calculate the overlap ratio ri-oj for image I' to image F as:
Number of valid matched pixels
Ti-j
Total number of pixels
4. Compute F1-+2 and r2-+1
5. Define the final overlap ratio as:
Foverlap mnin(r1-2,,
The resulting evaluation set and the code used for its generation will be made publicly available to facilitate further research in this area.
Statisties on the Evaluation Set. The number of scenes in RealEstate 10K (Zhou et al., 2018) for each overlap category is as follows: 1403 for small, 2568 for medium, and 1630 for large overlaps. In the ACID dataset (Liu et al., 2021), the number of scenes for each overlap category is: 249 for small, 644 for medium, and 448 for large overlaps.
Baseline Setup for Pose Estimation Tasks. For fair comparison with RoMa (Edstedt et al., 2024), we first resize and center-crop the input images to match the dimensions used in our model (256 x 256), then resize them to a coarse resolution (560 x 560) and an upsampled resolution (864 x 864) to fit RoMa's 's requirements. However, since DUS(3R (Wang et al., 2024b) and MASt3R (Leroy et al., 2024) are not trained with square image inputs, simply setting the input resolution to 256 x 256 leads to poor pose estimation performance. Therefore, we resize and center-crop their input images to 512 x 256, as officially supported by their models. Consequently, the image content visible to DUS13R and MASt3R is greater than that seen by RoMa and our model.
15

---

Published as a conference paper at ICLR 2025
Input Views
LEAP
Ours
GT
Figure 9: Object-level comparison on Objaverse dataset. Compared with the pose-free baseline method, LEAP, method shows significantly better novel view synthesis results object-level data.
Table 6: Performance comparison on Objaverse dataset.
PSNR
SSIM
LPIPS4
LEAP (Jiang et al., 2024b) Ours
20.559
0.853 0.935
0.144
28.378
0.053
Details on Pose Estimation. After obtaining the coarse camera pose, we apply photometric loss- based optimization to refine it. This optimization is performed for 200 steps per input image pair with learning rate x 10-3, taking approximately seconds. Notably, our pose estimation can also operate without the optimization refinement process, resulting in acceptable performance degradation but decreasing inference time. As shown in Tab. 9, despite the performance drop, our method still performs comparably to RoMa, which is trained on ScanNet,
in zero-shot setting.
Details on Applying to In-the-Wile Data. Our method requires camera intrinsic information as input for rendering novel views. When testing model on out-of-distribution datasets, such as Tanks and Temples (Knapitsch et al., 2017), we simply use the ground truth intrinsic data provided by the dataset. When tested photos taken with mobile phones, extract the focal length from their EXIF metadata. For data generated by SORA (OpenAI, 2024), we set the focal length to (H W)/2. We found that this heuristic setting works quite well, our model is relatively robust to focal length variations after training.
B APPLY TO ORJECT-LEVEL. DATA
To demonstrate our method's generalizability to object-level data, we evaluate its performance on the commonly used Objaverse dataset. Our experiments utilize four 256 x 256 unposed images as input for each object. Direct performance comparison with PF-LRM not feasible due to its unavailable source code and evaluation dataset. Therefore, compare method with LEAP, the state-of-the-art open-source pose-free method for object-level reconstruction.
To ensure fair comparison, retrain LEAP (Jiang al., 2024b) on the Objaverse dataset using their official implementation, maintaining identical training iterations and input image resolutions. The qualitative and quantitative results presented in Fig. 9 and Tab. 6 respectively. Our approach demonstrates significant performance improvements over LEAP, with the PSNR increasing from 20.559 28.378. highlighting robust generalization capabilities for object-level reconstruction.
16

---

Published as a conference paper at ICLR 2025
Table Ablation on different weight initialization. The results show that our method effectively learns pose-free inference capabilities during training, with appropriate weight initialization further enhancing performance. Notably, even with random initialization, our method significantly outper- forms the pose-free baseline (CoPoNeRF). Moreover, utilizing CroCov2 and DINOv2 pre-trained weights enables our method to surpass the previous SOTA pose-required method (MVSplat), despite neither initialization method providing prior knowledge of pose depth information.
PSNRT
SSIM
LPIPS
MASt3R
25.033
0.838
0.160
DUS13R
24.553
0.823
0.169
CroCov2
24.559
0.818
0.171
DINOv2
24.094
0.812
0.176
Random
23.487
0.779
0.189
Table 8: Ablation adding additional pose condition. We incorporate Plucker ray pose embed- dings into our pose-free model as additional input. The results show small performance gap after adding the embeddings, indicating the strong capability of our method in estimating pose.
Init Weights
Pose Condition
PSNR1
SSIM
LPIPS
Yes
25.033
0.838
0.160
MASt3R
No
25.080
0.844
0.158
Yes
23.708
0.788
0.173
Random
No
23.487
0.779
0.189
C MORE EXPERIMENTAL ANALYSIS
Ablations on Backbone Initialization. We initialize our backbone network with MASt3R (Leroy al., 2024) pre-trained weights in our main experiment. MASt3R is trained on datasets with ground truth depth annotation, but notably, its training data has no overlap with the training and evaluation datasets used in our experiments. Here, we also investigate the impact of different backbone initial- ization methods. Specifically, we compare the performance of our method using pre- pre-trained weights from DUSt3R (Wang et al., 2024b), CroCo V2 (Weinzaepfel et al., 2023) (DINOv2 is only used to initialize the encoder, the decoder is randomly initialized), as well as randomly initialized weights. DUSt3R is also trained with depth supervision but without fea- ture matching loss compared with MAS13R. CroCoV2 is pre-trained with pure 2D) image pairs with reconstruction loss (He et al., 2022), while DINOv2 (Oquab et al., 2023) is a self-supervised method trained on 2D images. Note that for CroCoV2, DINOv2, and random initialization, we warm up the training by adding point cloud distillation loss from the DUSt3R model for 1,000 steps, this aims to tell the network the goal is predict the Gaussians in the canonical space, otherwise, the learning target is too hard for the network to understand as we only train our network with photometric loss. These results in Tab. 7 demonstrate that our method effectively learns pose-free inference capabili- ties during training. with appropriate weight initialization further enhancing performance. Notably, even with random initialization, our method significantly outperforms the pose-free bascline (Co- PoNeRF). Moreover, utilizing CroCov2 and DINOv2 pre-trained weights enables our method to surpass the previous SOTA pose-required method (MVSplat), despite neither initialization method providing prior knowledge of pose or depth information.
al., 2023), and DINOv2 (Oquab
Ablations on Pose Condition. To further demonstrate the effectiveness of our method in inferring geometry without pose information, we conducted an ablation study by incorporating additional pose conditioning into our network. Keeping all other factors constant, we converted the pose in- formation into Plucker ray representations and concatenated them with the RGB images network input. The experimental results, shown in Tab. 8, reveal that while pose conditioning slightly im- proves performance compared to pose-free method, the enhancement is small especially when initialized with MASt3R weights. This highlights our method's 's robust ability to correlate image pairs even in the absence of explicit pose information.
Ablations on the Effectiveness of Two-Stage Pose Estimation. As shown in Tab. 9, relying solely PAP-RANSAC results in inaccurate pose estimates. However, if we skip the coarse camera
17

---

Published as a conference paper at ICLR 2025
Table 9: Ablation on the pose estimation method.
PnP
Photometric
50 f
10 1
20" f
0.318
0.538
0.717
0.287
0.506
0.692
0.017
0.027
0.051
Table 10: Performance comparison on the evaluation set pixelSplat.
PSNRT
SSIM
LPIPS
pixeINeRF
20.43
0.589
0.55
GPNR
24.11
0.793
0.255
AttnRend
24.78
0.82
0,213
pixelSplat
26.09
0.863
0.136
VVSllt
26.39
0.869
0.128
Ours
26.786
0.878
0.124
pose estimation stage and only use photometric loss-based optimization starting from [I 0), the performance significantly degrades when using the same number of optimization steps (200 steps). This is because optimizing from an initial pose from the target is more challenging.
Evaluate on the Evaluation Set of pixelSplat. We also present results based on the evaluation set used by pixelSplat (Charatan et al., 2024) and MVSplat (Chen et al., 2024). The results are shown in Tab. 10. However, we do not prioritize this evaluation set as it is relatively simple, with most input pairs exhibiting significant overlap, making it less suitable for more advanced future research.
Details Extension to 3 Input Views. Our method can be extended to an arbitrary number of input views. As shown in Tab. 5, using three input views improves the results. Specifically, we add the middle frame between the original tings, when generating the Gaussians of each view, we concatenate the feature tokens from all other views and perform cross-attention with these concatenated tokens in the ViT decoder stage, keeping all other operations the same as with two input views. Fig. 13 provides a qualitative comparison between using two and three input views.
input frames an additional view. For three-input set-
Experiments with More Input Views. We further evaluate the performance with varying numbers of input views. We train separate models for each configuration. The quantitative results are pre- sented in Fig. 10. To ensure fair comparison, we maintain consistent first and last views across all experiments, while sampling intermediate views as the number of input views increases. The results demonstrate significant performance improvements when increasing the input views from two to three or four. However, additional input views beyond four yield comparable performance. This is because four views are sufficient to capture most of the scene information. This finding under- scores our method's s robust capability to effectively process sparse input images. Figure 11 provides qualitative examples across different view configurations, corroborating the quantitative results.
Geometry Comparison with Splatt3R and MASt3R. In Figure 5, we compare our generated 3D 3D Gaussian Splatting methods, namely
Gaussians with state-of-the-arr pose-required pixelSplat and MVSplat. Additionally, we compare the geometric representations with MASt3R and Splatt3R. Specifically, we compare 3D Gaussians with Splatt3R's s 3D Gaussians and MASt3R's point cloud, as MASt3R only outputs point cloud representations. Notably, our method is trained solely using photometric loss, without ground truth point cloud supervision, whereas MASt3R uti- lizes ground truth point cloud for training, and Splatt3R directly employs MASt3R's 's point cloud outputs as Gaussian centers. The results in the first row of Figure 12 demonstrate that our method achieves better geometric details (e.g., straighter wall edges) despite the absence of point cloud supervision. The second row reveals that our method more coherently fuses the content of two put images, while significant quality discontinuities exist in the intersection areas of MAS13R and Splatt3R. Furthermore, MASt3R fails to capture the geometric relationships of distant objects, re- sulting in severe blurring in these regions when Splatt3R directly uses its point cloud as Gaussian centers.
18

---

Published as a conference paper at ICLR 2025
27.0
26.803
26.818
26.826
26.619
26.5
PSNR
26.0
25.5
25.033
25.0
2
5
6
Num of views
Figure 10: Novel view synthesis results with different number of input views.
Varying number of imputtiews
Novel View
Figure RealEstatel0k performance with different number of input views.
Addtional Comparison with Splatt3R. In Tab.1 of the main paper, we compare our method with the official model provided by the authors of Splatt3R(Smart et al., 2024), which freezes the MASt3R model and is then trained on the ScanNet++ (Yeshwanth et al., 2023) dataset. Here, we also train it on the RealEstate 10K (Zhou al., 2018) dataset at the same resolution other base- lines (256 x 256). However, when attempting to retrain it using the official code provided by the authors on RealEstate 10k, we find that the training fails because the original Splatt3R can only be trained on datasets with metric pose ground truth. This limitation arises because Splatt3R relies on a fixed pre-trained MASt3R (Leroy et al., 2024) model and its capability for metric depth predic- tion. As MASt3R is trained on ScanNet++ (Yeshwanth et al., 2023) using ground truth metric depth information, the poses in ScanNet++ are also metric. Consequently, the original Splatt3r cannot be trained on video datasets without metric pose information. We identify two main issues:
1. The camera poses provided by RealEstate 10K are up-to-scale, as they are estimated using COLMAP (Schonberger & Frahm, 2016). This results in scale misalignment between the ground truth pose and the scale of the estimated Gaussian field. Although Splatt3r predicts ad- ditional point offsets based on the point cloud estimated by MASt3R as the final Gaussian cen- ter, these offsets are typically small and unable to resolve the misalignment problem. This issue persists in datasets with metric camera poses, the point cloud provided by MASt3R is imperfect and not fully aligned with the ground truth pose scale. The training fails if the pro- vided COLMAP target camera poses are used render target views for the 3D Gaussians.
19

---

Published as a conference paper at ICLR 2025
Input Views
Ours
Splatt5R
MASIER
Figure 12: Geometry comparison with Splatt3R and MASt3R.
Table 11: Performance retrained Splatt3R (Smart ef al., 2024) model.
Small
Large
Average
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS.
PSNR1
SSIM
LPIPS].
PSNR1
SSIM
LPIPS,
Ours
22.514
0.784
0.210
24.899
0.839
27.411
0,883
0.119
25.033
Splau3R Offccial
14.352
0.475
0.472
15.529
0.502
0.425
15.817
0.483
0.421
15.318
0.490
0.436
Splam:3 Rotrain
0.385
0.685
0.288
To address this, we first estimate the camera poses of two input views using the point cloud from MASt3R, which is scale-consistent with the Gaussians. We then align the scale of the COLMAP target poses with the scale of the MASt3R point cloud by adjusting the scale of the input view poses to match the MASttR-estimated ones.
2. We also find that the intrinsic parameters estimated MASt3R (Leroy et al., 2024) do not align well with the ground truth intrinsic parameters. Using the ground truth intrinsic during training also causes failure. As result, opt to the intrinsics estimated by MASt3R.
Additionally, we ignore the loss mask used the original Splatt3R as requires ground truth depth to generate the mask, but the RealEstate 10K dataset lacks ground truth depth. The mask is unnec- essary for training on video datasets like RealEstate 10K, since during training. we use intermediate frames between the two input frames targets, and most of the image content in the target frames well covered by the input frames. For a fair comparison during evaluation, we apply the same evaluation-time pose alignment technique used in our method. The results are presented in Tab. 11. Although retraining the model on the RealEstate IOK dataset yields improved performance, it still significantly lags behind our approach. This performance gap can be attributed to the misalignment issues inherent in the fixed MASt3R model and the persistent scale ambiguity problem.
D LIMITATIONS
Our approach, like previous pose-free methods (Hong et al., 2024a; Fu et al., 2024), assumes known camera intrinsics. Although heuristically set intrinsic parameters prove effective for in-the-wild images, relaxing this requirement would enhance the robustness of real-world applications. Ad- ditionally,
feed-forward model is non-generative, it lacks the ability to reconstruct unseen regions of a scene with detailed geometry and texture, as demonstrated in the results of the 2-view model shown in Fig. 13. This limitation could potentially be mitigated by incorporating additional input views, which would enhance scene coverage. Finally, the current training data (limited to RealEstate 10K, ACID, and DL3DV) constrains the model's generalizability scenarios. Future work could explore training our model on scage, diverse indoor and out- door datasets, leveraging our method's independence from ground -ruth depth information during training.
diverse in-the-wild
E MORE VISUAL COMPARISONS
We present an additional comparison with previous SOTA pose-dependent and pose-free methods across various levels of image overlap: Fig. 14 14 for small input image overlap, Fig. 15 for medium input image overlap, and Fig. 16 for large input image overlap. Furthermore, we provide additional comparisons on the ACID dataset (Liu et al., 2021) in Fig.
20

---

Published as a conference paper at ICLR 2025
Ref.
Target View
Ours (3 Views)
Ours (2 Views)
Figure 13: Qualitative comparison on different numbers of input views. The model with 2 input views utilizes only the top and bottom reference images, whereas the 3-view model incorporates an additional intermediate view. Adding this extra reference view improves the quality of rendered novel views significantly, as it captures finer spatial details and reduces occlusion effects.
21

---

Published as a conference paper at ICLR 2025
Ref.
PixclSplat
MVSplat
CoPoNeRF
Splatt3R
Ours
GT
Figure 14: More comparisons of the RealEstate109 10K dataset with small overlap of input images.
22

---

Published as a conference paper at ICLR 2025
Ref.
pixclSplat
MVSplat
CoPoNeRF
Splatt3
Ours
GT
Figure 15: More comparisons of the RealEstate dataset with medium overlap of input images.
23

---

Published as a conference paper at ICLR 2025
Ref.
PixclSplat
MVSplat
CoPoNeRF
Splant3
Ours
GT
Figure 16: More comparisons of the RealEstate 10K dataset with large overlap of input images.
24

---

Published as a conference paper at ICLR 2025
Ref.
pixelSplat
MVSplat
CoPoNeRF
DUStä
Splatt3R
Ours
GT
m
Figure 17: More comparisons on the ACID dataset.
25


Reviews:
[{"review_id": "KXwnRS8SxR", "reviewer": "Reviewer_hSJP", "summary": "This submission proposes a sparse Gaussian splats prediction given very sparse unposed images. Comparing with prior works such as Master and Splatter, the method achieves good results without using supervision from depth, loss masks, or metric poses.\n\nAs this is a strong submission, the only weakness I could think of is that the paper should include quantitative comparisons on geometry reconstruction results with baselines such as Master and Splatter.\n\nIn short, my initial recommandation is **accept**, and I am willing to increase it to strong accept if the quantitative comparisons on geometry reconstruction are provided.", "questions": "1. Add quantitative results for geometry reconstruction?\n2. Clarifying questions (not affecting the paper rating):\n    1. It seems like the main difference between this and Splatter is using / not using depth supervision and metric poses, which leads to an \"impression\" that using depth supervision would limit the dataset choices and hence constrain the GS prediction performance?\n    2. L838-840: \"Using the ground truth intrinsic during training also causes failure\". I assume this is about training Splatter failed using the ground truth intrinsics?\n    3. What is the training hardware and training time?\n    4. The proposed method also loaded pre-trained Master weights, are they frozen or fine-tuned?\n3. Related work: maybe worth adding another concurrent work: ReconX (not affecting the paper rating)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The paper is very well written and easy to follow.\n* Good reconstruction and NVS results while no need for depth, loss masks, and metric poses comparing with prior work.\n* Good generalization capability on in-the-wild data.", "weaknesses": "It would be great to include quantitative results for geometry reconstruction, comparing it with Master and Splatter. Currenly only qualitative results are provided in Fig. 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TX3bD5cBfW", "reviewer": "Reviewer_wkJy", "summary": "The paper proposes a fast feed-forward model for sparse-view 3D reconstruction that is generalizable across scenes. Differently from recent work, the predictions (3DGS parameters) are performed in a common coordinate space directly without requiring any pose info. They perform evaluations on the RealEstate10k and ACID datasets and show improved performance over prior works. Their method also shows zero-shot generalizability to other datasets and in-the-wild scenarios. They also show relative pose estimation using a two-step approach: first with pnp and then with photometric optimization. The performance is benchmarked on similar datasets as well.", "questions": "- Contribution no. 4 is not a contribution. \n- Multi-view (N>2): Are there any qualitative results for input images more than 2? What would the upper limit on this be given compute constraints? How does the cross-attention in the ViT decoder work for this case since L215 mentions \"features from each view interact with those from all other views..\"? Also, adding qualitative and quantiative results for N>2 and performance tradeoff cuves would strengthen this method.\n- L274: \"By default, we utilize the intrinsic token option\". This needs to be \"Global Intrinsic Embedding - Concat option\".\n- Table 1, Fig4: Why is Dust3R and Mast3r part of the NVS comparison since they were never trained for novel view synthesis? What was the evaluation protocol for these methods? \n- In Fig4, its possible that the Gaussian structure masks the actual geometry of the scene when comparing to pointcloud outputs of Dust3r. To show geometry, it would be better to either show 3D points or depth maps. To show NVS, it would be better to remove Dust3r output since it is not a fair comparison.\n- Discrepancy in metrics:\n  * Both MVSplat and pixelSplat report higher PSNR numbers for RealEstate10k and ACID datasets. Since the same train-test split was used, why is there a huge discrepancy in the 'Average' column of the metrics?\n  * Table 4: why might be the Splatt3r numbers lower than those reported in the original paper?\n- Mentioning the training time and compute requirements would be useful additions to the paper.\n\nAppendix\n- Tab. 6 ablation does not provide the complete picture. The NVS metrics are dependent on the accuracy of the 3DGS field. Since the 3DGS parameter head is randomly initialized irrespective of the backbone, its contribution due to backbone initialization might not be as pronounced. A more accurate ablation would be on pose estimation given pure 3D points. \n- L773, L775 typo. \"experiences\" -> experiments", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is simple and easy to understand. More specifically, the highlighted advantages are:\n- A simple VIT approach is competitive to pretrained ViT backbones from Dust3r, and Mast3r when overlap is minimal.\n- The method does not require any ground-truth depth (explicit matching loss) during training thus allowing it to scale further. This is an important advantage since simple architectures w/ large datasets is a recipe for improving generalizability.\n- Using pretrained Mast3r weights with pure photometric losses on more datasets is a good strategy to scale up training and improve model performance on nvs and relative pose estimation.", "weaknesses": "The canonical space motivation: The central argument of the paper is that predicting the scene structure in a common coordinate space improves scene geometry and multi-view cohesion compared to local prediction->global fused prediction approach. Although I agree with this statement, the paper does not exploit this advantage:\n  * Given two views (which is the only input setting shown in the paper), the method still predicts per-pixel per-view 3DGS fields in a common coordinate (similar to Splatt3r). This is similar to Dust3r pointcloud variants where the pointclouds are predicted in a common coordinate space. I fail to see how this part is different from Dust3r-style prediction. Can you explain how this approach differs from those in the 2-view setting, if at all?\n  * The main advantage seems to come from finetuning the Mast3r backbone on more data using photometric loss alone. By directly predicting 3DGS fields, this approach can improve overall model performance. But this orthogonal to the motivation of the method. It would be good to highlight this as one of the main features of this method\n\nThis approach is quite similar to Splatt3r although it is concurrent work. The advantage of this approach is that it does not need ground-truth depth during training which is important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s8ei3YPNmt", "reviewer": "Reviewer_tg6H", "summary": "This paper proposes a feed-forward generalizable GS method from sparse and unposed images. It encodes images (with known intrinsics) into feature tokens and use a decoder with cross-attention layers for cross-view feature fusion. Then the gaussians are predicted in a canonical space. The proposed method is evaluated on RealEstate10K/ACID and tested generalization on DTU.", "questions": "Please first see comments above. \n\n**It is necessary to discuss prior works in an appropriate way.** I will give a reject first but I can raise the score to 5 if the author can differentiate their work from the missing related works and rephrase their contribution. \n\n**Besides, I strongly require the authors to perform experiment using the same training dataset with previous works.** For example, if you want to compare with MVSplat, you should only train on RealEstate10K+ACID. This is the correct way of doing scientific research. Otherwise, we cannot tell whether the performance gain comes from better model design or using larger and high-quality data (and also your MAST3R weight initialization should be ablated). The additionally used datasets in this paper are all from available resources and there is no need for data curation. It is different from LLM research where data collection and curation and be their contribution, and, thus, it is acceptable to directly compare LLMs trained using different data.\n\nI can raise the score to 6 or higher scores if the authors performs the asked three experiments (weakness 3) with promising results.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "- The paper proposes a simple and straightforward method.\n- The paper is well-written and is easy to follow.\n- The method shows a good performance, which is even better than previous pose-aware methods, e.g. MVSplat.", "weaknesses": "1. This paper is technically sound. However, basic literatures in this field are not discussed and the claimed novelty have already been studied in prior works. I don't understand why the authors ignore these works when the content and techniques are closely related. Please see details below.\n- PF-LRM [1] and LEAP [2] are the two earliest works for pose-free reconstruction, but none of them are mentioned. Almost all claimed novelty of the submssion, e.g. canonicalization, PnP pose estimation, using cross-attention for cross-view feature fusion, have already been studied in these two works (also true for DUST3R). The only difference is whether using 3DGS or NeRF as the 3D representation (which is actually also studied in GS-LRM [3] and GRM [4], see below). However, I don't think this is a solid contribution for a publication.\n- Besides, the architecture of the proposed work is almost the same with GS-LRM [3] with only two differences of design choice: (1) whether using a ViT encoder or directly tokenizing the images; (2) whether using cross-attention or using concatenation and self-attention for feature fusion. This is also true for GRM [4].\n\nThe paper is a direct combination of these previous works (pose-free feed-forward NVS from sparse-views + feed-forward 3DGS prediction).\n\n- Moreover, methods that use latent 3D representations [5,6,7] (SRT is the first pose-free NVS method), reconstruction using sparse-view pose estimation methods [8,9,10,11] and more scene-specific/optimization-based pose-free methods [12] should be discussed.\n\n[1] Wang, Peng et al. “PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction.” ICLR 2024.\n[2] Jiang, Hanwen et al. “LEAP: Liberate Sparse-view 3D Modeling from Camera Poses.” ICLR 2024.\n[3] Zhang, Kai et al. “GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting.” ECCV 2024.\n[4] Xu, Yinghao et al. “GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation.” ECCV 2024.\n[5] Sajjadi, Mehdi SM, et al. \"Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations.\" CVPR 2022.\n[6] Kani, Bharath Raj Nagoor, et al. \"UpFusion: Novel View Diffusion from Unposed Sparse View Observations.\" ECCV 2024.\n[7] Xu, Chao, et al. \"Sparp: Fast 3d object reconstruction and pose estimation from sparse views.\" ECCV 2024.\n[8] Zhang, Jason Y., Deva Ramanan, and Shubham Tulsiani. \"Relpose: Predicting probabilistic relative rotation for single objects in the wild.\" ECCV 2022.\n[9] Zhang, Jason Y., et al. \"Cameras as rays: Pose estimation via ray diffusion.\" ICLR 2024.\n[10] Jiang, Hanwen, et al. \"Few-view object reconstruction with unknown categories and camera poses.\" 3DV 2024.\n[11] Lin, Amy, et al. \"Relpose++: Recovering 6d poses from sparse-view observations.\" 3DV 2024.\n[12] Truong, Prune, et al. \"Sparf: Neural radiance fields from sparse and noisy poses.\" CVPR 2023.\n\n2. Moreover, there some other weaknesses as follows.\n- Inappropriate baselines. DUST3R is not designed for NVS but for reconstruction with point cloud. Evaluating NVS performance on DUST3R is misleading. This is also the same for MAST3R/Splatt3R. If you want to compare with these works, please compare the geometry quality, i.e. point cloud error, rather than NVS performance. Moreover, DUST3R and its following works don't require camera intrinsics.\n- Unfair comparison. The proposed method is trained on a joint set of RealEstate10K, DL3DV and ACID, while all the baselines are trained on a different training dataset. Moreover, the model uses the weight initialization of models trained on large dataset (MAST3R/DUST3R/Croco).\n\n3. As there are not a lot directly comparable works on scene-level, I require the authors to include additional five experiment results.\n- On object-level, train the proposed model using the same data with the previously mentioned pose-free works and compare their performance. This also verifies whether MAST3R weight initialization generalize to object-level data.\n- On scene-level, train with only RealEstate10K+ACID for comparing with MVSplat.\n- On scene-level, train a variant of the proposed model, which is also conditioned on the camera poses of inputs (using the plucker ray representation rather than 6D pose representation). If the performance gap is small enough, the model has a strong capability of correlating the two images with the missing poses.\n- On scene-level, ablate the weight initialization by training with no weight initialization (from MAST3R/DUST3R/Croco) using the current training set (ACID+RealEstate10K+DL3DV). This experiment ablates whether the pose-free inference capability comes from initialization or your model learning process. Please include results for both pose-free and pose-conditioned variants of your model.\n\n4. Besides, the authors should notify that during training, the method still requires camera poses for rendering. \n\n5. Camera intrinsics are still needed during inference. The proposed way of merging camera intrinsic information is the same as LRM/PF-LRM, but no discussion.\n\n6. The method is only tested with two views, while the author claim \"sparse-view\". The authors should perform experiments with more input views, e.g. up to 6-8 views, to support the claim. Otherwise, you should rename the paper as \"unposed two images\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)