Gaussian Scenes: Pose-Free Sparse-View Scene Reconstruction using Depth-Enhanced Diffusion Priors
Authors: Soumava Paul
Prakhar Kaushik
Alan Yuille
Venue: Accepted by TMLR ()

Abstract: In this work, we introduce a generative approach for pose-free (without camera parameters) reconstruction of 360 scenes from a sparse set of 2D images. Pose-free scene reconstruction from incomplete, pose-free observations is usually regularized with depth estimation or 3D foundational priors. While recent advances have enabled sparse-view reconstruction of large complex scenes (with high degree of foreground and background detail) with known camera poses using view-conditioned generative priors, these methods cannot be directly adapted for the pose-free setting when ground-truth poses are not available during evaluation. To address this, we propose an image-to-image generative model designed to inpaint missing details and remove artifacts in novel view renders and depth maps of a 3D scene. We introduce context and geometry conditioning using Feature-wise Linear Modulation (FiLM) modulation layers as a lightweight alternative to cross-attention and also propose a novel confidence measure for 3D Gaussian splat representations to allow for better detection of these artifacts. By progressively integrating these novel views in a Gaussian-SLAM-inspired process, we achieve a multi-view-consistent 3D representation. Evaluations on the MipNeRF360 and DL3DV-10K benchmark dataset demonstrate that our method surpasses existing pose-free techniques and performs competitively with state-of-the-art posed (precomputed camera parameters are given) reconstruction methods in complex 360 scenes. Our code and datasets will be open-sourced upon acceptance.

Published in Transactions on Machine Learning Research (07/2025)
Gaussian Scenes: Pose-Free Sparse-View Scene Reconstruc- tion using Bepth-Enhanee Diffusion Priors
Soumava Paul, Prakhar Kaushik, Alan Yuille CCVL, Johns Hopkins University {spau127, pkaushi1, ayuille1)@jhu.edu
Reviewed on OpenRevicw:
Input
MASE3R + 3DGS
COGS
O urs
Ground Truth
PSNR: 10.67
PSNR: 9.31
PSNR: 13.91
Figure 1: Given sparse pose-free images as input, GScenes reconstructs a 3D scene in 5 minutes by iteratively fusing novel view renders and depth maps with an underlying 3D Gaussian reprosentation. Typical pose-free baselines built with geometric priors struggle with reconstructing 360%scenes from sparse inpuls due to the absence of generative priors. GScenes comprises a latent diffusion model capable of inpainting missing details and removing Gaussian artifacts in novel view renders, thereby enabling generation of full
Abstract
In this work, we introduce a generative approach for pose-free (without camera parameters) reconstruction of scense from sparse set of 2D images. Pose-free scene reconstruction from incomplete, pose-free observations is usually regularized with depth estimation or 3D foundational priors. While recent advances have enabled sparse-view reconstruction of large complex scenes (with high degree of foreground and background detail) with known camera poses using view-conditioned generative priors, these methods cannot be directly adapted for the posc-free setting when ground-truth poses are not available during evaluation. To address this, we propose an maage-to-image generative model designed to inpaint missing details and remove artifacts in novel view renders and depth maps of 3D scene. We introduce context and geometry conditioning using Feature-wise Linear Modulation (FiLM) modulation layers as lightweight alternative to cross-attention and also propose a novel confidence measure for 3D Gaussian splat represenlations to allow for better detection of these artifacts. By progressively integrating these novel views in Gaussian- -LAiaan---MMM---- Minniiedeeed process, we achieve a multi-view-coniistont 3D representation. Evaluations on the MipNeRF336 and DL3DV- benchmark dataset demonstrate that our method surpasses existing pose-free techniques and performs competitively with state-of-the-art posed (precomputed camera parameters

---

Published in Transactions on Machine Learning Research (07/2025)
given) reconstruction methods in complex 360 Our project page' provides additional results, videos, and code.
1 Introduction
Reconstructing high-quality 3D scenes from sparse images remains fundamental challenge in computer vision. While recent methods employ various priors to stabilize NeRFs (Mildenhall el al., 2020) or Gaussian splats (3DGS) (Kerbl et al., 2023) in under-constrained scenarios, they typically require accurate camera parameters derived from dense observations- a restrictive assumption for real-world applications. Pose estimation from sparse views is inherently challenging; both traditional Structure from Motion and recent foundational models (Wang et al., 2024; Leroy et al., 2024) struggle with insufficient matching features. Current pose-free free 3DGS approaches integrate monocular depth (Ranftl al., 2020), semantic segmentation (Kirillov al., 2023), 3D priors (Wang et al., 2024), but fail on complex 360pscenes with sparse coverage, highlighting the need for additional generative regularization.
Although most previous sparse view 3D scene reconstruction works are posed and require ground-truth camera poses (DSNeRF (Deng et al., 2022), DNGaussian (Li et al., 2024), SparseGS (Xiong et al., 2023), SparseN- eRF (Wang et al., 2023a), DietNeRF (Jain et al., 2021), FreeNeRF (Yang et al., 2023), RegNeRF (Niemeyer et al., 2022), DiffusioNeRF (Wynn and Turmukhambetov, 2023)); recent works like COGS (Jiang ct al., 2024) and Ipsta (Fan al., 2024) have found initial success in the pose-free sparse view reconstruction problem. Although they have been successful in reducing 3D reconstructioo--elltee artifacts such as blur, floaters, color, and streak-like artifacts, they lack generative capabilities for complete reconstrructi,, which requires robust priors from powerful generative models (Saharia et al., 2022; Rombach et al., 2022). In addition, most of these methods lack mechanisms to robustly identify and locate these artifacts in the reconstructed scenes. Recent methods like ZeroNVS (Sargent et al., 2024), Reconfuusion (Wu et al., 2024), and CAT3D (Gao* et al., 2024) incorporate 3D view conditioning for realistic extrapolation but depend on accurate poses and cannot be trivially extended to the pose-free problem. Gaussian Object (Yang al., 2024), iFusion (Wu al., 2023), and UpFusion (Nagoor Kani et al., 2024) do provide pose-free generative solutions but remain limited to only object reconstruction rather than scene reconstruction. Other generative solutions like ZeroNVS, ReconFusion, and CAT3D use stronger priors over regularization-basee techniques like FreeNeRF, RegNeRF, and DietNeRF but rely on large-scale 3D datasets, video datasets, and compute resources not available to the average rescarcher. Furthermore, works like ReconFusion and CAT3D remain closed-source with no access to their models, data or other reproducibility data, which limits their usage for the community. We summarize such existing methods and their use cases and attributes in Fig 2.
To alleviate these challenges, we present GScenes, an elficient ap- proach using 3D foundational and RGBD (RGB image and depth) generative priors for pose-free sparse-view reconstruction of com- plex scens. We first esti- mate point cloud and approx- imate camera parameters using MASt3R (Leroy et al., 2024), then jointly optimize Gaussians and cameras with 3DGS. Novel views generated from an elliptical trajec- tory fitted to the training views contain artifacts that our diffusion prior refines to further optimize the scone. We condition a Stable Diffusion UNet (Rombach et al., 2022) on estimated cameras, context, 3DGS renders, depth maps, and a confidence map capturing artifacts and missing details. Our proposed confidence
Generative Priors
Scone Reconstruction
Method
Pose-free
O Open-source
FreeNeRF, NeeF RegNeRF, DN- SparseGS, SparseNoRF
X
/
x
DiffusioNeRP, ZeroNVS
x
/
ReconFusion, CAT3D
x
x
/
/
Gauasian Object, iFusion, UpFusion
/
/
/ /
InstantSplat, COGS
/
/ x
GScenes (Ours)
Comparison of sparse-view reconstruction methods. Meth- ods are grouped based on their requirement for accurate camera poses, open-source availability, need for generative priors, and applicability large- scale scene reconstruction.
Figure 2:
httpss//gaansianeenessggtttttt
2

---

Published in Transactions on Machine Learning Research (07 (025)
measure is designed by combining per-pixel light transmittance with Gaussian density, which provides reli- able conditioning signal to the Stable Diffusion UNet the presence of emply regions and Gaussian artifacts in novel view renders and depth maps. For training our diffusion prior, we create our own dataset of 171, 461 samples using scenes from open-source multiview datascts. Each sample is a pair of clean RGBD images and images with emply regions and Gaussian artifacts, along with other conditioning signals (confidence map, clip features, etc). Augmented variants of both the Stable Diffusion VAE and UNet are finetuned using this synthetic dataset. Despite using weaker and cheaper generative priors than pose-dependent methods, we demonstrate competitive performance against closed source methods like ReconFusion (Wu et al., 2024) and CAT3D (Gao* et al., 2024) while outperforming other techniques without requiring million-scale multi-view or video data or extensive compute resources. Our contributions include:
An image-to-image RGBD generative model for synthesizing plausible novel views from sparse pose-free images, using lightweight FiLM modulation (Perez al., 2018) instead of cross-attention
confidence measure to detect artifacts in novel view renders, guiding our diffusion model toward effective novel view synthesis
Integration of diffusion priors with MASt3R's geometry prior, enabling efficient scene reconstruction previously roquiring 3D-aware video diffusion
Superior performance compared to recent regularization and generative prior-based open-source methods for 3D scene reconstruction
An open-source low-cost solution with lower data and compute requirements compared state-of- the-art posed reconstruction methods.
See Appendix for further discussion and Related Works (Appendix B).
2 Method
Input tmages
CLIP Features Gay (R?39)
FILM
Plicekr Embeddings Gea (E?.)
DOWN
UNet co
Noisy Latent
Lpiff
Add noise
Encoder
Confidence Map
Novel View Render
Novel View Depth
Ground Truth RGBD at Navel View
Figure 3: Overview of GScenes. We render 3D Gaussians fitted to our sparse set of M views from a novel viewpoint. The resulting render and depth map have missing regions and Gaussian artifacts, which are rectified by an RGBD imago-to-image diffusion model. This then acts as pscudo ground truth to spawn and update 3D Gaussians and satisfy the new view roistraints. This process is repeated for several novel views spanning the 360"scene until the representation becomes multi-view consistent.
This section begins with an overview of our method Soc. 2.1, detailing our approach for reconstructing a 3D scene from sparse set of uncalibrated 2D images. In Sec. 2.1, we describe how we initialize Gaussian point. cloud using MAS13R and 3DGS to provide a coarse 3D representation. Sec. 2.2 introduces RGBD
3

---

Published in Transactions on Machine Learning Research (07 (07/2025)
image-to-image generative model, which refines rendered novel views by correcting artifacts and filling in missing details. We propose confidence measure (Sec. 2.3) based cumulative transmittance and Gaussian density to guide the generative model toward unreliable regions, Sec. 2.4 outlines our synthetic dataset creation process for training with high-quality RGBD supervision, followed by depth-augmented autocncoder fineluning (Sec. 2.5) and UNet finetuning with synthesized data (Sec. 2.5.1) 1) to improve generation quality. Finally, Secs. 2.5.2 and 2.6 detail the inference pipeline and Gaussian optimization process, and Sec. 2.7 describes test-time pose refinement step.
2.1 Algorithm Overview
Problem Setup Given set of M images I (1 12....... IM} of an underlying 3D scene with unknown intrinsics and extrinsics, our goal is to reconstruct the 3D scene, estimate the camera poses of monocular camera at the training views, and synthesize novel views at evaluation time given by N unseen test images {1M-1, 1......,, IM+N]
Algorithm Gaussian Scenes Training
Require:
Sparse image set I = {l...., Jarh, densification interval k, iterations N 1: Initialize 3D Gaussian primitives g from MASt3R 3D point cloud P
2: Optimize G using 3DGS on input views I and estimated cameras Ttrain = JM for 1k iterations
I<I
b Initialize training images
for t= t..,..,,N do if t mod k = 0 then
6:
Sample novel pose nnoeet from elliptical trajectory fitted to Ttrain (I, R (9)
7:
> Render image and depth map
8:
Compute confidence map (Eq. 2)
9:
Extract semantic context & geometry embeddings {Cetsp Cgeo
> Sec 2.2.2
10:
(Tref, Dref)= + Refine(I, D C, Celip: Cgeo)
Refine 1, D with generative priors (See 2.5.2) Synthesized novel view added training sample
11:
i- (ure)
12:
end if
13:
Sample pose Ti from Ttrain
14:
Render (/x, Dz) + (9)
15:
Optimize 3D Gaussians G
b Eq. or 3DGS loss
16:
end for
An overview of our method is given in Fig and Alg We initialize GSoenes with an incomplete dense Gaus- sian point cloud reconstruction from sparse input images using MASt3R and 1k iterations of 3DGS. Note that InstantSplat, proposes the same framework for sparse-view reconstruction, but with DUSt3R initialization. We choose MASt3R to initialize scene geometry instead due to its superior performance the sparse-view setting. We use this incomplete scene representation as an implicit geometric prior and sample novel views along smooth elliptical trajectory fitted to training views. An example tra- jectory is shown in Fig 4. We then use our RGBD generative prior to synthesize plausible novel views. In addition to CLIP features of source images for context and plücker embeddings of source and target cameras for geometric conditioning, we devise a novel 3DGS confidence measure effectively guide our generative model towards empty regions and potential artifacts novel view render. We then run 10k iterations of 3DGS optimization, sampling a novel view before each densification step (Fu et al., 2023) to obtain final scene representation.
18
65
Optimized Training P Poses from MASTER 3DGS
cameras eliptical trajectory
Figure Camera Trajectory Visualization for Novel View Synthesis in pose-free sparse-view setting.
4

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Gaussian Point Cloud Initialization The MASt3r initialization gives us pixel-aligned denso-stereo point cloud P E R camera intrinsies {K; E 1 Ji=l and extrinsios (E, = [R for our M input. images. Nevertheless, both P and the estimated poses demonstrate sub-optimal alignment compared to those generated by COLMAP from the dense observation dataset. We cannot also use COLMAP for camera or geometry estimation as the method fails when there is minimal overlap between training images, as in our setting. Consoquently, similar to the approach in InstantSplat, we initialize 3D Gaussians at each location in the globally aligned point cloud P. We then jointly optimize both the Gaussian altributes and camera parameters over the 1/ iterations. The MASt3R point cloud is already quite dense (15--MM points), which alleviates the need for any form of Adaptive Density Control (Kerbl et al., 2023).
2.2 RGBD Generative Priors for Novel View Synthesis (NVS)
Reconstructing complete 3D scenes from sparse observations requires inferring content in unobserved regions fundamental challenge that geometric regularization and 3D priors alone cannot adequately address. We introduce diffusion-based generative approach that leverages 2D image priors to synthesize plausible content.
2.2.1 Generative Model Architecture
Despite our initial geometric reconstruction, sparse input views inevitably result in regions with no Gaussian primitives ("O-Ganssians"). causing empty arcas and artifacts in novel views. Unlike reguaaizatton--asse methods that merely constrain optimization without generating content, our approach directly synthesizes missing scene details.
Our model comprises (1) A variational auloencoder (encoder &, decoder D) operating in compressed latent space. (2) A UNet denoiser Eo predicting noise in diffused latent 2t (3) multi-modal conditioning incorporating RGBD renders, confidence maps, semantic context, and geometric information.
The UNet ep receives four inputs: an artifact-laden RGBD image f, a confidence map C identifying unreliable regions, CLIP features Celip of Z providing semantic context, and camera encodings Cgeo establishing geometric relationships between views.
2.2.2 Multi-modal Conditioning
We initialize with pretrained
model weights and expand the first convolutional layer to accept additional inputs by concatenating the noisy latent 2,, the encoded RGBD image {f),, the confidence- weighted encoded image E(î ((f-C) C) and the downsampled confidence map Ć. To ensure view coherence and geometric consistency, we incorporate following conditioning signals:
Semantic Context:
CLIP features Gelip Mx from source images serve as semantic anchors, ensuring generated content remains consistent with observed scene elements.
Geometric Information:
For each camera with center o and forward axis d, we compute its plücker coordinates r (d, x d) E RR6 and apply frequency encoding for obtaining higher-dimensional features:
r ト [r, (rr),, cos(ffirr, sin(ffrr),
(1)
where K = 6 is the number of Fourier bands, and ft are equally spaced frequencies. This yields a 78- dimensional embedding for each camera (Croo E [R(M+1)x77H) capturing geometric relationships between viewpoints. Plücker coordinates were originally introduced by LFNs (Sitzmann et al., 2021) for per-pixel parameterization of a ray. We instead obtain a single representation per camera using extrinsic parameters E, for obtaining and d.
2.2.3 Parameter- Efficiter-Efficient FiLM Conditioning
We employ Feature-wise Linear Modulation (FiLM) instead of cross-attention for incorporating context and geometry information, achieving both computational efficiency and strong performance. We process context and geometry embeddings through self-attention to capture inter-view relationships as: G SelfAttention(s); E geo. We then generate scaling and shifting parameters via layer-specific
5

---

Published in Transactions on Machine Learning Research (07/2025)
networks as: 3 FC (catta) Finally, modulate the UNet feature maps through element- wise operations as: F mod = -00 F(() + 3(0
Critically, we apply FILM modulation only to down and mid blocks of the UNet- not up blocks based empirical evi- dence showing this selective application yields optimal results (Fig. 5).
MAS13R + 3DGS
Diffusion Sample (UpBlock Conditioning)
Diffusion Sample (Ours)
Our FiLM-bassed approach requires only 8.14M parameters (7.38M for CLIP lea- tures, 758K for pose embeddings) com- pared to 29.8M for equivalent cross- allention implementation tion while maintaining comparable qual- ity, enabling more efficient training and faster inference during iterative recon- struction.
3 views
3x reduc-
Figure 5: Incorporating context and geometry conditioning in the up blocks of the UNet negatively impacts latent and subsequent image reconstruction.
2.3 Pixel-Aligned Confidence Map
To guide our generative diffusion model in identifying problematic and artifact-ridden regions novel views, introduce pixel-aligned confidence measure combining transmittance with Gaussian density:
C = - log(T;; +) X Reontrib
where T, (1, a4) o;) represents light. transmission without Gaussian interac- tion, Neontrih counts contributing 3D Gaussians, and > 0 prevents logarith- mic singularities. This formulation cap- tures two complementary reliability sig- nals: (1) low transmittance indicates sig- nificant Gaussian interactions, suggest- ing higher rendering confidence, and (2) consensus among multiple Gaussians val- idates pixel reliability through primitive agreement.
MAS13R + 3DGS
Diffusion Sample (Conf. Diffusion Sample (Ours) Map (Liu et al., 2024))
views
Unlike 3DGS-Enhancer (Liu et al., 2024), (one of the few methods that propose con- fidence measures for sparse-view scene re- construction), which assumes that well- reconstructed areas contain small-scale Gaussians, our mcasure remains effective for monotonous lextures, where fine-grained Gaussian representation is unnecessary. Such regions are usually represented by a few high-opacity large-scale Gaussians (low Reontrib, but also low T;) and hence not flagged as low-confidence regions by our confidence measure. Fig. demonstrates how our approach accurately identifies both empty regions and reconstruction artifacts while avoiding false positives. The significance of this improved confidence measure is evident in Fig. 6, where models trained with previous confidence formulations produce implausible novel views due to misleading confidence signals.
Figure 6: As an ablation for our confidence mcasure, we train variant of our diffusion model using the confidence measure proposed in 3DGS Enhancer (Liu et al., 2024). Conditioning the UNet with this inaccurate confidence measure leads to implausible NVS.
2.4 RGBD Dataset Creation
6

---

Published in Transactions on Machine Learning Research (07/2025)
Sparse Render
Couf. Measure (Liu et al., 2021)
Conf. Measure (Ours)
Dense Render
3 views
Figure Confidence Measure comparison with 3DGS-En Enhancer (Liu et al., 2024). Our confidence map accurately identifies artifacts and O-Gaussian regions the sparse-view (darker pixels) while Liu et al. (2024) incorrectly attributes high confidence regions with overlap of small-scale Gaussians. NVS render from a densely fitted 3DGS representation is provided for reference.
For training the additional weights in Eg for RGBD image-to-image diffusion, we rely on a set X {(T', Cr,C', calip: (geo))ii), each containing a clean RGBD image I, an RGBD image with artifacts f and the corresponding confidence map Cτ CLIP features of source images Celip E R Mx78, and plücker embeddings of source and target cameras eyeo R(M++))x to "teach" " the diffusion model how to inpaint missing details and detect Gaussian artifacts guided by the confidence map, context and geometry features and generate clean version of the conditioning image. For this, build dataset generation pipeline comprising high-quality 3DGS model fitted to dense views, a low-quality 3DGS model fitted to few views, and camera interpolation and perturbation modules to use supervision of the high-quality model at viewpoints beyond ground truth camera poses. For given scene, we fit sparse models for M E (3,6,9 18) nunber of views. We render using the high-quality model and I' C using the low-quality model. We save the CLIP features Cetip and plücker embeddings of the M sparse views and target view per sample for conditioning co Our fino-tuning setup is illustrated in Fig 12 (Sec C).
2.5 Auggented Autoencoder Finetuning
Input RGB
Recon.
RGB
Input Depth
Recon. Depth
For encoding and decoding RGBD images, we customize a Variational AutoFncoder by introducing additional channels in the first and last convolutional layers of the Stable Diffusion VAF. A similar approach was followed by Stan et al. (2023), where their KL-autencoder was finetuned with triplets containing RGB images, depth maps, and captions to train the weights in the new channels. However, the depth maps used for fine-tuning this VAE were estimated using MiDaS (Ranftl et al., 2020), which are usually blurry monocular depth estimates. As such, reconstructing RGBD images using this VAE produces depth maps with extreme blur not ideal for a scene reconstruction problem. Hence, further finetune this VAE with our syn-
LDM3D VAE
Ours (finetuned)
Figure 8: RGBD reconstruction comparison of our finetuned VAE with the LDM3D VAE (Stan et al., 2023). Unlike LDM3D, our VAE finetuned synthetic dataset preserves sharp details and edges of input depth map while also preventing color artifacts in RGB.
7

---

Published in Transactions on Machine Learning Research (07 (075)
thetic dataset, which contains depth maps rendered by the differentiable 3DGS rasterizer, giving accurate pixel depth with high-frequency details. Specifically, we use the following objective:
Lautoencoder -min max (Cr max D(E(x))) - Laav Ladv(D(E(=))) + log (D)()) + E D2
(3)
where Free combination of L1, perceptual losses for the RGB channels, and Pearson Correlation Coefficient (PCC), TV regularization losses for the depth channels. Lady is the adversarial loss, D& is a patch-base based discriminator loss, and frgg, is the KL-rogularisation loss. The incorporation of PCC and TV terms for the depth channels leads to better retention of high- freeu-fryuency details in the reconstructed depth map, as observed in Fig 8. We finetune this VAE on a subset of our dataset for 5000 training steps with batch size 16 and learning rate le-05.
2.5.1 UNet Finetuning
With our finetuned autoencoder, we next train the UNet with the frozen VAE on X' with this objective:
Laifs = (AAU (lee coc
C)2 Catia catt )
(4)
2.5.2 RGBD Novel View Synthesis
At inference time, given render and depth map with artifacts, and confidence map, CLIP features and camera embeddings for conditioning, the finetuned UNet eg learns to predict the noise in latent z₁ according to ~ (tmins tmaz] as:
& = co 2,,0 2, clip Cattm: cattn
+ sr(e0 (2 C, C)2 Cattm: geo c t, Ø, C, Ø, clip
Cattas Cattn )
(5)
+ *cc(E0(2 1, 2, C, Ø Cattn (ate)) (0 I, Ø, R0, R, clip 200
clip
where 8, and se are the RGBD image and confidence map guidance scales, dictating how strongly the final multistep reconstruction agrees with the RGBD render î and the confidence map C, respectively. After k 20 DDIM (Song et al., 2021) sampling steps, we obtain our final RGBD render by decoding the denoised latent Fs [k, Dz] D(zo).
2.6 Scene Reconstruction with Generative Priors
Our generative diffusion model provides a generative prior to infer plausible detail in unobserved regions. Despite view conditioning using pose embeddings, the generated images at novel poses lack complete 3D consistency. For this, devise iterative strategy where first sample novel views along an elliptical trajectory fitted to the training views. We initialize the Gaussian optimization with the set of Gaussians G fitted to the training views (Sec 2.1). Each novel view is added to the training stack at the beginning of every densification step to encourage the optimization to adjust to the distilled scene priors. At every iteration, we sample either an observed or unobservod viewpoint from the current training stack. We bring back Adaptive Density Control to encourage densification of Gaussians in O-Gaussian regions. We employ the 3DGS objective for the training views. For novel views, we employ the SparseFusion (Zhou and Tulsiani, 2023) objective in the RGB space and PCC loss for the rendered and denoised depths.
Csampte (16) Er, (w(()(())=>>>> ix + C.p(I, 1+))
PCC(Dx, (.))
where Cp is the perceptual loss (Zhang et al., 2018), w(t) a noise-dependent weighting function, Lxs Drr are the rendered image and depth al novel viewpoint and f,, Dr are their rectified versions obtained with our diffusion prior. The PCC loss is defined as PCC(D,, D=) = 1 aD. Dz
Do)

---

Published in Transactions on Machine Learning Research (07 (07/2025)
2.7 Test-time Pose Alignment
GScenes reconstructs plausible 3D scene from pose-free source images. However, reconstruction with few views is inherently ambiguous as several solutions can satisfy the train view constraints. Hence, the reconstructed scene would probably be quile different from the actual scene from which M views were sampled. Hence, for a given set of test views, following prior work (Fan et al., 2024; Jiang al., 2024), we freeze the Gaussian attributes and optimize the camera pose for each target view by minimizing photometric loss between the rendered image and test view. Following this alignment step performed for 500 iterations per test image, we evaluate the NVS quality.
MASt3R 3DGS
RegNeRF
DiffusioNeRF
ZeroNVS* 0
COGS
Ours
Ground Truth
Homsail (0)
Counter (3)
Hicvcle(S) 33
Garden 66)
Figure 9: Qualitative comparison of GScenes with few-view methods on the MipNeRF360 dataset.
Our approach consistently fairs better in recovering image structure from foggy geomelry, where baselines typically struggle with "floaters" and color artifacts.
MASt3R + 3DGS ZeroNVS*
DiffusioNeRF 0
COGS
Ours
Ground Truth
WI
Figure 10: Qualitative comparison of GScenes with few-view methods on the DL3DV benchmark. Our method achieves plausible reconstruction in unobserved arcas of complex scenes where even posed reconstruction techniques struggle.

---

Published in Transactions on Machine Learning Research (07/ (025)
3 Experiments
We compare GScenes with state-of-the-art pose-free and posed sparse-view reconstruction methods in Fig 9, 10 and Table 1, 2. We also ablate the different components and design choices of our diffusion model.
3.1 Experimental Setup
Evaluation Dataset We evaluate GScenes on the scenes of the MipNeRF360 dataset (Barron al., 2022), and 15 scenes (out of 140) of the DL3DV-10K benchmark dataset. For MipNeRF3360, We pick the M-view splits proposed by ReconFusion and CAT3D and evaluate all baselines the official test views where every 8th image is held out for testing. For DL3DV-10K scenes, create M-view splits using a greedy view-selection heuristic for maximizing scene coverage given set of dense training views, similar to the heuristic proposed in Wu et al. (2024). For test views, we hold out every 8th image as in MipNeRFF300. Additionally, we pick the plant scene of CO3D for qualitative comparison with ReconFusion and CAT3D.
Fine-tuning Dataset We fine-tune our diffusion model on mix of 1043 scenes encompassing Tanks and Temples (Knapitsch al., 2017), CO3D (Reizenstein al., 2021), ; Deep Blending (Hedman et al., 2018), and the 11 subset of DL3DV-10K (Ling et al., 2024) to obtaín total of 171, 461 data samples. We first train 3DGS on sparse and dense subsets of each scene for M E {3, 6, 9, 18]. For M > 18, novel view renders and depth maps mostly show Gaussian blur as artifacts. Finetuming this model takes about 4-days on a single A6000 GPU.
Metrics Our quantitative metrics are used to evaluate two tasks - quality of novel views post roconstruction and camera pose estimation. For the former, we compute groups of metrics FID (Heusel et al., 2017) and KID (Bińkowski et al., 2018) due to the generative nature of our approach, perceptual metrics LPIPS (Zhang et al., 2018) and DISTS (Ding et al., 2020) to measure similarity in image structure and texture in the feature space, and pixel-aligned metrics PSNR and SSIM. However, PSNR and SSIM are not suitable evaluators of generative techniques (Chan et al., 2023; Sargent et al., 2024) as they favor pixel-aligned blurry estimates over high-frequency details.
Baselines We compare our approach against 8 baselines, but our main comparison is with 2 pose-free methods - COGS and our 3D reconstruction engine- MASt3R + 3DGS, The other 6 methods are posed and included for stronger comparison. Out of these, FreeNeRF, RegNeRF, and DiffusioNeRF are sparse-view regularization methods based on NeRFs. ZeroNVS reconstructs complete 3D scene from single image using novel camera normalization scheme and anchored SDS loss. We use the ZeroNVS* baseline introduced in ReconFusion, designed to adapt ZeroNVS to multi-view inputs. Further, we also provide the reported quantitative results from ReconFusion and CAT3D, which are closed source and unverifiable, to show that our method gets close to their performance despite using a fraction of their compute. We also pick the relevant. scenes and test views from the papers for qualitative comparison.
3.2 Implementation Details
GScenes is implemented in PyTorch 2.3.1 on single A6000 GPUs. Images and depth maps are rendered at 400-600 pixels to align with Stable Diffusion's 's resolution. The diffuusio model is finetuned for 100k iterations (batch size 16, learning rate le-4) with conditioning element dropout probability of 0.05 for CFG.
Following InstantSplat, we fit 3D Gaussians to sparse inputs and MASt3r point clouds for 1k iterations to obtain G. We use classifier-free guidance scales 81 = sc 3.0 and sample with k: DDIM steps. We linearly decay wa from 1 to 0.01 and Lsampie weight from 1 to 0.1 over 10k iterations. GScenes completes full 3D reconstruction in approximately 5 minutes on single A6000 GPU.
3.3
Comparative Results
We report qualitative and quantitative comparisons of GScenes against all related baselines in Fig 9, 10 and Tables and 2. Additional qualitative results are provided in Fig 14 and 15 (Sec E). Out of the basclines, only MASt3R 3DGS and COGS are pose-free techniques, but COGS relies on ground truth camera intrinsics while GSoenes and MAS13R + 3DGS do not. In classical metrics (Tab 1), baseline M MAS
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Render
Base
Base + Conf.
w/o CLIP
Bousai 3)
w/o Conf.
Cross-Attn
Ours
Ground Truth
Bunssi
Render
Base
Base + Conf.
w/o CLIP
Trechill
w/o Conf.
Cross- Attn
Ours
Ground Truth
Trechill33
Figure 11: Ablation Study with the 3-view splits of Bonsai and Treehill scenes from MipNeRFF300. Images show samples from different variants of our diffusion model based on conditioning.
Table 1: Quantitative comparison with state-of-the-art sparse-view reconstruction techniques on classical metries. Rod, orange, and yellow indicate best, socond-best, and third-best performing methods, respectively. indicates posed methods.
PSNR
SSIM t
LPIPS
Method
3-view
6-view
9-view
3-view
6-view
9-view
3-view
6-view
9-view
FreeNeRF 11.886
12.874
13.673
0.180
0.218
0.236
0.675
0.654
0.638
RegNeRF
12.294
13.204
13.796
0.182
0.207
0.217
0.668
0.656
0.625
MipNeRF360
DiffusioNeRII
9.996
12.025
12.567
0.159
0.211
0.213
0.715
0.648
0.659
NVS
11.991
11.817
11.729
0.142
0.129
0.128
0.710
0.705
0.694
ReconFusion
15.50
16.93
18.19
0.358
0.401
0.432
0.585
0.544
0.511
CAT3D
16.62
17.72
18.67
0.377
0.425
0.460
0.515
0.482
0.460
MAS.3R 3DDGs
T2.585
14.285
15.042
0.231
0.279
0.310
0.593
0.550
0.531
COGS
11.814
12.095
12.555
0.183
0.210
0.228
0.619
0.630
0.605
GScenes
13.809
14.507
14.831
0.265
0.275
0.282
0.547
0.530
0.521
DiffusioNeRI
11.715
13.245
14.504
0.245
0.317
0.376
0.606
0.544
0.500
DL3DV
12.145
12.560
12.774
0.202
0.206
0.209
0.679
0.658
0.648
MASt3R + 3DGS
13.217
16.417
17.840
0.371
0.476
0.531
0.443
0.359
0.326
COGS
11.057
12.350
13.300
0.241
0.282
0.318
0.598
0.564
0.539
GScenes
14.751
15.761
16.199
0.340
0.372
0.394
0.410
0.373
0.360
11

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 2: Quantitative comparison with few-view reconstruction techniques on metrics suited for generative reconstruction. Red, orange, and yellow indicate best, recond-best, and third-best performing methods.
FID 6-view
KID 6-view
DISTS + 6-view
Method
3-view
9-view
3-view
9-view
3-view
9-view
FreeNeRF
354.244
353.480
346.874
0.261
0.271
0.282
0.388
0.369
0.365
MipNeRF360
RegNeRF
358.855
360.380
338. 183
0.281
0.301
0.264
0.407
0.405
0.377
DiffusioNeRF
370.346
347.522 242.222
342.373
0.293
0.267
0.257
0.441
0.386
0,402
ZeroNVS* o
356.395
350.920
343.930
0.283
0.296
0.300
0.433
0.426
0.413
MASt3R + 3DGS
294.819
252.830
269
0.210
0.164
0.133
0.303
0.273
0,264
COGS
251.694
284.829
270.
0.158
0.183
0.159
0.297
0.313
0.303
GSeenes
163. 747
156.368
160.919
0.053
0.052
0.055
0.234
0.227
0,230
DiffusioNeRF
251.341
210.596
199.629
0.179
0.154
0.147
0.352
0.314
0.300
DL3DV
ZZroNNS* O
267.407
243.095
233.998
0.214
0.200
0.199
0.372
0.346
0.330
MAS+3R +3R3R 3DGS
174.665
136.013
118.675
0.117
0.077
0.062
0.247
0.204
0.187
COGS
249.530
224.605
207-782
0.198
0.159
0.128
0.308
0.295
0.283
GSeenes
107.750
107.329
105.871
0.036
0.046
0.046
0.193
0.190
0.189
3DGS shows better values than GScenes, as these metrics are usually inadequate for evaluating generative NVS techniques. Nevertheless, we include them for completeness and observe that either our method or the baseline is the best pose-free, open-source method. CAT3D and Reconfusion appear to do better in a posed setting, but we cannot confirm these values nor do a fair comparison in a pose-free setting as they are closcd-source. For metrics tailored for generative reconstruction (Tab 2), method comprehensively outperforms the baseline well as all related methods for both MipNeRF360 and DT.3DV-Benchmark datasots. Due to the unavailability of open-source code, evaluating ReconFusti and CAT3D these measures is unfortunately not possible. However, we do provide a qualitative comparison with both methods in Sec D. Qualitative comparisons with other open-source posed and pose-free free methods clearly show that GScenes far more adept at generating plausible scene content from partial observations.
3.4 Ablation Studies
In Fig 11 and Tab 3 (Sec F), thoroughly ablate different components of our diffusion model. We pick the bonsai and treehill scenes and their 3-view splits for this experiment. The 1s out of 8 columns shows the initial novel-view render obtained from the 3D reconstruction pipeline (MASt3r + 3DGS). The Base variant only performs image-to-image diffusion with no other conditioning, and this already provides strong baseline for inpainting and artifact elimination in novel-view renders. Without CLIP context guidance, the model fails to adhere to the semantics of the input images when inpainting missing details. Without our confidence measure, the model typically fails to differentiate between image structures and artifacts, often producing implausible images despite context and geometry conditioning. Additionally, we train Cross- variant where context and geometry conditionings are incorporated using cross- cross-attentin attention instead of FiLM
modulation layers in the down and middle blocks of the UNet. Our method achieves similar quality and FID (O//KID/DD KID/DISTS scores with more than 3x fewer additional trainable parameters. Similar to Tab 1, GSeenes performs& a little worse on PSNR, SSIM, and LPIPS compared to the baseline, as these metrics penalize the generation of plausible scene details, slightly dissimilar from the original scene. Nevertheless, the qualitative results clearly show the benefits of our proposed context, geometry, and confidence map conditioning.
4 Conclusion
In this work, we present GSeenes where we integrate image-to-image RGBD diffusion model with pose-frec reconstruction pipeline in MASt3R to reconstruct a 360 scene from few uncalibrated 2D images. We introduce context and geometry conditioning through FiLM layers, achieving similar performance as a cross-attention variant. We also introduce a pixel-aligned confidence measure to further guide the diffusion model in uncertain regions with missing details and artifacts. Our experiments show that GScenes outperforms existing pose-free reconstruction techniques in scene reconstruction and performs competitively with state-of-the-art posed sparse-view view ruction methods.
12

---

Published in Transactions on Machine Learning Research (07 2025)
5 Acknowledgement
This work is supported by Army Research Laboratory award W911NF2320008 and ONR with N00014-21-1- 2812.
References
Titas Anciukevičius, Fabian Manhardt, Federico Tombari, and Paul Henderson. Denoising diffusion via image- based rendering. In The Twelfth International Conference on Learning Representations, 2024. 18
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022. 10
Wenjing Bian, Zirui Wang, Kejie Li, Jiawang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. 2023, 18, 23
Mikotaj Binkkoski, Dongal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In ICLR, 2018. 10
Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein, GeNVS: Generative novel view synthesis with 3D-aware diffusion models. ICCV, 2023. 10
David Charatan, Sizhe Li, Audrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In CVPR, 2024. 18
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, 2021. 18
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction. In ICCV, 2023a. 18
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In ICCV, 2023b. 18
Yu Chen and Gim Hee Lee. Dbarf: Deep bundle-adjusting generalizable neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24-34, 2023. 18
Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. 2024. 18
Jaeyoung Chung, Jeongtaek Oh, and Kyoung Mu Lee. Deptlereguuarized optimization for 3d gaussian splatting in few-shot images. In Xiv 2023. 17
Congyue Deng, Chiyu Jiang, Charles R Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al. Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors. In CVPR, 2023. 18
Kangle Deng, Andrew Liu, Jun- Yan Zhu, and Deva Ramanan. Depth-supervised NeRE: Fewer views and faster training for free. In CVPR, 2022. 2, 17
Keyan Ding, Kede Ma, Shiqi Wang, and Eero Simoncelli. Image quality assessment: Unifying structure and texture similarity. In IEEE TPAMT, 2020. 10
Zhiwen Fan, Wenyan Cong, Kairun Wen, Kevin Wang, Jian Zhang, Xinghao Ding, Danfei Xu, Boris Ivanovie, Marco
Pavone, Georgios Pavlakos, Zhangyang Wang, and Yue Wang. Instantsplat: Unbounded sparse-view pose-free gaussian splatting in 40 seconds. In arXi 2024. 2, 9, 18
Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A. Efros, and Xiaolong Wang. Colmap-free 3d gaussian splatting. 2023. 4, 18
Ruiqi G Aleksander Holyuski*, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jouathan T. Barron, and Ben Poole", Cat3d: Create anything in 3d with multi-view diffusion models. arXiv, 2024. 2,
13

---

Published in Transactions on Machine Learning Research (07/ (07/2025)
Pengsheng Guo, Miguel Angel Bautisla, Alex Colburn, Liang Yang, Daniel Ulbricht, Joshua M. Susskind, and Qi Shan. Fast and explicit neural view syuthesis. In WACV, 2022. 18
Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. Deep blending for free-viewpoint image-based rendering. In ACM TOG, 2018. 10
Philipp Henzler, Jeremy Reizenstein, Patrick Labatut, Roman Shapovalov, Tobias Ritschel, Andrea Vedaldi, and David Novotny. Unsupervised learning of 3d object categories from videos in the wild. In CVPR, 2021. 18
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In NewrIPS, 2017. 10
Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Vitor Guizilini, Thomas Kollar, Adrien Gaidon, Zsolt Kira, and Rares Ambrus. Neo 360: Neural fields for sparse view syuthesis outdoor scenes. In ICCV, 2023.
Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on diet: Semantically consistent few-shot view synthesis. ICCV, 2021. 2, 17
Kaiwen Jiang, Yang Fu, Mukund Varma T, Yash Belhe, Xiaolong Wang, Hao Su, and Ravi Ramamoorthi. A construct-optimize approach to sparse view synthesis without camera pose. SIGGRAPH, 2024. 2, 9,
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering, In ACM TOG, 2023. 2, 5, 17, 18
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yer Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.026; 2023. 2, 17
Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: benchmarking large-scale scene reconstruction. In ACM TOG, 2017. 10
Vincent Leroy, Yohann Cabon, and Jerome Revaud. Grounding image matching in 3d with mast3r, 2024. 2, 17
Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization. In CVPR, 2024. 2, 17
Chen-Hsuua Lin, Wei-Chin Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In IEEE International Conference Computer Vision (ICCV), 2021. 18
Chen-Hsuar Lin, Jun Gao, Luminn Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung-Yi Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023a. 18
Kai-En Lin, Lin Chen, Wei-Sheng Lai, Tsung-Yi Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi. Vision transformer for nerf-based view synthesis from a single input image. In WACV, 2023b. 18
Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D13dv-10k: A large-scale scene dataset for deep learning-based 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22160-22169, 2024. 10
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to object. ICCV, 2023. 18
Xi Liu, Chaoyi Zhou, and Siyu Huang. 3dgs-cuhancer: Enhancing unbounded 3d gaussian splatting with view-consistent 2d diffusion priors. arXiv preprint 2ttb:://10.1166666 2024. 6,
Luke Keras-Kyriazi, Christian Rupprecht, and Andrea Vedaldi. Pc2: Projection-conditionee point clond diffusion for single-image reconstruction. In CVPR, 2023.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 17
Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias NieBner. Diffiff: Rendering-guidee 3d radiance field diffusion. In CVPR, 2023. 18
14

---

Published in Transactions on Machine Learning Research (07/ (07/2025)
Bharath Raj Nagoor Kani, Hsin- Lee, Sergey Tulyakov, and Shubham Tulsiani. Uplusion: Novel view diffusion from unposed sparse view observations. In European Conference on Computer Vision (ECCV), 2024. 2
Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, 2022. 2, 17
Soumava Paul, Christopher Wewer, Bernt Schiele, and Jan Eric Lenssen. Sp2360: Sparse-view 360 scene reconstruction using cascaded 2d diffusion priors. In ECCV 2024 Workshop on Wild SD: 3D Modeling, Reconstruction, and Generation in the Wild, 2024. 17, 18
Ethan Perez, Florian Strub, Harm Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with general conditioning layer. In AAAI, 2018. 3
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023. 18
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 17
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. In IEEE TPAMI, 2020. 2, 7, 17
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In ICCV, 2021.
10
Barbara Roessle, Jonathan T. Barron, Ben Mildenhall, Pratul P. Srinivasan, and Matthias Niefner. Dense depth priors for neural radiance fields from sparse input views. In CVPR, 2022. 17
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolttion image synthesis with latent diffusion models. CVPR, 2022. 2, 17, 19
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Chasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-inage diffusion models with deep language underslanding. In NewrIPS, 2022. 2
Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, and Jiajun Wu. ZeroNVS: Zero-shot 360-degree view synthesis from a single real image. In CVPR, 2024. 2, 10,
Philipp Schröppel, Christopher Wewer, Jan Eric Lenssen, Eddy 1lg, and Thomas Brox. Neural point cloud diffusion for disentangled 3d shape and appearance generation. In CVPR, 2024. 18
J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In CVPR, 2023. 18
Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Proc. NewrIPS, 2021. 5
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. 8
Gabriela Ben Melech Stan, Diana Wofk, Scottie Fox, Alex Redden, Will Saxton, Jean Yu, Estelle Aflalo, Shao-Yen Tseng, Fabio Nonato, Matthias Muller, and Vasudev Lal. Ldm3d: Latent diffusion model for 3d, 2023.
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Viewset diffusion: (0 -o-ditio-oond 3d generative models from 2d data. International Conference on Computer Vision 2023. 18
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Lin, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2024. 18
Alex Trevithick and Bo Yang, Grf: Learning a general radiance field for 3d scene representation and rendering. In ICCV, 2021. 18
15

---

Published in Transactions on Machine Learning Research (07/2025)
Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shol novel view synthesis. In JCCV, 2023a. 2, 17
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, 2023b. 18
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024. 2, 17
Christopher Wewer, Eddy 1gg, Bernt Schiele, and Jan Eric Lenssen. SimNP: Learning self-similarity priors between neural points. In ICCV, 2023. 18
Chin-Hsuan Wu, Yen-Chun Chen, Bolivar Solarte, Lu Yuan, and Min Sun. ifusion: Inverting diffusion for pose-free reconstruction from sparse views. arXiv preprint arXiv:2312. 10122.77225 17250, 2023. 2
Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Holynski. Reconfusion: 3d reconstruction with diffusson priors. In CVPR, 2024. 2, 3, 10, 18
Jamie Wynn and Daniyar Turmukhambetor. DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models. In CVPR, 2023. 2, 17
Haolin Xiong, Sairisheek Muttukuru, Rishi Upadhyay, Pradyumna Chari, and Achuta Kadambi. Spansegs: Real-time 360 sparse view synthesis using gaussian splatting. In arXiv, 2023. 2, 17
Chen Yang, Sikuang Li, Jiemin Fang, Ruofan Liang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian. Gaus- sianobject: High-quality 3d object reconstruction from four views with gaussian splatting. ACM Transactions on Graphics, 2024. 2
Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In CVPR, 2023. 2, 17
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one or few images. In CVPR, 2021. 18
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018. 8, 10
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through poxet-voxel diffusion. In ICCV, 2021. 18
Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. Tn CVPR, 2023. 8, 18
Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang, Fsgs: Real-Line few-shot view synthesis using gaussian splatting. In Xiv, 2023. 17
16

---

Published in Transactions on Machine Learning Research (07/2025)
A Problem Introduction and Discussion
Obtaining high-quality 3D reconstructions or novel views from a sparse set of images has been a long-standing goal in computer vision. Recent methods for sparse-view reconstruction often employ generative, geometric, or semantic priors to stabilize the optimization of NeRF's (Mildenhall et al., 2020) or Gaussian splats (Kerbl et al., 2023) in highly under-constrained scenarios. However, they typically assume access to accurate intrinsic and extrinsic parameters, often derived from dense observations. This reliance ground-truth poses is restrictive assumption, making these methods impractical for real-world applications. Morcover, pose estimation from sparse views is error-prone; both traditional Structure from Motion approaches and recent 3D foundational models (Wang et al., 2024; Leroy et al., 2024) struggle with insufficient matching features between image pairs. In response, recent posc-free approaches using 3D Gaussian splatting (3DGS) integrate monocular depth estimation (Ranftl et al., 2020), 2D semantic segmentation (Kirillov et al., 2023), or 3D foundational priors (Wang et al., 2024), optimizing 3D Gaussians and camera poses logether during training. However, these methods are typically designed for scenes with high view overlap, and they often fail to reconstruct complex, large-scale 360 scenes with sparse coverage. Additionally, despite extensive regularization to prevent overfitting, the limited observations impede coherent synthesis of unobserved regions. This challenge underlines the need for additional generative regularization to enable accurate extrapolation and complete scene reconstruction.
In the absence of poses estimated from dense observations, we instead rely on recent 3D foundational priors (Teroy et al., 2024) for scene initialization from sparse views and encode its estimated cameras for conditioning a Stable Diffuusin UNet (Rombach et al., 2022) during both training and evaluation. We also augment the UNet with additional channels for context, 3DGS render, depth maps with Gaussian artifacts, and a pixel-aligned confidence map capturing missing regions and reconstruction artifacts in the RGBD image. During inference, the model prodicts a clean, inpainted version of the conditioning artifact image and an aligned depth map. This formulation prevents the requirement of accurate ground truth poses for pose conditioning of a multiview diffusion model. This also alleviates dependency on large-scale 3D datasets which are usually synthetic and low quality compared to real-world scenes.
We present GScenes, efficient method that uses 3D foundational (Leroy et al., 2024) and RGBD diffusion priors for pose-free sparse-view reconstruction of complex 360 scenes. Our method leverages stronger priors than simple regularizers while not relying on million-scale multi-view data or huge compute resources to train a 3D-aware diffusion model. We also ablate all conditioning elements in our diffusion model and identify which conditioning features contribute the most towards coherent NVS from sparse views.
B Related Work
Reconstructing 3D scenes from limited observations requires generative priors or, more specifically, inpainting missing details in unseen regions in 3D and removing artifacts introduced through observing scene areas from a few observations. Our work builds on recent developments in 2D diffusion priors for 3D reconstruction (Paul et al., 2024), where knowledge learned from abundant 2D datasets is lifted to 3D for refining novel views rendered by sparse 3D models. Next, we discuss how our work is related to the current line of research.
Regularization Techniques Both NeRF and 3DGS rely on hundreds of scene captures for photorealistic novel view synthesis. When the input set becomes sparse, the problem becomes ill-posed, as several simultaneous 3D representations can agree with the training set. Regularization techniques are amongst the earliest techniques to address this limitation. Typical methods leverage depth from Structure-from wrwuurr--roo-Meee Motion (Den) et al., 2022; Rocssle et al., 2022), monocular cstimation(Li et al., 2024; Xiong et al., 2023; Zhu el al., 2023; Chung et al., 2023), or RGB-D sensors ( (Wang et al., 2023a). DietNeRP (Jain el al., 2021) uses a semantic consistency loss based on CLIP (Radford et al., 2021) features, while FreeNeRF (Yang et al., 2023) regularizes the frequency range of NeRF inputs. RegNeRF (Niemeyer et al., 2022) and DiffusioNeRF (Wynn and Turmukhambetov, 2023) maximize the likelihoods of rendered patches using normalizing flows or diffusion models, respectively. However, such techniques usually fail under extreme sparsity like 3, 6, or input images
17

---

Published in Transactions on Machine Learning Research (07/2025)
for 360 scene due to weaker priors. Generative priors can be viewed as stronger form of regularization they provide extrapolation capabilities for inferring details in unknown parts of scene.
Generalizable Reconstruction When only a few or single view is available, regularization techniques are often insufficient to resolve reconstruction ambiguities. To address this, recent research focuses on training priors for novel view synthesis across multiple scenes. pixeINeRR' (Yu et al., 2021) uses pixel-aligned CNN features as conditioning for a shared NeRF MLP, while other approaches (Trovithick and Yang, 2021; Chen el al., 2021; Henzler et al., 2021; Lin el. al., 2023b; Szymanowicz el al., 2023) condition NeRF on 2D or fused 3D features. Further priors have been learned on triplanes (Irshad al., 2023), voxel grids (Guo al., 2022), neural points (Wewer et al., 2023), and IB-planes Anciukevičius al. (2024). Leveraging 3D Gaussian Splatting (Kerbl et al., 2023), methods like pixelSplat (Charatan et al., 2024) and MVSplat (Chen et al., 2024) achieve state-of-the-art performance in stereo view interpolation. However, these regression-based techniques infer blurry novel views in case of high uncertainty.
Generative Priors for NVS For ambiguous novel views, predicting expectations over all recoustructions may be unreliable. Consequently, regression approaches fall short, whereas generative methods attempt to sample from a multi-modal distribution.
While diffusion models have been applied directly on 3D representations like triplanes (Shue et al., 2023; Chen et al., 2023a), voxel grids (Mäller al., 2023), or (neural) point clouds (Zhou et al., 2021; Melas-Kyriazi et al., 2023; Schröppel et al., 2024), 3D data is scarce. Given the success of large-scale diffusion models for image synthesis, there is a great research interest in leveraging them as priors for 3D reconstruction and generation. DreamFuusion (Poole et al., 2023) and follow-ups (Wang et al., 2023b; Lin et al., 2023a; Chen et al., 2023b; Deng et al., 2023; Tang et al., 2024) employ score distillation sampling (SDS) to iteratively maximize the likelihood of radiance field renderings under a conditional 2D diffusion prior. For sparse-view reconstruction, existing approaches incorporate niew-conditioning via epipolar fcature transform (Zhou and Tulsiani, 2023), cross-attention to encoded relative poses (Liu et al., 2023; Sargent et al., 2024), or pixelNeRF (Yu et al., 2021) feature renderings (Wu et al., 2024). However, this fine-tuning is expensive and requires large-scale multi-view data, which we circumvent with GScenes.
Pose-Free 3D Reconstruction For building a generalizable sparse-view reconstruction method, the assumption of camera poses during inference limits applications to real-world scenarios where usually only an uncalibrated set of 2D images are available with no known camera extrinsics or intrinsics. Several recent works (Jiang et al., 2024; Fan et al., 2024) have atlempled to solve reconstruction in a pose-free setting by jointly optimizing poses and NeRF or 3D Gaussian parameters during scene optimization. These mothods typically outperform previous techniques (Chen and Lec, 2023; Lin et al., 2021; Bian et al., 2023; Fu et al., 2023) where reconstruction is done in two stages first estimating poses and then optimizing the 3D representation, However, errors in the initial pose estimation harm subsequent scene optimization, resulting in inferior NVS quality. In our work, we use the MASt3R pipeline with 3DGS for predicting 3D Gaussians and camera parameters in global coordinate system from a set of unposed 2D images.
Our work is most closely related to Sp0 (Paul et al., 2024) where an instruction-following RGB diffusion model is finetuned for the task of rectifying novel views rendered by 3DGS fitted to sparse observations. We extend the problem setting to the more challenging pose-fres free scenario and jointly model RGB and depth aid optimization of 3D Gaussians during the distillation phase. Additionally, we introduce CLIP context and pose conditioning through FiLM layers and pixcl-aligned confidence measure for more accurate novel view synthesis.
C RGBD Dataset Creation
Our fine-tuning setup for the RGBD diffusion model is illustrated in Fig 12.
18

---

Published in Transactions on Machine Learning Research (07 (02))
Dense Model Render
Dense Model Depth
Cond. LDM Unet
Dense 3D G Bassians
Entedding
Camera Interpolation
CUP Testime
Perturbation
π
Render View
Sparse 3D Gaussians
Sparse Model Render
Sparse Model Depth
Confidence Map
Figure 12: Training our RGBD diffuusion model. Pairs of RGBD clean images and images with artifacts are obtained from 3DGS fitted to sparse and dense observations, respectively, across 1043 scenes. CLIP features provide semantic scene context, plücker embeddings of source and target cameras provide geometry information, and a confidence map additionally detects empty regions and artifacts the artifact image. The Stable Diffusion UNet (Rombach et al., 2022) is then fine-tuned with datasel of 171, 461 samples.
D Qualitative Comparison with ReconFusion / CAT3D
In Fig 13, we provide an additional qualitative comparison with ReconFusion and CAT3D. Their code, data, models, or training reproduction details are not available, and hence, we can not perform any evaluation across any of the test scenes in their paper. Despite this issue, from figures in the 2 papers, we pick the relevant test views for the treehill, flowers, bicycle scenes in MipNeRF330, and the plant scene from CO3Dv2 show how GSoenes compares with their reconstruction. We use the same training views as open-sourced in their data splits. Despite being pose-free pipeline using weaker generative priors, observe that GScenes compares competitively with both methods.
E
Additional Qualitative Results
Additional qualitative comparisons on MipNeRF360 and DI.3DV-bencharkk datasets are provided in Fig 14 and 15.
F
Ablation Studies
We report the quantitative performance of our method with different variants of the proposed diffusson model in Tab 3.
G Evaluating DiffusioNeRF in a pose-free setting
Since the performance ReconFusion and CAT3D in a pose-free setting cannot be ascertained due to a lack of open-source code models, we instcad evaluate DiffusioNeRF, an open-source posod reconstruction technique, on 3 scenes from the DL3DV-benchmad MASL3R-3DGS, and GScenes. We noisy poses estimated by MASt3R from sparse inputs training
the same pose-free evaluation protocol COGS,
19

---

Published in Transactions on Machine Learning Research (07 /025)
MAS13R + 3DGS
ReconFusion
CAT3D
Ours
Ground Truth
Treehill
Flowers (3)
Plant 3
Bicycle (9)
Figure 13: Qualitative comparison GScenes with ReconFusion and CAT3D (posed techniques). Despite being a prsc-free pipeline built with weaker diffusion priors, our method achioves better (flowers, treehill) similar (plant) NVS quality compared to ReconFusion on three out of examples above. We slightly worse compared to closed-source CAT3D (although their results cannot be validated), which closed-source, stronger video diffusion prior. No image available for CAT3D in the last row, hence kept blank.
MASt3R + 3DGS
RegNeRF 0
DiffusioNeRF
ZeroN O
COGS
Ours
Ground Truth
Stamp 3)
Bnom!
Kitchen
Room 31
Figure 14: Additional Qualitative comparisons on the Mi3Ne dataset.
20

---

Published in Transactions on Machine Learning Research (07 (025)
MAS(3R 3DGS
ZeroNVS*
DiflusioNeRIF
COGS
Ours
Ground Truth
views
vieass
vieges
Figure 15: Additional Qualitative comparisons on the DL3DV benchmark.
Table Ablation study. We report average performance across 3-view splits of bonsai and treehill scenes on metrics. Our proposed diffusson model variant with conditioning from clip features, plücker embeddings, and confidence maps achieves identical FID, KID, and DISTS scores with more parameter-hent cross-attention variant. Red, orange, and yellow indicate best, second-best, and third-best performing methods, respectively
Method
PSNR +
LPIPS +
SSIM +
FID +
KID +
DISTS +
Basc
12.024
0.271
0.574
264.562
0.105
0.272
Base + Conf.
11.529
0.261
0.572
249.204
0.099
0.274
w/o CLIP
12.166
0.312
0.578
177.707
0.059
0.279
w/o Conf.
12.100
0.270
0.572
265.564
0.131
0.271
12.740
0.298
0.543
152.961
0.042
0.225
MASt3R 3DGS
11.641
0.269
0.604
335.064
0.268
0.329
GScenes
12.871
0.297
0.545
154.769
0.039
0.228
21

---

Published in Transactions on Machine Learning Research (07 (075)
poses instead of ground truth COLMAP poses (posed setting) and show the difference in NVS quality test views in Fig 16. From this experiment, one can conclude that NVS quality achieved with posed techniques does not directly translate to similar performance in a pree-free setting.
DiffusioNeIRIF (posed)
DiffusioNeRRF (puse-free)
Ground Truth
Figure 16: DiffusioNeRF in a pose-free setting. We evaluate DiffusioNeRF using poses estimated by MASt3R instead of ground truth colmap poses estimated from dense views. The drop in NVS quality demonstrates that posed reconstruction technique cannot be trivially adapted to the posc-free setting.
H Effect of Generative Priors on Training Views
Table 4: Effect of Generative Priors on Training Views. We pick the Square scene from DL3DV-10K
and its 3, 6, 9 view splits to compare reconstruction performance of GScenes with our baseline MASt3R + 3DGS on the training views. Noise and 3D inconsistencies introduced from the generated novel views cause a minor drop-off the reconstruction quality of training views.
PSNR f
SSIM 1 6-view
LPIPS 6-view
3-view
6-view
9-view
3-view
9-view
3-view
I-view
MASt3R + 3DGS
23.34
22.32
22.96
0.73
0.62
0.63
0.20
0.21
0.23
GSeenes
22.44
21.52
21.70
0.75
0.57
0.56
0.18
0.25
0.26
In this section, analyze the effect of using generative priors for novel view synthesis on the reconstruction quality of the sparsc training views of 3D seenc. To ensure 3D consistency and limited artifacts in the reconstructed scene, our diffusion model integrales camera, context, and confidence map conditioning, and the autoregressive 2D-3D distillation process (Sec 2.6) progressively resolves inconsistencies across different novel views. The differences that cannot be reconciled result in artifacts like floaters, which also nogatively impact the reconstruction quality of the input views. In Tab and Fig 17, we pick the Square scene from DL3DV-10K benchmark and its 3, 6, and 9 view splits compare the performance of GScenes with
22

---

Published in Transactions on Machine Learning Research (07 (07/2025)
MASt3R + 3DGS (3)
GScenes (8)
MASt3R +
GSeenes (6)
MASt3R +
GSeenes (9)
Ground Truth
3DGS (6)
3DGS (9)
Figure 17: Effect of Generative Priors on Training Views.
baseline reconstruction engine MASt3R + 3DGS reconstruction of training views. As observe, while there is minor degradation both qualitatively and quantitatively due to distillation of generative priors, the original 3DGS objective for training views during autoregressive scene reconstruction ensures that the reconstructed sparse input views still faithfully represent the 3D scene.
Limitations & Future Work
Table 5: Pose estimation accuracy with GScenes. SIM-poses for training views estimated from the full observation set are used as ground truth. We report errors in camera rotation and translation using Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) as in Bian et al. (2023). Despite the MASt3r initialization of camera extrinsics and subsequent pose optimization, there large errors w.r.t COLMAP poses of the dense observation set, harming (est-pose alignment and subsequent NVS quality.
RPE,
RPE, 6-view
ATE
3-view
6-view
9-view
3-view
9-view
3-view
6-view
9-view
GScenes 37.684
27.893
30.916
124.501
58.915
40.424
0.261
0.294
0.266
GScenes is a first step towards a generative solution for posc-free sparso-view reconstruction of large complex scenes. However, not free of its fair share of limitations. The quality of the final reconstruction depends heavily on the initial relative pose estimation by the MASt3r pipeline, and even though the poses are further optimized jointly with Gaussian attributes during training, there are still large differences with the ground truth COLMAP poses estimated from dense views posed reconstruction methods, as even the test-time pose aligument step (Sec 3.5) cannot compensate for the initial errors in the pipeline. Our diffusion model, much like related methods, is not agnostic to the 3D representation from which novel view renders, depth, and confidence maps are obtained for fine-tuning the diffusion model for 3D-aware sperse-view NVS. Moreover, our diffusion model trained with 3DGS renders with MASt3R initialization would not be able rectify novel views from a 3DGS representation with SIM initialization due to the slight difference in the distribution of rendered images. To ensure multiview consistency across all synthesized views, employ view conditioning in the form of plucker embeddings and use a fixed noise latent across all novel views for multistep reconstruction. However, this does not alleviate the multiview consistency issue completely as novel views are synthesized in an autoregressive manner and not simultaneously synthesized like in video diflusion models. We aim to address these limitation of our diffusion model in future work.
show in Tab 5. This limits fair comparison with
23


Reviews:
[{"review_id": "bM3h7blum0", "reviewer": "Reviewer_3BVF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Novelty: Integrating depth-conditioned diffusion into a pose-free 3D reconstruction framework is an interesting direction. The FiLM-based conditioning elegantly reduces parameter count while preserving reconstruction quality.\n\n- Practicality: The end-to-end pipeline runs in minutes and does not require massive multi-view datasets or high-end GPUs. An open-source implementation will further accelerate adoption.\n\n- Extensive Evaluation: Quantitative results on two major benchmarks (MipNeRF360, DL3DV-10K), along with FID, KID, and DISTS scores, provide a comprehensive assessment of performance.\n\n**Weaknesses**\n- Limited Dataset Coverage: Several datasets used by key pose-free baselines—such as RealEstate10K, LLFF, DTU, and CO3D in CAT3D—are missing from the comparison, making it difficult to gauge relative gains.\n\n- Low PSNR Scores: All reported PSNR values fall below 20 dB, raising concerns about the method’s practical usefulness. The teaser images report values below 14 dB, which suggests poor fidelity.\n\n- Lack of Quantitative Ablation for the Confidence Map: While qualitative examples illustrate the confidence map’s utility, no quantitative ablation isolates its contribution to overall performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper proposes a novel, pose-free 360° scene reconstruction pipeline that fuses an initial point cloud generated by MASt3R with a 3DGS optimization stage, followed by diffusion-based inpainting.\n\n- It introduces a lightweight conditioning mechanism using FiLM layers—combining CLIP-based contextual features and Plücker-embedded geometric cues—to achieve performance comparable to cross-attention with fewer than one-third of the parameters.\n\n- A per-pixel confidence map is defined by combining transmittance and Gaussian density, enabling precise detection and correction of reconstruction artifacts.\n\n- On the MipNeRF360 and DL3DV-10K benchmarks, the method achieves state-of-the-art performance among pose-free approaches and approaches the accuracy of pose-aware techniques.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses outlined above.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7MjKO90Pz2", "reviewer": "Reviewer_LtUC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper attempts to address the problem of scene reconstruction under sparse view conditions. Leveraging generative priors to aid the reconstruction is a great idea. The writing and structure of the paper are also clear and easy to follow.\n\n\nHowever, there are some shortcomings: \nalthough the authors claim to have achieved state-of-the-art performance, the reconstruction quality is still not very good based on both the quantitative metrics and the visual results. In particular, the rendered images contain a large number of artifacts and do not align well with the real scenes. This raises the question of the method’s motivation. If the goal is to improve rendering for visualization purposes, the current results are not yet at a practically usable level. If the goal is 3D reconstruction, the integration of diffusion models has made the results unreliable.\n\n\nDuring training, the newly added novel views might be close to the test views. Could this lead to inaccurate results during testing?\n\nThe authors did not show whether the final optimized training camera poses deviate from the ground truth. In the proposed algorithm, newly generated images are added to the training set, which could potentially affect the result of joint camera parameter optimization.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission presents a method, GScenes, for pose-free 3D reconstruction of 360° scenes from sparse, uncalibrated images. The approach combines a 3D Gaussian Splatting framework with an RGBD image-to-image diffusion model to refine synthesized novel views. Key technical components include a FiLM-based conditioning mechanism for context and geometry, a pixel-aligned confidence measure to identify unreliable regions, and an iterative reconstruction strategy integrating synthesized views. Experimental results on standard benchmarks show that the method achieves competitive performance with state-of-the-art pose-free and some posed methods, using fewer views and lower computational resources.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It is necessary to provide a more practical justification for the motivation of the method and to complete the experiments mentioned in the weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0EGyRFuP5X", "reviewer": "Reviewer_ajat", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe underlying idea of performing sparse-view reconstruction by iteratively incorporating diffusion-generated views is simple and elegant\n\nThe specific approach contains some novel aspects, such as the approach to confidence estimation used in deciding how to weight information, and using RGB+D conditioning and output in the diffusion model (unlike ReconFusion/etc. which use RGB only)\n\nQuantitative results show the method exceeds the performance of two other pose-free approaches – just using MASt3R then gaussian splatting, and the recent COGS – on the standard reconstruction metrics (PSNR, SSIM, LPIPS) across two datasets (MipNeRF-360 and DL3DV)\n\nQualitative results are significantly better than a 'naive' baseline of MASt3R+3DGS, with plausible completion of partially-observed regions.\n\nThere is an ablation study measuring the benefit of several design decisions relating to conditioning in the diffusion model, giving some insight into what information is helpful to the model\n\nThere is a balanced discussion of limitations (albeit in an appendix)\n\nOverall the text is clear and readable; the paper is well structured; the figures are well-presented and appropriate\n\n**Weaknesses**\n\nThe technical novelty is fairly small – the proposed pipeline is rather reminiscent of ReconFusion, among others. While the pose-free setting differentiates it somewhat, there is very little in the method that is specific to this setting (only the MASt3R initialisation – which itself is not novel), so motivating the work on this basis is rather strange.\n\nQualitative results are somewhat weak – while the model synthesises 3D-consistent and somewhat plausible content in unobserved regions, this is lacking in detailed texture and notably weaker than e.g. CAT3D.\n\nWhile it is true that CAT3D and ReconFusion assume known camera poses, these could be estimated using Mast3R or similar; it is unclear whether such a pipeline would in fact perform better or worse (with perfect poses, they are apparently very good; with estimated ones, unknown). In particular, the quantitative and qualitative gap between the proposed method and CAT3D is large enough that it is plausible using it with estimated poses would still outperform the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method for 3D reconstruction from sparse views, without assuming access to camera parameters (i.e. it is pose-free). It fits gaussian splats (after MASt3R initialisation) to the input images, then renders from a novel viewpoint, and uses a conditional diffusion model to predict a 'tidied' version of this (RGB+D) rendering, which is then incorporated as pseudo-ground-truth in a further stage of splat optimisation. This process proceeds iteratively, adding more diffusion-generated views. The conditional diffusion model itself is fine-tuned from Stable Diffusion to support the relevant conditioning and RGB+D output, using a relatively small dataset. The method is evaluated on standard datasets of natural scenes, where it outperforms other pose-free methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There is some redundancy between 2.0 and the 2nd paragraph of 2.1; it might be worth merging these\n\nIt would be good if there were a related work section in the main paper rather than just an appendix\n\nThe \"Generative Priors for NVS\" paragraph of related work is missing references to diffusion models that incorporate explicit latent 3D – notably Viewset Diffusion [Szymanowicz, ICCV 2023] and Denoising Diffusion via Image-Based Rendering [Anciukevicius, ICLR 2024]\n\n\"competitive\" in fig 13 caption should perhaps be weakened – the proposed method's results are consistently worse than CAT3D", "broader_impact_concerns": "This work does not raise any specific concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)