ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model

## Abstract

Abstract Advancements in 3D scene reconstruction have transformed 2D images from the real world into 3D models, producing realistic 3D results from hundreds of input photos. Despite great success in dense-view reconstruction scenarios, rendering a detailed scene from insufficient captured views is still an ill-posed optimization problem, often resulting in artifacts and distortions in unseen areas. In this paper, we propose ReconX , a novel 3D scene reconstruction paradigm that reframes the ambiguous reconstruction challenge as a temporal generation task. The key insight is to unleash the strong generative prior of large pre-trained video diffusion models for sparse-view reconstruction. However, 3D view consistency struggles to be accurately preserved in directly generated video frames from pre-trained models. To address this, given limited input views, the proposed ReconX first constructs a global point cloud and encodes it into a contextual space as the 3D structure condition. Guided by the condition, the video diffusion model then synthesizes video frames that are both detail-preserved and exhibit a high degree of 3D consistency, ensuring the coherence of the scene from various perspectives. Finally, we recover the 3D scene from the generated video through a confidence-aware 3D Gaussian Splatting optimization scheme. Extensive experiments on various real-world datasets show the superiority of our ReconX over state-of-the-art methods in terms of quality and generalizability. Project Page: https://liuff19.github.io/ReconX .

## 1 Introduction

With the rapid development of photogrammetry techniques such as NeRF (Mildenhall et al., ) and 3D Gaussian Splatting (3DGS) (Kerbl et al., ), 3D reconstruction has become a popular research topic in recent years, finding various applications from virtual reality (Dalal et al., ) to autonomous navigation (Adamkiewicz et al., ) and beyond (Martin-Brualla et al., ; Liu et al., ; Wu et al., ; Charatan et al., ).
However, sparse-view reconstruction is an ill-posed problem (Gao et al., ; Yu et al., ) since it involves recovering a complex 3D structure from limited viewpoint information (*i.e., *even as few as two images) that may correspond to multiple solutions. This uncertain process requires additional assumptions and constraints to yield a viable solution.

Recently, powered by the efficient and expressive 3DGS (Kerbl et al., ) with fast rendering speed and high quality, several feed-forward Gaussian Splatting methods (Charatan et al., ; Szymanowicz et al., ; Chen et al., ) have been proposed to explore 3D scene reconstruction from sparse view images. Although they can achieve promising interpolation results by learning scene-prior knowledge from feature extraction modules (*e.g., *epipolar transformer (Charatan et al., )), insufficient captures of the scene still lead to an ill-posed optimization problem (Wu et al., ). As a result, they often suffer from severe artifact and implausible imagery issues when rendering the 3D scene from novel viewpoints, especially in unseen areas.

To address the limitations, we propose ReconX, a novel 3D scene reconstruction paradigm that reformulates the inherently ambiguous reconstruction problem as a generation problem. Our key insight is to unleash the strong generative prior of pre-trained large video diffusion models (Blattmann et al., ; ; Xing et al., ) to create more observations for the downstream reconstruction task. Despite the capability to synthesize video clips featuring plausible 3D structure (Gao et al., ), recovering a high-quality 3D scene from current video diffusion models is still challenging due to the poor 3D view consistency across generated 2D frames. Grounded by theoretical analysis, we explore the potential of incorporating 3D structure guidance into the video generative process, which bridges the gap between the under-determined 3D creation problem and the fully-observed 3D reconstruction setting. Specifically, given sparse images, we first build a global point cloud through a pose-free stereo reconstruction method. Then we encode it into a rich context representation space as the 3D condition in cross-attention layers, which guides the video diffusion model to synthesize detail-preserved frames with 3D consistent novel observations of the scene. Finally, we reconstruct the 3D scene from the generated video through Gaussian Splatting with a 3D confidence-aware and robust scene optimization scheme, which further deblurs the uncertainty in video frames effectively. Extensive experiments verify the efficacy of our framework and show that ReconX outperforms existing methods for high quality and generalizability, revealing the great potential to craft intricate 3D worlds from video diffusion models. The overview and examples of reconstructions are shown in Figure 1.

Figure: Figure 1: An overview of our ReconX framework for sparse-view reconstruction. Unleashing the strong generative prior of video diffusion models, we can create more observations for 3D reconstruction and achieve impressive performance.
Refer to caption: x1.png

In summary, our main contributions are:

- •
We introduce ReconX, a novel sparse-view 3D scene reconstruction framework that reframes the ambiguous reconstruction challenge as a temporal generation task.
- •
We incorporate the 3D structure guidance into the conditional space of the video diffusion model to generate 3D consistent frames and propose a 3D confidence-aware optimization scheme in 3DGS to reconstruct the scene given the generated video.
- •
Extensive experiments demonstrate that our ReconX outperforms existing methods for high-fidelity and generalizability on a variety of real-world datasets.

## 2 Related Work

Sparse-view reconstruction.
NeRF and 3DGS typically demand hundreds of input images and rely on the multi-view stereo reconstruction (MVS) approaches (*e.g., *COLMAP (Schönberger & Frahm, )) to estimate the camera parameters. To address the issue of low-quality 3D reconstruction caused by sparse views, PixelNeRF (Yu et al., ) proposes using convolutional neural networks to extract features from the input context. Moreover, FreeNeRF (Yang et al., ) adopts the frequency and density regularized strategies to alleviate the artifacts caused by insufficient inputs without any additional cost. To mitigate the overfitting to input sparse views in 3DGS, FSGS (Zhu et al., ) and SparseGS (Xiong et al., ) employ a depth estimator to regularize the optimization process. However, these methods all require known camera intrinsics and extrinsics, which is not practical in real-world scenario. Benefiting from the existing powerful 3D reconstruction model (*i.e., *DUSt3R (Wang et al., )), InstantSplat (Fan et al., ) is able to acquire accurate camera parameters and initial 3D representations from unposed sparse-view inputs, leading to the efficient and high-quality 3D reconstruction.

Regression model for generalizable view synthesis.
While NeRF and 3DGS are optimized per-scene, a line of research aims to train feed-forward models that output a 3D representation directly from a few input images, bypassing the need for time-consuming optimization.
Splatter image (Szymanowicz et al., ) performs an efficient feed-forward manner for monocular 3D object reconstruction by predicting a 3D Gaussian for each image pixel. Meanwhile, pixelSplat (Charatan et al., ) proposes predicting the scene-level 3DGS from the image pairs, using the epipolar transformer to better extract scene features. Following that, MVsplat (Chen et al., ) introduces the cost volume and depth refinements to produce a clean and high-quality 3D Gaussians in a faster way. To reconstruct a complete scene from a single image, Flash3D (Szymanowicz et al., ) adopts a hierarchical 3DGS learning policy and depth constraint to achieve high-quality interpolation and extrapolation view synthesis. Although these methods leverage the 3D data priors, they are limited by the scarcity and diversity of 3D data. Consequently, these methods struggle to acquire high-quality renderings in unseen areas, especially when OOD data is used as input.

Generative models for 3D reconstruction.
Constructing comprehensive 3D scenes from limited observations demands generating 3D content, particularly for unseen areas. Earlier studies distill the knowledge in the pre-trained text-to-image diffusion models (Rombach et al., ; Saharia et al., ; Ramesh et al., ) into a coherent 3D model. Specifically, the Score Distillation Sampling (SDS) technique (Wu et al., ; Lin et al., ; Liu et al., ; Wang et al., ) is adopted to synthesize a 3D object from the text prompt.
To enhance the 3D consistency, several approaches (Wu et al., ; Shi et al., ; Liu et al., ) inject the camera information into diffusion models, providing strong multi-view priors. Furthermore, ZeroNVS (Sargent et al., ) and CAT3D (Gao et al., ) extend the multi-view diffusion to the scene level generation. More recently, video diffusion models (Blattmann et al., ; Xing et al., ) have shown an impressive ability to produce realistic videos and are believed to implicitly understand 3D structures (Liu et al., ). SV3D (Voleti et al., ) and V3D (Chen et al., ) explore fine-tuning the pre-trained video diffusion model for 3D object generation. Meanwhile, MotionCtrl (Wang et al., ) and CameraCtrl (He et al., ) achieve scene-level controllable video generation from a single image by explicitly injecting the camera pose into video diffusion models. However, they don’t work for the unconstrained sparse-view reconstruction, which requires strong 3D consistency.

## 3 Preliminaries

Video Diffusion Models.
Diffusion models (Ho et al., ; Song et al., ) have emerged as the cutting-edge paradigm to generate high-quality videos. These models learn the underlying data distribution by adding and removing noise on the clean data. The forward process aims to transform a clean data sample $\bm{x}_{0}\sim p(\bm{x})$ to a pure Gaussian noise $\bm{x}_{T}\sim\mathcal{N}(0,I)$, following the process:

$$ $\bm{x}_{t}=\sqrt{\bar{\alpha}_{t}}\bm{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon ,\quad\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{1}),$ (1) $$

where $\bm{x}_{t}$ and $\bar{\alpha}_{t}$ denotes the noisy data and noise strength at the timestep $t$. The denoising neural network $\epsilon_{\theta}$ is trained to predict the noises added in the forward process, which is achieved by the MSE loss:

$$ $\mathcal{L}=\mathbb{E}_{\bm{x}\sim p,\epsilon\sim\mathcal{N}(0,I),c,t}\left[ \left\|\epsilon-\epsilon_{\theta}\left(\bm{x}_{t},t,c\right)\right\|_{2}^{2} \right],$ (2) $$

where $c$ represents the embeddings of conditions like text or image prompt. For the video diffusion models, Latent Diffusion Models (LDMs) (Rombach et al., ), which compress images into the latent space, are commonly employed to mitigate the computation complexity while maintaining competitive performance.

3D Gaussian Splatting.
3DGS (Kerbl et al., ) represents a scene explicitly by utilizing a set of 3D Gaussian spheres, achieving a fast and high-quality rendering. A 3D Gaussian is modeled by a position vector $\bm{\mu}\in\mathbb{R}^{3}$, a covariance matrix $\bm{\Sigma}\in\mathbb{R}^{3\times 3}$, an opacity $\alpha\in\mathbb{R}$, and spherical harmonics (SH) coefficient $\bm{c}\in\mathbb{R}^{k}$ (Ramamoorthi & Hanrahan, ). Moreover, the Gaussian distribution is formulated as the following:

$$ $G(x)=e^{-\frac{1}{2}(x-\bm{\mu})^{T}\bm{\Sigma}^{-1}(x-\bm{\mu)}},$ (3) $$

where $\bm{\Sigma}=\bm{R}\bm{S}\bm{S}^{T}\bm{R}^{T}$, $\bm{S}$ denotes the scaling matrix and $\bm{R}$ is the rotation matrix.

In the rendering stage, the 3D Gaussian spheres are transformed into 2D camera planes through rasterization (Zwicker et al., ). Specifically, given the perspective transformation matrix $\bm{W}$ and Jacobin of the projection matrix $\bm{J}$, the 2D covariance matrix in the camera space is computed as

$$ $\bm{\Sigma}^{{}^{\prime}}=\bm{J}\bm{W}\bm{\Sigma}\bm{W}^{T}\bm{J}^{T}.$ (4) $$

For every pixel, the Gaussians are traversed in depth order from the image plane, and their view-dependent colors $c_{i}$ are combined through alpha compositing, leading to the pixel color ${C}$:

$$ ${C}=\sum_{i\in N}c_{i}\alpha_{i}\prod_{j=1}^{i-1}\left(1-\alpha_{i}\right).$ (5) $$

End-to-end Dense Unconstrained Stereo. DUSt3R (Wang et al., ) is a new model to predict a dense and accurate 3D scene representation solely from image pairs without any prior information about the scene. Given two unposed images $\{\bm{I}_{1},\bm{I}_{2}\}$, this end-to-end model is trained to estimate the point maps $\{P_{1,1},P_{2,1}\}$ and confidence maps $\{\mathcal{C}_{1,1},\mathcal{C}_{2,1}\}$, which can be utilized to recover the camera parameters and dense point cloud. The training procedure for view $v\in\{1,2\}$ is formulated as a regression loss:

$$ $\mathcal{L}=\left\|\frac{1}{z_{i}}\cdot{P}_{v,1}-\frac{1}{\hat{z}_{i}}\cdot \hat{{P}}_{v,1}\right\|,$ (6) $$

where $P$ and $\hat{P}$ denote the ground-truth and prediction point maps, respectively. The scaling factors $z_{i}$ = norm$(P_{1,1},P_{2,1})$ and $\hat{z}_{i}$=norm$(\hat{P}_{1,1},\hat{P}_{2,1})$ are adopted to normalize the point maps, which merely indicate the mean distance $D$ of all valid points from the origin:

$$ $\operatorname{norm}\left({P}_{1,1},{P}_{2,1}\right)=\frac{1}{\left|{D}_{1} \right|+\left|{D}_{2}\right|}\sum_{v\in\{1,2\}}\sum_{i\in{D}_{v}}\left\|{P}_{v }^{i}\right\|.$ (7) $$

## 4 Motivation for ReconX

In this paper, we focus on the fundamental problem of 3D scene reconstruction and novel view synthesis (NVS) from very sparse view (*e.g., *as few as two) images. Most existing works (Chen et al., ; Yu et al., ; Charatan et al., ; Szymanowicz et al., ) utilize 3D prior and geometric constraints (*e.g., *depth, normal, cost volume) to fill the gap between observed and novel regions in sparse-view 3D reconstruction. Although capable of producing highly realistic images from the given viewpoints, these methods often struggle to generate high-quality images in areas not visible from the input perspectives due to the inherent problem of insufficient viewpoints and the resulting instability in the reconstruction process. To address this, a natural idea is to create more observations to collapse the under-determined 3D creation problem into a fully constrained 3D reconstruction setting. Recently, video generative models have shown promise for synthesizing video clips featuring 3D structure (Voleti et al., ; Blattmann et al., ; Xing et al., ). This inspires us to unleash the strong generative prior of large pre-trained video diffusion models to create temporal consistent video frames for sparse-view reconstruction. Nevertheless, it is non-trivial as the main challenge lies in poor 3D view consistency among video frames, which significantly limits the downstream 3DGS training process. To achieve 3D consistency within video generation, we first analyze the video diffusion modeling from a 3D distributional view.
Let $\bm{x}$ be the set of rendering 2D images from any 3D scene in the world, $q(\bm{x})$ be the distribution of the rendering data $\bm{x}$, our goal is to minimize the divergence $\mathcal{D}$:

$$ $\min_{\bm{\theta}\in\Theta,\psi\in\Psi}\mathcal{D}\left(q(\bm{x})\|p_{\bm{ \theta},\psi}(\bm{x})\right),$ (8) $$

where $p_{\bm{\theta},\psi}$ be a diffusion model parameterized by $\bm{\theta}\in\Theta$ (the parameters in the backbone) and $\psi\in\Psi$ (any embedding function shared by all data). In vanilla video diffusion model (Xing et al., ), they choose CLIP (Radford et al., ) model $g$ to add an image-based condition (*i.e., *$\psi=g$). However, in sparse-view 3D reconstruction, only limited 2D images as the condition cannot provide sufficient guidance for approximating $q(\bm{x})$ (Charatan et al., ; Chen et al., ; Wu et al., ). Motivated by this, we explore the potential of incorporating the native 3D prior (denoted by $\mathcal{F}$) to find a more optimal solution in Equation 8 and derive a theoretical formulation for our analysis in Proposition 1.

###### Proposition 1 .

Let $\bm{\theta}^{*},\psi^{*}=g^{*}$ be the optimal solution of the solely image-based conditional diffusion scheme and $\bm{\tilde{\theta}}^{*},\tilde{\psi}^{*}=\{{g}^{*},\mathcal{F}^{*}\}$ be the optimal solution of diffusion scheme with native 3D prior. Suppose the divergence $\mathcal{D}$ is convex and the embedding function space $\Psi$ includes all measurable functions, we have
$\mathcal{D}(q\left(\bm{x}\right)\|p_{\bm{\tilde{\theta}}^{*},\tilde{\psi}^{*}}
(\bm{x}))<\mathcal{D}\left(q\left(\bm{x}\right)\|p_{\bm{\theta}^{*},\psi^{*}}(
\bm{x})\right)$.
(Proof in our Appendix)

Towards this end, we reformulate the inherently ambiguous reconstruction problem as a generation problem by incorporating 3D native structure guidance into the diffusion process.

###### Proposition 1 .

## 5 Method

Figure: Figure 2: Pipeline of ReconX. Given sparse-view images as input, we first build a global point cloud and project it into 3D context representation space as 3D structure guidance. Then we inject the 3D structure guidance into the video diffusion process and guide it to generate 3D consistent video frames. Finally, we reconstruct the 3D scene from the generated video through Gaussian Splatting with a 3D confidence-aware and robust scene optimization scheme. In this way, we unleash the strong power of the video diffusion model to reconstruct intricate 3D scenes from very sparse views.
Refer to caption: x2.png

### 5.1 Overview of ReconX

Given $K$ sparse-view (*i.e., *as few as two) images $\mathcal{I}=\left\{\bm{I}^{i}\right\}_{i=1}^{K},\left(\bm{I}^{i}\in\mathbb{R}^
{H\times W\times 3}\right)$, our goal is to reconstruct the underlying 3D scene, where we can synthesize novel views of unseen viewpoints. In our framework ReconX, we first build a global point cloud $\mathcal{P}=\left\{\bm{p}_{i},1\leq i\leq N\right\}\in\mathbb{R}^{N\times 3}$ from $\mathcal{I}$ and project $\mathcal{P}$ into the 3D context representation space $\mathcal{F}$ as the structure guidance $\mathcal{F}(\mathcal{P})$ (Sec. 5.2). Then we inject $\mathcal{F}(\mathcal{P})$ into the video diffusion process to generate 3D consistent video frames $\mathcal{I}^{\prime}=\left\{\bm{I}^{i}\right\}_{i=1}^{K^{\prime}},(K^{\prime}>K)$, thus creating more observations (Sec. 5.3). To alleviate the negative artifacts caused by the inconsistency among generated videos, we utilize the confidence maps $\mathcal{C}=\left\{\mathcal{C}_{i}\right\}_{i=1}^{K^{\prime}}$ from the DUSt3R model and LPIPS loss (Zhang et al., ) to achieve a robust 3D reconstruction (Sec. 5.4). In this way, we can unleash the full power of the video diffusion model to reconstruct intricate 3D scenes from very sparse views. Our pipeline is depicted in Figure 2.

### 5.2 Building the 3D Structure Guidance

Grounded by the theoretical analysis in Sec. 4, we leverage an unconstrained stereo 3D reconstruction method DUSt3R (Wang et al., ) with point-based representations to build the 3D structure guidance $\mathcal{F}$. Given a set of sparse images $\mathcal{I}=\left\{\bm{I}^{i}\right\}_{i=1}^{K}$, we first construct a connectivity graph $\mathcal{G}(\mathcal{V},\mathcal{E})$ of $K$ input views similar to DUSt3R, where vertices $\mathcal{V}$ and each edge $e=(n,m)\in\mathcal{E}$ indicates that the images $\bm{I}^{n}$ and $\bm{I}^{m}$ shares visual contents.
Then we use $\mathcal{G}$ to recover a globally aligned point cloud $\mathcal{P}$. For each image pair $e=(n,m)$, we predict pairwise pointmaps ${P}^{n,n},{P}^{m,n}$ and their corresponding confidence maps $\mathcal{C}^{n,n},\mathcal{C}^{m,n}\in\mathbb{R}^{H\times W\times 3}$. For clarity, let’s denote ${P}^{n,e}:={P}^{n,n}$ and ${P}^{m,e}:=\mathcal{P}^{m,n}$. Since we aim to rotate all pairwise predictions into a shared coordinate frame, we introduce transformation matrix $T_{e}$ and scaling factor $\sigma_{e}$ associated with each pair $e\in\mathcal{E}$ to optimize global point cloud $\mathcal{P}$ as:

$$ $\mathcal{P}^{*}=\underset{\mathcal{P},T,\sigma}{\arg\min}\sum_{e\in\mathcal{E} }\sum_{v\in e}\sum_{i=1}^{HW}\mathcal{C}_{i}^{v,e}\left\|\mathcal{P}_{i}^{v}- \sigma_{e}T_{e}{P}_{i}^{v,e}\right\|.$ (9) $$

More details of the point cloud extraction can be found in Wang et al. (). Having aligned the point clouds $\mathcal{P}$, we now project it into a 3D context representation space $\mathcal{F}$ through a transformer-based encoder for better interaction with latent features of the video diffusion model. Specifically, we embed the input point cloud $\mathcal{P}$ into a latent code using a learnable embedding function and a cross-attention encoding module:

$$ $\mathcal{F}(\mathcal{P})=\text{FFN}\left(\text{CrossAttn}(\text{PosEmb}(\tilde {\mathcal{P}}),\text{PosEmb}(\mathcal{P}))\right),$ (10) $$

where $\tilde{\mathcal{P}}$ is a down-sampled version of $\mathcal{P}$ at $1/8$ scale to efficiently distill input points to a compact 3D context space. Finally, we get the 3D structure guidance $\mathcal{F}(\mathcal{P})$ which contains sparse structural information of the 3D scene that can be interpreted by the denoising U-Net.

### 5.3 3D Consistent Video Frames Generation

In this subsection, we incorporate the 3D structure guidance $\mathcal{F}(\mathcal{P})$ into the video diffusion process to obtain 3D consistent frames. To achieve consistency between generated frames and high-fidelity rendering views of the scene, we utilize the video interpolation capability to recover more unseen observations, where the first frame and the last frame of input to the video diffusion model are two reference views. Specifically, given sparse-view images $\mathcal{I}=\left\{\bm{I}^{i}_{\text{ref}}\right\}_{i=1}^{K}$ as input, we aim to render consistent frames $f(\bm{I}^{i-1}_{\text{ref}},\bm{I}^{i}_{\text{ref}})=\{\bm{I}^{i-1}_{\text{ref
}},\bm{I}_{2},...,\bm{I}_{T},\bm{I}^{i}_{\text{ref}}\}\in\mathbb{R}^{(T+2)
\times 3\times H\times W}$ where $T$ is the number of generated novel frames. To unify the notation, we denote the embedding of image condition in the pretrained video diffusion model as $F_{g}=g(\bm{I_{\text{ref}}})$ and the embedding of 3D structure guidance as $F_{\mathcal{F}}=\mathcal{F}(\mathcal{P})$. Subsequently, we inject the 3D guidance into the video diffusion process by interacting with the U-Net intermediate feature $F_{\text{in}}$ through the cross-attention of spatial layers:

$$ $F_{\text{out}}=\text{Softmax}(\frac{QK^{T}_{g}}{\sqrt{d}})V_{g}+\lambda_{ \mathcal{F}}\cdot\text{Softmax}(\frac{QK^{T}_{\mathcal{F}}}{\sqrt{d}})V_{ \mathcal{F}},$ (11) $$

where $Q=F_{\text{in}}W_{Q},K_{g}=F_{g}W_{K},V_{g}=F_{g}W_{V}$, $K_{\mathcal{F}}=F_{\mathcal{F}}W_{K}^{\prime},V_{\mathcal{F}}=F_{\mathcal{F}}W
_{V}^{\prime}$ are the query, key, and value of 2D and 3D embeddings respectively. $W_{Q},W_{K},W_{K}^{\prime},W_{V},W_{V}^{\prime}$ are the projection matrices and $\lambda_{\mathcal{F}}$ denotes the coefficient that balances image-conditioned and 3D structure-conditioned features. Given the first and last two views condition $c_{\text{view}}$ from $F_{g}$ and 3D structure condition $c_{\text{struc}}$ from $F_{\mathcal{F}}$, we apply the classifier-free guidance (Ho & Salimans, ) strategy to incorporate the condition and our training objective is:

$$ $\mathcal{L}_{\text{diffusion}}=\mathbb{E}_{\bm{x}\sim p,\epsilon\sim\mathcal{N }(0,I),c_{\text{view}},c_{\text{struc}},t}\left[\left\|\epsilon-\epsilon_{ \theta}\left(\bm{x}_{t},t,c_{\text{view}},c_{\text{struc}}\right)\right\|_{2}^ {2}\right],$ (12) $$

where $\bm{x}_{t}$ is the noise latent from the ground-truth views of the training data.

### 5.4 Confidence-Aware 3DGS Optimization.

Built upon the well-designed 3D structure guidance, our video diffusion model generates highly consistent video frames, which can be used to reconstruct the 3D scene. As conventional 3D reconstruction methods are originally designed to handle real-captured photographs with calibrated camera metrics, directly applying these approaches to these generated videos are not perfect to recover the coherent scene due to the uncertainty of unconstrained images (Wang et al., ; Fan et al., ). To alleviate the uncertainty issue, we adopt a confidence-aware 3DGS mechanism to reconstruct the intricate scene. Different from recent approaches (Martin-Brualla et al., ; Ren et al., ) which model the uncertainty in per-image, we instead focus on a global alignment among a series of frames. For the generated frames $\left\{\bm{I}_{i}\right\}_{i=1}^{K^{\prime}}$, we denote $\hat{C}_{i}$ and $C_{i}$ the per-pixel color value for generated and ground-truth view $i$. Then, we model the pixel values as a Gaussian distribution in our 3DGS, where the mean and variance of $I_{i}$ are $C_{i}$ and $\sigma_{i}$. The variance $\sigma_{i}$ measures the discrepancy between the generated view and the real view. Our target is to minize the following negative log-likelihood among all frames:

$$ $\mathcal{L}_{I_{i}}=-\log\left(\frac{1}{\sqrt{2\pi\sigma_{i}^{2}}}\exp\left(- \frac{\|\hat{C}_{i}^{{}^{\prime}}-C_{i}\|_{2}^{2}}{2\sigma^{2}}\right)\right).$ (13) $$

where $\hat{C}_{i}^{{}^{\prime}}=\mathcal{A}(\hat{C}_{i},\{\hat{C}_{i}\}_{i=1}^{K^{
\prime}}\setminus\hat{C}_{i})$ and $\mathcal{A}$ is the global alignment function. Through empirical study, we find a well-aligned mapping function $\mathcal{A}$ from the transformer decoder of DUSt3R, which builds the confidence maps $\left\{\mathcal{C}_{i}\right\}_{i=1}^{K^{\prime}}$ for each generated frames $\left\{\bm{I}_{i}\right\}_{i=1}^{K^{\prime}}$. Specifically, the confidence score tends to be lower in areas that are difficult to estimate (*e.g., *regions with solid colors) while the score will be higher in areas with less uncertainty. Moreover, we introduce LPIPS (Zhang et al., ) loss to remove the artifacts and further enhance the visual quality. Towards this end, we formulate the confidence-aware 3DGS loss as:

$$ $\mathcal{L}_{\text{conf}}=\sum_{i=1}^{K^{\prime}}\mathcal{C}_{i}\left(\lambda_ {\text{rgb}}\mathcal{L}_{1}(\hat{\bm{I}_{i}},\bm{I}_{i})+\lambda_{\text{ssim}} \mathcal{L}_{\text{ssim}}(\hat{\bm{I}_{i}},\bm{I}_{i})+\lambda_{\text{lpips}} \mathcal{L}_{\text{lpips}}(\hat{\bm{I}_{i}},\bm{I}_{i})\right).$ (14) $$

where $\mathcal{L}_{1}$, $\mathcal{L}_{\text{ssim}}$, and $\mathcal{L}_{\text{lpips}}$ denote the $L_{1}$, SSIM, and LPIPS loss, respectively, with $\lambda_{\text{rgb}}$, $\lambda_{\text{ssim}}$, and $\lambda_{\text{lpips}}$ being their corresponding coefficient parameters. In comparison to the photometric loss (*e.g., *$\mathcal{L}_{1}$ and $\mathcal{L}_{\text{ssim}}$), LPIPS loss is mainly focused on the high-level semantic information.

**Table 1: Quantitative comparisons for small angle variance in input views. For each scene, the model takes two views as input and renders three novel views for evaluation. We outperform all baseline methods in terms of PSNR, LPIPS, and SSIM for novel view synthesis on the real-world RealEstate10K (Zhou et al., ) and ACID (Liu et al., ) datasets.**
| Method | RealEstate10K | ACID |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |  |
| pixelNeRF (Yu et al., ) | 20.43 | 0.589 | 0.550 | 20.97 | 0.547 | 0.533 |
| GPNR (Suhail et al., ) | 24.11 | 0.793 | 0.255 | 25.28 | 0.764 | 0.332 |
| AttnRend (Du et al., ) | 24.78 | 0.820 | 0.213 | 26.88 | 0.799 | 0.218 |
| MuRF (Xu et al., ) | 26.10 | 0.858 | 0.143 | 28.09 | 0.841 | 0.155 |
| pixelSplat (Charatan et al., ) | 25.89 | 0.858 | 0.142 | 28.14 | 0.839 | 0.150 |
| MVSplat (Chen et al., ) | 26.39 | 0.839 | 0.128 | 28.25 | 0.843 | 0.144 |
| ReconX (Ours) | 28.31 | 0.912 | 0.088 | 28.84 | 0.891 | 0.101 |

Figure: Figure 3: Qualitative comparison for small angle variance in input views. The first four rows are from RealEstate10K (Zhou et al., ), while the last one is from ACID (Liu et al., ). Given two nearby input sparse views, our approach produces more accurate and perceptually appealing images in test scenes than the baselines.
Refer to caption: x3.png

## 6 Experiments

In this section, we conduct extensive experiments to evaluate our sparse-view reconstruction framework ReconX. We first present the setup of the experiment (Sec 6.1). Then we report our qualitative and quantitative results compared to representative baseline methods in various settings such as different angle variances from input views and cross-dataset generalization (Sec 6.2). Finally, we conduct ablation studies to further verify the efficacy of our framework design (Sec 6.3). Please refer to our project page for more comparisons and visualizations.

### 6.1 Experiment Setup

Implementation Details. In our framework, we choose DUSt3R (Wang et al., ) as our unconstrained stereo 3D reconstruction backbone and I2V model DynamiCrafter (Xing et al., ) (@ $512\times 512$ resolution) as the video diffusion backbone. We first finetune the image cross-attention layers with 2000 steps on the learning rate $1\times 10^{-4}$ for warm-up. Then we incorporate the 3D structure condition $c_{\text{struc}}$ into the video diffusion model and further finetune the spatial layers with 30K steps on the learning rate of $1\times 10^{-5}$. Our video diffusion was trained on 3D scene datasets by sampling 32 frames with dynamic FPS at the resolution of $512\times 512$ in a batch. The AdamW (Loshchilov & Hutter, ) optimizer is employed for optimization. At the inference of our video diffusion, we adopt DDIM sampler (Song et al., ) using multi-condition classifier free guidance (Ho & Salimans, ). Similar to Xing et al. (), we adopt tanh gating to learn $\lambda_{\mathcal{F}}$ adaptively. The training is conducted on 8 NVIDIA A800 (80G) GPUs in two days. In 3DGS optimization stage, we choose the point maps of first and end frames as the initial global point cloud and all 32 generated frames are used to reconstruct the scene. Our implementation follows the pipeline of the original 3DGS (Kerbl et al., ), but unlike this method, we omit the adaptive control process and attain high-quality renderings in just 1000 steps. The coefficients $\lambda_{\text{rgb}}$, $\lambda_{\text{ssim}}$, and $\lambda_{\text{lpips}}$ are set to 0.8, 0.2, and 0.5, respectively.

**Table 2: Quantitative comparison for large angle variance in input views. Given input views with large angle variance, our results have larger improvements than SOTA methods.**
| Method | ACID | RealEstate10K |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ |  |
| pixelSplat (Charatan et al., ) | 16.83 | 0.476 | 0.494 | 19.62 | 0.730 | 0.270 |
| MVSplat (Chen et al., ) | 16.49 | 0.466 | 0.486 | 19.97 | 0.732 | 0.245 |
| ReconX (Ours) | 24.53 | 0.847 | 0.083 | 23.70 | 0.867 | 0.143 |

Figure: Figure 4: Qualitative comparison for large angle variance in input views. Given more challenging settings (*i.e., *input views are far away from each other as the frame ID shows), our method ReconX demonstrates more improvements and superiority than baseline methods.
Refer to caption: x4.png

Datasets. The video diffusion model of ReconX is trained on three datasets: RealEstate-10K (Zhou et al., ), ACID (Liu et al., ), and DL3DV-10K (Ling et al., ) based on the pretrained model. RealEstate-10K is a dataset downloaded from YouTube, which is split into 67,477 training scenes and 7,289 test scenes. The ACID dataset consists of natural
landscape scenes, with 11,075 training scenes and 1,972 testing scenes. DL3DV-10K is a large-scale outdoor dataset containing 10,510 videos with consistent capture standards. For each scene video, we randomly sample 32 contiguous frames with random skips and serve the first and last frames as the input for our video diffusion model. To further validate our cross-data generalizability, we also directly evaluate our method on DTU (Jensen et al., ) dataset and NeRF-LLFF (Mildenhall et al., ) dataset which are two popular multi-view datasets in NVS task.

Baselines and Metrics. We compare our ReconX with original 3DGS (Kerbl et al., ) and several representative baselines that focus on sparse-view 3D reconstruction, including: NeRF-based pixelNeRF (Yu et al., ) and MuRF (Xu et al., ); Light Field based GPNR (Suhail et al., ) and AttnRend (Du et al., ); and the recent state-of-the-art 3DGS-based pixelSplat (Charatan et al., ) and MVSplat (Chen et al., ). For quantitative results, we report the standard metrics in NVS, including PSNR, SSIM (Wang et al., ), LPIPS (Zhang et al., ). For a fair comparison, we conduct experiments on $256\times 256$ resolutions following existing methods (Charatan et al., ; Chen et al., ).

### 6.2 Comparison with Baselines

Comparison for small angle variance in input views. For fair comparison with baseline methods like MuNeRF (Xu et al., ), pixelSplat (Charatan et al., ), and MVSplat (Chen et al., ), we first compare our reconX with baseline method from sparse views with small angle variance (see Table 1 and Figure 3). We observe that our ReconX surpasses all previous state-of-the-art models in terms of all metrics on visual quality and qualitative perception.

Comparison for large angle variance in input views. As MVSplat and pixelSplat are much better than previous baselines, we conduct thorough comparisons with them in more difficult settings. In more challenging settings (*i.e., *given sparse views with large angle variance), our proposed ReconX demonstrate more significant improvement than baselines, especially in unseen and generalized viewpoints (see Table 2 and Figure 4). This clearly shows the effectiveness of ReconX in creating more consistent observations from video diffusion to mitigate the inherent ill-posed sparse-view reconstruction problem.

Cross-dataset generalization. Unleashing the strong generative power of the video diffusion model through 3D structure guidance, our ReconX is inherently superior in generalizing to out-of-distribution novel scenes. To demonstrate our strong generalizability, we conduct two cross-dataset evaluations. For a fair comparison, we train the models solely on the RealEstate10K and directly test them on two popular NVS datasets (*i.e., *NeRF-LLFF (Mildenhall et al., ) and DTU (Jensen et al., )). As shown in Table 3 and Figure 5, the competitive baseline methods MVSplat (Chen et al., ) and pixelSplat (Charatan et al., ) fail to render such OOD datasets which contain different camera distributions and image appearance, leading to dramatic performance degradation. In contrast, our ReconX shows impressive generalizability and the gain is larger when the domain gap from training and test data becomes larger.

Assessing more-view quality. Our framework ReconX is agnostic to the number of input views. Specifically, given $N$ views as input, we sample a plausible camera trajectory to render image pairs using our video diffusion models and finally optimize the 3D scene from all generated frames. For a fair comparison with Chen et al. (), we verify this by testing on DTU with three context views. Our results are PSNR: 22.83, SSIM: 0.512, LPIPS: 0.317, MVSplat’s are PSNR: 14.30, SSIM: 0.508, LPIPS: 0.371, and pixelSplat’s are PSNR: 12.52, SSIM: 0.367, LPIPS: 0.585. Compared to the two-view results (Table 3), our ReconX and MVSplat both achieve better performance given more input views while we are much better than MVSplat. However, pixelSplat performs worse when using more views also shown in Chen et al. (). More visualization of our results with more input views can be found in our project page.

### 6.3 Ablation Study and Analysis

We carry out ablation studies on RealEstate10K to analyze the design of our ReconX framework in Table 4 and Figure 6. A naive combination of pretrained video diffusion model and Gaussian Splatting is regarded as the “base”. Specifically, we ablation on the following aspects of our method: 3D structure guidance, confidence-aware optimization, and LPIPS loss. The results indicate that the omission of any of these elements leads to a degradation in terms of quality and consistency. Notably, the basic combination of original video diffusion model and 3DGS leads to significant distortion of the scene. The absence of 3D structure guidance causes inconsistent generated frames especially in distant input views, resulting in blur and artifact issues. The lack of confidence-aware optimization leads to suboptimal results in some local detail areas. Adding LPIPS loss in confidence-aware 3DGS optimization would provide clearer rendering views. This illustrates the effectiveness of our overall framework (Figure 2), which drives generalizable and high-fidelity 3D reconstruction given only sparse views as input.

Figure: Figure 5: Qualitative results in cross-dataset generalization. Models trained on the source dataset RealEstate10K are tested on NeRF-LLFF and DTU datasets.
Refer to caption: x5.png

Figure: Figure 6: Visualization results of ablation study. We ablate the design choices of 3D structure guidance, confidence-aware optimization, and the LPIPS loss.
Refer to caption: x6.png

**Table 3: Quantitative results in cross-dataset generalization. Models trained on the source dataset RealEstate10K are tested on unseen scenes from target datasets NeRF-LLFF (Mildenhall et al., ) and DTU (Jensen et al., ), without any fine-tuning.**
| Training data | Method | NeRF-LLFF | DTU |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| RealEstate10K (Zhou et al., ) | pixelSplat (Charatan et al., ) | 11.42 | 0.312 | 0.611 | 12.89 | 0.382 | 0.560 |
| MVSplat (Chen et al., ) | 11.60 | 0.353 | 0.425 | 13.94 | 0.473 | 0.385 |  |
|  | ReconX (Ours) | 21.05 | 0.768 | 0.178 | 19.78 | 0.476 | 0.378 |

**Table 4: Quantitative results of ablation study. We report the quantitative metrics in ablations of our framework in real-world data RealEstate10K (Zhou et al., ).**
| Setup | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| --- | --- | --- | --- |
| ReconX (Ours) | 28.31 | 0.912 | 0.088 |
| base | 19.70 | 0.789 | 0.229 |
| w/o 3D structure guidance | 25.13 | 0.901 | 0.131 |
| w/o confidence-aware opt. | 26.83 | 0.897 | 0.097 |
| w/o LPIPS loss | 27.47 | 0.906 | 0.111 |

## 7 Conclusion

In this paper, we introduce ReconX, a novel sparse-view 3D reconstruction framework that reformulates the inherently ambiguous reconstruction problem as a generation problem. The key to our success is that we unleash the strong prior of video diffusion models to create more plausible observations frames for sparse-view reconstruction. Grounded by empirical study and theoretical analysis, we propose to incorporate 3D structure guidance into the video diffusion process for better 3D consistent video frames generation. What’s more, we propose a 3D confidence-aware scheme to optimize the final 3DGS from generated frames, which effectively address the uncertainty issue. Extensive experiments demonstrate the superiority of our ReconX the latest state-of-the-art methods in terms of high quality and strong generalizability in unseen data.

Limitations and Future Work. Although our ReconX achieves remarkable reconstruction results in novel viewpoints, the quality still seems to be limited to the backbone itself as we choose the U-Net based diffusion model DynamiCrafter (Xing et al., ) in this work. We expect them to be solved with open-sourced larger video diffusion models (*e.g., *DiT-based framework). In the future, it is interesting to integrate 3DGS optimization directly with the video generation model, enabling more efficient end-to-end 3D scene reconstruction. We are also interested in exploring consistent 4D scene reconstruction. We believe that ReconX provides a promising research direction to craft intricate 3D worlds from video diffusion models and will inspire more works in the future.

## Appendix A Theoretical Proof

Proposition 1.
Let $\bm{\theta}^{*},\psi^{*}=g^{*}$ be the optimal solution of the solely image-based conditional diffusion scheme and $\bm{\tilde{\theta}}^{*},\tilde{\psi}^{*}=\{{g}^{*},\mathcal{F}^{*}\}$ be the optimal solution of diffusion scheme with native 3D prior. Suppose the divergence $\mathcal{D}$ is convex and the embedding function space $\Psi$ includes all measurable functions, we have
$\mathcal{D}(q\left(\bm{x}\right)\|p_{\bm{\tilde{\theta}}^{*},\tilde{\psi}^{*}}
(\bm{x}))<\mathcal{D}\left(q\left(\bm{x}\right)\|p_{\bm{\theta}^{*},\psi^{*}}(
\bm{x})\right)$.

Proof. According to the convexity of $\mathcal{D}$ and Jensen’s inequality $\mathcal{D}(\mathbb{E}[X])\leq\mathbb{E}[\mathcal{D}(X)]$, where $X$ is a random variable, we have:

$$ $\displaystyle\mathcal{D}\left(q(\bm{x})\|p_{\bm{\tilde{\theta}}^{*},\tilde{ \psi}^{*}}(\bm{x})\right)$ $\displaystyle=\mathcal{D}\left(\mathbb{E}_{q(s)}q(\bm{x}|s)\|\mathbb{E}_{q(s)} p_{\bm{\tilde{\theta}}^{*},\tilde{\psi}^{*}}(\bm{x}|s)\right)$ (15) $\displaystyle\leq\mathbb{E}_{q(s)}\mathcal{D}\left(q(\bm{x}|s)\|p_{\bm{\tilde{ \theta}}^{*},\tilde{\psi}^{*}}(\bm{x}|s)\right)$ $\displaystyle=\mathbb{E}_{q(s)}\mathcal{D}\left(q(\bm{x}|s)\|p_{\bm{\tilde{ \theta}}^{*},{g}^{*},\mathcal{F}^{*}}(\bm{x}|s)\right),$ $$

where we incorporate an intermediate variable $s$, which represents a specific scene. $q(\bm{x}|s)$ indicates the conditional distribution of rendering data $\bm{x}$ given the specific scene $s$. According to the definition of $\bm{\tilde{\theta}}^{*},{g}^{*},\mathcal{F}^{*}$, we have:

$$ $\displaystyle\mathbb{E}_{q(s)}\mathcal{D}\left(q(\bm{x}|s)\|p_{\bm{\tilde{ \theta}}^{*},{g}^{*},\mathcal{F}^{*}}(\bm{x}|s)\right)$ $\displaystyle=\min_{\bm{{\theta}},{g},\mathcal{F}}\mathbb{E}_{q(s)}\mathcal{D} \left(q(\bm{x}|s)\|p_{\bm{{\theta}},{g},\mathcal{F}}(\bm{x}|s)\right)$ (16) $\displaystyle=\min_{\bm{{\theta}}}\mathbb{E}_{q(s)}\min_{{g}(s),\mathcal{F}(s) }\mathcal{D}\left(q(\bm{x}|s)\|p_{\bm{{\theta}},{g}(s),\mathcal{F}(s)}(\bm{x})\right)$ $\displaystyle=\min_{\bm{{\theta}}}\mathbb{E}_{q(s)}\min_{{g},E}\mathcal{D} \left(q(\bm{x}|s)\|p_{\bm{{\theta}},{g},E}(\bm{x})\right),$ $$

where $E$ is the general 3D encoder in 3D structure conditional scheme while it is a redundant embedding in solely image-based conditional scheme, *i.e., *$\psi=\{g,E(\varnothing)\}$. Combining Equation 15 and 16, we have:

$$ $\displaystyle\mathcal{D}\left(q(\bm{x})\|p_{\bm{\tilde{\theta}}^{*},\tilde{ \psi}^{*}}(\bm{x})\right)$ $\displaystyle\leq\min_{\bm{{\theta}}}\mathbb{E}_{q(s)}\min_{{g},E}\mathcal{D} \left(q(\bm{x}|s)\|p_{\bm{{\theta}},{g},E}(\bm{x})\right)$ (17) $\displaystyle<\min_{\bm{\theta},g,E}\mathcal{D}\left(q(\bm{x})\|p_{\bm{\theta} ,g,E}(\bm{x})\right)=\min_{\bm{\theta},g,E(\varnothing)}\mathcal{D}\left(q(\bm {x})\|p_{\bm{\theta},g,E(\varnothing)}(\bm{x})\right)$ $\displaystyle=\min_{\bm{\theta},\psi}\mathcal{D}\left(q(\bm{x})\|p_{\bm{\theta },\psi}(\bm{x})\right)=\mathcal{D}\left(q\left(\bm{x}\right)\|p_{\bm{\theta}^{ *},\psi^{*}}(\bm{x})\right).$ $$

The second inequality holds because given any specific scene $s$ in any parameter $\bm{\theta}\in\Theta$, approximating $q(\bm{x}|s)$ is simpler than $q(\bm{x})$ by only tuning the encoder $E$ of $p_{\bm{\theta},g,E}$(^1^11A simple verifiable case is to optimize the parameters of 3DGS by only 2D images (solely image-based conditional learning) or using a SFM initialization from collected images (native 3D conditional learning) before optimization. The latter provides a more constrained and optimal solution space.), *i.e., *$\min_{E}\mathcal{D}\left(q(\bm{x}|s)\|p_{\bm{{\theta}},{g},E}(\bm{x})\right)<
\min_{E}\mathcal{D}\left(q(\bm{x})\|p_{\bm{\theta},g,E}(\bm{x})\right)$.

Consequently, the proof of Proposition 1 has been done.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - Is the mentioned counterexample for the proof correct or am I missing something?
- What exactly is the implementation of the learnable embedding function PosEmb in equation 3?
- What is meant with "uncertainty of unconstrained images" (line 265)? Does this mean that generated images are not fully 3D-consistent?
- Regarding the confidence-aware 3DGS optimization in section 4.4, how is the loss in equation 6 still used / relevant for the final loss in equation 7, which does not use equation 6 anymore?
- Is there any specific reason why you limit yourself to the case of view interpolation and do not evaluate extrapolation?
soundness: 3
strengths: - The paper tackles an important and very challenging task of reconstructing general scenes from very sparse views.
- The two-stage approach leverages the strong prior of video diffusion models and the technical contributions fit well into this framework.
- The experimental evaluation shows strong results consistently outperforming state-of-the-art baselines on both
   - in training distribution data 
     - significant quantitative and qualitative advantage for small angle variance in input views (table 1, figure 3)
     - even larger gap for large angle variance in input views (table 2, figure 4)
  - and cross-dataset generalization
    - baselines fail completely, while the proposed method achieves reasonable metric scores (table 4)
    - impressive qualitative results (figure 6)
- The ablation study validates the effectiveness of the proposed 3D structure conditioning and the confidence-aware 3DGS optimization (especially convincing qualitative ablation in figure 5).
- The paper is mostly well-written and easy to follow.
- The supplemental video provides additional convincing qualitative reconstruction results.
summary: The paper proposes a method or 3D scene reconstruction from as few as two views via a two-stage approach. First, a finetuned video diffusion model generates multiple view that interpolate the camera trajectory between the given input views. In order to improve 3D consistency of these generated images, the authors introduce a 3D structure conditioning by encoding a point cloud obtained from an existing learned stereo reconstruction method (DUSt3R). In a second stage, the generated views are fused into a 3D representation, for which the authors extend 3D Gaussian Splatting by a weighting based on confidence of DUSt3R.
Experimental evaluation results show strong performance for both in training distribution data as well as cross-dataset generalization, outperforming state-of-the-art baselines. Ablation studies validate the effectiveness of the technical contributions.
weaknesses: - The lack of precise mathematical formulation raises doubt about proposition 1 and its proof:
  - The main inequality in equation 17 (appendix) is justified only verbally and is not obvious to me.
  - A counterexample for the inequality in line 841 could be a dataset mainly consisting of dark rooms with all ground truth renderings being black except for one where the lights are on (with possibly very complex geometry or even transparent and reflective materials). For this case, fitting the marginal distribution can be easier than fitting the conditional distribution, if the given scene s happens to be the one with lights on.
  - Moreover, if the 3D structure (point cloud) is obtained from 2D images only via stereo reconstruction (DUSt3R [1]), then an image encoder should be theoretically able to encode the images in the same way as the composition of DUSt3R and the point cloud encoder used in the method. Following this argument, a strict inequality as in proposition 1 is not reasonable.

- Some lack of clarity:
  - What exactly is the implementation of the learnable embedding function PosEmb in equation 3?
  - What is meant with "uncertainty of unconstrained images" (line 265)? Does this mean that generated images are not fully 3D-consistent?
  - Regarding the confidence-aware 3DGS optimization in section 4.4, it is unclear how the loss in equation 6 is still used / relevant for the final loss in equation 7, which does not use equation 6 anymore.

- Somewhat unfair comparison with baselines:
  - The proposed method relies heavily on the strong priors learned by DUSt3R [1] and the video diffusion model.
  - All baselines are trained more or less from scratch on small-scale datasets compared to the large-scale datasets for training video diffusion models.
  - Therefore, it is not that surprising that the proposed method generalizes much better to other datasets out of the training/finetuning data distribution.

- Restriction to view interpolation:
  - Given that the method makes use of a pre-trained video diffusion model, the restriction to interpolation of the camera trajectory between two input views is unsatisfactory.
  - In many cases, two views as conditioning already eliminate uncertainty in the reconstruction task entirely such that deterministic approaches like pixelSplat [4] and MVSplat [5] already produce detailed and sharp reconstructions.
  - The proposed generative approach with a strong prior trained on large-scale single view data should be able to extrapolate from input views to some extent, which the paper misses to evaluate.
  - This is especially critical, if the paper motivates the method with the limitation of generalizable 3D reconstruction methods that "struggle to generate high-quality images in areas not visible from the input perspectives" (lines 143f.).

- Missing relevant related work and possible baseline:
  - latentSplat [2] aims to bridge the gap between regression models for generalizable NVS and generative models for 3D reconstruction and is therefore a very relevant work and potential baseline.
  - GeNVS [3] was one of the first approaches to generative novel views with a diffusion model conditioned on 3D-aware (pixelNeRF) features and is therefore important related work.

Minor comments:
- The paper sometimes uses the terms "conditioning" and "guidance" interchangeably, although they have different meanings to me (conditioning: giving something as additional input to the diffusion model; guidance: using CFG or also gradients (e.g. classifier guidance) to affect the sampling process). A more precise use of these terms would be helpful.
- The ablation study in table 3 could be extended to contain all different combinations of leaving out individual components in order to find possible dependencies between them.

References:
- [1] DUSt3R: Geometric 3D Vision Made Easy. CVPR 2024
- [2] latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstruction. ECCV 2024
- [3] Generative Novel View Synthesis with 3D-Aware Diffusion Models. ICCV 2023
- [4] pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction. CVPR 2024
- [5] MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images. ECCV 2024

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Since DUSt3R is a pose-free stereo estimation pipeline, the estimated point cloud would not be aligned with the ground truth poses of benchmarks like LLFF/DTU. To obtain synthesized novel views, how is the reconstructed Gaussian point cloud aligned with the COLMAP cameras during inference? I did not find any information regarding that.

2. Does the choice of the video diffusion model have a major impact on the final reconstruction? How would Stable Video Diffusion, for example, work in combination with the proposed 3D structure conditioning? Is there any specific reason behind picking DynamiCrafter?

3. How many scenes from the 3 datasets - RealEstate10k, ACID, and DL3DV-10K are used for training the video diffusion model? I do not see a mention of the total count, so I am assuming all?

4. Can you please provide some more details on the transformer-based encoder for embedding the DUSt3R point cloud? Is it one of the pretrained point Transformer architectures (or maybe the point-SAM encoder), and is it also finetuned along with the video diffusion model? From Figure 2, I am guessing not, but such encoders are usually trained with point clouds from Scannet-like benchmarks, and DUSt3R point clouds would most likely be OOD in terms of overall structure and density.
soundness: 3
strengths: 1. The proposed 3D point cloud conditioning for ensuring the 3D consistency of generated frames is novel and intuitive.
2. The video diffusion architecture is well-ablated, and each of the highlighted contributions impacts the final reconstruction quality positively.
3. Extensive experiments demonstrate ReconX's ability to achieve high-quality reconstructions that outperform related state-of-the-art methods, particularly in challenging scenarios where there is a large angle variance between input views. ReconX also shows robust generalizability across datasets, a notable limitation of baselines pixelSplat and MVSplat.
summary: ReconX proposes video diffusion priors conditioned with 3D structure guidance for reconstruction and novel view synthesis in a 2-view setting. Specifically, the authors leverage the DUSt3R method to obtain a global dense point cloud and encode it into the latent space of the video diffusion model with point transformers for providing sufficient 3D context and obtaining a 3D-consistent novel view coherent with the input images. Finally, a 3D-confidence-aware optimization scheme with 3DGS robustly reconstructs the scene in a few iterations. Experiments show that ReconX outperforms related 2-view baselines - pixelSplat and MVSplat on both in-distribution and OOD benchmarks.
weaknesses: 1. Weak Benchmarks: ReconX shows strong sparse-view reconstruction capabilities for scenes from LLFF and DTU datasets, but these are relatively simpler benchmarks, and since the method uses strong video diffusion priors, I would expect a comparison on more challenging benchmarks like MipNeRF360 (or Tanks and Temples). CAT3D / ReconFusion already provides data splits for 3, 6, and 9 view settings for this dataset, so comparisons with a generalized N-view setting would strengthen the submission further.
One can also estimate a DUSt3R point cloud at low cost for N <= 9, so I do not find any bottlenecks in that regard.

2. Weak Baselines: Both baselines - pixelSplat and MVSplat are non-generative in nature. As such, comparison with methods integrating video diffusion priors for sparse-view reconstruction like CAT3D and V3D seems quite relevant.

3. Missing Ablation: The method uses a DUSt3R point cloud initialization for optimizing 3DGS. However, the baseline involving optimization of 3DGS with this dense stereo point cloud is missing from Table 3. Since the authors optimize 3D Gaussians for only 1000 iterations without Adaptive Density Control, my guess is that the DUSt3R point cloud already provides a very strong scene structure. As such, assessing the impact of the video diffusion priors becomes more important. I assume the variant denoted as “base” in Table 3 already integrates the DUSt3R priors with 3DGS.

References:

1. CAT3D: Create Anything in 3D with Multi-View Diffusion Models, Ruiqi Gao* and Aleksander Holynski* and Philipp Henzler and Arthur Brussee and Ricardo Martin-Brualla and Pratul P. Srinivasan and Jonathan T. Barron and Ben Poole*, 2024
2. V3D: Video Diffusion Models are Effective 3D Generators, Zilong Chen and Yikai Wang and Feng Wang and Zhengyi Wang and Huaping Liu, 2024

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Please address the issues raised in the weaknesses section. Specifically:

- If there is time, please provide experimental comparisons with per-scene sparse input NVS methods as elaborated above, in terms of quantitative results, and if possible side-by-side video results.
  - If there are good reasons why it is not appropriate to compare to these baselines, please explain.
- Define the global alignment function $\mathcal{A}$ and explain its role.
- Clarify the vague statement in L293-295, regarding confidence maps for generated images.
- Clarify how the DL3DV-10K dataset is used in the experiments.

REVISED review: Given the major effort the authors have put into the response and revision, I have raised my rating to 5. It is not higher, however, because many of the issues were very obvious (e.g. comparison to feed-forward methods only), and could have been avoided at point of submission. As such, there is insufficient time during the rebuttal period for proper due diligence, on both the part of the authors and reviewers.
soundness: 3
strengths: There does not appear to be previous work combining ideas from Dust3r, video diffusion and 3DGS, so this work is novel in that sense, although there are concurrent ones (ViewCrafter [1], LM-Gaussian [2], 3DGS-Enhancer [3], MVSplat360 [4]) that explore similar ideas.

The proposed methodology appears to be sound, and mostly well-justified. Although this perhaps explains convergent ideas as highlighted by the concurrent works.

Most of the paper is mostly well-written and diagrams are clear, except for a few parts highlighted in the Weaknesses section.

References
- [1] Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien-Tsin Wong, Ying Shan, Yonghong Tian, ViewCrafter: Taming video diffusion models for high-fidelity novel view synthesis, arXiv preprint arXiv:2409.02048, 3 Sep 2024.
- [2] Hanyang Yu, Xiaoxiao Long, Ping Tan, LM-Gaussian: Boost sparse-view 3D Gaussian splatting with large model priors, arXiv preprint arXiv:2409.03456v1, 5 Sep 2024.
- [3] Xi Liu, Chaoyi Zhou, Siyu Huang, 3DGS-Enhancer: Enhancing unbounded 3D Gaussian splatting with view-consistent 2D diffusion priors, NeurIPS 2024. Also as arXiv preprint arXiv:2410.16266, 21 Oct 2024.
- [4] Yuedong Chen, Chuanxia Zheng, Haofei Xu, Bohan Zhuang, Andrea Vedaldi, Tat-Jen Cham, Jianfei Cai, MVSplat360: Benchmarking 360° generalizable 3D novel view synthesis from sparse views, NeurIPS 2024.
summary: This paper proposes a framework for obtaining a 3DGS representation of a scene from sparse views, intended for novel view synthesis. It first uses Dust3r to recover a point cloud representation of the scene, which is only a rough estimate of the 3D structure due to the sparsity of views. The point cloud is converted into a learned 3D embedding through an FFN, for injecting as a condition into a video diffusion model to generate intermediate novel views that are consistent with the 3D embedding. Next, each generated view appears to have its pixel values modified by some alignment function, and also assigned a confidence map. The final 3DGS representation is then obtained by standard 3DGS optimization except weighted by the confidence values. Although the proposed method involves per-scene 3DGS optimization, the comparisons are primarily made to feed-forward methods (e.g. MuRF, pixleSplat, MVSplat), and unsurprisingly outperforms them.
weaknesses: **Problems with experimental settings**

The primary weakness of this paper lies in the experimental section. The baseline methods chosen seem to be the wrong ones to compare against, as they are all _feed-forward_ methods that produce 3DGS representations (pixelSplat, MVSplat) or radiance fields (MuRF, pixelNeRF, GPNR), done _in a single pass_. In contrast, the proposed method is based on per-scene 3DGS optimization, which unsurprisingly will perform better.

Furthermore, the choice of datasets for testing is limited. For the main results, only RealEstate10k and ACID, with LLFF and DTU only tested on for cross-dataset generalization (which, as an experiment setting itself, is much less impacted in a per-scene optimization method, compared to feed-forward methods). The paper also mentions DL3DV-10K (L373), but it is unclear how and where this is used in the experiments.

Instead, the appropriate baselines, settings and choice of datasets should follow that of per-scene sparse view NVS papers, such as conducted in ReconFusion (Wu 2024b) and Cat3D (Gao 2024), for a fairer comparison.

**Other weaknesses**

A global alignment function is mentioned in L293, but this is neither defined nor is there explanation for its purpose. Based on the brief expression in L293, it appears to do some kind of color adjustment of the video diffusion generated images.

L293-295:
> Through empirical study, we find a well-aligned mapping function $\mathcal{A}$ from the transformer decoder of DUSt3R, which builds the confidence maps ...

This statement is very vague and contradictory. Does $\mathcal{A}$ do color adjustment, or does it produce confidence maps? Is it a handcrafted function? How does it relate to the Dust3r transformer decoder?

The utility of Proposition 1 (L185-189) is unclear. It appears to express that having a 3D condition is less ill-posed than a 2D condition, which is already universally accepted. If so, it would be a distraction from the key ideas of the paper, so I would suggest leaving this out of the paper.

Minor point, but for equation 5 (L256), it would seem that $c_{view}$ and $c_{struc}$ should be deterministically tied to any sampled $x\sim p$. So there is no separate marginalization, and should not appear in the subscript of the expectation $\mathbb{E}$.

Also minor formatting issue, but having Table 4 (L432-436) placed above Table 3 (L468-478) is strange and breaks the flow.

The supplementary video only includes results from the authors' method, but not from the baselines. To better judge qualitative improvements, it would be much better to include video results for all compared methods side-by-side. The few frames in the main paper are not really sufficient, and can easily be cherry-picked.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Please refer to the Weakness.
soundness: 2
strengths: 1. Using temporal consistency generation to solving the multi view consistency problem in sparse view reconstruction 
2. Including the point cloud as an additional condition for the video generation model and achieving great performance is novel and effective.
3. Applying confidence-aware strategy to optimize the 3DGS to improve the artifacts introduced by the diffusion model.
4. Comprehensive experiments have been done, including different number of views, in and cross-domain, to demonstrate great performance of this work.
summary: This paper presents ReconX, which is aim to convert the sparse view 3D reconstruction to the temporal consistency generation by leveraging the video diffusion model as the prior. Moreover, the author uses the 3D structure guidance besides the common multi view images as the guidance, along with the mathematical prove that such new condition can improve the performance. Then, the confidence-aware 3DGS optimization strategy further ensure the performance to have the final novel view synthesis. Comprehensive experiments have been done to demonstrate the in and cross-domain ability of the model.
weaknesses: 1. More outdoor unbounded dataset can be included such as MipNeRF 360 in the experiment.
2. The performance relies heavily on the point cloud condition. But in the far camera distance case, which is common in MipNeRF 360, the point cloud may not be achieved from the spare input views or the point cloud may not include enough high-quality information, the performance of the diffusion model cannot be preserved. 
3. Even though the confidence-aware can boost the performance of 3DGS optimization, but the reason why transformer decoder from  DUSt3R can provide the confidence is not fully investigated while it only draws from empirical study.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weakness part. I will be glad to raise my rate if my concerns can be addressed!
soundness: 4
strengths: 1. The idea of repurposing a video diffusion model as a spatial interpolator sounds reasonable and can provide generative priors to unseen regions.
2. How this work injects structure-aware guidance by using the point cloud from DUSt3R seems critical and can provide insights for future work.
3. This work achieves visually better results than previous methods and seems to generalize better to large viewpoint changes.
4. The paper is well-written and should be easy to follow as long as the code is released.
summary: This paper reframes the ambiguous sparse-view reconstruction task as a temporal generation task and unleashes the strong generative prior of large pre-trained video diffusion models. To maintain consistency while generating the video clip, this work proposes to first construct a global point cloud using DUSt3R and then encode it into a contextual space to provide the 3D structure condition. Afterward, it will set the first frame as well as the last frame and repurpose the video diffusion model as an interpolator to synthesize plausible frames. Finally, it will incorporate a confidence-aware 3DGS optimization for reconstruction. Experiments show the paper's superiority in terms of quality and generalization capability to larger viewpoint change compared with previous work.
weaknesses: 1. One strength of directly applying a feed-forward reconstruction like pixelSplat[1] or MVSplat[2] is the inference speed. The proposed method needs to incorporate DUSt3R, a video diffusion model, and a confidence-aware optimization to get the final result. I wonder about the total time consumption during inference compared with baseline models.
2. From the reviewer's perspective, there seem to be multiple solutions given the first frame and the last frame of a video, would it be better to incorporate camera pose information during the training and inference of the video diffusion model? Or on the other side, is the video diffusion model trying to learn a certain type of camera motion due to the limitation of fine-tuning dataset? 
3. This paper proposes to utilize a global point cloud as the 3D structure guidance. From the ablation study in Fig.5, the quality seems to be poor without this guidance, showing the importance of incorporating this point cloud. While DUSt3R itself can provide a global point cloud as the reconstruction result, I would like to see a comparison with DUSt3R apart from MVSplat and pixelSplat. Moreover, pixelSplat and MVSplat do not incorporate a global point cloud as input. 
4. This work mainly deals with static scene reconstruction, it would be interesting to reconstruct a dynamic scene in future work, which is also mentioned by the authors in the future work part. 

[1] pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction, CVPR 2024 

[2] MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images, ECCV 2024

[3] DUSt3R: Geometric 3D Vision Made Easy, CVPR 2024

Meta-review: The paper presents ReconX, a novel framework for sparse-view 3D reconstruction that uses video diffusion models as spatial interpolators. It incorporates 3D structure conditioning using point clouds generated by DUSt3R to ensure consistency and robustness in the synthesis of novel views. Confidence-aware 3D Gaussian Splatting optimization refines the reconstruction by mitigating artifacts from the diffusion process. ReconX achieves state-of-the-art performance in interpolation tasks, shows promising extrapolation capabilities, and demonstrates robustness on challenging datasets such as MipNeRF360 and Tank & Temples.

Strengths:
- Strong performance: ReconX achieves state-of-the-art results on standard datasets, especially for view interpolation tasks. Incremental extrapolation experiments show promising extensions for 360-degree reconstructions.
- Technical Novelty: Novel combination of video diffusion models for 3D reconstruction, 3D structure conditioning, and confidence-aware optimization for 3DGS.
- Clear presentation: The methodology is well motivated and generally easy to follow, supported by qualitative and quantitative results.

Weaknesses:
- Concerns about experimental validity: discrepancies in Fig. 5(a) during the rebuttal raised doubts about the rigor of the experimental protocols.
Questions remain about adherence to the train-test split and the validity of certain representations.
- Limited evaluations: while interpolation results are strong, the paper lacks 360-degree video reconstructions for MipNeRF360 scenes and presents limited qualitative comparisons with baselines, particularly for video outputs.
- Inference time: ReconX is slower than other feedforward methods due to the inclusion of DUSt3R, video diffusion, and 3DGS optimization.

The main reasons for rejection are concerns about experimental validity and incomplete evaluations. The discrepancy in Figure 5(a) raised doubts about the adherence to train-test splits and the rigor of the experimental protocols. Despite the authors' transparency in addressing errors, these issues undermine confidence in the paper's claims and its ability to provide a thorough, reproducible evaluation of its contributions.

Decision: Reject