Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis

Authors: qitao  zhao
shubham  tulsiani
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 00:28:46

Abstract: Inferring the 3D structure underlying a set of multi-view images typically requires solving two co-dependent tasks -- accurate 3D reconstruction requires precise camera poses, and predicting camera poses relies on (implicitly or explicitly) modeling the underlying 3D. The classical framework of analysis by synthesis casts this inference as a joint optimization seeking to explain the observed pixels, and recent instantiations learn expressive 3D representations (e.g., Neural Fields) with gradient-descent-based pose refinement of initial pose estimates. However, given a sparse set of observed views, the observations may not provide sufficient direct evidence to obtain complete and accurate 3D. Moreover, large errors in pose estimation may not be easily corrected and can further degrade the inferred 3D. To allow robust 3D reconstruction and pose estimation in this challenging setup, we propose SparseAGS, a method that adapts this analysis-by-synthesis approach by: a) including novel-view-synthesis-based generative priors in conjunction with photometric objectives to improve the quality of the inferred 3D, and b) explicitly reasoning about outliers and using a discrete search with a continuous optimization-based strategy to correct them. We validate our framework across real-world and synthetic datasets in combination with several off-the-shelf pose estimation systems as initialization. We find that it significantly improves the base systems' pose accuracy while yielding high-quality 3D reconstructions that outperform the results from current multi-view reconstruction baselines.

Main paper: 
 introduction: 
 Consider the images of the robot shown in Fig. 1 . From just these few images, we humans can easily understand the 3D structure of this object -it has a cylindrical base supporting a tall body from which an arm extends to the front. We do this by aggregating the information across images into a consistent 3D mental model, e.g., the "front" view informs us of the width of the body and the "side" view(s) about the extended arm. But how do we know which image is to the "front" or to the "side" to begin with? As evidenced in seminal research of mental rotation [30] , we understand viewpoints by forming mental 3D models. Thus, to form mental 3D models, we need to understand the (relative) viewpoints across images, but doing so in turn requires a mental 3D model! This co-dependency in inferring shape and pose is one that any computational approach aiming to recover 3D from multiple views also needs to deal with. Indeed, classical approaches like Structurefrom-Motion (SfM) [29] tackle the two together and infer 3D and camera viewpoints. However, these correspondence-based methods can only infer sparse 3D representations and are not robust given a small set of images with limited overlap. To allow 3D inference in such sparse-view settings, recent learning-based approaches have pursued sparse-view reconstruction approaches [52, 45] , but assuming known precise camera poses. Separately, there have been several methods [18, 47, 48, 31] which predict camera viewpoints given a set of images. While these methods have led to impressive results for both 3D reconstruction and pose inference, their singular focus on only one task without tackling the other limits their utility -the 3D reconstruction methods requiring precise cameras cannot be easily used in real-world applications, and pose estimation methods that do not model 3D are typically limited in their accuracy.We present SparseAGS, a framework that unifies the advances in learning-based pose estimation and 3D reconstruction -using the former as an initialization and building on the latter for obtaining accurate 3D reconstruction. Specifically, we adopt an "analysis by synthesis" approach where we jointly optimize pose and 3D to explain the observed pixels. However, unlike prior methods [19, 42] which simply leverage photometric-error-driven gradient-based updates for pose and 3D, we additionally leverage generative priors [8, 33] for more complete (and accurate) 3D despite input images that may only partially capture the object. However, current off-the-shelf novel-view generative models [20] only allow 3-DoF camera parameterization which is insufficient beyond synthetic settings, we finetune a SoTA model to allow 6-DoF camera variation when querying novel views. We find that such generative priors not only contribute to the 3D reconstruction quality but also result in more accurate camera poses. Moreover, we also explicitly account for large possible errors in initial camera estimation and prevent these from degrading 3D reconstruction via identifying outliers, and also improve poses via a combination of a discrete search and continuous optimization.Compared to prior joint reconstruction and pose estimation methods that are designed to improve near-perfect initial cameras [19, 38] , SparseAGS can leverage off-the-shelf pose estimates, thereby allowing robust inference in real-world scenarios. We demonstrate its efficacy using both, realworld and synthetic datasets in conjunction with several state-of-the-art pose estimation methods as initialization. We find that our approach significantly improves the initial camera estimates and yields high-fidelity 3D reconstructions (and novel view renderings). In summary, our contributions are:• We introduce an analysis-by-generative-synthesis framework that jointly estimates 3D and camera viewpoints given a sparse set of input images, by integrating a 6-DoF novel-view generative prior in an analysis-by-synthesis approach • Our approach allows leveraging any off-the-shelf pose estimation system and can robustly estimate 3D and viewpoints despite large errors in the initial estimates. • We present results across datasets and initializations and show clear improvements over the initializations as well as outperform prior sparse-view 3D reconstruction baselines. 
related work: 
 Sparse-view Pose Estimation. Traditional correspondence-based Structure-from-Motion [32, 29] methods often fail to estimate camera poses in sparse-view settings. Several approaches instead seek to leverage data-driven priors, for example learning energy-based [48, 18] or denoising diffusion [39] models to predict cameras. While these approaches predict global camera representations, some works have demonstrated the benefit of denser camera parametrizations by predicting raymaps [47] or pointmaps [41, 17] . As an alternative paradigm to direct camera prediction, some recent methods [3, 43, 34] instead estimate relative poses by inverting the view-conditioned synthesis capabilities of diffusion models [20] . While these methods have led to remarkable improvements in camera estimation, these are still susceptible to some imprecision and occasional outliers which our 3Dreasoning-based approach can correct.Sparse-view 3D Reconstruction. This line of work aims to recover 3D from sparsely sampled views, aiming to infer complete 3D representations that faithfully reflect the content captured by the input images while making reasonable guesses for invisible areas. The progress of diffusion models [8, 33] has greatly advanced this direction, as they are capable of learning strong natural image priors from data. Inspired by DreamFusion [23] , which generates 3D scenes given textual descriptions leveraging a text-to-image diffusion model [26] , SparseFusion [52] learns a viewconditioned diffusion model on multi-view image collections for novel view synthesis and then distills the learned novel-view distributions into a single consistent 3D representation. DreamSparse [45] further improves the performance by utilizing internet-scale natural image priors learned by Stable Diffusion [25] . Although these methods present impressive results, they assume precise camera poses are available, which limits their applications. FORGE [11] addresses this by jointly inferring both camera poses and 3D structure in a single forward pass, though the quality of its reconstructions remains constrained by pose estimation accuracy without further refinement or correction.Pose-free Sparse-view 3D Reconstruction. Some recent works [27, 40, 12, 13] have attempted to bypass the reasoning about camera poses and directly infer novel views or 3D representations from unposed images. An unposed variant of the Scene Representation Transformer [27] encodes a set of input images into latent features and synthesizes novel views given the corresponding query rays (w.r.t. the viewpoint of the first image) using a transformer encoder and decoder. UpFusion [13] improved upon this by learning a diffusion model and distilling a consistent 3D representation via Score Distillation Sampling [35] , whereas LEAP [12] and PF-LRM [40] can directly predict (volumetric or triplane) 3D representations in a feedforward manner. While these methods demonstrate promising results, their geometry-free approach cannot easily capture the specific details across input images and they struggle to improve the 3D estimation with additional input images.Analysis-by-synthesis Approaches. Approaching visual perception as an inverse graphics task is classical idea in computer vision [15, 46] , and has been leveraged for inferring scene properties (e.g., object pose) by synthesizing visual content as close to observations as possible [2, 16, 51, 50] . Closer to our setup, prior approaches jointly optimize camera pose and 3D representation (e.g., NeRF [22] ) to explain the observed images [19, 42] but these methods are designed for dense observations and only handle small pose errors. Closer to our work, SPARF [38] focuses on the sparse-view setup, leveraging estimated pixel correspondence [37] as prior knowledge in addition to the standard photometric loss. However, reliably extracting such correspondences can be challenging, and false match estimates may even confuse pose refinement, leading to degraded 3D reconstruction. 
overview: 
 Analysis by Synthesis. Given a set of sparse-view images, denoted as I ≡ {I i } N i=1 , our goal is to reconstruct the underlying 3D structure θ and infer the camera poses corresponding to the input images Π ≡ {π i } N i=1 . This can be done by solving an analysis-by-synthesis problemEQUATIONwhere f θ is a rendering function. Eq. 1 demonstrates that we want to find a scene description consisting of a 3D representation θ and camera configurations Π that well explain the observed input images. If f θ is differentiable, we can jointly optimize the 3D representation and camera poses via gradient descent [19, 42] . However, this approach may not work well in the sparse-view setting [38] (i.e., N is small) as the 3D representation can overfit to the input images without forming a plausible structure, degrading both, pose estimation and 3D reconstruction.ℒ 456 = Average Prediction *+, " = 1, … , E 2 # (4 |C, 6 $ , ∆8 $ ) Outlier Removal 7ℒ !"# 7# $ × : {2 $ } ! ! (# $ * ) % " * 9 %$ * + … (a) (b) : %$ * ℰ( MV-DG H %$ * , I %$ * , H %$ * , I %$ * ) ℰ( MV-Analysis by Generative Synthesis. To address this issue, we propose to introduce generative priors into analysis by synthesis, so we term our method analysis by generative synthesis. In addition to the known-view objective (Eq. 1), we leverage diffusion priors [8, 33] to optimize renderings from randomly sampled novel views (π) as wellEQUATIONwhere p ϕ is the likelihood of the novel view rendering conditioned on the viewpoint π and inputs (I, Π), modeled by the diffusion model ϕ. The gradients for this objective can be obtained via Score Distillation Sampling (SDS) [23] , and intuitively, they encourage the renderings of the 3D representation to be plausible based on image distributions learned by the diffusion model.In the following, we first introduce a few preliminaries about an efficient single-view-to-3D approach (Sec. 3.2), on which we build our multi-view reconstruction method, MV-DreamGaussian, enabling analysis by generative synthesis in the wild (Sec. 3.3). Then, we present our complete framework that involves dealing with imprecise cameras (Sec. 3.4 ). An illustration of our approach is in Fig. 2 .DreamGaussian [36] generates 3D from a single image with a two-stage approach, achieving a satisfactory trade-off between speed and fidelity. The first stage optimizes 3D Gaussians [14] (parameterized by θ) using a combination of photometric loss (Eq. 1, except that the camera pose is not optimized) and SDS loss (Eq. 3) with a view-conditioned diffusion model, Zero-1-to-3 [20] . Specifically, for a randomly sampled novel view π, scheduled noise ϵ at timestep t is added to the latent of its rendering (the noisy latent is denoted by z t ). The training objective minimizes the difference between the predicted noise and the added noise, approximating the negative log-likelihood of the rendered image. The gradient of SDS loss is given byEQUATIONwhere w(t) is a weighting function, ϵ ϕ (•) is a U-Net trained to predict the added noise given the noisy latent z t , conditioned on the timestep t, reference image I 1 , and the relative camera pose ∆π. This stage efficiently builds the geometry of the object with rough texture, which takes 500 training steps (in about 1 minute). In the second stage, 3D Gaussians are converted to a textured mesh with Marching Cubes [21] , and only its texture is optimized. This stage takes another 50 steps and can finish within 30 seconds on a single GPU.We find DreamGaussian to be a suitable starting point to perform analysis by generative synthesis, but note that it has some key limitations: (1) 3-DoF Camera Parameterization. DreamGaussian adopts a 3-DoF camera parameterization (i.e., radius, elevation, and azimuth) to accommodate the camera definition in Zero-1-to-3 [20] . While this parameterization is sufficient for synthetic data, it cannot well represent the 6-DoF camera poses of real-world images. (2) Single Input Image.DreamGaussian is designed for the singe-view-to-3D task. In contrast, we aim for the reconstructed 3D to reflect the details captured by multiple input images faithfully. This requires an approach to handling information from multi-view images.We adapt DreamGaussian's two-stage method and extend it to (1) handle real-world images with 6-DoF camera parameters and (2) utilize sparse-view images as input.Generative Priors in the Wild. Zero-1-to-3 [20] offers desirable generative priors that enable single-view-to-3D generation of DreamGaussian. However, it assumes no in-plane camera rotation and that all possible camera poses are strictly directed toward a common origin. We find these assumptions are over-restrictive for real-world images. Therefore, we propose to replace the 3-DoF camera condition in Zero-1-to-3 with a 6-DoF one, represented as an 18-dimensional vector:EQUATION) where π rel is the relative extrinsic matrix (4×4) between the source view and target view, and f x rel (f y rel ) is the ratio of the focal length along the x-(y-) axis between them. We include the focal length term to account for the object scale change due to cropping. This simple camera parameterization effectively represents 6-DoF cameras in the wild. Details regarding finetuning Zero-1-to-3 for 6-DoF camera conditioning are deferred to Sec. C in the appendix. We note that recent work, ZeroNVS [28] , also discussed this 3-DoF issue of Zero-1-to-3 and proposed a "6-DoF+1" camera parameterization for scene-level novel view synthesis. However, this approach is not directly applicable to our objectcentric setting, as it is trained using images with complex backgrounds and leverages depth priors to address scale ambiguity.Leveraging the Generative Priors from Multiple Views. DreamGaussian only uses the generative priors from a single reference image via SDS loss. To make L SDS aware of the visual cues from multiple input images, we modify Eq. 3 asEQUATIONEQUATIONN is the total number of input views, I i is the i th input image, and ∆π i is its relative camera pose w.r.t. the sampled novel view π. We average the noise predictions from all input views that share the same timestep t. This method draws inspiration from the implementation of Stable-Dreamfusion [35] , but we do not weigh the predicted noises based on the relative closeness of camera poses. The rationale behind this is that the camera poses in our setting are not always reliable, and relying too heavily on "close" views could introduce significant conflicts during the 3D optimization process.With these modifications, our multi-view reconstruction approach, termed MV-DreamGaussian, is capable of reconstructing 3D from sparse images in the wild by leveraging diffusion priors. When describing its use in our overall framework, we use the notation θ = MV-DG(I, Π) to denote the 3D representation inferred via this pipeline given a set of input images I and associated viewpoints Π.Here, we introduce the complete framework of SparseAGS (see Fig. 2 for an overview) that: a) leverages off-the-shelf pose estimation methods and b) incorporates our multi-view reconstruction approach MV-DG (Sec. 3.3) to jointly infer accurate 3D and camera viewpoints. A key challenge we seek to overcome is that the estimated camera viewpoints may have significant errors and that naively using all images to infer 3D can result in suboptimal estimates.Pose Refinement via Gradient Descent. During 3D reconstruction via MV-DreamGaussian, we backpropagate gradients from the photometric loss (Eq. 1) back to update camera poses (implementing custom CUDA kernels to enable this gradient computation). This process allows the camera poses to become more precisely aligned as 3D reconstruction progresses. We denote by Π ′ = GD(I, θ, Π) the resulting camera viewpoints from this optimization given the set of input images I, and 3D reconstruction θ and initial poses Π.With this pose-and-3D co-optimization, we can instantiate a version of our analysis-by-generativesynthesis framework by iteratively refining poses and reconstructing 3D given initial pose estimates Π 0 from an off-the-shelf system:EQUATIONFor clarity, we present separate formulas for the reconstructed 3D θ k and the updated poses Π k , though they are part of the same optimization process. Notably, in each iteration, we initialize the camera poses using the output from the previous iteration (Π k-1 ), while the 3D representation (θ k ) is reset and reconstructed from scratch.Dealing with Outliers. Although the above iterative optimization framework can allow us to infer consistent poses and 3D reconstructions, it is susceptible to local optima and not robust to large errors in initial pose estimates. To overcome this, we additionally detect "outliers", i.e., images with possibly large pose errors that degrade the quality of 3D reconstruction. We then modify our approach to leverage only the estimated inliers for 3D reconstruction while also separately performing a discrete search to update the outlier viewpoints.Iterative Outlier Identification. Our key insight is that an "outlier" image not only exhibits high reprojection error, making it difficult to reconstruct on its own, but also that including it as a training image for 3D reconstruction degrades the overall quality, thus leading to poorer reconstruction even from other views! We operationalize this insight by classifying an image as an outlier if removing it from training significantly improves performance on other images. More formally, let I -i denote the set of images after removing the i th one and let E(θ, I, Π) denote the average reprojection error of a 3D representation θ over images I with (predicted) poses Π. We consider an image i as an outlier if E(MV-DG(I, Π),EQUATIONi.e., adding the image to training set significantly increases the error for other views. For efficiency, instead of considering all images as outlier candidates, we iterate over images in decreasing order of reprojection error. Given this procedure to detect outliers, at each iteration (except k = 1) we modify the above framework to first filter out the outliers found in previous iterations (along with the new "outlier candidate" that gives the largest reprojection error at the last iteration):EQUATIONand only use the estimated inliers for optimizing 3D:θ k = MV-DG(I inlier k-1 , Π inlier k-1). This filter-andreconstruct loop stops when either the selected outlier candidate is determined to be an inlier (i.e., the condition 8 is not satisfied) or the number of remaining inliers falls below a threshold (e.g., 4).Correcting Outlier Poses. While identifying the outliers allows us to prevent them from influencing the 3D inference, the finally recovered model may not capture the details from all images. We thus also attempt to "correct" the pose estimates for the outliers via a discrete search (followed by continuous optimization). Using the currently estimated 3D (reconstructed from only the inliers), for each image in the outlier set, we re-estimate its camera pose via render-and-compare. We first densely sample pose candidates on a sphere and render images from the current 3D. We rank the pose candidates by measuring both pixel-space error (i.e., MSE) and perception error (i.e., LPIPS [49] ). The pose candidate with the highest cumulative rank is selected as the optimal solution. Once all identified outliers are corrected, another reconstruction is performed to form a consistent 3D representation with all images using the updated poses.Our overall framework is very efficient (largely due to an efficient implementation of the reconstruction step), typically taking 5-10 minutes given 8 input images, with increased inference time depending on the number of estimated outliers. We include a brief analysis of the inference time of our system in Sec. B. [47] and RelPose++ [18] ), the refined poses from SPARF [38] are compared with the output of SparseAGS. The estimated cameras are aligned with ground truth (in black) with an optimal similarity transform. More results are available in Fig. 8 . [38] . We use three pose estimation baselines (RelPose++ [18] , Ray Diff. [47] , DUSt3R [41] ) and measure rotation accuracy at two thresholds (5 and 15 degrees) and camera center accuracy at a threshold of 0.1 (of the scene scale). Eight images are used.Rot.@5°↑ Rot.@15°↑ CC@0. Datasets. We primarily evaluate our method on a real-world multi-view object-centric dataset NAVI [9] . This dataset includes high-quality foreground masks, precise camera poses, and 3D meshes. For each of the 35 objects in NAVI, we randomly select 5 multi-view sequences for pose estimation and reconstruction. Additionally, we assess our method on synthetic datasets, including GSO [7] , ABO [4] , and OmniObject3D [44] . Results for the synthetic datasets are provided in Sec. E of the appendix.Baselines. To evaluate camera pose accuracy, we select three sparse-view pose estimation baseline methods: RelPose++ [18] , Ray Diffusion [47] , and DUSt3R [41] . The first two are trained exclusively on CO3D [24] , while DUSt3R is trained on a mixture of eight datasets, representing different levels of precision in initial camera poses. Our method initializes and improves the pose estimates from these baselines, and we also compare with SPARF [38] , a sparse-view pose-NeRF co-optimization method.For evaluation of novel view synthesis, we mainly compare our method with unposed sparse-view reconstruction approaches, LEAP [12] and UpFusion [13] (we include comparison with SPARF in Sec. D). We conduct experiments with varying numbers of input images (N = 6, 8, 10, 16).Metrics. For pose accuracy, we follow prior works [18, 47] and report the following metrics: (1) Rotation accuracy: we compare pairwise relative rotation between the predicted cameras and ground truth. We report the proportion of samples with errors less than a specified threshold, such as 5 and 15 degrees. (2) Camera center accuracy: we align the predictions and ground truth using an optimal similarity transform and report the proportion of camera centers within 10% of the scene Figure 4 : Qualitative Comparison with LEAP [12] on Novel View Synthesis. We use two pose estimation baselines (Ray Diffusion [47] and DUSt3R [41] ). SparseAGS better preserves details from the input images and shows enhanced performance with more accurate initial camera poses. More results are available in Fig. 9 of the appendix.scale relative to the ground truth. We evaluate our 3D representation via novel-view synthesis and report PSNR and LPIPS [49] for the rendered views. In our ablation study, we also assess the 3D geometry using the F1 score, comparing our recovered geometry against the ground truth 3D meshes. Rot.@5°Rot.@15°CC@0.1 Rot.@5°Rot.@15°CC@0.1 Rot.@5°Rot.@15°CC@0.  
evaluation: 
 Camera Pose Accuracy. We compare SparseAGS with SPARF [38] on pose accuracy given eight input images quantitatively in Tab. 1 (numbers are in percentage) and qualitatively in Fig. 3 . We find that SparseAGS consistently yields larger improvements than SPARF, which sometimes even leads to degraded accuracy (marked by red numbers). We attribute this to the unreliable correspondences extracted by SPARF (we include an example in Fig. 7 ), as the input images in NAVI may exhibit more significant viewpoint changes compared to scene-level datasets, e.g., DTU [10] where SPARF Table 3 : Quantitative Comparison of 3D Reconstruction on NAVI [9] . We compare our method with two unposed approaches: LEAP [12] and UpFusion [13] , using varying numbers of input images (N). We adopt two pose initializations (Ray Diff. [47] , DUSt3R [41] ) reporting PSNR and LPIPS. We vary the number of input images (N = 6, 10, 16) and report camera pose accuracy in Tab. 2 (we only test RelPose++ [18] with six images as inference with more than eight images is not supported). SparseAGS consistently enhances baseline performance for both rotation and camera center accuracy, with particularly significant gains for stricter metrics (e.g., Rot.@5°). Moreover, the improvements tend to further increase with the number of input images. These results demonstrate that our method is robust to varying levels of initial camera poses and generalizes well across different input numbers. Figure 5 : Qualitative Comparison with Up-Fusion [13] on Novel View Synthesis. We use two pose estimation baselines (Ray Diffusion [47] and DUSt3R [41] ) as in Fig. 4 . Note that the left eye and symbol 2 ⃝ of the Chicken Racer is missing in UpFusion's output, probably because of the "first-image bias", while SparseAGS preserves these details.3D Reconstruction. In Table 3 , we compare SparseAGS with two unposed approaches, LEAP [12] and UpFusion [13] , reporting metrics for 3D reconstruction (novel view synthesis). Our method consistently outperforms both baselines across different numbers of input images and with two pose estimation initializations. While SparseAGS shows continuous improvements with an increasing number of input images, the performance of LEAP and UpFusion nearly saturates in terms of both PSNR and LPIPS. We hypothesize that unposed methods struggle to utilize additional input images beyond their training capacity without further training adjustments (LEAP is trained using five views, while UpFusion is trained with a maximum of six images). In contrast, our method is flexible w.r.t. the number of input images, eliminating the need for re-training. A qualitative comparison with LEAP and UpFusion is presented in Fig. 4 and Fig. 5 , respectively. The results show that SparseAGS better preserves the details in input images by explicitly modeling cameras and produces higher-quality novel view synthesis with more precise initial camera poses. Table 4 : Ablation Study. Using initial poses from Ray Diffusion [47] for eight input images, we ablate the effect of each proposed component of our approach.Rot.@5°↑ Rot.@15°↑ CC@0.1↑ PSNR↑ LPIPS↓ F1@0.01↑ We ablate the effectiveness of each component in our approach (Tab. 4) using initial pose estimates from Ray Diffusion [47] with eight input images. In addition to camera pose metrics and novel-view (Appropriate) Generative Priors Improve Analysis by Synthsis. We find that adding generative priors (Eq. 2) to naive pose-3D co-optimization (Eq. 1) improves both pose accuracy and 3D reconstruction quality (comparison between (1) and (3) shows consistent improvements in all metrics). However, vanilla Zero-1-to-3 [20] is not suitable for providing such priors in real-world scenarios, as we observed a drop in camera rotation and center accuracy (compare (1) with (2) in Rot.@15°and CC@0.1). This is because 3-DoF camera parameterization cannot well represent the camera poses in the wild. Although the numerical improvements may appear marginal (e.g., in PSNR), Fig. 6 presents a qualitative comparison of 3D reconstruction with and without our 6-DoF novel-view generative priors. Supervision on novel views via SDS helps form a consistent 3D representation.Outlier Removal and Correction. The presence of outlier initial cameras introduces significant challenges to pose-3D co-optimization. Our iterative outlier removal and correction pipeline effectively addresses this issue. For instance, comparing (3) with (4) shows a substantial improvement: Rot.@5°i ncreased from 34.6% to 60.3% (25.7% absolute improvement), PSNR improved from 13.44 to 15.30, and F1@0.01 increased from 57.2 to 68.2 (11.0 point absolute improvement). These results confirm the effectiveness of our approach.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We include our experiment setup in Sec. 4.1.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 In this work, we presented SparseAGS, a framework for joint pose estimation and 3D reconstructioncombining off-the-shelf pose estimation methods with a novel-view synthesis generative prior for robust inference in real-world sparse-view captures.Limitations. While our experiments demonstrated clear improvements over initializations and stronger performance compared to prior 3D reconstruction methods, there are several challenges that remain. First, our approach does rely on some reasonable off-the-shelf pose estimates and cannot succeed if a large fraction of the predictions have a large error. Secondly, SparseAGS (similar to existing baselines) does not deal with truncation or occlusion and cannot be directly applied to scenarios with close-up images of parts of objects or cluttered scenes with one object occluding the other. Finally, we focused here on an object-centric setting, and it would be interesting to extend our approach to broader settings, e.g., deploying our framework in conjunction with methods that learn novel-view generative priors for scenes. 70 sequences (2 sequences per object) in these two experiments. Notably, for a direct comparison on NVS, we removed backgrounds from the input images, whereas no masking was applied in Tab. 1. This may slightly disadvantage SPARF, as backgrounds provide additional cues for pose registration and correspondences.The results indicate that SparseAGS outperforms SPARF in pose accuracy and novel view quality as well. In fact, we found that SPARF can often make the poses worse compared to the (relatively accurate) DUSt3R initialization (measured via Improvement Rate that indicates the percentage of sequences with reduced pose error). This is likely because the correspondences leveraged by SPARF in its optimization are not robust and are susceptible to false matches -see Fig. 7 for an example.As an attempt to compare novel view quality despite the difference in pose accuracy, we report *PSNR and *LPIPS, which are measured only on the sequences where SPARF improves pose accuracy and find that even in these, our approach outperforms it. We also observed that while SPARF works well on novel views close to the input, floaters constantly appear with significant viewpoint changes. In contrast, our generative prior leads to a more consistent 3D representation. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We, unfortunately, are not able to clear data and codes before the camera-ready deadline. However, we will release our code as well as the data used soon. Guidelines:• The answer NA means that paper does not include experiments requiring code. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: Although our method leverages diffusion priors, we do not generate images directly. Instead, we reconstruct the 3D images from user-specified images. We will require the users to strictly follow the guidelines when our models are publicly available. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Table 6 : Evaluation of Rotation Accuracy on Three Synthetic Datasets (GSO [7] , ABO [4] and OmniObject3D [44] ). We test our method on ID-Pose [3] with eight images as input. We measure rotation accuracy at two thresholds (15 and 30 degrees).Rot.@15°Rot.@30°Rot.@15°Rot.@30°Rot.@15°Rot.@30°I D-Pose Though our main focus is real-world data, we apply our approach to three synthetic datasets (GSO, ABO, OmniObject3D) for a complete evaluation of our approach. We use ID-Pose [3] as a baseline, which inverses the novel-view-synthesis ability of Zero-1-to-3 [20] for pose estimation and adopts a 3-DoF camera parameterization. Here, we also use the vanilla Zero-1-to-3 for multi-view SDS loss to accommodate this camera definition and for fair comparison. We report the results on camera rotation accuracy in Tab. 6. Across these datasets, our approach consistently improves performance on two metrics, even though ID-Pose uses the same backbone model (vanilla Zero-1-to-3) as we do. These results demonstrate that our approach more effectively leverages the generative priors from Zero-1-to-3, achieving better pose accuracy. Plus, we also show that our method is applicable to synthetic data, showing strong generalization abilities across different datasets.See Fig. 8 .See Fig. 9 .SparseAGS (w/ 3D) SPARF Figure 8 : More Qualitative Comparison on Camera Pose Accuracy. Given initial camera poses from off-the-shelf methods, the refined poses from SPARF [38] are compared with the output of SparseAGS. We align the estimated cameras with ground truth (in black) with an optimal similarity transform. Figure 9 : More Qualitative Comparison with LEAP [12] on Novel View Synthesis. We use two pose estimation baselines (Ray Diffusion [47] and DUSt3R [41] ). SparseAGS better preserves details in the input images and shows enhanced performance with more accurate initial camera poses.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 


Decision: The paper introduces a novel optimization-based method for sparse-view 3D reconstruction from unposed images. The method uses an off-the-shelf pose estimator for pose initialization, then optimizes both the pose and 3D reconstruction using rendering loss and generative priors. The method demonstrates satisfactory results on the evaluation data, and the ablation study highlights the effectiveness of each proposed technique. The initial ratings were three borderline accepts and one reject. The key concerns included a lack of novelty, unfair comparisons in terms of 3D reconstruction, increased inference time, and missing related work. In the rebuttal, additional 3D reconstruction comparisons were provided, and discussions on novelty, new related work, and inference time were included. The final ratings were four weak accepts. Given the consensus, the AC recommends acceptance.

Reviews: review: paper_summary: paper_summary: This paper presents a method named MV-DreamGaussian for tackling the problem of 3D reconstruction from sparse multi-view inputs. In particular, the paper extends the DreamGaussian work to use multi-view images as the inputs and proposes a scheme to optimize the inaccurate camera poses of the multi-view images.
strength_weakness: strength_weakness: - This paper presents very limited novelty in the reconstruction part with a trivial extension to DreamGaussian to use multi-view images, which is already implemented in a public repository [stable-dreamfusion](https://github.com/ashawkey/stable-dreamfusion).
- The major weakness of the paper is the lack of fair comparisons in terms of the 3D reconstruction. The authors only compared with LEAP for the 3D reconstruction. However, LEAP is a work that **does not require any pose inputs**, whereas the proposed work needs relatively good pose initialization (e.g., Dust3r) and conduct refinement on it. In addition, the underlying 3D representation is different, too: LEAP uses NeRF while the proposed work uses 3D Gaussian. I'm confused as to why the authors did not compare with SPARF for the reconstruction quality too since SPARF shares the same input setup as the proposed work. Besides, the very recent work DMV3D would also be a good method to compare with.
questions: questions: - I'm quite curious what the reconstruction quality the method can achieve without 3D generative prior but with the proposed refinement. Namely the combination of (1) and (4) in Table 4.
- How are the poses used for generative prior sampled in addition to the input views?
- How are the thresholds for pose outlier removal tuned?
limitations: limitations: The authors have discussed the limitations of the paper and I generally agree with them.
paper_summary: paper_summary: This paper proposes a method for the joint reconstruction of camera poses and 3D objects given sparse input views. The core idea is to use a pose-conditioned diffusion model (Zero-123) as a prior, impose the SDS loss, and jointly optimize the poses and objects, similar to the approach in ID-pose. To improve the robustness and quality of the optimization, the authors made several modifications: (1) Using a 6 DoF pose-conditioned diffusion model instead of a 3 DoF model. (2) Adding strategies for outlier detection and correction. (Although somewhat empirical, it proves effective.)

This approach requires initial camera poses (from methods such as RelPose++, RayDiffusion, etc.) and is not capable of reconstructing poses from scratch (e.g., purely random camera poses). Experimental results demonstrate that, compared to SPARF and ID-pose, the proposed method achieves better pose estimation quality. Additionally, it provides better object reconstruction in terms of novel view synthesis quality compared to LEAP.
strength_weakness: strength_weakness: (1) This optimization-based method requires more time compared to a feed-forward model, taking about 5-10 minutes. Additionally, the writing discussing this aspect is somewhat unclear: the paper states, “with increased inference time depending on the number of outliers.” Could this statement be more specific? How much does the time increase with the number of outliers? The correction of outliers may be time-consuming as it requires dense searches of initial camera poses.

(2) (Minor) The method focuses only on object-level reconstruction, which makes the scope seem narrow.

(3) The authors do not sufficiently discuss experiments in a more “standard” sparse-view setting, such as using 3 or 4 views. The reported experiments use at least 6 views, which is not a particularly small number.
questions: questions: (1) A related work is lacking in discussion:  Sun, Yujing, et al. "Extreme Two-View Geometry From Object Poses with Diffusion Models." arXiv preprint arXiv:2402.02800 (2024).

(2) Is the testing data included in the training set for fine-tuning the 6-DoF diffusion model?
limitations: limitations: As discussed in the weakness.
paper_summary: paper_summary: This paper proposes a framework for joint 3D reconstruction and pose refinement. Specifically, given estimated camera poses from off-the-shelf models, the proposed method first leverages diffusion priors and rendering loss for 3D reconstruction. The 3D reconstruction is further used to refine the current pose parameters. The 3D reconstruction and pose refinement are conducted in an alternative way. An outlier identification and correction strategy is also introduced to make full use of the given image while mitigating the adverse effect of noisy camera estimations at the same time. Experimental comparison with several pose estimation baselines shows that the proposed method can refine inaccurate pose estimation effectively.
strength_weakness: strength_weakness: 1. The proposed method is compared with SPARF only in the setting of using pose from different pose estimation baselines. However, it would be more convincing to also present the results using the same setting of SPARF, which adds noise into the GT camera pose. This will be a direct comparison with SPARF’s original results reported in their paper.
2. The proposed method is compared with LEAP for 3D reconstruction results. However, the comparison is a bit unfair since LEAP does not require any initial camera poses. 
3. The description of how to effectively detect the outliers (line 212 - line 214) is not very clear. Similarly, the procedure of how to correct the outlier poses (line 223 - line 225) is not very clear either. How the MSE and LPIPS are computed and compared since there is no correspondence?
questions: questions: 1. The proposed method is evaluated on the NAVI dataset. It seems that the dataset is quite simple as shown in Fig. 3 and Fig. 4. The reviewer is wondering about the performance of the proposed method on more complex scenes?
2. The reviewer is wondering about the separate ablation results on the outlier removal and correction.
limitations: limitations: Limitations are addressed in the paper.
paper_summary: paper_summary: The paper introduces a novel optimization-based method for sparse-view 3D reconstruction from unposed images. The method uses off-the-shelf pose estimator to get pose initialization, then it uses rendering loss and generative priors to optimize the pose and 3D reconstruction. In detail, the generative priors involve a multi-view SDS loss on generated novel views using Zero123. The method demonstrates satisfying results on the evaluation data, and the ablation study shows the effectiveness of each proposed technique.
strength_weakness: strength_weakness: - Missing baseline. For the reconstruction methods, the only baseline is LEAP, which is a feedforward method. In contrast, the proposed method is an optimization-based method, which introduces pose-processing to estimated poses. I would suggest adding baseline of SPARF [1] and using the same pose initialization. Moreover, why not comparing with UpFusion?
- Unknown inference speed. Will the joint optimization of pose and shape be slow? Could you provide a analysis of inference time?
- Related work. One related work is iFusion [2], which uses generative priors for pose estimation and is very relevant to the philosophy of the proposed method. Another related work is FORGE [3], which introduces pose optimization for sparse view reconstruction. Moreover, the authors should discuss the prior sparse-view reconstruction from unposed images works with more details, the authors should provide more comparison and contrast with prior work. The current discussion is too short (Line 90-92).
- Ablation study. The ablation study is performed with the Ray Diffusion pose initialization. How will it look like using Dust3r initialization? This is important as the ablation should be performed with the best base model.


[1] Truong, Prune, et al. "Sparf: Neural radiance fields from sparse and noisy poses." CVPR 2023.
[2] Wu, Chin-Hsuan et al. “iFusion: Inverting Diffusion for Pose-Free Reconstruction from Sparse Views.” ArXiv 2023.
[3] Jiang, Hanwen et al. “Few-View Object Reconstruction with Unknown Categories and Camera Poses.” 3DV 2024.
questions: questions: - The introduction spends a lot space discussing the chicken-and-egg problem of pose estimation and reconstruction. However, I don't think it is quite related to the technical part, as the proposed method still need pose initialization using off-the-shelf methods. The method doesn't provide a novel perspective regarding how to solve the chicken-and-egg problem, and using pose initialization is quite common in prior works, e.g., SPARF, FORGE, and FvOR or even traditional SfM methods. Why the authors want to emphasize this?
- Is it possible to evaluate the outlier removal method? For example, the authors can evaluate the correlation between the removal and the pose error. If the proposed method works well, they should have strong correlations. Moreover, it will be good to provide any statistics on the outlier removal method, e.g., how many images are removed in average.
limitations: limitations: Please see weaknesses and questions.

Comments: comment: Dear Reviewer,

Thanks for the feedback. We believe there are two schools of thought for developing approaches for sparse-view-3D in-the-wild: a) methods that side-step explicit pose estimation (e.g. LEAP, UpFusion), and b) methods that rely on and improve initial pose estimation (e.g. SPARF and ours). We do already include a comparison to SPARF as a baseline that also uses the same initial poses to show that for methods following ideology (b), our work improves over the current SOTA.


Regarding the comparison to LEAP, we certainly agree that LEAP not using poses is perhaps a reason why it is not competitive -- but this is exactly the point we wish to make! In particular, we hope that one key take-away for a reader is that methods following ideology (a) are limited in their performance. Based on their comments, this maybe something the reviewer feels is obvious, but this is not universally agreed! For example, LEAP positions itself as ‘liberating’ sparse-view 3D reconstruction from a dependence on pose estimation and even begins the abstract with a question that “Are camera poses necessary for multi-view 3D modeling?” and argues that they are not needed (UpFusion also follows a similar philosophy). Our comparison to LEAP (and the added comparison to UpFusion in response to Reviewer XaBT) seeks to make the counterpoint that “Actually, such poses are indeed helpful!”. 

We would be happy to present the results with this context more clearly outlined in the text if the reviewer feels that maybe helpful.
Thanks for the authors' detailed responses. Most of my concerns are resolved in the rebuttal. For the comparison on 3D reconstruction, it would be more convincing to compare under the same condition, namely based on an initial pose. However, LEAP is not using pose information, hence putting LEAF at disadvantage for the comparison.
We are glad to see that the reviewer's concerns were resolved, and we will update our paper based on our discussion with the reviewer.
Thanks for the reply. The response makes sense to me. I will raise my score.
We greatly appreciate the reviewer for raising the score! We will carefully revise our paper based on the reviewer’s valuable feedback. Thank you again for providing such insightful comments!
We would like to thank the reviewer for the prompt response and the willingness to raise our score!

Yes, we indeed found that correspondence is crucial for SPARF's pose optimization, and we observed that SPARF often fails to obtain reliable correspondence in our experiments. In **Fig. 3** of our rebuttal PDF file, we included an example where false estimated correspondence makes SPARF's pose estimation fail, and therefore, it produced degraded novel views. 

To demonstrate the influence of foreground masks on correspondence (which is reflected by pose accuracy), we present a comparison of SPARF with and without using masks below.

| N=8              | Avg Rot Error | Rot@5° | Rot@15° | CC@10% | Improvement Rate |
| :--------------- | :------------ | :----- | :------ | :----- | :--------------- |
| DUSt3R           | 8.71          | 51.6   | 92.9    | 79.5   | /                |
| SPARF (w/ mask)  | 18.17         | 57.3   | 75.7    | 68.6   | 0.54             |
| SPARF (w/o mask) | 10.10         | 58.1   | 85.9    | 81.2   | 0.59             |
| Ours             | 5.99          | 81.7   | 95.3    | 92.5   | 0.93             |

While including background did improve SPARF's pose accuracy, it failed to improve the average rotation error and Rot@15° over the baseline as the w/ mask setting did. Even in this setup, we still observed false estimated correspondence used by SPARF, which may cause large pose errors. Given these results, we may attribute the SPARF's counter-intuitive performance more to the distribution of NAVI data instead of the use of masks. In NAVI, images have limited overlap, making it inherently hard to find reliable correspondence. This differs significantly from SPARF's testing data in the original paper, which may show notable overlaps and a strong forward-facing feature (e.g., the DTU dataset).

In our General Response, we also tried to compare novel view synthesis with SPARF despite differences in pose accuracy by introducing `*PSNR` and `*LPIPS`.  We used these metrics to factor out the negative influence of failure pose optimization on SPARF due to unavailable/false correspondence. We refer the reviewer to our General Response for more details! We hope this could address the reviewer's concern regarding the influence of masks (and thus the influence of pose accuracy) on NVS.

Finally, we would like to clarify an implementation detail relevant to the reviewer’s question: in the original submission, when reporting pose error, we did give SPARF access to the background as it gave stronger results. Unfortunately, for NVS experiments, this was not trivial as it would require post-processing to remove background from the renderings before comparing NVS metrics, so we gave it images w/o background. Although this may be slightly suboptimal, we hope that our `*PSNR` and `*LPIPS` (which explicitly focus on sequences where SPARF improved pose, thus minimizing the effect of sequences where SPARF was not robust) make it clear that our system does outperform SPARF.
Thanks for the detailed reply. The authors' response addressed my major concerns about the evaluation and I decided to raise my rating to acceptance.
Thanks for the detailed reply! The rebuttal resolves most of my concerns. I would like to raise the score of this paper. 

Before that, I have a last question. I noticed that in the reply to Reviewer mujy, with using SPARF, the pose error becomes larger, which is counter-intuitive. Could the authors provide more insight into this? Is there any problem with the estimated correspondence, e.g. does the object-centric image makes the correspondence fail? If so, is it reasonable to use images with original background as the input of SPARF, and after rendering the novel views, we use foreground segmentation to evaluate the NVS metrics?