Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis
Authors: Qitao Zhao
Shubham Tulsiani
Venue: NeurIPS 2024 poster (2024)
Keywords: computer vision
3D vision
3D reconstruction
camera pose estimation
analysis-by-synthesis
generative model
Primary area: machine_vision

Abstract: Inferring the 3D structure underlying a set of multi-view images typically requires solving two co-dependent tasks -- accurate 3D reconstruction requires precise camera poses, and predicting camera poses relies on (implicitly or explicitly) modeling the underlying 3D. The classical framework of analysis by synthesis casts this inference as a joint optimization seeking to explain the observed pixels, and recent instantiations learn expressive 3D representations (e.g., Neural Fields) with gradient-descent-based pose refinement of initial pose estimates. However, given a sparse set of observed views, the observations may not provide sufficient direct evidence to obtain complete and accurate 3D. Moreover, large errors in pose estimation may not be easily corrected and can further degrade the inferred 3D. To allow robust 3D reconstruction and pose estimation in this challenging setup, we propose SparseAGS, a method that adapts this analysis-by-synthesis approach by: a) including novel-view-synthesis-based generative priors in conjunction with photometric objectives to improve the quality of the inferred 3D, and b) explicitly reasoning about outliers and using a discrete search with a continuous optimization-based strategy to correct them. We validate our framework across real-world and synthetic datasets in combination with several off-the-shelf pose estimation systems as initialization. We find that it significantly improves the base systems' pose accuracy while yielding high-quality 3D reconstructions that outperform the results from current multi-view reconstruction baselines.

Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis
Qitao Zhao
Shubham Tulsiani
Carnegie Mellon University
Project page:
Input
Camera Poses w/
Novel Views
Figure 1: Given set of unposed input images, SparseAGS jointly infers the corresponding camera poses and underlying 3D, allowing high-fidelity 3D inference in the wild.
Abstract
Inferring the 3D structure underlying of multi-view images typically requires solving two co-dependent tasks accurate 3D reconstruction requires precise camera poses, and predicting camera poses relies (implicitly explicitly) modeling the underlying 3D. The classical framework of analysis by synthesis casts this inference as a joint optimization seeking to explain the observed pixels, and recent instantiations learn expressive 3D representations (e.g., Neural Fields) with gradient-descen based pose refinement of initial pose estimates. However, given sparse set of observed views, the observations may not provide sufficient direct evidence obtain complete and accurate 3D. Moreover, large errors in pose estimation may not be easily corrected and can further degrade the inferred 3D. allow robust 3D reconstruction and pose estimation in this challenging setup, approach
we propose SparseAGS, method that adapts this malysis-by-synthees by: a) including novel-view-sye riew-synthesis- based generative priors in conjunction with photometric objectives to improve the quality of the inferred 3D, and b) explicitly
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

reasoning about outliers and using discrete search with continuous optimization- based strategy correct them. We validate our framework across real-world and synthetic datasets in combination with several off-the-shelf pose estimation systems as initialization. We find that it significantly improves the base systems' pose accuracy while yielding high-quality 3D reconstructions that outperform the results from current multi-view reconstruction baselines.
1 Introduction
Consider the images of the robot shown in Fig. 1. From just these few images, we humans can easily understand the 3D structure of this object it has cylindrical base supporting tall body from which an arm extends to the front. We do this by aggregating the information across images into a consistent 3D mental model, e.g., the "front" view informs of the width of the body and the "side" view(s) about the extended arm. But how do we know which image is to the "front" or to the "side" to begin with? As evidenced in seminal research of mental rotation [30],
understand viewpoints by forming mental 3D models. Thus, to form mental 3D models, we need to understand the (relative) viewpoints across images, but doing in turn requires mental 3D model!
This co-dependency in inferring shape and pose is one that any computational approach aiming to recover 3D from multiple views also needs to deal with. Indeed, classical approaches like Structure- from-Motion (SfM) [29] tackle the two together and infer 3D and camera viewpoints. However, these correspondence- -bsed methods can only infer sparse 3D representations and are not robust given a small set of images with limited overlap. To allow 3D inference in such sparse-view settings, recent learning- basedd approaches have pursued sparse- view reconstruction approaches [52, 45], but assuming known precise camera poses. Separately, there have been several methods [18, 47, 48, 31] which predict camera viewpoints given a set of images. While these methods have led to impressive results for both 3D reconstruction and pose inference, their singular focus on only one task without tackling the other limits their utility- the 3D reconstruction methods requiring precise cameras cannot be easily used in real-world applications, and pose estimation methods that do not model 3D are typically limited in their accuracy.
We present ApS, a framework that unifies the advances learning- pose estimation and 3D reconstruction using the former an initialization and building the latter for obtaining accu- rate 3D reconstruction. Specifically, we adopt an "analysis by synthesis" approach where we jointly optimize pose and 3D explain the observed pixels. However, unlike prior methods [19, 42] which simply leverage
gradient-basee updates for pose and 3D, we additionally leverage generative priors [8, 33] for more complete (and accurate) 3D despite input images that may only partially capture the object. However, current off-the-shelf novel-view generative models [20] only allow 3-DoF camera parameterization which is insufficient beyond synthetic settings, finetune a SoTA model to allow 6-DoF camera variation when querying novel views. We find that such generative priors not only contribute the 3D reconstruction quality but also result in more accurate camera poses. Moreover, we also explicitly account for large possible errors in initial camera estimation and prevent these from degrading 3D reconstruction via identifying outliers, and also improve poses via a combination of a discrete search and continuous optimization.
Compared to prior joint reconstruction and pose estimation methods that are designed to improve near-perfect initial cameras [19, 38), SparseAGS can leverage off-the-shelf pose estimates, thereby allowing robust inference in real-world scenarios. We demonstrate its efficacy using both, real- world and synthetic datasets in conjunction with several satte-of-the-ate pose estimation methods as initialization. We find that our approach significantly improves the initial camera estimates and yields high-fidelity 3D reconstructions (and novel view renderings). In summary, our contributions are:
We introduce analysis-by-generees gewerative-synneeneee viewpoints given a sparse set of input images, by integrating a 6-DoF novel-view generative prior in an seisyysi----yyttheee approach
framework that jointly estimates 3D and camera
Our approach allows leveraging any off-the-shelf pose estimation system and can robustly estimate 3D and viewpoints despite large errors in the initial estimates.
We present results across datasets and initializations and show clear improvements over the initializations as well outperform prior sparse-view 3D reconstruction baselines.

---

2 Related Work
Atructure-from-Mottie [32, 29] methods often fail to estimate camera poses in sparse-view settings. Several approaches instead seek to leverage data-driven priors, for example learning energy- baay- [48, 18] or denoising diffusion [39] models to predict cameras. While these approaches predict global camera representations, some works have demonstrated the benefit of denser camera parametrizations by predicting raymaps [47] or pointmaps [41, 17]. As an alternative paradigm to direct camera prediction, some recent methods [3, 43, 34] instead estimate relative poses by inverting the view-conditioned synthesis capabilities of diffusion models [20]. While these methods have led to remarkable improvements in camera estimation, these are still susceptible to some imprecision and occasional outliers which our 3D- reasoning-based approach can correct.
Sparse-view Pose Estimation. Traditional correspondence-basee
Sparse-view 3D Reconstruction. This line of work aims to recover 3D from sparsely sampled views, aiming to infer complete 3D representations that faithfully reflect the content captured by the input images while making reasonable guesses for invisible areas. The progress of diffusion models [8, 33] has greatly advanced this direction, as they are capable of learning strong natural image priors from data. Inspired by DreamFusion [23], which generates 3D scenes given textual descriptions leveraging text-to-image diffusion model [26], SparseFusion [52] learns view- conditioned diffusion model on multi-view image collections for novel view synthesis and then distills the learned novel-view distributions into a single consistent 3D representation. DreamSparse [45] further improves the performance by utilizing internet-scale natural image priors learned by Stable Diffusion [25]. Although these methods present impressive results, they assume precise camera poses are available, which limits their applications. FORGE [1i] addresses this by jointly inferring both camera poses and 3D structure in single forward pass, though the quality of its reconstructions remains constrained by pose estimation accuracy without further refinement or correction.
Pose-free Sparse-view 3D Reconstruction. Some recent works [27, 40, 12, 13] have attempted to bypass the reasoning about camera poses and directly infer novel views or 3D representations from unposed images. An unposed variant of the Scene Representation Transformer [27] encodes a set of input images into latent features and synthesizes novel views given the corresponding query rays (w.r.t the viewpoint of the first image) using a transformer encoder and decoder. UpFusion [13] improved upon this by learning a diffusion model and distilling a consistent 3D representation via Score Distillation Sampling [35]. whereas LEAP [12] and PF-LRM [40] can directly predict (volumetric triplane) 3D representations in a feedforward manner. While these methods demonstrate promising results, their geometry free approach cannot easily capture the specific details across input images and they struggle to improve the 3D estimation with additional input images.
Analysis-yy-yynheeis Approaches. Approaching visual perception as an inverse graphics task is classical idea in computer vision [15, 46), and has been leveraged for inferring scene properties (e.g., object pose) by synthesizing visual content as close to observations possible [2, 16, 51, 50]. Closer to our setup, prior approaches jointly optimize camera pose and 3D representation (e.g., NeRF [22]) to explain the observed images [19, 42] but these methods are designed for dense observations and only handle small pose errors. Closer to our work, SPARF [38] focuses on the sparse-vier setup, leveraging estimated pixel correspondence [37] prior knowledge in addition to the standard photometric loss. However, reliably extracting such correspondences can be challenging, and false match estimates may even confuse pose refinement, leading to degraded 3D reconstruction.
3 Method
3.1
Overview
Analysis by Synthesis. Given a of sparse-view images, denoted Il = ( our goal is to reconstruct the underlying 3D structure 0 and infer the camera poses corresponding to the input images II { This can be done by solving an mualysis-by-syntheet problem
min 0.11
I1 fo(
where So is a rendering function. Eq. demonstrates that we want to find scene description consisting of 3D representation 0 and camera configurations II that well explain the observed input
3

---

Input Images (4p
Camera Poses [mT]
Representation (0)
Rerovaa
K
OFE-tho--ahllf
ADT
Render ing
Generatiove Pripew
LgG
Zg
ビ- L-(O.T)
for L....N
Average Prediction
112
Eg (c.1,,
(e)
Novel View fo(
L Multi-view SDS
(T()))
(b)
RGr dm;
E( ((NV-D((=",n-)).--
lr fo(me)
Does using I.- for inferring 3D incur larger errors other views?
&( ((XVDGG1.n)) M).1".
Figure 2: (a) Overview of SpSrseAGS: Given estimated camera poses from off-the-shelf models, method iteratively reconstructs 3D and optimizes poses leveraging diffusion priors. (b) Detailed View of Each Component: We use rendering loss and multi-view SDS loss for 3D reconstruction while the rendering loss propagated back refine camera poses. At the end of each reconstruction iteration, we identify outliers by checking if their involvement in 3D inference yields larger errors in other views, implying the inconsistency of their poses with others.
images. If fo is differentiable, we can jointly optimize the 3D representation and camera poses via gradient descent [19, 42]. However, this approach may work well in the sparse-view setting [38] (i.e,, N small) as the 3D representation can overfit to the input images without forming plausible structure, degrading both, pose estimation and 3D reconstruction.
Analysis by Generative Synthesis. To address this issue, we propose to introduce generative priors into analysis by synthesis, so term method analysis by generative synthesis. In addition to the known-view objective (Eq. 1), we leverage diffusion priors [8, 33] to optimize renderings from randomly sampled novel views (π) well
min Ex log P IT, II,1
(2)
where Pe is the likelihood of the novel view rendering conditioned the viewpoint and inputs (I, II), modeled by the diffusion model ø. The gradients for this objective can be obtained via Score Distillation Sampling (SDS) [23], and intuitively, they encourage the renderings of the 3D representation to be plausible based on image distributions learned by the diffusion model.
In the following, we first introduce few preliminaries about efficient single-view-to-3D approach (Sec. 3.2), on which we build our multi-view reconstruction method, VV-DreamGaussian, enabling analysis by generative synthesis in the wild (Sec. 3.3). Then, we present complete framework that involves dealing with imprecise cameras (Sec. 3.4). An illustration of our approach is in Fig. 2.
Preliminaries: DreamGaussian
DreamGaussian [36] generates 3D from single image with two-stage approach, achieving
satisfactory trade-off between speed and fidelity. The first stage optimizes 3D Gaussians [14] (parameterized by 0) using combination of photometric loss (Eq. 1, except that the camera pose not optimized) and SDS loss (Eq. 3) with a view-conditioned diffusion model, 7Zro--1to-3 [20]. Specifically, for randomly sampled novel view π, scheduled noise at timestep t is added to the latent of its rendering (the noisy latent is denoted by 2,). The training objective minimizes the difference between the predicted noise and the added noise, approximating the negative log-likelihood of the rendered image. The gradient of SDS loss is given by
VoLSDS = ASDS E, w(t)(Es(zzitt, A. 0fe(T)
(3)
00

---

where w(t) is weighting function, Ep(.) is a U-Net trained to predict the added noise given the noisy latent Z,. conditioned on the timestep t. reference image I, and the relative camera pose A This stage efficiently builds the geometry of the object with rough texture, which takes 500 training steps (in about 1 minute). In the second stage, 3D Gaussians are converted to a textured mesh with Marching Cubes [21], and only its texture is optimized. This stage takes another 50 steps and can finish within 30 seconds on a single GPU.
We find DreamGaussian to be a suitable starting point to perform analysis by generative synthesis, but note that has some key limitations: (1) 3-DoF Camera Parameterization. DreamGaussian adopts 3-DoF camera parameterization (i.e., radius, elevation, and azimuth) to accommodate the camera definition Zero- 1-to-3 [20]. While this parameterization is sufficient for synthetic data, it cannot well represent the 6-DoF camera poses of real-world images. (2) Single Input Image. DreamGaussian is designed for the singe- view-to-3 task, In contrast, we aim for the reconstructed 3D to reflect the details captured by multiple input images faithfully. This requires an approach to handling information from multi-view images.
3.3 Leveraging Generative Priors for Sparse-view 3D in the Wild
We adapt DreamGaussian's 's two-stage method and extend it to (1) handle real-world images with 6-DoF camera parameters and (2) utilize sparse-viee view images as input.
Generative Priors in the Wild. Zero-I-to-3 [20] offers desirable generative priors that enable single-view-to-3D generation of DreamGaussian. However, it assumes no in-plane camera rotation and that all possible camera poses are strictly directed toward a common origin. We find these assumptions are over-restrictive for real-world images. Therefore, we propose to replace the 3-DoF camera condition in Zero-I-to-3 with a 6-DoF one, represented as an 18-dimensional vector:
(4) where Tel is the relative extrinsic matrix (4 x 4) between the source view and target view, and fi (f2) is the ratio of the focal length along the x- (y-) axis between them. We include the focal length term account for the object scale change due cropping. This simple camera parameterization effectively represents 6-DoF cameras in the wild. Details regarding finetuning Zero-1-to-3 for 6-DoF camera conditioning are deferred to Sec. C in the appendix. We note that recent work, ZeroNVS [28], also discussed this 3-DoF issue of Zero-I-to-2 and proposed a "6-DoF+1" I" camera parameterization for scene-level novel view synthesis. However, this approach is not directly applicable to our object- centric setting, as it is trained using images with complex backgrounds and leverages depth priors to address scale ambiguity.
Flattenn( Tie)), log()) log(ffi),, log(f%))
Leveraging the Generative Priors from Multiple Views. DreamGaussian only uses the generative priors from single reference image via SDS loss. To make Lsns aware of the visual cues from multiple input images, we modify Eq. 3 as
(
a LoLaiiiivieww SDS Asns Er,,,, w())(Ep -)- {
where
(5)
00
Es E N Σ
1 N Ezizi I,, AT,)
(6)
N is the total number of input views, I, is the 8th input image, and AT; is its relative camera pose w.r.t the sampled novel view π. We average the noise predictions from all input views that share the same timestep L. This method draws inspiration from the implementation of GabbleDreamfusioo [35], but we do not weigh the predicted noises based on the relative closeness of camera poses. The rationale behind this is that the camera poses in our setting are not always reliable, and relying too heavily on "close" views could introduce significant conflicts during the 3D optimization process.
With these modifications, our multi-view reconstruction approach, termed MV-DrramGauussinn is capable of reconstructing 3D from sparse images the wild by leveraging diffusion priors. When describing its use in our overall framework, we use the notation 0 = MV MV- II) to denote the 3D representation inferred via this pipeline given set of input images I and associated viewpoints II.
3.4 3D Reconstruction with Imperfect Cameras
Here, we introduce the complete framework of SparseAGS (see Fig. 2 for an overview) that: a) leverages off-the-shelff pose estimation methods and b) incorporates our multi-view reconstruction
5

---

approach MV-DG (Sec. 3.3) to jointly infer accurate 3D and camera viewpoints. A key challenge we seek to overcome is that the estimated camera viewpoints may have significant errors and that naively using all images to infer 3D can result in suboptimal estimates.
Pose Refinement via Gradient Descent. During 3D reconstruction via MV-DreamCaussian back- propagate gradients from the photometric loss (Eq. 1) back to update camera poses (implementing custom CUDA kernels to enable this gradient computation). This process allows the camera poses to become more precisely aligned as 3D reconstruction progresses, We denote by II' = GD 0, II) the resulting camera viewpoints from this optimization given the set of input images I, and 3D reconstruction i and initial poses II.
With this pose-and-30 co-optimization, we can instantiate version of our analysis- by-generative- synthesis framework by iteratively refining poses and reconstructing 3D given initial pose estimates Th from off-the-shelf system:
For k 1.
K:
02 = -V II. = GD 0x 1Ix-1)
(7)
For clarity, we present separate formulas for the reconstructed 3D 0g and the updated poses T..- though they are part of the same optimization process. Notably, in each iteration, we initialize the camera poses using the output from the previous iteration (I1x-1), while the 3D representation (0x) is reset and reconstructed from scratch.
Dealing with Outliers. Although the above iterative optimization framework can allow us to infer consistent poses and 3D reconstructions, errors in initial pose estimates. To overcome this, we additionally detect "outliers", i.e,, images with possibly large pose errors that degrade the quality of 3D reconstruction. We then modify our approach to leverage only the estimated inliers for 3D reconstruction while also separately performing a discrete search to update the outlier viewpoints.
susceptible to local optima and not robust to large
Iterative Outlier Identification. Our key insight is that an "outlier" image not only exhibits high reprojection error, making difficult to reconstruct on its own, but also that including it as a training image for 3D reconstruction degrades the overall quality, thus leading to poorer reconstruction even from other views! We operationalize this insight by classifying an image as an outlier if removing from training significantly improves performance on other images. More formally, let I denote the set of images after removing the 8th one and let E0, I, II) denote the average reprojection error of a 3D representation 0 over images II with (predicted) poses II. We consider an image as an outlier if
(MV-
>> (MVV-DG "II "I''",""
(8)
ie., adding the image to training set significantly increases the error for other views. For efficiency, instead of considering all images outlier candidates, we iterate over images in decreasing order of reprojection error. Given this procedure to detect outliers, at each iteration (except k = 1) we modify the above framework to first filter out the outliers found in previous iterations (along with the new "outlier candidate" " that gives the largest reprojection eтror at the last iteration):
(9) and only use the estimated inliers for optimizing 3D: 0₂ DV-DG _ij This filter-and- reconstruct loop stops when either the selected outlier candidate is determined to be an inlier (i.e. the condition 8 is not satisfied) the number of remaining inliers falls below threshold (e.g., 4).
1 _I
filter-outliersee outliers(I, 11-1)
Correcting Outlier Poses. While identifying the outliers allows us to prevent them from influencing the 3D inference, the finally recovered model may not capture the details from all images. We thus also attempt to "correct" the pose estimates for the outliers via a discrete search (followed by continuous optimization). Using the currently estimated 3D (reconstructed from only the inliers), for each image in the outlier set, we re-estimate its camera pose via render-and-comperr densely sample pose candidates a sphere and render images from the current 3D. We rank the pose candidates by measuring both pixel-space error (i.e., MSE) and perception error (i.e., LPIPS [49]). The pose candidate with the highest cumulative rank is selected as the optimal solution. Once all identified outliers are corrected, another reconstruction is performed to form a consistent 3D representation with all images using the updated poses.
We first
Our overall framework is very efficient (largely due to an efficient implementation of the recon- struction step), typically taking 5-10 minutes given 8 input images, with increased inference time depending on the number of estimated outliers. We include brief analysis of the inference time of our system in Sec. B.

---

Inpot
Initial Pose
SPARF
Sparse (w/ 3D)
7aom
0-1
Figure 3: Qualitative Comparison on Camera Pose Accuracy. Given initial poses from off-the- shelf methods (top to bottom: DUSt3R [41], Ray Diff. [47] and RelPose++ [18]). the refined poses from SPARF [38] are compared with the output of SparseAGS. The estimated cameras are aligned with ground truth (in black) with an optimal similarity transform. More results are available in Fig. 8.
Experiments
4.1 Experimental Setup
Datasets. We primarily evaluate our method on a real-world multi-view object-centric dataset NAVI [9]. This dataset includes high-quality foreground masks, precise camera poses, and 3D meshes. For each of the 35 objects in NAVI, we randomly select 5 multi-view sequences for pose estimation and recon- struction. Additionally, we assess our method on synthetic datasets, including GSO [7], ABO [4], and OmniObject3D [44]. Results for the synthetic datasets provided in Sec. E of the appendix.
Table 1: Comparison of Camera Rotation and Center Accuracy with SPARF [38]. We use three pose estima- tion baselines (RelPose++ [18], Ray Diff. [47], DUSt3R [4] and measure rotation accuracy at thresholds (5 and 15 degrees) and camera center accuracy at a threshold of 0.1 (of the scene scale). Eight images used.
Method
Rot. @5
Rot. 15
CC@0.1 1
RelPose++
10.9
56.4
26.0
w/ SPARF
28.6(+17.7)
51.9(-4.5)
37.9(+1199)
w/ SparseAGS
42.1(431.2)
67.6(+11.2)
53.3(427.3)
Ray Diff.
13.5
73.5
38.3
w/ SPARF
46.0(+32.5)
76.1(+2.6)
65.8(+27.5)
w/ SparseAGS
60.3(+46.8)
88.2(+14.7)
80.4(442.1)
Baselines. To evaluate camera pose ac-
curacy, we select three sparse-view pose estimation baseline methods: RelPoset+
DUSI3R
52.3
93.8
82.2
w/ SPARF
59.7(+7.4)
87.8(-6.0)
81.9(-0.3)
[18], Ray Diffusion [47], and DUS(3R
w/ SparseAGS
83.7(+31.4)
96.2(+2.4)
93.5(+11.3))
[41]. The first two are trained exclusively
on CO3D [24], while DUSt3R is trained on mixture of eight datasets, representing different levels of precision in initial camera poses. Our method initializes and improves the pose estimates from these baselines, and we also compare with SPARF [38], sparse-view view posc-NeRF co-optimization method. For evaluation novel view synthesis, we mainly compare our method with unposed sparse-viee reconstruction approaches, LEAP [12] and UpFusion [13] (we include comparison with SPARF in
Sec. D). We conduct experiments with varying numbers input images (N = 6, 8, 10, 16).
For pose accuracy. we follow prior works [18, 47] and report the following metrics: (1) Rotation accuracy: we compare pairwise relative rotation between the predicted cameras and ground truth. We report the proportion of samples with enrors less than specified threshold, such as and 15 degrees. (2) Camera center accuracy: we align the predictions and ground truth using an optimal similarity transform and report the proportion of camera centers within 10% of the scene
Metrics.
7

---

SparseAGS (Ray DIE)
SparseAGS (DUSER)
Input
GT
LEAP
Zoom
Figure 4: Qualitative Comparison with LEAP [12] on Novel View Synthesis. We use two pose estimation baselines (Ray Diffusion [47] and DUSt3R [41]). SparseAGS better preserves details from the input images and shows enhanced performance with more accurate initial camera poses. More results available in Fig. 9 of the appendix.
scale relative to the ground truth. We evaluate our 3D representation via novel-view synthesis and report PSNR and LPIPS [49] for the rendered views. In our ablation study, we also assess the 3D geometry using the Fl score, comparing our recovered geometry against the ground truth 3D meshes.
Table 2: Evaluation of Camera Pose Accuracy with Varying Numbers of Input Images NAVI [9]. Here we use the same evaluation protocols as Tab.
N
N
N =
Method
15"
CC@0011
CC#0.1
Rot
Rot, 15"
CC#0.1
RelPoses+ 11.0
$7.0
28.6
SpanseAGS
27.3(+16.3)
60.2()3.2)
47.0x-18.4)
Ray Diff.
13.3
74.3
44.1
12.9
73.4
36.1
12.6
74.0
34,0
SparseAGS 44.9(+3166)
83.6(+9.3)
73.6(=29.5)
70.0x+57.1)
89.6(+162)
83.4(+47.3)
82.3(+69.7)
93.2(-19.2)
89.0(+55.0)
DUSL3R
50.3
93.4
82.1
52.9
95.0
84.4
55.5
94.9
84.2
SpansAGS
74.3(+24.0)
95.1(+17)
92.3(-10.2)
87.00-4411)
97.3(+2.3)
94.3(49.9)
91.1(+35.6)
97.7(-28)
95.0k+10.8)
4.2 Evaluation
Camera Pose Accuracy. We compare SparseAGS with SPARF [38] on pose accuracy given eight input images guantitatively in Tab. 1 (numbers are in percentage) and qualitatively in Fig. 3. We find that SparseAGS consistently yields larger improvements than SPARE, which sometimes even leads to degraded accuracy (marked by red numbers). We attribute this to the unreliable correspondences extracted by SPARF (we include an example Fig. 7), as the input images in NAVI may exhibit more significant viewpoint changes compared scene-level datasets, e.g., DTU [10] where SPARF

---

Table 3: Quantitative Comparison of 3D Reconstruction on NAVI [9]. We compare our method
with two unposed approaches: LEAP [12] and UpFusion [13], using varying numbers of input images (N). We adopt two pose initializations (Ray Diff. [47], DUSt3R [41]) reporting PSNR and LPIPS.
Initial
N =
= 10
Method
Cam. Pose
PSNR t
LPIPS 4
PSNR t
LPIPS
PSNR 1
LPIPS +
LEAP
X
12.84
0.2918
12.93
0.2902
12.98
0.2890
UpFusion
X
13.30
0.2747
13.27
0.2744
/
SparseAGS
Ray Diff. DUS13R
13.63
0.2697
15.30
0.2304
16.80
0.1960
SparseAGS
15.56
0.2173
17.03
0.1870
18.03
0.1660
originally tested. Note that training SPARF (or other NeRF-based methods) is far more expensive than ours, and it may take more than hours. Whereas our method typically finishes in 5-10 minutes. More analysis and detailed comparisons with SPARF pose accuracy and novel view synthesis are in Sec. D.
We vary the number of input images (N = 6, 10, 16) and report camera pose accuracy in Tab. 2 (we only test RelPose++ [18] with six images as inference with more than eight images is not supported). SparseAGS consistently enhances baseline performance for both rotation and camera center accuracy, with particularly significant gains for stricter metrics (e.g., Rot. @5"). Moreover, the improvements tend to further increase with the number of input images. These results demonstrate that our method is robust to varying levels of initial camera poses and generalizes well across different input numbers.
3D Reconstruction. In Table 3. we compare SparseAGS with two unposed approaches, LEAP [12] and UpFusion [13], reporting metrics for 3D reconstruction (novel view synthesis). Our method consistently outperforms both baselines across dif- ferent numbers of input images and with two pose estimation initializations. While SparseAGS shows continuous improvements with an increasing num- ber of input images, the performance of LEAP and UpFusion nearly saturates in terms of both PSNR and LPIPS. We hypothesize that unposed methods struggle to utilize additional input images beyond their training capacity without further training ad- justments (LEAP is trained using five views, while UpFusion is trained with maximum of six images). In contrast, our method is flexible w.r.t. the number of input images, eliminating the need for re-training- A qualitative comparison with LEAP and UpFusion presented in Fig. and Fig. respectively. The results show that Sparse better preserves the details in input images by explicitly modeling cameras and produces higher-quality novel view synthesis with more precise initial camera poses.
lepal
SpansAis
Spurw 4CS
Figure 5: Qualitative Comparison with Up- Fusion [13] on Novel View Synthesis. We use two pose estimation baselines (Ray Diffu- sion [47] and DUSt3R [41]) as in Fig. 4. Note that the left eye and symbol @ of the Chicken Racer is missing in UpFusion's output, prob- ably because of the "first-image bias", while Sparse preserves these details.
Table 4: Ablation Study. Using initial poses from Ray Diffusion [47] for eight input images, we
ablate the effect of each proposed component of our approach.
Method
Rot_@5
Rot, 1I..@15 15
CC@0.11
PSNRT
LPIPS,
F1@00011
Ray Diffusion
13.5
73.5
38.3
Pose-3DCCo Co-opt. (w/ SDS)
(1)
28.4
79.9
57.7
12.72
0.3100
46.3
SDS (vanilla 23 (200)
(2)
30.2
78.3
57.3
13.04
0.2999
49.9
SDS (Our 6-DoF Zero
(3)
34.6
83.1
65.3
13.44
0.2793
57.2
Outlier Removal & Correction
(4)
60.3
88.2
80.4
15.30
0.2304
68.2
4.3 Ablation Study
We ablate the effectiveness of each component in our approach (Tab. 4) using initial pose estimates from Ray Diffusion [47] with eight input images. In addition to camera pose metrics and nivel-view

---

Input
Novel Views Normal
GT
SBS
SDS
wio SDS
SDS
Figure 6: Qualitative Comparison with No-Generative-Pine Setup (N =). Novel Views (NV) & Normal: From left to right- GT, NV w/o SDS, NV w/ SDS, Normal w/o SDS, Normal w/ SDS, Leveraging generative priors in the form of SDS contributes to consistent 3D representation.
metrics, we report the Fl score of reconstructed meshes, which reflects their alignment with ground truth meshes.
(Appropriate) Generative Priors Improve Analysis by Synthsis. We find that adding generative priors (Eq. 2) to naive pose-3D co-optimization (Eq. 1) improves both pose accuracy and 3D reconstruction quality (comparison between (1) and (3) shows consistent improvements in all metrics). However, vanilla Zero-1-to-3 [20] is not suitable for providing such priors in real-world scenarios, as we observed a drop in camera rotation and center accuracy (compare (1) with (2) Rot. @ 15" and CC (C@001). This is because 3-DoF camera parameterization cannot well represent the camera poses in the wild. Although the numerical improvements may appear marginal (e.g., PSNR), Fig. presents qualitative comparison of 3D reconstruction with and without our 6-DoF novel-view generative priors. Supervision on novel views via SDS helps form a consistent 3D representation.
Outlier Removal and Correction. The presence of outlier initial cameras introduces significant chal- lenges to pose-3D 3D co-optimization. Our iterative outlier removal and correction pipeline effectively addresses this issue. For instance, comparing (3) with (4) shows a substantial improvement: Rot. @ increased from 34.6% to 60.3% (25.7% absolute improvement), PSNR improved from 13.44 to 15.30, and F1 @0.01 increased from 57.2 to 68.2 (11.0 point absolute improvement). These results confirm the effectiveness of our approach.
Conclusion
In this work, we presented ApS, a framework for joint pose estimation and 3D reconstruction combining off-the-shelff pose estimation methods with novel-view synthesis generative prior for robust inference in real-world vierse-view captures.
Limitations. While our experiments demonstrated clear improvements over initializations and stronger performance compared to prior 3D reconstruction methods, there are several challenges that remain. First, our approach does rely on some reasonable off-the-shelf pose estimates and cannot succeed a large fraction of the predictions have large error. Secondly, SparseAGS (similar to existing baselines) does not deal with truncation or occlusion and cannot be directly applied to scenarios with close-up images of parts objects or cluttered scenes with one object occluding the other. Finally, we focused here on an object-centric setting, and it would be interesting to extend our approach broader settings, e.g., deploying our framework in conjunction with methods that learn novel-view generative priors for scenes.
10

---

Acknowledgements
We thank Zihan Wang and the members of the Physical Perception Lab CMU for their valuable discussions. We especially appreciate Amy Lin and Zhizhuo (Z) Zhou for their assistance in creating figures, as well as Yanbo Xu and Jason Zhang for their feedback on the draft.
This work was supported in part by NSF Award IIS-2345610. This work used Bridges- 2 [1] at Pitts- burgh Supercomputing Center through allocation CIS240166 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
References
[1] Shawn T Brown, Paola Buitrago, Edward Hanna, Sergiu Sanielevici, Robin Scibek, and Nicholas A Nystrom. Bridges-2: A platform for rapidly-evolving and data intensive research. In Practice and experience in advanced reseurch computing, 2021.
[2] Xu Chen, Zijian Dong, Jie Song, Andreas Geiger, and Otmar Hilliges. Category level object pose estimation via neural analysis-by-synteest In ECCV, 2020.
[3] Weihao Cheng. Yan-Pei Cao, and Ying Shan. Id-pose: Sparse- view camera pose estimation by inverting diffusion models, 2023. arXiv preprint arXiv:2306.17140.
[4] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, et al. Abo: Dataset and benchmarks for real-world 3d object understanding. In CVPR, 2022.
[5] Matt Deitke, Ruoshi Liu, Matthew Wallingford. Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: universe of 10m+ 3d objects. In NeurlPS, 2023.
[6] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023.
[7] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Rey- mann, Thomas McHugh, and Vincent Vanhoucke. Google scanned objects: high-quality dataset of 3d scanned household items. In ICRA, 2022.
[8] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[9] Varun Jampani, Kevis-Kokitsi Maninis, Andreas Engelhardt, Arjun Karpur, Karen Truong, Kyle Sargent, Stefan Popov, André Araujo, Ricardo Martin Brualla, Kaushal Patel, et al. Navi: Category-agnosse image collections with high-quality 3d shape and pose annotations. In NeurIPS, 2023.
[10] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In CVPR, 2014.
[11] Hanwen Jiang, Zhenyu Jiang, Kristen Grauman, and Yuke Zhu. Few-view object reconstruction with unknown categories and camera poses. In 3DV, 2024.
[12) Hanwen Jiang, Zhenyu Jiang, Yue Zhao, and Qixing Huang. LEAP: Liberate sparse-view 3d modeling from camera poses. In ICLR, 2024.
[13] Bharath Raj Nagoor Kani, Hsin-Ying Lee, Sergey Tulyakov, and Shubham Tulsiani. Upfusion: Novel view diffusion from unposed sparse view observations. In ECCV, 2024.
[14] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. In ACM TOG, 2023.
[15] Daniel Kersten, Pascal Mamassian, and Alan Yuille. Object perception as bayesian inference. In Annu. Rev. Psychol., 2004.

---

[16] Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, Justin Carpentier, Mathieu Aubry, Dieter Fox, and Josef Sivie. Megapose: 6d pose estimation of novel objects via render & compare. In CoRL., 2022.
[17] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r, 2024. arXiv preprint arXiv:2406.09756.
[18] Amy Lin, Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose++: Recovering 6d poses from sparse-view observations. In 3DV, 2024.
[19] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey, Barf: Bundle-adjusting neural radiance fields. In ICCV, 2021.
[20] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: sero-shot one image to 3d object. In ICCV, 2023.
[21] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In ACM SIGGRAPH, 1987.
[22] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[23] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
[24] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny, Common objects 3d: Large-scale learning and evaluation real-life 3d category reconstruction. In ICCV, 2021.
[25] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[26] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic -xk-tt--mmae diffusion models with deep language understanding. In NeurIPS, 2022.
[27] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, al. Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations. In CVPR, 2022.
[28] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, et al. Zeronvs: Zero-shot 360-degree view synthesis from single real image. In CVPR, 2024.
[29] Johannes L Schonberger and Jan-Michad Frahm. Stmuctureffrom-mottie revisited. In CVPR, 2016.
[30] Roger N Shepard and Jacqueline Metzler. Mental rotation of three-dimensional objects. In Science, 1971.
[31] Samarth Sinha, Jason Y Zhang, Andrea Tagliasacchi, Igor Gilitschenski, and David B Lindell. Sparsepose: Sparse-vier view camera pose regression and refinement. In CVPR, 2023.
[32] Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM SIGGRAPH. 2006.
[33] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[34] Yujing Sun, Caiyi Sun, Yuan Liu, Yuexin Ma, and Siu Ming Yiu. Extreme two-view geometry from object poses with diffusion models, 2024. arXiv preprint arXiv:2402.02800.
12

---

[35] Jiaxiang Tang.
Stable :rabfl-dra Text-to-3d with stable-diffusion, 2022. http:///thhbb.oooo/aaaawwwe///// drwabbd-sdddmmusion...
[36] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2024.
[37] Prune Truong, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning accurate dense correspondences and when to trust them. In CVPR, 2021.
[38] Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. CVPR, 2023.
[39] Jianyuan Wang, Christian Rupprecht, and David Novotny. Posediffusion: Solving pose estima- tion via diffusion-aiin aided bundle udjustment. In ICCV, 2023.
[40] Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. PF-LRM: Pose-free large reconstruction model for joint pose and shape prediction. In ICLR,, 2024.
[41] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024.
[42] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf- Neural radiance fields without known camera parameters, 2021. arXiv preprint arXiv:2102.07064
[43]
Wu, Yen-Chun Chen, Bolivar Solarte, Lu Yuan, and Min Sun. ifusion: Inverting dif- fusion for pose-free reconstruction from sparse views, 2023. arXiv preprint arXiv:2312.17250.
[44] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang. Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al. Omniobject3d: Large-vocabulary object dataset for realistic perception, reconstruction and generation. In CVPR, 2023.
[45] Paul Yoo, Jiaxian Guo, Yutaka Matsuo, and Shixiang Shane Gu. Dreamsparse: Escaping from plato's 's cave with 2d diffusion model given sparse views. In NeurIPS, 2023.
[46] Alan Yuille and Daniel Kersten. Vision as bayesian inference: analysis by synthesis? In Trends in cognitive sciences, 2006.
[47] Jason Y. Zhang, Amy Lin, Moneish Kumar, Tru-Hsuan Yang, Deva Ramanan, and Shubham Tulsiani. Cameras as rays: Sparse-view pose estimation via ray diffusion. In ICLR, 2024.
[48] Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose: Predicting probabilistic relative rotation for single objects in the wild. In ECCV, 2022.
[49] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018.
[50] Chen Zhao, Tong Zhang, and Mathieu Salzmann. 3d-aware hypothesis & verification for generalizable relative object pose estimation. In ICLR, 2024.
[51] Guangyao Zhou, Nishad Gothoskar, Lirui Wang. Joshua B Tenenbaum, Dan Gutfreund, Miguel Låzaro-Greede Dileep George, and Vikash K Mansinghka. 3d neural embedding likelihood: Probabilistic inverse graphics for robust 6d pose estimation. In ICCV, 2023.
[52] Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for
3d reconstruction. In CVPR, 2023.
13

---

Appendix
Sec. A: Broader social impacts.
Sec. B: Analysis of inference time.
Sec. C: Implementation details.
Sec. D: More detailed comparisons with SPARF [38].
Sec. E: Additional results on three synthetic datasets.
Sec. F; More qualitative comparison camera pose accuracy with SPARF [38].
Sec. G: More qualitative comparison on novel view synthesis with LEAP [12].
A Broader Impacts
Our method leverages generative priors from diffusion models, enabling 3D reconstruction and pose estimation in the wild. This may benefit the generation of 3D assets for common users. However, we acknowledge that the web-scale data used for training these diffusion models may include content with potential negative social impacts, such biased representations or harmful stereotypes. Therefore, while our approach benefits from the richness of the data, we must remain vigilant about the ethical implications and strive to mitigate any adverse effects.
B Analysis of Inference Time
For 8-image inference using a single RTX A5000 GPU, one reconstruction with MVVDreamGaussian takes about 2 minutes complete, and the "render-and-comparc" for each outlier takes around minute. Our full pipeline (using RayDiffusion initialization) detected an average of 0.94 outliers per sequence on NAVI, resulting in an inference time of around 9 minutes. We believe additional engineering efforts can further optimize and reduce inference time.
C Implementation Details
Finetuning Zero-I-to-3 with 6-DoF Camera Conditioning. To learn the 6-DoF camera condi- tioning for novel view synthesis in the wild, we first initialize the weights of Zero- 1-to-3 using the Zero123-XX checkpoint [5] and replace the original camera condition with ours. We then only finetune the layers associated with camera conditioning (i.e., the linear projection and cross-attention layers) while freezing all other layers. This approach is more efficient than all-layer finetuning. To alleviate the synthetic data bias learned by the vanilla Zero-1-to-3, 1-to-3, we include the training samples from CO3D [24] along with the Objaverse [6] renderings provided by Liu et al. [20] for finetuning. For computational resources, we used 8 V100 GPUs, setting batch size of 36 per GPU with gradient accumulation of The model was trained for 23,500 iterations, taking -4 days.
Iterative Outlier Removal Details. For the outlier condition specified in inequality 8, we employ LPIPS the reprojection error metric, applying a threshold of 0.05. The reconstruction loop terminates when the average reprojection eтror reduction falls below this threshold or if the number of estimated inliers drops below predefined count. Specifically, we use threshold of inliers for N 6 and N 8, inliers for N = 10, and 12 inliers for N = 16. These iteration counts generally suffice handle outliers given the current capabilities of state-of-the-an pose estimation systems.
Comparing with Pose-free Methods. To compare with pose-free methods, we follow these steps to obtain their novel view renderings: First, we normalize the ground truth camera poses match the scale of the coordinate systems used by these methods. Next, we render target images from novel views using their relative camera poses with respect to the first input image. Additionally, we adjust the camera intrinsics (focal length and principal point) during inference to align the foreground mask of the rendered images with the ground truth to reduce scale difference.
D More Detailed Comparisons with SPARF
In addition to our main text comparing to SPARF for pose estimation, here we also present Novel View Synthesis (NVS) metrics. We report the results Tab. 5, using DUSt3R pose (N = 6, 8) as initialization. Due to SPARF's long training time (about 10 hours per instance), could only include
14

---

Table 5: Expanded Comparison of Pose Accuracy and Novel View Synthesis with SPARF [38]. In addition the primary metries presented in the main text, we report Average Rotation Error and Improvement Rate (IR), which indicates the percentage of sequences with reduced pose error. See Sec. D for further analysis and detailed explanations.
Pose Metries
NVS Metries
6
Avg Rot. Err
Rot.
CC@0.
IR
PSNR
LPIPS
*PSNR
*LPIPS
DUSI3R
7.90
48.5
80.2
/
1
SPARF
17.07
47.9
67.6
0.41
12.80
0.3201
15.30
0.2478
w/ SparseAGS
5.82
73,9
92.1
0.80
15.52
0.2179
16.23
0.1844
Pose Metries
NVS Metries
=
Avg Rot. Err
&ot
CC@0.1
IR
PSNR
LPIPS
*PSNR
*LPIPS
DUSBR w/ SPARF
8.71
51.6
79.5
1
18.17
57.3
68.6
0.54
13,42
0.3059
15.44
0.2584
w/ SparseAGS
5.99
81.7
92.5
0.93
17.02
0.1874
17.48
0.1695
70 sequences (2 sequences per object) in these two experiments. Notably, for a direct comparison on NVS, removed backgrounds from the input images, whereas no masking was applied in Tab. 1. This may slightly disadvantage SPARE, as backgrounds provide additional cues for pose registration and correspondences.
The results indicate that Sparse outperforms SPARF in pose accuracy and novel view quality well. In fact, found that SPARF can often make the poses worse compared to the (relatively accurate) DUSt3R initialization (measured via Improvement Rate that indicates the percentage of sequences with reduced pose error). This is likely because the correspondences leveraged by SPARF in its optimization are not robust and are susceptible to false matches see Fig. 7 for an example.
As an attempt compare novel view quality despite the difference in pose accuracy, we report *PSNR and LLPISS which are measured only on the sequences where SPARF improves pose accuracy and find that even in these, our approach outperforms it. We also observed that while SPARF works well on novel views close to the imput, floaters constantly appear with significant viewpoint changes. In contrast, generative prior leads to a more consistent 3D representation.
SPARF Leveraged False Estimated Correspondences
Input
Sparse AGS
SPARF
Sparse SPARF
Figure SPARF Fails When Incorrect Correspondence is Leveraged. Rightmost Section: From left to right Source, Target, Warped Source to Target Based on Estimated Correspondence, Confidence Map (yellow indicates high confidence). The estimated false correspondence due to symmetric patterns causes pose optimization to fail, leading degraded novel views in SPARF.
E Evaluation on Synthetic Datasets
Table 6: Evaluation of Rotation Accuracy on Three Synthetic Datasets (GSO [7], ABO [4] and OmniOhject3D [44]). We test method ID-Pose [3] with eight images as input. We measure rotation accuracy at two thresholds (15 and 30 degrees).
GSO
ABO
OmniObject3D
Method
Rot. @ 15
Rot. @30
Rot. 15"
Rot. @ @
Rot. @ 15"
Rot. @ 30
ID-Pose
52.5
59.2
47.3
52.7
55.1
62.4
w/ SparseAGS 68.8(+16.3)
74.8(+15.6)
64.4(+17.1)
69.0(+16.3)
75.1(420.0)
79.2(+16.8)
Though our main focus real-world data, we apply our approach three synthetic datasets (GSO, ABO, OmniObject3D) for complete evaluation of our approach. We use ID-Pose [3] as a baseline,
15

---

which inverses the novel-view wiww-synthesiiisss ability of Zero-1-to-3 [20] for pose estimation and adopts 3-DoF camera parameterization. Here, we also use the vanilla Zero-I-to-3 for multi-view SDS loss to accommodate this camera definition and for fair comparison. We report the results on camera rotation accuracy Tab. 6. Across these datasets, our approach consistently improves performance on two metrics, even though ID-Pose uses the same backbone model (vanilla Zero- 1-t-3) as we do. These results demonstrate that our approach more effectively leverages the generative priors from Zero-I-to-3, achieving better pose accuracy. Plus, also show that our method is applicable to synthetic data, showing strong generalization abilities across different datasets.
Additional Visualizations for Inferred Poses
See Fig. 8.
G
Additional Visualizations for Novel View Synthesis
See Fig. 9.
16

---

Inpur
Sparse AG (w) 3D)
10
Figure 8: More Qualitative Comparison on Camera Pose Accuracy, Given initial camera poses from tfi-the-shelf methods, the refined poses from SPARF [38] are compared with the output of SparseAGS. We align the estimated cameras with ground truth (in black) with an optimal similarity transform.
17

---

SparseAGS (Ruy Dift)
SparseA (DUSI3R)
Input
GT
LEAP
Beda
Figure 9: More Qualitative Comparison with LEAP [12] on Novel View Synthesis. We use two pose estimation baselines (Ray Diffusion [47] and DUSt3R [41]). SparseAGS better preserves details the input images and shows enhanced performance with more accurate initial camera poses.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide extensive experiments in Sec. 4 to support our claims. Guidelines:
The answer NA means that the abstract and introduction do not include the claims
made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include the discussions of our limitations in Sec. 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well- sell-specffication, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include theoretical results, and we conduct experiments to verify the effectiveness of our method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
19

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include our experiment setup in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibilityy which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., s. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We, unfortunately, are not able to clear data and codes before the camera-ready deadline. However, we will release our code as well as the data used soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/ /gwidee////////eeee/eeeeeee wwt////obbiiiiiiiooooooooo httpaiiissiooooiiii....oo Policyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
20

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
wwtppcccppcccccccccccccccccccccc htlllii//uidesssee / /CodeSubmissss ssisssiii assonPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include experiment setup in Sec. 4.1 and details of our method in
Sec. C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them. The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification: We follow the standard practice in the community to evaluate methods, which do not include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We include such information in Sec. B and Sec. C. Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https htte:://neuurpppe ec wttp:/cccppc....eeeeeeeec Ceaelicc//hiiiee www.iiiiiiieeiimemmeeee ines?
Answer:
[Yes]
Justification: We followed the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the social impacts in Sec. A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No]
Justification: Although our method leverages diffusion priors, we do not generate images directly. Instead, we reconstruct the 3D images from user-specified images. We will require the users to strictly follow the guidelines when our models are publicly available.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring
22

---

that users adhere to usage guidelines or restrictions to access the model or implementing
safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors
should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We give proper credit to the resources we used mainly via citations and textual descriptions.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We currently not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their
submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer:[Na [NA]
Justification: Our paper does not include such content. Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
23

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not include the related content.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "VO6TAj1KwL", "reviewer": "Reviewer_ePgG", "summary": "This paper presents a method named MV-DreamGaussian for tackling the problem of 3D reconstruction from sparse multi-view inputs. In particular, the paper extends the DreamGaussian work to use multi-view images as the inputs and proposes a scheme to optimize the inaccurate camera poses of the multi-view images.", "questions": "- I'm quite curious what the reconstruction quality the method can achieve without 3D generative prior but with the proposed refinement. Namely the combination of (1) and (4) in Table 4.\n- How are the poses used for generative prior sampled in addition to the input views?\n- How are the thresholds for pose outlier removal tuned?", "limitations": "The authors have discussed the limitations of the paper and I generally agree with them.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well written and I can follow smoothly.\n- The authors proposed a finetuned version of Zero-1-to-3 with 6 DoF camera parametrization which shows an advantage over 3 DoF camera parameterization in the original paper.\n- The proposed pose refinement scheme is novel and very effective according to the authors' experiments compared with SPARF as well as the ablation study which shows that adding the proposed pose refinement improves the pose accuracy and reconstruction quality significantly. The design of the outlier removal based on photometric error ranking and discrete search is empirical but works quite well.", "weaknesses": "- This paper presents very limited novelty in the reconstruction part with a trivial extension to DreamGaussian to use multi-view images, which is already implemented in a public repository [stable-dreamfusion](https://github.com/ashawkey/stable-dreamfusion).\n- The major weakness of the paper is the lack of fair comparisons in terms of the 3D reconstruction. The authors only compared with LEAP for the 3D reconstruction. However, LEAP is a work that **does not require any pose inputs**, whereas the proposed work needs relatively good pose initialization (e.g., Dust3r) and conduct refinement on it. In addition, the underlying 3D representation is different, too: LEAP uses NeRF while the proposed work uses 3D Gaussian. I'm confused as to why the authors did not compare with SPARF for the reconstruction quality too since SPARF shares the same input setup as the proposed work. Besides, the very recent work DMV3D would also be a good method to compare with.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> This paper presents very limited novelty in the reconstruction part with a trivial extension to DreamGaussian to use multi-view images, which is already implemented in a public repository stable-dreamfusion.\n\nWe respectfully disagree with the reviewer's assessment that our extension to DreamGaussian is trivial. While using multiple input images for DreamGaussian has been implemented in the stable-dreamfusion GitHub repository (as mentioned in L173-174 of our main paper), our work goes further by enabling the handling of 6-DoF camera poses and gradient-based pose optimization. To achieve this, we fine-tuned Zero-123 with a novel 6-DoF camera parameterization and implemented customized CUDA kernels, as the official Gaussian Splatting codebase did not support this feature.\n\nWe appreciate that the reviewer recognizes the novelty of our pose refinement strategy. However, we kindly ask the reviewer to also consider the broader contributions of our proposed system and the specific problem we aim to address. Specifically, we integrate gradient-based pose optimization with generative priors to tackle sparse-view pose refinement, a scenario prone to overfitting. Additionally, we introduced an outlier identification and correction system to manage significant initial pose errors so that our method can work in real-world scenarios where only estimated camera poses from off-the-shelf methods are available. To the best of our knowledge, no existing work effectively addresses this challenging yet practical setup.\n\n> The major weakness of the paper is the lack of fair comparisons in terms of the 3D reconstruction.\n\nPlease refer to our general response, where we include an evaluation of SPARF regarding novel view synthesis. We apologize for not including these in our earlier version -- SPARF training is slow (the overall training takes more than 10 hours **per instance** on a single GPU such as V100) and due to limited resources, we only compared to SPARF's pose optimization stage (which takes 30% iterations). We also note that the suggested baseline DMV3D is not applicable in our setup -- it generates multiple views as output but does not allow multiple unposed views as input! Moreover, their implementation is not open-sourced.\n\n> I'm quite curious what the reconstruction quality the method can achieve without 3D generative prior but with the proposed refinement. Namely the combination of (1) and (4) in Table 4.\n\nWe thank the reviewer for suggesting this experiment. We conducted the ablation study under the same settings as Table 4 and compared the results with our full system:\n\n| Full Method |  Rot@5°  | Rot@15°  |  CC@10%  |    F1   |   PSNR    |   LPIPS     |   PSNR*    |   LPIPS*    |\n| ----------- | :------: | :------: | :------: | :-------: | :--------: |:-------: | :--------: | --------- |\n| w/o SDS     | **66.2** |   86.8   |   77.8   | 63.57     | **16.05** | **0.2277** |   17.90   |   0.1990   |\n| w/ SDS      |   60.3   | **88.2** | **80.4** | **68.19** |   15.30   |   0.2304   | **18.18** | **0.1778** |\n\nIn this setup, the quality of the 3D reconstructions is worse as there are more artifacts (significant roughness and holes) in the geometry. However, we found that this setup does achieve high pose accuracy (even being more precise on Rot@5° than our full system). Analyzing further, we found the no-SDS setup makes it easier to remove outliers (removing one input image more significantly reduces the reprojection errors). Quantitatively, **97.1%** of sequences in this setup involve at least one identified outlier, compared to **67.2%** in the full system. \n\nPerhaps more surprisingly, we found that this setup also yielded slightly better view synthesis results (PSNR and LPIPS), but this did not correspond to the qualitative results (see **Fig. 2** in our rebuttal PDF file). We hypothesize that this discrepancy is because we compute a single global alignment between our optimized camera poses and GT poses before evaluating NVS, and this alignment maybe slightly more precise for the baseline leading to improved NVS metrics even though the 3D/view-synthesis quality is worse. To mitigate the effects of alignment in NVS evaluation, we also report PSNR*, and LPIPS* -- where we locally optimize camera pose for each novel view for a few steps using photometric error while keeping our 3D representation fixed. We see that our full system does yield better predictions, highlighting that the use of SDS does indeed improve the novel-view generations.\n\n> How are the poses used for generative prior sampled in addition to the input views?\n\nWe preprocess the initial camera poses provided by off-the-shelf methods to align the world origin with the approximate intersection of their optical axes. Next, we create a sphere centered at the origin, with a radius equal to the average distance of all cameras from the origin. When rendering novel views for SDS, we sample points on the sphere for camera translation using random azimuth and elevation angles. For rotation, we orient the sampled camera towards the origin.\n\n> How are the thresholds for pose outlier removal tuned?\n\nDuring the early stages of our work, we conducted initial experiments on a few samples from the CO3D dataset and found that a threshold of 0.05 for LPIPS generally worked well. This setting was maintained for all our in-the-wild evaluations on NAVI reported in the paper. We did not tune the threshold further for these results, though a fine-grained search might yield improved performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bfe3uEMTe6", "reviewer": "Reviewer_uJqP", "summary": "This paper proposes a method for the joint reconstruction of camera poses and 3D objects given sparse input views. The core idea is to use a pose-conditioned diffusion model (Zero-123) as a prior, impose the SDS loss, and jointly optimize the poses and objects, similar to the approach in ID-pose. To improve the robustness and quality of the optimization, the authors made several modifications: (1) Using a 6 DoF pose-conditioned diffusion model instead of a 3 DoF model. (2) Adding strategies for outlier detection and correction. (Although somewhat empirical, it proves effective.)\n\nThis approach requires initial camera poses (from methods such as RelPose++, RayDiffusion, etc.) and is not capable of reconstructing poses from scratch (e.g., purely random camera poses). Experimental results demonstrate that, compared to SPARF and ID-pose, the proposed method achieves better pose estimation quality. Additionally, it provides better object reconstruction in terms of novel view synthesis quality compared to LEAP.", "questions": "(1) A related work is lacking in discussion:  Sun, Yujing, et al. \"Extreme Two-View Geometry From Object Poses with Diffusion Models.\" arXiv preprint arXiv:2402.02800 (2024).\n\n(2) Is the testing data included in the training set for fine-tuning the 6-DoF diffusion model?", "limitations": "As discussed in the weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The approach is technically sound, and I believe the reported results are reproducible.\n\n(2) The reconstructed results look good and represent the state-of-the-art in object-level pose-free reconstruction.\n\n(3) The paper is well-written, making it easy to read and understand.", "weaknesses": "(1) This optimization-based method requires more time compared to a feed-forward model, taking about 5-10 minutes. Additionally, the writing discussing this aspect is somewhat unclear: the paper states, “with increased inference time depending on the number of outliers.” Could this statement be more specific? How much does the time increase with the number of outliers? The correction of outliers may be time-consuming as it requires dense searches of initial camera poses.\n\n(2) (Minor) The method focuses only on object-level reconstruction, which makes the scope seem narrow.\n\n(3) The authors do not sufficiently discuss experiments in a more “standard” sparse-view setting, such as using 3 or 4 views. The reported experiments use at least 6 views, which is not a particularly small number.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> This optimization-based method requires more time compared to a feed-forward model, taking about 5-10 minutes. Additionally, the writing discussing this aspect is somewhat unclear: the paper states, “with increased inference time depending on the number of outliers.” Could this statement be more specific? How much does the time increase with the number of outliers? The correction of outliers may be time-consuming as it requires dense searches of initial camera poses.\n\nWe apologize for the lack of clarity and address the inference time in our general response above, and will update the final version to include these details. We agree that our system is slower than feed-forward methods (e.g LEAP), but allows significantly more accurate generations. Compared to prior NeRF-based optimization methods however, our gaussian-splatting based system is far more efficient e.g. SPARF takes 10hrs per instance whereas our systems takes ~9min.\n\n> The authors do not sufficiently discuss experiments in a more “standard” sparse-view setting, such as using 3 or 4 views. The reported experiments use at least 6 views, which is not a particularly small number.\n\nWe chose 6-8 images as a setting representative of online marketplaces, but agree with the reviewer that it is important to also study even fewer views and report an experiment analyzing N=4 below.\n\n| Method  | Rot@5° | Rot@15° | CC@10% |\n| ------- | :----: | :-----: | :----: |\n| DUSt3R  |  52.0  |  90.3   |  79.0  |\n| w/ Ours |  57.8  |  90.5   |  84.5  |\n\nWhile our system does lead to consistently improved poses, the gains are less prominent compared to settings with more views. We believe this is because our approach does rely on multiple images 'co-observing' common 3D regions (to guide pose correction), but these are not common if we randomly sample a small set of views around an object, thus leading to diminishing benefits with very few images. \n\n> A related work is lacking in discussion: Sun, Yujing, et al. \"Extreme Two-View Geometry From Object Poses with Diffusion Models.\" arXiv preprint arXiv:2402.02800 (2024).\n\nWe thank the reviewer for sharing relevant work, which we will incorporate in our final version. The process of matching the generated images with the target image is similar to our outlier correction procedure at a high level, but we leverage a reconstructed 3D representation to render novel views instead of purely using diffusion models to generate them. Regarding this perspective, our novel views may be more detailed and faithful as we leverage multiple input views.\n\n>  Is the testing data included in the training set for fine-tuning the 6-DoF diffusion model?\n\nNo, we did **not** include NAVI in our data to fine-tune Zero-123 with 6-DoF camera conditions.\n\n> The core idea is to use a pose-conditioned diffusion model (Zero-123) as a prior, impose the SDS loss, and jointly optimize the poses and objects, similar to the approach in ID-pose.\n\nWe would like to clarify that our method is not similar to ID-Pose. ID-Pose estimates camera poses by optimizing the relative pose conditions in Zero-123 for image pairs while it does not form any 3D representation. In fact, we adopted ID-Pose as initialization for our experiments on three synthetic datasets in the supplementary and show that our systems leads to significant gains over it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Na2Dv7a3OL", "reviewer": "Reviewer_mujy", "summary": "This paper proposes a framework for joint 3D reconstruction and pose refinement. Specifically, given estimated camera poses from off-the-shelf models, the proposed method first leverages diffusion priors and rendering loss for 3D reconstruction. The 3D reconstruction is further used to refine the current pose parameters. The 3D reconstruction and pose refinement are conducted in an alternative way. An outlier identification and correction strategy is also introduced to make full use of the given image while mitigating the adverse effect of noisy camera estimations at the same time. Experimental comparison with several pose estimation baselines shows that the proposed method can refine inaccurate pose estimation effectively.", "questions": "1. The proposed method is evaluated on the NAVI dataset. It seems that the dataset is quite simple as shown in Fig. 3 and Fig. 4. The reviewer is wondering about the performance of the proposed method on more complex scenes?\n2. The reviewer is wondering about the separate ablation results on the outlier removal and correction.", "limitations": "Limitations are addressed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper tackles a practical problem in real-world scenarios, where ground truth camera poses are not always available.\n2. The proposed method is shown to be effective when applying to different pose estimation baselines.\n3. The proposed outlier removal and correction is effective from the ablation study results in Table 4.", "weaknesses": "1. The proposed method is compared with SPARF only in the setting of using pose from different pose estimation baselines. However, it would be more convincing to also present the results using the same setting of SPARF, which adds noise into the GT camera pose. This will be a direct comparison with SPARF’s original results reported in their paper.\n2. The proposed method is compared with LEAP for 3D reconstruction results. However, the comparison is a bit unfair since LEAP does not require any initial camera poses. \n3. The description of how to effectively detect the outliers (line 212 - line 214) is not very clear. Similarly, the procedure of how to correct the outlier poses (line 223 - line 225) is not very clear either. How the MSE and LPIPS are computed and compared since there is no correspondence?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The proposed method is compared with SPARF only in the setting of using pose from different pose estimation baselines. However, it would be more convincing to also present the results using the same setting of SPARF, which adds noise into the GT camera pose. This will be a direct comparison with SPARF’s original results reported in their paper.\n\nThanks for the advice! We experimented with the suggested setting using eight images, where Gaussian noise is added to ground truth camera poses (so the initial poses have a rotation error of about 9.71°). A comparison between the proposed method and SPARF on  pose accuracy is presented below:\n\n| Method         | Avg Rot Error | Rot@5° | Rot@15° | CC@10% | Improvement  Rate |\n| :------------- | :------------ | :----- | :------ | :----- | :------------ |\n| Initialization | 9.71          | 11.9   | 89.1    | 56.3   | /             |\n| w/ SPARF       | 14.47         | 43.6   | 73.5    | 66.5   | 0.53          |\n| w/ Ours        | 3.63          | 82.4   | 96.9    | 94.4   | 0.92          |\n\nWhile our method yields consistent improvements over all metrics, we observed that SPARF can sometimes degrade performance compared to the initialization.  As SPARF leverages correspondence, its pose optimization is ineffective (or even diverges) when no reliable correspondence (or false correspondence) is found. The results are consistent with our experiments on DUSt3R poses, which we include in our General Response. We refer the reviewer to the top-level response for more details. Please note that the experiments in the original SPARF paper were done using image sets with significant overlap (e.g.  DTU dataset where all images observe the same aspect of an object).\n\n> The proposed method is compared with LEAP for 3D reconstruction results. However, the comparison is a bit unfair since LEAP does not require any initial camera poses.\n\nFor an additional baseline, please see our general response where we additionally compare to SPARF on novel-view synthesis. We would also be happy to report any additional comparisons the reviewer suggests.\n\n> The description of how to effectively detect the outliers (line 212 - line 214) is not very clear. Similarly, the procedure of how to correct the outlier poses (line 223 - line 225) is not very clear either. How the MSE and LPIPS are computed and compared since there is no correspondence?\n\nWe apologize for being unclear in our submission and will update the text based on the reviewer's feedback. In outlier correction, we do render-and-compare for each identified outlier, where we resue the reconstructed 3D from only the inliers (obtained in *Iterative Outlier Identification*). By sampling pose candidates, we can render images that can be compared with the target image (of the outlier) with metrics such as MSE and LPIPS. Therefore, correspondence is not required in the process. We included detailed explanations for outlier removal and correction in our General Response, and we hope that this will resolve the reviewer's concern. \n\n> The proposed method is evaluated on the NAVI dataset. It seems that the dataset is quite simple as shown in Fig. 3 and Fig. 4. The reviewer is wondering about the performance of the proposed method on more complex scenes?\n\nWe primarily tested on NAVI because it provides ground truth 3D meshes that are unavailable in other datasets. This enables the evaluation of 3D metrics such as F1 for our 3D reconstruction. In addition, NAVI offers highly accurate object masks and high-quality sparse-view image sequences. However, we include qualitative results of our approach on more challenging scenarios in **Fig. 1** of our rebuttal PDF, where we show two self-captures and four challenging instances from the held-out set of CO3D. These instances have more complex textures and shapes than the NAVI samples, which verifies the proposed approach's generalization ability.\n\n> The reviewer is wondering about the separate ablation results on the outlier removal and correction.\n\nWe would like to thank the reviewer for mentioning this ablation. We followed our setup in Table 4 of the main paper, but we only do *Iterative Outlier Identification* without the following correction stage.\n\n| Method                          |   Rot@5°  |   Rot@15°  | CC@10%  | PSNR  | LPIPS  | F1    |\n|---------------------------------|-----------|------------|---------|-------|--------|-------|\n|   Ray Diffusion                 |   13.5    | 73.5       | 38.3    | /     |   /    |   /   |\n|   Ours (w/o Outlier Correction) | 47.8      | 86.6       | 75.2    | 14.45 | 0.2494 | 65.41 |\n| Ours (Full System)              | 60.3      | 88.2       | 80.4    | 15.30 | 0.2304 | 68.19 |\n\nCompared to the baseline Ray Diffusion numbers, this experiment setup benefits from iterative pose-3D co-optimization, improving the base pose accuracy by a notable margin. Moreover, the results verify that an additional outlier correction is necessary for higher pose accuracy and high-fidelity novel views.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f8pa41zXMR", "reviewer": "Reviewer_XaBT", "summary": "The paper introduces a novel optimization-based method for sparse-view 3D reconstruction from unposed images. The method uses off-the-shelf pose estimator to get pose initialization, then it uses rendering loss and generative priors to optimize the pose and 3D reconstruction. In detail, the generative priors involve a multi-view SDS loss on generated novel views using Zero123. The method demonstrates satisfying results on the evaluation data, and the ablation study shows the effectiveness of each proposed technique.", "questions": "- The introduction spends a lot space discussing the chicken-and-egg problem of pose estimation and reconstruction. However, I don't think it is quite related to the technical part, as the proposed method still need pose initialization using off-the-shelf methods. The method doesn't provide a novel perspective regarding how to solve the chicken-and-egg problem, and using pose initialization is quite common in prior works, e.g., SPARF, FORGE, and FvOR or even traditional SfM methods. Why the authors want to emphasize this?\n- Is it possible to evaluate the outlier removal method? For example, the authors can evaluate the correlation between the removal and the pose error. If the proposed method works well, they should have strong correlations. Moreover, it will be good to provide any statistics on the outlier removal method, e.g., how many images are removed in average.", "limitations": "Please see weaknesses and questions.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Good performance. The reconstruction quality and pose estimation accuracy are satisfying.\n- The paper is well-written and is easy to follow.\n- The idea of rejecting images with large pose error is interesting.\n- The technical part of the paper is solid.", "weaknesses": "- Missing baseline. For the reconstruction methods, the only baseline is LEAP, which is a feedforward method. In contrast, the proposed method is an optimization-based method, which introduces pose-processing to estimated poses. I would suggest adding baseline of SPARF [1] and using the same pose initialization. Moreover, why not comparing with UpFusion?\n- Unknown inference speed. Will the joint optimization of pose and shape be slow? Could you provide a analysis of inference time?\n- Related work. One related work is iFusion [2], which uses generative priors for pose estimation and is very relevant to the philosophy of the proposed method. Another related work is FORGE [3], which introduces pose optimization for sparse view reconstruction. Moreover, the authors should discuss the prior sparse-view reconstruction from unposed images works with more details, the authors should provide more comparison and contrast with prior work. The current discussion is too short (Line 90-92).\n- Ablation study. The ablation study is performed with the Ray Diffusion pose initialization. How will it look like using Dust3r initialization? This is important as the ablation should be performed with the best base model.\n\n\n[1] Truong, Prune, et al. \"Sparf: Neural radiance fields from sparse and noisy poses.\" CVPR 2023.\n[2] Wu, Chin-Hsuan et al. “iFusion: Inverting Diffusion for Pose-Free Reconstruction from Sparse Views.” ArXiv 2023.\n[3] Jiang, Hanwen et al. “Few-View Object Reconstruction with Unknown Categories and Camera Poses.” 3DV 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Missing baseline.\n\nWe would like to thank the reviewer for the suggestion on baselines. We include the comparison between the proposed approach and SPARF in General Response. In additions, we also compare our method with UpFusion on novel view synthesis:\n\n| Method   | Init Cam Pose | (N=6) PSNR | LPIPS  | (N=8) PSNR | LPIPS  |\n| :------- | :------------ | :--------- | :----- | :--------- | :----- |\n| LEAP     | /             | 12.84      | 0.2918 | 12.93      | 0.2902 |\n| UpFusion | /             | 13.30      | 0.2747 | 13.27      | 0.2744 |\n| Ours     | Ray Diffusion | 13.63      | 0.2697 | 15.30      | 0.2304 |\n| Ours     | DUSt3R        | 15.56      | 0.2173 | 17.03      | 0.1870 |\n\nUpFusion performs slightly better than LEAP, while our approach constantly outperforms both pose-free baselines with two different pose initialization and view numbers. Our method improves consistently by leveraging more precise initial camera poses and more observed input images. However, LEAP and UpFusion show little performance increase when more views are available due to their geometry unawareness. We also include a qualitative comparison between our approach and UpFusion in **Fig. 4** of the rebuttal PDF file.\n\n> Unknown inference speed.\n\nWe apologize for the ambiguity in our earlier version. Our shape-pose optimization is fast, benefiting from the efficient Gaussian Splatting compared to NeRF baselines, and on average our inference time is about 9 minutes, whereas SPARF may take more than 10 hours to train a full model. We discuss inference time more thoroughly in our General Response and will revise our paper according to the review's feedback.\n\n> Related work.\n\nThanks for mentioning related work! We will incorporate our discussion into our updated version. \n\n+ iFusion leverages diffusion priors from the diffusion model (i.e., Zero-123) for pose estimation in a similar manner to ID-Pose where relative camera poses are parameterized as conditioning to the diffusion model and are refined via gradient descent from diffusion noise prediction. Compared to iFusion, we also leverage diffusion priors from Zero-123 but we optimize camera poses directly via photometric error from differentiable rendering. In this process, we explicitly form a 3D representation (3DGS) for optimization.\n+ FORGE (which is earlier work from the authors of LEAP) uses the synergy of pose estimation and 3D reconstruction by inferring 3D features that are shared by both tasks. Our work shares a similar high-level idea in that we make use of the estimated camera poses for 3D reconstruction, and the reconstructed 3D, in turn, is exploited to help refine camera poses and identify outliers. By doing this, we also hope to make both sub-tasks to benefit from each other.\n\nIn our final version, we will add detailed discussions about pose-free approaches (e.g., LEAP, UpFusion) in our Related Work section.\n\n> Ablation study.\n\nWe thank the reviewer for suggesting an ablation study on DUSt3R, and present the results below:\n\n|                                 | **Rot@5°** | **Rot@15°** | **CC@10%** | PSNR  | LPIPS  | **F1** |\n| ------------------------------- | ---------- | ----------- | ---------- | ----- | ------ | ----------- |\n| DUSt3R                          | 52.3       | 93.8        | 82.2       | /     | /      |             |\n| \\+ Pose-3D Co-opt. (w/o SDS)    | 80.9       | 95.8        | 91.6       | 16.79 | 0.2072 | 64.70       |\n| \\+ SDS (vanilla Zero123)        | 78.7       | 95.3        | 90.2       | 16.38 | 0.2097 | 69.45       |\n| \\+ SDS (Our 6-DoF Zero123)      | 81.1       | 95.9        | 91.0       | 16.75 | 0.1941 | 75.04       |\n| \\+ Outlier Removal & Correction | 83.7       | 96.2        | 93.5       | 17.03 | 0.1870 | 75.80       |\n\n> The introduction spends a lot space discussing the chicken-and-egg problem of pose estimation and reconstruction... Why the authors want to emphasize this?\n\nWe agree with the reviewer that classical SfM methods did indeed tackle this chicken-and-egg problem. Our introduction was instead motivated by ignorance of the chicken-and-egg nature in the more recent learning-based sparse-view reconstruction methods. For example, approaches like LEAP/UpFusion seek to side-step the pose estimation task altogether. On the other hand, methods that explore generative-prior-based reconstruction (e.g. ReconFusion/SparseFusion) assume perfect poses. Even methods that jointly optimize 3D and pose (e.g. BARF/SPARF/FvOR) are designed to use noisy ground-truth poses or near-perfect initial poses (as opposed to ones output by real systems with possibly large errors). \n\nWe agree this emphasis maybe obvious to any reader familiar with the rich history of the field, but believe this is useful for readers only familiar with a recent history.  That said, we would be happy to revise the introduction if the reviewer suggests.\n\n> Statistics of Outlier Removal.\n\nWe again thank the reviewer for this advice. With the same setting as Table 4 in our main paper, we investigated the relationship between the identified outlier numbers and the corresponding sequence numbers in our dataset (174 in total) and their initial average rotation error.\n\n| # of Outliers            | 0     | 1     | 2     | 3     | 4     |\n|--------------------------|-------|-------|-------|-------|-------|\n| Init Rot Error | 14.19 | 13.09 | 20.52 | 17.16 | 33.17 |\n|   # of Sequences         | 56    | 82    | 28    | 7     | 1     |\n\nIn general, sequences with higher rotation errors tend to be identified with more outliers. In this setup, our method identified ~0.94 outliers per sequence. We also computed the rotation error at an image level for outliers and inliers with the average over all samples as reference:\n\nAll: 14.92°\nOutlier: 19.75°\nInlier: 13.53°\n\nThe results indicate that the outlier we found indeed has a higher error than others, verifying the effectiveness of our outlier identification approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We appreciate the reviewers' insightful comments and valuable feedback. We are glad that the reviewers appreciated the results, the practicality of the setup, and found the paper well written. In this response, we address some of the common points raised by the reviewers (additional NVS comparison and clarifications on inference), and address the more specific comments via separate responses to each review.

### Comparison with SPARF on Novel View Synthesis (NVS)

In addition to our submission comparing to SPARF for pose estimation, reviewers ePgG and XaBT recommended comparison to SPARF via NVS metrics. We report these below, using DUSt3R pose (N=6, 8) as initialization. Due to SPARF's long training time (about 10 hours per instance), we could only include 70 sequences (2 sequences per object) in these two experiments. 

| N = 6     | Avg Rot Error | Rot@5°  | Rot@15° | CC@10%  | Improvement Rate | PSNR     | LPIPS     | *PSNR    | *LPIPS    |
|-----------|---------------|---------|---------|---------|--------------|----------|-----------|----------|-----------|
|   DUSt3R  |   7.90        |   48.5  |   93.3  |   80.2  |   /          |   /      |   /       |   /      |   /       |
|   SPARF   |   17.07       |   47.9  |   68.7  |   67.6  |   0.41       |   12.80  |   0.3201  |   15.30  |   0.2478  |
|   Ours    |   5.82        |   73.9  |   94.5  |   92.1  |   0.80       |   15.52  |   0.2179  |   16.23  |   0.1844  |

| N = 8     | Avg Rot Error | Rot@5°  | Rot@15° | CC@10%  | Improvement Rate | PSNR     | LPIPS     | *PSNR    | *LPIPS    |
|-----------|---------------|---------|---------|---------|--------------|----------|-----------|----------|-----------|
|   DUSt3R  |   8.71        |   51.6  |   92.9  |   79.5  |   /          |   /      |   /       |   /      |   /       |
|   SPARF   |   18.17       |   57.3  |   75.7  |   68.6  |   0.54       |   13.42  |   0.3059  |   15.44  |   0.2584  |
|   Ours    |   5.99        |   81.7  |   95.3  |   92.5  |   0.93       |   17.02  |   0.1874  |   17.48  |   0.1695  |

The results indicate that our method outperforms SPARF in pose accuracy and novel view quality. In fact, we found that SPARF can often make the poses worse compared to the (relatively accurate) DUSt3R initialization (measured via `Improvement Rate` that indicates the percentage of sequences with reduced pose error).  We found that this is because the correspondences leveraged by SPARF in its optimization are not robust and are susceptible to false matches -- please see **Fig. 3** of the rebuttal PDF for an example.

As an attempt to compare novel view quality despite the difference in pose accuracy, we report *PSNR and *LPIPS, which are measured *only on the sequences where SPARF improves pose accuracy* and find that even in these, our approach outperforms it. We also observed that while SPARF works well on novel views close to the input, floaters constantly appear with significant viewpoint changes. In contrast, our generative prior leads to a more consistent 3D representation.

## Inference Time and Details
### (1) Iterative Outlier Identification
Given N input images, our method first reconstructs 3D using the proposed MV-DG approach. The image with the largest reprojection error is considered as a candidate for being an outlier, and we perform another reconstruction after removing it (using remaining N-1 inputs). This candidate is considered a valid outlier if the average reprojection error in others views reduces significantly (we use 0.05 in LPIPS as a threshold). If it is an outlier, we remove it and repeat the process to identify outliers among the remaining N-1 images, stopping the process when no outliers remain. In general, if O outliers are detected, we need to perform O+2 reconstructions.

### (2) Outlier Correction via Render-and-compare
Any identified outlier in (1) requires pose correction. We do this by combining discrete search and continuous optimization. Specifically, we reuse the reconstructed 3D from (1) with only the inliers involved. For each outlier, we sample pose candidates evenly on a sphere around and targeted at the reconstructed 3D. Next, we optimize these pose candidates via gradient descent from the photometric error given our reconstructed 3D. To find out the most likely target pose, we select the optimized pose candidate whose rendering has the lowest discrepancy with the target input image, which is measured by both MSE and LPIPS. As MV-DG outputs a 3D mesh, rendering and propagating gradients to camera poses are fast. Each outlier takes roughly one minute to do pose correction. 

If any outliers were corrected, another reconstruction is performed using the updated poses.

### (3) Inference time:
For N=8 images using a single RTX A5000, one reconstruction with MV-DG takes about 2 minutes to complete, and the 'render-and-compare' for each outlier takes around a minute. Our full pipeline (using RayDiffusion initialization) detected an average of 0.94 outliers per sequence, resulting in an inference time of around 9 minutes.

We will update the paper to include these details more clearly as well as release our implementation for reproducibility.

Meta-review: The paper introduces a novel optimization-based method for sparse-view 3D reconstruction from unposed images. The method uses an off-the-shelf pose estimator for pose initialization, then optimizes both the pose and 3D reconstruction using rendering loss and generative priors. The method demonstrates satisfactory results on the evaluation data, and the ablation study highlights the effectiveness of each proposed technique. The initial ratings were three borderline accepts and one reject. The key concerns included a lack of novelty, unfair comparisons in terms of 3D reconstruction, increased inference time, and missing related work. In the rebuttal, additional 3D reconstruction comparisons were provided, and discussions on novelty, new related work, and inference time were included. The final ratings were four weak accepts. Given the consensus, the AC recommends acceptance.
Decision: Accept (poster)