Title: Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis
Venue: NeurIPS 2024
Accepted: yes
Keywords: computer vision

3D vision

3D reconstruction

camera pose estimation

analysis-by-synthesis

generative model
Abstract: Inferring the 3D structure underlying a set of multi-view images typically requires solving two co-dependent tasks -- accurate 3D reconstruction requires precise camera poses, and predicting camera poses relies on (implicitly or explicitly) modeling the underlying 3D. The classical framework of analysis by synthesis casts this inference as a joint optimization seeking to explain the observed pixels, and recent instantiations learn expressive 3D representations (e.g., Neural Fields) with gradient-descent-based pose refinement of initial pose estimates. However, given a sparse set of observed views, the observations may not provide sufficient direct evidence to obtain complete and accurate 3D. Moreover, large errors in pose estimation may not be easily corrected and can further degrade the inferred 3D. To allow robust 3D reconstruction and pose estimation in this challenging setup, we propose SparseAGS, a method that adapts this analysis-by-synthesis approach by: a) including novel-view-synthesis-based generative priors in conjunction with photometric objectives to improve the quality of the inferred 3D, and b) explicitly reasoning about outliers and using a discrete search with a continuous optimization-based strategy to correct them. We validate our framework across real-world and synthetic datasets in combination with several off-the-shelf pose estimation systems as initialization. We find that it significantly improves the base systems' pose accuracy while yielding high-quality 3D reconstructions that outperform the results from current multi-view reconstruction baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel optimization-based method for sparse-view 3D reconstruction from unposed images. The method uses off-the-shelf pose estimator to get pose initialization, then it uses rendering loss and generative priors to optimize the pose and 3D reconstruction. In detail, the generative priors involve a multi-view SDS loss on generated novel views using Zero123. The method demonstrates satisfying results on the evaluation data, and the ablation study shows the effectiveness of each proposed technique.\n\nSTRENGTHS:\n- Good performance. The reconstruction quality and pose estimation accuracy are satisfying.\n- The paper is well-written and is easy to follow.\n- The idea of rejecting images with large pose error is interesting.\n- The technical part of the paper is solid.\n\nWEAKNESSES:\n- Missing baseline. For the reconstruction methods, the only baseline is LEAP, which is a feedforward method. In contrast, the proposed method is an optimization-based method, which introduces pose-processing to estimated poses. I would suggest adding baseline of SPARF [1] and using the same pose initialization. Moreover, why not comparing with UpFusion?\n- Unknown inference speed. Will the joint optimization of pose and shape be slow? Could you provide a analysis of inference time?\n- Related work. One related work is iFusion [2], which uses generative priors for pose estimation and is very relevant to the philosophy of the proposed method. Another related work is FORGE [3], which introduces pose optimization for sparse view reconstruction. Moreover, the authors should discuss the prior sparse-view reconstruction from unposed images works with more details, the authors should provide more comparison and contrast with prior work. The current discussion is too short (Line 90-92).\n- Ablation study. The ablation study is performed with the Ray Diffusion pose initialization. How will it look like using Dust3r initialization? This is important as the ablation should be performed with the best base model.\n\n\n[1] Truong, Prune, et al. \"Sparf: Neural radiance fields from sparse and noisy poses.\" CVPR 2023.\n[2] Wu, Chin-Hsuan et al. “iFusion: Inverting Diffusion for Pose-Free Reconstruction from Sparse Views.” ArXiv 2023.\n[3] Jiang, Hanwen et al. “Few-View Object Reconstruction with Unknown Categories and Camera Poses.” 3DV 2024.\n\nQUESTIONS:\n- The introduction spends a lot space discussing the chicken-and-egg problem of pose estimation and reconstruction. However, I don't think it is quite related to the technical part, as the proposed method still need pose initialization using off-the-shelf methods. The method doesn't provide a novel perspective regarding how to solve the chicken-and-egg problem, and using pose initialization is quite common in prior works, e.g., SPARF, FORGE, and FvOR or even traditional SfM methods. Why the authors want to emphasize this?\n- Is it possible to evaluate the outlier removal method? For example, the authors can evaluate the correlation between the removal and the pose error. If the proposed method works well, they should have strong correlations. Moreover, it will be good to provide any statistics on the outlier removal method, e.g., how many images are removed in average.\n\nLIMITATIONS:\nPlease see weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework for joint 3D reconstruction and pose refinement. Specifically, given estimated camera poses from off-the-shelf models, the proposed method first leverages diffusion priors and rendering loss for 3D reconstruction. The 3D reconstruction is further used to refine the current pose parameters. The 3D reconstruction and pose refinement are conducted in an alternative way. An outlier identification and correction strategy is also introduced to make full use of the given image while mitigating the adverse effect of noisy camera estimations at the same time. Experimental comparison with several pose estimation baselines shows that the proposed method can refine inaccurate pose estimation effectively.\n\nSTRENGTHS:\n1. The paper tackles a practical problem in real-world scenarios, where ground truth camera poses are not always available.\n2. The proposed method is shown to be effective when applying to different pose estimation baselines.\n3. The proposed outlier removal and correction is effective from the ablation study results in Table 4.\n\nWEAKNESSES:\n1. The proposed method is compared with SPARF only in the setting of using pose from different pose estimation baselines. However, it would be more convincing to also present the results using the same setting of SPARF, which adds noise into the GT camera pose. This will be a direct comparison with SPARF’s original results reported in their paper.\n2. The proposed method is compared with LEAP for 3D reconstruction results. However, the comparison is a bit unfair since LEAP does not require any initial camera poses. \n3. The description of how to effectively detect the outliers (line 212 - line 214) is not very clear. Similarly, the procedure of how to correct the outlier poses (line 223 - line 225) is not very clear either. How the MSE and LPIPS are computed and compared since there is no correspondence?\n\nQUESTIONS:\n1. The proposed method is evaluated on the NAVI dataset. It seems that the dataset is quite simple as shown in Fig. 3 and Fig. 4. The reviewer is wondering about the performance of the proposed method on more complex scenes?\n2. The reviewer is wondering about the separate ablation results on the outlier removal and correction.\n\nLIMITATIONS:\nLimitations are addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for the joint reconstruction of camera poses and 3D objects given sparse input views. The core idea is to use a pose-conditioned diffusion model (Zero-123) as a prior, impose the SDS loss, and jointly optimize the poses and objects, similar to the approach in ID-pose. To improve the robustness and quality of the optimization, the authors made several modifications: (1) Using a 6 DoF pose-conditioned diffusion model instead of a 3 DoF model. (2) Adding strategies for outlier detection and correction. (Although somewhat empirical, it proves effective.)\n\nThis approach requires initial camera poses (from methods such as RelPose++, RayDiffusion, etc.) and is not capable of reconstructing poses from scratch (e.g., purely random camera poses). Experimental results demonstrate that, compared to SPARF and ID-pose, the proposed method achieves better pose estimation quality. Additionally, it provides better object reconstruction in terms of novel view synthesis quality compared to LEAP.\n\nSTRENGTHS:\n(1) The approach is technically sound, and I believe the reported results are reproducible.\n\n(2) The reconstructed results look good and represent the state-of-the-art in object-level pose-free reconstruction.\n\n(3) The paper is well-written, making it easy to read and understand.\n\nWEAKNESSES:\n(1) This optimization-based method requires more time compared to a feed-forward model, taking about 5-10 minutes. Additionally, the writing discussing this aspect is somewhat unclear: the paper states, “with increased inference time depending on the number of outliers.” Could this statement be more specific? How much does the time increase with the number of outliers? The correction of outliers may be time-consuming as it requires dense searches of initial camera poses.\n\n(2) (Minor) The method focuses only on object-level reconstruction, which makes the scope seem narrow.\n\n(3) The authors do not sufficiently discuss experiments in a more “standard” sparse-view setting, such as using 3 or 4 views. The reported experiments use at least 6 views, which is not a particularly small number.\n\nQUESTIONS:\n(1) A related work is lacking in discussion:  Sun, Yujing, et al. \"Extreme Two-View Geometry From Object Poses with Diffusion Models.\" arXiv preprint arXiv:2402.02800 (2024).\n\n(2) Is the testing data included in the training set for fine-tuning the 6-DoF diffusion model?\n\nLIMITATIONS:\nAs discussed in the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method named MV-DreamGaussian for tackling the problem of 3D reconstruction from sparse multi-view inputs. In particular, the paper extends the DreamGaussian work to use multi-view images as the inputs and proposes a scheme to optimize the inaccurate camera poses of the multi-view images.\n\nSTRENGTHS:\n- This paper is well written and I can follow smoothly.\n- The authors proposed a finetuned version of Zero-1-to-3 with 6 DoF camera parametrization which shows an advantage over 3 DoF camera parameterization in the original paper.\n- The proposed pose refinement scheme is novel and very effective according to the authors' experiments compared with SPARF as well as the ablation study which shows that adding the proposed pose refinement improves the pose accuracy and reconstruction quality significantly. The design of the outlier removal based on photometric error ranking and discrete search is empirical but works quite well.\n\nWEAKNESSES:\n- This paper presents very limited novelty in the reconstruction part with a trivial extension to DreamGaussian to use multi-view images, which is already implemented in a public repository [stable-dreamfusion](https://github.com/ashawkey/stable-dreamfusion).\n- The major weakness of the paper is the lack of fair comparisons in terms of the 3D reconstruction. The authors only compared with LEAP for the 3D reconstruction. However, LEAP is a work that **does not require any pose inputs**, whereas the proposed work needs relatively good pose initialization (e.g., Dust3r) and conduct refinement on it. In addition, the underlying 3D representation is different, too: LEAP uses NeRF while the proposed work uses 3D Gaussian. I'm confused as to why the authors did not compare with SPARF for the reconstruction quality too since SPARF shares the same input setup as the proposed work. Besides, the very recent work DMV3D would also be a good method to compare with.\n\nQUESTIONS:\n- I'm quite curious what the reconstruction quality the method can achieve without 3D generative prior but with the proposed refinement. Namely the combination of (1) and (4) in Table 4.\n- How are the poses used for generative prior sampled in addition to the input views?\n- How are the thresholds for pose outlier removal tuned?\n\nLIMITATIONS:\nThe authors have discussed the limitations of the paper and I generally agree with them.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}