ReconFusion: 3D Reconstruction with Diffusion Priors
Authors: name: Rundi Wu; name: Ben Mildenhall; name: Philipp Henzler; name: Keunhong Park; name: Ruiqi Gao; name: Daniel Watson; name: Pratul P. Srinivasan; name: Dor Verbin; name: Jonathan T. Barron; name: Ben Poole; name: Aleksander Holynski
Published: 2023-12-05 18:59:58

Abstract: 3D reconstruction methods such as Neural Radiance Fields (NeRFs) excel at
rendering photorealistic novel views of complex scenes. However, recovering a
high-quality NeRF typically requires tens to hundreds of input images,
resulting in a time-consuming capture process. We present ReconFusion to
reconstruct real-world scenes using only a few photos. Our approach leverages a
diffusion prior for novel view synthesis, trained on synthetic and multiview
datasets, which regularizes a NeRF-based 3D reconstruction pipeline at novel
camera poses beyond those captured by the set of input images. Our method
synthesizes realistic geometry and texture in underconstrained regions while
preserving the appearance of observed regions. We perform an extensive
evaluation across various real-world datasets, including forward-facing and
360-degree scenes, demonstrating significant performance improvements over
previous few-view NeRF reconstruction approaches.