UniSDF: Unifying Neural Representations for High-Fidelity 3D
  Reconstruction of Complex Scenes with Reflections
Authors: name: Fangjinhua Wang; name: Marie-Julie Rakotosaona; name: Michael Niemeyer; name: Richard Szeliski; name: Marc Pollefeys; name: Federico Tombari
Published: 2023-12-20 18:59:42

Abstract: Neural 3D scene representations have shown great potential for 3D
reconstruction from 2D images. However, reconstructing real-world captures of
complex scenes still remains a challenge. Existing generic 3D reconstruction
methods often struggle to represent fine geometric details and do not
adequately model reflective surfaces of large-scale scenes. Techniques that
explicitly focus on reflective surfaces can model complex and detailed
reflections by exploiting better reflection parameterizations. However, we
observe that these methods are often not robust in real unbounded scenarios
where non-reflective as well as reflective components are present. In this
work, we propose UniSDF, a general purpose 3D reconstruction method that can
reconstruct large complex scenes with reflections. We investigate both
view-based as well as reflection-based color prediction parameterization
techniques and find that explicitly blending these representations in 3D space
enables reconstruction of surfaces that are more geometrically accurate,
especially for reflective surfaces. We further combine this representation with
a multi-resolution grid backbone that is trained in a coarse-to-fine manner,
enabling faster reconstructions than prior methods. Extensive experiments on
object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF
360 and Ref-NeRF real demonstrate that our method is able to robustly
reconstruct complex large-scale scenes with fine details and reflective
surfaces. Please see our project page at
https://fangjinhuawang.github.io/UniSDF.