PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce
  Lidar
Authors: name: Tzofi Klinghoffer; name: Xiaoyu Xiang; name: Siddharth Somasundaram; name: Yuchen Fan; name: Christian Richardt; name: Ramesh Raskar; name: Rakesh Ranjan
Published: 2023-12-21 18:59:53

Abstract: 3D reconstruction from a single-view is challenging because of the ambiguity
from monocular cues and lack of information about occluded regions. Neural
radiance fields (NeRF), while popular for view synthesis and 3D reconstruction,
are typically reliant on multi-view images. Existing methods for single-view 3D
reconstruction with NeRF rely on either data priors to hallucinate views of
occluded regions, which may not be physically accurate, or shadows observed by
RGB cameras, which are difficult to detect in ambient light and low albedo
backgrounds. We propose using time-of-flight data captured by a single-photon
avalanche diode to overcome these limitations. Our method models two-bounce
optical paths with NeRF, using lidar transient data for supervision. By
leveraging the advantages of both NeRF and two-bounce light measured by lidar,
we demonstrate that we can reconstruct visible and occluded geometry without
data priors or reliance on controlled ambient lighting or scene albedo. In
addition, we demonstrate improved generalization under practical constraints on
sensor spatial- and temporal-resolution. We believe our method is a promising
direction as single-photon lidars become ubiquitous on consumer devices, such
as phones, tablets, and headsets.