Adaptive Shells for Efficient Neural Radiance Field Rendering
Authors: name: Zian Wang; name: Tianchang Shen; name: Merlin Nimier-David; name: Nicholas Sharp; name: Jun Gao; name: Alexander Keller; name: Sanja Fidler; name: Thomas Müller; name: Zan Gojcic
Published: 2023-11-16 18:58:55

Abstract: Neural radiance fields achieve unprecedented quality for novel view
synthesis, but their volumetric formulation remains expensive, requiring a huge
number of samples to render high-resolution images. Volumetric encodings are
essential to represent fuzzy geometry such as foliage and hair, and they are
well-suited for stochastic optimization. Yet, many scenes ultimately consist
largely of solid surfaces which can be accurately rendered by a single sample
per pixel. Based on this insight, we propose a neural radiance formulation that
smoothly transitions between volumetric- and surface-based rendering, greatly
accelerating rendering speed and even improving visual fidelity. Our method
constructs an explicit mesh envelope which spatially bounds a neural volumetric
representation. In solid regions, the envelope nearly converges to a surface
and can often be rendered with a single sample. To this end, we generalize the
NeuS formulation with a learned spatially-varying kernel size which encodes the
spread of the density, fitting a wide kernel to volume-like regions and a tight
kernel to surface-like regions. We then extract an explicit mesh of a narrow
band around the surface, with width determined by the kernel size, and
fine-tune the radiance field within this band. At inference time, we cast rays
against the mesh and evaluate the radiance field only within the enclosed
region, greatly reducing the number of samples required. Experiments show that
our approach enables efficient rendering at very high fidelity. We also
demonstrate that the extracted envelope enables downstream applications such as
animation and simulation.