NeRF-Casting: Improved View-Dependent Appearance with Consistent
  Reflections
Authors: name: Dor Verbin; name: Pratul P. Srinivasan; name: Peter Hedman; name: Ben Mildenhall; name: Benjamin Attal; name: Richard Szeliski; name: Jonathan T. Barron
Published: 2024-05-23 17:59:57

Abstract: Neural Radiance Fields (NeRFs) typically struggle to reconstruct and render
highly specular objects, whose appearance varies quickly with changes in
viewpoint. Recent works have improved NeRF's ability to render detailed
specular appearance of distant environment illumination, but are unable to
synthesize consistent reflections of closer content. Moreover, these techniques
rely on large computationally-expensive neural networks to model outgoing
radiance, which severely limits optimization and rendering speed. We address
these issues with an approach based on ray tracing: instead of querying an
expensive neural network for the outgoing view-dependent radiance at points
along each camera ray, our model casts reflection rays from these points and
traces them through the NeRF representation to render feature vectors which are
decoded into color using a small inexpensive network. We demonstrate that our
model outperforms prior methods for view synthesis of scenes containing shiny
objects, and that it is the only existing NeRF method that can synthesize
photorealistic specular appearance and reflections in real-world scenes, while
requiring comparable optimization time to current state-of-the-art view
synthesis models.