Object-Centric Neural Scene Rendering

1 INTRODUCTION . Synthesizing images of dynamic scenes is an important problem in computer vision and graphics , with applications in AR/VR and robotics ( Savva et al. , 2019 ; Xia et al. , 2020 ) . For synthetic scenes , a user typically designs a set of 3D objects separately , then composes them into scenes to be rendered with specified camera , material , and lighting parameters . While this traditional graphics approach allows for flexible scene compositions , it requires detailed models of geometry , lighting , materials , and cameras , which can be difficult to obtain for real-world scenes . To render real-world scenes without computer graphics models , recent works have explored using neural implicit methods ( Lombardi et al. , 2019 ; Sitzmann et al. , 2019a ; b ) . Most notably , Mildenhall et al . ( 2020 ) proposed neural radiance fields ( NeRF ) , which achieve photorealistic quality by implicitly modeling the volumetric density and directional emitted radiance of a scene . However , as shown in Figure 1 , NeRF can not generalize beyond the scene it was trained on , because it assumes static scenes and fixed illumination and learns a radiance field , which estimates only the resulting radiance along a ray after all light transport has occurred in a scene . Thus , for dynamic scenes where lights and objects can move , a separate NeRF-based model is needed for each new scene configuration . To address this issue , we propose Object-Centric Neural Scattering Functions ( OSFs ) to synthesize dynamic scenes of objects learned from 2D images ( Figure 2 ) . We represent each object as a learned 7D scattering function with inputs ( x , y , z , φi , θi , φo , θo ) , where ( x , y , z ) is the spatial location , ( φi , θi ) is the incoming light direction , and ( φo , θo ) is the outgoing light direction . The function outputs the volumetric density as well as the fraction of light arriving from direction ( φi , θi ) that scatters in outgoing direction ( φo , θo ) . Each OSF models all light bounces ( reflections ) and occlusions ( shadows ) within an object . Since each object ’ s scattering function is a radiance transfer function rather than a radiance field , it is intrinsic to the object ( independent of the scene it is in ) and can be reused across different object placements and lighting conditions without retraining . We emphasize that because NeRFs are radiance fields , they can not be composed , and can not generalize beyond one scene . In contrast , we can render infinitely many scenes . We can build a library of OSFs trained independently for different objects to be composed into scenes with different object placements , camera , and lighting . To model light transport between objects , we integrate our implicit object functions with volumetric path tracing . Like NeRF , we evaluate the radiance and volumetric density at 5D samples along every primary ray to the camera and composite them with an over operator . However , unlike NeRF , we estimate the radiance for each 5D sample by integrating our 7D OSF across the 2D sphere of incoming light directions . We estimate the integral with Monte Carlo path tracing ( Kajiya , 1986 ) to reproduce shadows and indirect illumination effects . Our key idea is to decompose the rendering problem into ( i ) a learned component ( per-object asset creation ) , and ( ii ) a non-learned component ( per-scene path tracing ) . The learned component models intra-object light transport ( e.g. , bounces from the seat of a chair to the back of the chair ) . The non-learned component handles inter-object light transport ( e.g. , bounces from a wall to a chair ) . Together , they model the full rendering equation ( Kajiya , 1986 ) ( except for occluders or light sources that intrude the object ’ s convex hull ( Sloan et al. , 2002 ) ) . Since only the inter-object light transport changes as objects and lights move , no re-training is required for different scene arrangements . Experimental results indicate that our method is capable of rendering images with novel scene compositions and lighting conditions better than alternative learned approaches . In summary , our contributions are : 1 . Learning Object-Centric Neural Scattering Functions ( OSFs ) that model intra-object light transport implicitly using a lighting- and view-dependent neural network . 2 . Integrating implicitly learned object scattering functions with volumetric path tracing to model inter-object light transport . 3 . A rendering algorithm that enables rendering scenes with moving objects , lights and cameras , using implicit functions . 2 RELATED WORK . Classical object-centric representations . Factoring light transport into intra- and inter-object illumination has a long history in traditional computer graphics ( Dutre et al. , 2018 ) . In most cases , the motivation is to improve rendering efficiency by approximating intra-object lighting factors with simple transfer functions ( e.g. , linear ) for simple radiance fields ( e.g. , spherical harmonics ) derived from from computer graphics models , as in precomputed radiance transfer ( PRT ) ( Sloan et al. , 2002 ) , ambient occlusion ( Miller , 1994 ) , or virtual walls ( Arnaldi et al. , 1994 ) . In other cases , the motivation is to insert captured , real-world radiance fields into synthetic scenes , as in Light Field Transfer ( Cossairt et al. , 2008 ) . These methods generally store the radiance field for objects in a discrete representation ( e.g. , a sampled 2D or 4D grid ) . As a result , they can not reproduce accurate inter-object light transport , especially for objects with intersecting bounding volumes . In contrast , we focus on learning radiance transfer from images in order to model complex real-world scattering accurately , and utilize volumetric rendering techniques to account for inter-object illumination . Novel view synthesis . Traditional methods for synthesizing novel views of a scene from captured images include using Structure-From-Motion ( Hartley & Zisserman , 2003 ) and bundle adjustment ( Triggs et al. , 1999 ) to predict a sparse point cloud and camera parameters of the scene . More recently , a number of learning-based novel view synthesis methods have been presented but require 3D geometry as inputs ( Hedman et al. , 2018 ; Thies et al. , 2019 ; Meshry et al. , 2019 ; Aliev et al. , 2020 ; Martin-Brualla et al. , 2018 ) . Others use multiplane images as proxies for novel view synthesis , but their viewing ranges are limited to interpolated input views ( Flynn et al. , 2016 ; Zhou et al. , 2018 ; Srinivasan et al. , 2019 ; Mildenhall et al. , 2019 ) . Some works represent scenes as coarse voxel grids and use a CNN-based decoder for differentiable rendering , but lack view consistency due to the use of 2D convolutional kernels ( Nguyen-Phuoc et al. , 2018 ; 2019 ; 2020 ) . Recently , volume rendering approaches have been used to render scenes represented as voxel grids that are more view-consistent ( Lombardi et al. , 2019 ; Sitzmann et al. , 2019a ) . However , the rendering resolution of these methods are limited by the time and computational complexity of discretely sampled volumes . To address this issue , Neural Radiance Fields ( NeRF ) ( Mildenhall et al. , 2020 ) directly optimizes a continuous radiance field representation using a multi-layer perceptron . This allows synthesizing novel views of realistic images at an unprecedented level of fidelity . To make NeRF more efficient , Neural Sparse Voxel Fields ( Liu et al. , 2020 ) have been proposed as a sparse voxel octree variant of NeRF and demonstrate the ease of composing learned NeRFs with their voxel representation . See ( Dellaert & Yen-Chen , 2020 ) for survey . While these implicit methods produce high-quality novel views of a scene , their models assume a static scene with fixed illumination . Our method enables synthesizing dynamic scenes with novel viewpoint , lighting , and object configurations . Relighting . Learning-based methods that relight images without explicit geometric reasoning have been proposed , but lack the ability to recover hard shadows ( Sun et al. , 2019 ; Xu et al. , 2018 ; Zhou et al. , 2019 ) . Other works use geometric representations that facilitate shadowing computation , but require 3D geometry as input ( Philip et al. , 2019 ; Zhang et al. , 2021 ; Oechsle et al. , 2020 ; Rematas & Ferrari , 2020 ) . Deep Reflectance Volumes ( Bi et al. , 2020b ) reconstructs a voxelized representation of a scene and predict per-voxel BRDFs , but the fixed resolution of voxel grids limits the quality in the rendered images . Similarly , Neural Reflectance Fields ( Bi et al. , 2020a ) predicts the parameters of a BRDF model , but demonstrate higher fidelity rendering by learning a continuous scene representation . However , Neural Reflectance Fields focuses on relighting single objects , and requires manual specification of the BRDF model . Parametric BRDF models are unable to handle complex scattering functions , including real-world scattering phenomena that are difficult to model . In contrast , our method is capable of learning all scattering functions , and can render multiple objects in dynamic scenes . 3 PRELIMINARIES . 3.1 VOLUME RENDERING . To render an image of a scene with arbitrary camera parameters , camera rays are sent into the scene , through each pixel on the image plane . The expected color of each pixel is computed as the radiance along each camera ray . Volume rendering is an approach for computing the radiance traveling along rays traced in a volume . Let r ( t ) = x0 + ωot be a point along a ray r with origin x0 and direction ωo , where t ∈ R is a 1D location along the ray , and the o in ωo denotes “ outgoing ” direction . For our purposes , we assume non-emissive and non-absorptive volumes . From Novák et al . ( 2018 ) , the volume rendering equation to compute the radiance L ( x0 , ωo ) of the ray is defined as : L ( x0 , ωo ) = ∫ tf tn τ ( t ) σ ( r ( t ) ) Ls ( r ( t ) , ωo ) dt , where τ ( t ) = exp ( − ∫ t tn σ ( r ( u ) ) du ) , ( 1 ) where tn and tf are near and far integration bounds , σ ( r ( t ) ) denotes the volume density of point r ( t ) , and τ ( t ) denotes the accumulated transmittance from tn to t. The termLs ( r ( t ) , ωo ) is the light scattered at point r ( t ) along direction ωo , defined as the integral over all incoming light directions : Ls ( x , ωo ) = ∫ S L ( x , ωl ) fp ( x , ωl , ωo ) dωl , ( 2 ) where S is a unit sphere and fp is a phase function that evaluates the fraction of light incoming from direction ωl at a point x that scatters out in direction ωo . In NeRF , Mildenhall et al . ( 2020 ) assume fixed illumination and do not consider any form of Equation 2 . We consider a more general form of the volume rendering equation that explicitly models light paths within and between objects . This is important for dynamic scenes , where lighting and objects can move with respect to one another .

Summary: The paper proposes a decomposition of a 3D scene into object-level neural radiance fields. This allows to apply rigid transforms to each object independently and to rearrange the scene. Crucially, light transport is modelled, which turns the radiance fields into scattering functions. Direct lighting and shadowing, as well as indirect illumination from several light bounces are taken into account, such that the illumination of the modified scene looks correct. As long as the scattering fields are trained on a sufficient number of light positions, the lights can also be moved around at test time.