Object-Centric Neural Scene Rendering

1 INTRODUCTION . Synthesizing images of dynamic scenes is an important problem in computer vision and graphics , with applications in AR/VR and robotics ( Savva et al. , 2019 ; Xia et al. , 2020 ) . For synthetic scenes , a user typically designs a set of 3D objects separately , then composes them into scenes to be rendered with specified camera , material , and lighting parameters . While this traditional graphics approach allows for flexible scene compositions , it requires detailed models of geometry , lighting , materials , and cameras , which can be difficult to obtain for real-world scenes . To render real-world scenes without computer graphics models , recent works have explored using neural implicit methods ( Lombardi et al. , 2019 ; Sitzmann et al. , 2019a ; b ) . Most notably , Mildenhall et al . ( 2020 ) proposed neural radiance fields ( NeRF ) , which achieve photorealistic quality by implicitly modeling the volumetric density and directional emitted radiance of a scene . However , as shown in Figure 1 , NeRF can not generalize beyond the scene it was trained on , because it assumes static scenes and fixed illumination and learns a radiance field , which estimates only the resulting radiance along a ray after all light transport has occurred in a scene . Thus , for dynamic scenes where lights and objects can move , a separate NeRF-based model is needed for each new scene configuration . To address this issue , we propose Object-Centric Neural Scattering Functions ( OSFs ) to synthesize dynamic scenes of objects learned from 2D images ( Figure 2 ) . We represent each object as a learned 7D scattering function with inputs ( x , y , z , φi , θi , φo , θo ) , where ( x , y , z ) is the spatial location , ( φi , θi ) is the incoming light direction , and ( φo , θo ) is the outgoing light direction . The function outputs the volumetric density as well as the fraction of light arriving from direction ( φi , θi ) that scatters in outgoing direction ( φo , θo ) . Each OSF models all light bounces ( reflections ) and occlusions ( shadows ) within an object . Since each object ’ s scattering function is a radiance transfer function rather than a radiance field , it is intrinsic to the object ( independent of the scene it is in ) and can be reused across different object placements and lighting conditions without retraining . We emphasize that because NeRFs are radiance fields , they can not be composed , and can not generalize beyond one scene . In contrast , we can render infinitely many scenes . We can build a library of OSFs trained independently for different objects to be composed into scenes with different object placements , camera , and lighting . To model light transport between objects , we integrate our implicit object functions with volumetric path tracing . Like NeRF , we evaluate the radiance and volumetric density at 5D samples along every primary ray to the camera and composite them with an over operator . However , unlike NeRF , we estimate the radiance for each 5D sample by integrating our 7D OSF across the 2D sphere of incoming light directions . We estimate the integral with Monte Carlo path tracing ( Kajiya , 1986 ) to reproduce shadows and indirect illumination effects . Our key idea is to decompose the rendering problem into ( i ) a learned component ( per-object asset creation ) , and ( ii ) a non-learned component ( per-scene path tracing ) . The learned component models intra-object light transport ( e.g. , bounces from the seat of a chair to the back of the chair ) . The non-learned component handles inter-object light transport ( e.g. , bounces from a wall to a chair ) . Together , they model the full rendering equation ( Kajiya , 1986 ) ( except for occluders or light sources that intrude the object ’ s convex hull ( Sloan et al. , 2002 ) ) . Since only the inter-object light transport changes as objects and lights move , no re-training is required for different scene arrangements . Experimental results indicate that our method is capable of rendering images with novel scene compositions and lighting conditions better than alternative learned approaches . In summary , our contributions are : 1 . Learning Object-Centric Neural Scattering Functions ( OSFs ) that model intra-object light transport implicitly using a lighting- and view-dependent neural network . 2 . Integrating implicitly learned object scattering functions with volumetric path tracing to model inter-object light transport . 3 . A rendering algorithm that enables rendering scenes with moving objects , lights and cameras , using implicit functions . 2 RELATED WORK . Classical object-centric representations . Factoring light transport into intra- and inter-object illumination has a long history in traditional computer graphics ( Dutre et al. , 2018 ) . In most cases , the motivation is to improve rendering efficiency by approximating intra-object lighting factors with simple transfer functions ( e.g. , linear ) for simple radiance fields ( e.g. , spherical harmonics ) derived from from computer graphics models , as in precomputed radiance transfer ( PRT ) ( Sloan et al. , 2002 ) , ambient occlusion ( Miller , 1994 ) , or virtual walls ( Arnaldi et al. , 1994 ) . In other cases , the motivation is to insert captured , real-world radiance fields into synthetic scenes , as in Light Field Transfer ( Cossairt et al. , 2008 ) . These methods generally store the radiance field for objects in a discrete representation ( e.g. , a sampled 2D or 4D grid ) . As a result , they can not reproduce accurate inter-object light transport , especially for objects with intersecting bounding volumes . In contrast , we focus on learning radiance transfer from images in order to model complex real-world scattering accurately , and utilize volumetric rendering techniques to account for inter-object illumination . Novel view synthesis . Traditional methods for synthesizing novel views of a scene from captured images include using Structure-From-Motion ( Hartley & Zisserman , 2003 ) and bundle adjustment ( Triggs et al. , 1999 ) to predict a sparse point cloud and camera parameters of the scene . More recently , a number of learning-based novel view synthesis methods have been presented but require 3D geometry as inputs ( Hedman et al. , 2018 ; Thies et al. , 2019 ; Meshry et al. , 2019 ; Aliev et al. , 2020 ; Martin-Brualla et al. , 2018 ) . Others use multiplane images as proxies for novel view synthesis , but their viewing ranges are limited to interpolated input views ( Flynn et al. , 2016 ; Zhou et al. , 2018 ; Srinivasan et al. , 2019 ; Mildenhall et al. , 2019 ) . Some works represent scenes as coarse voxel grids and use a CNN-based decoder for differentiable rendering , but lack view consistency due to the use of 2D convolutional kernels ( Nguyen-Phuoc et al. , 2018 ; 2019 ; 2020 ) . Recently , volume rendering approaches have been used to render scenes represented as voxel grids that are more view-consistent ( Lombardi et al. , 2019 ; Sitzmann et al. , 2019a ) . However , the rendering resolution of these methods are limited by the time and computational complexity of discretely sampled volumes . To address this issue , Neural Radiance Fields ( NeRF ) ( Mildenhall et al. , 2020 ) directly optimizes a continuous radiance field representation using a multi-layer perceptron . This allows synthesizing novel views of realistic images at an unprecedented level of fidelity . To make NeRF more efficient , Neural Sparse Voxel Fields ( Liu et al. , 2020 ) have been proposed as a sparse voxel octree variant of NeRF and demonstrate the ease of composing learned NeRFs with their voxel representation . See ( Dellaert & Yen-Chen , 2020 ) for survey . While these implicit methods produce high-quality novel views of a scene , their models assume a static scene with fixed illumination . Our method enables synthesizing dynamic scenes with novel viewpoint , lighting , and object configurations . Relighting . Learning-based methods that relight images without explicit geometric reasoning have been proposed , but lack the ability to recover hard shadows ( Sun et al. , 2019 ; Xu et al. , 2018 ; Zhou et al. , 2019 ) . Other works use geometric representations that facilitate shadowing computation , but require 3D geometry as input ( Philip et al. , 2019 ; Zhang et al. , 2021 ; Oechsle et al. , 2020 ; Rematas & Ferrari , 2020 ) . Deep Reflectance Volumes ( Bi et al. , 2020b ) reconstructs a voxelized representation of a scene and predict per-voxel BRDFs , but the fixed resolution of voxel grids limits the quality in the rendered images . Similarly , Neural Reflectance Fields ( Bi et al. , 2020a ) predicts the parameters of a BRDF model , but demonstrate higher fidelity rendering by learning a continuous scene representation . However , Neural Reflectance Fields focuses on relighting single objects , and requires manual specification of the BRDF model . Parametric BRDF models are unable to handle complex scattering functions , including real-world scattering phenomena that are difficult to model . In contrast , our method is capable of learning all scattering functions , and can render multiple objects in dynamic scenes . 3 PRELIMINARIES . 3.1 VOLUME RENDERING . To render an image of a scene with arbitrary camera parameters , camera rays are sent into the scene , through each pixel on the image plane . The expected color of each pixel is computed as the radiance along each camera ray . Volume rendering is an approach for computing the radiance traveling along rays traced in a volume . Let r ( t ) = x0 + ωot be a point along a ray r with origin x0 and direction ωo , where t ∈ R is a 1D location along the ray , and the o in ωo denotes “ outgoing ” direction . For our purposes , we assume non-emissive and non-absorptive volumes . From Novák et al . ( 2018 ) , the volume rendering equation to compute the radiance L ( x0 , ωo ) of the ray is defined as : L ( x0 , ωo ) = ∫ tf tn τ ( t ) σ ( r ( t ) ) Ls ( r ( t ) , ωo ) dt , where τ ( t ) = exp ( − ∫ t tn σ ( r ( u ) ) du ) , ( 1 ) where tn and tf are near and far integration bounds , σ ( r ( t ) ) denotes the volume density of point r ( t ) , and τ ( t ) denotes the accumulated transmittance from tn to t. The termLs ( r ( t ) , ωo ) is the light scattered at point r ( t ) along direction ωo , defined as the integral over all incoming light directions : Ls ( x , ωo ) = ∫ S L ( x , ωl ) fp ( x , ωl , ωo ) dωl , ( 2 ) where S is a unit sphere and fp is a phase function that evaluates the fraction of light incoming from direction ωl at a point x that scatters out in direction ωo . In NeRF , Mildenhall et al . ( 2020 ) assume fixed illumination and do not consider any form of Equation 2 . We consider a more general form of the volume rendering equation that explicitly models light paths within and between objects . This is important for dynamic scenes , where lighting and objects can move with respect to one another .

Summary: This paper proposes a NeRF based method for composing photo-realistic scenes from captured images. The proposed method learns object-centric neural scattering functions (OSFs) to implicitly model per-object light transport using a lighting- and view-dependent neural network. Multiple objects can be rendered with volumetric path tracing. The proposed method has been evaluated on both synthetic and real-world datasets.