Neural Relightable Participating Media Rendering

1 Introduction . From natural phenomenons like fog and cloud to ornaments like jade artworks and wax figures , participating media objects are pervasive in both real life and virtual content like movies or games . Inferring the bounding geometry and scattering properties of participating media objects from observed images is a long-standing problem in both computer vision and graphics . Traditional methods addressed the problem by exploiting specialized structured lighting patterns [ 1 , 2 , 3 ] or using discrete representations [ 4 ] . These methods , however , require the bounding geometry of participating media objects to be known . Learning neural radiance fields or neural scene representations [ 5 , 6 , 7 ] has achieved remarkable progress in image synthesis . They are able to optimize the representations with the assistance of a differentiable ray marching process . However , these methods are mostly designed for novel view synthesis and have baked in materials and lighting into the radiance fields or surface color . Therefore , they can hardly support downstream tasks such as relighting and scene editing . Recent work [ 8 , 9 ] has taken initial steps to disentangle the lighting and materials from radiance . For material , their methods are primarily designed for solid objects with opaque surfaces , thus they assume an underlying surface at each point with a normal and a BRDF . The assumed prior , however , does not apply to non-opaque participating media which has no internal surfaces . For lighting , neural reflectance field [ 8 ] simulates direct illumination from a single point light , whereas NeRV [ 9 ] handles direct illumination and one-bounce indirect illumination . They generally suffer from the energy loss issue due to ignoring the high-order indirect illumination . However , indirect lighting from multiple scattering plays a significant role in the final appearance [ 10 ] of participating media . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we propose a novel neural representation for learning relightable participating media . Our method takes as input a set of posed images with varying but known lighting conditions and designs neural networks to learn a disentangled representation for the participating media with physical properties , including volume density , scattering albedo and phase function parameter . To synthesize images , we embed a differentiable physically-based ray marching process in the framework . In addition , we propose to simulate global illumination by embedding the single scattering and multiple scattering estimation into the ray marching process , where single scattering is simulated by Monte Carlo ray tracing and the incident radiance from multiple scattering is approximated by spherical harmonics ( SH ) . Without supervising with ground-truth lighting decomposition , our method is able to learn a decomposition of direct lighting and indirect lighting in an unsupervised manner . Our comprehensive experiments demonstrate that our method achieves better visual quality and higher numerical performance compared to state-of-the-art methods . Meanwhile , our method can generalize to handle solid objects with opaque surfaces . We also demonstrate that our learned neural representations of participating media allow relighting , scene editing and insertion into another virtual environment . To summarize , our approach has the following contributions : 1 . We propose a novel method to learn a disentangled neural representation for participating media from posed images and it is able to generalize to solid objects . 2 . Our method deals with both single scattering and multiple scattering and enables the unsupervised decomposition of direct illumination and indirect illumination . 3 . We demonstrate flexibility of the learned representation of participating media for relighting , scene editing and scene compositions . 2 Related Work . Neural scene representations . Neural scene representations [ 11 , 7 , 6 ] are important building blocks for the recent progress in synthesizing realistic images . Different from representations of components such as ambient lighting and cameras [ 12 , 13 , 14 ] of scenes , neural scene representation [ 11 ] learns an embedding manifold from 2D images and scene representation networks [ 7 ] aim to infer the 3D context of scenes from images . Classic explicit 3D representations , such as voxels [ 15 , 16 , 6 ] , multiplane images [ 17 , 18 ] and proxy geometry [ 19 ] are exploited to learn neural representations for specific purposes . These explicit representations generally suffer from the intrinsic resolution limitation . To sidestep the limitation , most recent approaches shift towards implicit representations , like signed distance fields [ 20 , 21 ] , volumetric occupancy fields [ 22 , 23 , 24 , 25 ] , or coordinatebased neural networks [ 26 , 7 , 27 , 28 ] . By embedding a differentiable rendering process like ray marching [ 6 , 5 ] or sphere tracing [ 7 , 29 ] into these implicit representations , these methods are capable of optimizing the scene representations from observed images and synthesizing novel views after training . While they generally show improved quality compared to interpolation based novel view synthesis methods [ 30 , 31 ] , the learned representations are usually texture colors and radiance , without separating lighting and materials . By contrast , we propose to learn a neural representation with disentangled volume density , scattering properties and lighting , which allow the usages in relighting , editing and scene composition tasks . Volume geometry and properties capture . Acquiring geometry and scattering properties of participating media has long been of the interest to the computer vision and graphics community . Early methods utilize sophisticated scanning and recording devices [ 32 , 2 ] and specialized lighting patterns [ 3 , 1 ] to capture volume density . Computational imaging methods [ 33 , 34 ] frame the inference of scattering properties from images as an inverse problem , but they require that the geometries of objects are known . Based on the differentiable path tracing formulation [ 35 ] , the inverse transport method [ 36 ] incorporates a differentiable light transport [ 37 ] module within an analysis-by-synthesis pipeline to infer scattering properties , but it aims for known geometries and homogeneous participating media . In contrast , our method learns the geometries and scattering properties of participating media simultaneously and our method can deal with both homogeneous and heterogeneous participating media . Relighting . Neural Radiance Field ( NeRF ) [ 5 ] and its later extensions [ 38 ] encode the geometry and radiance into MLPs and leverage ray marching to synthesize new views . While they achieve realistic results , they are limited to synthesize views under the same lighting conditions as in training . To mitigate this , appearance latent code [ 39 , 40 ] are used to condition on the view synthesis . Recent approaches [ 8 , 9 ] decompose materials and lighting by assuming an opaque surface at each point , but this does not apply to participating media . After training , density and materials can be extracted [ 41 ] to render new views using Monte Carlo methods [ 42 , 43 , 44 ] . Instead , our method models participating media as a field of particles that scatter and absorb light , which are in accordance with its nature . Neural reflectance field [ 45 , 8 ] requires collocated cameras and lights during training and simulates only direct illumination . NeRV [ 9 ] and OSF [ 46 ] simulate merely one-bounce indirect light transport because of the prohibitive computation cost for long paths . However , ignoring the high-order indirect illumination leads to the potential problem of energy loss . By contrast , we use Monte Carlo ray tracing to compute direct lighting and propose to learn a spherical harmonic field for estimating the complete indirect lighting . The PlenOctree [ 47 ] uses spherical harmonics to represent the outgoing radiance field as in NeRF , but it does not allow relighting . With both direct illumination and indirect illumination properly estimated , our method enables a principled disentanglement of volume density , scattering properties , and lighting . 3 Background . Volume rendering . The radiance carried by a ray after its interaction with participating media can be computed based on the radiative transfer equation [ 48 ] Lo ( r0 , rd ) = ∫ ∞ 0 τ ( r ( t ) ) σ ( r ( t ) ) L ( r ( t ) , −rd ) dt . ( 1 ) Here , r is a ray starting from r0 along the direction rd and r ( t ) = ro+ t ·rd denotes a point along the ray at the parametric distance1 t. Lo is the received radiance at r0 along rd . σ denotes the extinction coefficient that is referred to as volume density . The τ ( r ( t ) ) is the transmittance between r ( t ) and r0 and it can be computed by exp ( − ∫ t 0 σ ( r ( s ) ) ds ) . The L ( · ) inside the integral stands for the in-scattered radiance towards r0 along −rd . NeRF [ 5 ] models the in-scattered radiance L ( Eq . 1 ) as a view dependent color c , but it ignores the underlying scattering event and incident illumination . Since the learned radiance field of NeRF bakes in the lighting and materials , it allows merely view synthesis under the original fixed lighting , without the support for relighting . Ray marching . The integral in Equation 1 can be solved with the numerical integration method , ray marching [ 49 ] . This is generally done by casting rays into the volume and taking point samples along each ray to collect volume density and color values [ 5 , 6 ] . The predicted color of a ray is computed by Lo ( r0 , rd ) = ∑ j τj · αj · L ( r ( tj ) , −rd ) , where αj = 1 − exp ( −σj · δj ) , δj = ∥tj+1 − tj∥2 and τj = ∏j−1 i=1 ( 1− αi ) . 4 Neural Relightable Participating Media . In this work , we aim to learn neural representations of participating media with disentangled volume density and scattering properties . We model the participating media as a field of particles that absorb and scatter light-carrying rays . Below we first describe our disentangled neural representation based on a decomposition with single scattering and multiple scattering . Then , we depict our neural network design , followed by details of a volume rendering process for synthesizing images and the details of training . 4.1 Lighting Decomposition . We firstly write the in-scattered radiance L ( r ( t ) , −rd ) in Equation 1 as an integral of light-carrying spherical directions over a 4π steradian range L ( r ( t ) , −rd ) = ∫ Ω4π S ( r ( t ) , −rd , ωi ) Lin ( r ( t ) , ωi ) dωi , ( 2 ) 1While the integration accounts for a t going to ∞ , t covers only the range with participating media in practice . where Lin is the incident radiance at r ( t ) from the direction ωi . S ( · ) is a scattering function which determines the portion of lighting that is deflected towards the −rd . Previous methods [ 8 , 9 , 41 ] assume a surface prior with a normal at every point and account for 2π hemispherical incident directions . Accordingly , they define the scattering function S as a BRDF function . This assumption , however , can hardly match the participating media objects which have no internal surfaces and normals . By contrast , we deal with light-carrying directions over the full 4π steradian range . Specifically , we define S = a ( r ( t ) ) · ρ ( −rd , ωi , g ) to account for the scattering over the full spherical directions . Here , a ( r ( t ) ) is the scattering albedo . ρ is the Henyey-Greenstein ( HG ) [ 50 ] phase function2 that decides the scattering directionality ( Appendix A ) , where g is an asymmetry parameter in ( −1 , 1 ) . For brevity , we omit g in the notation of ρ . Then , we can rewrite the radiance of ray r as the disentangled form : Lo ( r0 , rd ) = ∫ ∞ 0 τ ( r ( t ) ) σ ( r ( t ) ) ∫ Ω4π a ( r ( t ) ) ρ ( −rd , ωi ) Lin ( r ( t ) , ωi ) dωi dt , ( 3 ) from which the volume density σ decides the geometries , and the albedo a along with the phase function ρ controls the scattering of rays . We propose to train neural networks to learn the volume density and scattering properties . To compute Lo , we additionally need to estimate Lin ( Eq . 3 ) . This can be conducted by recursively substituting the Lo into Lin and expressing the radiance for a pixel k as an integral over all paths Pk = ∫ Ψ gk ( x̄ ) dµ ( x̄ ) , where x̄ is a light-carrying path , gk denotes a measurement contribution function , µ is the measure for the path space , and Ψ is the space of paths with all possible lengths . Pk is then computed by a summation of the contributions of all paths . The contribution of a path with length i can be written as Pk , i = ∫ l1 ∫ Ω · · · ∫ li−1 ∫ Ω Le ( x ( ti−1 ) ) V ( x ( ti−1 ) , ωi−1 ) T ( Pk , i ) dt1 dω1 · · · dti−1 dωi−1 , ( 4 ) where x ( ti−1 ) is a point along the ( i− 1 ) -th ray segment with the length li−1 , ωi−1 is the scattering direction in the space Ω and ω0 = rd , Le denotes the emitted radiance towards x ( ti−1 ) from a light source , V accounts for the transmittance between x ( ti−1 ) and the light source , T ( Pk , i ) =∏i−1 j=1 V ( x ( tj ) , ωj ) σ ( x ( tj ) ) · ∏i−1 j=1 ρ ( −ωj−1 , ωj ) a ( x ( tj ) ) is called the path throughput . Volumetric path tracing [ 44 ] utilizes Monte Carlo method to estimate the integral but its computational cost increases quickly when using high sampling rates of paths with many bounces . To reduce the cost , NeRV [ 9 ] truncates the paths and only considers up to one indirect bounce , namely i = 3 in Eq . 4 . This , however , leads to energy loss since high-order indirect illumination are neglected . Instead of tracing infinitely long paths or truncating the paths , we propose to decompose the inscattered radiance L in Eq . 1 as L = Ls + Lm , where Ls is the single-scattering contribution and Lm denotes the multiple-scattering contribution . Therefore , Lo can be split into two integrals : Lo , s = ∫∞ 0 τ ( r ( t ) ) σ ( r ( t ) ) Ls dt and Lo , m = ∫∞ 0 τ ( r ( t ) ) σ ( r ( t ) ) Lm dt , that can be evaluated separately . Single scattering . To compute the Lo , s , we evaluate the following integral at each sample point r ( t ) along the camera ray r Ls = ∫ Ω4π a ( r ( t ) ) ρ ( −rd , ωi ) Le ( r ( t ) , ωi ) V ( r ( t ) , ωi ) dωi . ( 5 ) Here , Le is the emitted radiance from a light source to the point r ( t ) and V is the transmittance between r ( t ) and a light source . The transmittance can be computed with another integral of volume density as described in Section 3 , but computing the integral for all points leads to high computation cost during training and inference . Therefore , we train a visibility neural network to regress the transmittance value as done in [ 9 ] . Multiple scattering . For Lo , m , we evaluate the Lm = ∫ Ω4π a ( r ( t ) ) ρ ( −rd , ωi ) Lin ( r ( t ) , ωi ) dωi . Lm aggregates the incoming radiance of rays that have been scattered at least once in the participating media . Since the distribution of incident radiance from multiple scattering is generally smooth , we propose to represent the incident radiance Lin as a spherical harmonics expansion : Lin ( ωi ) = 2Both the incident and outgoing directions point away from a scattering location in this paper . F ( ∑lmax l=0 ∑l m=−l c m l Y m l ( ωi ) ) , where lmax is the maximum spherical harmonic band , cml ∈ R3 are spherical harmonic coefficients for the RGB spectrum , Y ml are spherical harmonic basis functions and F ( x ) = max ( 0 , x ) . Therefore , we compute the multiple-scattering contribution with Lm = ∫ Ω4π a ( r ( t ) ) ρ ( −rd , ωi ) · F ( lmax∑ l=0 l∑ m=−l cml Y m l ( ωi ) ) dωi . ( 6 ) We employ a neural network to learn spherical harmonic coefficients . By using spherical harmonics for the incident radiance from multiple scattering , we sidestep the lengthy extension of the path integral ( Eq . 4 ) . Since we introduce the approximation of multiple scattering at the primary rays , we sidestep the explosion of rays . Figure 1 visualizes the explosion of rays when computing indirect illumination under an environment lighting . Multiple shadow rays are needed to account for the directional emission from the light source . The brute-force ray splitting approach leads to explosion of rays and is impractical . NeRV reduces the ray count by tracing up to one indirect bounce , but it has a complexity of O ( M ·N ) , where M is the number of first indirect bounces ( red ) and N is the number of shadow rays ( blue ) . Our method uses spherical harmonics to handle indirect illumination as a whole and its complexity is O ( k ·M ) , where k is the sampling rates along the primary rays . Network architectures . Figure 2 presents our overall architecture . Our neural networks are based on the coordinate-based MLP strategy , and we use frequency-based positional encoding [ 5 , 51 ] E to map an input coordinate p to a higher dimensional vector E ( p ) before sending it to the neural networks . Specifically , we use the MLP Rϑ to predict volume density σ ( 1D ) , scattering albedo a ( 3D ) , and asymmetry parameter g ( 1D ) . Meanwhile , we employ the MLP Sϕ to learn spherical harmonic coefficients cml . Here , Sϕ is conditioned on the point light location ζ , the point light intensity I , and a binary indicator e which is one-hot encoded to indicate the existence of environment lighting . Since both the property network Rϑ and the SH network Sϕ takes as input the encoded coordinate , we introduce a feature network Fθ to predict shared features for downstream MLPs . To get the visibility for shadow rays , we train another MLP Vφ , which takes in the encoded direction d in addition to the encoded coordinate p , to learn visibility values for the estimation of single scattering . In summary , we have Rϑ : Fθ ( E ( p ) ) → ( σ , a , g ) Sϕ : ( e , ζ , I , Fθ ( E ( p ) ) ) → { cml } Vφ : ( r ( t ) , E ( d ) ) → τ . ( 7 ) Note that the above Rϑ learns per-location asymmetry parameter g ( Appendix A ) . Yet , for scenes with a single participating media object , we use a singe g and optimize it during training .

Summary: The paper proposes a method to learn a volumetric representation for scenes with  participating media from a set of images. It builds on the NeRF architecture.  The difference part is that it applies a phase function to model how light is  scattered at point instead of using radiance field or BRDF. It also models the  incident multi-scatterings at each point using spherical harmonics to avoid path  tracing. They show better results than previous methods and demo applications   such as scene composition and editing.