Neural Relightable Participating Media Rendering

1 Introduction . From natural phenomenons like fog and cloud to ornaments like jade artworks and wax figures , participating media objects are pervasive in both real life and virtual content like movies or games . Inferring the bounding geometry and scattering properties of participating media objects from observed images is a long-standing problem in both computer vision and graphics . Traditional methods addressed the problem by exploiting specialized structured lighting patterns [ 1 , 2 , 3 ] or using discrete representations [ 4 ] . These methods , however , require the bounding geometry of participating media objects to be known . Learning neural radiance fields or neural scene representations [ 5 , 6 , 7 ] has achieved remarkable progress in image synthesis . They are able to optimize the representations with the assistance of a differentiable ray marching process . However , these methods are mostly designed for novel view synthesis and have baked in materials and lighting into the radiance fields or surface color . Therefore , they can hardly support downstream tasks such as relighting and scene editing . Recent work [ 8 , 9 ] has taken initial steps to disentangle the lighting and materials from radiance . For material , their methods are primarily designed for solid objects with opaque surfaces , thus they assume an underlying surface at each point with a normal and a BRDF . The assumed prior , however , does not apply to non-opaque participating media which has no internal surfaces . For lighting , neural reflectance field [ 8 ] simulates direct illumination from a single point light , whereas NeRV [ 9 ] handles direct illumination and one-bounce indirect illumination . They generally suffer from the energy loss issue due to ignoring the high-order indirect illumination . However , indirect lighting from multiple scattering plays a significant role in the final appearance [ 10 ] of participating media . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we propose a novel neural representation for learning relightable participating media . Our method takes as input a set of posed images with varying but known lighting conditions and designs neural networks to learn a disentangled representation for the participating media with physical properties , including volume density , scattering albedo and phase function parameter . To synthesize images , we embed a differentiable physically-based ray marching process in the framework . In addition , we propose to simulate global illumination by embedding the single scattering and multiple scattering estimation into the ray marching process , where single scattering is simulated by Monte Carlo ray tracing and the incident radiance from multiple scattering is approximated by spherical harmonics ( SH ) . Without supervising with ground-truth lighting decomposition , our method is able to learn a decomposition of direct lighting and indirect lighting in an unsupervised manner . Our comprehensive experiments demonstrate that our method achieves better visual quality and higher numerical performance compared to state-of-the-art methods . Meanwhile , our method can generalize to handle solid objects with opaque surfaces . We also demonstrate that our learned neural representations of participating media allow relighting , scene editing and insertion into another virtual environment . To summarize , our approach has the following contributions : 1 . We propose a novel method to learn a disentangled neural representation for participating media from posed images and it is able to generalize to solid objects . 2 . Our method deals with both single scattering and multiple scattering and enables the unsupervised decomposition of direct illumination and indirect illumination . 3 . We demonstrate flexibility of the learned representation of participating media for relighting , scene editing and scene compositions . 2 Related Work . Neural scene representations . Neural scene representations [ 11 , 7 , 6 ] are important building blocks for the recent progress in synthesizing realistic images . Different from representations of components such as ambient lighting and cameras [ 12 , 13 , 14 ] of scenes , neural scene representation [ 11 ] learns an embedding manifold from 2D images and scene representation networks [ 7 ] aim to infer the 3D context of scenes from images . Classic explicit 3D representations , such as voxels [ 15 , 16 , 6 ] , multiplane images [ 17 , 18 ] and proxy geometry [ 19 ] are exploited to learn neural representations for specific purposes . These explicit representations generally suffer from the intrinsic resolution limitation . To sidestep the limitation , most recent approaches shift towards implicit representations , like signed distance fields [ 20 , 21 ] , volumetric occupancy fields [ 22 , 23 , 24 , 25 ] , or coordinatebased neural networks [ 26 , 7 , 27 , 28 ] . By embedding a differentiable rendering process like ray marching [ 6 , 5 ] or sphere tracing [ 7 , 29 ] into these implicit representations , these methods are capable of optimizing the scene representations from observed images and synthesizing novel views after training . While they generally show improved quality compared to interpolation based novel view synthesis methods [ 30 , 31 ] , the learned representations are usually texture colors and radiance , without separating lighting and materials . By contrast , we propose to learn a neural representation with disentangled volume density , scattering properties and lighting , which allow the usages in relighting , editing and scene composition tasks . Volume geometry and properties capture . Acquiring geometry and scattering properties of participating media has long been of the interest to the computer vision and graphics community . Early methods utilize sophisticated scanning and recording devices [ 32 , 2 ] and specialized lighting patterns [ 3 , 1 ] to capture volume density . Computational imaging methods [ 33 , 34 ] frame the inference of scattering properties from images as an inverse problem , but they require that the geometries of objects are known . Based on the differentiable path tracing formulation [ 35 ] , the inverse transport method [ 36 ] incorporates a differentiable light transport [ 37 ] module within an analysis-by-synthesis pipeline to infer scattering properties , but it aims for known geometries and homogeneous participating media . In contrast , our method learns the geometries and scattering properties of participating media simultaneously and our method can deal with both homogeneous and heterogeneous participating media . Relighting . Neural Radiance Field ( NeRF ) [ 5 ] and its later extensions [ 38 ] encode the geometry and radiance into MLPs and leverage ray marching to synthesize new views . While they achieve realistic results , they are limited to synthesize views under the same lighting conditions as in training . To mitigate this , appearance latent code [ 39 , 40 ] are used to condition on the view synthesis . Recent approaches [ 8 , 9 ] decompose materials and lighting by assuming an opaque surface at each point , but this does not apply to participating media . After training , density and materials can be extracted [ 41 ] to render new views using Monte Carlo methods [ 42 , 43 , 44 ] . Instead , our method models participating media as a field of particles that scatter and absorb light , which are in accordance with its nature . Neural reflectance field [ 45 , 8 ] requires collocated cameras and lights during training and simulates only direct illumination . NeRV [ 9 ] and OSF [ 46 ] simulate merely one-bounce indirect light transport because of the prohibitive computation cost for long paths . However , ignoring the high-order indirect illumination leads to the potential problem of energy loss . By contrast , we use Monte Carlo ray tracing to compute direct lighting and propose to learn a spherical harmonic field for estimating the complete indirect lighting . The PlenOctree [ 47 ] uses spherical harmonics to represent the outgoing radiance field as in NeRF , but it does not allow relighting . With both direct illumination and indirect illumination properly estimated , our method enables a principled disentanglement of volume density , scattering properties , and lighting . 3 Background . Volume rendering . The radiance carried by a ray after its interaction with participating media can be computed based on the radiative transfer equation [ 48 ] Lo ( r0 , rd ) = ∫ ∞ 0 τ ( r ( t ) ) σ ( r ( t ) ) L ( r ( t ) , −rd ) dt . ( 1 ) Here , r is a ray starting from r0 along the direction rd and r ( t ) = ro+ t ·rd denotes a point along the ray at the parametric distance1 t. Lo is the received radiance at r0 along rd . σ denotes the extinction coefficient that is referred to as volume density . The τ ( r ( t ) ) is the transmittance between r ( t ) and r0 and it can be computed by exp ( − ∫ t 0 σ ( r ( s ) ) ds ) . The L ( · ) inside the integral stands for the in-scattered radiance towards r0 along −rd . NeRF [ 5 ] models the in-scattered radiance L ( Eq . 1 ) as a view dependent color c , but it ignores the underlying scattering event and incident illumination . Since the learned radiance field of NeRF bakes in the lighting and materials , it allows merely view synthesis under the original fixed lighting , without the support for relighting . Ray marching . The integral in Equation 1 can be solved with the numerical integration method , ray marching [ 49 ] . This is generally done by casting rays into the volume and taking point samples along each ray to collect volume density and color values [ 5 , 6 ] . The predicted color of a ray is computed by Lo ( r0 , rd ) = ∑ j τj · αj · L ( r ( tj ) , −rd ) , where αj = 1 − exp ( −σj · δj ) , δj = ∥tj+1 − tj∥2 and τj = ∏j−1 i=1 ( 1− αi ) . 4 Neural Relightable Participating Media . In this work , we aim to learn neural representations of participating media with disentangled volume density and scattering properties . We model the participating media as a field of particles that absorb and scatter light-carrying rays . Below we first describe our disentangled neural representation based on a decomposition with single scattering and multiple scattering . Then , we depict our neural network design , followed by details of a volume rendering process for synthesizing images and the details of training . 4.1 Lighting Decomposition . We firstly write the in-scattered radiance L ( r ( t ) , −rd ) in Equation 1 as an integral of light-carrying spherical directions over a 4π steradian range L ( r ( t ) , −rd ) = ∫ Ω4π S ( r ( t ) , −rd , ωi ) Lin ( r ( t ) , ωi ) dωi , ( 2 ) 1While the integration accounts for a t going to ∞ , t covers only the range with participating media in practice . where Lin is the incident radiance at r ( t ) from the direction ωi . S ( · ) is a scattering function which determines the portion of lighting that is deflected towards the −rd . Previous methods [ 8 , 9 , 41 ] assume a surface prior with a normal at every point and account for 2π hemispherical incident directions . Accordingly , they define the scattering function S as a BRDF function . This assumption , however , can hardly match the participating media objects which have no internal surfaces and normals . By contrast , we deal with light-carrying directions over the full 4π steradian range . Specifically , we define S = a ( r ( t ) ) · ρ ( −rd , ωi , g ) to account for the scattering over the full spherical directions . Here , a ( r ( t ) ) is the scattering albedo . ρ is the Henyey-Greenstein ( HG ) [ 50 ] phase function2 that decides the scattering directionality ( Appendix A ) , where g is an asymmetry parameter in ( −1 , 1 ) . For brevity , we omit g in the notation of ρ . Then , we can rewrite the radiance of ray r as the disentangled form : Lo ( r0 , rd ) = ∫ ∞ 0 τ ( r ( t ) ) σ ( r ( t ) ) ∫ Ω4π a ( r ( t ) ) ρ ( −rd , ωi ) Lin ( r ( t ) , ωi ) dωi dt , ( 3 ) from which the volume density σ decides the geometries , and the albedo a along with the phase function ρ controls the scattering of rays . We propose to train neural networks to learn the volume density and scattering properties . To compute Lo , we additionally need to estimate Lin ( Eq . 3 ) . This can be conducted by recursively substituting the Lo into Lin and expressing the radiance for a pixel k as an integral over all paths Pk = ∫ Ψ gk ( x̄ ) dµ ( x̄ ) , where x̄ is a light-carrying path , gk denotes a measurement contribution function , µ is the measure for the path space , and Ψ is the space of paths with all possible lengths . Pk is then computed by a summation of the contributions of all paths . The contribution of a path with length i can be written as Pk , i = ∫ l1 ∫ Ω · · · ∫ li−1 ∫ Ω Le ( x ( ti−1 ) ) V ( x ( ti−1 ) , ωi−1 ) T ( Pk , i ) dt1 dω1 · · · dti−1 dωi−1 , ( 4 ) where x ( ti−1 ) is a point along the ( i− 1 ) -th ray segment with the length li−1 , ωi−1 is the scattering direction in the space Ω and ω0 = rd , Le denotes the emitted radiance towards x ( ti−1 ) from a light source , V accounts for the transmittance between x ( ti−1 ) and the light source , T ( Pk , i ) =∏i−1 j=1 V ( x ( tj ) , ωj ) σ ( x ( tj ) ) · ∏i−1 j=1 ρ ( −ωj−1 , ωj ) a ( x ( tj ) ) is called the path throughput . Volumetric path tracing [ 44 ] utilizes Monte Carlo method to estimate the integral but its computational cost increases quickly when using high sampling rates of paths with many bounces . To reduce the cost , NeRV [ 9 ] truncates the paths and only considers up to one indirect bounce , namely i = 3 in Eq . 4 . This , however , leads to energy loss since high-order indirect illumination are neglected . Instead of tracing infinitely long paths or truncating the paths , we propose to decompose the inscattered radiance L in Eq . 1 as L = Ls + Lm , where Ls is the single-scattering contribution and Lm denotes the multiple-scattering contribution . Therefore , Lo can be split into two integrals : Lo , s = ∫∞ 0 τ ( r ( t ) ) σ ( r ( t ) ) Ls dt and Lo , m = ∫∞ 0 τ ( r ( t ) ) σ ( r ( t ) ) Lm dt , that can be evaluated separately . Single scattering . To compute the Lo , s , we evaluate the following integral at each sample point r ( t ) along the camera ray r Ls = ∫ Ω4π a ( r ( t ) ) ρ ( −rd , ωi ) Le ( r ( t ) , ωi ) V ( r ( t ) , ωi ) dωi . ( 5 ) Here , Le is the emitted radiance from a light source to the point r ( t ) and V is the transmittance between r ( t ) and a light source . The transmittance can be computed with another integral of volume density as described in Section 3 , but computing the integral for all points leads to high computation cost during training and inference . Therefore , we train a visibility neural network to regress the transmittance value as done in [ 9 ] . Multiple scattering . For Lo , m , we evaluate the Lm = ∫ Ω4π a ( r ( t ) ) ρ ( −rd , ωi ) Lin ( r ( t ) , ωi ) dωi . Lm aggregates the incoming radiance of rays that have been scattered at least once in the participating media . Since the distribution of incident radiance from multiple scattering is generally smooth , we propose to represent the incident radiance Lin as a spherical harmonics expansion : Lin ( ωi ) = 2Both the incident and outgoing directions point away from a scattering location in this paper . F ( ∑lmax l=0 ∑l m=−l c m l Y m l ( ωi ) ) , where lmax is the maximum spherical harmonic band , cml ∈ R3 are spherical harmonic coefficients for the RGB spectrum , Y ml are spherical harmonic basis functions and F ( x ) = max ( 0 , x ) . Therefore , we compute the multiple-scattering contribution with Lm = ∫ Ω4π a ( r ( t ) ) ρ ( −rd , ωi ) · F ( lmax∑ l=0 l∑ m=−l cml Y m l ( ωi ) ) dωi . ( 6 ) We employ a neural network to learn spherical harmonic coefficients . By using spherical harmonics for the incident radiance from multiple scattering , we sidestep the lengthy extension of the path integral ( Eq . 4 ) . Since we introduce the approximation of multiple scattering at the primary rays , we sidestep the explosion of rays . Figure 1 visualizes the explosion of rays when computing indirect illumination under an environment lighting . Multiple shadow rays are needed to account for the directional emission from the light source . The brute-force ray splitting approach leads to explosion of rays and is impractical . NeRV reduces the ray count by tracing up to one indirect bounce , but it has a complexity of O ( M ·N ) , where M is the number of first indirect bounces ( red ) and N is the number of shadow rays ( blue ) . Our method uses spherical harmonics to handle indirect illumination as a whole and its complexity is O ( k ·M ) , where k is the sampling rates along the primary rays . Network architectures . Figure 2 presents our overall architecture . Our neural networks are based on the coordinate-based MLP strategy , and we use frequency-based positional encoding [ 5 , 51 ] E to map an input coordinate p to a higher dimensional vector E ( p ) before sending it to the neural networks . Specifically , we use the MLP Rϑ to predict volume density σ ( 1D ) , scattering albedo a ( 3D ) , and asymmetry parameter g ( 1D ) . Meanwhile , we employ the MLP Sϕ to learn spherical harmonic coefficients cml . Here , Sϕ is conditioned on the point light location ζ , the point light intensity I , and a binary indicator e which is one-hot encoded to indicate the existence of environment lighting . Since both the property network Rϑ and the SH network Sϕ takes as input the encoded coordinate , we introduce a feature network Fθ to predict shared features for downstream MLPs . To get the visibility for shadow rays , we train another MLP Vφ , which takes in the encoded direction d in addition to the encoded coordinate p , to learn visibility values for the estimation of single scattering . In summary , we have Rϑ : Fθ ( E ( p ) ) → ( σ , a , g ) Sϕ : ( e , ζ , I , Fθ ( E ( p ) ) ) → { cml } Vφ : ( r ( t ) , E ( d ) ) → τ . ( 7 ) Note that the above Rϑ learns per-location asymmetry parameter g ( Appendix A ) . Yet , for scenes with a single participating media object , we use a singe g and optimize it during training .

Summary: The authors present a neural implicit representation for participating media. This requires (i) extending neural radiance field approaches that are limited to a fixed lighting and therefore less flexible regarding relighting and scene editing, or (ii) overcoming methods for disentangling reflectance and illumination that are typically limited to solid objects without subsurface scattering and direct lighting. The presented approach takes input data in terms of a set of images with camera calibration data taken under varying but known lighting conditions. Then they use a NeRF-like approach for learning a disentangled representation for the participating media with volume density and scattering albedo. Here, they use a differentiable physically-based ray marching where they embed single scattering (using Monte Carlo ray tracing) and multiple scattering (using spherical harmonics approximation) separately to handle global illumination at reduced computational burden. This allows learning the decomposition of direct lighting and indirect lighting in an unsupervised manner.  In their evaluation, the authors demonstrate the learned representation of participating media to outperform related state-of-the-art techniques (in both quantitative and qualitative comparisons) and to be suited for scene relighting, scene editing and scene composition.