Neural Relightable Participating Media Rendering
Year: 2021

Abstract:
Learning neural radiance fields of a scene has recently allowed realistic novel view synthesis of the scene, but they are limited to synthesize images under the original fixed lighting condition. Therefore, they are not flexible for the eagerly desired tasks like relighting, scene editing and scene composition. To tackle this problem, several recent methods propose to disentangle reflectance and illumination from the radiance field. These methods can cope with solid objects with opaque surfaces but participating media are neglected. Also, they take into account only direct illumination or at most one-bounce indirect illumination, thus suffer from energy loss due to ignoring the high-order indirect illumination. We propose to learn neural representations for participating media with a complete simulation of global illumination. We estimate direct illumination via ray tracing and compute indirect illumination with spherical harmonics. Our approach avoids computing the lengthy indirect bounces and does not suffer from energy loss. Our experiments on multiple scenes show that our approach achieves superior visual quality and numerical performance compared to state-of-the-art methods, and it can generalize to deal with solid objects with opaque surfaces as well.

heading: 1 Introduction
text: From natural phenomenons like fog and cloud to ornaments like jade artworks and wax figures, participating media objects are pervasive in both real life and virtual content like movies or games. Inferring the bounding geometry and scattering properties of participating media objects from observed images is a long-standing problem in both computer vision and graphics. Traditional methods addressed the problem by exploiting specialized structured lighting patterns [1, 2, 3] or using discrete representations [4]. These methods, however, require the bounding geometry of participating media objects to be known.
Learning neural radiance fields or neural scene representations [5, 6, 7] has achieved remarkable progress in image synthesis. They are able to optimize the representations with the assistance of a differentiable ray marching process. However, these methods are mostly designed for novel view synthesis and have baked in materials and lighting into the radiance fields or surface color. Therefore, they can hardly support downstream tasks such as relighting and scene editing. Recent work [8, 9] has taken initial steps to disentangle the lighting and materials from radiance. For material, their methods are primarily designed for solid objects with opaque surfaces, thus they assume an underlying surface at each point with a normal and a BRDF. The assumed prior, however, does not apply to non-opaque participating media which has no internal surfaces. For lighting, neural reflectance field [8] simulates direct illumination from a single point light, whereas NeRV [9] handles direct illumination and one-bounce indirect illumination. They generally suffer from the energy loss issue due to ignoring the high-order indirect illumination. However, indirect lighting from multiple scattering plays a significant role in the final appearance [10] of participating media.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
In this paper, we propose a novel neural representation for learning relightable participating media. Our method takes as input a set of posed images with varying but known lighting conditions and designs neural networks to learn a disentangled representation for the participating media with physical properties, including volume density, scattering albedo and phase function parameter. To synthesize images, we embed a differentiable physically-based ray marching process in the framework. In addition, we propose to simulate global illumination by embedding the single scattering and multiple scattering estimation into the ray marching process, where single scattering is simulated by Monte Carlo ray tracing and the incident radiance from multiple scattering is approximated by spherical harmonics (SH). Without supervising with ground-truth lighting decomposition, our method is able to learn a decomposition of direct lighting and indirect lighting in an unsupervised manner.
Our comprehensive experiments demonstrate that our method achieves better visual quality and higher numerical performance compared to state-of-the-art methods. Meanwhile, our method can generalize to handle solid objects with opaque surfaces. We also demonstrate that our learned neural representations of participating media allow relighting, scene editing and insertion into another virtual environment. To summarize, our approach has the following contributions:
1. We propose a novel method to learn a disentangled neural representation for participating media from posed images and it is able to generalize to solid objects.
2. Our method deals with both single scattering and multiple scattering and enables the unsupervised decomposition of direct illumination and indirect illumination.
3. We demonstrate flexibility of the learned representation of participating media for relighting, scene editing and scene compositions.

heading: 2 Related Work
text: Neural scene representations. Neural scene representations [11, 7, 6] are important building blocks for the recent progress in synthesizing realistic images. Different from representations of components such as ambient lighting and cameras [12, 13, 14] of scenes, neural scene representation [11] learns an embedding manifold from 2D images and scene representation networks [7] aim to infer the 3D context of scenes from images. Classic explicit 3D representations, such as voxels [15, 16, 6], multiplane images [17, 18] and proxy geometry [19] are exploited to learn neural representations for specific purposes. These explicit representations generally suffer from the intrinsic resolution limitation. To sidestep the limitation, most recent approaches shift towards implicit representations, like signed distance fields [20, 21], volumetric occupancy fields [22, 23, 24, 25], or coordinatebased neural networks [26, 7, 27, 28]. By embedding a differentiable rendering process like ray marching [6, 5] or sphere tracing [7, 29] into these implicit representations, these methods are capable of optimizing the scene representations from observed images and synthesizing novel views after training. While they generally show improved quality compared to interpolation based novel view synthesis methods [30, 31], the learned representations are usually texture colors and radiance, without separating lighting and materials. By contrast, we propose to learn a neural representation with disentangled volume density, scattering properties and lighting, which allow the usages in relighting, editing and scene composition tasks.
Volume geometry and properties capture. Acquiring geometry and scattering properties of participating media has long been of the interest to the computer vision and graphics community. Early methods utilize sophisticated scanning and recording devices [32, 2] and specialized lighting patterns [3, 1] to capture volume density. Computational imaging methods [33, 34] frame the inference of scattering properties from images as an inverse problem, but they require that the geometries of objects are known. Based on the differentiable path tracing formulation [35], the inverse transport method [36] incorporates a differentiable light transport [37] module within an analysis-by-synthesis pipeline to infer scattering properties, but it aims for known geometries and homogeneous participating media. In contrast, our method learns the geometries and scattering properties of participating media simultaneously and our method can deal with both homogeneous and heterogeneous participating media.
Relighting. Neural Radiance Field (NeRF) [5] and its later extensions [38] encode the geometry and radiance into MLPs and leverage ray marching to synthesize new views. While they achieve
realistic results, they are limited to synthesize views under the same lighting conditions as in training. To mitigate this, appearance latent code [39, 40] are used to condition on the view synthesis. Recent approaches [8, 9] decompose materials and lighting by assuming an opaque surface at each point, but this does not apply to participating media. After training, density and materials can be extracted [41] to render new views using Monte Carlo methods [42, 43, 44]. Instead, our method models participating media as a field of particles that scatter and absorb light, which are in accordance with its nature. Neural reflectance field [45, 8] requires collocated cameras and lights during training and simulates only direct illumination. NeRV [9] and OSF [46] simulate merely one-bounce indirect light transport because of the prohibitive computation cost for long paths. However, ignoring the high-order indirect illumination leads to the potential problem of energy loss. By contrast, we use Monte Carlo ray tracing to compute direct lighting and propose to learn a spherical harmonic field for estimating the complete indirect lighting. The PlenOctree [47] uses spherical harmonics to represent the outgoing radiance field as in NeRF, but it does not allow relighting. With both direct illumination and indirect illumination properly estimated, our method enables a principled disentanglement of volume density, scattering properties, and lighting.

heading: 3 Background
text: Volume rendering. The radiance carried by a ray after its interaction with participating media can be computed based on the radiative transfer equation [48]
Lo (r0, rd) = ∫ ∞ 0 τ (r(t))σ (r(t))L (r(t),−rd) dt. (1)
Here, r is a ray starting from r0 along the direction rd and r(t) = ro+ t ·rd denotes a point along the ray at the parametric distance1 t. Lo is the received radiance at r0 along rd. σ denotes the extinction coefficient that is referred to as volume density. The τ (r(t)) is the transmittance between r(t) and r0 and it can be computed by exp ( − ∫ t 0 σ(r(s)) ds ) . The L(·) inside the integral stands for the in-scattered radiance towards r0 along −rd. NeRF [5] models the in-scattered radiance L (Eq. 1) as a view dependent color c, but it ignores the underlying scattering event and incident illumination. Since the learned radiance field of NeRF bakes in the lighting and materials, it allows merely view synthesis under the original fixed lighting, without the support for relighting.
Ray marching. The integral in Equation 1 can be solved with the numerical integration method, ray marching [49]. This is generally done by casting rays into the volume and taking point samples along each ray to collect volume density and color values [5, 6]. The predicted color of a ray is computed by Lo (r0, rd) = ∑ j τj · αj · L (r(tj),−rd), where αj = 1 − exp (−σj · δj), δj = ∥tj+1 − tj∥2
and τj = ∏j−1 i=1 (1− αi).

heading: 4 Neural Relightable Participating Media
text: In this work, we aim to learn neural representations of participating media with disentangled volume density and scattering properties. We model the participating media as a field of particles that absorb and scatter light-carrying rays. Below we first describe our disentangled neural representation based on a decomposition with single scattering and multiple scattering. Then, we depict our neural network design, followed by details of a volume rendering process for synthesizing images and the details of training.

heading: 4.1 Lighting Decomposition
text: We firstly write the in-scattered radiance L (r(t),−rd) in Equation 1 as an integral of light-carrying spherical directions over a 4π steradian range
L (r(t),−rd) = ∫ Ω4π S (r(t),−rd, ωi)Lin(r(t), ωi) dωi, (2)
1While the integration accounts for a t going to ∞, t covers only the range with participating media in practice.
where Lin is the incident radiance at r(t) from the direction ωi. S(·) is a scattering function which determines the portion of lighting that is deflected towards the −rd. Previous methods [8, 9, 41] assume a surface prior with a normal at every point and account for 2π hemispherical incident directions. Accordingly, they define the scattering function S as a BRDF function. This assumption, however, can hardly match the participating media objects which have no internal surfaces and normals. By contrast, we deal with light-carrying directions over the full 4π steradian range. Specifically, we define S = a(r(t)) · ρ(−rd, ωi, g) to account for the scattering over the full spherical directions. Here, a(r(t)) is the scattering albedo. ρ is the Henyey-Greenstein (HG) [50] phase function2 that decides the scattering directionality (Appendix A), where g is an asymmetry parameter in (−1, 1). For brevity, we omit g in the notation of ρ. Then, we can rewrite the radiance of ray r as the disentangled form:
Lo (r0, rd) = ∫ ∞ 0 τ (r(t))σ (r(t)) ∫ Ω4π a(r(t))ρ(−rd, ωi)Lin(r(t), ωi) dωi dt, (3)
from which the volume density σ decides the geometries, and the albedo a along with the phase function ρ controls the scattering of rays. We propose to train neural networks to learn the volume density and scattering properties.
To compute Lo, we additionally need to estimate Lin (Eq. 3). This can be conducted by recursively substituting the Lo into Lin and expressing the radiance for a pixel k as an integral over all paths Pk = ∫ Ψ gk(x̄)dµ(x̄), where x̄ is a light-carrying path, gk denotes a measurement contribution function, µ is the measure for the path space, and Ψ is the space of paths with all possible lengths. Pk is then computed by a summation of the contributions of all paths. The contribution of a path with length i can be written as
Pk,i = ∫ l1 ∫ Ω · · · ∫ li−1 ∫ Ω Le (x(ti−1))V (x(ti−1), ωi−1) T (Pk,i) dt1 dω1 · · · dti−1 dωi−1, (4)
where x(ti−1) is a point along the (i− 1)-th ray segment with the length li−1, ωi−1 is the scattering direction in the space Ω and ω0 = rd, Le denotes the emitted radiance towards x(ti−1) from a light source, V accounts for the transmittance between x(ti−1) and the light source, T (Pk,i) =∏i−1
j=1 V (x(tj), ωj)σ(x(tj)) · ∏i−1
j=1 ρ(−ωj−1, ωj)a(x(tj)) is called the path throughput. Volumetric path tracing [44] utilizes Monte Carlo method to estimate the integral but its computational cost increases quickly when using high sampling rates of paths with many bounces. To reduce the cost, NeRV [9] truncates the paths and only considers up to one indirect bounce, namely i = 3 in Eq. 4. This, however, leads to energy loss since high-order indirect illumination are neglected.
Instead of tracing infinitely long paths or truncating the paths, we propose to decompose the inscattered radiance L in Eq. 1 as L = Ls + Lm, where Ls is the single-scattering contribution and Lm denotes the multiple-scattering contribution. Therefore, Lo can be split into two integrals: Lo,s = ∫∞ 0 τ (r(t))σ (r(t))Ls dt and Lo,m = ∫∞ 0
τ (r(t))σ (r(t))Lm dt, that can be evaluated separately.
Single scattering. To compute the Lo,s, we evaluate the following integral at each sample point r(t) along the camera ray r
Ls = ∫ Ω4π a(r(t))ρ(−rd, ωi)Le(r(t), ωi)V (r(t), ωi) dωi. (5)
Here, Le is the emitted radiance from a light source to the point r(t) and V is the transmittance between r(t) and a light source. The transmittance can be computed with another integral of volume density as described in Section 3, but computing the integral for all points leads to high computation cost during training and inference. Therefore, we train a visibility neural network to regress the transmittance value as done in [9]. Multiple scattering. For Lo,m, we evaluate the Lm = ∫ Ω4π
a(r(t))ρ(−rd, ωi)Lin(r(t), ωi)dωi. Lm aggregates the incoming radiance of rays that have been scattered at least once in the participating media. Since the distribution of incident radiance from multiple scattering is generally smooth, we propose to represent the incident radiance Lin as a spherical harmonics expansion: Lin(ωi) =
2Both the incident and outgoing directions point away from a scattering location in this paper.
F (∑lmax
l=0 ∑l m=−l c m l Y m l (ωi) ) , where lmax is the maximum spherical harmonic band, cml ∈ R3
are spherical harmonic coefficients for the RGB spectrum, Y ml are spherical harmonic basis functions and F(x) = max(0, x). Therefore, we compute the multiple-scattering contribution with
Lm = ∫ Ω4π a(r(t))ρ(−rd, ωi) · F ( lmax∑ l=0 l∑ m=−l cml Y m l (ωi) ) dωi. (6)
We employ a neural network to learn spherical harmonic coefficients. By using spherical harmonics for the incident radiance from multiple scattering, we sidestep the lengthy extension of the path integral (Eq. 4). Since we introduce the approximation of multiple scattering at the primary rays, we sidestep the explosion of rays. Figure 1 visualizes the explosion of rays when computing indirect illumination under an environment lighting. Multiple shadow rays are needed to account for the directional emission from the light source. The brute-force ray splitting approach leads to explosion of rays and is impractical. NeRV reduces the ray count by tracing up to one indirect bounce, but it has a complexity of O(M ·N), where M is the number of first indirect bounces (red) and N is the number of shadow rays (blue). Our method uses spherical harmonics to handle indirect illumination as a whole and its complexity is O(k ·M), where k is the sampling rates along the primary rays.
Network architectures. Figure 2 presents our overall architecture. Our neural networks are based on the coordinate-based MLP strategy, and we use frequency-based positional encoding [5, 51] E to map an input coordinate p to a higher dimensional vector E(p) before sending it to the neural networks. Specifically,
we use the MLP Rϑ to predict volume density σ (1D), scattering albedo a (3D), and asymmetry parameter g (1D). Meanwhile, we employ the MLP Sϕ to learn spherical harmonic coefficients cml . Here, Sϕ is conditioned on the point light location ζ , the point light intensity I , and a binary indicator e which is one-hot encoded to indicate the existence of environment lighting. Since both the property network Rϑ and the SH network Sϕ takes as input the encoded coordinate, we introduce a feature network Fθ to predict shared features for downstream MLPs. To get the visibility for shadow rays, we train another MLP Vφ, which takes in the encoded direction d in addition to the encoded coordinate p, to learn visibility values for the estimation of single scattering. In summary, we have
Rϑ : Fθ(E(p)) → (σ, a, g) Sϕ : (e, ζ, I, Fθ(E(p))) → {cml } Vφ : (r(t), E(d)) → τ. (7)
Note that the above Rϑ learns per-location asymmetry parameter g (Appendix A). Yet, for scenes with a single participating media object, we use a singe g and optimize it during training.

heading: 4.2 Volume Rendering
text: Based on the above decomposition, we employ ray marching (Sec. 3) to numerically estimate Lo,s and Lo,m. Hence, the final radiance Lo of the camera ray r in Eq. 1 can be computed as:
Lo (r) = Σ N j=1τ (r(t)) (1− exp (−σ (r(t)) · δtj)) (Ls + Lm) , (8)
where we sample N = 64 query points in a stratified way along the ray r and δtj = ∥r(tj+1) − r(tj)∥2 is the step size. For each point sample, we query the MLPs to obtain its scattering properties and SH coefficients for computing single scattering and multiple scattering.
We compute single scattering at a point based on Eq. 5. We shoot shadow rays towards light sources to get the emitted radiance Le according to the light types. For environment lighting, we sample 64 directions stratified over a sphere around the point to obtain incident radiance. For a point light, we directly connect the query point to the light source. To account for the attenuation of light radiance, we query Vφ to get the visibility to the light source from the query point.
For multiple scattering, we uniformly sample K = 64 random incident directions over the sphere around each query point, evaluate the incident radiance Lin along each direction using the learned spherical harmonic coefficients, and estimate the integral in Eq. 6 with a Monte Carlo integration Lm = 1/K ∑K i=1 aρ(ωi)Lin(ωi). For brevity, we omit the r notation. Note that the visibility towards the light source is not needed in the computation.

heading: 4.3 End-to-end Learning
text: Based on the fully differentiable pipeline, we can end-to-end learn a neural representation for each scene. The learning requires a set of posed RGB images and their lighting conditions. During each training iteration, we trace primary rays through the media. Along each primary ray, we estimate single scattering using shadow rays and compute multiple scattering contribution via the learned spherical harmonic coefficients as described in Sec. 4.2. We optimize the parameters of Fθ, Rϑ, and Sϕ by minimizing a rendering loss between the predicted radiance Lo(r) from ray marching and the radiance L̂o(r) from input images. To train the visibility network Vφ, we use the transmittance V̂ϑ computed from the learned volume density as the ground truth and minimize the visibility loss between the prediction Vφ and the ground truth. Therefore, our loss function includes two parts:
L = ∑ r∈R ∥Γ(Lo(r))− Γ(L̂o(r))∥22 + µ · ∑ r∈R,t ∥Vφ(r(t), rd)− V̂ϑ(r(t), rd)∥22, (9)
where Γ(L) = L/(1 +L) is a tone mapping function, R is a batch of camera rays and µ = 0.1 is the hyperparameter to weight the visibility loss.

heading: 4.4 Implementation Details
text: Our feature MLP Fθ has 8 fully-connected ReLU (FC-ReLU) layers with 256 channels per layer. The downstream Sϕ consists of 8 FC-ReLU layers with 128 channels per layer, whereas the Rϑ uses one such layer with 128 channels. The visibility MLP Vφ has 4 FC-ReLU layers with 256 channels per layer. We set the maximum positional encoding frequency to 28 for coordinates p, 21 for directions d, and 22 for the 3D location of the point light.
We train all neural networks together to minimize the loss (Eq. 9). We use the Adam [52] optimizer with its default hyperparameters and schedule the learning rate to decay from 1× 10−4 to 1× 10−5 over 200K iterations. For each iteration, we trace a batch of 1200 primary rays. Note we stop the gradients from the visibility loss to the property network and the feature network so that they do not compromise the learning to match the visibility network.

heading: 5 Experiments
text: We firstly evaluate our method by comparing it with state-of-the-art methods on simultaneous relighting and view synthesis. Then, we demonstrate that our learned neural representations allow flexible editing and scene compositions, followed by ablation studies of this approach. Please refer to the appendices for additional results.

heading: 5.1 Experiment Settings
text: Compared methods. We compare our method with state-of-the-art baselines [8, 9]. They are designed for scenes with solid objects and do not trivially extend to handle participating media, so we implement them with the new functionality to handle participating media. Please refer to the Appendix D for the implementation details.
Datasets. We produce datasets from seven synthetic participating media scenes. The Cloud scene is heterogeneous media and the others are homogeneous media. Bunny4-VaryA and Buddha3 are set with spatially varying albedo. Bunny4-VaryG and Buddha3 are configured with spatially varying asymmetry parameters. Each scene is individually illuminated with two lighting conditions. The first one “point” has a white point light with varying intensities sampled within 50 ∼ 900 and its location is randomly sampled on spheres with the radius ranging from 3.0 to 5.0; The second one “env + point” contains a fixed environment lighting and a randomly sampled white point light. Each dataset contains 180 images, from which we use 170 images for training and the remaining for validation. In addition, we prepare a test set with 30 images for each scene to test the trained models. Each test image is rendered with a new camera pose and a new white point light that is located on a sphere of the radius 4.0. Since Bi’s method [8] requires a collocated camera and light during training, we additionally generated such datasets for it. For the “env + point” datasets, we randomize the usage of environment lighting across images and record a binary indicator for each image.

heading: 5.2 Results
text: Relighting comparisons. We show the qualitative comparisons of simultaneous relighting and view synthesis on the test data in Fig. 3. The left half is trained with the “point” lighting condition, whereas the right half is trained with the “env + point”. Bi’s method shows artifacts in each case as it has no mechanisms to simulate the multiple scattering that significantly affects the appearance of participating media. NeRV handles the environment illumination properly but shows artifacts on the participating media objects. Our method achieves realistic results on all test sets with
either homogeneous media or heterogeneous media. Table 1 and Table 2 present the corresponding quantitative measurements, where our method overtakes the compared methods on each test set.
Using the same batch size, our training with 200K iterations on a Nvidia Quadro RTX 8000 GPU takes one day, whereas Bi’s method and NeRV takes 22h and 46h. For a 400× 400 image, our average inference time is 7.9s, while Bi’s method and NeRV takes 53.2s and 21.9s, respectively.
Learned lighting decomposition. Without using any ground-truth lighting decomposition data, our method is able to learn the decomposition of lighting in an unsupervised way.
Figure 4 presents our decomposed results on a test view of the Cow scene, with the single-scattering component (direct lighting) and the multiple-scattering component (indirect lighting), and the corresponding ground-truth images.
Scene editing and scene composition. Our method learns neural representations for the participating media scenes. After training, we can query the neural networks to obtain the volume density, the albedo, and the phase function parameter. This allows flexible editing to achieve desired effects or insertion into a new virtual environment for content creation. In addition, we can leverage a standard rendering engine to render these data. Figure 5 compares the rendering
of the learned Bunny with ray marching using the neural network and with a volumetric path tracer. To render with the path tracer, we first queried the neural network to obtain 128 x 128 x 128 data volumes of volume density and albedo. Both rendered results are visually similar to the ground truth. Figure 6 demonstrates an editing of the red channel of the albedo to achieve the red cloud and another editing of the volume density to make the cloud thinner.
We show in Fig. 7 that we can compose a scene consisting of our learned cow and a gold sculpture described by traditional meshes and materials (Fig. 7 bottom). Similarly, we can construct a scene composed entirely of our learned objects (Fig. 7 top). To render the composed scenes, we slice out discrete volumes with a resolution 128× 128× 128 from the volume density field and the albedo field and conduct the Monte Carlo rendering in Mitsuba [54].
the glossy dragon better.

heading: 5.3 Ablation Studies
text: Spherical harmonic bands. We analyze the effect of the maximum spherical harmonics band lmax of Eq. 6 based on the Buddha scene. Figure 9 compares the same test view of each case on the left and tabulates the average quality measurements over the test set on the right. Removing the spherical harmonics from our method leads to quality drop and color shift is observed in its result. Based on the numerical metrics and inference timings, we select the lmax = 5 for other experiments.
Scattering function. We show in Fig. 8 that our method with the HG phase function can be applied to a scene of solid objects. In addition, we conduct an ablation by applying the variant “Ours+BRDF” to the Bunny scene of participating media. Figure 10 shows that the variant has difficulty in learning the volume density and leads to many cracks, while the proposed method performs robustly on this scene.

heading: 6 Limitations and Future Work
text: Real-world scenes. In this work, we learn neural representations from synthetic datasets with varying but known lighting conditions. Also, the camera
poses are available to the method. It would be interesting to extend this method to handle participating media captured from real-world scenes with unknown lighting conditions and unknown camera poses. In that case, the illumination and camera poses of the scenes need to be estimated in the first place.
Glossy reflections. For scenes with glossy solid objects (Fig. 8), our method tends to reproduce a smooth appearance and the glossy highlights are not as sharp as the ground truth. An avenue for future research would be to develop methods to recover the glossy reflections better.
Generalizability. Our ray marching with the trained neural network generalizes well to unseen light intensities and light locations that are in the range of the training data. That said, its generalization is in an interpolation manner. For light intensities and light locations that are outside of the training range, the generalization quality of the neural network gradually decreases. Please refer to the Appendix H for the analysis on the generalization quality.
Media within refractive boundaries. Our method achieves realistic results for participating media without refractive boundaries, like cloud, fog, and wax figure. Applying our method to participating media within refractive boundaries, like wine in a glass, entails further work as the refractive boundaries cause ambiguities due to deflecting the camera rays and the light rays.

heading: 7 Conclusion
text: We have proposed a novel method for participating media reconstruction from observed images with varying but known illumination. We propose to simulate direct illumination with Monte Carlo ray tracing and approximate indirect illumination with learned spherical harmonics. This enables our approach to learn to decompose the illumination as direct and indirect components in an unsupervised manner. Our method learns a disentangled neural representation with volume density, scattering albedo and phase function parameters for participating media, and we demonstrated its flexible applications in relighting, scene editing and scene compositions.
Acknowledgments and Disclosure of Funding
We acknowledge the valuable feedback from reviewers. This work was supported by Research Executive Agency 739578 and CYENS Phase 2 AE739578.

Review summaries:
The authors present a neural implicit representation for participating media. This requires (i) extending neural radiance field approaches that are limited to a fixed lighting and therefore less flexible regarding relighting and scene editing, or (ii) overcoming methods for disentangling reflectance and illumination that are typically limited to solid objects without subsurface scattering and direct lighting. The presented approach takes input data in terms of a set of images with camera calibration data taken under varying but known lighting conditions. Then they use a NeRF-like approach for learning a disentangled representation for the participating media with volume density and scattering albedo. Here, they use a differentiable physically-based ray marching where they embed single scattering (using Monte Carlo ray tracing) and multiple scattering (using spherical harmonics approximation) separately to handle global illumination at reduced computational burden. This allows learning the decomposition of direct lighting and indirect lighting in an unsupervised manner.  In their evaluation, the authors demonstrate the learned representation of participating media to outperform related state-of-the-art techniques (in both quantitative and qualitative comparisons) and to be suited for scene relighting, scene editing and scene composition. 

The paper proposes a method to learn a volumetric representation for scenes with  participating media from a set of images. It builds on the NeRF architecture.  The difference part is that it applies a phase function to model how light is  scattered at point instead of using radiance field or BRDF. It also models the  incident multi-scatterings at each point using spherical harmonics to avoid path  tracing. They show better results than previous methods and demo applications   such as scene composition and editing. 

This paper presents a neural volumetric inverse rendering approach for reconstructing a 3D representation of a scene with participating media (volumetric scattering) that can be rendered from novel viewpoints under novel lighting conditions. The paper builds on NeRV (an extension of NeRF that estimates relightable models of opaque objects) and modifies NeRV so that it can also recover relightable models of scenes with participating media. The main contributions of the paper are replacing the BRDF used in NeRV with a parametric phase/scattering function, and a technique to efficiently account for multiple scattering by training an MLP that predicts the "global illumination" at any location, represented by spherical harmonics, that is conditioned on a representation of the incoming lighting.