Neural Relightable Participating Media Rendering
Authors: Quan Zheng
Gurprit Singh
Hans-peter Seidel
Venue: NeurIPS 2021 Poster (2021)
Keywords: Neural rendering
Neural representation
Participating media
Volume Rendering
Relighting

Abstract: Learning neural radiance fields of a scene has recently allowed realistic novel view synthesis of the scene, but they are limited to synthesize images under the original fixed lighting condition. Therefore, they are not flexible for the eagerly desired tasks like relighting, scene editing and scene composition. To tackle this problem, several recent methods propose to disentangle reflectance and illumination from the radiance field. These methods can cope with solid objects with opaque surfaces but participating media are neglected. Also, they take into account only direct illumination or at most one-bounce indirect illumination, thus suffer from energy loss due to ignoring the high-order indirect illumination. We propose to learn neural representations for participating media with a complete simulation of global illumination. We estimate direct illumination via ray tracing and compute indirect illumination with spherical harmonics. Our approach avoids computing the lengthy indirect bounces and does not suffer from energy loss. Our experiments on multiple scenes show that our approach achieves superior visual quality and numerical performance compared to state-of-the-art methods, and it can generalize to deal with solid objects with opaque surfaces as well.

Neural Relightable Participating Media Rendering
Quan Zheng ?, Gurprit Singh', Hans-Peter Seidel
Max Planck Institute for Informatics, 66123 Saarbrücken, Germany Institute of Software, Chinese Academy of Sciences, 100190 Beijing, China {qzheng, gsingh, hpseide1) (mppeeimmmmmmmmmi inff-inf... mmg.
Abstract
Learning neural radiance fields scene has recently allowed realistic novel view synthesis of the scene, but they are limited to synthesize images under the original fixed lighting condition. Therefore, they are not flexible for the eagerly desired tasks like relighting, scene editing and scene composition. To tackle this problem, several recent methods propose to disentangle reflectance and illumination from the radiance field. These methods can cope with solid objects with opaque surfaces but participating media are neglected. Also, they take into account only direct illumination or most one-bounce indirect illumination, thus suffer from energy loss due to ignoring the high-order indirect illumination. We propose to learn neural representations for participating media with complete simulation of global illumination. We estimate direct illumination via ray tracing and compute indirect illumination with spherical harmonics. Our approach avoids computing the lengthy indirect bounces and does not suffer from energy loss. Our experiments on multiple scenes show that our approach achieves superior visual quality and numerical performance compared to state-of-the-art methods, and it can generalize to deal with solid objects with opaque surfaces as well.
Introduction
From natural phenomenons like fog and cloud to ornaments like jade artworks and wax figures, participating media objects pervasive in both real life and virtual content like movies games. Inferring the bounding geometry and scattering properties of participating media objects from observed images is a long-stann standing problem in both computer vision and graphics. Traditional methods addressed the problem by exploiting specialized structured lighting patterns 2 or using discrete representations 4 These methods, however, require the bounding geometry of participating media objects to be known.
Learning neural radiance fields or neural scene representations [5 has achieved remarkable progress in image synthesis. They are able to optimize the representations with the assistance of a differentiable ray marching process. However, these methods are mostly designed for novel view synthesis and have baked in materials and lighting into the radiance fields or surface color. Therefore, they can hardly support downstream tasks such relighting and scene editing, Recent work (8] has taken initial steps to disentangle the lighting and materials from radiance. For material, their methods are primarily designed for solid objects with opaque surfaces, thus they assume underlying surface at each point with normal and a BRDF. The assumed prior, however, does not apply to non-opaque participating media which has no internal surfaces. For lighting, neural reflectance field [ simulates direct illumination from single point light, whereas NeRV [9] handles direct illumination and one-bounce indirect illumination. They generally suffer from the energy loss issue due ignoring the high-order indirect illumination. However, indirect lighting from multiple scattering plays significant role in the final appearance [10 of participating media.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this paper, we propose novel neural representation for learning relightable participating media. Our method takes as input a set of posed images with varying but known lighting conditions and designs neural networks to learn a disentangled representation for the participating media with physical properties, including volume density, scattering albedo and phase function parameter. To synthesize images, we embed a differentiable physically-basee ray marching process in the framework. In addition, we propose to simulate global illumination by embedding the single scattering and multiple scattering estimation into the ray marching process, where single scattering is simulated by Monte Carlo ray tracing and the incident radiance from multiple scattering is approximated by spherical harmonics (SH). Without supervising with ground-truie -ruth lighting decomposition, our method is able to learn a decomposition of direct lighting and indirect lighting in an unsupervised manner.
Our comprehensive experiments demonstrate that our method achieves better visual quality and higher numerical performance compared to sfate-of-the-art methods. Meanwhile, our method can generalize to handle solid objects with opaque surfaces. We also demonstrate that our learned neural representations of participating media allow relighting, scene editing and insertion into another virtual environment. To summarize, our approach has the following contributions:
1. We propose a novel method to learn a disentangled neural representation for participating media from posed images and is able generalize to solid objects.
2. Our method deals with both single scattering and multiple scattering and enables the unsupervised decomposition of direct illumination and indirect illumination.
3. We demonstrate flexibility of the learned representation of participating media for relighting, scene editing and scene compositions.
2 Related Work
Neural scene representations. Neural scene representations | are important building blocks for the recent progress in synthesizing realistic images. Different from representations of components such as ambient lighting and cameras 12 13 14 of scenes, neural scene representation [1] learns an embedding manifold from 2D images and scene representation networks 77] aim to infer the 3D context of scenes from images. Classic explicit 3D representations, such as voxels [ 16 6).
multiplane images [17 [8] and proxy geometry [19] are exploited to learn neural representations for specific purposes. These explicit representations generally suffer from the intrinsic resolution limitation. To sidestep the limitation, most recent approaches shift towards implicit representations,
like signed distance fields 20 21 volumetric occupancy fields [22 23 24 25). or coordinate- based neural networks 26
28 By embedding differentiable rendering process like ray 29 into these implicit representations, these methods are capable of optimizing the scene representations from observed images and synthesizing novel views after training. While they generally show improved quality compared to interpolation based novel view synthesis methods [30] 3 the learned representations
27
marching 6 5] or sphere tracing 7
usually texture colors and radiance, without separating lighting and materials. By contrast, we propose to learn a neural representation with disentangled volume density, scattering properties and lighting, which allow the usages in relighting, editing and scene composition tasks.
Volume geometry and properties capture. Acquiring geometry and scattering properties of participating media has long been of the interest to the computer vision and graphics community. Early methods utilize sophisticated scanning and recording devices [32] 2 and specialized lighting patterns B [ to capture volume density. Computational imaging methods [33 34 frame the inference of scattering properties from images as an inverse problem, but they require that the geometries of objects are known. Based on the differentiable path tracing formulation [35]. the inverse transport method (36) incorporates a differentiable light transport
(37) module within an pipeline to infer scattering properties, but it aims for known geometries and homogeneous participating media. In contrast, our method learns the geometries and scattering properties of participating media simultaneously and our method can deal with both homogeneous and heterogeneous participating media.
Relighting. Neural Radiance Field (NeRF) S and its later extensions and radiance into MLPs and leverage ray marching synthesize new views. While they achieve
38 encode the geometry
2

---

realistic results, they are limited to synthesize views under the same lighting conditions as in training. To mitigate this, appearance latent code [39 40) are used condition on the view synthesis. Recent approaches R | decompose materials and lighting by assuming an opaque surface at each point, but this does not apply to participating media. After training, density and materials can be extracted [41 to
render new views using Monte Carlo methods 42 43 44]. Instead, our method models participating media as a field of particles that scatter and absorb light, which are in accordance with its nature. Neural reflectance field 445 8 requires collocated cameras and lights during training and simulates only direct illumination. NeRV [ and OSF [46] simulate merely one-bounce indirect light transport because of the prohibitive computation cost for long paths. However, ignoring the high-order indirect illumination leads to the potential problem of energy loss. By contrast, we use Monte Carlo ray tracing compute direct lighting and propose to learn spherical harmonic field for estimating the complete indirect lighting. The PlenOctree [47] uses spherical harmonics to represent the outgoing radiance field as in NeRF, but it does not allow relighting. With both direct illumination and indirect illumination properly estimated, our method enables principled disentanglement of volume density, scattering properties, and lighting.
Background
Volume rendering. The radiance carried by a ray after its interaction with participating media can be computed based on the radiative transfer equation 48
Lo (ro, ra) -/o (r(()) (r(t)) (r(t) (r(t), -ra) dt.
(1)
Here, r is a ray starting from ro along the direction ra and r(t) = ro + r. denotes point along the ray at the parametric distance f. Lo is the received radiance at To along ra. a denotes the extinction coefficient that is referred to as volume density. The (r())) is the transmittance between r(t) and To and it can be computed by exp (- (r(s)) ds). The L(-) inside the integral stands for the in-scattered radiance towards ro along rra NeRF (5) models the in-scattered radiance (Eq. D as view dependent color but it ignores the underlying scattering event and incident illumination. Since the learned radiance field NeRF bakes in the lighting and materials, allows merely view synthesis under the original fixed lighting, without the support for relighting.
Ray marching. The integral in Equation can be solved with the numerical integration method, ray marching (49 This is generally done by casting rays into the volume and taking point samples along each ray to collect volume density and color values (5 6). The predicted color of a ray is computed by Lo (ro, ra)
E3 Ta ag aj- L (r(tz), -ra). where aj oz).
exp (-o, -g). oj
||ts+1 llll
and r
Neural Relightable Participating Media
In this work, aim to learn neural representations of participating media with disentangled volume density and scattering properties. We model the participating media as a field of particles that absorb and scatter light-carrying rays. Below first describe our disentangled neural representation based on a decomposition with single scattering and multiple scattering. Then, we depict our neural network design, followed by details of volume rendering process for synthesizing images and the details of training.
4.1 Lighting Decomposition
We firstly write the in-scattered radiance L (r(1), -raa) in Equation as an integral of light-carrying spherical directions over 4m steradian range
/
L (r(t), -ra)
S (r(t), -(t,),-ra,,)) wir Lin (r((), w:) dwi,
Daz
Whiiee the integration accounts for a going co, covers only the range with participating media in practice.
3

---

where Lin is the incident radiance at r(t) from the direction wy. S(-) is scattering function which determines the portion of lighting that is deflected towards the -ra- Previous methods (8] 9 41 assume surface prior with a normal at every point and account for 2x hemispherical
incident directions. Accordingly, they define the scattering function S as BRDF function. This assumption, however, can hardly match the participating media objects which have no internal surfaces and normals. By contrast, we deal with light-carrying directions over the full 4m steradian range. Specifically, we define S = =a(((t)) p(-rá, wi.9) .) to account for the scattering over the full spherical directions. Here, a(r(t)) is the scattering albedo. P is the Henyey- Henyey-Greenstein (HG) [50] phase function that decides the scattering directionality (Appendix A), where 9 is asymmetry parameter in (-1,1). .. For brevity, we omit g in the notation of p. Then, we can rewrite the radiance of ray as the disentangled form:
Lo (r) = (r(1)) / ((r(t)))(-raas) Ta. Li) (r((), wi) dw; dt, Pur
(3)
from which the volume density a decides the geometries, and the albedo a along with the phase function p controls the scattering of rays. We propose to train neural networks to learn the volume density and scattering properties.
To compute Lg. we additionally need to estimate Lin (Eq. This can be conducted by recursively substituting the Lo into Lin and expressing the radiance for pixel is integral over all paths Ps = Je Ih (3) { where I is a light-carrying path, Is denotes a measurement contribution function, 1s is the measure for the path space, and V/ is the space of paths with all possible lengths. Ps is then computed by a summation of the contributions of all paths. The contribution of a path with length i be written
PE.: = I. J.J
I Le (x((x=1)) (x((;-1), ((Ph.) df, dw1 ... ....f-111 dw;-1,
(4)
where r((;-1) is point along the (i - 1)-th ray segment with the length 4-1, Wi-1 is the scattering direction in the space and we rı. Le denotes the emitted radiance towards xxt;-1) from light source, V accounts for the transmiltance between x(t;-1) and the light source, (Px,,) = v(((s,, Jo (x((t)) p(-- wj) ())((((()))))) is called the path throughput. Volumetric path tracing 44l utilizes Monte Carlo method to estimate the integral but its computational cost increases quickly when using high sampling rates of paths with many bounces. To reduce the cost, NeRV [9] truncates the paths and only considers up to one indirect bounce, namely i 3 in Eq. 4 This, however, leads to energy loss since high-order indirect illumination are neglected.
Instead of tracing infinitely long paths or truncating the paths, we propose to decompose the in- scattered radiance in Eq. as I. = L., + Lim, where L, is the single-scattering contribution and L., denotes the nuattele-scattering contribution. Therefore, Lo can be split into two integrals: Lo.s [ T (r(t)) a (r(t)) L, dt and Lo, = So T (r(t)) a (r(t)) Lm dt, that can be evaluated separately.
Single seattering. To compute the Lo.s, we evaluate the following integral at each sample point r()) along the camera ray r
L = (
Ja.-
dws.
(5)
Here, Le is the emitted radiance from light source to the point r(t) and V is the transmittance between r(t) and a light source. The transmittance can be computed with another integral of volume density as described in Section 3 but computing the integral for all points leads to high computation cost during training and inference. Therefore, we train visibility neural network to regress the transmittance value as done in 9
Multiple scattering. For Lo.m, we evaluate the Lm = Ss..
(i) wi )dw;
Lm aggregates the incoming radiance of rays that have been scattered at least once in the participating media. Since the distribution of incident radiance from multiple scattering is generally smooth, we propose to represent the incident radiance Lin as spherical harmonics expansion: Lin (w;) =
Both the incident and outgoing directions point away from scattering location in this paper.

---

PE
ViD
Vis.
predict
p PE
Fg
R
Single scatter.
Render loss
e
One-hat
5
S
{cl")
Multiple scatter.
PE
observe
Figure 2: Our overall architecture for learning neural participating media. "PE" denotes the positional encoding and "one-hot" denotes the one-hot encoding. The rendering loss and the visibility loss correspond to the summands in Eq.
F F _m=-, oi' ym (w:)) where lma is the maximum spherical harmonic band, cT" R are spherical harmonic coefficients for the RGB spectrum, Y," are spherical harmonic basis functions and F(x) max(0, z). Therefore, we compute the multiple-scattering contribution with
Lm / a(r(())(-raauu) F Bar
Amn Σ I=0
c (TY dus..
m=-!
We employ neural network to learn spherical harmonic coefficients. By using spherical harmonics for the incident radiance from multiple scattering, we sidestep the lengthy extension of the path integral (Eq. 4. Since we introduce the approximation multiple scattering at the primary rays, we sidestep the explosion of rays. Figure visualizes the explosion of rays when computing indirect illumination under an environment lighting. Multiple shadow rays are needed to account for the directional emission from the light source. The brute-force ray splitting approach leads to explosion of rays and is impractical. NeRV reduces the ray count by tracing up to one indirect bounce, but has complexity of O(M N), where M is the number of first indirect bounces (red) and N is the number shadow rays (blue). Our method uses spherical harmonics to handle indirect illumination whole and its complexity O(k M), where & is the sampling rates along the primary rays.
Network architectures. Figure 2 presents our overall architecture. Our neural networks based the coordinate-bazt MLP strat- egy, and we use frequency- based positional en- coding [5] 51 E to map input coordinate p to a higher dimensional vector E(p) before sending it to the neural networks. Specifically, we use the MIP Ra to predict volume density a (1D), scattering albedo (3D), and asymmetry parameter (1D). Meanwhile, Here, S is conditioned on the point light location Ç. the point light intensity J, and a binary indicator which is one-hot encoded indicate the existence of environment lighting. Since both the property network Rig and the SH network Sg takes as input the encoded coordinate, we introduce a feature network Fg to predict shared features for downstream MLPs. To get the visibility for shadow rays, train another MLP Vg. which takes in the encoded direction d in addition to the encoded coordinate p. to learn visibility values for the estimation of single scattering. summary, we have
Brute force
Figure I: Visualization of path bounces.
employ the MLP S₂ to learn spherical harmonic coefficients e/"
Rg (p() (a, a. So (, - { Vo (r(t), E(d)) -T
Note that the above R. learns per-location asymmetry parameter (Appendix A). Yet, for scenes with single participating media object, singe g and optimize it during training.
5

---

4.2 Volume Rendering
Based on the above decomposition, we employ ray marching (Sec. 3 to numerically estimate Lo,s and Lom- Hence, the final radiance L. of the camera ray in Eq. can be computed as;
Lo (r) = "3=1" (r(()) (1 exp (-a (r(t)) 8tz)) (I, + Lm),
(8)
where we sample N 64 query points in stratified way along the ray r and öt, = (t((++)) r(1;)l2 2|2 is the step size. For each point sample, we query the MLPs to obtain its scattering properties and SH coefficients for computing single scattering and multiple scattering.
We compute single scattering at point based on Eq. 5 We shoot shadow rays towards light sources get the emitted radiance Le according the light types. For environment lighting, we sample 64 directions stratified over a sphere around the point to obtain incident radiance. For a point light, we directly connect the query point the light source. To account for the attenuation of light radiance, we query Ve to get the visibility to the light source from the query point.
For multiple scattering, we uniformly sample K = 64 random incident directions the sphere around each query point, evaluate the incident radiance Lin along each direction using the learned spherical harmonic coefficients, and estimate the integral in Eq. with a Monte Carlo integration Im 1/ K Σ sp(ww)) (u:)). For brevity, we omit the notation. Note that the visibility towards the light source is not needed in the computation.
4.3 End-to-end Learning
Based on the fully differentiable pipeline, we can end-to-end learn neural representation for each scene. The learning requires a set of posed RGB images and their lighting conditions. During each training iteration, single scattering using shadow rays and compute multiple scattering contribution via the learned spherical harmonic coefficients described in Sec. 4.2 We optimize the parameters of Fs. Rø, and Sz by minimizing a rendering loss between the predicted radiance (r)r) from ray marching and the radiance (r) from input images. To train the visibility network Vp. we use the transmittance Vo computed from the learned volume density as the ground truth and minimize the visibility loss between the prediction VPp and the ground truth. Therefore, our loss function includes two parts:
trace primary rays through the media. Along each primary ray, estimate
-=
Σ PER
r((a)))) p. Σ (V((r((), Ve(r(1, ra)22 PER.
where I(L) = L/(1 L) a tone mapping function, R is batch of camera rays and μ 0.1 the hyperparameter to weight the visibility loss.
4.4 Implementation Details
Our feature MLP Fg has 8 fully-connected ReLU (FC-ReLU) layers with 256 channels per layer. The
downstream Sy consists of 8 FC-ReLU layers with 128 channels per layer, whereas the Ra uses one such layer with 128 channels. The visibility MIPP Vi has 4 FC-ReLU layers with 256 channels per layer. We set the maximum positional encoding frequency to 2 for coordinates p, 2 for directions d, and 2 for the 3D location of the point light.
We train all neural networks together to minimize the loss (Eq. 9). We use the Adam 52 optimizer with its default hyperparameters and schedule the learning rate decay from 10 to 1 10-5 over 200K iterations. For each iteration, we trace a batch of 1200 primary rays. Note we stop the gradients from the visibility loss to the property network and the feature network so that they do not compromise the learning to match the visibility network.
5 Experiments
We firstly evaluate our method by comparing it with state-of-the-art methods on simultaneous relighting and view synthesis. Then, we demonstrate that our learned neural representations allow flexible editing and scene compositions, followed by ablation studies of this approach. Please refer to the appendices for additional results.

---

Point
Env
Point
Cow
Cloud
Bunny4-A
Bunny4.
Buddha3
Bi
NeRV
Ours
View GT
Bi et al.
NeRV
Ours
View GT
Figure Qualitative comparisons of simultaneous view synthesis and relighting. The training illumination for the left half is "point" which contains single point light. The training illumination for the right half "env + point". GT denotes the ground truth image.
5.1 Experiment Settings
Compared methods. designed for scenes with solid objects and do not trivially extend to handle participating media, so we implement them with the new functionality to handle participating media. Please refer to Appendix D for the implementation details.
compare our method with state-of-the-art baselines 9 They are
Datasets. We produce datasets from seven synthetic participating media scenes. The Cloud scene is heterogeneous media and the others are homogeneous media. Bunny4- VaryA and Buddha3 are set with spatially varying albedo. Bunny4- VaryG and Buddha3 are configured with spatially varying asymmetry parameters. Each scene is individually illuminated with two lighting conditions. The first one "point" has a white point light with varying intensities sampled within 50 ~ 900 and its location is randomly sampled point" contains a fixed environment lighting and randomly sampled white point light. Each dataset contains 180 images, from which use 170 images for training and the remaining for validation. In addition, we prepare a test set with 30 images for each scene to test the trained models. Each test image is rendered with new camera pose and new white point light that is located on sphere of the radius 4.0. Since Bi's method [8] requires a collocated camera and light during training, we additionally generated such datasets for it. For the "env point" datasets, randomize the usage of environment lighting across images and record a binary indicator for each image.
spheres with the radius ranging from 3.0 to 5.0; The second one "env +
5.2 Results
Relighting comparisons. We show the qualitative comparisons simultaneous relighting and view synthesis on the test data in Fig. 3 The left half is trained with the "point" lighting condition, whereas the right half is trained with the "env + point". Bi's method shows artifacts each case as it has no mechanisms to simulate the multiple scattering that significantly affects the appearance of participating media. NeRV handles the environment illumination properly but shows artifacts on the participating media objects. Our method achieves realistic results on all test sets with

---

Table 1: Quantitative comparisons on the test data for training on the "point" illumination. We
measure image qualities with PSNR (t), SSIM (f) and ELPIPS (4) [53] ELPIPS values below have a scale of x10-2 Note the tabulated values are the mean values over all images of test set.
Point
Cow
Cloud
Bunny4-Varyy VaryA
Bunny4-VaryG
Buddha3
Method
PSNR
SSIM
EL .IPP
PSNR
SSIM
ELPPPS
PSNR
SSIM
ELPIPS
PSNR SSIM
ELPIPS
PSNR
SSIM
EL.PPS
Bi et al.
24.70
0.958
0.465
20.92
0.921
0.783
27.29
0.960
0.378
29.40
0.971
0.334
29.47
0.970
0.299
NeRV
25.20
0.960
0.540
25.68
0.949
0.526
27.67
0.969
0,306
26.76
0.968
0.419
28.69
0.969
0.315
Ours
0.983
0.184
33.51
34.75
0.980
0.189
0.257
33.77
0.975
0.245
Table 2: Quantitative comparisons on the test data for training on the "env + point" datasets.
Env+Point
Cow
Cloud
Bunny4-Vargy
Guunn-4Varyye
Buddha3
Method
PSNR
SSIM
ELPIPS
PSNR
SSIM
ELPIPS
PSNR
SSIM
ELPIPS
PSNR SSIM
ELPIPS
PSNR
SSIM ELPIPS
Bi al.
24.84
0.960
0.501
22.18
0.934
0.709
26.65
0.958
0.464
30.03
0,974
0.285
23.41
0.938
0.679
NeRV
27.83
0.974
0.413
26.07
0.950
0,476
28.18
0.968
0.301
27.97
0.975
0.339
28.99
0.969
0.299
Ours
33.32
0.982
0.209
32.64
0.969
0.353
34.47
0.979
0.191
34,09
0.982
0.243
34.03
0.975
0.261
either homogeneous media heterogeneous media. Table D and Table 2 present the corresponding quantitative measurements, where our method overtakes the compared methods on each test set.
Using the same batch size, training with 200K iterations on Nvidia Quadro RTX 8000 GPU takes one day, whereas Bi's method and NeRV takes 22h and 46h. For a 400 x 400 image, our average inference time is 7.9s, while Bi's method and NeRV takes 53. 28 and 21. 21,9s, respectively.
Learned lighting decom- position. Without using any ground-truth lighting decomposition data, method is able to learn the decomposition of light- ing in an unsupervised way. Figure 4 presents our decomposed results on a test view of the Cow scene, with the single-scattering component (direct lighting) and the multiple-scatterig component (indirect lighting), and the corre- sponding ground-trun trutt images.
drecct
Indrecc
Irdrect
Figure 4: Lighting decompositions.
Scene editing and scene composition. Our method learns neural representations for the participating me- dia scenes. After training, we can query the neural networks to obtain the volume density, the albedo, and the phase function parameter. This allows flexi- ble editing to achieve desired effects or insertion into new virtual environment for content creation. In addition, we can leverage standard rendering engine to render these data. Figure [ the rendering of the learned Bunny with ray marching using the neural network and with a volumetric path tracer. To render with the path tracer, we first queried the neural network to obtain 128 x 128 128 data volumes of volume density and albedo. Both rendered results are visually similar to the ground truth. Figure o demonstrates editing of the red channel of the albedo to achieve the red cloud and another editing of the volume density to make the cloud thinner.
ray marching
path tracer
Figure 5: Ray marching vs. volume path tracing.
We show in Fig. that we can compose a scene con- sisting of our learned cow and gold sculpture de- scribed by traditional meshes and materials (Fig. 7 bottom). Similarly, we can construct scene com- posed entirely of our learned objects (Fig. 7 top). To render the composed scenes, we slice out discrete volumes with resolution 128 128 x 128 from the volume density field and the albedo field and conduct the Monte Carlo rendering in Mitsuba [54]
w/o esi
alicdd
Edit densitv
Figure 6: Edit the learned cloud.
8

---

Biet al.
NeRV
Ours-BRDF
Ours
GT
PSNR 20.20
25.22
28.02
27.82
SSIM
0.907
0.909
0.924
0.994
Figure 8:
Comparisons on test view of the solid Dragon with glossy surfaces. PSNR and SSIM
metrics for this view are listed below images. The training illumination is from single point light and the test illumination is new white point light.
PSNR1
SSIMT
ELPIPS!
Time (s)
SH
25.91
0,9301
0.683
3.91
SH-1
32.53
0.9728
0.175
5.16
SH-3
32.70
0.9739
0.159
5.75
SH--5
32.87
0.9743
0.154
7.91
SH-7
32.80
0.9740
0.152
13.00
SH-9
32.61
0.9739
0.157
16.58
w/o SH
SH
SH 5
SH
GT
Figure 9: Image quality and mean inference timings of different number of spherical harmonic
bands. ELPIPS metrics have scale of 10-2. The full images, including the SH-3 and SH-7, documented in Appendix E.
Scene of solid objects. Beyond the scenes with participating media, our method can be used for scenes with solid objects. Figure shows parison between our method and the baselines on the Dragon scene which contains glossy opaque surfaces. "Ours+BRDF" is a variant of our method that uses SH for indirect illumination, but adopts classical BRDF model 55 and trains the neural network to predict parameters of the BRDF model as method produces an overexposed appearance and the shadow on the floor gets faint. Our method achieves smooth appearance and higher numerical metrics, whereas "Ours BRDF" recovers the highlights on the glossy dragon better.
9. Bi's
Ours
GT
Figure 7: Scene compositions.
5.3 Ablation Studies
Spherical harmonic bands. We analyze the effect of the maximum spherical harmonics band mmoz of Eq. 6 based on the Buddha scene. Figure 9 compares the same view of each case on the left and tabulates the average quality measurements over the test set on the right. Removing the spherical harmonics from our method leads quality drop and color shift is observed in its result. Based on the numerical metrics and inference timings, we select the Imax 5 for other experiments.
Scattering function. We show in Fig. that our method with the HG phase function to a scene of solid objects. In addition, we conduct an ablation by applying the variant "Ours+BRDF" the Bunny scene of participating media. Figure IO shows that the variant has difficulty in learning the volume density and leads to many cracks, while the proposed method performs robustly on this scene.
applied
Variant with BRDF
Ours
Figure 10: Compare Ours+BRDF and Ours with phase function.
Limitations and Future Work
Real-world scenes. In this work, we learn neural
representations from synthetic datascts with varying but known lighting conditions, Also, the camera

---

poses are available to the method. It would be interesting to extend this method to handle participating media captured from In that case, the illumination and camera poses of the scenes need to be estimated in the first place.
scenes with unknown lighting conditions and unknown camera poses.
Glossy reflections. For scenes with glossy solid objects (Fig. 8p. our method tends to reproduce
a smooth appearance and the glossy highlights are not as sharp as the ground truth. An avenue for future research would be develop methods to recover the glossy reflections better.
Generalizability. Our ray marching with the trained neural network generalizes well to unseen light intensities and light locations that are in the range of the training data. That said, its generalization in an interpolation manner. For light intensities and light locations that are outside of the training range, the generalization quality of the neural network gradually decreases. Please refer to the Appendix H for the analysis on the generalization quality.
Media within refractive boundaries. Our method achieves realistic results for participating media without refractive boundaries, like cloud, fog, and wax figure. Applying our method to participating media within refractive boundaries, like wine in a glass, entails further work as the refractive boundaries cause ambiguities due to deflecting the camera rays and the light rays.
7 Conclusion
We have proposed a novel method for participating media reconstruction from observed images with varying but known illumination. We propose to simulate direct illumination with Monte Carlo ray tracing and approximate indirect illumination with learned spherical harmonics. This enables our approach to learn to decompose the illumination as direct and indirect components in unsupervised manner. Our method learns a disentangled neural representation with volume density, scattering albedo and phase function parameters for participating media, and demonstrated its flexible applications in relighting, scene editing and scene compositions.
Acknowledgments and Disclosure of Funding
We acknowledge the valuable feedback from reviewers. This work was supported by Research Executive Agency 739578 and CYENS Phase 2 AE739578.
References
[1] Jinwei Gu, Shree Nayar, Eitan Grinspun, Peter Belhumeur, and Ravi Ramamoorthi. Compressive structured light for recovering inhomogeneous participating media. In European Conference Computer Vision, pages 845 858. Springer, 2008.
[2] Tim Hawkins, Per Einarsson, and Paul Debevec. Acquisition of time-varying participating media. ACM Transactions on Graphics (ToG), 24(3):812 815, 2005.
Christian Fuchs, Tongbo Chen, Michael Goesele, Holger Theisel, and Hans-Peter Seidel. Density estimation for dynamic volumes. Computers & Graphics, 2222:205--11, 2007.
[4] Samuel W Hasinoff and Kiriakos N Kutulakos. Photo-consisten reconstruction of semitransparent scenes by density-shect decomposition. IEEF Transactions on Pattern Analysis and Machine Intelligence, 29(5): 870-885, 2007.
[5] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. European Conference on Computer Vision, pages 405-421. Springer, 2020.
[6] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz. Andreas Lehrmann, and Yaser Sheikh. Neural volumes: learning dynamic renderable volumes from images. ACM Trunsactions on Graphics (TOG), 38(4):1-14, 2019.
[7] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3D ED-structure-awee neural scene representations. In Advances in Neural Information Processing Systems. pages 1121-1132, 2019.
10

---

[8] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Milos Hašan, Yannick Hold- Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008. 03824, 2020.
[9] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron, NeRV: Neural reflectance and visibility fields for relighting and view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7495-7504, 2021.
[10] Yasuhiro Mukaigawa, Yasushi Yagi, and Ramesh Raskar. Analysis of light transport in scattering media. In 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pages 153-160. IEEE, 2010.
[11] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 360(6394):1204 1210, 2018.
[12] Pinar Satilmis. Thomas Bashford- Alan Chalmers, and Kurt Debattista. A weaiiiee--aannnnneeee sky model. IEEE Computer Graphics and Applications, 37(1):80 91, 2016.
[13] Quan Zheng and Changwen Zheng. NeuroLens: Data-driven camera lens simulation using neural networks. Computer Graphics Forum, 390 2017.
[14] Quan Zheng and Changwen Zheng. Adaptive sparse polynomial regression camera lens simulation. The Visual Computer, (1866-8)77157722 2017.
[15] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. DeepVoxels: Learning persistent 3D feature embeddings. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2437-2446. 2019.
[16] Songyou Peng. Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 523 540. Springer, 2020.
[17] John Flynn, Michael Broxton, Paul Debevec, Matthew Vall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Viep:iew: View synthesis with learned gradient descent. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3376, 2019.
[18] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images. ACM Transactions on Gruphics (TOG), 37(4):1-12. 2018.
[19] Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts- Escolano, Philip Davidson. Christoph Rhemann, Paul Debevec, et al. Neural light transport for relighting and view synthesis. ACM Transactions on Graphics (TOG), 40(1):1-17, 2021.
[20] Jeong Joon Park, Peter Florence, Julian Straub. Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[21] Rohan Chabra, Jan E Lenssen, Eddy Ilg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe, Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In European Conference on Computer Vision, pages 608-625. Springer, 2020.
[22] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9939-5948, 2019.
[23] Lars Mescheder, Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460 4470, 2019.
[24] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33, 2020.
[25] Shunsuke Saito, Zeng Huang. Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel- Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 2304-2314, 2019.
[26] Qiangeng Xu, Weiyue Wang. Duygu Ceylan, Radomir Mech, and UIrich Neumann. DISN: Deep implicit surface network for high quality single-view 3D reconstruction. Advances in Neural Information Processing Systems, $22492-502, 2019.
11

---

[27] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
[28] Quan Zheng, Vahid Babaei. Gordon Wetzstein, Hans-Peter Seidel. Matthias Zwicker, and Gurprit Singh. Neural light field 3D printing. ACM Transactions Graphics (TOG), 39(6):1-12, 2020.
[29] Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Mare Pollefeys, and Zhaopeng Cui. DIST: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2019-2028, 2020.
[30] Pratul P Srinivasan, Tongzhou Wang, Ashwin Sreelal, Ravi Ramamoorthi, and Ren Ng. Learning to synthesize a 4D RGBD light field from a single image. In Proceedings of the IEEE International Conference on Computer Vision, pages 2243 2251, 2017.
[31] Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely, Pushing the boundaries of view extrapolation with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 175 184, 2019.
[32] Michael Goesele, Hendrik P. A. Lensch, Jochen Lang, Christian Fuchs, and Hans-Peter Seidel. DISCO: Acquisition of translucent objects. In ACM SIGGRAPH 2004 Papers, pages 835-844, New York, NY, USA, 2004. Association for Computing Machinery.
[33] loannis Gkioulekas, Anat Levin, and Todd Zickler. An evaluation of computational imaging techniques for heterogeneous inverse scattering. In European Conference Computer Vision, pages 685-701. Springer, 2016.
[34] loannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. Inverse volume rendering with material dictionaries. ACM Transactions on Graphics (TOG), 32(6):1-13, 2013.
[35] Dejan Azinovic, Tzu-Mao Li, Anton Kaplanyan, and Matthias NieBner. Inverse path tracing for joint material and lighting estimation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2447-244 2456, 2019.
[36] Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and loannis Gkioulekas. Towards learning-based inverse subsurface scattering. In 2020 IEEE International Conference on Computational Photography (ICCP), pages 12. IEEE, 2020.
[37] Cheng Zhang, Lifan Wu, Changxi Zheng, loannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. differential theory of radiative transfer. ACM Transactions on Graphics (TOG), 38(6):1-16, 2019.
[38] Frank Dellaert and Lin Yen-Chen. Neural volume rendering: NeRF and beyond. arXiv preprint Siki::2001.02204 2020.
[39] Ricardo -rruala, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210-7219. 2021.
[40] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. GRAF: Generative radiance fields for 3D-aware image synthesis. Advances in Neural Information Processing Systems, 33, 2020.
[41] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. NeRD: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12684 12694, 2021.
[42] Wojciech Jarosz, Derek Nowrouzezabrai. Iman Sadeghi, and Henrik Wann Jensen. A comprehensive theory of volumetric radiance estimation using photon points and beams. ACM Transactions Graphics (TOG), 30(1):1-19, 2011.
[43] Quan Zheng and Chang Wen Zheng. Visual importance-basee adaptive photon tracing. The Visual Computer, 100110000000100 2015.
Chang-Wen
[44] Jan Novák, lliyan Georgiev, Johannes Hanika, and Wojciech Jarosz. Monte Carlo methods for volumetric light transport simulation. Computer Graphics Forum, 37(2):551-576, 2018
[45] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold Geoffroy, David Kriegman, and Ravi Ramamoorthi. Deep reflectance volumes: Relightable reconstructions from multi view photometric images. In Computer -idon-ECVV22 2020: 16th European Conference, Glasgow, UK, August 23-28. 28, 2020, Proceedings, Part III 16, pages 294-311. Springer, 2020.
12

---

[46] Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser: Object-centric neural scene rendering. arXiv preprint arXiv:2012.08503. 2020.
[47] Alex Yu, Ruilong Li, Matthe Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
[48] Eric P Lafortune and Yves D Willems. Rendering participating media with bidirectional path tracing. In Eurographics Workshop on Rendering Techniques, pages 91-100. Springer, 1996.
[49] Joe Kniss, Simon Premoze, Charles Hansen, Peter Shirley, and Allen McPherson. A model for volume lighting and modeling. IEEE Transactions on Visualization and Computer Graphics, 122) 2003.
[50] Louis G Henyey and Jesse Leonard Greenstein. Diffuse radiation in the galaxy. The Astrophysical Journal, 93:70-83, 1941.
[51] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Kan -eil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions low dimensional domains. NeurlPS, 2020.
[52] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Truck Proceedings, 2015.
[53] Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. E-LPIPS: robust perceptual image similarity via random transformation ensembles. arXiv preprit arXiv: 1906.03973 2019.
[54] Wenzel Jakob. Mitsuba renderer, 2010. hutp:/wwww.iisubb.renderrrrrr
[55] Brian Karis. Real shading in unreal engine 4. Proc. Physically Based Shading Theary Practice, 4:3, 2013.
13


Reviews:
[{"review_id": "kxTdwTg0WQa", "reviewer": "Reviewer_8Z2s", "summary": "The paper proposes a method to learn a volumetric representation for scenes with \nparticipating media from a set of images. It builds on the NeRF architecture. \nThe difference part is that it applies a phase function to model how light is \nscattered at point instead of using radiance field or BRDF. It also models the \nincident multi-scatterings at each point using spherical harmonics to avoid path \ntracing. They show better results than previous methods and demo applications   such as scene composition and editing.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strengths\n1. The usage of spherical harmonics to model incident lighting is technically sound,\nand reduces the computation cost to account for multi-scatterings.\n\n2. The results on synthetic data are better than NRF and NeRV, and better match the \nground truth.   \n\n### Weakness\n1. The paper says that spherical harmonics can better represent the multi-scattering \nterms. If that's the case, I would expect if the method changes the phase function to \nBRDF, it would achieve better performance than NeRV, since the proposed method has \na more accurate lighting representation. However, from Figure 6, and the Figure at \nLine 325, the proposed method is worse than NeRV. It make me wonder whether the proposed \nmulti-scattering representation is really beneficial. Why does the proposed method not \nwork well with BRDF even for solid objects?\n\n2. At Line 280, the paper shows the images under single-scattering and multi-scattering. \nWithout giving the corresponding ground truth, it's difficult to interpret whether the \nshown images are correct or not. In fact, I feel that the single-scattering is too dark.\nFor points at the surface, I would expect the single-scattering plays a dominant role.\nHowever, it seems multi-scattering is much brighter than single-scattering. To really validate \nthis, ground truth images need to be provided.  \n\n3. While the proposed method estimates per-point albedo, all the synthetic scenes\nhave a constant albedo. How does the proposed method work for participating media with \nvarying albedo?\n\n4. To predict the SH coefficients, the network takes in the point-light position and intensity \nas input. This does not make sense to me. I don't think such a design can generalize to \narbitrary lighting intensity and position, considering that you only have a single set of \nlighting condition for the training images? \nAlso what if you have multiple point lights with an envmap in training?  In fact, it's not clear to \nme how the proposed method performs the testing.  How is the multi-scattering rendered \nduring relighting? Given a new lighting condition, the SH for multi-scattering \nwould change, and the originally predicted SH cannot be used. Is the paper using path tracers \nto generate images in Figure 3, 4? Or the SH are re-predicted with the new lighting condition? \n\n5. For the renderings with Mitsuba, the objects such as the bunny and the buddha are more \ntranslucent than they should be. What is the reason?\n\n6. The qualitative results are not fully convincing. The generated results tend to be blurry and \nmissing details, also the shading looks incorrect when the light is at an grazing \nangle (video 02:28). While the paper acknowledges the ringing artifacts, such artifacts \nalso limit the contributions of the paper. \n\nOverall, while I agree the usage of SH for multi-scatterings is reasonable, I am doubtful about \nthe design of the network to predict the SH and whether it can generalize to arbitrary lighting \nposition and intensity. The quality of the results also needs further improvement, and more \nvalidations against GT under more diverse lighting conditions are also needed.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "c9u2Vc_jywH", "reviewer": "Reviewer_2Q3Q", "summary": "The authors present a neural implicit representation for participating media. This requires (i) extending neural radiance field approaches that are limited to a fixed lighting and therefore less flexible regarding relighting and scene editing, or (ii) overcoming methods for disentangling reflectance and illumination that are typically limited to solid objects without subsurface scattering and direct lighting. The presented approach takes input data in terms of a set of images with camera calibration data taken under varying but known lighting conditions. Then they use a NeRF-like approach for learning a disentangled representation for the participating media with volume density and scattering albedo. Here, they use a differentiable physically-based ray marching where they embed single scattering (using Monte Carlo ray tracing) and multiple scattering (using spherical harmonics approximation) separately to handle global illumination at reduced computational burden. This allows learning the decomposition of direct lighting and indirect lighting in an unsupervised manner.\n\nIn their evaluation, the authors demonstrate the learned representation of participating media to outperform related state-of-the-art techniques (in both quantitative and qualitative comparisons) and to be suited for scene relighting, scene editing and scene composition.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n- The method seems novel and reasonable with plausible results for the datasets and offers benefits over state-of-the-art techniques regarding quality and inference time.\n\n\nEvaluation\n- The authors provide quantitative and qualitative comparisons for a diverse set of examples including homogeneous and inhomogeneous media.\n- They also include related techniques that they adapted to handle scattering characteristics and show that their proposed approach outperforms these approaches.\n- Further results address the editing of scene characteristics such as editing of albedo and density or scene compositing where the proposed approach provides reasonable results.\n- Ablation studies address the impact of the number of spherical harmonics bands and exchanging the scattering function.\n- The authors provide a discussion of limitations. Besides relying on known illumination and known camera poses, there are some further limitations. For solid glossy objects, the reconstruction preserves the base colors but faces difficulties in preserving the specular highlights. The consideration of refractive boundaries is left for future work. Still, this seems to be ok for the current submission.\n\n\nExposition\n- The paper is well-structured and readable. Figures/tables and captions are informative.\n\n\nReproducibility\n- The paper seems reproducible for a more experience person. The authors mention that they intend to release the code.\n\n\nReferences\n- References seem to be adequate.\n\n\nPost-rebuttal comments:\nI appreciate that the authors share further insights on other reviewers’ comments, report results for tests on scenes of solid objects and promise to release code and data. I remain at my previous recommendation \"7: Good paper, accept\".\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide a discussion of limitations. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "QHzDUkMpJM", "reviewer": "Reviewer_wnsz", "summary": "This paper proposes a (inverse) neural rendering method for participating media that can disentangle volume density, scattering albedo and illumination from posed images and known light positions. Compared with previous NeRF methods, such as NeRF and NeRV, this approach can better deal with scattering media, such as fog and wax, in the meanwhile, generalizing to solid material. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n\n1. As for (inverse) neural rendering for participating media, the learning-based problem formulation is novel, and this method can jointly estimate volume density, scattering albedo and lighting in an unsupervised manner. \n   \n2. Compared with previous NeRF-based methods, the proposed method can capture more accurate global illuminations of participating media.\n\n3. A new synthetic training and evaluation dataset for participating media is proposed.\n\n\n### Weaknesses:\n\n1. My major concern about this paper is lack of experiments on real data, and the authors mentioned this in limitations as well. Because the real-world environment and material are much more complex than simulation, it is hard to tell how well this method can generalize to real-world scenes.\n\n2. The writing needs polish, e.g., the authors listed four contributions in L53-60, but they are basically saying the same thing, and I think the 2nd and 3rd ones can be merged. As for the 4th contribution, the authors can emphasize that the proposed method outperforms previous arts. In addition, L203 mentioned parameter g in Eq. 3, but I did not find g in Eq.3.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See weaknesses in main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "G4tGrPVeRjY", "reviewer": "Reviewer_uER8", "summary": "This paper presents a neural volumetric inverse rendering approach for reconstructing a 3D representation of a scene with participating media (volumetric scattering) that can be rendered from novel viewpoints under novel lighting conditions. The paper builds on NeRV (an extension of NeRF that estimates relightable models of opaque objects) and modifies NeRV so that it can also recover relightable models of scenes with participating media. The main contributions of the paper are replacing the BRDF used in NeRV with a parametric phase/scattering function, and a technique to efficiently account for multiple scattering by training an MLP that predicts the \"global illumination\" at any location, represented by spherical harmonics, that is conditioned on a representation of the incoming lighting. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nI think this paper is generally well-written, and the proposed method is logical and well-motivated. The experimental results validate the main idea of the paper that extending NeRV to handle participating media (by changing the BRDF to a scattering phase function and including a way to handle full indirect illumination) produces better results than existing techniques. \n\nWeaknesses:\n\nI'm a bit confused at the idea of conditioning the network that predicts spherical harmonics coefficients for multiple scattering on the binary e that represents the existence of environment lighting (lines 197-198). Shouldn't the multiple scattering depend on the actual environment lighting distribution, intensity, and color? How would the scene be rendered under arbitrary environment maps? Could the authors please discuss this?\n\nI'm curious how this method would work when the asymmetry parameter g and volumetric albedo can also vary across the scene. For example, what if there are two volumetric objects with different properties next to each other?\n\nAre there any results where the scattering albedo is not constant across the entire volume? I think that showing such experiments would make the visual results much more compelling and convincing. Page 7 mentions that the cloud scene is heterogeneous, but I think that choosing an example with more obvious albedo and asymmetry parameter changes across the scene would be more convincing.\n\nI'm curious why \"Ours+BRDF\" seems to work worse than \"Ours\" in the dragon in Figure 6 where the material should be better modeled as a BRDF than as a phase function. Could the authors please discuss this more?\n\nMinor edits/suggestions:\n\nI would recommend changing the title to remove the word \"Implicit\", as this may be confusing since there is no implicit surfaces in this method. Maybe \"Neural Volumetric Relightable Participating Media\" could be better? \n\nLine 41 mentions that this method learns a representation with volume density and scattering albedo. This might confuse readers by suggesting you are assuming a single known phase function, so I suggest clarifying this and maybe mentioning that the phase function parameter is optimized.\n\nI would recommend not calling NeRF-like methods that use MLPs to represent continuous volumes \"implicit\". It makes more sense to call neural occupancy fields or SDFs \"implicit\" because they represent shapes as implicit functions (surfaces are level sets of a function), but NeRF is not trying to represent a single surface so I don't recommend calling it \"implicit\".\n\nLine 68: most approaches --> most recent approaches\n\nLine 79: the interest of the community --> of interest to the computer vision community\n\nLine 83: based on the inverse rendering fashion --> based on differentiable pathtracing\n\nLine 88: later enhancement --> extensions\n\nLine 88: I'm not sure why just these two specific NeRF follow-ups are cited here, when there has been a great deal of extensions to NeRF (https://github.com/yenchenlin/awesome-NeRF plus even more). \n\nLine 115: I'm not sure what \"glosses\" is supposed to mean here.\n\nLine 190: fashion --> technique/strategy\n\nI recommend proofreading the references section to fix references that are cited as \"arXiv\" but were actually published at other venues. For example, [9] was published at CVPR, [17] was published at SIGGRAPH, [23] was published at NeurIPS, etc.\n\n[5], [9], and [41] should be capitalized as NeRF, NeRV, and NeRD in the references. [16] should be capitalized as DeepView. \"3D\" in [7] should be capitalized.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does a reasonable job discussing limitations, including limitations in the type of input data required and limitations in performance on certain types of scenes.\n\nThe paper does not discuss potential negative societal impacts. However, I do not think that this necessarily applies to the paper, because the paper addresses a more theoretical problem, and the recovery of relightable volumetric models from images does not seem to have any immediate safety/privacy/fairness concerns.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews were split: 3 reviewers gave 7 and one gave 4. Those who supported acceptance acknowledge the novelty of the paper as the method can jointly estimate volume density, scattering albedo and lighting in an unsupervised manner, and can capture more accurate global illuminations of participating media. The reviewer who gave 4 had concerns that he proposed method's using a network to predict the spherical harmonics approximation of indirect illumination/multiple scattering does not necessarily generalize well, and there was lengthy discussion between the authors and this reviewer, who was not convinced eventually. The AC agrees with the majority that the paper presented an interesting extension of previous "relightable NeRF" to participating media. The novelty and results warrant acceptance. 
Decision: Accept (Poster)