Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a (inverse) neural rendering method for participating media that can disentangle volume density, scattering albedo and illumination from posed images and known light positions. Compared with previous NeRF methods, such as NeRF and NeRV, this approach can better deal with scattering media, such as fog and wax, in the meanwhile, generalizing to solid material. 
main_review: ### Strengths:

1. As for (inverse) neural rendering for participating media, the learning-based problem formulation is novel, and this method can jointly estimate volume density, scattering albedo and lighting in an unsupervised manner. 
   
2. Compared with previous NeRF-based methods, the proposed method can capture more accurate global illuminations of participating media.

3. A new synthetic training and evaluation dataset for participating media is proposed.


### Weaknesses:

1. My major concern about this paper is lack of experiments on real data, and the authors mentioned this in limitations as well. Because the real-world environment and material are much more complex than simulation, it is hard to tell how well this method can generalize to real-world scenes.

2. The writing needs polish, e.g., the authors listed four contributions in L53-60, but they are basically saying the same thing, and I think the 2nd and 3rd ones can be merged. As for the 4th contribution, the authors can emphasize that the proposed method outperforms previous arts. In addition, L203 mentioned parameter g in Eq. 3, but I did not find g in Eq.3.
limitations_and_societal_impact: See weaknesses in main review.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a method to learn a volumetric representation for scenes with 
participating media from a set of images. It builds on the NeRF architecture. 
The difference part is that it applies a phase function to model how light is 
scattered at point instead of using radiance field or BRDF. It also models the 
incident multi-scatterings at each point using spherical harmonics to avoid path 
tracing. They show better results than previous methods and demo applications   such as scene composition and editing.

main_review: 
### Strengths
1. The usage of spherical harmonics to model incident lighting is technically sound,
and reduces the computation cost to account for multi-scatterings.

2. The results on synthetic data are better than NRF and NeRV, and better match the 
ground truth.   

### Weakness
1. The paper says that spherical harmonics can better represent the multi-scattering 
terms. If that's the case, I would expect if the method changes the phase function to 
BRDF, it would achieve better performance than NeRV, since the proposed method has 
a more accurate lighting representation. However, from Figure 6, and the Figure at 
Line 325, the proposed method is worse than NeRV. It make me wonder whether the proposed 
multi-scattering representation is really beneficial. Why does the proposed method not 
work well with BRDF even for solid objects?

2. At Line 280, the paper shows the images under single-scattering and multi-scattering. 
Without giving the corresponding ground truth, it's difficult to interpret whether the 
shown images are correct or not. In fact, I feel that the single-scattering is too dark.
For points at the surface, I would expect the single-scattering plays a dominant role.
However, it seems multi-scattering is much brighter than single-scattering. To really validate 
this, ground truth images need to be provided.  

3. While the proposed method estimates per-point albedo, all the synthetic scenes
have a constant albedo. How does the proposed method work for participating media with 
varying albedo?

4. To predict the SH coefficients, the network takes in the point-light position and intensity 
as input. This does not make sense to me. I don't think such a design can generalize to 
arbitrary lighting intensity and position, considering that you only have a single set of 
lighting condition for the training images? 
Also what if you have multiple point lights with an envmap in training?  In fact, it's not clear to 
me how the proposed method performs the testing.  How is the multi-scattering rendered 
during relighting? Given a new lighting condition, the SH for multi-scattering 
would change, and the originally predicted SH cannot be used. Is the paper using path tracers 
to generate images in Figure 3, 4? Or the SH are re-predicted with the new lighting condition? 

5. For the renderings with Mitsuba, the objects such as the bunny and the buddha are more 
translucent than they should be. What is the reason?

6. The qualitative results are not fully convincing. The generated results tend to be blurry and 
missing details, also the shading looks incorrect when the light is at an grazing 
angle (video 02:28). While the paper acknowledges the ringing artifacts, such artifacts 
also limit the contributions of the paper. 

Overall, while I agree the usage of SH for multi-scatterings is reasonable, I am doubtful about 
the design of the network to predict the SH and whether it can generalize to arbitrary lighting 
position and intensity. The quality of the results also needs further improvement, and more 
validations against GT under more diverse lighting conditions are also needed.


limitations_and_societal_impact: See the main review.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present a neural implicit representation for participating media. This requires (i) extending neural radiance field approaches that are limited to a fixed lighting and therefore less flexible regarding relighting and scene editing, or (ii) overcoming methods for disentangling reflectance and illumination that are typically limited to solid objects without subsurface scattering and direct lighting. The presented approach takes input data in terms of a set of images with camera calibration data taken under varying but known lighting conditions. Then they use a NeRF-like approach for learning a disentangled representation for the participating media with volume density and scattering albedo. Here, they use a differentiable physically-based ray marching where they embed single scattering (using Monte Carlo ray tracing) and multiple scattering (using spherical harmonics approximation) separately to handle global illumination at reduced computational burden. This allows learning the decomposition of direct lighting and indirect lighting in an unsupervised manner.

In their evaluation, the authors demonstrate the learned representation of participating media to outperform related state-of-the-art techniques (in both quantitative and qualitative comparisons) and to be suited for scene relighting, scene editing and scene composition.

main_review: Originality
- The method seems novel and reasonable with plausible results for the datasets and offers benefits over state-of-the-art techniques regarding quality and inference time.


Evaluation
- The authors provide quantitative and qualitative comparisons for a diverse set of examples including homogeneous and inhomogeneous media.
- They also include related techniques that they adapted to handle scattering characteristics and show that their proposed approach outperforms these approaches.
- Further results address the editing of scene characteristics such as editing of albedo and density or scene compositing where the proposed approach provides reasonable results.
- Ablation studies address the impact of the number of spherical harmonics bands and exchanging the scattering function.
- The authors provide a discussion of limitations. Besides relying on known illumination and known camera poses, there are some further limitations. For solid glossy objects, the reconstruction preserves the base colors but faces difficulties in preserving the specular highlights. The consideration of refractive boundaries is left for future work. Still, this seems to be ok for the current submission.


Exposition
- The paper is well-structured and readable. Figures/tables and captions are informative.


Reproducibility
- The paper seems reproducible for a more experience person. The authors mention that they intend to release the code.


References
- References seem to be adequate.


Post-rebuttal comments:
I appreciate that the authors share further insights on other reviewers’ comments, report results for tests on scenes of solid objects and promise to release code and data. I remain at my previous recommendation "7: Good paper, accept".

limitations_and_societal_impact: The authors provide a discussion of limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a neural volumetric inverse rendering approach for reconstructing a 3D representation of a scene with participating media (volumetric scattering) that can be rendered from novel viewpoints under novel lighting conditions. The paper builds on NeRV (an extension of NeRF that estimates relightable models of opaque objects) and modifies NeRV so that it can also recover relightable models of scenes with participating media. The main contributions of the paper are replacing the BRDF used in NeRV with a parametric phase/scattering function, and a technique to efficiently account for multiple scattering by training an MLP that predicts the "global illumination" at any location, represented by spherical harmonics, that is conditioned on a representation of the incoming lighting. 
main_review: Strengths:

I think this paper is generally well-written, and the proposed method is logical and well-motivated. The experimental results validate the main idea of the paper that extending NeRV to handle participating media (by changing the BRDF to a scattering phase function and including a way to handle full indirect illumination) produces better results than existing techniques. 

Weaknesses:

I'm a bit confused at the idea of conditioning the network that predicts spherical harmonics coefficients for multiple scattering on the binary e that represents the existence of environment lighting (lines 197-198). Shouldn't the multiple scattering depend on the actual environment lighting distribution, intensity, and color? How would the scene be rendered under arbitrary environment maps? Could the authors please discuss this?

I'm curious how this method would work when the asymmetry parameter g and volumetric albedo can also vary across the scene. For example, what if there are two volumetric objects with different properties next to each other?

Are there any results where the scattering albedo is not constant across the entire volume? I think that showing such experiments would make the visual results much more compelling and convincing. Page 7 mentions that the cloud scene is heterogeneous, but I think that choosing an example with more obvious albedo and asymmetry parameter changes across the scene would be more convincing.

I'm curious why "Ours+BRDF" seems to work worse than "Ours" in the dragon in Figure 6 where the material should be better modeled as a BRDF than as a phase function. Could the authors please discuss this more?

Minor edits/suggestions:

I would recommend changing the title to remove the word "Implicit", as this may be confusing since there is no implicit surfaces in this method. Maybe "Neural Volumetric Relightable Participating Media" could be better? 

Line 41 mentions that this method learns a representation with volume density and scattering albedo. This might confuse readers by suggesting you are assuming a single known phase function, so I suggest clarifying this and maybe mentioning that the phase function parameter is optimized.

I would recommend not calling NeRF-like methods that use MLPs to represent continuous volumes "implicit". It makes more sense to call neural occupancy fields or SDFs "implicit" because they represent shapes as implicit functions (surfaces are level sets of a function), but NeRF is not trying to represent a single surface so I don't recommend calling it "implicit".

Line 68: most approaches --> most recent approaches

Line 79: the interest of the community --> of interest to the computer vision community

Line 83: based on the inverse rendering fashion --> based on differentiable pathtracing

Line 88: later enhancement --> extensions

Line 88: I'm not sure why just these two specific NeRF follow-ups are cited here, when there has been a great deal of extensions to NeRF (https://github.com/yenchenlin/awesome-NeRF plus even more). 

Line 115: I'm not sure what "glosses" is supposed to mean here.

Line 190: fashion --> technique/strategy

I recommend proofreading the references section to fix references that are cited as "arXiv" but were actually published at other venues. For example, [9] was published at CVPR, [17] was published at SIGGRAPH, [23] was published at NeurIPS, etc.

[5], [9], and [41] should be capitalized as NeRF, NeRV, and NeRD in the references. [16] should be capitalized as DeepView. "3D" in [7] should be capitalized.
limitations_and_societal_impact: The paper does a reasonable job discussing limitations, including limitations in the type of input data required and limitations in performance on certain types of scenes.

The paper does not discuss potential negative societal impacts. However, I do not think that this necessarily applies to the paper, because the paper addresses a more theoretical problem, and the recovery of relightable volumetric models from images does not seem to have any immediate safety/privacy/fairness concerns.
review_title: null

Meta-review: The reviews were split: 3 reviewers gave 7 and one gave 4. Those who supported acceptance acknowledge the novelty of the paper as the method can jointly estimate volume density, scattering albedo and lighting in an unsupervised manner, and can capture more accurate global illuminations of participating media. The reviewer who gave 4 had concerns that he proposed method's using a network to predict the spherical harmonics approximation of indirect illumination/multiple scattering does not necessarily generalize well, and there was lengthy discussion between the authors and this reviewer, who was not convinced eventually. The AC agrees with the majority that the paper presented an interesting extension of previous "relightable NeRF" to participating media. The novelty and results warrant acceptance. 
Decision: Accept (Poster)