Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method to estimate the geometry, material and lighting 
of an object from a set of multi-view images. It builds on top of previous work 
nvdiffrec. Different from nvdiffrec that applies split-sum approximation for 
direct lighting, the proposed method applies Monte Carlo ray tracing  and make the rendering more physically accurate. To reduce the 
noise caused by Monte Carlo integration when
there are few samples, the paper proposed to use denoisers to reduce the
variance and make the training more efficient. The experiment results show that 
the proposed method generates better results than baseline methods such as
nvdiffrec and nerfactor.

strengths_and_weaknesses: Strengths:

The paper combines neural inverse rendering and traditional Monte Carlo ray
tracing, which enjoys the benefits of both world.
The idea of using denoisers to reduce variance in Monte Carlo integration 
is interesting and helps make the training more efficient. The paper did
multiple experiments to show that it helps improve material, geometry and 
lighting estimation.

Weakness/Questions:
1. As mentioned in the paper, while using Monte Carlo integration, the paper 
didn't consider multi-bounces or shadow gradients, which makes the method fail
to handle indirect illumination. As a stress test, it would be interesting to
see how it works when increasing the number of bounces.

2. For shadow gradients, how the light leakage term is set? Is it a fixed
parameter for all experiments? 

3. In Section 3 of the supp, it's not clear to me how the visibility gradients
are calculated?  Is the paper using the methods based on warped area/edge sampling? 
It's worth adding a more detailed discussion here to differentiate the referred
methods and the method used by the paper.

4. How robust is the denoiser in the optimization? At the early stage of the
optimization, when the rendering images don't make too much sense, will denoiser
make the optimization worse? The paper talks about ramping up the spatial
footprint and linearly blending the noisy and denoised images in supp (Line 60), 
how robust are those operations? Do they need to be fine-tuned for each scene?

5. During optimization, since it minimizes the difference between the denoised
images and the GT images, I am wondering whether it's possible that the noisy 
images have some artifacts that are not caused by low samples but incorrect
estimations. Such artifacts will be covered by the denoiser, and will be visible
during testing. Is such a case possible? During testing, can we get high-quality
noise-free results by increasing the number of samples without using denoisers? 


6. In Line 266, the paper says view interpolation degrades with improved
material and lighting estimations. More materials/evidence are needed to support this
claim. 

Overall, I think the idea of combining neural inverse rendering, traditional 
Monte Carlo integration and denoisers is interesting and novel. The paper did 
thorough evaluations on the method and performs detailed ablation study. 
Therefore, I vote for accepting the paper.


questions: See above.
limitations: The limitations are well discussed.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an algorithm for the inverse rendering based on the neural fields. Unlike existing neural-fields-based inverse rendering relies on the simple shading model, the paper introduced the ray tracing with Monte Carlo integration that enables the inverse rendering more general and practical. To make the optimization tractable, the paper proposed to introduce multiple importance sampling and denoising for the differentiable ray tracing. The extensive evaluation shows that the proposed method can accurately recover physical attributes in more physically interpretable manner.
strengths_and_weaknesses: Strength:

- The introduction of this work is very clear which is supported by an extensive survey of previous studies. 

- Utilizing the denoiser and importance sampling for stable and efficient differentiable rendering makes a lot of sense. I expect that this research will lead the computer vision community the active utilization of real-time rendering technics, which are being actively researched and developed in the field of computer graphics.

- Though most of the techniques for forward-rendering in this paper came from previous works, but their application to inverse rendering is not obvious. I think the beauty of this paper is that in every technical component, the reader can learn why it was introduced, including its surroundings thanks to the proper citations.

- The evaluation successfully demonstrated the benefit of the proposed denoising.

- The extensive ablation study justifies the algorithmic choice. It is really helpful to me to see the comparison of denoisers with detailed discussions in the supplementary material.

Weakness:

- As described in the paper, the method still considers the single bounce of light, therefore essentially inter-reflection is still a problem.

- Though qualitatively demonstrated, the reconstruction accuracy of individual attributes (such as normal or specular components) was not quantitatively evaluated.

- As the authors repeatedly mentioned, the training time is not as fast as expected even with the small sample count per pixel.

- Experiments show that the proposed method does a better job of recovering materials and lighting than the existing methods, but on the other hand, the increase in computational cost is an issue. Therefore, it is very helpful to clarify which kinds of scenes the existing method is not good at and the proposed method is good at handling so that a user can choose the proper method depending on the scene.

- The writing of the paper is very good, and the proposed method is undoubtedly very effective in recent inverse rendering based on Neural Fields. On the other hand, however, I had the impression that the paper uses existing techniques at a very high level rather than making innovative proposals, so I have slightly downgraded it from the highest evaluation.
questions: - The paper claimed that the proposed method scales linearly with sample count but the Table at line 275 shows the iteration time doesn’t linear to the sample count. What does “scale linearly” mean?

- Can the proposed method be applied to the recovery of translucent or transparent objects with some realistic constraints (e.g., known background)?

limitations: The limitation is properly described.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles the challenge of learning the shape, light & material properties of a scene from multiview images. The main idea is to consider a general lighting model for the rendering equation, which in turn is approximated using Monte Carlo (MC). As MC introduces large variance gradient estimates for optimization, the paper suggests importance sampling techniques. In addition, an image denoising model is incorporated into the system. The method is evaluated on standard scene datasets.
strengths_and_weaknesses: Strengths
-------------
The paper is well written and easy to follow. The introduction is concisely informative.
The suggested system seems to improve previous work in learning scene properties decomposition.
The evaluation of synthetic data and real data is adequate.

Weaknesses
----------------

The paper focuses on the different solutions the proposed system consists of. However, I am missing a discussion on some of the challenges or design choices made in designing such a system. For example, the choice of the importance sampling techniques is given with any discussion. Some questions that arise are: Can the variance reduction be somehow quantified (bounded above)? Are there other existing techniques?
Another example of a missed discussion is how much the work is coupled with NVDIFFREC. Could the suggested techniques be incorporated into methods like [1] as well? 

In addition, the discussion on the bias introduced (154-162) is unclear. The seems like an important issue that can also promote further research but is very shortly discussed. 

The paper could be improved by being more self-contained. For example, the sampling techniques used are only referenced.

The effect of the new regularizer is not tested in an ablation study. 

[1]: PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting, CVPR 2021.
[2]: 
questions: Please address the weakness stated above.
limitations: The paper does not include such a discussion.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes an inverse rendering pipeline for recovering geometry, reflectance and environment map from a set of multi-view images. The key contributions is a denoising step that significantly reduces variance of Monte Carlo light integration at lower sampling rates. The method achieved superior results compared to a direct lighting baseline (nvdiffrec) and Nerfactor.
strengths_and_weaknesses: Strengths:

- The idea to denoise Monte Carlo rendering is well motivated and well implemented. To the best of my knowledge this idea is original.
- Results are convincing. Particularly the comparison with nvdiffrec baseline shows considerable improvement in novel rendering and relighting. Compared to SG and SH representations, the recovered environment map contains high fidelity details, evident in Figure 2 in the appendix.
- Limitations have been discussed in detail. I appreciate authors honesty in this.


Weakness:
- Evaluations primarily focus on novel rendering/relighting but lacks in individual intrinsic components such as 3D geometry (e.g. depth, chamfer, normal errors etc) and environment maps. 
- Comparison with recent Monte Carlo inverse rendering method [31,37] especially [37] would strengthen this paper. (Although it should be noted [31] does not solve for geometry, while [37] assumes known lighting and does not have full code release).
- I think the paper is better suited for CV or graphics venues than NeurIPS. There are machine learning components in this paper but the main components (denoising/Monte Carlo rendering) do not seem very well aligned with NeurIPS.
questions: I wonder to what extent the proposed method could be used to handle inter-reflections and light refraction in e.g. translucent objects. To me this seems to be one of the biggest advantage of Monte Carlo integration versus SH/SG but I could not find relevant discussions in paper.
limitations: Yes limitations have been well discussed.
review_title: null

Meta-review: metareview: The paper addresses the task of reconstructing 3D meshes, materials and lighting from multi-view images. To this end Monte Carlo ray tracing is used in combination with denoisers for training efficiency. Experiments show improvement w.r.t. SOTA nvdiffrec and nerfactor.
Reviewers like the combination of denoisers for training and MC rays tracing, the thorough evaluation and ablation studies. All reviewers recommend the paper for acceptance and so do I.
confidence: Certain
Decision: Accept