Title: RobIR: Robust Inverse Rendering for High-Illumination Scenes
Venue: NeurIPS 2024
Accepted: yes
Keywords: Inverse Rendering

BRDF Estimation

Illumination Estimation

NeRF

Inverse Graphics
Abstract: Implicit representation has opened up new possibilities for inverse rendering. However, existing implicit neural inverse rendering methods struggle to handle strongly illuminated scenes with significant shadows and slight reflections. The existence of shadows and reflections can lead to an inaccurate understanding of the scene, making precise factorization difficult. To this end, we present RobIR, an implicit inverse rendering approach that uses ACES tone mapping and regularized visibility estimation to reconstruct accurate BRDF of the object. By accurately modeling the indirect radiance field, normal, visibility, and direct light simultaneously, we are able to accurately decouple environment lighting and the object's PBR materials without imposing strict constraints on the scene. Even in high-illumination scenes with shadows and specular reflections, our method can recover high-quality albedo and roughness with no shadow interference. RobIR outperforms existing methods in both quantitative and qualitative evaluations.


Peer reviews:
comments: SUMMARY:
This paper addresses inverse rendering in high-illumination scenes with strong shadows where past methods bake shadows and highlights into estimation results. This paper proposes to use ACES tone mapping and makes it scene-dependent for inverse rendering in high-illumination scenes. This paper also proposes to directly estimate the visibility of each spherical Gaussian of direct illumination instead of a visibility field, which enables an accurate representation of shadows at the edge. The experimental results on the synthetic and real-world datasets show that the proposed method can estimate accurate albedos, surface roughness, and illumination without artifacts in the high-illumination scenes.

STRENGTHS:
+ This paper proposes a novel regularized visibility estimation that enables an accurate representation of shadows at the edge.
+ Experimental results show that the proposed method successfully estimates BRDF and illumination while existing methods suffer from artifacts. They also indicate the effectiveness of ACES tone mapping compared with log tone mapping methods.

WEAKNESSES:
- It is unclear why the ACES tone mapping, of which usage is the key contribution, enables the robust inverse rendering of high-illumination scenes with strong shadows.
- The proposed method loses the albedo of the detailed texture due to smoothness loss in Eq. 10. For example, Bear in Fig. 7 and truck in Fig. 4.

QUESTIONS:
- A more detailed explanation of the effects of ACES tone mapping is expected. Why is it more suitable for high-illumination scenes than other tone mapping methods such as sRGB and log tone mapping? How does it affect the loss and the optimization?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
This paper proposes a method for the inverse rendering of high-illumination and highly reflective scenes. There are two training phases, in the first phase, it trains by Neus, to get geometry and compute visibility by octrees. In the second phase, it decomposes lighting as SGs and material by MLPs.

STRENGTHS:
Direct and indirect lighting, visibilities are presented by SGs, which is compact. 
From the results, shadows and specularities are decomposed well. 
Tone mapping is used, as NeRF in the Dark, which improves results for high-illumination scenes.

WEAKNESSES:
Figure 2 gives comparisons with and without smooth loss, however, the one w/o smooth loss is better, while the loss may over-smooth the details. 
Figure 11 shows results where the lighting consists of the original colors.

QUESTIONS:
How does the relighting work? Do you use original indirect lighting? 
Why is visibility divided into two stages, and why not directly use the results calculated by the Neus octree as the ground truth to supervise SG? Instead, why is the Neus octree result used as the ground truth to supervise the MLP, and then the MLP's distribution used to supervise the SG results? What are the differences?

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RobIR, an inverse rendering approach that can better tackle “high-illumination” scenes. RobIR first leverages the existing neural field model (NeuS) to represent 3D geometry information including normal, visibility, and indirect illumination. It then utilizes these geometry priors to decompose rendering attributes of the scene through the approximated rendering equation with spherical Gaussian. This work introduces an optimizable ACES tone-mapping and regularized visibility estimation model to better handle HDR color and shadow occlusions, respectively. Their experiment demonstrates some impressive results on shadow disentanglement.\n\nSTRENGTHS:\n1. This work has some further thoughts on color tone mapping for the typical multi-view inverse rendering. The proposed optimizable ACES tone mapping looks very effective in improving inverse rendering results.\n2. The proposed visibility representation (RVE) also plays an important role in the final results. RVE with its neural net accumulates and denoises Monte Carlo samples to achieve more accurate and stable visibility evaluation. Their efforts to refine the visibility should be appreciated.\n3. I am glad the authors clearly point out they use the original NeRF rendering of Hotdog and Lego, instead of the NeRFactor’s.\n\nWEAKNESSES:\n1. This paper still follows a commonly used multi-stage inverse rendering strategy with neural fields. The geometry representation is based on NeuS; the rendering formulation (SG rendering, visibility, and indirect illumination) is mainly based on InvRender; The proposed optimizable ACES tone and REV are more like incremental improvements over InvRender. The key rendering formulation and optimization remain the same as the prior works. Therefore, the novelty of this work is moderate.\n2. The description of RVE in Sec. 3.4 is not very clear.  It seems that MLP $Q(x, \\tau)$ directly outputs N visibility ratios, thus $\\eta(x)$ in Eq. 12 should also be an N-dim vector instead of a scalar value.\n3. The proposed method is quite time-consuming. Training time even without NeuS is around 5 hours.\n4. The proposed regularization terms may hurt the high-frequency details in real-world scenes (Fig. 7).\n5. This method is limited to dielectric materials, without the consideration of metallic and glossy objects, as already pointed out by the authors.\n6. The paper should include some inverse rendering methods with differentiable path tracing, as these methods can explicitly handle visibility, for example, NvDiffRecMC, Mitsuba, etc.\n7. Minor errors:\n    * L275 accuurate -> accurate\n    * Figure 8: Hotdog label is wrong.\n\nQUESTIONS:\n1. What are the differences between the final optimized ACES tone mapping and sRGB gamma tone mapping? How does this optimized tone mapping vary from scene to scene? It would be better if tone-mapping curves were included in the paper.\n2. I tested this ACES tone mapping mentioned in the paper, it seems that the proposed learnable ACES tonemapping (an S-curve) cannot well approximate the existing concave tone-mapping curves that are potentially used for rendering Blender objects (e.g., sRGB curve, Filmic curve, AgX curve, etc.). Given this limitation, how does the paper address the color mismatch in the low-illumination color space?\n2. The paper does not show the metrics for rendering (NVS) with decomposed attributes. I’d like to see the rendering quality metrics for those NeRF scenes.\n3. It would be better if the author could show some video examples of moving shadows while rotating envmaps in the final release.\n4. Why does the learnable parameter $\\gamma$ have an exponent 0.2?\n\nLIMITATIONS:\nThe limitations are adequately discussed in the last paragraph of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RobIR, an inverse rendering approach designed to handle strong or directional illumination scenes with strong shadows and specular reflections. \nThe proposed method aims to decouple environment lighting and object materials, with the goal of producing high-quality albedo without baked shadow.\n\nBuilding on top of prior inverse rendering methods such as InvRender, RobIR introduces two components that further boost the reconstruction quality: (1) ACES tone mapping with an optimizable gamma parameter to better capture the image formation process; (2) regularization for visibility estimation. \n\nRobIR demonstrates better performance over existing methods in quantitative and qualitative evaluations.\n\nSTRENGTHS:\n- The model design choices are valid and sensible. \n- The experiments and ablation study are thorough. \n- The paper is well written and easy to follow.\n\nWEAKNESSES:\n[W1] The benefit of ACES tone-mapping is a bit surprising. Despite a more accurate formulation for the image formation process, the task of inverse rendering is inherently still an ill-posed problem. It’s a surprising conclusion that a tone-mapping formulation can robustly and significantly benefit shadow removal. \n\nWith many other regularization terms entangled, it’s a bit hard to evaluate the correctness of this specific component. \nI’d hope to know more details in the following aspects: \n\n\n[W1.1] Missing visualization of the tone-mapping curve. Despite an important contribution, the estimation results of the tone-mapping curve are missing. What does the default ACES tonemapping look like, and what does the final tonemapping look like with the optimizable gamma? \n\nIn the revised version, the curves should be qualitatively visualized and included in main paper. From the existing experiments (such as the PSNR metrics), the audience cannot intuitively understand how well the tonemapping is estimated. \n\n[W1.2] Missing evaluation of the tone-mapping curve. \n\nAs many datasets do not have GT tonemapping, it’s unclear how accurately the tone-mapping approximate the GT tonemapping. \n\nOne way to evaluate is to add additional tone adjustment to the input dataset. Assume with the original dataset images $\\{ I \\}$ and the method originally reconstructs a tonemapping curve $f$. Given a new tone adjustment function, e.g. $g(x) = x^\\kappa$, the adjusted dataset images become $\\{g(I)\\}$. Re-running the method can get a newly reconstructed tonemapping curve $f_\\kappa $. The consistency between $g \\circ f$  and $f_\\kappa$ can indicate how well the model can approximate the additional introduced tone adjustment function. $\\kappa$ can be set to values like 0.5 or 2. \n\n[W1.3] The evaluation metric on the Albedo is flawed. As albedo estimation/optimization often involve an unknown scale, PSNR alone is not a proper evaluation for Albedo. Check out [1] for more analysis and more appropriate scale-invariant metrics.\n\n[W1.4] Most of the results are from synthetic datasets, where the GT tonemapping could potentially be close to ACES. For real-world results in Fig.7, the albedo estimation looks strongly regularized and over-smoothed. \n\n[W1.5] Are the radiance values (before tonemapping) and indirect illumination in HDR? If so, what is the activation function? \n\n[W2] The proposed method involves a complicated training pipeline (two stages with each stage have its own loss scheduling), and the novelty is relatively limited. \n\n\n**References** \n\n[1] Grosse et al., Ground truth dataset and baseline evaluations for intrinsic image algorithms, ICCV 2009.\n\nQUESTIONS:\nThis paper addresses the shadow baking issue in inverse rendering by proposing two straightforward but effective techniques: optimizable tone-mapping and visibility regularization.\n\nHowever, I am not fully convinced that optimizable tone-mapping significantly benefits shadow removal, and without further clarification it’s challenging to evaluate the technical correctness of this component. In the rebuttal, please prioritize addressing Weaknesses 1.1-1.4. \n\n**[Post rebuttal update]**\n\nThe rebuttal and related discussion address my concerns. I update my rating to Weak Accept.\n\nLIMITATIONS:\nThe paper discussed limitations in the end of paper. It’ll be beneficial to include failure cases in Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-19"}