RobIR: Robust Inverse Rendering for High-Illumination Scenes

Authors: ziyi  yang
yanzhen  chen
xinyu  gao
yazhen  yuan
yu  wu
xiaowei  zhou
xiaogang  jin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-30 05:21:35

Abstract: Implicit representation has opened up new possibilities for inverse rendering. However, existing implicit neural inverse rendering methods struggle to handle strongly illuminated scenes with significant shadows and slight reflections. The existence of shadows and reflections can lead to an inaccurate understanding of the scene, making precise factorization difficult. To this end, we present RobIR, an implicit inverse rendering approach that uses ACES tone mapping and regularized visibility estimation to reconstruct accurate BRDF of the object. By accurately modeling the indirect radiance field, normal, visibility, and direct light simultaneously, we are able to accurately decouple environment lighting and the object's PBR materials without imposing strict constraints on the scene. Even in high-illumination scenes with shadows and specular reflections, our method can recover high-quality albedo and roughness with no shadow interference. RobIR outperforms existing methods in both quantitative and qualitative evaluations.

Main paper: 
 introduction: 
 Inverse rendering, the task of extracting the geometry, materials, and lighting of a 3D scene from 2D images, is a longstanding challenge in computer graphics and computer vision. Previous methods, such as providing geometry for the entire scene [35, 46] , modeling shape representation [21, 34, 52, 14] or pre-providing multiple known light information [10] , have achieved plausible results using prior information. To achieve clear albedo and roughness decomposition, factors such as light obscuration, reflection, or refraction must be taken into account. Among these, hard and soft shadows are particularly challenging to eliminate, as they play a critical role not only in obtaining cleaner material but also in accurately modeling geometry and light sources. Although some data-driven approaches [22, 39] have performed plausible shadow removal at the image level, these methods are not generally applicable for inverse rendering.Since the advent of NeRF [32] , implicit representation has garnered significant interest in portraying scenes as neural radiance fields. By applying implicit neural representation to inverse rendering [3, 19, 54] , plausible factorization can be achieved in simple scenes with weak light intensity. Thanks to NeRFactor [55] and its relevant work [8] , which extend previous works by explicitly representing visibility, implicit inverse rendering can be improved with simple shadow removal and clear edge in albedo and roughness. Recently, InvRender [56] has taken the scene factorization problem to a new level by modeling indirect illumination, serving as the baseline in our experiment.However, in high-illumination scenarios with strong shadows or subtle specular reflections, the current methods for implicit inverse rendering have shown limitations in accurately modeling each decomposed part for BRDF estimation. Especially, it will lead to shadow baking in albedo and roughness, thereby causing serious artifacts in relighting and other downstream applications. To deal with such scenes, the following challenges arise in order to obtain high-quality physically based rendering (PBR) materials.First, previous methods for inverse rendering struggle to correctly decouple environment lighting, shadows, and the object's PBR materials. While these methods perform well in scenes with weak light intensity, where shadows and specular reflections are minimal, they struggle to accurately reconstruct BRDF of the object in scenarios with intense lighting. As shown in Fig. 4 and Fig. 5 , shadow and specular reflection lead to poor albedo and messy environment map. To address the aforementioned challenge, we propose a novel approach that applies Academy Color Encoding System (ACES) [1] tone mapping [1] to nonlinearly and monotonically convert the PBR color output from the rendering equation to a range within [0, 1]. Specifically, we introduce a scaled parameter γ to adjust the standard ACES tone mapping curve for specific scenes, better adapting to varying lighting conditions. Unlike previous methods, which either directly output PBR color within [0, 1] [56] , or convert linear PBR color outputted within [0, 1] to sRGB color also lying in [0, 1] [17, 26] , our method can calculate PBR color over a broader value range. For areas with extremely strong or weak lighting, ACES tone mapping can reduce information loss in reconstruction through more refined contrast control, thereby better estimating BRDF without baking shadow or specular highlights.Second, existing methods encounter difficulties in accurately modeling visibility. Typically these methods [56, 17] model the visibility field V (x, ω) through a learned SDF field and sphere tracing, which takes position and view direction as inputs. However, the visibility field is not compatible with direct light modeled based on Spherical Gaussian (SG), resulting in many stubborn shadows remaining at the edges. To address this, we introduce a regularized visibility estimation (RVE) distilled from the visibility field to directly predict the visibility for each SG to achieve more accurate visibility. This technique significantly contributes to the BRDF estimation, enabling the separation of environment maps, albedo, and roughness without the baked shadows. We also apply octree tracing instead of sphere tracing to improve the precision of the visibility field modeling. In summary, the major contributions of our work are:• A novel scene-dependent ACES tone mapping for inverse rendering. It enables the high-quality albedo and roughness reconstruction in scenes with intense lighting and strong shadows. • A novel regularized visibility estimation designed for direct SGs. It improves the visibility accuracy for each direct SG and reduces shadow residue, enhancing the overall BRDF quality of the ill-posed inverse rendering. • The first neural field-based inverse rendering framework to achieve robust shadow removal in BRDF estimation under high-illumination scenes.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: I am sure that the abstract and introduction accurately reflect the paper's contributions and scope. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
implicit neural representation: 
 Neural rendering has gained popularity due to its ability to produce photorealistic images. Recently, NeRF [32] enables photo-realistic novel view synthesis using MLPs. It can handle complex light scattering and reconstruct high-quality scenes for downstream tasks.Subsequent work has enhanced NeRF's efficiency in various ways, elevating it to new heights and enabling its use in other domains. Structure-based techniques [51, 13, 37, 15, 9, 12] have explored ways to improve inference or training efficiency by caching or distilling implicit neural representation into the efficient data structure. Hybrid methods [25, 27, 42, 43, 7] aim to improve the efficiency by incorporating explicit voxel-based data structures. Among them, Instant-NGP [33] achieves minute training by additionally incorporating hash encoding. In addition, some follow-up methods [36, 47, 50] are dedicated to recovering clear surfaces for scenes with complex solid objects by modeling a learnable SDF network, the value of which indicates the minimum distance between the input coordinate and surfaces in the scene.In our work, we employ NeuS [47] , an SDF-based volume rendering framework, to learn geometry priors for inverse rendering. Furthermore, drawing inspiration from PlenOctree [51] , we construct an Octree tracer from the SDF to improve inference efficiency and accuracy compared to sphere tracing. The pipeline of our method. During the pre-processing stage, we reconstruct the scene as an implicit representation by NeuS [47] . From the implicit representation, we extract scene priors such as normal, visibility, and indirect illumination. During BRDF estimation, we optimize environmental lighting, the scaled parameter γ, albedo a, and roughness r, to minimize reconstruction loss under the constraint of the rendering equation. After 100 epochs, we perform regularized visibility estimation and employ an MLP to learn the visibility ratio Q of the direct SGs to obtain more accurate visibility specified for SGs, which is critical for eliminating stubborn shadows at the edges and boundaries.Inverse rendering is a process in computer graphics that aims to derive an understanding of the physical properties of a scene from a set of images. Because the problem is highly ill-posed, most previous works have incorporated priors such as illumination, shape, and shadow, as well as additional observations such as scanned geometry [35, 38, 20] and known light conditions [10] . Simplified approaches, such as those assuming outdoor and natural light [40] or white light [30] , aim to reduce the number of fitting parameters in an ill-posed problem.Recently, there has been a surge of interest in implicit inverse rendering, building on the success of NeRF and its fully differentiable implicit representation. To model spatially-varying bidirectional reflectance distribution function (SVBRDF) under more casual capture conditions, many recent methods [3, 19, 5, 4, 49, 53, 54] have relied on implicit representation. Other works [55, 41, 48, 17, 26] have focused on physical-based modeling for complex scenes via visibility prediction. L-Tracing [8] introduced a new algorithm for estimating visibility without training, while NeRFactor [55] proposed a canonical normal and BRDF smoothness to address NeRF's poor geometric quality. InvRender [56] extends previous work by modeling indirect illumination. Relightable-GS [11] and GS-IR [24] , based on the representation of 3D-GS [18] , have achieved real-time inverse rendering. However, none of these methods are able to decouple shadows and materials under high-illuminance conditions.For non-emitted object, the color c of the surface point x is calculated by the rendering equation:EQUATIONwhere c(x, ω o ) is the output color leaving point x in the view direction ω o , f r (x, ω i , ω o ) is the BRDF function, L(x, ω i ) is the incoming radiance at point x from direction ω i , and n is the surface normal. Following PhySG [54] and InvRender [56] , we use spherical Gaussians (SGs) to efficiently approximate the rendering equation shown in Eq. ( 1). An SG is a spherical function that takes the following form:EQUATIONwhere ξ ∈ R 3 is the lobe axis, λ ∈ R 1 is the lobe sharpness, and µ ∈ R 3 is the lobe amplitude. Please refer to the supplementary material for the complete details.In NeuS [47] , we can determine the surface point x along a specific direction using sphere tracing. By substituting the color function with the shading function based on Eq. ( 1), we can achieve BRDF decomposition through image loss.3 MethodologyQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: I am sure that the paper discuss the limitations of the work. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview: 
 Given a set of multi-view RGB images with known camera poses as input, our target is to reconstruct BRDF of the object even under high-illuminance scenes. As shown in Fig. 1 , the pipeline of RobIR consists of two stages. In the pre-processing stage, we train NeuS S(x, ω) as the representation of the scene, which can provide scene priors like normals, visibility, and indirect illumination (Sec. 3.2).In the BRDF estimation stage, we fix the scene priors and optimize the direct illumination and scaled parameter to compute an accurate BRDF of the object under the constraint of rendering equation (Sec. 3.3). To improve the visibility accuracy for direct illumination and decomposition stability, we introduce the regularized visibility estimation after 100 epochs (Sec. 3.4).In this stage, we adopt the same neural SDF representation and the volume rendering as NeuS [47] to reconstruct the scene. Then we can obtain the necessary prior information for the BRDF estimation stage, such as normal, visibility, and indirect illumination from NeuS.Normal smoothing. In our framework, the accuracy of normal is crucial for BRDF estimation. However, we observed that normals estimated from NeuS tend to be noisy. To overcome this, we drew inspiration from Ref-NeRF [45] and employ a spatial MLP N(x) to predict smooth normals aligned with the density gradient normals (See Fig. 2 ) obtained from NeuS using L 2 loss. We further employ a smooth loss to fix the broken normals caused by specular reflection:EQUATIONwhere N denotes the normal at point x learned by MLP, n denotes the supervision normal from NeuS, and ϵ is a 0.02× Gaussian noise.Visibility and indirect illumination. With the availability of NeuS SDF, we can use sphere tracing to model secondary shading effects such as visibility and indirect illumination. However, performing sphere tracing requires a significant amount of time and memory. Inspired by PlenOctree [51] , we use an octree tracer derived from the NeuS SDF, replacing sphere tracing to accelerate the tracing and achieve more precise intersection results. Moreover, We can further improve the inference efficiency by compressing the visibility and indirect illumination field into MLP.As for indirect illumination, we follow InvRender [56] and model the indirect radiance field L I (x, ω i ) using M = 24 SGs under the supervision of NeuS radiance field. At point x, we first perform octree tracing along direction ω i to get the second intersection point x. Then the indirect radiance field can be supervised by the out-going radiance S(x, -ω i ) from NeuS. Then, the indirect illumination L I is computed by:EQUATIONwhere we use an MLP Γ to output the jth indirect SG parameters, and γ denotes the scaled parameter, which will be illustrated in Sec. 3.3.As for visibility, we learn an MLP that maps the point x and direction ω to visibility V (x, ω), which is supervised by the result of octree tracer from point x along direction ω. The L indir and L vis are optimized by L 1 and binary cross entropy loss as follows:EQUATIONwhere BCE(p i ∥ y i ) represents the binary cross-entropy (BCE) loss, LI is the radiance value at the second intersection point x obtained by querying NeuS, and V (x, ω) is obtained using an octree tracer from point x along direction ω.So far, we have faithfully reconstructed the prior information of the scene such as the normal, visibility and the indirect illumination. In this stage, we aim to accurately evaluate the rendering equation in order to precisely estimate the surface BRDF i.e. albedo a, roughness r and direct environment light with the fixed priors from stage 1. However, previous approaches tend to leave shadow and specular reflection in PBR materials under scenes with high illumination. Thus, we apply a scene-specific ACES tone mapping to the PBR color output by the rendering equation. The ACES tone mapping can calculate the PBR color over a broader value range, better estimating BRDF without baking shadow through more refined contrast control. We adopt SGs to efficiently approximate the rendering equation as PhySG [54] . See complete SGs approximation in the supplementary materials.Scene-specific ACES tone mapping. We adopt the widely used the ACES tone mapping [1] , which is a type of high dynamic range (HDR) tone mapping. Several recent works [16, 31] have incorporated HDR tone mapping into NeRF for specific applications. Specifically, we apply the ACES tone mapping F to convert the PBR color e lying in [0, +∞) to color lying in [0, 1]:F(e) =(2.51e + 0.03)e (2.43e + 0.59)e + 0.14 ,whereas the ACES inverse tone mapping F I is given by:EQUATIONGiven that the light intensity varies across different scenes, applying ACES tone mapping universally is not feasible. Thus, we introduce an additional learnable parameter γ ∈ (0, 1]. This scaled parameter modifies the ACES tone mapping curve, enabling it to automatically adapt to each scene's unique illumination intensity. The resulting deformed tone mapping function is defined as follows:EQUATIONIndirect illumination with scaled parameter. In Sec. 3.2, we model the indirect illumination under the supervision from NeuS's radiance field. To convert indirect illumination to the same value range as BRDF estimation, we need to map the supervised values from NeuS through ACES inverse tone mapping F γ I . Since we are not certain of the γ that best fits the scene during stage 1, we train indirect illumination using randomly sampled γ to obtain indirect illumination under all possible γ settings. Consequently, the loss function L indir in Eq. ( 5) is then revised to include γ as follows:EQUATIONThen in stage 2, we stop training the indirect illumination and treat γ as a learnable parameter. The optimal γ for the current scene will be determined as the decomposition model converges.BRDF estimation. We use the simplified Disney BRDF [6] model with albedo, roughness, and environment light as parameters and assume dielectric materials with a fixed Fresnel term value of F 0 = 0.02. During the BRDF estimation stage, we adopt N = 128 learnable SGs to model direct illumination and represent the PBR materials using an encoder-decoder network. The network initially encodes the input surface point x into its corresponding latent code z and then decodes it into albedo a and roughness r. To further reduce noise in materials, we incorporate the smooth loss similar to Eq. ( 3) to both the albedo and roughness, and apply sparsity loss to z to ensure that most of the channels are close to zero:EQUATION)where D is the decoder of the PBR material network, KL(ρ ∥ ρ) = ρlog ρ ρ + (1 -ρ)log 1-ρ 1-ρ represents Kullback-Leibler (KL) divergence loss that measures the relative entropy of two distributions. We compare our method to InvRender [56] , NVDiffrec [34] , TensoIR [17] , NeRO [26] , Relightable-GS [11] , and GS-IR [24] . The results show that our method outperforms previous approaches without baking specular highlights and shadows into albedo.SGs approximation for rendering equation. In RobIR, we follow PhySG [54] and adopt SGs to approximate the rendering equation in Eq. ( 1):EQUATIONwhere G is the direct SGs learned in this stage, G I is the indirect SGs learned in stage 1, n is the surface normal, η(x)= S i=0 G(ωi)V (x,ωi) S i=0 G(ωi)signifies the visibility for direct SGs obtained by randomly sampling S directions, f s denotes the specular component that can be converted to a single SG. Then, we can integrate the multiplication of these SGs in closed-form [29] to compute the final PBR color ω o . For more details about f s , please see the supplementary materials.One of our primary goals is to achieve clean albedo with no residual shadows, which are typically caused by inaccurate visibility. Despite all efforts of the previous modeling, a small amount of stubborn visibility errors still exist. Therefore, after 100 epochs of BRDF estimation, we introduce regularized visibility estimation, directly using an MLP Q(x, τ ) to predict the visibility of x relative to N direct SGs instead of η calculated through previously learned visibility network V (x, ω). Specifically, Q(x, τ ) is a visibility prediction network learned from scratch under the supervision of η, while τ represents the N × N identity matrix used to add information for N direct SGs and x ∈ R 3 is expanded to R N ×3 to predict visibility for each direct SG. Since visibility errors primarily occur at the edges, which are also sparse in the scene, we leverage the edge loss to make the residual sparse:EQUATIONIn the first 100 epochs, we fix V (x, ω) using η to obtain a stable visibility estimate, avoiding the early collapse of BRDF estimation caused by directly using Q(x, τ ). After 100 epochs, with a rough BRDF estimation in place, we introduce regularized visibility estimation. By using V (x, ω) to distill Q(x, τ ), we directly predict the visibility of point x relative to direct SGs, circumventing errors caused by the sampling direction when calculating η. Thus, we can achieve a more accurate visibility estimate designed for direct SGs (See Fig. 3 ). We present the results of the synthetic scenes. We color each cell as best , second best , and third best . Our method can produce high-quality albedo, roughness, and environment map while maintaining the relighting fidelity. EQUATIONwhere C pbr is the physically-based color from the rendering equation, F γ is the scene-specific ACES tone mapping, C gt is the ground-truth color. In our experiments, λ smooth , λ sparse , and λ edge are set to 0.001, 0.01, and 1.0 respectively.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 In this section, we present the experimental evaluation of our methods. To assess the effectiveness of our approach, we collect synthetic and real-world datasets from NeRF and NeuS without any post-processing. In addition, we use Blender to render our own datasets to further demonstrate the superiority of our methods in high-illumination scenes. It should be noted that unlike previous methods [17, 55] that used a hotdog scene with reduced illumination, we use the original hotdog from NeRF [32] without reduced illumination. See more comparison in the supplementary materials. We compare our method with previous state-of-the-art neural field-based inverse rendering approaches: NVDiffrec [34] , InvRender [56] , TensoIR [17] , NeRO [26] , Relightable-GS [11] , and GS-IR [24] .As shown in Fig. 4 and Fig. 6 , our method can truly achieve robust BRDF estimation, correctly decoupling shadows, ambient lighting, and PBR materials without baking shadows and specular highlights into albedo and roughness. Other methods tend to bake shadows into albedo, which also affects the correct decomposition of object roughness, reflecting their inability to properly separate the various components of BRDF estimation. Even in more challenging real-world scenarios shown in Fig. 7 , our method can achieve robust decomposition results without baking shadows and specular highlights into albedo and roughness.The estimated environment maps are shown in Fig. 5 . Our method can accurately estimate the position of the light source and generate more precise light intensity in high-illumination scenes. As far as we know, we are the first to incorporate the accuracy of the estimated environment map into the quality assessment of neural field-based inverse rendering.Tab. 1 shows the accuracy of the albedo, roughness, relighting, and environment map averaged over synthetic scenes. We did not measure the relighting of Relightable-GS because it does not support relighting of a single object. The term "Log" refers to the use of sigmoid mapping instead of ACES. We can observe that our method achieve the best results in all inverse rendering tasks. Inaccurate BRDF estimation significantly affects the results of relighting, causing methods with high-quality reconstruction to bake shadows and thus leading to a decline in rendering quality during relighting. Overall, our approach can achieve robust inverse rendering in high-illumination scenes.We perform an ablation study to analyze the importance of the key components in our proposed method. As illustrated in Fig. 8 , our method is unable to eliminate both shadows or specular reflection in the absence of ACES tone mapping. Without regularized visibility estimation, inaccurate predictions of direct SGs visibility results in residual shadows. The "Log Tone" result indicates that ACES is a more effective tone mapping than the sigmoid to remove shadow within our framework. Finally, our full method can correctly estimate BRDF of the object, resulting in the best performance.De-shadowing. De-shadowing is a challenging task in the field of inverse rendering, often requiring strong priors and large data-driven models. Our proposed method correctly understands various lighting effects and is capable of effectively eliminating strong and irregular shadows, particularly in scenes with intense lighting. As shown in Fig. 9 , by setting the visibility of direct SGs to 1, we can remove the shadow caused by direct light occlusion. It should be noted that our method cannot remove the areas with reflections and the dark regions caused by the backlighting phenomenon.Relighting. To demonstrate the practical utility of the materials from our method, we conducted relighting experiments. As shown in Fig. 10 , our estimated BRDF results can be accurately relighted in various lighting environments without shadow or illumination artifacts. 
conclusions and discussions: 
 We presented a novel inverse rendering framework for estimating BRDF of the object under highillumination scenes. The key innovation lies in the use of ACES tone mapping, which shifts the calculation of PBR color to a wider value range, significantly reducing the impact of shadows and specular parts on BRDF estimation. In addition, regularized visibility estimation are employed to ensure more acuurate visibility for direct SGs. Experiment results on both synthetic and real-world data show that our method outperforms previous approaches in eliminating shadows and specular reflection under high-illumination scenes.Currently, the proposed method has some limitations. First, non-solid, translucent, and thin objects cannot be correctly handled due to the limitations of NeuS. Second, the employment of SGs to model both direct and indirect lighting presents challenges in dealing with anisotropic objects, consequently leading to our method's deficiency in incorporating the metallic learnable parameters present in the Disney BRDF model. Third, we have not considered scenes with dynamic lighting like [28, 44] . Finally, our method's prior information is limited to multi-view images. We will consider integrating with LLM models in the future work. Table 2 : Quantitative albedo comparison on synthetic dataset. We compare our method to several previous approaches: NVDiffrec [14] , InvRender [56] , TensoIR [17] , Relightable-GS [11] and GS-IR [24] . We report PSNR, SSIM, LPIPS(VGG) and color each cell as best , second best and third best . 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Yes, they do. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: More qualitative results. Our method can effectively remove shadows baked into albedo and roughness, thanks to our accurate modeling of each decomposition component. Therefore, our method can certainly handle scenes with less intense lighting. Fig. 11 shows the results of our method on real-world datasets and some synthetic datasets, including scenes with shadows and specular, as well as diffuse objects. Our method can robustly perform inverse rendering in any situation without baking shadows and illumination into PBR materials.Per-scene albedo results. We present the complete metrics of our method compared to other methods in Tab. 2. The estimated albedo in our method surpasses existing SOTA methods in every synthetic scene.More relighting. We show more relighting results in Fig. 12 and Fig. 14 . The two scenes demonstrate that our method can accurately estimate the BRDF of the object under scenes with specular highlights and severe shadows. Visualization and evaluation on tone mapping. We first visualize the vanilla ACES and sRGB tone mapping in Fig. 18 , indicating that ACES curve has much wider input range. Then in Fig. 19 , we show the scene-specific tone mapping curve with different γ, enabling the ACES curve to fit other settings with different tone mapping methods. Finally, we evaluate the optimized ACES curve (with γ = 0.42) in chessboard scene with GT tone mapping (sRGB) in Fig. 20 . The results show that our scene-specific ACES tone mapping can stretch to sRGB curve, demostrating the effectiveness of our method. NeurIPS Paper Checklist 


Decision: The paper presents a new approach to inverse rendering, specifically designed for challenging high-illumination scenarios. The method introduces scene-specific tone mapping and regularized visibility estimation to effectively decouple environment lighting from an object's BRDF, resulting in improved albedo and roughness estimations without shadow interference. The paper is well-written and supported by comprehensive experiments, demonstrating superior performance over existing methods.

While the proposed approach is novel, some reviewers noted that it builds incrementally on existing frameworks like NeuS and InvRender, and the complexity of the method may limit its practical application. Additionally, there were concerns about the loss of high-frequency details due to the smooth loss used in the model, which the authors addressed by attributing it to the rendering precision of NeuS. Despite these concerns, the authors provided thorough rebuttals that clarified the benefits of their approach, particularly the advantages of ACES tone mapping in HDR contexts.

After the discussions, some reviewers raised their initial scores, recognizing the paper's contributions. Given the strengths of the work and the constructive engagement with reviewers, the paper is recommended for acceptance as a poster presentation.

Reviews: review: paper_summary: paper_summary: This paper introduces RobIR, an inverse rendering approach designed to handle strong or directional illumination scenes with strong shadows and specular reflections. 
The proposed method aims to decouple environment lighting and object materials, with the goal of producing high-quality albedo without baked shadow.

Building on top of prior inverse rendering methods such as InvRender, RobIR introduces two components that further boost the reconstruction quality: (1) ACES tone mapping with an optimizable gamma parameter to better capture the image formation process; (2) regularization for visibility estimation. 

RobIR demonstrates better performance over existing methods in quantitative and qualitative evaluations.
strength_weakness: strength_weakness: [W1] The benefit of ACES tone-mapping is a bit surprising. Despite a more accurate formulation for the image formation process, the task of inverse rendering is inherently still an ill-posed problem. It’s a surprising conclusion that a tone-mapping formulation can robustly and significantly benefit shadow removal. 

With many other regularization terms entangled, it’s a bit hard to evaluate the correctness of this specific component. 
I’d hope to know more details in the following aspects: 


[W1.1] Missing visualization of the tone-mapping curve. Despite an important contribution, the estimation results of the tone-mapping curve are missing. What does the default ACES tonemapping look like, and what does the final tonemapping look like with the optimizable gamma? 

In the revised version, the curves should be qualitatively visualized and included in main paper. From the existing experiments (such as the PSNR metrics), the audience cannot intuitively understand how well the tonemapping is estimated. 

[W1.2] Missing evaluation of the tone-mapping curve. 

As many datasets do not have GT tonemapping, it’s unclear how accurately the tone-mapping approximate the GT tonemapping. 

One way to evaluate is to add additional tone adjustment to the input dataset. Assume with the original dataset images $\{ I \}$ and the method originally reconstructs a tonemapping curve $f$. Given a new tone adjustment function, e.g. $g(x) = x^\kappa$, the adjusted dataset images become $\{g(I)\}$. Re-running the method can get a newly reconstructed tonemapping curve $f_\kappa $. The consistency between $g \circ f$  and $f_\kappa$ can indicate how well the model can approximate the additional introduced tone adjustment function. $\kappa$ can be set to values like 0.5 or 2. 

[W1.3] The evaluation metric on the Albedo is flawed. As albedo estimation/optimization often involve an unknown scale, PSNR alone is not a proper evaluation for Albedo. Check out [1] for more analysis and more appropriate scale-invariant metrics.

[W1.4] Most of the results are from synthetic datasets, where the GT tonemapping could potentially be close to ACES. For real-world results in Fig.7, the albedo estimation looks strongly regularized and over-smoothed. 

[W1.5] Are the radiance values (before tonemapping) and indirect illumination in HDR? If so, what is the activation function? 

[W2] The proposed method involves a complicated training pipeline (two stages with each stage have its own loss scheduling), and the novelty is relatively limited. 


**References** 

[1] Grosse et al., Ground truth dataset and baseline evaluations for intrinsic image algorithms, ICCV 2009.
questions: questions: This paper addresses the shadow baking issue in inverse rendering by proposing two straightforward but effective techniques: optimizable tone-mapping and visibility regularization.

However, I am not fully convinced that optimizable tone-mapping significantly benefits shadow removal, and without further clarification it’s challenging to evaluate the technical correctness of this component. In the rebuttal, please prioritize addressing Weaknesses 1.1-1.4. 

**[Post rebuttal update]**

The rebuttal and related discussion address my concerns. I update my rating to Weak Accept.
limitations: limitations: The paper discussed limitations in the end of paper. It’ll be beneficial to include failure cases in Appendix.
paper_summary: paper_summary: This paper introduces RobIR, an inverse rendering approach that can better tackle “high-illumination” scenes. RobIR first leverages the existing neural field model (NeuS) to represent 3D geometry information including normal, visibility, and indirect illumination. It then utilizes these geometry priors to decompose rendering attributes of the scene through the approximated rendering equation with spherical Gaussian. This work introduces an optimizable ACES tone-mapping and regularized visibility estimation model to better handle HDR color and shadow occlusions, respectively. Their experiment demonstrates some impressive results on shadow disentanglement.
strength_weakness: strength_weakness: 1. This paper still follows a commonly used multi-stage inverse rendering strategy with neural fields. The geometry representation is based on NeuS; the rendering formulation (SG rendering, visibility, and indirect illumination) is mainly based on InvRender; The proposed optimizable ACES tone and REV are more like incremental improvements over InvRender. The key rendering formulation and optimization remain the same as the prior works. Therefore, the novelty of this work is moderate.
2. The description of RVE in Sec. 3.4 is not very clear.  It seems that MLP $Q(x, \tau)$ directly outputs N visibility ratios, thus $\eta(x)$ in Eq. 12 should also be an N-dim vector instead of a scalar value.
3. The proposed method is quite time-consuming. Training time even without NeuS is around 5 hours.
4. The proposed regularization terms may hurt the high-frequency details in real-world scenes (Fig. 7).
5. This method is limited to dielectric materials, without the consideration of metallic and glossy objects, as already pointed out by the authors.
6. The paper should include some inverse rendering methods with differentiable path tracing, as these methods can explicitly handle visibility, for example, NvDiffRecMC, Mitsuba, etc.
7. Minor errors:
    * L275 accuurate -> accurate
    * Figure 8: Hotdog label is wrong.
questions: questions: 1. What are the differences between the final optimized ACES tone mapping and sRGB gamma tone mapping? How does this optimized tone mapping vary from scene to scene? It would be better if tone-mapping curves were included in the paper.
2. I tested this ACES tone mapping mentioned in the paper, it seems that the proposed learnable ACES tonemapping (an S-curve) cannot well approximate the existing concave tone-mapping curves that are potentially used for rendering Blender objects (e.g., sRGB curve, Filmic curve, AgX curve, etc.). Given this limitation, how does the paper address the color mismatch in the low-illumination color space?
2. The paper does not show the metrics for rendering (NVS) with decomposed attributes. I’d like to see the rendering quality metrics for those NeRF scenes.
3. It would be better if the author could show some video examples of moving shadows while rotating envmaps in the final release.
4. Why does the learnable parameter $\gamma$ have an exponent 0.2?
limitations: limitations: The limitations are adequately discussed in the last paragraph of the paper.
paper_summary: paper_summary: This paper proposes a method for the inverse rendering of high-illumination and highly reflective scenes. There are two training phases, in the first phase, it trains by Neus, to get geometry and compute visibility by octrees. In the second phase, it decomposes lighting as SGs and material by MLPs.
strength_weakness: strength_weakness: Figure 2 gives comparisons with and without smooth loss, however, the one w/o smooth loss is better, while the loss may over-smooth the details. 
Figure 11 shows results where the lighting consists of the original colors.
questions: questions: How does the relighting work? Do you use original indirect lighting? 
Why is visibility divided into two stages, and why not directly use the results calculated by the Neus octree as the ground truth to supervise SG? Instead, why is the Neus octree result used as the ground truth to supervise the MLP, and then the MLP's distribution used to supervise the SG results? What are the differences?
limitations: limitations: See above.
paper_summary: paper_summary: This paper addresses inverse rendering in high-illumination scenes with strong shadows where past methods bake shadows and highlights into estimation results. This paper proposes to use ACES tone mapping and makes it scene-dependent for inverse rendering in high-illumination scenes. This paper also proposes to directly estimate the visibility of each spherical Gaussian of direct illumination instead of a visibility field, which enables an accurate representation of shadows at the edge. The experimental results on the synthetic and real-world datasets show that the proposed method can estimate accurate albedos, surface roughness, and illumination without artifacts in the high-illumination scenes.
strength_weakness: strength_weakness: - It is unclear why the ACES tone mapping, of which usage is the key contribution, enables the robust inverse rendering of high-illumination scenes with strong shadows.
- The proposed method loses the albedo of the detailed texture due to smoothness loss in Eq. 10. For example, Bear in Fig. 7 and truck in Fig. 4.
questions: questions: - A more detailed explanation of the effects of ACES tone mapping is expected. Why is it more suitable for high-illumination scenes than other tone mapping methods such as sRGB and log tone mapping? How does it affect the loss and the optimization?
limitations: limitations: Yes.

Comments: title: Additional questions
Fix typo errors in Q1
comment: We thank the reviewer for the valuable comments and are keen to follow up on the provided suggestion to include NVS metrics in the supplement. 

Additionally, we want to emphasize that the NVS results are highly dependent on the base method. We tried applying our method to 3D-GS, and the NVS results are shown in the table below. 

|         | PSNR      | SSIM       | LPIPS      |
| ------- | --------- | ---------- | ---------- |
| Ficus   | 32.35     | 0.9773     | 0.0191     |
| Hotdog  | 35.66     | 0.9806     | 0.0297     |
| Lego    | 34.51     | 0.9766     | 0.0208     |
| Mic     | 34.01     | 0.9871     | 0.0129     |
| Average | **34.13** | **0.9804** | **0.0206** |
| Ours    | 30.11     | 0.9466     | 0.0528     |

Although the NVS results are significantly higher than ours, the decomposition of each component is completely chaotic, with shadows baked into the albedo, the environment map being meaningless, and so on. This issue arises because the geometric quality of 3D-GS is far inferior to that of NeuS.

In the future, we will attempt to base our method on models like 2D-GS, which possess both good geometric and rendering quality. We believe that strong geometric quality is the foundation for the correct decomposition in inverse rendering, facilitating accurate BRDF estimation and yielding better PBR materials and environment maps for relighting and shadow removal. Improved rendering quality will also enhance NVS quality and further improve the rendering quality of relighting.
Thank you for your recognition and adjustment of the score. We are glad that our rebuttal effectively addressed your concerns. We also appreciate your recognition of our methods. Thank you again for your constructive feedback and guidance throughout the review process.
Thanks for the authors' detailed responses. I am generally satisfied with these responses. 
I'll retain or possibly increase my score later.
I still think NVS rendering quality is also an important factor that should be considered. I encourage you to include these metrics in the supplement.
Thank you for your response. I acknowledge the novel idea of scene-specific tone mapping to handle both HDR and non-HDR scenes as well as the effectiveness of the visibility estimation. I raise my rating from Borderline reject to Accept.
Thank you very much for the constructive feedback. We hope that our response below will address your concerns.

**Q1: Unknown tone mapping assumption.**

Our proposed method only takes a collection of images with camera poses as input, and the ground truth tone mapping of these images is unknown. All the methods we compare in our paper use datasets with unknown ground truth tone mapping. We introduce scene-specific ACES tone mapping to approximate various tone mapping scenarios; for example, HDR scenes are typically optimized to vanilla ACES tone mapping. Therefore, if the tone mapping method were known, theoretically, we wouldn’t need to use our proposed scene-specific ACES. However, this is nearly impossible because different physics engines employ different tone mappings, and tone mapping itself involves numerous parameters. Additionally, variations camera parameters in real-world scenes make it even more difficult for us to determine the ground truth tone mapping.

**Q2: The problem of the existing inverse rendering method.**

The existing inverse rendering methods, such as NeRO, use sRGB tone mapping. However, in high-illumination scenes, we typically need tone mapping that can handle HDR inputs to produce the final image. We are the **first** to apply ACES tone mapping in inverse rendering, which fundamentally enables our method to handle inverse rendering in high-illumination scenes. Additionally, the introduction of scene-specific ACES tone mapping allows us to approximate other tone mappings (e.g., sRGB), making our method compatible with non-high-illumination scenes as well. 

Building on more accurate tone mapping, we achieve more precise visibility modeling through Regularized Visibility Estimation (RVE) and NeuS Octree, which effectively removes shadows and disentangles other parts of the BRDF estimation more accurately.


Therefore, the failure (especially shadow baking) of existing inverse rendering methods in high-illumination scenes stems from 1) the mismatch of tone mapping between image processing and optimization, as well as 2) inaccurate visibility estimation.
I appreciate your addressing the questions. After reading the rebuttal, I have additional questions about tone mapping to clarify the contributions of this paper.
1. Does the proposed method assume an unknown method of tone mapping for the input images? If so,
* Is this assumption common for applications of inverse rendering?
* Is the proposed scene-dependent tone mapping not needed when the tone mapping method is known and can be used in optimization?
2. Would you like to claim that the existing inverse rendering method misses the mismatch of tone mapping between image processing (image formation) and optimization, resulting in failure in high-illumination scenes?
**Q1: Why ACES is more suitable than other tone mapping like sRGB and sigmoid.**

As previously mentioned, high-illumination scenes often require HDR tone mapping to produce the final image, mapping values from $[0, +\infty]$ into the range of $[0, 1]$. However, sRGB is not designed for processing HDR input. ACES allows PBR output colors to be in a larger value range, which better distinguishes very dark shadow regions and aids in the decoupling of object materials. 

Additionally, ACES has significant advantages over sigmoid curves. Although both map an infinite range to $[0,1]$, ACES is a more commonly used HDR tone mapping curve in graphics, consistent with many rendering engines, and provides better contrast control while preserving more detail in highlights and shadows.