RobIR: Robust Inverse Rendering for High-Illumination Scenes
Authors: Ziyi Yang
Chenyanzhen
Xinyu Gao
YazhenYuan
Wu Yu
Xiaowei Zhou
Xiaogang Jin
Venue: NeurIPS 2024 poster (2024)
Keywords: Inverse Rendering
BRDF Estimation
Illumination Estimation
NeRF
Inverse Graphics
Primary area: machine_vision
TL;DR: A NeuS-based inverse rendering method for shadow and specular highlight removed BRDF estimation under high illumination scenes.

Abstract: Implicit representation has opened up new possibilities for inverse rendering. However, existing implicit neural inverse rendering methods struggle to handle strongly illuminated scenes with significant shadows and slight reflections. The existence of shadows and reflections can lead to an inaccurate understanding of the scene, making precise factorization difficult. To this end, we present RobIR, an implicit inverse rendering approach that uses ACES tone mapping and regularized visibility estimation to reconstruct accurate BRDF of the object. By accurately modeling the indirect radiance field, normal, visibility, and direct light simultaneously, we are able to accurately decouple environment lighting and the object's PBR materials without imposing strict constraints on the scene. Even in high-illumination scenes with shadows and specular reflections, our method can recover high-quality albedo and roughness with no shadow interference. RobIR outperforms existing methods in both quantitative and qualitative evaluations.

RobIR: Robust Inverse Rendering for High-Illminntion Scenes
Ziyi Yang Yanzhen Chen'
Xinyu Gao! Xiaogang Jin It
Yazhen Yuan?
Yu Wu?
Xiaowei Zhou'
State Key Lab of CAD&CG, Zhejiang University
Tencent
Abstract
Implicit representation has opened up possibilities for inverse rendering. How- ever, existing implicit neural inverse rendering methods struggle to handle strongly illuminated scenes with significant shadows and slight reflections. The existence of shadows and reflections can lead to an inaccurate understanding of the scene, making precise factorization difficult. To this end, we present RobIR, implicit inverse rendering approach that uses ACES tone mapping and regularized visibility estimation reconstruct accurate BRDF of the object. By accurately modeling the indirect radiance field, normal, visibility, and direct light simultancously, we are able accurately decouple environment lighting and the object's 's PBR mate- rials without imposing strict constraints on the scene. Even in high-illumination scenes with shadows and specular reflections, our method can recover high-quality albedo and roughness with no shadow interference. RobIR outperforms existing methods both quantitative and qualitative evaluations. Code is available https://github.com/nngg 4m/RobIR
Introduction
Inverse rendering, the task of extracting the geometry, materials, and lighting of 3D scene from 2D images, longstanding challenge in computer graphics and computer vision. Previous methods, such as providing geometry for the entire scene [35] 46). modeling shape representation [21] (34]
52
pre-providing multiple known light information [10]. have achieved plausible results using prior information. To achieve clear albedo and roughness decomposition, factors such as light obscuration, reflection, refraction must be taken into account. Among these, hard and soft shadows are particularly challenging to eliminate, they play critical role not only in obtaining cleaner material but also accurately modeling geometry and light sources. Although some data-driven approaches 22 39 have performed plausible shadow removal at the image level, these methods are not generally applicable for inverse rendering.
Since the advent of NeRF (32), implicit representation has garnered significant interest in portraying scenes as neural radiance fields. By applying implicit neural representation to inverse rendering [ 19 54], plausible factorization can be achieved in simple scenes with weak light intensity. Thanks to NeRFactor (55) and its relevant work (8), which extend previous works by explicitly representing visibility, implicit inverse rendering can be improved with simple shadow removal and clear edge in albedo and roughness. Recently, InvRender (56) has taken the scene factorization problem to new level by modeling indirect illumination, serving as the baseline in our experiment.
However, in high-illa -lhiimilminniion scenarios with strong shadows subtle specular reflections, the current methods for implicit inverse rendering have shown limitations in accurately modeling each decomposed part for BRDF estimation. Especially, it will lead to shadow baking in albedo and roughness, thereby causing serious artifacts relighting and other downstream applications. To
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

deal with such scenes, the following challenges arise in order to obtain high-quality physically based rendering (PBR) materials.
First, previous methods for inverse rendering struggle to correctly decouple environment lighting, shadows, and the object's PBR materials. While these methods perform well in scenes with weak light intensity, where shadows and specular reflections are minimal, they struggle to accurately reconstruct BRDF of the object in scenarios with intense lighting. As shown in Fig. 4 and Fig. shadow and specular reflection lead to poor albedo and messy environment map. To address the aforementioned challenge, we propose a novel approach that applies Academy Color Encoding System (ACES) M tone mapping [ to nonlinearly and monotonically convert the PBR color output from the rendering equation to a range within [0. 1]. Specifically,, we introduce a scaled parameter " to adjust the standard ACES tone mapping curve for specific scenes, better adapting to varying lighting conditions. Unlike previous methods, which either directly output PBR color within [0. [56], or convert linear PBR color outputted within [0, 1] to sRGB color also lying in [0, 1] 17 26.. our method can calculate PBR color over a broader value range. For areas with extremely strong or weak lighting, ACES tone mapping can reduce information loss in reconstruction through more refined contrast control, thereby better estimating BRDF without baking shadow or specular highlights.
Second, existing methods encounter difficulties in accurately modeling visibility. Typically these methods [56] I7 model the visibility field V(x, w) through a learned SDF field and sphere tracing, which takes position and view direction as inputs. However, the visibility field is not compatible with direct light modeled based on Spherical Gaussian (SG), resulting many stubborn shadows remaining at the edges. To address this, we introduce a regularized visibility estimation (RVE) distilled from the visibility field to directly predict the visibility for each SG achieve more accurate visibility. This technique significantly contributes to the BRDF estimation, enabling the separation of environment maps, albedo, and roughness without the baked shadows. We also apply octree tracing instead of sphere tracing to improve the precision of the visibility field modeling.
In summary, the major contributions of our work are:
A novel scene-dependent ACES tone mapping for inverse rendering, It enables the high-quality albedo and roughness reconstruction in scenes with intense lighting and strong shadows.
A novel regularized visibility estimation designed for direct SGs. It improves the visibility accuracy for each direct SG and reduces shadow residue, enhancing the overall BRDF quality of the ill pos-p inverse rendering.
The first neural beld-based inverse rendering framework to achieve robust shadow removal in
BRDF estimation under high-illumination scenes.
2
Related Work
2.1 Implicit Neural Representation
Neural rendering has gained popularity due to its ability produce photorealistic images. Recently, NeRF (32] enables photo-realistic novel view synthesis using MLPs. It can handle complex light scattering and reconstruct high-quality scenes for downstream tasks.
Subsequent work has enhanced NeRF's efficiency various ways, elevating to new heights and enabling its use in other domains. Structure- based techniques [51] 13 37 15]
9 12 have explored ways to improve inference or training efficiency by caching or distilling implicit neural representation into the efficient data structure. Hybrid methods 225 27 42 43 aim to improve the efficiency by incorporating explicit voxel-based data structures. Among them, Instant-NGP [3] achieves minute training by additionally incorporating hash encoding. In addition, some follow-up methods (3 47 50 are dedicated to recovering clear surfaces for scenes with complex solid objects by modeling learnable SDF network, the value of which indicates the minimum distance between the input coordinate and surfaces in the scene.
In our work, we employ NeuS [47, an SDF-based volume rendering framework, to learn geometry priors for inverse rendering. Furthermore, drawing inspiration from PlenOctree (51). we construct an Octree tracer from the SDF to improve inference efficiency and accuracy compared to sphere tracing.

---

NeuS
2 fla r)
Direct Light
Indromclagte
Recomstracion
Vailiiit
X
optinize Indrectillunitaaii with varsus
Strect Vie
Scaled
Y
Mapped
Radiane Fodd
Eavtecemenntll Hup
Staled Panundter
Boughtees
Pre Processing
BRDF Estimation
Figure I: The pipeline of method. During the pre-processing stage, reconstruct the scene as an implicit representation by NeuS [7]] From the implicit representation, we extract scene priors such as normal, visibility, and indirect illumination. During BRDF estimation, we optimize environmental lighting. the scaled parameter y, albedo a, and roughness r, to minimize reconstruction loss under the constraint of the rendering equation. After 100 epochs, we perform regularized visibility estimation and employ MLP learn the visibility ratio Q of the direct SGs obtain more accurate visibility specified for SGs, which is critical for eliminating stubborn shadows at the edges and boundaries.
2.2 Inverse Rendering
Inverse rendering is a process in computer graphics that aims to derive an understanding of the physical properties of scene from set of images. Because the problem is highly ill- ill-posed, most
previous works have incorporated priors such as illumination, shape, and shadow, as well as additional
observations such scanned geometry [35] 38] [20 and known light conditions [1O]. Simplified
approaches, such as those assuming outdoor and natural light [40] or white light [30, aim to reduce
the number of fitting parameters in ill-posed problem.
Recently, there has been surge of interest in implicit inverse rendering, building on the success of NeRF and its fully differentiable implicit representation. To model spatially-varying bidirectional reflectance distribution function (SVBRDF) under more casual capture conditions, many recent meth-
ods [3] 19 5 4 49 (53 54] have relied on implicit representation. Other works [55] 41
48
17
26
have focused on physical-bass based modeling for complex scenes via visibility prediction. L- L-Tracing B introduced
algorithm for estimating visibility without training. while NeRFactor (55] proposed
a canonical normal and BRDF smoothness to address NeRF's poor geometric quality. InvRender [56] extends previous work by modeling indirect illumination. Relightable GS [IT] and GS-IR 22],
based on the representation of 3D-GS [8]. have achieved real-time inverse rendering. However, none of these methods are able decouple shadows and materials under high-illuminance conditions.
2.3 The Rendering Equation
For non-emitted object, the color of the surface point is calculated by the rendering equation:
(x, = = fr (Wo, Wi, x) L (x, wi) (w: n)dw,,
where c(x, w.) is the output color leaving point x in the view direction Wor fr (x, wi, w is the BRDF function, L(x, wi) is the incoming radiance point from direction Wi, and the surface normal. Following PhySG [54] and InvRender [56), we use spherical Gaussians (SGs) to efficiently approximate the rendering equation shown Eq. (T. An SG spherical function that takes the following form:
G(w; &, A. swo-1)
(2)
where & R3 is the lobe axis, À R! is the lobe sharpness, and μ R²3 is the lobe amplitude. Please refer to the supplementary material for the complete details.
we can determine the surface point along a specific direction using sphere tracing. By substituting the color function with the shading function based on Eq. ( we can achieve BRDF decomposition through image loss.
In NeuS 47

---

3 Methodology
3.1 Overview
Given a set of multi-view RGB images with known camera poses as input, our target is to reconstruct BRDF of the object even under high-illuminance scenes. As shown in Fig. T the pipeline of RobIR consists of two stages. In the pre-processing stage, we train NeuS S(r, as the representation of the scene, which can provide scene priors like normals, visibility, and indirect illumination (Sec. (.2).. In the BRDF estimation stage, we fix the scene priors and optimize the direct illumination and scaled parameter compute an accurate BRDF of the object under the constraint of rendering equation (Sec. 3.3). To improve the visibility accuracy for direct illumination and decomposition stability, we introduce the regularized visibility estimation after 100 epochs (Sec. 3.4.
3.2 Stage 1: Pre- Pre-processing
In this stage, we adopt the same neural SDF representation and the volume rendering as NeuS 47 to reconstruct the scene. Then can obtain the necessary prior information for the BRDF estimation stage, such as normal, visibility, and indirect illumination from NeuS.
Normal smoothing. In our framework, the accuracy of normal is crucial for BRDF estimation. However, we observed that normals estimated from NeuS tend to be noisy. To overcome this, we drew inspiration from Ref-NeRF [ and employ a spatial ML.P N(x) to predict smooth normals aligned with the density gradient normals (See Fig- P obtained from NeuS using L2 loss. We further employ smooth loss to fix the broken normals caused by specular reflection:
muorm [ á + [N(x) - N(x + 1)
where N denotes the normal at point x learned by MLP, ñ denotes the supervision normal from NeuS, and is 0.02x Gaussian noise.
Visibility and indirect illumination. With the availability of NeuS SDF, we can use sphere tracing model secondary shading effects such as visibility and indirect illumination. However, performing sphere tracing requires a significant amount of time and memory. Inspired by PlenOctree [51], we an octree tracer derived from the NeuS SDF, replacing sphere tracing accelerate the tracing and achieve more precise intersection results. Moreover, We can further improve the inference efficiency by compressing the visibility and indirect illumination field into MLP.
As for indirect illumination, we follow InvRender [56] and model the indirect radiance field Lr wi) using M 24 SGs under the supervision of NeuS radiance field. At point x, first perform octree tracing along direction wi to get the second intersection point a. Then the indirect radiance field can be supervised by the out-going radiance S(i, ww;) from NeuS. Then, the indirect illumination Ly is computed by:
M
Lı(x, I) = Σ G(w;
j=1
where we use an MLP to output the jth indirect SG parameters, and denotes the scaled parameter, which will be illustrated in Sec. 3.3
As for visibility, learn an MLP that maps the point and direction to visibility V(x, w), which is supervised by the result of octree tracer from point x along direction w. The Lindir and Lyis are optimized by En and binary cross entropy loss as follows:
Lindir = ||
Lr ll C... = BCE(V (x,,), V (x, w)),
(5)
where BCE(p: y:) represents the binary cross-entropy (BCE) loss, L, is the radiance value at the second intersection point å obtained by querying NeuS, and V V(x w) is obtained using an octree tracer from point x along direction
3.3 Stage 2: BRDF Estimation
So far, we have faithfully reconstructed the prior information the scene such as the normal, visibility and the indirect illumination. In this stage, aim to accurately evaluate the rendering equation in

---

Inpur
Before Estim
After
smooth lows
smooth low
Figure 2: Smooth loss fix broken part.
Figure 3: Visualization of direct SGs.
order to precisely estimate the surface BRDF albedo a, roughness and direct environment light with the fixed priors from stage 1. However, previous approaches tend to leave shadow and specular reflection in PBR materials under scenes with high illumination. Thus, we apply scene-specific ACES tone mapping to the PBR color output by the rendering equation. The ACES tone mapping can calculate the PBR color over a broader value range. better estimating BRDF without baking shadow through more refined contrast control. We adopt SGs to efficiently approximate the rendering equation as PhySG (44 See complete SGs approximation in the supplementary materials.
Scene-specific ACES tone mapping. We adopt the widely used the ACES tone mapping M. which is type of high dynamic range (HDR) tone mapping. Several recent works [16] 31 have incorporated HDR tone mapping into NeRF for specific applications. Specifically, we apply the ACES tone mapping FF convert the PBR color lying [0, +o0) color lying in [0. 1]:
(2.51e + 0.03)e
F(e) =
(6)
(2.43e 0.59)e + 0.14
whereas the ACES inverse tone mapping Fi is given by:
Fi(c) = 0.59c - 0.03 + 1.0127c + 1.3702e - 0.0009
(7)
2(2.51 2.43c)
Given that the light intensity varies across different scenes, applying ACES tone mapping universally is not feasible. Thus, we introduce an additional learnable parameter (0, 1). This scaled parameter modifies the ACES tone mapping curve, enabling it to automatically adapt to each scene's 's unique illumination intensity. The resulting deformed tone mapping function is defined as follows:
F) - 0.2 F(e), (T))) Filc %.3).
8
Indirect illumination with scaled parameter. In Sec. 3.2 we model the indirect illumination under the supervision from NeuS's 's' radiance field. To convert indirect illumination to the same value range as BRDF estimation, we need map the supervised values from NeuS through ACES inverse tone mapping FP. Since we are not certain of the that best fits the scene during stage 1, we train indirect illumination using randomly sampled y obtain indirect illumination under all possible ? settings. Consequently, the loss function Lindir in Eq. (5) is then revised to include as follows:
Lindir =
Lrlli-
Then in stage 2, we stop training the indirect illumination and treat learnable parameter. The optimal 7 for the current scene will be determined as the decomposition model converges.
BRDF estimation. We use the simplified Disney BRDF [6] model with albedo, roughness, and environment light as parameters and assume dielectric materials with fixed Fresnel term value of Fo 0.02. During the BRDF estimation stage, we adopt N 128 learnable SGs to model direct illumination and represent the PBR materials using an encoder-decoder network. The network initially encodes the input surface point x into its corresponding latent code and then decodes it into albedo and roughness r. To further reduce noise in materials, we incorporate the smooth loss similar to Eq. (3) to both the albedo and roughness, and apply sparsity loss to to ensure that most of channels are close to zero:
Camnorh [zz), (2z + e) ypare = KL(z 0.05),
(10)
where D is the decoder of the PBR material network. KL(p || P) plog + (1 p)log represents Kullback-Leible (KL) divergence loss that measures the relative entropy of two distributions.
5

---

Input
InvRender NVDiffrec
TensolR
NeRO
Relight-GS
GS-IR
Ours
GT
Hotdog
Lego
Helmet
Truck
Figure 4: Albedo in synthetic scenes. We compare our method to InvRender [56], NVDiffrec [4], TensoIR (17), NeRO (26). Relightable-GS (M]) and GS-IR (24). The results show that method
outperforms previous approaches without baking specular highlights and shadows into albedo.
SGs approximation for rendering equation. approximate the rendering equation in Eq. (T)
In RobIR, follow PhySG [54] and adopt SGs to
a
fr (w wi. x
+ fa (W.. w..r
Wi n = G (w: 0.0315, n, 30.7080) 31.7003,
(11)
M
L (xx Σ G En Ae, 7xx)))
Σ Gr(w;:((xs1))
where G is the direct SGs learned in this stage, Gr is the indirect SGs learned in stage 1, n is the surface normal, 7(() = Σ G(w:)V(x.w.) signifies the visibility for direct SGs obtained by -i-o G(w,)
randomly sampling S directions, fa denotes the specular component that be converted single SG. Then, we can integrate the multiplication of these SGs in closed-form (29 to compute the final PBR color w.. For more details about Ss. please see the supplementary materials.
3.4 Regularized Visibility Estimation
One of our primary goals is to achieve clean albedo with no residual shadows, which are typically caused by inaccurate visibility. Despite all efforts of the previous modeling, a small amount of stubborn visibility emrоrs still exist. Therefore, after 100 epochs of BRDF estimation, introduce regularized visibility estimation, directly using MLP Q(x, r) predict the visibility of relative to N direct SGs instead of r calculated through previously learned visibility network Vxx, w). Specifically, @(x, T) is a visibility prediction network learned from scratch under the supervision of 1), while represents the N N identity matrix used to add information for N direct SGs and x E R² is expanded to R. N3 to predict visibility for each direct SG. Since visibility errors primarily occur the edges, which also sparse in the scene, leverage the edge loss to make the residual sparse:
Ledge KL(Ộ((,T) - (x) 0.01).
(12)
In the first 100 epochs, fix V(x,w) w) using 1) obtain stable visibility estimate, avoiding the early collapse of BRDF estimation caused by directly using Q(x, 7). After 100 epochs, with rough BRDF estimation in place, we introduce regularized visibility estimation. By using V(x, w) to distill @(x,T) we directly predict the visibility of point x relative to direct SGs, circumventing errors caused by the sampling direction when calculating 7). Thus, we can achieve a more accurate visibility estimate designed for direct SGs (See Fig. 3)

---

Albedo
Env Map
Relighting
Roughness
Method
PSNR
SSIM T
LPIPS
PSNR 7
SSIM
MAE
PSNR T
SSIM
LPIES
MAE
NVDitfree
16.89
0.8252
0.1965
6.63
0.1397
0.3897
17.33
0.8235
0.2008
0.112
InvRender
19.12
0.8757
0.1652
13.47
0,5796
0.1624
22.57
0.8967
0.1354
0,073
TensolR
20.52
0.8679
0.1537
5.19
0.4064
0.4903
18.66
0.8260
0.1981
0.066
Relightable-GS
17.63
0.8343
0.1695
9.96
0.3354
0.2413
0.104
GS-IR
14.88
0.7618
0.2170
5.10
0.1369
0.4530
17.18
0.8307
0,1891
0.142
Ours-no no aces
21.24
0.8851
0.1421
10.50
0.5446
0.2379
23.61
0.9059
0.1221
0.065
Ours-no
18.51
0.8786
0.1403
10.10
0,5650
0.2486
23.20
0.8981
0.1243
0.059
Ours-Log 21.13
0,8883
1294
17.07
0.6431
0.1091
24.07
0.9003
0.1095
0.077
Ours
25.09
0.9303
0,0972
16.32
0.6351
0.1215
24.65
0.9118
0.0972
0.045
Table 1: Quantitative evaluations. We present the results of the synthetic scenes. We color each
cell as best second best and third best Our method can produce high-quality albedo, roughness,
and environment map while maintaining the relighting fidelity.
(a) avdiffiree (b) InvRender
(c) TensoIR
(d) GS-IR
(e) Relight-GS
ours
(g) gt
helmet hotdog
Figure Environment map. Compared existing approaches, our method can truly achieve high-quality environment light decoupling, avoiding messy results.
Final loss. After incorporating regularized visibility estimation into inverse rendering, our final loss function in the BRDF estimation stage is:
L = |F |F (Cpb)), Carl Asmooth Camoohh Xsparse Exparse + Redge Cedees
(13)
where Cpbr is the physically- basedd color from the rendering equation, FF is the scene-specific ACES tone mapping. Cgr is the ground-trute -ruth color. our experiments, Asmooth. Apparse. and Aedge are set to 0.001, 0.01, and 1.0 respectively.
Experiments
In this section, present the experimental evaluation of our methods. To assess the effectiveness of our approach, we collect synthetic and real-world datasets from NeRF and NeuS without any post-processing. In addition, we use Blender render our own datasets further demonstrate the superiority of our methods in high-illumination scenes. It should be noted that unlike previous methods [17 (55 that used hotdog scene with reduced illumination, we use the original hotdog from NeRF 32 without reduced illumination. See more comparison in the supplementary materials.
Our model hyperparameters consisted of batch size of 1024, with 200k iterations for the NeuS training. The model was implemented in PyTocch and optimized with the Adam optimizer at learning rate of 5c-1. All tests were conducted on single Tesla V100 GPU with 32GB memory. The training time without NeuS is around hours.
Input
InvRender
NVDiffrec
TensoIR
NeRO
Relight-GS
GS-IR
Ours
Chess
Truck
a
G
Figure 6: Roughness in synthetic scenes. The results show that our method can achieve clean roughness, even in scenes with intense shadow interference.

---

Input
InvRender
TensoIR Relight-GS
Ours
InvRender
TensolR
Relight-GS
Ours
Man
Clock
Bear
Figure 7: Comparisons on real-world scenes. Columns 2 to 5 are albedo, the last four columns are roughness. Even complex real-world scenarios, our method can robustly decouple shadow and material, resulting in high-quality albedo and roughness.
Input
InvRender NVDiffree TensoIR Relight GS Log Tone No ACES No RVE
Ours
Truck
Chess
Figure Ablation. We conduct ablation experiments on the key components in the BRDF estimation stage. The ablation results emphasize the critical importance of each component in our proposed framework for attaining high- nigh-quality albedo.
4.1 Comparisons with previous methods
We compare our method with previous state-of-the-ar neural field-based inverse rendering approaches:
NVDifffre (34). InvRender (56). TensoIR [17], NeRO (26). Relightable-Gi GS [[]] and GS-IR (24).
As shown in Fig. and Fig. 6 our method can truly achieve robust BRDF estimation, correctly decoupling shadows, ambient lighting, and PBR materials without baking shadows and specular highlights into albedo and roughness. Other methods tend bake shadows into albedo, which also affects the correct decomposition of object roughness, reflecting their inability to properly separate the various components of BRDF estimation. Even in more challenging real-world scenarios shown in Fig. 7 our method can achieve robust decomposition results without baking shadows and specular highlights into albedo and roughness.
The estimated environment maps are shown in Fig. Our method can accurately estimate the position of the light source and generate more precise light intensity in high-illumination scenes. As far as we know, we are the first to incorporate the accuracy of the estimated environment map into the quality assessment of neural feld-based inverse rendering.
Tab. shows the accuracy of the albedo, roughness, relighting, and environment map averaged over synthetic scenes. We did not measure the relighting of Relightable-GG because it does not support relighting of a single object. The term "Log" refers to the use of sigmoid mapping instead of ACES. We can observe that our method achieve the best results in all inverse rendering tasks. Inaccurate BRDF estimation significantly affects the results of relighting, causing methods with high-quality reconstruction to bake shadows and thus leading a decline in rendering quality during relighting. Overall, our approach can achieve robust inverse rendering in high-illumination scenes.
4.2 Ablation Studies
We perform an ablation study to analyze the importance of the key components in our proposed method. As illustrated in Fig. g our method is unable to eliminate both shadows or specular
8

---

Input
Render
Deshadow
Input
Render
Deshadow
Input
Render
Deshadow
Figure 9: De-shadow, Given an input image from a specific viewpoint, our proposed method can
accurately remove shadows caused by direct light occlusion without sacrificing rendering quality.
Input
GT
Ours
InvRender
TensoIR
GS-IR
Light 0
Light
Figure 10: Relighting. Our method not only achieves high-quality relighting results in scenarios with specular highlights but can also robustly decouple shadows, obtaining high-quality relighting outcomes without baked shadows even in scenes with severe shadows.
reflection in the absence of ACES tone mapping. Without regularized visibility estimation, inaccurate predictions of direct SGs visibility results in residual shadows. The "Log Tone" result indicates that ACES is a more effective tone mapping than the sigmoid to remove shadow within our framework. Finally, our full method can correctly estimate BRDF of the object, resulting in the best performance.
4.3 Application
De-shadowing. De-shadowing is a challenging task in the field of inverse rendering, often requiring strong priors and large data-driven models. Our proposed method correctly understands various lighting effects and is capable of effectively eliminating strong and irregular shadows, particularly in scenes with intense lighting. As shown in Fig. 9 by setting the visibility of direct SGs to can remove the shadow caused by direct light occlusion. It should be noted that our method cannot remove the areas with reflections and the dark regions caused by the backlighting phenomenon.
To demonstrate the practical utility of the materials from our method, we conducted relighting experiments. As shown in Fig. [0] our estimated BRDF results can be accurately relighted in various lighting environments without shadow or illumination artifacts.
Relighting.
5 Conclusions and Discussions
We presented novel inverse rendering framework for estimating BRDF of the object under high- illumination scenes. The key innovation lies in the use of ACES tone mapping, which shifts the calculation of PBR color to a wider value range, significantly reducing the impact of shadows and specular parts on BRDF estimation. In addition, regularized visibility estimation are employed to ensure more acuurate visibility for direct SGs. Experiment results on both synthetic and real-world data show that our method outperforms previous approaches in eliminating shadows and specular reflection under high-illi nigh-illumination scenes.
Currently, the proposed method has some limitations. First, non-solid, translucent, and thin objects cannot be correctly handled due to the limitations of NeuS. Second, the employment of SGs model both direct and indirect lighting presents challenges in dealing with anisotropic objects, consequently leading to our method's deficiency in incorporating the metallic learnable parameters present the Disney BRDF model. Third, we have not considered scenes with dynamic lighting like [28 44). Finally, our method's prior information is limited to multi-view images. We will consider integrating with LLM models in the future work.

---

6 Acknowlegements
This work was supported by Key R&D Program of Zhejiang (No. 2024C01069). We thank Wenxin Sun for her help in pipeline illustration. We also thank Yuan Liu and Wen Zhou for the constructive suggestions.
References
[1] Walter Arrighetti. The academy color encoding system (aces): A professional color management framework for production, post productioo and archival of still and motion pictures. Journal of Imaging, 3(4):00, 2017.
[2] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick hold-Geeffroy, David Kriegman, and Ravi Ramamoorthi. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images.
In European Conference Computer Vision, pages 294-311. Springer, 2020.
[3] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pages 12684-12694, 2021.
[4] Mark Boss, Andreas Engelhardt, Abhishek Kar. Yuanzhen Li, Deqing Sun, Jonathan Barron, Hendrik Lensch, and Varun Jampani. Samurai: Shape and material from unconstrained real-world arbitrary image collections. Advances in Neural Information Processing Systems, 55:26389-264403 2022.
[5] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural-pil: Neural pre-integrated lighting for reflectance decomposition. Advances in Neural Information Processing
Systems, 34:
2021.
[6] Brent Burley and Walt Disney Animation Studios. Physically-based shading at disney. In Acm Siggraph, volume 2012, pages 1-7. vol. 2012, 2012.
[7] Anpei Chen. Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV). 2022.
[8] Ziyu Chen, Chenjing Ding, Jianfci Guo, Dongliang Wang, Yikang Li, Xuan Xiao, Wei Wu, and Li Song. L-tracing: Fast light visibility estimation on neural surfaces by sphere tracing. In European Conference on Computer Vision, pages 217-233. Springer, 2022.
[9] Zhiqin Chen, Thomas Funkhouser. Peter Hedman, and Andrea Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. arXiv preprint
20iv:2208..00277 2022.
[10] Ziang Cheng. Hongdong Li, Yuta Asano, Yingiang Zheng, and Imari Sato. Multi-view 3d reconstruction of texture-less smooth surface of unknown generic reflectance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16226-16235, 2021.
[11] Jian Gho,Chun Gu, Youtian Lin, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. Relightable 3d gaussian: Real-time point cloud relighting with brdf decomposition and ray tracing. arXiv:2311. I.16033, 2023.
Chun
[12] Xinyu Gao. Ziyi Yang, Yunlu Zhao, Yuxiang Sun, Xiaogang Jin, and Changqing Zou. A general implicit framework for fast nerf composition and rendering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1833-1841, 2024.
[13] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14346-14355, 2021.
[14] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkherg. Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. arXiv:2206.03333 2022.
[15] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1975-5884, 2021.
[16] Xin Huang, Qi Zhang. Ying Feng, Hongdong Li, Xuan Wang, and Qing Wang. Hdr-nerf: High dynamic range neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18398-18408. 2022.
[17] Haian Jin. Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang
Xu, and Hao Su. Tensoir: Tensorial inverse rendering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[18] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
[19] Julian Knodt, Joe Bartusek, Seung-Hwan Back, and Felix Heide. Neural ray-tracing: Learning surfaces and reflectance for relighting and view synthesis. arXiv preprint arXiv:2 13562, 2021.
[20] Hendrik PA Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich. and Hans-Peter Seidel. Image- based reconstruction of spatial appearance and geometric detail. ACM Transactions on Graphics (TOG). 22(2):234-257, 2003.
[21] Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. ACM Transactions on Graphics (TOG), 3-16):1111, 2018
[22] Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes; Shape, spatially-varying lighting and svbrdf from a single
10

---

image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2475-2484, 2020.
[23] Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning
to reconstruct shape and spatially-varying reflectance from a single image. ACM Transactions on Graphics (TOG), (7(6))1-11, 2018.
[24]
Zhihao Liang. Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering. arXiv preprint arXiv:2311.
2023.
[25] Lingjie Liu, Jiatao Gu. Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33: 15651-15663, 2020.
[26] Yuan Liu, Peng Wang. Cheng Lin, Xiaoxiao Long. Jiepeng Wang, Lingji Liu, Taku Komura, and Wenping Wang. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. In SIGGRAPH. 2023.
[27] Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Erie R. Chan, Marco Monteiro, and Gordon Wetzstein.
Acorn: Adaptive coordinate networks for neural scene representation. ACM Trans. Gruph. (SIGGRAPH).
40(4). 2021.
[28]
Ricardo Martin-Brualla Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and
Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210-7219, 2021.
[29] Julian Meder and Beat D. Brüderlin. Hemispherical gaussians for accurate light integration. In International Conference on Computer Vision and Graphics, 2018 88
[30] Abhimitra Meka, Mohammad Shafiei, Michael Zollhöfer, Christian Richardt, and Christian Theobalt. Real-time global illumination decomposition of videos. ACM Trunsactions Gruphics, 40(3), aug 2021. [31] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Brualla, Pratul P Srinivasan, and Jonathan T Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16190-16199, 2022.
[32] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[33] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., ((4)::02:11102::11
July 2022.
[34] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1280-82900 2022.
[35] Merlin Nimier-David, Zhao Dong, Wenzel Jakob, and Anton Kaplanyan. Material and lighting reconstruc-
tion for complex indoor scenes with texture-space differentiable rendering. 2021.
[36] Michael Occhslc, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and Proceedings of the IEEE/CVF International Conference
radiance fields for multi-view reconstruction.
on Computer Vision. pages 5589-5599, 2021.
[37] Christian Reiser. Songyou Peng. Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335-14345, 2021.
[38] Carolin Schmitt, Simon Donne, Gernot Riegler, Vladlen Koltun, and Andreas Geiger. On joint estimation of pose, geometry and svbrdf from a handheld scanner. In Proceedings of the IEEL/CVF Conference on Computer Vision and Puttern Recognition, pages 3493-3503, 2020.
[39] Soumyadip Sengupta, Jinwei Gu, Kihwan Kim. Guilin Liu, David W Jacobs, and Jan Kautz. Neural inverse rendering of an indoor scene from single image. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 8598-8607, 2019.
[40] Shuang Song and Rongjun Qin. A novel intrinsic image decomposition method to recover albedo for aerial images in photogrammetry processing. 2022.
[41] Pratul Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7495-7504, 2021. [42] Cheng Sun. Min Sun. and Hwann- Tzong Chen. Direct voxel grid optimization: faser-fast& convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5459-5469, 2022.
[43] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Improved direct voxel grid optimization for radiance fields reconstruction. arXiv preprins arXiv:2206. 00000,00055.55 2022.
[44] Jiaming Sun, Xi Chen, Qianqian Wang, Zhengqi Li, Hadar Averbuch- -lor, Xiaowei Zhou, and Noah Snavely. Neural 3d reconstruction in the wild. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-9, 2022.
[45] Dor Verbin, Peter Hedman. Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref-nerf: structured view-dependent appearance for neural radiance fields. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 5481-5490. IEEF, 2022.
[46] Delio Vicini, Sébastien Speierer, and Wenzel Jakob. Differentiable signed distance function rendering.
ACM Transactions on Graphics (TOG), 41(4):1-18, 2022.
[47] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurlPS, pages 27171-27183,
11

---

2021.
[48] Wenqi Yang. Guanying Chen, Chaofeng Chen, Zhenfang Chen, and Kwan-Yee K Wong. Ps-nerf: Neural inverse rendering for multi-viww photometric stereo. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part pages 266-284. Springer. 2022.
[49] Yao Yao, Jingyang Zhang. Jingbo Liu, Yihang Qu, Tian Fang. David McKinnon, Yanghai Tsin, and Long Quan. Neilf: Neural incident light field physically-based material estimation. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXXI, pages 700-716. Springer, 2022.
[50] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural
Information Processing Systems, 33:2492-2502. 2020.
[51] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li. Ren Ng. and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings the IEEF/CVF International Conference on Computer
Vision, pages 5752-5761, 2021.
[52] Jason Zhang. Gengshan Yang. Shubham Tulsiani, and Deva Ramanan. Ners: neural reflectance surfaces for sparse-view 3d reconstruction the wild. Advances in Neural Information Processing Systems,
34:29835-29844 2021.
[53] Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 5565-552 5574, 2022.
[54] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5453-5462, 2021.
[55] Xiuming Zhang. Pratul Srinivasan, Boyang Deng. Paul Debevec, William T Freeman, and Jonathan Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (TOG), 40(6):1-118 18, 2021.
[56] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei 7hou. Modeling indirect illumination for inverse rendering. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18652, 2022.
12

---

Appendix
This supplementary document provides some implementation details and further results that accom- pany the paper.
Section introduces the differences between the dataset used by our method and those used by previous methods.
Section introduces more details of the SG approximation for the rendering equation.
Section provides additional results, including more visualizations and results on more datasets.
A High-illumination Dataset
Currently, neural field-based inverse rendering methods, such as InvRender [56), NeRFactor [53] and TensoIR [17), generally use scenes with almost no high-intensity ambient light (See Fig. 13 The advantage of these scenes is that the object's BRDF estimation is not affected by self-occlusion shadows, making albedo and color quite similar. As result, even if each part of the BRDF estimation is somewhat messy, plausible results can still be obtained. However, when the scene has intense illumination and shadows, these methods will fail correctly model the object's s BRDF. Therefore. to more accurately evaluate the robustness of inverse rendering, we choose more challenging high-illa nigh-illumination dataset.
B SG Approximation for the Rendering Equation
Following the methodology from (54), we employ the inner product of SGs approximate the computation of the rendering equation. The position x is dropped in the following equation due to the distant illumination assumption. Specifically, the term w; approximated by SG as follows:
W. n G (w,; 0.0315, n, 32.700) 31.7003.
(14)
As for the specular component fs. employ the simplified Disney BRDF model previous methods 6 23
fo (Wor wi) = MM (wo, wi) D(h).
W. + Wi
(15)
Iw + w
where M represents the Fresnel with shadowing effects, and D is the normalized distribution function. To simplify the computation, we assume an isotropic specular BRDF, and adapt D and M follows:
M (wos wi) F (w., w;) g (wo:
(n W.) w;:
F FWWo wi) = 8 + (1 8) 2 (4. 146.816) Wn n
w
9 (wo,
Wo n(1
k) + k
Wi n(1 k)
k (r + 1)²
8
D(h) G h; 2
where s E [0, 1}3 the specular factor, and r denotes the roughness. Finally, we can compute the rendering equation through the fast inner product of SGs (29).
13

---

Fieus
Stool
Mic
Headset
Toy
(a) gt
(b) rendering (c) normal
(d) light
(e) albedo (f) roughness
env-map
Figure 11: Other results of our method. In each scene, present the input ground-truth image
(a), our rendering result (b), normal (c), light (d), alhedo (e), and roughness (f) obtained through our method. These experiments illustrate the generalizability of our method across diverse datasets and demonstrate its ability to produce high-quality results.
Input
GT
Ours
InvRender
TensoIR
GS-IR
Light
Light 2
Light 3
Light 4
Figure 12: Helmet Relighting, Our method achieves high-quality relighting results in scenarios with specular highlights and slight specular reflections.
14

---

Ours-hotdog
TensoIR-
Ours-lego
TensoIR-lego
Figure 13: Dataset Comparison. We choose a more challenging iighiillminatiinn dataset, which exposed the inability of previous neural field-based inverse rendering methods to decouple shadows from the object's PBR materials.
Input
GT
Ours
InvRender
TensoIR
GS-IR
Light 3
Light 4
Figure 14: Hotdog Relighting. Our method achieves high-quality relighting results in scenarios with
severe shadows.
Hotdog SSIM
Lego SSIM
Helmet SSIM
Method
PSNR1
LPIPS
PSNRT
LPIPS.
PSNR1
LPIPS
NVDiftrec
20.60
0.8872
0.1777
18.52
0.8299
0.1616
12.06
0.7866
0.2274
InvRender
15.76
0.8575
0.2029
20.75
0.8606
0.1656
19.50
0.8761
0.1697
TensolR
16.01
0.8496
0.2047
20.74
0.8493
0.1541
16.95
0.8341
0.1759
Relightable-GS
15.34
0.8453
0.2111
20.07
0.8030
0.1580
14.97
0.7946
0.1963
GS-IR
9.72
0.6382
0.3139
13.03
0.6860
0.2386
13.72
0.7774
0,2538
Ours-no aces
22.24
0,8582
0.1779
22.00
0.8675
0.1497
19.98
0.9173
0.1202
Ours-no rve
19.04
0,8487
0.1489
20.17
0.8659
0.1437
14.30
0.8769
0.1493
Ours-Log
19.01
0.8570
0.1560
21.43
0.8596
0.1504
21.87
0.9078
0.1012
Ours
24.25
0.9185
0.0970
24.63
0.9175
0.1071
24.14
0.9427
0.1122
Truck
Stool
Average
Method
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS.,
PSNR
SSIM
LPIPS
NVDiffrec
19.59
0.9010
0.1437
13.69
0.7213
0.2722
16.89
0.8252
0.1965
InvRender
21.68
0.9023
0.1440
17.90
0.8822
0.1440
19.12
0.8757
0.1652
TensoIR
24.06
0.9375
0.1071
24.81
0.8692
0.1269
20.51
0.8679
0.1537
Relightable-GS
19.17
0.8720
0.1528
18.62
0.8567
0.1296
17.63
0.8343
0.1696
GS-IR
19.03
0.8379
0.1729
18.92
0.8695
0.1061
14.88
0.7618
0,2171
Ours-no aces
20.81
0.9177
0.11II
21.15
0.8647
0.1518
21.24
0.8851
0.1421
Ours-no rve
20.59
0.9200
0.1108
18.46
0.8814
0.1511
18.51
0.8786
0.1408
Ours-Log
24.04
0.9418
0.0997
19.29
0.8755
0.1395
21.13
0.8883
0.1294
Ours
27.46
0.9592
0.0647
24.98
0.9136
0.1051
25.09
0.9303
0.0972
Table 2: Quantitative albedo comparison synthetic dataset. We compare method to
several previous approaches: NVDiffrec [14), InvRender (56). TensoIR [17), Relightable-Gr
and GS-IR (24). We report PSNR, SSIM, LPIPS(VGG) and color each cell as best second best and third best.
C Additional Results
More qualitative results. Our method can effectively remove shadows baked into albedo and roughness, thanks to our accurate modeling of each decomposition component. Therefore, our method can certainly handle scenes with less intense lighting. Fig. shows the results of our method on real-world datasets and some synthetic datasets, including scenes with shadows and specular, as
15

---

well as diffuse objects. Our method can robustly perform inverse rendering in any situation without baking shadows and illumination into PBR materials.
Per-scene albedo results. We present the complete metrics of our method compared to other methods Tab. 2 The estimated albedo in our method surpasses existing SOTA methods in every synthetic scene.
More relighting. We show more relighting results in Fig. 12 and Fig. 14 The scenes demon- strate that our method can accurately estimate the BRDF of the object under scenes with specular highlights and severe shadows.
Additional comparison with NVDiffreMMC. We show more albedo, roughness, and environment
map comparison with NVDiffrecMC [14) in Figs. 15 17
Input
NVDiffRecMC
Ours
GT
Figure 15: Albedo comparison with NyDiffRecMC on synthetic scenes. NVDiffRecMC cannot achieve the decouple of shadow, indirect illumination, and the PBR materials of the objects.
NVDiffRecMC
Ours
NVDiffRecMC
Ours
Figure 16: Roughness comparison with NyDiffRecMC. NyDiffRecMC cannot obtain high-quality roughness in high illumination scenes.
NVDilIRecMC
Ours
GT
Figure 17: Environment map comparison with NyDiffRecMC.
16

---

Visualization and evaluation on tone mapping. We first visualize the vanilla ACES and sRGB tone mapping in Fig. [8] indicating that ACES curve has much wider input range. Then Fig. [9 we show the scene-specific tone mapping curve with different 7, enabling the ACES curve to fit other settings with different tone mapping methods. Finally, we evaluate the optimized ACES curve (with 0.42) in chessboard scene with GT tone mapping (sRGB) in Fig. 20 The results show that our scene-specific ACES tone mapping can stretch to sRGB curve, demostrating the effectiveness of our method.
ACES and U0C0 Durs
44
4.0
Figure 18: Comparison on ACES and sRGB curve.
Figure 19: Visualization of ACES tone mapping with different 7-
Evaluation of the Tone aan-iMaping Curve
1BC6 Tore Nagsing
1.2
Chesstaaddip
1.0
0.0
0.5
Figure 20: Evaluation tone mapping in chessboard. The ACES tone mapping with 0.42 matches well with the sRGB curve.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: I am sure that the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: I am sure that the paper discuss the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
Yes, the paper does.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, the paper does.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [No]
Justification:
The code can be released upon acceptance, but now it's not a clean version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwiii/gguie///eeeeeeoeic CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper contains details about the training model.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We evaluate the results through PSNR, SSIM and LPIPS.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: They are presented in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: ftttp:://neurippe http///urrrs.ccppui//i/e httpp//c.cccpp.....ttttttcteeeee
Answer: [Yes]
Justification: Yes, do.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
21

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, they do.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthcee....c... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
22

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "u4eU7dmdht", "reviewer": "Reviewer_ySN8", "summary": "This paper introduces RobIR, an inverse rendering approach designed to handle strong or directional illumination scenes with strong shadows and specular reflections. \nThe proposed method aims to decouple environment lighting and object materials, with the goal of producing high-quality albedo without baked shadow.\n\nBuilding on top of prior inverse rendering methods such as InvRender, RobIR introduces two components that further boost the reconstruction quality: (1) ACES tone mapping with an optimizable gamma parameter to better capture the image formation process; (2) regularization for visibility estimation. \n\nRobIR demonstrates better performance over existing methods in quantitative and qualitative evaluations.", "questions": "This paper addresses the shadow baking issue in inverse rendering by proposing two straightforward but effective techniques: optimizable tone-mapping and visibility regularization.\n\nHowever, I am not fully convinced that optimizable tone-mapping significantly benefits shadow removal, and without further clarification it’s challenging to evaluate the technical correctness of this component. In the rebuttal, please prioritize addressing Weaknesses 1.1-1.4. \n\n**[Post rebuttal update]**\n\nThe rebuttal and related discussion address my concerns. I update my rating to Weak Accept.", "limitations": "The paper discussed limitations in the end of paper. It’ll be beneficial to include failure cases in Appendix.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The model design choices are valid and sensible. \n- The experiments and ablation study are thorough. \n- The paper is well written and easy to follow.", "weaknesses": "[W1] The benefit of ACES tone-mapping is a bit surprising. Despite a more accurate formulation for the image formation process, the task of inverse rendering is inherently still an ill-posed problem. It’s a surprising conclusion that a tone-mapping formulation can robustly and significantly benefit shadow removal. \n\nWith many other regularization terms entangled, it’s a bit hard to evaluate the correctness of this specific component. \nI’d hope to know more details in the following aspects: \n\n\n[W1.1] Missing visualization of the tone-mapping curve. Despite an important contribution, the estimation results of the tone-mapping curve are missing. What does the default ACES tonemapping look like, and what does the final tonemapping look like with the optimizable gamma? \n\nIn the revised version, the curves should be qualitatively visualized and included in main paper. From the existing experiments (such as the PSNR metrics), the audience cannot intuitively understand how well the tonemapping is estimated. \n\n[W1.2] Missing evaluation of the tone-mapping curve. \n\nAs many datasets do not have GT tonemapping, it’s unclear how accurately the tone-mapping approximate the GT tonemapping. \n\nOne way to evaluate is to add additional tone adjustment to the input dataset. Assume with the original dataset images $\\{ I \\}$ and the method originally reconstructs a tonemapping curve $f$. Given a new tone adjustment function, e.g. $g(x) = x^\\kappa$, the adjusted dataset images become $\\{g(I)\\}$. Re-running the method can get a newly reconstructed tonemapping curve $f_\\kappa $. The consistency between $g \\circ f$  and $f_\\kappa$ can indicate how well the model can approximate the additional introduced tone adjustment function. $\\kappa$ can be set to values like 0.5 or 2. \n\n[W1.3] The evaluation metric on the Albedo is flawed. As albedo estimation/optimization often involve an unknown scale, PSNR alone is not a proper evaluation for Albedo. Check out [1] for more analysis and more appropriate scale-invariant metrics.\n\n[W1.4] Most of the results are from synthetic datasets, where the GT tonemapping could potentially be close to ACES. For real-world results in Fig.7, the albedo estimation looks strongly regularized and over-smoothed. \n\n[W1.5] Are the radiance values (before tonemapping) and indirect illumination in HDR? If so, what is the activation function? \n\n[W2] The proposed method involves a complicated training pipeline (two stages with each stage have its own loss scheduling), and the novelty is relatively limited. \n\n\n**References** \n\n[1] Grosse et al., Ground truth dataset and baseline evaluations for intrinsic image algorithms, ICCV 2009.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad and appreciate that the reviewer recognize that the results of RobIR is thorough. Our response to the reviewer’s comments is below:\n\n**Q1: Visualization of the tone-mapping curve.**\n\nGreat suggestion. We highly appreciate the suggestion, which can improve the readability of the article. We have already submitted visualizations with different tone mappings and different gamma settings in the submitted rebuttal PDF.\n\n\n\n**Q2: Missing evaluation of the tone-mapping curve.**\n\nThis is also a very good suggestion. We would like to thank the reviewer for the in-depth understanding and for attempting to provide a method to evaluate real-world scenarios without GT tone mapping. Here, we provide tone mapping evaluations for two our rendered scenes: truck and chessboard.\n\n- Truck: We rendered the scene in Blender and used Blender's built-in Filmic tone mapping to handle the high dynamic range. The final optimized $\\gamma$ for the `truck` scene is 1.0, meaning the optimized tone mapping is the vanilla ACES. Considering that the Filmic curve is modified from ACES and their distributions are generally consistent with only some differences in detail, we believe that the optimization of tone mapping for this scene is accurate.\n- Chessboard: We also rendered this scene in Blender, but used sRGB to process the BRDF output. The final optimized $\\gamma$ value is 0.42. In the submitted PDF, we compared the differences between the ACES curve with $\\gamma=0.42$ and the sRGB curve. It can be seen that $\\gamma^{-0.2}$ stretches the ACES curve very well, making it closer to the distribution of the sRGB curve. Combined with more accurate visibility modeling (NeuS Octree and Regularized Visibility Estimation), we can remove the baked shadows and reflections from the PBR material in the chessboard scene (See Fig. 6 and Fig. 8).\n\n\n\n**Q3: Albedo quantitative comparison.**\n\nWe greatly appreciate the reviewer's attention to the hue differences in albedo in inverse rendering, as directly using PSNR might lead to unfair comparisons. We carefully studied the \"Ground Truth Dataset and Baseline Evaluations for Intrinsic Image Algorithms\" for evaluating and decomposing albedo and found that their decomposition formula is $I(x) = S(x)R(x)+C(x), \\text{where}\\  S(x), R(x), C(x)\\  \\text{denotes illumination, albedo, specular respectively} $, which is not compatible with our method. Nonetheless, we are grateful to the reviewer for suggesting a more reasonable evaluation of albedo.\n\n\n\nRegarding the evaluation of albedo, we believe that qualitative assessment is more important. Therefore, in Fig. 4, we provide extensive comparisons to demonstrate that our method does not bake shadows into the albedo. The removal of shadows cannot be solved merely by adjusting the hue (scale). Quantitative experiments only serve as a supplement to albedo evaluation. Although every method has issues with hue affecting PSNR to some extent, our method remains the closest and does not bake shadows. Furthermore, we also provide SSIM and LPIPS metrics, which are less sensitive to hue, and these metrics consistently show that our method achieves higher quality albedo, in agreement with Fig. 4.\n\n**Q4: Albedo estimation in real-world scenes looks over-smoothed.**\n\nIn Fig. 7, the `Man` scene is actually successful in removing complex lighting and shadows, resulting in a smooth albedo, as the material of the sculpture is uniform. However, in the `Bear` scene, there is indeed an issue of over-smoothing. As mentioned in the common response, the smooth loss does have some impact, but it is more attributable to the rendering precision of NeuS. We are very much looking forward to considering structures that balance both rendering quality and geometric quality, such as 2DGS, in our future work.\n\n\n\n**Q5: About the radiance values and indirect illumination.**\n\nYour understanding is very accurate. The radiance values (before tone mapping) and indirect illumination are in HDR, which are then mapped to the $[0,1]$ using scene-specific ACES tone mapping. The activation functions for albedo and roughness are **sigmoid**, while SG-based direct light is modeled as `nn.Parameter`. The indirect light is supervised by NeuS (with NeuS supervision values $[0, 1]$ are mapped back to the same range in BRDF estimation stage using the inverse ACES mapping in Eq. 7).\n\n**Q6: The statement on the role of scene-specific ACES tone mapping.**\n\nIn Figure 8, we conducted a rigorous ablation study on the effect of ACES. Introducing scene-specific tone mapping indeed plays a crucial role in shadow removal.\n\nFinally, we would like to thank the reviewer once again. Many of these points were things we had not noticed before, and these suggestions will significantly improve the readability of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SONb9Q5T41", "reviewer": "Reviewer_GRUR", "summary": "This paper introduces RobIR, an inverse rendering approach that can better tackle “high-illumination” scenes. RobIR first leverages the existing neural field model (NeuS) to represent 3D geometry information including normal, visibility, and indirect illumination. It then utilizes these geometry priors to decompose rendering attributes of the scene through the approximated rendering equation with spherical Gaussian. This work introduces an optimizable ACES tone-mapping and regularized visibility estimation model to better handle HDR color and shadow occlusions, respectively. Their experiment demonstrates some impressive results on shadow disentanglement.", "questions": "1. What are the differences between the final optimized ACES tone mapping and sRGB gamma tone mapping? How does this optimized tone mapping vary from scene to scene? It would be better if tone-mapping curves were included in the paper.\n2. I tested this ACES tone mapping mentioned in the paper, it seems that the proposed learnable ACES tonemapping (an S-curve) cannot well approximate the existing concave tone-mapping curves that are potentially used for rendering Blender objects (e.g., sRGB curve, Filmic curve, AgX curve, etc.). Given this limitation, how does the paper address the color mismatch in the low-illumination color space?\n2. The paper does not show the metrics for rendering (NVS) with decomposed attributes. I’d like to see the rendering quality metrics for those NeRF scenes.\n3. It would be better if the author could show some video examples of moving shadows while rotating envmaps in the final release.\n4. Why does the learnable parameter $\\gamma$ have an exponent 0.2?", "limitations": "The limitations are adequately discussed in the last paragraph of the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work has some further thoughts on color tone mapping for the typical multi-view inverse rendering. The proposed optimizable ACES tone mapping looks very effective in improving inverse rendering results.\n2. The proposed visibility representation (RVE) also plays an important role in the final results. RVE with its neural net accumulates and denoises Monte Carlo samples to achieve more accurate and stable visibility evaluation. Their efforts to refine the visibility should be appreciated.\n3. I am glad the authors clearly point out they use the original NeRF rendering of Hotdog and Lego, instead of the NeRFactor’s.", "weaknesses": "1. This paper still follows a commonly used multi-stage inverse rendering strategy with neural fields. The geometry representation is based on NeuS; the rendering formulation (SG rendering, visibility, and indirect illumination) is mainly based on InvRender; The proposed optimizable ACES tone and REV are more like incremental improvements over InvRender. The key rendering formulation and optimization remain the same as the prior works. Therefore, the novelty of this work is moderate.\n2. The description of RVE in Sec. 3.4 is not very clear.  It seems that MLP $Q(x, \\tau)$ directly outputs N visibility ratios, thus $\\eta(x)$ in Eq. 12 should also be an N-dim vector instead of a scalar value.\n3. The proposed method is quite time-consuming. Training time even without NeuS is around 5 hours.\n4. The proposed regularization terms may hurt the high-frequency details in real-world scenes (Fig. 7).\n5. This method is limited to dielectric materials, without the consideration of metallic and glossy objects, as already pointed out by the authors.\n6. The paper should include some inverse rendering methods with differentiable path tracing, as these methods can explicitly handle visibility, for example, NvDiffRecMC, Mitsuba, etc.\n7. Minor errors:\n    * L275 accuurate -> accurate\n    * Figure 8: Hotdog label is wrong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review as well as the suggestions for improvement. We will revise the typo errors in the paper based on these insightful suggestions. Our response to the reviewer’s comments is below:\n\n**Q1: Comparison with inverse rendering methods with differentiable path tracing.**\n\nGreat suggestion. We will add citations to these papers in our paper. We have shown a comparison with `NvDiffRecMC` in the rebuttal PDF. As can be seen, NvDiffRecMC still cannot remove baked shadows and indirect illumination from PBR materials, demonstrating the robustness of our method.\n\n**Q2: Show some video examples of moving shadows while rotating envmaps.**\n\nOf course, no problem. Unfortunately, we are unable to submit a video during the rebuttal period. However, we can provide other examples to demonstrate that our results are indeed strong. In Fig. 9, we offer several examples that illustrate the robustness of our method in shadow removal, which can partially demonstrate our capability to achieve the task you mentioned.\n\n**Q3: This method is limited to dielectric materials.**\n\nWe are very grateful to the reviewer for pointing out this issue and noticing that we discussed it in the Limitations section. We want to emphasize that our approach is fundamentally a general method that can be integrated with other NeuS-based inverse rendering methods for glossy objects, such as NeRO. In the NeRO paper, there is also an issue with shadow baking in PBR materials for glossy objects. We believe that our approach can help NeRO achieve higher quality PBR material decomposition for metallic and glossy objects.\n\n**Q4: Novel view synthesis on NeRF synthetic scenes.**\n\nFrom our perspective, we believe that novel view synthesis (NVS) is not the primary focus of inverse rendering. Instead, we are more concerned with relighting and the decoupled BRDF components and PBR materials. Using the rendering equation to constrain the process can negatively impact NVS performance, resulting in rendering metrics that are less competitive than methods that directly output color. Additionally, NVS performance is highly sensitive to the base method used; for example, approaches based on NeuS are likely to perform worse in NVS compared to methods based on TensoRF or 3D-GS, which can introduce a degree of unfairness. Despite this, we still provide NVS results (on hotdog, lego, ficus, and mic) to demonstrate that the fidelity of our reconstruction is acceptable.\n\n|           | PSNR  | SSIM   | LPIPS  |\n| --------- | ----- | ------ | ------ |\n| Ours      | 30.11 | 0.9466 | 0.0528 |\n| NVDiffrec | 29.81 | 0.9732 | 0.0345 |\n| InvRender | 27.64 | 0.9056 | 0.0982 |\n| TensoIR   | 32.47 | 0.9637 | 0.0377 |\n\nWe want to emphasize once again that, although our NVS quality may not be the highest, the image supervision is sufficient to decouple high-quality and smooth PBR materials. These can be effectively used for tasks that are of greater interest in inverse rendering, such as relighting and shadow removal.\n\n**Q5: Why does the learnable parameter $\\gamma$ have an exponent 0.2.**\n\nThis is a very detailed question. It's an engineering trick to allow ACES to stretch across a wider range.\n\n**Q6: The use of $\\eta (x)$ in Eq. 12 is inaccurate.**\n\nThank you very much for pointing out this issue. Your understanding is completely correct. In Eq. 12, the supervision value for $Q(x, \\tau)$ should be an N-dimensional vector, representing the visibility of N direct SGs at coordinate **x**. We will replace $\\eta (x)$ with a different mathematical symbol in the paper.\n\n\n\nFinally, we would like to thank the reviewer once again. Many of the reviewer's comments were very accurate, which truly made us very happy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OcPdwkNjFc", "reviewer": "Reviewer_QVta", "summary": "This paper proposes a method for the inverse rendering of high-illumination and highly reflective scenes. There are two training phases, in the first phase, it trains by Neus, to get geometry and compute visibility by octrees. In the second phase, it decomposes lighting as SGs and material by MLPs.", "questions": "How does the relighting work? Do you use original indirect lighting? \nWhy is visibility divided into two stages, and why not directly use the results calculated by the Neus octree as the ground truth to supervise SG? Instead, why is the Neus octree result used as the ground truth to supervise the MLP, and then the MLP's distribution used to supervise the SG results? What are the differences?", "limitations": "See above.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Direct and indirect lighting, visibilities are presented by SGs, which is compact. \nFrom the results, shadows and specularities are decomposed well. \nTone mapping is used, as NeRF in the Dark, which improves results for high-illumination scenes.", "weaknesses": "Figure 2 gives comparisons with and without smooth loss, however, the one w/o smooth loss is better, while the loss may over-smooth the details. \nFigure 11 shows results where the lighting consists of the original colors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive review as well as the suggestions for improvement. Our response to the reviewer’s comments is below:\n\n**Q1: How does the relighting work.**\n\nWe imported the reconstructed albedo and roughness maps into Blender and performed relighting using Blender's scripting. Fig. 10 aims to demonstrate that we can extract high-quality PBR materials that can be directly applied in creative tools like Blender.\n\n**Q2: Why use an MLP instead of directly using the results from the NeuS Octree.**\n\nGood question. This is because querying Octree tracing is extremely time-consuming. By using an MLP, we can cache the results of the NeuS Octree and speed up inference.\n\n**Q3: Over-smooth details in Fig. 2.**\n\nThis is a finding we want to highlight: smoothing helps correct geometric errors. Although the smooth loss results in some loss of detail, it effectively fixes geometric breakages caused by reflections or shadows in the normal map. Addressing these issues will aid in the decoupling of shadows and indirect illumination.\n\n**Q4: Environment map consists of the object color.**\n\nThe estimation of environment map is still not robust in current IR frameworks. However, we want to emphasize that, as shown in Figure 5, our method has significantly improved the accuracy of environment map estimation compared to previous approaches. Additionally, we are one of the few methods that quantitatively compare environment map metrics (see Table 1). We will focus on addressing the estimation of environment maps in cases like those shown in Figure 11 in our future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hgxArhFlht", "reviewer": "Reviewer_ijRb", "summary": "This paper addresses inverse rendering in high-illumination scenes with strong shadows where past methods bake shadows and highlights into estimation results. This paper proposes to use ACES tone mapping and makes it scene-dependent for inverse rendering in high-illumination scenes. This paper also proposes to directly estimate the visibility of each spherical Gaussian of direct illumination instead of a visibility field, which enables an accurate representation of shadows at the edge. The experimental results on the synthetic and real-world datasets show that the proposed method can estimate accurate albedos, surface roughness, and illumination without artifacts in the high-illumination scenes.", "questions": "- A more detailed explanation of the effects of ACES tone mapping is expected. Why is it more suitable for high-illumination scenes than other tone mapping methods such as sRGB and log tone mapping? How does it affect the loss and the optimization?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper proposes a novel regularized visibility estimation that enables an accurate representation of shadows at the edge.\n+ Experimental results show that the proposed method successfully estimates BRDF and illumination while existing methods suffer from artifacts. They also indicate the effectiveness of ACES tone mapping compared with log tone mapping methods.", "weaknesses": "- It is unclear why the ACES tone mapping, of which usage is the key contribution, enables the robust inverse rendering of high-illumination scenes with strong shadows.\n- The proposed method loses the albedo of the detailed texture due to smoothness loss in Eq. 10. For example, Bear in Fig. 7 and truck in Fig. 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad and appreciate that the reviewer recognizes the novel ideas of RobIR and the proposed method successfully estimates BRDF and illumination. Since many of the questions have already been answered in the common response, our additional response to the reviewer’s comments is below:\n\n**Q1: Why ACES is more suitable than other tone mapping like sRGB and sigmoid.**\n\nAs previously mentioned in the common response, high-illumination scenes often require HDR tone mapping to produce the final image, mapping values from 0 to A into the range of $[0, +\\infty]$. However, sRGB is not designed for processing HDR input. ACES allows PBR output colors to be in a larger value range, which better distinguishes very dark shadow regions and aids in the decoupling of object materials. \n\nAdditionally, ACES has significant advantages over sigmoid curves. Although both map an infinite range to $[0,1]$, ACES is a more commonly used HDR tone mapping curve in graphics, consistent with many rendering engines, and provides better contrast control while preserving more detail in highlights and shadows.\n\n**Q2: Lost details in albedo.**\n\nAs previously mentioned in the common response, although the smooth loss does cause some loss of detail, the rendering precision of NeuS is a bigger factor in this loss. In the submitted rebuttal PDF, we included the results of NeuS rendering, which clearly show some blurring. This blurring affects other BRDF components supervised by NeuS, exacerbating the over-smoothing issue. Therefore, we are very eager to use methods with higher rendering and geometric quality, such as 2DGS, in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable comments. We are glad and appreciate that the reviewers recognize that our proposed regularized visibility estimation and ACES tone mapping are novel, and our experiments are thorough and impressive. We will further polish our paper and release our codes. 



We would first like to clarify the contributions of RobIR and provide the common response for the explanation of details absence and ACES tone mapping. Following that, we will address the specific questions posed by each reviewer.



**1. Contribution:**

We are very grateful to reviewer `GRUR` for pointing out that we used the original NeRF rendering of Hotdog and Lego instead of NeRFactor’s. This actually highlights the fundamental contribution of our work. Currently, most neural field-based inverse rendering methods are applied to scenes with low light intensity (as shown in Fig. 13), which avoids the challenge of decoupling shadows, indirect illumination, and PBR materials. This reflects that the components decomposed by current inverse rendering methods are often messy and fail to decouple shadows from PBR materials in high illumination scenes (see Fig. 4).

Through this paper, we aim to draw the community's attention to this basic issue in current IR frameworks, and we propose our solution by explicitly introducing ACES tone mapping and more accurate visibility modeling.



**2. The absence of the texture details in Fig. 4 and Fig. 7.**

Reviewers `ijRb` and `GRUR` believe that our regularization terms may lead to a loss of high-frequency details. The loss of some details is indeed an issue brought about by smooth loss. Additionally, we believe that the rendering accuracy of NeuS is a more significant reason for the loss of details. However, we want to emphasize that **the quality of geometry** is crucial for correctly understanding the scene and decoupling materials and shadows. Only through NeuS's high-quality geometric reconstruction and smooth loss correction of geometric errors can we correctly decouple shadows from the object's PBR materials, ensuring that shadows or indirect illumination are not baked into the PBR materials (**No shadow residual** in Fig. 7 bear and Fig. 4 truck). We look forward to incorporating work that excels in both geometric reconstruction and rendering quality, such as 2DGS, in future endeavors to enhance the details of PBR materials.



**3. Why ACES can make a difference in BRDF estimation.**

We are very grateful that almost every reviewer raised this issue; it is a very profound question. We believe that the introduction of scene-specific ACES tone mapping has two benefits for BRDF estimation. 

- First, as reviewer `ySN8`mentioned, incorporating ACES tone mapping undoubtedly provides a more accurate formulation for the image formation process, especially for high-illumination scenes (since rendering in such scenes often requires HDR tone mapping). 
- Second, and what we consider a more direct reason in neural field-based inverse rendering, is that it offers a broader value range for the calculation of PBR color (Lines 159-163). In the submitted rebuttal PDF, we provided a comparison **between ACES and sRGB tone mapping**. It can be observed that the input for sRGB is within the range of $[0, 1]$, whereas ACES extends from $[0, +\infty]$. This allows the color obtained through PBR to have a greater contrast when aided by ACES tone mapping, thereby helping to decouple the shadows in extremely dark areas from the PBR material.

However, not all scenes with shadows or high illumination undergo HDR tone mapping, and there are differences between various HDR tone mapping methods (as reviewer `GRUR` mentioned, such as Filmic and AgX curves). Therefore, we introduced a $\gamma$ correction (see Eq. 8) to stretch the ACES curve, allowing it to adapt to different light intensities and to approximate various tone mapping curves. We also included a comparison of curves under different $\gamma$ values with other tone mapping methods in the submitted rebuttal PDF.



Overall, we believe that the introduction of scene-specific ACES tone mapping undoubtedly provides a foundation for more precise BRDF estimation. Its inclusion is essential for the decoupling of shadows, and when combined with our proposed Regularized Visibility Estimation (RVE), it enables more robust shadow removal. As shown in the Fig. 8 ablation study, neither ACES nor RVE alone can achieve reliable shadow removal, but together they can truly make a difference.

Meta-review: The paper presents a new approach to inverse rendering, specifically designed for challenging high-illumination scenarios. The method introduces scene-specific tone mapping and regularized visibility estimation to effectively decouple environment lighting from an object's BRDF, resulting in improved albedo and roughness estimations without shadow interference. The paper is well-written and supported by comprehensive experiments, demonstrating superior performance over existing methods.

While the proposed approach is novel, some reviewers noted that it builds incrementally on existing frameworks like NeuS and InvRender, and the complexity of the method may limit its practical application. Additionally, there were concerns about the loss of high-frequency details due to the smooth loss used in the model, which the authors addressed by attributing it to the rendering precision of NeuS. Despite these concerns, the authors provided thorough rebuttals that clarified the benefits of their approach, particularly the advantages of ACES tone mapping in HDR contexts.

After the discussions, some reviewers raised their initial scores, recognizing the paper's contributions. Given the strengths of the work and the constructive engagement with reviewers, the paper is recommended for acceptance as a poster presentation.
Decision: Accept (poster)