SplitNeRF: Split Sum Approximation Neural Field for Joint Geometry, Illumination, and Material Estimation
Authors: Jesus Zarzar
Bernard Ghanem
Venue: NeurIPS 2024 poster (2024)
Keywords: inverse rendering
neural rendering
physically-based
Primary area: other

Abstract: We present a novel approach for digitizing real-world objects by estimating their geometry, material properties, and environmental lighting from a set of posed images with fixed lighting. Our method incorporates into Neural Radiance Field (NeRF) pipelines the split sum approximation used with image-based lighting for real-time physically based rendering. We propose modeling the scene's lighting with a single scene-specific MLP representing pre-integrated image-based lighting at arbitrary resolutions. We accurately model pre-integrated lighting by exploiting a novel regularizer based on efficient Monte Carlo sampling. Additionally, we propose a new method of supervising self-occlusion predictions by exploiting a similar regularizer based on Monte Carlo sampling. Experimental results demonstrate the efficiency and effectiveness of our approach in estimating scene geometry, material properties, and lighting. Our method attains state-of-the-art relighting quality after only ${\sim}1$ hour of training in a single NVIDIA A100 GPU.

SplitNeRF: Split Sum Approximation Neural Field for Joint Geometry, Illumination, and Material Estimation
Jesus Zarzar KAUST
Bernard Ghanem
KAUST
Abstract
We present novel approach for digitizing real-world objects by estimating their ge- ometry, material properties, and environmental lighting from set of posed images with fixed lighting. Our method incorporates into Neural Radiance Field (NeRF) pipelines the split sum approximation used with image-based lighting for real-time physically based rendering. We propose modeling the scene's 's lighting with a single scene-specific MLP representing pre- rreiinteratted image-based based lighting arbitrary resolutions. We accurately model pre-integrated lighting by exploiting a novel regularizer based efficient Monte Carlo sampling. Additionally, propose a new method of supervising self-occlusion predictions by exploiting a similar regularizer based Monte Carlo sampling. Experimental results demonstrate the efficiency and effectiveness of our approach in estimating scene geometry, material properties, and lighting. Our method attains state-of-the-ae relighting quality after only ~ hour of training in single NVIDIA A100 GPU.
Ground Truth Env. Map
Predictions
Env. Map
Metalness
Roughness
Render
Render
Albedo
Normals
Relighting
Courtyard
Interior
Sunrise
Sunset
Figure We visualize the lighting, material properties (albedo, metalness, and roughness), and geometry predicted by model in addition to four relighting predictions of the 'materials' scene. Our method predicts high-frequency illumination with only ~ hour of training thus enabling the efficient digitization of relightable objects.
Introduction
The idea of creating realistic and immersive digital environments has piqued the imagination of countless science fiction authors, science fiction directors, and scientists. In the past few years, the fields of computer graphics and computer vision have advanced so much that we are capable of
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

creating photo-realistic environments 28,28 21] as well as capturing real-world environments in a way that allows to render new photo-realistic views [2]. However, the creation of digital twins [9] of objects that can be integrated within photo-realistic environments still requires artists to meticulously hand-design realistic object meshes, materials, and lighting. While this is feasible for generating few scenes, large-scale digitization requires automatic ways of reconstructing real-world objects along with their corresponding material properties.
In this work, we address the problem of object inverse rendering: extracting object geometry, material properties, and environment lighting from a set of posed images of the object. Inverse rendering enables the seamless integration of virtual objects into different environments with varying illumination conditions from simple image captures taken by commonplace camera sensors.
Neural rendering methods, such as Neural Radiance Fields (NeRF) 18 novel view synthesis, 3D reconstruction from images, and inverse rendering. By directly modeling outgoing radiance at each point in 3D space, NeRF methods excel at accurately recovering scene geometry and synthesizing novel views. However, a drawback of this approach is that the learned radiance representation entangles environment lighting with the rendered scene's 's properties, making it challenging to recover material properties and illumination. Due to the success of NeRFs in reconstructing scenes, several works have proposed modifications to enable inverse rendering (2 3 These works build upon NeRF by decomposing radiance into function of illumination and material properties but differ in their ways of modeling lighting and reflections. We build upon these methods with the main goal of efficiency without sacrificing reconstruction quality or the ability to recover higu-frey illumination details.
have revolutionized
16 I0 41
To achieve these goals, we rely the split sum approximation 11] which is commonly used in efficient image-based lighting techniques and has been successfully applied for inverse rendering before 3 20). This approximation involves splitting the surface reflectance equation into two factors: one responsible for pre-integrating illumination and the other for integrating material properties. The separation allows us estimate pre-integrated illumination using Multi-Layer Perceptron (MLP). This manner of modeling the pre-integrated illumination function is inspired by the modeling of radiance fields, which model a complex integral of lighting and material properties using an MLP. Correspondingly, our illumination representation inherits beneficial properties observed with the modeling of radiance fields such as smoothness. While MLPs representing pre-integrated illumination have been previously exploited [14 13 previous works do not supervise the MLP's learning, leading to physically inaccurate illumination representations. We propose a novel regularizer based on Monte Carlo sampling to ensure accurate learning of illumination.
However, the split sum approximation on its own does not take into account self-occlusions. This hinders material property estimation since shadows tend to be incorrectly attributed to objects' albedo. Thus, we derive an occlusion factor to alleviate the effect of self ceff-occlusions. This factor then approximated via Monte Carlo sampling and used to supervise an occlusion MLP.
Altogether, our method is capable of attaining training on a single NVIDIA A100 GPU.
relighting results with under an hour of
Contributions. We claim the following contributions:
(i We propose novel representation for pre-integrated illumination single MLP with corre- sponding regularization to ensure learning a physically-meaninghhh representation.
(ii) We derive method for approximating the effect of self-occlusions pre-integrated lighting and use to supervise an occlusion MLP improving material estimation.
(iii) We demonstrate the effectiveness of our method in extracting environmental lighting. geometry, and material properties by achieving competitive reconstruction and relighting quality on both synthetic and real data with under one hour of training on a single NVIDIA A100 GPU.
Related works
The problem of digitizing real-world objects and environments has long been a subject of active research in computer vision and computer graphics. We approach this problem through the lenses of neural rendering and neural inverse rendering; paradigms with lots of recent attention. We now provide brief overview of related works in these areas.

---

2.1 Neural rendering and 3D reconstruction
Novel view synthesis the task of rendering new views of a scene given set of observations of
the scene. Neural Radiance Fields (NeRF) [88] and its variants [1] 19 4 26] have demonstrated remarkable success in the task of novel view synthesis. NeRF directly models the volumetric scene function by predicting radiance and density at each 3D point in space while supervising learning with photometric reconstruction loss. Due to its success implicitly learning accurate 3D reconstructions, several works have branched out to reconstruct accurate meshes through neural rendering [22 31].
Signed Distance Funetion (SDF)-based methods [6 40,
37 12 model density as a function of
the SDF to obtain well-defined surfaces. By increasing sharpness during training in the conversion from SDF to density these methods can transition from volume rendering to surface rendering as they train. While effective, these methods suffer from entangled representations of scene geometry, material properties, and lighting. Our work follows the surface rendering pipeline proposed in [36] but reformulates the radiance prediction to disentangle environment lighting and material properties.
2.2 Neural inverse rendering
The task of inverse rendering is long-standing problem in computer graphies which consists of estimating the properties of a 3D scene such as shape, material, and lighting from a set of image observations. The success of neural rendering methods for novel view rendering and 3D
reconstruction has led to variety of works [2] 45
43
29
3
20
46 42
14
16 30 exploiting
neural rendering for inverse rendering. Due to the challenging nature of this problem, multiple simplifying assumptions have been adopted. Some works simplify the modeling of lighting by using low-frequency representations such as spherical gaussians 2 45
43
29
46
10 or low-resolution 39
environment maps [2] 45 46). While this approximation generally allows for closed-form solutions of the rendering integral, it does not capture natural high-frequency illumination. Our work leverages the split sum approximation [Н.) proposed for real-time rendering of image-based global illumination to enable the learning of high-frequency environment lighting. The split sum approximation has 13 14]. These works represent pre- integrated lighting with an autoencoder-base based illumination network 3 13), with a set of learnable images for different roughness levels (20 17.. or with an MLP with integrated spherical harmonic encoding as input [1]. Autoencoder- based methods rely on learnt illumination features incompatible susceptible to noise and require re-integrating illumination whenever the base illumination is updated. Lastly, using integrated encodings to avoid integrating light leads to representation that is not physically based. In contrast, we propose modeling pre-integrated lighting as the output of an MLP paired with a novel regularization, which ensures the network correctly learns to represent physically-based pre-integrated lighting. An issue arising from the split sum approximation is that pre-integrated illumination is blind to geometry and does not account for the occlusion of light sources due geometry throughout the scene. Our work tackles this issue by supervising the prediction of ambient occlusion through Monte Carlo sampling.
been adopted by several inverse rendering methods K 20
with existing rendering pipelines. Learnable images
3 Methodology
Our method aims to extract a scene's geometry, material properties, and illumination from a set of posed images of the scene. We accomplish this by incorporating a decomposed formulation of radiance into surface rendering pipeline. In the following sections, begin with overview of the surface rendering pipeline. We then detail the physically-bar based radiance formulation, which allows us decompose radiance into illumination and material properties. Next, describe our proposed MI. representation for illumination along with the additional loss term it requires. Afterward, we derive method for estimating occlusion factor account for visibility within the split sum approximation. Finally, we describe additional regularization used to facilitate learning.
3.1 Overview of neural rendering
Neural volume rendering relies on learning two functions: o (x; f) R + R which maps a point in space> x onto a density a, and L,,xx, wo; 0) R 3 + R that maps point x viewed from direction wi onto outgoing radiance L,, The parameters 0 that define the density and radiance functions are typically optimized to represent a single scene by using multiple posed views of the scene. To learn these functions, they are evaluated at multiple points along a ray r(t) two. t [tn, t defined
3

---

Fixed
Spatial
Learnable
Network
Grad
Reflect
Illu. MLP
9
Render (Fq.8)
L.
Ilum. 0
Figure 2: Proposed architecture. A spatial network maps spatial coordinates x into geometry (o)), material properties (albedo å, metalness m, and roughness P), and occlusion factors (6). The pre-integrated illumination MLP predicts both specular 9s (Wr,, and diffuse da (f, 1) terms by using the predicted normals n, roughness, and the reflection vector w, of view direction wo- Finally, the specular and diffuse terms are combined with material properties to compute output radiance L.
by the camera origin E R2, pixel viewing direction wo, and camera near and far clipping planes tn and ly. pixel color for the ray can then obtained through volume rendering via:
C(r:0) T(t) T(t) ((r(t) (o(t),,, dt, where T(t) exp (- (-jotr(s)) ds ds)
(I)
In practice, summation of discrete samples along the ray is used to approximate the integral. This volume rendering process allows us to supervise the learning of implicit functions L, and o, in pixel-wise fashion through the reconstruction loss:
(oo(R;;) =
1
C(;))
(2)
R
Fell
where R is a batch of rays generated from a random subset of pixels from training images.
The learned geometry can be improved if, instead of directly predicting density o,, a signed distance field (SDF) is learned and then mapped to density. To this end, we follow the SDF formulation proposed in NeuS [36]. Learning valid SDF requires the use of an additional Eikonal loss term LER. For more details, please refer to (36]
Since volume density depends only depends on both position and viewing direction, neural rendering networks are typically split into spatial network and radiance network. As shown in fig. 2 maintain the spatial network to estimate density along with additional material properties but rely on a physically-based 23 radiance estimation instead of a radiance network.
point's position in space while output radiance Lo
3.2 Physically-based rendering
Given knowledge of a scene's geometry, material properties, and illumination, it is possible to model the outgoing radiance (x wi) reflected at any position of object's 's surface in direction wa by integrating over the hemisphere Q defined by the surface's normal n using the reflectance equation:
/
(k. a + [.) n)dwi,
L,
where L, is the incoming radiance, a is the object's diffuse albedo, and k, and f, are material properties dependent on the object's 's Bidirectional Reflectance Distribution Function (BRDF). For clarity, we omit from notation the dependency of incoming radiance on W; as well as the dependency of material properties on position x. Radiance L,, has diffuse and specular components L, and L. Image-based lighting methods often employ the split sum approximation to calculate specular lighting L, by splitting the integral into two components: one containing the incoming light L,, and one depending only material properties. We use the Disney || microfacet BRDF parameterized

---

by albedo, metalness, and roughness. The specular component is modeled with the Cook Torrance GGX 33 35 BRDF, leading the following approximation:
{ [Dw Wri p) L) (WW), n)dw;
fu(w;, n)dw; = glur, p) / f, (wis n)dwi,
Lg Rz
(4)
D [ Wny p) (wi, n)dw,,
52
12
!2
where ((w,, wir p) is the microfacet normal distribution function dependent on the direction of light reflection wy as well as the surface roughness The term on the right can be pre-computed as in [ since it depends only
the BRDF and not scene's ' lighting. The term the left in Equation depends on the scene's 's lighting and the microfacet distribution function D( Wr, p). The following
sections refer to this term (ww,, p) and aim estimate it with MLP representation. As shown we estimate an object's s albedo å metalness m, and roughness ò from the spatial network.
in fig. 2
Roughness 0.0
Roughness = 0.2
Roughness
0.5
Roughness = 1.0
Ground Truth
Ours
Figure 3: Pre-integrated environment illumination. We visualize the pre-integrated illumination g(w,, p) for varying roughness values along our model's prediction for the 'toaster' scene. Our pre-integrated illumination MLP accurately approximates pre-integrated lighting across roughness values thanks to our novel regularization loss based on Monte Carlo sampling.
3.3 Pre-integrated illumination MLP representation
We propose to estimate the pre-integrated lighting g(wr, p) at different roughness levels through pre-integrated illumination MLP ((w,, p). Please refer Appendix A.2 for details how ((w,, p) is used to calculate L. and L. The predictions 9 should accurately represent the environment lighting at different levels of roughness. We achieve this through loss term based on Monte Carlo estimates of the original integral for varying roughness and reflected directions using the predicted environment map L. (w) which can be extracted from 9 querying perfect specular reflections g(w, 0).
D(w Wa: (s)g(we, (w1,
(o)
1 S
Σ (lø(s) SES
g(s) with g(s)
wien
(5)
D(wi, Was Pe (Ww, Wiel
Σ
where the set S consists of paired samples of directions w, and roughness Pa- Directions taken uniformly on sphere, and half the roughness samples are taken uniformly in the range [0. 1 with the other half fixed to 1 ensure correct learning of diffuse lighting. Please refer Appendix [.. A. detailed derivation of g(s). The set Ω of light direction samples is also taken uniformly on sphere. While different sampling could lead reduced variance, utilize uniform spherical sampling for w; to be more computationally efficient. Uniform spherical sampling allows us to share light samples across the batch of predictions, thus reducing the number of evaluation calls the light function g(w, 0). We visualize both g and ğ in fig. for specific scene.
3.4 Occlusion f'actors
The split approximation does not consider the occlusion of light sources due geometry. Occlusions can be incorporated by multiplying incoming light L; by a binary visibility function Vú
L
K4 π
+ If),, Vilwe, n)dwi,
(6)
5

---

Albedo Ground Truth
Albedo Ours
Occlusion Ours
Albedo
Ocelusiio / Occlusion Loss
w/o Occlusion Loss
Occlusion loss visualization. We visualize the albedo and occlusion predicted by our method with and without the proposed occlusion regularization loss. When regularization is used, we observe that the occlusion prediction fails disentangling shadows from the albedo. Additionally, darker materials might wind up with lighter albedos due to occlusion overcompensation.
Figure 4:
with V; taking value of 1 when there are no occlusions and when incoming light is occluded by geometry. Both diffuse and specular integrals be rewritten to incorporate visibility via occlusion factors (x( and o, (x) which multiply the split sum diffuse and specular lighting terms respectively. We propose learning the occlusion factors o(x) with MLP by supervising them with Monte Carlo estimates 6xx) using the predicted geometry. Please refer to Appendix A.3 for the derivation of Monte Carlo estimates 6xx). Given the Monte Carlo estimates, supervise the predicted occlusion terms 6(x) as follows:
(0 Σ w w1(((x) - 6(x)
where the sample set X' is a random subset of the points sampled for volume rendering, and the weights the corresponding normalized volume rendering weights. Weighting the loss function by the volume rendering weights is required so that the occlusion prediction focuses only on learning surface points. The output radiance at each point in space is then calculated follows:
i. 762= L. + ô, L.),
(8)
where y maps the predicted output radiance L, from linear sRGB space.
3.5 Material regularization
To better learn material properties, we introduce soft regularizer reduce the prediction of metallic materials. This encourages the model to prefer explaining outgoing radiance through albedo and roughness whilst still allowing the prediction of metallic materials. We implement this regularization as weighted L2 loss with the same weighting as for the occlusion loss in Equation (7). That is,
(m)
Σ w (1)
X
Experiments
4.1 Baselines
We compare against Nerfactor
45
NVDiffRec (20] NVDiffRecMC 8 NeRO 14
NMF [66
and TensoTR [10]. Due to the differing evaluation methodologies among these works, we train baseline methods following publicly released code and report metries as detailed in the following.

---

Table I: NeRFactor metries. We evaluate the reconstruction quality of our method against the baselines using 20 test images and 8 low-frequency illumination maps for each scene from the NeRFactor dataset. We scale albedo and relit images with per-channel factor before computing metries. Our method attains competitive performance across all metrics with low runtime.
Normals
Albedo
Relighting
A Average
Method
MAE 4
PSNR f
SSIM t
LPIPS
PSNK t
SSIM t
LPIPS 4
Runtime
NerFactor
30.49
23.53
0,910
0.109
23.66
0.895
0.120
>20
NVDiffRec
26.47
23.05
0.901
0.123
21.88
0.880
0.111
0.98 hr.
NVDiffRecMC
25.98
23.84
0.918
0.114
24.06
0.902
0.099
2.95 hr.
NeRO
30.59
22.83
0.897
0.117
23.68
0.907
0.093
18.38 hr.
NMF
24.14
22.23
0.895
0.093
2,91 hr.
TensoIR
22.90
25.21
0,929
0.087
23.78
0.907
0.100
3.53 hr.
Ours
17.52
25.29
0.924
0.108
27.31
0.941
0.061
0.81 hr.
Table 2: Blender and Shiny Blender metries. We report the average of relighting reconstruction metrics and normal error for our extended Blender and Shiny Blender datasets. Metrics are computed as the average of 20 test views across 7 -igeufrry illumination conditions for each scene. We scale images by per-channel factor for relighting metrics. Our method outperforms the baselines across all metrics for the Blender dataset and has a higher PSNR for the Shiny Blender dataset.
Blender
Shiny Blender
Method
Normals
Relighting
Normals
Relighting
MAE 1
PSNR
SSIM t
LPIPS
MAE 1
PSNR
SSIM t
LPIPS
NVDilRRee
26.32
20.11
0.857
0.138
23.64
21.39
0.848
0.177
NVDiIfReeMC
24.74
22.50
0.884
0.136
13.75
24.60
0,911
0.151
NeRO
31.59
18.47
0.847
0.158
04.11
16.82
0.844
0.203
NMF
20.66
21.21
0.881
0.118
06.85
24.20
0,908
0.136
TensoIR
18.05
22.58
0.891
0.120
13.14
22.33
0.840
0.193
Ours
16.18
22.73
0.906
0.106
09.07
24.96
0.904
0.144
4.2
Experimental setup
445 dataset along with extended versions of the NeRF Blender [18] (Blender) and the RefNeRF Shiny Blender [34] (Shiny Blender) datasets. The NeRFactor dataset consists of four synthetic scenes, where test images are rendered under eight different low-freyuency lighting conditions. The Blender dataset consists of eight synthetic scenes representing mix of glossy, specular, and Lambertian objects. while the Shiny Blender dataset consists of six highly reflective synthetic scenes. To showcase the ability of our model to estimate high-frequency environment lighting, we extend the Blender and Shiny Blender datasets by rendering all objects under seven novel high-freee sighffreueecc lighting conditions. All models are trained using 100 posed images, and evaluated on 20 test images consisting of novel views for each lighting condition.
Datasets. We report results using the NeRFactor
Finally, we report qualitative results on real-world objects using the CO3D [24] dataset. Each object in the dataset consists of a set of images captured along a circular path along with automatically extracted foreground segmentation masks. We estimate each image's 's camera pose with Colmap 27
Relighting evaluation. We extract geometry from our model in the form of triangular mesh by using marching cubes (15). At each predicted mesh vertex, we estimate material properties in the form of an albedo, metalness, and roughness. We then render the predicted geometry using Blender's [5] physically-based shader. Material properties across faces are obtained by interpolating the predicted vertex material properties. We utilize the same Blender rendering pipeline to compute relighting metrics for baselines where explicit meshes and material properties are extracted. Otherwise, predictions are rendered using the provided relighting methodology. Before evaluating metrics, per-channel scaling factor is computed for each scene to compensate for the albedo-lightnng ambiguity. We evaluate the predicted scenes for the NeRFactor, Blender, and Shiny Blender datasets and report the average Peak Signal-to-Noii Ratio (PSNR), Structural Similarity Index Measure (SSIM) (38), and Learned Perceptual Image Patch Similarity (LPIPS) 44 in table [ and table 2 Metrics are reported as an average across 20 test images and across all illumination maps for each dataset. This metric gives an aggregated performance measure for geometry and material property estimation. Our method attains competitive relighting performance while maintaining low runtime.
7

---

Ground Truth
Pred. Render
Albedo
Metalness
Roughness
Normals
Pred. Env. Map
Figure 5: Qualitative real-world results. We present qualitative results on four scenes from the CO3D dataset. Our method can successfully recover object geometry, material properties, and illumination even for challenging scenes captured in the wild.
Albedo evaluation. In addition to overall relighting quality, we evaluate the ability of our method to recover albedo. We report reconstruction metries (PSNR, SSIM, and LPIPS) on the predicted albedo in table []] As with the relighting metrics, we apply a per-scaling factor to the albedo predictions before computing reconstruction metrics. Metrics are reported as an average across all 20 images for each scene in the NeRFactor dataset. We do not report albedo metrics for other datasets due to the lack of ground truth. We exclude results from NMF [6] since the albedo in their lighting formulation is not comparable to the other methods. Thanks to our proposed occlusion factor and material regularization, our method is on average better able to reconstruct albedo.
Normals evaluation. We measure the pixel-wise Mean Absolute Error (MAE) between ground truth and predieted normal images to evaluate geometric reconstruction quality. The MAE is weighted by ground truth alpha values to lower the effect of prediction errors at object borders. Our method recovers good estimate of geometry for most scenes evidenced in tables and 2
Real-world qualitative results. In world scenes, the far-field illumination assumption violated and objects don't follow any specific BRDF model as opposed to synthetic scenes. Both of these differences make inverse rendering from real-world data a much more challenging task than with synthetic data. Therefore, captures from the CO3D dataset. It can be observed that even in this challenging scenario our method capable of recovering accurate object geometry, well providing reasonable estimation of material properties and environment maps.
provide qualitative results in fig. 5 to validate our method on real world
5
Discussions
5.1
Ablations
Occlusion loss. We visualize the effects of the proposed occlusion loss in fig. Learning an occlusion factor without supervision leads to errors in the albedo predictions due to the inability to disentangle shadows from object color. By explicitly supervising an occlusion factor observe better albedo color predictions such as visualized in the blue box the hotdog example, and all boxes in the drums example. Additionally, shadows are better disentangled from albedo observed in the red and green boxes for the hotdog example. Quantitatively, we measure the importance of adding the occlusion loss to our model in table where it improves both relighting and albedo reconstruction.
Occlusion averaging. The occlusion factor we derive per-channel factor that depends on estimated lighting. However, since both are being learned jointly, we observe that training can be noisy. We find in table that relighting and albedo reconstruction both improve when we supervise the occlusion factors 0s and ô, with their per-channel averages instead. Assuming that all channels of the occlusion factor are equal is equivalent to assuming only white light with varying intensities, which reduces noise during training and uses fewer parameters.

---

Table 3: NeRFactor ablation results. We report reconstruction and relighting metrics for different variations of our methodology on the NeRFactor dataset. While the proposed regularizations do not have a noticeable effect on geometry, they all lead to improvements in albedo and relighting quality.
Normals
Albedo
Relighting
Method
MAE 4
PSNR f
SSIM t
LPIPS 4
PSNR
SSIM t
LPIPS 4
Ours (w/o Occ. Avg.)
17.23
24.73
0.919
0.109
27.25
0.941
0.061
Ours (w/ Occ. Loss)
17.29
22.13
0,900
0.122
26.40
0.936
0.064
Ours (w/ Met. Reg.)
17.82
23.92
0.916
0.123
26.56
0.936
0.066
Ours
17.52
25.29
0.924
0.108
27.31
0.941
0.061
Ground
NVDilitee
NVDiRRecMC
NMF
TensoIR
Helmet Materials Drums
Teapot
Figure 6: Blender and Shiny Blender illumination visualizations. We visualize the predicted illumination environment maps for our method and baselines for two scenes in the Blender dataset and two scenes in the Shiny Blender dataset. Illumination is scaled by a per-channel factor account for albedo-illumination ambiguity. Our proposed illumination inherits smoothness from the MLP representation but still captures high-quality details such trees.
Material regularization. We measure the effect of the material regularization in table 3 Penalizing metalness prediction discourages our model from explaining radiance through environment lighting with overpredicted metallic surfaces. This leads to improved albedo predictions as shown in table However, as visualized in fig, [ the loss coefficient is small enough such that our model is still capable of correctly predicting metallic surfaces.
6 Conclusion and limitations
In conclusion, we present a novel and efficient method for inverse rendering based on neural surface rendering and the split sum approximation for masee lighting. Owing to our proposed integrated illumination MLP. we can jointly estimate geometry, lighting, and material properties in under one hour using single NVIDIA A100 GPU. Physical accuracy of our pre-integrated MLP representation ensured thanks to the proposed illumination regularization. Additionally, we define occlusion factors for diffuse and specular lighting so that self-occlusions are accounted for with the split sum approximation. Finall, we propose a way of supervising occlusion MLPs to learn the proposed occlusion estimators. Altogether, our method produces high-quality estimates of geometry, lighting, and material properties as measured by rendering objects under unseen views and lighting conditions.
However, due to the highly complex problem that inverse rendering presents, our method comes with some limitations. The major assumptions we rely on come from using image-based lighting, the split sum approximation, and Monte Carlo sampling. Image-basee lighting assumes that light sources are located infinitely far away from the scene, leading to errors when this assumption violated. While we have tackled the problem of missing self-occlusions within the split sum approximation, we disregard the effect of indirect illumination. This has a noticeable impact on albedo for reflective surfaces such as the 'toaster' scene. Additionally, we only consider the reflection of light and don't t model transmission and subsurface scattering effects. Finally, we use low number of uniform Monte Carlo samples for the occlusion loss leading to errors due to strong and small light sources. This is mostly noticeable albedo predictions for objects in the synthetie Blender dataset, where shadows can still be noticed in the albedo predictions. We hope future works will tackle these limitations.

---

7 Acknowledgements
The research reported in this publication was supported by funding from King Abdullah University of Science and Technology (KAUST)- Center of Excellence for Generative AI, under award number 5940.
References
[1] Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: A multiscale representation for anti anti-aliasing neural radiance fields. ICCV (2021)
[2] Boss, M., Braun, R., Jampani, V., Barron, J.T., Liu, C., Lensch, H.P.: Nerd: Neural reflectance decomposition from image collections. In: IEEE International Conference on Computer Vision (ICCV) (2021)
[3] Boss, M., Jampani, V., Braun, R., Liu, C., Barron, J.T., Lensch, H.P.: Neural-pil: Neural pre-integrated lighting for reflectance decomposition. In: Advances in Neural Information Processing Systems (NeurIPS) (2021)
[4] Chen, A., Xu, Z., Geiger, A., Yu, J., Su, H.: Tensorf: Tensorial radiance fields. In: European Conference Computer Vision. pp. 333-350. Springer (2022)
[5] Community, B.O.: Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam (2018), http: //ww:: blender. org
[6] Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V:: CARLA: An open urban driving simulator. In: Proceedings of the 1st Annual Conference on Robot Learning. pp. 1-16 (2017)
[7] Guo, Y.C.: Instant neural surface reconstruction (2022), http://gginbbbcomm/memmaa/aaaaa nsr-pl
[8] Hasselgren, J.. Hofmann, N., Munkberg, J.: Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. arXiv:2206.033: 03388 (2022)
[9] Jiang, Y., Yin, S., Li, K., Luo, H.,, Kaynak, O.: Industrial applications of digital twins. Philo- sophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
379, 20200360 (08 2021). https://doi.or hot::/oooooo///oo 10. (098/rsta.22200553
[10] Jin, H., Liu, I., Xu, P., Zhang, X., Han, S., Bi, S., Zhou, X., Xu, Z., Su, H.: Tensoir: Tensorial inverse rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
[11] Karis, B., Games, E.: Real shading in unreal engine 4. Proc. Physically Based Shading Theory Practice 4(3). (2013)
[12] Li, Z., Müller, T., Evans, A., Taylor, R.H., Unberath, M., Liu, M.Y., Lin, C.H.: Neuralangelo: figh-fidelity neural surface reconstruction. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
[13] Liang, R., Chen, H., Li, C., Chen, F., Panneer, S., Vijaykumar, N.: Envidr: Implicit differentiable renderer with neural environment lighting. arXiv preprint arXiv:2303. 13022 (2023)
[14] Liu, Y., Wang, P., Lin, C., Long, X., Wang, J., Liu, L., Komura, T., Wang. W.: Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. In: SIGGRAPH (2023)
[15] Lorensen, W.E., Cline, H.E.:
Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics 21(4), 163-169 (1987)
[16] Mai, A., Verbin, D., Kuester, F., Fridovich-Kan Keil, S.: Neural microfacet fields for inverse rendering (2023)
[17] Mao, S., Wu, C., (Shen, Z., Zhang, L.: Neus-pir: Learning relightable neural surface using pre-integrated rendering. CoRR abs/2306.07632 (2023). hattps:/ddos 1tt.oi.org//0..88/0 48550//r55i arxii
2306.07660
[18] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng. R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications the ACM 65(1), 90-106660 (2021)
10

---

[19] Müller, T., Evans, A., Schied, C.. Keller, A.: Instant neural graphics primitives with multireso- lution hash encoding. ACM Transactions on Graphies (ToG) 41(4), 1-15 (2022)
[20] Munkberg, J., Hasselgren, J., Shen, T., Gao, J., Chen, W., Evans, A., Müller, T., Fidler, S.: Extracting Triangular 3D Models, Materials, and Lighting From Images. In: Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 8280-8290 (June 2022)
[21] Müller, M., Casser, V., Lahoud, J., Smith, N., Ghanem, B.: Sim4cv: photo-realistic simulator for computer vision applications. International Journal of Computer Vision 126(9), 902-919 (2018). https://doi.org//i 1007/s1 1263-018-1073-7
[22] Oechsle, M., Peng. S., Geiger, A.: Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In: International Conference on Computer Vision (ICCV) (2021)
[23] Pharr, M., Jakob, W., Humphreys, G.: Physically Based Rendering: From Theory to Implemen- tation. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 3rd edn. (2016)
[24] Reizenstein, J., Shapovalov, R., Henzler, P., Sbordone, L., Labatut, P., Novotny, D.: Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In: International Conference on Computer Vision (2021)
[25] Rematas, K., Liu, A., Srinivasan, P.P., Barron, J.T., Tagliasacchi, A., Funkhouser, T., Ferrari, V.: Urban radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12932-12942 (2022)
[26] Rojas, S., Zarzar, J., Pérez, J.C., Sanakoyeu, A., Thabet, A., Pumarola, A.,, Ghanem, B.: Re-rend: Real-time rendering of nerfs across devices. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 3632-3641 (October 2023)
[27] Schonberger, J.L.,, Frahm, J.M.: cenntttrre-frommeee
revisited. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4100-4113 (2016)
[28] Shah, S., Dey, D., Lovett, C., Kapoor, A.: Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In: Field and Service Robotics (2017), https: /taxxi /axivv org/ /bsss/ 1705.05065
[29] Srinivasan, P.P., Deng, B.. Zhang, X., Tancik, M., Mildenhall, B., Barron, J.T.: Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In: CVPR (2021)
[30] Sun, C., Cai, G., Li, Z., Yan, K.. Zhang, C., Marshall, C., Huang, J.B., Zhao, S., Dong, Z.: -eural-pbir reconstruction of shape, material, and illumination. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 18046-18056 (October 2023)
[31] Sun, J., Chen, X., Wang, Q., Li, Z., Averbuch-Eua Elor, H., Zhou, X., Snavely, N.: Neural 3d reconstruction in the wild. In: ACM SIGGRAPH 2022 Conference Proceedings. pp. 1-9 (2022)
[32] Tancik, M., Casser, V., Yan, X., Pradhan, S., Mildenhall, B., Srinivasan, P., Barron, J.T., Kretzsehmar, H.: Block-NeRF: Scalable large scene neural view synthesis. arXiv (2022)
[33] Torrance, K.E., Sparrow, E.M.: Theory for off- off-specular reflection from roughened surfaces=. J.
Opt. Soc. Am. 57(9), 1105-1114 (Sep 1967). /rtps://ooi.org/11.. 1ttp///0....4/0...55111 3344/J0SA..7.001000 105, https //opg.optice org/ abstract...t.................a.. wtt.rraactc.cccu.s.....
57-91105
[34] Verbin, D., Hedman, P., Mildenhall, B., Zickler, T., Barron, J.T., Srinivasan, P.P.: Ref-NeRF: Structured view-dependent appearance for neural radiance fields. CVPR (2022)
[35] Walter, B., Marschner, S.R., Li, H., Torrance, K.E.: Microfacet models for refraction through rough surfaces. In: Proceedings of the 18th Furographics Conference on Rendering Techniques.
p. 195-206. EGSR'07, 07, Furographics Association, Goslar, DEU (2007)
[36] Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106. 10689 (2021)
[37] Wang, Y., Skorokhodov, I., Wonka, P.: Hf-neus: Improved surface reconstruction using high- frequency details. Advances in Neural Information Processing Systems 35, 1966-1978 (2022)
[38] Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visi- bility to structural similarity. IEEE Transactions on Image Processing 13(4), 600-612 (2004). http:///bi.og/1/0.1109/11P100.00
11

---

[39] Wu, H., Hu, Z., Li, L., Zhang, Y., Fan, C., Yu, X: Nefii: Inverse rendering for re-
flectance decomposition with near-field indirect illumination. In: 2023 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR). pp. 4295-430 (2023). http:///oi.org/10.1111 htt.:///o///.0111//55/57799999 29012722..2020044111
[40] Yariv, L.. Gu, J., Kasten, Y., Lipman, Y:: Volume rendering of neural implicit surfaces. In: Thirty-Fifth Conference on Neural Information Processing Systems (2021)
[41] Zhang, J., Yao, Y., Li, S., Liu, J., Fang, T., McKinnon, D., Tsin, Y., Quan, L.: Neilf++:
Inter-reflectable light fields for geometry and material estimation. International Conference Computer Vision (ICCV) (2023)
[42] Zhang, K., Luan, F., Li. Z., Snavely, N.: Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In: Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. pp. 5565-5574 (2022)
[43] Zhang, K., Luan, F., Wang, Q., Bala, K., Snavely, N.: PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
[44] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effective- perceptual metric. In: 2018 IEEE/CVF Conference Com- puter Vision and Pattern Recognition (CVPR). pp. 586-595, IEEE Computer Society, Los Alamitos, CA, USA (jun 2018).
ness of deep features
http:////ooorr////// 109/CVPR 20P.R.0018.0006 https: //doi
Leeecomputersociety. org/10r..100 10. 1109/CVPE 2018.00068
[45] Zhang, X., Srinivasan, P.P., Deng, B., Debevec, P., Freeman, W.T., Barron, J.T.: Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (TOG) (2021)
[46] Zhang, Y., Sun, J., He, X., Fu, H., Jia, R., Zhou, X.: Modeling indirect illumination for inverse rendering, In: CVPR (2022)
A Appendix
A.1 Derivation of illumination loss
In this section, go through the derivation for the Monte Carlo approximation of pre-integrated illumination g used in eq. (5. We first split the specular light integral into two terms:
f L, (wi, n)
/ Eo (wir, n)dwi-
L
(10)
f f. (wi, n)dw; 0
As mentioned in section 3.2 the term the right be precomputed focus calculating approximation for the term on the left, which we denote g(wr, p).
f L.L.(Wa n)dw;
g(wr, p) =
(11)
f. (wi, n)dws
This term requires us to make two approximations to the Cook-Torrance GGX BRDF f,
DFG
f,
(12)
4(wo. n) n)
to be able to approximate g as blurred environment maps as per the split sum approximation. The first approximation on the BRDF consists of assuming the multiplication between fresnel and geometric shadowing terms approximately equal to the dot product between the normal and viewing directions: FG z (ii, n). Thus, we have that
12

---

DLe(wi n)dwi
D 4/wo: n)
fg
g(wr, p)
(13)
D (Ws, n)
as shown in Equation as:
The GGX (Trowbridge-Reitz) microfacet distribution function D defined
p² Wo n, T (h, 2p² 1) + 1)2
(14)
where h is the half vector between wi and wio The second approximation assumes the normal and viewing directions be equal the reflection direction. That n wy and w. wy. This leaves us with the following simplified D:
p²
D(wi, Wr, p) R0
(15)
I[(
(p² 1) 1)2"
which now does not depend on either the normal or viewing directions. Approximating both integrals with Monte Carlo sampling and taking the same number of samples, arrive the following expression:
Σ D(wi, wr: P) (i (w)
g(wr p) x
(16)
Σ (Wii p) (Wi Wr)
We finally obtain the expression for I in eq. (5) by replacing L; with the estimates from Ô querying perfect specular reflections g(w, 0). side effect of the approximations used is that for p = in eq. 15). we have that D(wi, Wr, 1) + and together with eq. 13 we obtain that
1
g(n 1)
Li(wi, n)dwi.
(17)
12
This allows us to reuse the same network 9 used to approximate L, also approximate L, as follows:
La = -00 1)k,ā.
(18)
A.2 BRDF details
The only remaining terms compute for obtaining diffuse and specular components are the precom- puted BRDF integral and the diffuse coefficient k, which we compute following [T]. Given our material property estimates we can compute K, follows:
F = (1 m) * 0.04 + m a,
F, = F. + (1 -- Fo) (1 (n, F.).
(19)
k, (1 m)
Finally, we store two coefficients Fj and Fz in two-dimensional lookup table as per compute the BRDF integral follows:
11 and
f fylui = F, F1 F2.
(20)
The final expressions for the diffuse and specular coefficients are thus
L g(. 1)ka L. g(ārr, p) (f,- Fi + F2).
(21)
13

---

A.3 Derivation of occlusion factor approximation
We now go over the derivation of the occlusion factor Monte Carlo approximation. We aim to approximate occlusion factors ox(x) and (x) such that the diffuse/specular components with visibility, L and L can be computed as a multiplication between the diffuse/specular components without visibility, La and Lg, and their respective occlusion factors. For the diffuse component,
L Lio,, n)dwe
Ly kd a
LiVi(un, n)dw;
ka
/
Li(wris n)dwi.
(22)
L Lelwr, n)dw;
52
52
L Vilwy, ni/wui
Thus, Ly o,(xxLa with 0a(x)
82
(23)
L Li(ws, n)dwi
52
To incorporate this information, we learn this factor for the diffuse radiance. We propose learning the occlusion factor (x(x) with an MLP, supervising it with Monte Carlo estimates ox(x) using the predicted geometry. We approximate (x(x) with Monte Carlo sampling by taking the same number of wi samples for both integrals:
Σ L.V.
ox(x)=
WiEn
(24)
Σ L
WEER
with W; taken from a cos-weighted sampling of the hemisphere around the normal n at location x. The probability density function sampled is given by:
(w), n)
pdf(w,; n)
(25)
π
This cos-weighted sampling aids in reducing variance by eliminating the dot product factor from the estimation. Similarly, for the specular component have that
f,L, Vilue, d)
LV
T,L;Ve(ws, n)dw;
fgLe nidwss
(26)
f, L, (wo, n n)dw
Q
12
f. Vi (ww, n) dur
Thus, L = (,)L,, with (x)
(27)
f L. (Wi, n)dls
13 leading the following Monte Carlo estimate for We then use the approximation for f, in eq. (x)::
Σ
L, L,Vi(ur, n)
(x(x)
WEe
(28)
Σ
L, n)
W.ER
where is now obtained by sampling the GGX distribution to reduce variance by eliminating the factor from both integrals. The probability density function sampled in this case is the following:
D(w,, Wrs (n wn)
pdf((w;; wr p)
(29)
4 4(n,
which relies on the second approximation used in the previous section, and where wi the half-vector angle between Wi and Wo.
14

---

A.4 Implementation details
We embed our proposed lighting decomposition within an efficient implementation of NeuS M. We train our models for 20, 000 steps using a warmup learning rate scheduler for the first 500 steps followed by an exponential decay scheduler. After every 2000 steps, we estimate the current geometry by using marching cubes [15] to extract the isosurface at SDF level-set O. The estimated geometry used with 64 samples for the Monte Carlo estimation of occlusion factors. We use random subset of 10% of the points from volume rendering to supervise the occlusion network to reduce time and memory requirements. 8129 light samples are used for computing illumination loss Monte linear combination of the proposed losses, with the following coefficients: Are = 10.0, Ap 10.0, A = 0.01, Apik = 0.1, and Am = 0.001. We run all experiments single A100 GPU with 60GB RAM and 6 CPU workers using AMD EPYC 7713 64-Core processor for a total training time of ~ hour.
Carlo estimates. The final loss is calculated
Network implementation details. We implement the spatial network using the progressive hash grid encoding from [1]. The hash grid consists of 16 levels with features per level and hashmap size of 2 entries. The base grid spatial resolution is 32 voxels, increasing by ~~1.2 each level. An MLP with a single 64-channel hidden layer is used produce spatial features with 13 channels along with the SDF predictions. Spatial features are then input to an MLP with two hidden layers of 256 channels each and ReIUU activations to produce material property (metalness, roughness, and albedo) predictions. A separate but identical MLP is used to produce occlusion factor predictions. A sigmoid is used to map the MLP outputs to the occlusion factor and material properties' ranges of [o, 1]. The illumination network consists of an MLP with five hidden layers with 256 channels each and ReLU activations. Both the direction and roughness vectors used as input to the illumination network are first positionally encoded proposed in [18), using 10 frequencies for the directional input and 5 for the roughness input. A softplus function is used to map the illumination network's 's output to the range (0, inf).
A.5 Per-scene quantitative results
We report per-scene metries for geometry, albedo, and relighting reconstruction using the NeRFactor dataset in tables 4 to
6 We report per-scene metrics for geometry and relighting using the Blender
dataset in tables 7 to 10, and using the Shiny Blender dataset in tables IT to 14 Additionally, we present qualitative results of our method visualizing the learnt illumination, material properties (metalness, roughness, and albedo), geometry, and relit renderings from our method's predictions for the Blender dataset in figs. to and for the Shiny Blender dataset in figs. 15 19
A.6 CO3D qualitative relighting results
We present qualitative results of our method visualizing the learnt illumination, material properties (metalness, roughness, and albedo), geometry, and relit renderings from our method's predictions for the CO3D dataset in figs. 20 to 23 Even in this challenging real-world setting where images are taken from a user-captured video, our method provides good-quality results.
A.7 Additional qualitative videas
Please refer to the included video files for additional qualitative videos showing predicted renders, material properties, and relighting for the Blender, Shiny Blender, and CO3D datasets. All videos follow the same camera trajectory at fixed distance from the center of the scene. Please note that due objects in CO3D lacking training images viewing the upper lower surfaces of objects, the reconstruction quality at these locations suffers. This issue could be alleviated with better video-capturing trajectories.
15

---

MAE +
Method
avg.
drums
ficus
hotdog
lego
NerFaactor
30,49
30.27
45.37
16.95
29.37
NVDifTRee
26.47
26.37
29.39
13.28
36.86
NVDiffRecMC
25.98
28.81
30.88
13.12
31.11
NeRO
30.59
22.00
50.69
19.78
29.90
NMF
24.14
20.62
39.00
10.85
26.10
TensoIR
22.90
18.30
36.38
14.21
22.74
Ours
17.52
18.33
17.19
10.41
24.13
Table 4: NeRFactor per-scene MAE.
PSNR
SSIM
LPIPS
avg.
botdlog
lego
ayg.
Brus
hotdog
lego
avg.
hotdog
lepo
NerFactor
23.06
20.01
23.71
26.15
21.77
0.895
0.8.7
0.932
0.914
0.854
0.120
0.1.0
0.090
0.115
0.14I
NVDiMRee
21.88
20.72
20.09
24.64
22.09
0.880
0.890
0.900
0.892
0.831
0.111
0.097
0.085
0.124
0.137
NVDNITRecMC.
29,05
74.24
0.902
0.910
0.099
0.079
NMF
22.23
21.54
21.36
22.47
23.58
0.895
0.906
0.934
0.876
0.863
0.093
0.075
0.063
0.120
0.116
TemolR
23.78
22.49
23.07
25.58
23.97
0.907
0.915
0.933
0.895
0.886
0.100
0.077
0.081
0.129
0.113
Ours
27.31
24.72
27.45
29.04
28.02
0.941
0.935
0.964
0.947
0.929
0.061
0,058
0.041
0,069
0,075
Table 5: NeRFactor per-scene relighting metrics.
Method
PSNR
SSIM
LPIPS
drums
Biens
boldog
lego
drrms
Biess
hotdog
lego
drums
Bieus
botdog
lego
NerFactor
23.33
20.75
27.75
23.55
NYDiIRee
23.05
19:47
23.67
28.20
20.87
0.901
0,938
0.844
0.123
0.090
0.110
NVIRITReMMC
23.84
20.28
22.16
29.27
23.65
0.918
0.892
0.923
0.950
0.905
0.114
0.118
0.105
0.100
0.134
NeRO
2283
20.52
20.05
26.70
24.03
0.897
0.889
0.910
0.923
0.868
0.117
0.116
0.106
0.100
0.147
NMP
Ours
25.29
23.73
29.41
24.68
23.33
0.921
0.923
0.974
0.929
0.870
0.108
0.099
0.051
0.121
0,160
Table 6: NeRFactor per-scene albedo metrics.
MAE
Method
avg.
chair
drums
ficus
hotdog
lego
materials
mic
ship
NVDiffRec
26.52
20.71
28.70
28.96
15,46
40.53
22.37
21.39
34.04
NVDiffRecMC
24.74
16.47
28.69
38.18
16.36
34.84
12.04
21.04
30.27
NeRO
31.59
13.47
29.45
51.33
13.13
29.20
71.22
19.00
25.95
NMF
20.66
13.18
20.24
36.61
13.97
26.38
08.69
20.68
25.56
TensoIR
18.05
11.69
17.88
30.63
15.16
19.18
13.51
17.31
19.02
Ours
16.18
11.65
20.96
17.38
13.20
21.17
08.93
14.37
21.76
Table 7: Blender per-scene MAE.
PSNR
Method
avg.
chair
drums
fieus
hotdog
lego
materials
mic
ship
NVDifReee
20.11
21.70
19.41
20.32
21.17
21.84
21.26
18.48
16.69
NVDiffRecMC
22.50
24.64
20.57
21.27
26.37
24.77
24.57
18.95
18.91
NeRO
18.47
22.73
13.51
21.12
19.99
21.81
12.73
17.82
18.04
NMF
21.21
22.53
21.38
22.62
20.52
22.05
24.87
18.34
17.41
TensoIR
22.58
25.21
22.10
23.90
22.01
26.18
23.00
18.79
19.45
Ours
22.73
25.00
22.62
26.40
20.94
23.88
25.38
18.75
18.88
Table 8: Blender per-scene PSNR.
16

---

SSIM
Method
avg.
chair
drums
ficus
hotdog
lego
materials
mic
ship
NVDiffRee
0.857
0.886
0.852
0.902
0.894
0.834
0.866
0.927
0.695
NVDiffRecMC
0.884
0.918
0.877
0.902
0.930
0.865
0.904
0,924
0.750
NeRO
0.847
0.905
0.774
0.912
0,899
0.856
0.755
0,920
0.754
NMF
0.881
0.908
0.890
0.934
0.890
0.863
0.913
0,926
0.722
TensoIR
0.891
0.929
0.899
0.935
0.891
0.906
0.871
0,934
0.763
Ours
0.906
0.937
0.908
0.952
0.914
0.901
0.930
0.937
0.769
Table 9: Blender per-scene SSIM.
LPIPS
Method
avg.
chair
drums
ficus
hotdog
lego
materials
mic
ship
NVDiffRec
0.138
0.099
0.134
0.083
0.141
0.141
0.132
0.092
0.283
NVDiffRecMC
0.136
0.084
0.135
0.093
0.118
0.147
0.112
0.097
0.306
NeRO
0.158
0.099
0.229
0.088
0.115
0.132
0.218
0.095
0.291
NMF
0.118
0.093
0.103
0.066
0.135
0.108
0.081
0.087
0.271
TensoIR
0.120
0.082
0.102
0.076
0.148
0.087
0.130
0,088
0.251
Ours
0.106
0.065
0.096
0.050
0.116
0.097
0.075
0.077
0.269
Table 10: Blender per-scene LPIPS.
MAE +
Method avg.
car
coffee
helmet
teapot
toaster
NVDiffRec
23.64
40.01
21.46
18.14
07.73
30.86
NVDiffRecMC
13.75
10.83
23.34
12.73
08.77
13.08
NeRO
04.11
05.72
04.72
01.29
03.27
05.54
NMF
06.85
07.38
12.22
02.42
04.67
07.58
TensoIR
13.14
11.44
09.56
18.00
10.97
15.72
Ours
09.07
07.21
23.44
02.56
02.29
09.86
Table 11: Shiny Blender per-scene MAE.
PSNR
Method avg.
coffee
helmet
teapot
toaster
NVDitfRee
21.39
24.85
18.29
19.07
29.39
15.36
NVDifRReMC
24.60
24.25
22.93
22.86
32.70
20.25
NeRO
16.82
15.36
17.21
14.09
25.31
12.12
NMF
24.20
24.58
17.88
27.48
30.66
20.41
TensoIR
22.33
25.31
18.42
19.45
31.35
17.14
Ours
24.96
26.86
18.70
21.51
38.13
19.62
Table 12: Shiny Blender per-scene PSNR.
SSIM 1
Method
avg.
coffee
helmet
teapot
toaster
NVDiffRec
0.848
0.913
0.799
0.848
0.972
0.705
NVDiffRecMC
0.911
0.917
0.921
0.908
0.983
0.827
NeRO
0.844
0.837
0.867
0.835
0.964
0.717
NMF
0.908
0.917
0.843
0.946
0.982
0.849
TensoIR
0.840
0.899
0.857
0.784
0.970
0.692
Ours
0.904
0.942
0.883
0.877
0.993
0.827
Table 13: Shiny Blender per-scene SSIM.
17

---

LPIPS +
Method
avg.
car
coffee
helmet
teapot
toaster
NVDiffRec
0.177
0.102
0.237
0.209
0.046
0.290
NVDiffRecMC
0.151
0.101
0.195
0.182
0.039
0.241
NeRO
0.203
0.152
0.248
0.259
0.049
0.305
NMF
0.136
0.092
0.208
0.142
0.032
0.206
TensoIR
0.193
0.125
0.203
0.277
0.048
0.313
Ours
0.144
0.072
0.204
0.195
0.017
0.234
Table 14: Shiny Blender per-scene LPIPS.
GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrisc
Sunset
Figure 7: Qualitative results on Blender 'chair' scene.
18

---

GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Sunset
Figure 8: Qualitative results on the Blender 'drums' scene.
GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Sunset
Figure 9; Qualitative results on the Blender 'ficus' scene.
19

---

GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Sunset
Figure 10: Qualitative results on the Blender 'hotdog' scene.
GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Sunset
Figure 11: Qualitative results on the Blender 'lego' scene.
20

---

GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Sunset
Figure 12: Qualitative results on the Blender 'materials' scene.
GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Sunset
Figure 13: Qualitative results on the Blender 'mic' scene.
21

---

GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Sunset
Figure 14: Qualitative results on the Blender "ship' scene.
GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Sunset
Figure 15: Qualitative results on the Shiny Blender 'car' scene.
22

---

GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Sunset
Figure 16: Qualitative results on the Shiny Blender 'coffee' scene.
GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Sunset
Figure 17: Qualitative results on the Shiny Blender 'helmet' scene.
23

---

GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Sunset
Figure 18: Qualitative results on the Shiny Blender 'teapot' scene.
GT Envmap
Pred Envmap
Metalness
Roughness
GT Render
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Sunset
Figure 19: Qualitative results the Shiny Blender 'toaster' scene.
24

---

Pred Envmap
Metalness
Roughness
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Figure 20: Qualitative results on the CO3D car scene $421 1112551_
Pred Envmap
Metalness
Roughness
Pred Render
Albedo
Normals
Courtyard
Tall
Interior
Sunrise
Figure 21: Qualitative results on the CO3D car scene '112 13250 22955'.
25

---

Pred Envmap
Metalness
Roughness
Pred Render
Albedo
Normals
Courtyard
O F
Interior
Sunrise
Figure Qualitative results on the CO3D ball scene $373_4166 41665 001444166116666666 83166'.
Pred Envmap
Metalness
Roughness
Pred Render
Albedo
Normals
Courtyard
Interior
Sunrise
Figure 23: Qualitative results on the CO3D cup scene '34 1428 4472'.
26

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
Claims made in the abstract and introduction are backed by experimental
results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide discussion of the limitations of our work in section 0
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
We provide the derivation along as assumptions made for our proposed
illumination regularization and occlusion factors appendices A. and A.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Additionally, we provide the code as supplementary material and will make the code public upon acceptance. All datasets used are publicly available.
Justification: We provide the implementation details in appendix A.4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
28

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code is provided as zip file along with the supplementary material. All datasets used are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//ipsscccppbllii wwtppbblcggguiiieecceeeeeeooeeee http://ddese/eee/e SooodeSuumisssionPP..
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the implementation details in appendix A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not provide error bars due the runtime cost. While training for scene is relatively fast for our method A100 GPU hour), the total runtime including training and evaluation for all baselines in our main experiment becomes prohibitively expensive for computing error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the average compute time and computational resources used for each experiment. [TODO]
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htttap/e//orrrpss.ccpuu////teeie
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential positive socictal impacts in section I
Guidelines:
The answer NA means that there is no societal impact of the work performed.
30

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We properly credit the owners of data used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "z9YwZI5KS2", "reviewer": "Reviewer_ApFY", "summary": "The paper introduces SplitNeRF, a method that integrates the split sum approximation into Neural Radiance Field (NeRF) pipelines. This approach optimizes object geometry, material properties, and environmental lighting. The method employs a Multi-Layer Perceptron (MLP) to model scene-specific pre-integrated image-based lighting at arbitrary resolutions. The model is regularized using efficient Monte Carlo sampling to accurately capture pre-integrated lighting and self-occlusion effects. Experimental results indicate that SplitNeRF achieves state-of-the-art relighting quality with only about an hour of training on a single NVIDIA A100 GPU.", "questions": "See weaknesses 1, 2.", "limitations": "Limitations discussed in the conclusions section.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The method is efficient and achieves high-quality relighting results after only about an hour of training on a single NVIDIA A100 GPU, which is highly efficient compared to existing methods.\n2. By using Monte Carlo sampling for regularization, the method accurately models pre-integrated lighting and self-occlusion, leading to high-fidelity reconstruction of scene geometry and material properties. The integration of split sum approximation into NeRF pipelines is novel and effective, allowing the method to disentangle environmental lighting from material properties.\n3. The method demonstrates competitive performance on both synthetic and real datasets, showing its state-of-the-art performance in material decomposition and relighting.", "weaknesses": "I do not have a major concern about this paper- the technical claims are sound and the paper is overall well presented. Several minor points:\n\n1. Adding material editing visualizations in the paper could further strengthen the results and inspire downstream applications.\n2. Normal could contain a significant error in e.g. lego and coffee scenes. Can you provide more insight and analysis into this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Estimating geometry together with materials and illumination is a very complex and unconstrained problem. Because of this, the optimization can sometimes get stuck in local minima. We believe that is happening for those scenes. For example, it is possible to model reflections via small variations in geometry rather than higher frequency changes in the illumination. We hypothesize that is happening in the 'coffee' scene.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f7YJ4LNZ9M", "reviewer": "Reviewer_h9Hn", "summary": "This paper introduces a method for digitizing real-world objects by estimating their geometry, material properties, and environmental lighting from posed images under fixed lighting. Integrating image-based lighting techniques into Neural Radiance Field (NeRF) pipelines, the method uses a scene-specific MLP for pre-integrated lighting at arbitrary resolutions. A Monte Carlo sampling-based regularizer ensures accurate lighting representation and self-occlusion predictions.", "questions": "In the discussion of occlusion loss, why is only the albedo prediction addressed? The paper claims that the occlusion is designed to improve the overall material estimation.", "limitations": "The proposed method decomposes lighting and material using surface reflection. However, real-world objects often have multiple material layers, such as the car showcased in the qualitative results. The clear coat reflects stronger specular light, while the lower paint layer includes both specular and diffuse reflections. The proposed model does not account for these complexities.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The method proposes a method to approximate the effect of self-occlusion to improve the material estimation", "weaknesses": "- The proposed method, particularly in terms of material and lighting representation, is trivial.\n- The correctness of the decomposed material is flawed in the qualitative results. The albedo clearly has specular and shadow bake-in, and the metalness and roughness do not match the material in the ground truth.\n- The occlusion introduced in the paper to improve material estimation is not robust enough to validate its contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We only adress the albedo prediction since albedo is the only material property that is shared across most commonly used BRDF models. The synthetic datasets we rely on come from 3D models which were hand-designed using a variety of complex BRDF models with properties which can't be directly translated into out model's \"metalness\" and \"roughness\". Due to this, there doesn't exist a ground truth \"metalness\" or \"roughness\" to evaluate against. Qualitatively, it is easier to visualize the effects of the occlusion loss on albedo due to its colored nature, which makes it more intuitive to evaluate.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dm1ZBsuEZm", "reviewer": "Reviewer_UpzA", "summary": "This paper tackles the problem of inverse rendering, which reconstructs geometry, material, and environmental lighting from a set of posed images with fixed lighting. It proposes two contributions: 1) representing pre-integrated illumination as a single MLP, 2) approximating self-occlusion on pre-integrated lighting and use it to supervise an occlusion MLP to disentangle shadows and materials.\n\nThe proposed method achieves state-of-the-art results while being very fast to train (less than one hour).", "questions": "1. The paper mentions that the relighting results were obtained via Blender's PBR shader. Does the shader use global illumination? If so it would be unfair for some of the baselines (TensorIR, NMF) as they can only render direct illumination. A more fair way would be run another experiment where all methods do not use global illumination.", "limitations": "The limitations and potential negative societal impact are discussed properly.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of representing pre-integrated illumination as MLP is interesting. The regularization is particularly novel and clever. I really like the way how pure specular MLP (roughness=0) is used to regularize the training of the illumination MLP (Eq. 5)\n\n2. The proposed occlusion factor estimation method is also very interesting and novel. Instead of ambient occlusion, the paper uses simple trick to factor out occlusion as an independent scalar for light integral. This scalar can be computed with MC integration and is distilled into a neural network for efficient inference.\n\n3. The quantitative result shows that the improvement over SOTA is significant, especially on relatively diffuse scene (NeRFactor dataset).", "weaknesses": "1. More datasets could be tested, e.g. the TensorIR dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The shader uses global (indirect) illumination. We decided to evaluate using blender's PBR shader, as done in previous works, since our aim is to generate relightable meshes for use in existing rendering pipelines. Evaluating the rendering quality from one such pipeline is the most direct way of benchmarking that goal. This is especially important in the context of radiance fields since the volumetric formulation supervised mostly through rendering allows for artifacts in 3D to be hidden in renders. For example, material properties could be distributed along rays near the surface of objects, leading to good quality renders but bad quality 3D representations. Both TensoIR and NMF take into account indirect illumination, although they restrict its calculation to two ray bounces.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received three mixed reviews -- one 4 (borderline reject), one 6 (weak accept), and one 7 (accept). 

While there was general appreciation for the proposed idea of representing scene's illumination map using an MLP (and the associated regularization and optimization techniques used for reconstruction), and the quality of the results, including improvements over the current state-of-the-art methods. 

While there were some concerns raised about the effectiveness of the proposed method in separating the albedo components correctly, which the authors attempted to address in the rebuttal. 

The opinion remained split post-rebuttal, but given that the paper tackles an important problem of reconstructing the scene's geometry, material properties and environmental lighting and demonstrates reasonable results with novel insights and mathematical techniques, on balance, an accept decision was reached.
Decision: Accept (poster)