Multi-times Monte Carlo Rendering for Inter-reflection Reconstruction
Authors: Zhu Tengjie
Zhuo Chen
Jingnan Gao
Yichao Yan
Xiaokang Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Inverse rendering
3D reconstruction
Indirect illumination
Physically based rendering
Monte Carlo Rendering
Ray tracing
Primary area: machine_vision

Abstract: Inverse rendering methods have achieved remarkable performance in reconstructing high-fidelity 3D objects with disentangled geometries, materials, and environmental light. However, they still face huge challenges in reflective surface reconstruction. Although recent methods model the light trace to learn specularity, the ignorance of indirect illumination makes it hard to handle inter-reflections among multiple smooth objects. In this work, we propose Ref-MC2 that introduces the multi-time Monte Carlo sampling which comprehensively computes the environmental illumination and meanwhile considers the reflective light from object surfaces. To address the computation challenge as the times of Monte Carlo sampling grow, we propose a specularity-adaptive sampling strategy, significantly reducing the computational complexity. Besides the computational resource, higher geometry accuracy is also required because geometric errors accumulate multiple times. Therefore, we further introduce a reflection-aware surface model to initialize the geometry and refine it during inverse rendering. We construct a challenging dataset containing scenes with multiple objects and inter-reflections. Experiments show that our method outperforms other inverse rendering methods on various object groups. We also show downstream applications, e.g., relighting and material editing, to illustrate the disentanglement ability of our method.

Multi-times Monte Carlo Rendering for Inter-reflection Reconstruction
Tengjie Zhu
Zhuo Chen
Jingnan Gao
Yichao Yan
Xiaokang Yang
MoE Key Lab Artificial Intelligence, Al Institute, Shanghai Jiao Tong University {zhutengjie, ningei5252, gjn0310, yanyichao, ass tu. edu cn
Abstract
Inverse rendering methods have achieved remarkable performance in reconstructing high-fidelity 3D objects with disentangled geometries, materials, and environmental light. However, they still face huge challenges in reflective surface reconstruction. Although recent methods model the light trace to learn specularity, the ignorance of indirect illumination makes it hard to handle inter-reflections among multiple smooth objects. In this work, we propose Ref-MC² that introduces the multi- times Monte Carlo sampling which comprehensively computes the environmental illumination and meanwhile considers the reflective light from object surfaces. To address the computation challenge as the times of Monte Carlo sampling grow, we propose a specularity-adaptin sampling strategy, significantly reducing the computational complexity. Besides the computational resource, higher geometry accuracy is also required because geometrie errors accumulate multiple times. Therefore, further introduce reflection-awarr aware surface model initialize the geometry and refine it during inverse rendering. We construct a challenging dataset containing scenes with multiple objects and inter-reflections. Experiments show that our method outperforms other inverse rendering methods on various object groups. We also show downstream applications, e-g., relighting and material editing, to illustrate the disentanglement ability of our method. Our project page: https: // pettps:/zzuuteee htt////htttnnjiti/ ie. github. 1o/Ref-MCC
Introduction
Neural Radiance Fields (NeRF) 26 and 3DGS [16] have demonstrated their excellent performance on novel view synthesis. However, it is difficult to directly apply their reconstructed 3D model to the current industrial pipeline, leading the lack of flexibility in many downstream applications, e-g.. relighting and material editing. To better cooperate with mature techniques, inverse rendering bases physical rendering (24 and utilizes the neural network learn disentangled materials that can be seamlessly plunged into the industrial pipeline for further manipulation.
Previous methods [28 45 57] have explored how disentangle geometry, diffuse, roughness, metalness, and environmental light from multi-view images, but they still face challenges in shading and reflective objects. Specifically, Nydiffrec 22 proposes a differentiable pipeline that enables gradient-based optimization on both meshes and volumetric textures. These methods ignore the shadow when modeling the illumination, resulting in failed disentanglement for diffuse and shading appearances. To model more realistic shadings, Nydiffrecme [9] further incorporates ray tracing and Monte Carlo sampling (27 into inverse rendering, significantly improving the decomposition of shape, materials, and lighting. However, it ignores the indirect illuminations during path tracing
"Equal contribution.
t "oorrsponndigg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

and treats rays attacking object surfaces as sheltered illumination. As a consequence, methods like Nydiffrecme increase the ambiguity of geometry reconstruction and undermine material learning. Unfortunately, the scene with multiple inter-reflections is common in the real world. The failure in these scenes hinders the wider applications of these methods. Recent methods [ 40 22 take inter-reflections into account and can reconstruct the reflective objects well. However, the implicit representations of these methods for materials and renders sacrifice their scalability to downstream tasks. Recently, Nefii [45] further incorporated the implicit neural radiance to estimate ray tracing, alleviating the ambiguity between materials and indirect illuminations. However, this leads to a huge computational consumption for path tracing. Neural Microfacet Fields [2] employs two-bounce sampling for indirect light modeling, but its geometry field limits flexibility for downstream tasks.
In this paper, we proposed full inverse rendering method, Ref-MC2, which considers inter-reflections during ray tracing to improve the decomposition of explicit materials and environmental lighting. The core of our method is to use Multi-times Monte Carlo integration [27] and BRDF [2 rendering to approximate the indirect illumination at multiple reflection points along the light propagation path. Although our method takes advantage of bardware-accelerateed ray tracing to model the indirect illumination, we still face two major challenges brought by the multi-times Monte Carlo sampling. I) Efficiency: the explosive computational growth from multiple times is too heavy for the hardware algorithm only. 2) Geometry: the geometry quality greatly affects the calculation of indirect light and the decomposition of the materials, because the error will accumulate over and over again the times of Monte Carlo sampling increase.
To address these challenges, we correspondingly propose two strategies. 1) For efficiency, we flatten the multi-times sampling into sequential single-time sampling. In a Lambert model [1., the diffuse light is independent of the direction of reflection and thus can be presented as map to query at any time. When we trace the indirect light from an object, instead of recalculating the diffuse component, we can take the value directly from the diffuse map. It can be optimized through self-supervision. Therefore, we only need to sample the specular component within a small lobe along around the reflective direction. 2) For geometry, we refer to the SDF-based methods [4 6 and replace the common positional encoding with the Sphere Gaussian encoding to get an accurate initial geometry for reflective objects. We use this geometry to initialize Flxicubes (35) that optimizes surface meshes based on the gradient, and further fine-tunes the Flexicubes in a differentiable pipeline. To evaluate framework, we construct dataset containing difficult scenes which contain strong inter-reflections between multiple surfaces. Extensive experiments demonstrate that our framework successfully decompose indirect illumination and materials. In summary, our contributions are:
We propose a full inverse rendering method that employs multi-times Monte Carlo sampling to correctly decompose indirect illumination and materials.
We reduce the computational consumption when tracing indirect illumination by self- supervising the diffuse map based on the Lambert model.
We refine the SDF-based architecture with Spherical Gaussian encoding to obtain a high-quality initial geometry which further releases the accumulated error during multi-times sampling.
We construct a dataset to evaluate the performance on indirect illumination.
2
Related Work
2.1 Implicit Neural Representations
Neural implicit representations 23 26
4 58 36 7 have achieved impressive
success in many computer vision and computer graphics tasks. These methods use neural radiance fields to capture color and volume density, generating photo-realistic novel views through volume rendering [15]. However, the unconstrained volumetric representation of the original NeRF method
leads to low-quality geometry. Recent 3D Gaussian Splatting (3DGS) [66] 12
20 53 has
gained popularity in novel view synthesis. Different from NeRF, is an explicit representation that involves the optimization of multiple Gaussians to reconstruct 3D objects. 3DGS learns color and density in a volumetric point cloud, but it also fails to produce accurate geometry due to its discrete representation. Besides, these methods are all entangled learning, integrating all inherent materials and the environment map into the appearance. This limits the downstream applications in the current
industrial pipeline. To produce high-quality geometry, follow-up works
[48 43
51 30 use

---

function to associate signed distance field (SDF) and volume density. The surface mesh can be extracted from neural implicit surfaces by Marching Cubes [9]. and this 3D asset can be further applied in other applications. However, they usually perform badly in reconstructing the reflective objects due to the ambiguity of reflective appearance.
2.2 Neural Inverse Rendering
Although neural implicit surfaces have achieved impressive performance in geometry reconstructing and novel views synthesis, they do not obtain the fundamental materials of PBR which limits their
flexibility in downstream tasks. Neural inverse rendering methods [17 46 8 54
57
38
21 introduce the physical rendering equation to estimate the disentangled diffuse and specular
component from RGB images. They approximate the rendering equation based on neural networks
or basis functions, e-g-, Spherical Gaussians 42
47
49
[54) and Spherical Harmonies [5] 1 37
52 Nvdiffree [28) introduces the full inverse rendering that estimates shape, materials, and environmental light into gradient-basee optimization. However, it does not consider the shadows, leading to the entanglement of materials. Recent works [ 4 extend SDF-based architectures with an additional appearance branch to model the reflections on the object. RefNeuS [6] introduces a reparametrizatioon method to distinguish the reflective appearance from the diffuse appearance by direction-relatiee process, but it fails faithfully reconstruct non-reflective objects. UniSDF [4] proposes to use a weight-MLP to balance the reflective and non-reflective branches for different objects. However, these methods still face challenges in scenes with complex inter-reflections. Nydiffrecme [9] extends Nydiffrec with Monte Carlo sampling to trace the light path but still ignores indirect illumination between objects. Further methods [7] 45) consider indirect illumination in their design. ENVIDR [17 employs neural renderer to learn the physical light interaction, without explicitly formulating the rendering equation. NeRO [8] applies the split-sum approximation to approximate the shading effects of both direct and indirect lights. Nefii 45 introduces ray tracing to the radiance field to model indirect illuminations. Neural Microfacet Fields [22] employs two-bounce sampling to accurately calculate indirect illumination. However, NMF represents geometry by density field, which limits the scalability and flexibility for downstream tasks. In addition, these aforementioned methods do not fully disentangle the materials from RGB images, but decompose the appearance into reflective and diffuse color. It is hard to apply to the current industrial pipeline directly.eq: rendering equation In contrast, our Ref-MC2 is full inverse rendering method that also considers indirect illumination.
Method
3.1 Preliminaries
The rendering equation 14 is commonly used to compute the outgoing radiance Lo(p, w.) from the point p in outgoing direction wp:
Lo (p. w.) I.. (p, I (p, wa; w.)
) (n wi) dwy,
(1)
where L,(p, w;) is the incoming radiance into p from the direction Wi, is the normal of the point p. Ω is the hemisphere of directions above p. (p, wi, w.) is the BSDF evaluated for w; and the current incoming direction w; The GGX (24 physics-based BSDF function is proposed to decompose the function into several physical terms. The function described as:
DFG
(p. wi w.) fa fo = fa fa+ 4 (n - w:) (n-w))'
(2)
where fa is the diffuse term and fa is the specular term. D, F, and G are the microfacet distribution function, the Fresnel reflection coefficient, and the geometric attenuation, respectively.
Previous works 28 use the split sum function 25 to approximate the rendering equation, but it inevitably omits the shadow and indirect illumination. In contrast, Monte Carlo integration 27] is a simple but unbiased estimation method that comprehensively considers all the physical terms for the outgoing radiance. The Monte Carlo integration rendering equation
N
Li (p. wi) f (p. wi, wo) (w: n)
Lo wo) R
(w:)
3

---

Path traccing
Diffuse
light
Dir nct Iil Hination
Diffuse map
+
Indirect illumination
gradients
Speculer
Environment light
light
Intteed
Partial MC sampling
Selective sempling
minatinaion
ka
Rorı
gradients
Rendering results
Truth
normals
Figure I: We perform Monte Carlo sampling at the viewpoint. When the sampling ray from the point is not blocked, it is the direct illumination from environmental lighting. When the sampling ray hits an object, we divide this indirect illumination from the object into diffuse light and specular light. We sample the diffuse light from diffuse map that is optimized through self supervision. For specular light, we only need partially trace the rays small specular lobe along the reflective direction. The gradients are backward along the tracing path, and are passed to optimize ka, korm: normals, and environment maps.
where w; is the ith sample drawn from density As the number of samples grows, the estimation variance reduces, but the computation increases. Multiple importance sampling [39] (MIS) is proposed to inhibit the computational consumption. When function can be expressed as multiplication of functions, it draws n, samples wi, from n sampling distributions p; turn. The MIS Monte Carlo estimator for the rendering equation is:
F; (P. (P. Wi.s)
Lo (p. wo) = Σ Σ
W
(4)
Pi
where pt the i-th multiplication function of the integrated function of the rendering equation and W; is the the balance heuristic weighting function.
3.2 Multi-times Monte Carlo Sampling
As discussed in Sec. 3.1 the split sum approximation ignores the shadows and the indirect illumina- tion. Nydiffrecmc notes this and accounts for shadows using the Monte Carlo integration method. However, they give up continuously tracing the light rays attacking the surface of objects and them as zero illumination. It leads to bad performance in scenes with ieter-reflections. Therefore, we propose the Ref-MC2 method trace the light rays continuously. When taking into account indirect illumination, based on [14), the rendering equation can be expressed in this version:
La wn) = / Li (r (p, wi), - -w:) I (p, wi
(p, Wp) ) (n wi) dw..
(5)
In this equation, r (p, wi) represents the location of a surface point on the object surface hit by ray cast from p in direction w, for the first time. The corresponding Monte Carlo integration that considers the indirect illumination can then expressed as:
1
L. (r w:) of (p, wi, wa) (was n)
Lo (p)
(6)
p p(w))
can interpreted as that when the sampling ray is not blocked, can be seen as direct illumination from environmental lighting. When the surface point of an object blocks the sampling ray, it needs to be continuously traced from this surface point. This continuous tracing is an iterated process of Monte Carlo integration. It is noteworthy that when continuous sampling the blocking surface point is

---

Difluse MLP
Cd
9G
Cd
SDF value
Wp
SDF velue
(a)
Reflection
Rellection
b
Reflective
Reflective
MLP
SGE
MLP
(3) The architecture of previous SDF works
(b) The architecture for inter- interrreflection
Geumeety Comparison
Figure 2: Differences between previous SDF architectures and the architecture for inter-reflections.
unnecessary to consider the indirect illumination like calculating the outgoing radiance at initial point p. This because the energy of light gradually degrades and the impact of light after twice reflections negligible compared to the explosively increased computational load. However, the additional computational load of ray tracing at depth two is still huge. To reduce our computational load, further propose to approximate the diffuse light and transfer the computations.
In Disney PBR equation 24 the diffuse term fa is:
Caiff
1 (FD90 1) (n wx)))) (1+ (FD90 (1 (n (o)))))))
π
F 0.5 + 2r cos² 0a.
(8)
where can is the diffuse albedo of the material. In this equation, f is related to the direction of the incoming radiance, which can yield more realistic results. However, using this equation to calculate the diffuse light for indirect illumination creates unnecessary extra computation. Because the term fa related to the normal and the direction of rays Wo. we cannot get the diffuse map before deep ray tracing. The following term is to introduce the roughness for diffuse light to avoid too dark edges in extremely low grazing angles. However, using this equation to compute the diffuse light for indirect illumination introduces unnecessary extra computation. To reduce the burden, we can follow H and approximate using Lambertian diffuse lighting.:
find
Where find is the diffuse light of the indirect illumination. In this approximation, fini is independent on the incoming direction wi and the outgoing direction wi- Based on this, the diffuse part of indirect light can be simplified as:
ind (p)
Caiff
( Li wi) (n wi) dw,,
(10)
π
where Lnd is the diffuse part of the indirect illumination. no longer relates to the direction of the sampling ray, we can present it via MLP M. Compared to the diffuse part, the specular part is strongly related to the ray direction. It is a disadvantage that we cannot approximate the specular part like the diffuse part, but it also advantage that only need sample minor rays in small specular lobe along the reflective directions. Therefore, we significantly reduce the computations of both the diffuse part and the specular part during the multi-times Monte Carlo Sampling.
3.3 Geometry Initialization
As seen in the equation 5 our indirect lighting highly depends on the geometry. However, the strong ambiguity of reflections makes it hard directly learn high-quality disentangled shapes. Therefore, we flatten the joint learning process of both geometry and materials into separate learning of them and thus reduce the number of physical terms to be disentangled in each learning stage.
18 to learn an initial geometry. The two branches of We first utilize SDF-based architectures 48
the diffuse and reflective networks well disambiguate the appearance with reflections and empower the
5

---

SDF network to produce high-qualigg geometry, shown in Fig. (a). However, this architecture encounters challenges of indirect illumination due the expressive capacity of Integrated Positional Encoding (IPE). It performs well in general scenes but shows limitations in representing inter- reflections. Due to the capacity Spheical Gaussians (SG) [50 to represent radiance directions, we introduce SG encoding instead of IPE to enhance the expressive capacity of reflective MLP, as shown Fig. (b). Besides, we directly parameterize diffuse appearance as SG coefficients which is suitable for objects with multiple reflective surfaces. The geometry comparison is shown in Fig. 2
The learned SDF can be then converted into a surface mesh using the Marching Cubes [9 which supports further tuning in the following inverse rendering pipeline. Different from Nvdiffrecmc which adopts DMTET for differentiable mesh optimization, we introduce Flexicubes 35] into our inverse rendering pipeline. Flexicubes use the SDF, weight, and the deformation for vertexes of the grid cells to extract the surface mesh using the DMC [29]. The mesh is converted into a differentiable representation that can be optimized based on gradients.
3.4 Training Objectives
The main objective is to minimize the photometric loss between rendered images and ground truth:
Lrgb = ||C Cgt 1|
(11)
where C is the rendering result, and Cyt is the corresponding ground truth. Following previous work [56] 28 19. adopt a smoothness loss for diffuse and material regularization:
Ld
Ika (Xaurt) ) - Kid. (Xsurf + +))l,
(12)
Xauor
Corm
Σ 1oomm (Xsurt) Korm (Xaurf + e)]
(13)
where Xsui represents the world coordinates of points on the object's surface. ka(xaur and korm (Xsut) is the material of this point. e is randomly distributed vector of tiny deformations. A self-supervised loss used regulate the learned diffuse color:
Lait Legh (Caift, Kan (I
(14)
where Cair is the diffuse light from the object surface obtained during rendering. kanr Xourt) is the diffuse light obtained by the MIP. Overall, the full loss function is:
= Lrgb + wi La + weform W3 Leill,
(15)
where W1, We and Wy are three predefined scalars.
Experiments
4.1
Implementation Details
Dataset. We construct a dataset multiple reflective objects based on the existing single objects to evaluate the performance. Our dataset consists of 16 groups of object compositions, most of which contain indirect illumination between reflective objects. We render the composed objects with various environmental lighting in the Blender engine. Each group contains 300 images, with 200 for the training set and 100 for the test set.
Experiment setup. We optimize the 3D model on RTX 3090 GPU with 24G memory. We use the Adam optimizer for the material and the environment map with an initial learning rate of 0.03. The coefficients of loss function W1, w2, and wy set to 0.1, 0.05, and 1, respectively. The rate of Monte Carlo sampling is commonly set to 128 consistent with the setting in Nydiffrecme 9
4.2 Comparison with Baseline
In this section, we compare our method with Nydiffrec 28
Nydiffrecme 9 and Nefi 45 on our
constructed reflective dataset, and the results are shown Fig. B Nydiffrec achieves peato-realistic results in the metal balls but fails on the smooth and glossy table horse. Besides, Nydiffrrec cannot

---

Ground Truth
Nan
Nan
Nan
Nan
Ours
Nydiffrec
Nydiffreemc
Nefi
renderings
materials
detail&probes
renderings
materials
details&probes
Figure 3: Qualitative comparison. The results of renderings, materials, and environment maps presented. Note that, the material of Nefii contains only roughness without metalness. Our method achieves the best renderings with clear reflections, compared to other inverse rendering methods. Our method is also superior others in the disentanglement of materials and environment maps.
Table 1: Quantitative Comparisons. Ours (x) means the times of sampling in method. Ours (1) is about 1.5h longer than Nydiffrecme 9
as we need additional time to learn the initial geometry.
NDR
NDRMC
Nefii
Ours
Ours
Ours (3)
Ours
Method
Ours
28
9
45
(w/o Acc.)
(w/o Geo.)
(1)
PSNR*
26.9
25.7
22.3
28.1
28.0
24.6
28.4
27.3
Training timel
30min
45min
20h
6.5h
11.5h
5h
26.5h
2.25h
well disentangle the materials because it does not consider the shading. In contrast, Nydiffreeme performs well in material learning, while its rendering results are bad. These two methods also suffer from indirect illumination from the reflection of inner objects, leading to low-quality environment maps. Nefii is the recent work that considers the indirect illumination in the radiance field, but it tends to produce low-reflective results and performs badly these high specular objects. Compared to these methods, our method can achieve both photo-realistic rendering results and well-disentangled material learning. We can handle highly specular objects, e.g., the metal table. The environment maps learned by methods are also superior others.

---

Depth=1
GT
Depth=2
Depth-1
Depth=3
Depth=1
Depth-2
Depth=2
Depth=3
Depth=3
Toaster bread
Table materials
Detail
kd
Korm
Probe
Figure 4: Ablation study on the depth of ray tracing, i.e., the times of Monte Carlo sampling. The results with depth=1 show fewer and darker reflections compared to the ground truth. ka maps also illustrate the limited capacity to disentangle the material from environmental light, for example, mistaking the diffuse color of the table as the color of the sky. In contrast, with depth=2 or 3, results show more realistic renderings and disentangled materials.
w/o
w/
w/o
w/
GT
w/o initial
w/ initial
kg
Kørm
Normal
Probe
Figure 5: Ablation study on geometric initialization. As shown, better-quality geometry can significantly improve the material learning and also refine the rendering results.
4.3 Multi-times Monte Carlo Sampling
In this section, we conduct an ablation study the times of Monte Carlo Sampling, i.e depth of ray tracing. Multi-times sampling considers the indirect illumination, and thus successfully disentangles the environment light and inner reflective light. As shown in Fig. the environment

---

maps learned by single-time sampling contain noise and shades that are actually the inner reflection. In contrast, multi-times sampling significantly releases the problem. producing clearer environment map. Single-time sampling also undermines the rendering results, with darker inverted reflections. When the sampling ray is sheltered by objects during path tracing, the ray returns no light, resulting in dark point. Multi-times sampling returns the reflective light of the sheltering points containing both diffuse and specular light. As a consequence, the rendering results of multi-times sampling show realistic appearance and reflections. Besides, it can be seen that results with 2-time sampling comparable to results with 3-time sampling. Tab. aalso quantitatively shows minimal improvement by deeper tracing. However, each additional sampling time induces a 4-times computation increment. Therefore, a 2-times sampling is a more cost-effective setting. Additionally, we also compare the efficiency between the sampling with and without acceleration. As shown in Tab. II our method can make the training speed twice as fast while keeping a comparable PSNR. The degree of acceleration is related to the complexity of geometry, and here we adopt the median in our dataset.
4.4 Geometry
In this section, we further explore the necessity of the initial geometry discussed in Sec. 3.3 and present our geometry reconstruction results. As shown in Fig. [ without initial geometry. the rendering results show bumpy surfaces with badly learned materials in the area of hollow shape. Because multi-times sampling amplifies the errors introduced by geometry, the Iearned materials are even worse than the single-time sampling. When applying a good-quality initialization, the network bypasses the ambiguity brought by the geometry. It helps our method learn well well-disentangled materials and finally produces high-quality rendering results. Quantitative comparison in Tab. [ showease real-scene geometry reconstruction result of the NeRO [18] dataset in Fig. O We further use Chamfer Distance quantitatively evaluate our geometric quality and present the results in Tab. -
also demonstrate the necessity of geometry initialization. In addition,
Real Image
GT
Ours
NeRO
Nydiffreeme
Nvdiffree
Figure 6: Real scene geometry comparison. We compare our reconstructed geometry with NeRO, Nvdiffree and Nydiffrecmc. Our reconstructed geometry demonstrates superior results.
Table 2: Geometry qualitative comparison. We use the Chamfer Distance ()) to evaluate our reconstructed geometry. As shown in the table, we obtain the best results in multiple scenes.
Dataset
Nydiffrec [28
Nydiffrecme [
NeRO [8]
Ours
Materials 0.016
0.016
0,0057
0.0030
Coral
0.28
0.25
0.13
0.13
4.5 Relighting and Material Editing
The disentangled environment map and material empower our methods to relight and edit recon- structed objects in the downstream application. Our method can also be seamlessly plugged into the industrial pipeline. As shown in Fig. 7 our method enables the arbitrary combination of reconstructed objects. We can easily edit their metalness, roughness, and albedo color by manipulating the learned materials. Due to the well-diseetanggled shading, our rendering results are natural and realistic in all five environment maps, even the point light in a dark environment. They also perform well after editing materials thanks to the well-learned material map. For example, after we increase the metalness and reduce the roughness of the teapot, the teapot clearly reflects the neighboring objects on its surface. In another case, where we extremely increase the reflectance and change the base color of the toaster, it accomplishes to reflect the scene and other objects. To showcase further applications

---

Relighting
Materials edits
Figure 7: Relighting and editing. We compose the reconstructed objects in unified scene and change the environmental light. The relighting results show our strong ability to disentangle the light and shading. We also perform material edits which shows the flexibility in wide applications.
NeRO
NeRO
Figure 8: Relighting results of real scenes. Our method demonstrates more realistic results under several lighting conditions than NeRO.
of our method, we conduct experiments on real-world datasets captured by NeRO [8] and present the comparison results in Fig- & can be that Ref-MC achieves realistic results under different lighting conditions.
Conclusion and Limitations
In conclusion, our Ref-MC2 introduces multi-times Monte Carlo sampling into the inverse rendering pipeline to model the indirect illumination. It improves the performance in scenes with complex inter- reflections. However, increment of sampling times significantly increases computational consumption and makes the pipeline highly scometry-sensithee based on the Lambert model, we simplify the BRDF for indirect lighting, which allows us to reduce the number of ray traces. To improve the geometry quality, we adopt SDF-based architecture to get an initial geometry and refine a design with Spherical Gaussian encoding for reflective objects. We further use Flexicubes to take the initial mesh into the differentiable rendering pipeline that learns disentangled materials. Our Ref-MC" still has several limitations. The major limitation is that the 2-time sampling cannot handle extremely reflective objects, e.g., mirrors, the specular energy hardly degrades after reflections, Besides, the training time needs to further reduce in future time.
To solve the challenge of computational efficiency,
Acknowledgements
This work supported NSFC (62201342) and Shanghai Science and Technology Major Project (202ISHZDZX0102), We also thank Student Innovation Center of SJTU for providing GPUs.
10

---

References
[1] Ronen Basri and David W. Jacobs. Lambertian reflectance and linear subspaces. IEFE Trans, Pattern Anal.
Mach. Intell., 25(2):218-233, 2003.
[2] Mark Boss. Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik P. A. Lensch. Neural-pil: Neural pre-integrated lighting for reflectance decomposition. In NewrIPS, pages 10691-10704,
2021.
[3] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In
ECCV, volume 13692, pages 333-350, 2022.
[4] Zhang Chen, Zhong Li. Liangchen Song. Lele Chen, Jingyi Yu, Junsong Yuan, and Yi Xu. Neurbf: A neural fields representation with adaptive radial basis functions. In ICCV. pages 4159-4171, 2023.
[5] Sara Kridovich-Keil. Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.
Plenoxels: Radiance fields without neural networks. CVPR, 2022.
[6] Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, and Ting-Cong Chen. Ref-neus: Ambiguity reduced neural
implicit surface learning for multi-vieu reconstruction with reflection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 4251 12260, 2023.
[7] Yuan- Yuan-Chen Guo, Yan-Pei Cao, Chen Wang. Yu He. Ying Shan, and Song-Hai Zhang. Vmesh: Hybrid volume-mesh representation for efficient view synthesis. In SIGGRAPH Asia, pages 17:1-17:11, 2023. [8] Yuan-Chen Guo, Di Kang, Linchao Bao, Yu Hc, and Song-Hai Zhang. Nerfren: Neural radiance fields with reflections. In CVPR. pages 18388-18397 1899.. 2022.
[9] Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, light, and material decomposition from images using monte carlo rendering and denoising. Advances in Neural Information Processing Systems,
35:22856-22885 2022.
[10] Wenho Hu, Yuling Wang, Lin Ma, Bangbang Yang. Lin Gao, Xiao Liu, and Yuewen Ma. Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In ICCV, pages 19717-19726, 2023,
[11] Binbin Huang. Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields, In SIGGRAPH, 2024.
[12] Vingwenqi Jiang. Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang. and Yuexin Ma. Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces. arXiv preprint arXiv:2311.1999 17977. 2023.
[13] Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowci Zhou, Zexiang Xu,
and Hao Su. Tensoir: Tensorial inverse rendering. In CVPR, pages 165-174, 2023. [14] James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques. pages
1986.
[15] James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics. 1653):1651174, 1984.
[16] Bernhard Kerbl. Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139:1-1399 2023.
[17] Ruofan Liang. Huiting Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, and Nandita Vijaykumar. ENVIDR: implicit differentiable renderer with neural environment lighting. In ICCV, pages 79-89, 2023. [18] Yuan Liu, Peng Wang. Cheng Lin. Xiaoxiao Long. Jiepeng Wang, Lingjie Liu, Taku Komura, and Wenping Wang. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. ACM Transactions on Graphics (TOG), 42(4):1-22, 2023.
[19] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphicx: pioneering efforts that shaped the field, pages 347-353, 1998. [20] Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold-gs: Structured 3d gaussians for view-adantive rendering. CVPR, 2024.
[21] Jipeng Lv. Heng Guo, Guanying Chen, Jinxiu Liang. and Boxin Shi. Non-lambertian multispectral photometric stereo via spectral reflectance decomposition. In IJCAI, 2023.
[22] Alexander Mai, Dor Verbin, Falko Kuester, and Sara Fridovich-Kell Neural microfacet fields for inverse rendering, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 408-44 -418, 2023.
[23] Ricardo Martin Bartin-Brualla, Noha Radwan, Mehdi S. Sajjadi. Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR,
pages 7210-7219, 2021.
[24] Stephen McAuley. Stephen Hill, Naty Hoffmman, Yoshiharu Gotanda, Brian Smits, Brent Burley, and Adam Martinez. Practical physically-bast -bssed shading in film and game production. In ACM SIGGRAPH 2012 Courses, pages
papess 1-7. 2012.
[25] Stephen McAuley, Stephen Hill, Adam Martinez, Ryusuke Villemin, Matt Pettineo, Dimitar Lazarov, David Neubelt, Brian Karis, Christophe Hery, Naty Hoffman, al. Physically based shading in theory and practice. In ACM SIGGRAPH 2013 Courses, pages 1-8. 2013.
[26] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[27] William J Morokoff and Russel Caflisch. Quasi-monte carlo integration. Journal of computational
physics, 122(2):218-230 1995.
[28] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8280-8800000 8290, 2022.
11

---

[29] Gregory M Nielson. Dual marching cubes. In IEEE visualizution 2004, pages 489-496. IEEE. 2004.
[30] Michael Occhsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5589-5599, 2021.
[31]
Michael Oren and Shree K Nayar. Generalization of lambert's reflectance model. In Proceedings of the
21s annual conference on Computer graphics and interactive techniques, pages 239-246. 1994.
[32] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Steven M. Seitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. In ICCV, pages 5845-5854, 2021.
[33] Christian Reiser, Stephan Garbin, Pratul Srinivasan, Dor Verbin, Richard Szeliski, Ben Mildenhall, Jonathan Barron, Peter Hedman, and Andreas Geiger. Binary opacity grids: Capturing fine geometric detail for mesh- based view synthesis. ACM Transactions on Graphics (TOG), 43(4): 14, 2024.
[34] Christian Reiser, Richard Szeliski, Dor Verbin, Pratul P. Srinivasan, Ben Mildenhall, Andreas Geiger, radiance fields for real-time view Jonathan T. Barron, and Peter Hedman. MERF: memory-
synthesis in unbounded scenes. ACM Truns. Gruph. 42(4h).44(44)88))) 42:4))89: 1-89:12. 2023.
[35] Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcie, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flcxible isosurface extraction for gradient-based mesh optimization. ACM Transactions on Graphics (TOG), 42(4):1-16, 2023.
[36] Zixi Shu, Ran Yi, Yuqi Meng, Yutong Wu, and Lizhuang Ma. Rt-octree: Accelerate plenoctree rendering with batched regular tracking and neural denoising for real-time neural radiance fields. In SIGGRAPH Asia, pages 99:1 19:1-99:11. 2023.
[37] Peter-Pike J. Sloan, Jan Kautz, and John M. Snyder. Precomputed radiance transfer for real-time rendering in dynamie. low-frequency lighting environments. ACM Tranx. Gruph., 21(3):527-536, 2002.
[38] Jiajun Tang, Haofeng Zhong, Shuchen Weng, and Boxin Shi. Luminaire: Illumination-awann conditional image repainting for lighting-reatin
generation. In NewrIPS, 2023.
[39] Fric Veach and Leonidas J Guibas. Optimally combining sampling techniques for monte carlo rendering. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 419-428, 1995.
[40] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5481-5490 IEEE, 2022.
[41] Fangjinhua Wang. Murie-Julie Rakotosaona, Michael Niemeyer, Richard Szeliski, Marc Pollefeys, and Federico Tombari. Unisdf: Unifying neural representations for high-fidelity 3d reconstruction of complex scenes with reflections. arxiv preprint arXiv:2312. 13285, 2023.
[42] Jiaping Wang, Peiran Ren. Minmin Gong, John M. Snyder, and Baining Guo. All-frequency rendering of dynamic, spatially-varying reflectance. ACM Trans, Gruph., 28(5):133, 2009.
[43] Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106. 10689, 2021.
[44] Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance
fields without known camera parameters. arXiv preprint arXiv:2102.0706. 2021.
[45] Haoqian Wu, Zhipeng Hu, Lincheng Li, Yongqiang Zhang, Changjic Fan, and Xin Yu. Nefii Inverse rendering for reflectance decomposition with near-field indirect illumination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4295-4304, 2023.
[46] Xiuchao Wu, Jiamin Xu, Zihan Zhu, Hujun Bao, Qixing Huang. James Tompkin, and Weiwei Xu. Scalable
neural indoor scene rendering. ACM Trans. Graph. 41(4):98:1 199:161.111 2022.
[47] Kun Xu, Wei-L.un Sun, 7hao Dong, DannYong Zhao, Run-Dong Wu, and Shi-Min Hu. Anisotropic
spherical gaussians. ACM Trans. Graph., 32(6):209:1 1-29:111 2013.
[48] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces.
Advances in Neural Information Processing Systems, 34:4805 44-4800-4815.. 2021.
[49] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, and Ben Mildenhall. Bakedsdf: Meshing neural sdfs for real-time view synthesis. In SIGGRAPH,
pages 46:1-46:9, 2023.
[50] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul Srinivasan, Richard Szeliski, Jonathan Barron, and Ben Mildenhall. Bakedsdf: Mcshing neural sdfs for real-time view synthesis. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-9, 2023.
[51] Lior Yariv, Yoni Kasten, Dror Moran, Mcirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Advances in Neural
Information Processing Systems, 33:2492 2502, 2020.
[52] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. Plenoctrees for real-time
rendering of neural radiance fields. ICCV, pages 5732-5741, 2021.
[53] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d
gaussian splatting. CVPR, 2024.
[54] Kai Zhang. Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg:
Inverse rendering with
spherical gaussians for physics-basee based material editing and relighting. In CVPR, pages
2021.
[55] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladien Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 2020.
12

---

[56] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng. Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under unknown illumination. ACM
Transactions on Graphics (ToG), 40(6):1-11 2021.
[57]
Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling indirect illumination inverse rendering. In CVPR, pages 18622-18631, 2022.
[58] Youjia Zhang, Teng Xu, Junqing Yu. Yuteng Ye. Yanqing Jing, Junle Wang. Jingyi Wei Yang.
Nemf: Inverse volume rendering with neural microflake field. ICCV. pages 22862-22872 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction accurately reflect the contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [No]
Justification: The data and code have not been provided to reviewers.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification:
We will release the data and code a few weeks.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pippe.cc/pullic/guuii wwwtc//ggue////eeeeceoocc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the implement details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Results are accompanied by error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report compute resources in the experiment set up.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:: httppss:///riiss../e/ http:/p......p.iiiittcccccccllea
Answer: [Yes]
Justification:
The research conforms with that.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss them in the introduction and experiments.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee wwworeewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
18

---

Answer: [Yes]
Justification:
We have discussed them in section experiment.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "afeOgnXpjE", "reviewer": "Reviewer_9PRG", "summary": "This paper presents a method for learning disentangled scene representations from images. The proposed pipeline has two stages: the first recovers geometry using an SDF-based model to learn geometry from the images through differentiable rendering. The second applies differentiable ray tracing to predict material parameters and environment lighting. The proposed method differs from previous approaches in that it renders inter-reflections between objects which create indirect lighting rather than only reflections of the environment and shadows. This is shown to produce better results than existing methods in scenes where these inter-reflections are present.", "questions": "Would it be possible to run on the real capture dataset from NeRD, similar to nvdiffrec?", "limitations": "The authors are clear about limitations, and I think they cover them well.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The core idea is solid, and seems to have the expected effect on improving both recovery of environment lighting and rendering quality. I think it would also be fairly applicable to other pipelines which use ray tracing to recover lighting, most likely with similar benefits.\n\nThe explanation is clear, and I think the method should be reproducible from the paper, and the authors have also said they will release code.\n\nGenerally, I think this is a good contribution tackling a quite difficult problem. The strategies of diffuse pre-computation and spherical Gaussian radiance models will likely be useful to future efforts in this direction.", "weaknesses": "It would have been nice to see some results on real data. The synthetic examples seem to show a clear improvement, but it would help to see the difference in a more practical setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***More experiments on real-world datasets***  \n\nThank you for your valuable suggestions. Experiments on real-world datasets are important for assessing the practicality of inverse rendering tasks. Therefore, we have supplemented the experiments on two real-world datasets, i.e., NeRD and NeRO. On the NeRD dataset, we compared our method with nvdiffrec and nvdiffrecmc. Additionally, the NeRO dataset includes several high-reflection objects, making it a challenging dataset. Conducting experiments on this dataset allows for a convenient comparison with the strong baseline NeRO. Thus, we also performed experiments on this dataset, with results shown in Figures 1, 2, and 3 in the rebuttal pdf.\n\n\nThese results demonstrate that our method can generate detailed geometry and accurate BRDF materials from multi-view images, even when dealing with challenging real-world objects. This capability results in impressive relighting outcomes. However, our results on the NeRD dataset did not show significant improvements over nvdiffrec and nvdiffrecmc, which could be due to two main reasons: first, the object's base color is very strong, making it difficult for indirect lighting to have a significant impact. Second, the object has a predominantly convex shape, making it challenging to receive indirect light reflected from other areas. Therefore, our method's strengths are not fully demonstrated on this particular object.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xemRCisXwt", "reviewer": "Reviewer_VkB8", "summary": "The authors propose an inverse rendering method that reconstructs the geometry, materials, and lighting of 3D objects from 2D images, effectively handling scenes with multiple reflections and inter-reflections. To address the high computation cost of Monte Carlo sampling, the authors propose a specularity-adaptive sampling strategy, significantly reducing the computational complexity. The authors also introduce a reflection-aware surface model to initialize the geometry and refine it during inverse rendering, to improve the inverse rendering reconstruction quality. Experiments show the proposed method outperforms other inverse rendering methods.", "questions": "1. Please show some results on real data with some baseline comparison.\n2. Please provide more baseline comparisons and quantitative comparisons as discussed in the weakness section.\n3. Please answer the question related to SG encoding.", "limitations": "The limitations have been discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand.\n2. The proposed methods are intuitive and the results look nice.", "weaknesses": "1. The validation on real data is missing. The paper only shows its results on synthetic data. It would be better to test this work on challenging real data, such as the specular real data captured by NeRO.\n\n2. The selected baselines are incomplete. There exists a strong baseline candidate, NeRO, which can also reconstruct the geometry and materials of specular objects. (Although its main focus is geometry reconstruction). Since it has released all of its code, it would be better if the authors could also compare this work with NeRO.\n\n3. Need more analysis on the SG encoding. SG encoding looks like a very important part of the final quality(because it plays an important role in the geometry reconstruction). However, the authors only use a small part of Fig. 2 to showcase its effectiveness. More visual comparison and quantitative comparison would be appreciated. In the meanwhile, SG encoding seems to be very similar to [1], which was released on Arxiv last year. The authors should explain the difference between its SG encoding and [1]'s encoding. Since the authors didn't cite [1], I assume that SG encoding should be the author's original contribution.\n\n4. The quantitative comparison is insufficient. More quantitative comparisons on the relighting results and reconstruction of BRDF and geometry should be included.\n\n[1] SpecNeRF: Gaussian Directional Encoding for Specular Reflections", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Missing validation on real data:***  \nThank you very much for your instructive feedback, which pointed out the oversight in our experiments. The NeRO real-world dataset is indeed a challenging dataset because the color of the object's surface can change significantly with different viewing angles, causing ambiguities in both material and geometry. We have supplemented the results on the two NeRO real-world datasets, bear and coral. On the bear dataset, we detailed our rendering results, PBR materials, and reconstructed geometry, as shown in Figure 1. Figure 2 shows our relighting results on the bear and coral datasets. From the relighting results, it can be seen that the disentangled PBR materials are reasonable, and our method effectively distinguishes between the object's material and the reflections on its surface. Figure 3 shows our geometric reconstruction results on the coral dataset. Our method achieves results very close to the ground truth, similar to NeRO, and shows significant improvements over inverse rendering methods like nvdiffrec and nvdiffrecmc.\n\n\n***Comparisons with NeRO:***   \nNeRO is a recently outstanding work for object reconstruction, focusing on the challenging scenario of high-reflection objects and achieving excellent results. The difficulty of high-reflection objects lies mainly in two aspects: First, due to the nature of specular reflection, the observed color of an object changes significantly with the viewing angle, leading to view inconsistency. Second, for smooth objects, especially those with low roughness, indirect lighting has a greater impact and can even play a major role from certain angles. However, the calculation of indirect lighting is closely related to the object's geometry, the material properties of other regions of the object, and ambient lighting, making it a relatively difficult value to model and compute.\n\nIn the rebuttal pdf, we compared our work with NeRO in terms of rendering quality, PBR materials, geometric reconstruction, and relighting. In Figure 1, we provide a detailed comparison on the NeRF synthetic dataset materials and the NeRO real-world dataset bear. Our method achieved excellent results on both materials and bear. While NeRO performed well on bear, it encountered issues on materials. It can be seen that indirect lighting caused significant ambiguities in their geometry. Both our method and NeRO's method consider indirect lighting, but their indirect lighting is implicitly learned through neural networks, whereas ours is explicitly calculated using ray tracing and Monte Carlo sampling, which may explain the differences in our results.\nFigures 2 and 3 show the geometric reconstruction results on coral and our relighting results on bear and coral, respectively. These results demonstrate that our method is not inferior to NeRO in terms of geometric and material reconstruction of high-reflection objects, and in scenarios where light continuously reflects between multiple objects, we can achieve even better results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "phyiraSKut", "reviewer": "Reviewer_UfeL", "summary": "This paper proposes an inverse rendering method that handles multi-time Mante Carlo integration which models indirect illumination. It reduces the computational cost by pre-computing the diffuse map based on a Lambertian model. It also proposes to use spherical Gaussian encoding to improve the initial SDF reconstruction of scenes.", "questions": "Please see the weaknesses.", "limitations": "A crucial limitation, i.e. diffuse shadows are not handled by the proposed method, are omitted and not properly discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The presentation is clear and easy to follow.\n\n2. Modeling indirect illumination is so far under-explored in inverse rendering", "weaknesses": "1. Overclaim of novelty: multi-bounce MC integration has already been employed in earlier inverse rendering work [1]. On the other hand, the diffuse Fresnel term (Eq. 7) is not commonly used in recent inverse rendering works. Most related works (TensoIR, nvdiffrec/nvdiffrecmc) use exactly Eq 9, so the claim of transferring from Eq. 7 to Eq. 9 should not be counted as a contribution.\n\n2. Continuing from (1), using a pre-computed diffuse map also prevents correct modeling of diffuse shadows, this is not discussed nor mentioned in the paper.\n\n3. Inaccurate/incomplete description of the geometry module: IPE is used for encoding input positions to MLPs, while judging by the text (line 181) spherical Gaussian should be used for encoding viewing/reflecting directions. However, in this work, spherical Gaussian is actually used to replace IPE, which is very confusing. Also in Figure (2b) the entire Diffuse MLP is replaced with \"SG\" and there is no explanation of what this stands for. Also in the same figure, the term \"SGE\" is not explained (maybe spherical Gaussian encoding? Then what is the difference between SGE and SG?)\n\n4. Weak evaluation: there is only a single table in the entire paper. Only PSNR and training time are reported. It is not clear how the PNSR metric is computed (is it novel-view synthesis? Is it relighting?). There is no evaluation of other intrinsic properties that are commonly evaluated in other inverse render papers, e.g. albedo, normal error, etc. The proposed spherical Gaussian encoding is not properly ablated - only qualitative comparison is presented.\n\n5. Missing datasets: it would make the evaluation more complete if the method could evaluate more standard datasets (NeRFactor dataset, TensoIR dataset) where many other works have been evaluated. Also, there is no real-world dataset tested, making it questionable whether the proposed algorithm can work in real-world scenarios.\n\nFactual errors: GGX has nothing to do with the diffuse term used in Disney BRDF. GGX is a microfacet normal distribution function that models specifically the specular part of the Disney BRDF. The diffuse Fresnel term in Disney BRDF is invented by the original Disney BRDF paper and is often omitted in real-time rendering engines and replaced with the Lambertian model.\n\n[1] Mai et. al. Neural Microfacet Fields for Inverse Rendering, ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Overclaim of novelty***   \nThe paper Neural Microfacet Fields for Inverse Rendering is indeed an excellent work that considers indirect lighting, but their method for computing indirect lighting is significantly different from ours. As stated in their paper, they approximate the rendering equation into form Equ(15), but a key variable in this equation is obtained by calculating the environment light map. This is fundamentally different from our approach. Our method involves performing Monte Carlo sampling again, essentially repeating the same method to compute it anew. Their method essentially belongs to the implicit learning of indirect lighting, which I believe does not involve computing indirect lighting using the multi-bounce MC integration method.\nFurthermore, in our paper, we list Equations 7 to 9 to explain the principle of approximating the diffuse part of our indirect lighting using the diffuse map. Although real-time rendering \n\n***Diffuse map***  \nOur precomputed diffuse map is used only to approximate the diffuse part of the indirect lighting. Although the diffuse map does influence the modeling of shadows, it affects the modeling of light within the shadows rather than the light directly entering the viewer's eye. In our Table 1, we also demonstrate the impact of using the diffuse map on the results. The results show that using our acceleration structure has almost no impact on the accuracy of the final results, while significantly saving computation time.\n\n***Weak evaluation and Missing dataset:*** \nThank you for pointing out the shortcomings of our quantitative comparative experiments. The PNSR metric is computed in novel-view synthesis. Indeed, our current judgment on the quality of target material decoupling is indeed at the level of subjective judgment based on life experience, so I use the table-horse data set in our data set , generated the albedo and roughness of his GT, and used this data set to quantitatively measure the decoupling effect of our work. The influence of geometry on the final effect does require quantitative experiments to prove the effectiveness of the method we use. Therefore, we have made some supplements to the experiments in the geometry part. We have put the quantitative comparison of geometry on the final effect in the PDF. Figure 5.\n\n\nOtherwise, thank you for pointing out the inaccuracies in our paper. According to Disney's article and the book Real time rendering, GGX is indeed described as a function of the surface normal vector distribution, and it does not represent the BRDF proposed by Disney. Thanks again for finding our error and correcting it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bDbWEM4x4x", "reviewer": "Reviewer_izde", "summary": "This paper proposes a comprehensive inverse rendering method that incorporates indirect illumination to enhance the decomposition quality of environment lighting and BRDF materials. Specifically, this work uses multi-time Monte Carlo integration to model light transport and devises algorithms to accelerate computation by precomputing the diffuse term and leveraging an SDF-based geometry for initialization. Both qualitative and quantitative results are promising, demonstrating realistic shadows on glossy surfaces and higher PSNR values.", "questions": "In general I enjoyed reading this paper. The content is well written and clear to me. Therefore, I’m on the positive side of this submission. A few questions regarding the technical details:\n1. Will the geometry obtained from Sec. 3.3 be optimized in inverse rendering? The writeup suggests this (Lines 189-190), but I didn't find any experiments addressing it. I am curious about potential geometry improvements.\n2. Regarding the material editing mentioned in Section 4.5, I have some questions about its implementation. It seems that material properties are represented as MLP. How do you intuitively edit different material properties, such as roughness? Or are these properties explicitly exported and stored as UV maps?\n3. What does “orm” stand for in k_orm?", "limitations": "Limitations and potential negative social impact are well discussed.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The results exhibit a decent quality. By incorporating indirect illumination, the modeling of shadows and glossy surfaces achieves a remarkable degree of photorealism.\n2. The underlying theory is sound to me. The GGX appearance model is de facto in industry, and the approximation applied for the diffuse term is standard and commonly used in real-time rendering.\n3. As mentioned, I believe that some insights are derived from real-time rendering techniques and I personally appreciate it as accurately modeling indirect illumination is also important for effective inverse rendering.", "weaknesses": "1. The optimization time is relatively long compared to other methods. \n2. Due to computational limitations, the number of bounces is restricted, thus limiting the ability to model shiny surfaces and confining the results to glossy surfaces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. This is indeed a current limitation of our approach. Using Monte Carlo integration to calculate the reflection equation itself is more computationally intensive than the real-time rendering split sum method used by nvdiffrec, because we have to use ray tracing and integration. To make matters worse, as the tracking depth increases, the number of rays we need to emit will increase exponentially, but fortunately, as we show in Figure 4 in the paper, a tracking depth of 2 is enough to handle most In the scene, the tracking depth of 3 will greatly increase the amount of calculation, but it cannot significantly improve the final effect.\n\n2. We will optimize the geometry in subsequent calculations. But unfortunately only PSNR can be improved. Compared with non-optimized geometry, optimized geometry generally improves PSNR by 1 to 2 dB in the new perspective verification set, but the geometry cannot be improved. We speculate that this is the process of using the implicit method to learn geometry, and there is no rasterization step. Our differentiable rendering requires rasterization, so it should be the direct application of geometric PSNR caused by the difference in the forward process. lower. Therefore, we can overcome this shortcoming and obtain higher PSNR by optimizing the geometry in subsequent inverse rendering.\n\n3. This is a map, where R, G, and B channels represent alpha, roughness, and metallic respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their constructive comments. Your suggestions have been invaluable in refining and strengthening our work. In this general response, we will address the three important parts that were commonly mentioned in the discussions, namely the experiment on the real-world dataset, the comparison experiment with a strong baseline candidate, and the experiment on the geometry module.


***Experiments on real-world dataset***  

We acknowledge the importance of evaluating our proposed method on the real-world dataset to demonstrate its applicability and generalization. We have taken your advice into consideration and conducted extensive experiments on a diverse and challenging real-world dataset. For ease of comparison with the current strong baseline NeRO, we conducted experiments on the high-reflection real-world datasets bear and coral captured by them. The results are shown in Figures 1, 2, and 3. Additionally, we conducted experiments on the NeRD real-world dataset, with results shown in Figure 4. Through these experiments, we aim to demonstrate the effectiveness and generalizability of our approach in handling challenging real-world datasets.

***Comparison experiment with a strong baseline candidate***   

We appreciate your suggestion to compare our work with the latest outstanding work. NeRO is an excellent contribution to the field of object reconstruction, focusing on high-reflection objects, much like our work. To demonstrate the effectiveness of our method, we conducted a comprehensive comparison with NeRO from three aspects. Figure 1 presents a thorough comparison on the NeRF synthetic dataset materials and NeRO’s real-world dataset bear. Figure 2 shows our relighting results on real-world datasets. Figure 3 displays our geometric reconstruction results on the NeRO real-world dataset coral. These results clearly demonstrate that our method performs on par with NeRO on their real-world datasets, and we achieve even better results in scenarios like the materials dataset, where light continuously reflects between objects.

***Experiment on the geometry module***  
Thank you very much for pointing out the shortcomings in our geometric experiments. In our method, high-quality geometry is indeed a critical factor affecting the final results, as our approach to calculating indirect lighting relies on accurate geometry. Ambiguities in geometry can also lead to ambiguities in the object's material and surface color. Therefore, in Figure 5, we present the direct impact of learning geometry with our improved method compared to the previous method on subsequent results, as well as the quantitative comparison in terms of PSNR.


We believe that these revisions and additional experiments will strengthen our paper and contribute to the advancement of the field. We are grateful for your valuable comments and appreciate the opportunity to enhance the quality and impact of our work.

Meta-review: This paper received four positive leaning reviews --- one 5 (borderline accept), one 6 (weak accept), and two 7s (accept). 

There was broad appreciation for the technical soundness of the work, clean presentation, high quality of the results, and the overall importance of the problem addressed herein (modeling interreflections in inverse rendering). 

There were some concerns raised around limited real experiments, and other technical issues which the authors addressed in a comprehensive rebuttal. There was substantial post-rebuttal discussion, following which, an accept consensus was reached. Hence, the decision to accept the paper. 

The authors are strongly urged to incorporate the reviewer feedback and include the additional comparisons / validation promised in the rebuttal when preparing the final camera-ready version.
Decision: Accept (poster)