Multi-times Monte Carlo Rendering for Inter-reflection Reconstruction

Authors: tengjie  zhu
zhuo  chen
jingnan  gao
yichao  yan
xiaokang  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 16:31:06

Abstract: Inverse rendering methods have achieved remarkable performance in reconstructing high-fidelity 3D objects with disentangled geometries, materials, and environmental light. However, they still face huge challenges in reflective surface reconstruction. Although recent methods model the light trace to learn specularity, the ignorance of indirect illumination makes it hard to handle inter-reflections among multiple smooth objects. In this work, we propose Ref-MC2 that introduces the multi-time Monte Carlo sampling which comprehensively computes the environmental illumination and meanwhile considers the reflective light from object surfaces. To address the computation challenge as the times of Monte Carlo sampling grow, we propose a specularity-adaptive sampling strategy, significantly reducing the computational complexity. Besides the computational resource, higher geometry accuracy is also required because geometric errors accumulate multiple times. Therefore, we further introduce a reflection-aware surface model to initialize the geometry and refine it during inverse rendering. We construct a challenging dataset containing scenes with multiple objects and inter-reflections. Experiments show that our method outperforms other inverse rendering methods on various object groups. We also show downstream applications, e.g., relighting and material editing, to illustrate the disentanglement ability of our method.

Main paper: 
 introduction: 
 Neural Radiance Fields (NeRF) [26] and 3DGS [16] have demonstrated their excellent performance on novel view synthesis. However, it is difficult to directly apply their reconstructed 3D model to the current industrial pipeline, leading to the lack of flexibility in many downstream applications, e.g., relighting and material editing. To better cooperate with mature techniques, inverse rendering bases the physical rendering [24] and utilizes the neural network to learn disentangled materials that can be seamlessly plunged into the industrial pipeline for further manipulation.Previous methods [28, 9, 45, 57] have explored how to disentangle geometry, diffuse, roughness, metalness, and environmental light from multi-view images, but they still face challenges in shading and reflective objects. Specifically, Nvdiffrec [28] proposes a differentiable pipeline that enables gradient-based optimization on both meshes and volumetric textures. These methods ignore the shadow when modeling the illumination, resulting in failed disentanglement for diffuse and shading appearances. To model more realistic shadings, Nvdiffrecmc [9] further incorporates ray tracing and Monte Carlo sampling [27] into inverse rendering, significantly improving the decomposition of shape, materials, and lighting. However, it ignores the indirect illuminations during path tracing and treats rays attacking object surfaces as sheltered illumination. As a consequence, methods like Nvdiffrecmc increase the ambiguity of geometry reconstruction and undermine material learning. Unfortunately, the scene with multiple inter-reflections is common in the real world. The failure in these scenes hinders the wider applications of these methods. Recent methods [17, 40, 22] take inter-reflections into account and can reconstruct the reflective objects well. However, the implicit representations of these methods for materials and renders sacrifice their scalability to downstream tasks. Recently, Nefii [45] further incorporated the implicit neural radiance to estimate ray tracing, alleviating the ambiguity between materials and indirect illuminations. However, this leads to a huge computational consumption for path tracing. Neural Microfacet Fields [22] employs two-bounce sampling for indirect light modeling, but its geometry field limits flexibility for downstream tasks.In this paper, we proposed a full inverse rendering method, Ref-MC 2 , which considers inter-reflections during ray tracing to improve the decomposition of explicit materials and environmental lighting. The core of our method is to use Multi-times Monte Carlo integration [27] and BRDF [24] rendering to approximate the indirect illumination at multiple reflection points along the light propagation path. Although our method takes advantage of hardware-accelerated ray tracing to model the indirect illumination, we still face two major challenges brought by the multi-times Monte Carlo sampling. 1) Efficiency: the explosive computational growth from multiple times is too heavy for the hardware algorithm only. 2) Geometry: the geometry quality greatly affects the calculation of indirect light and the decomposition of the materials, because the error will accumulate over and over again as the times of Monte Carlo sampling increase.To address these challenges, we correspondingly propose two strategies. 1) For efficiency, we flatten the multi-times sampling into sequential single-time sampling. In a Lambert model [31] , the diffuse light is independent of the direction of reflection and thus can be presented as a map to query at any time. When we trace the indirect light from an object, instead of recalculating the diffuse component, we can take the value directly from the diffuse map. It can be optimized through self-supervision. Therefore, we only need to sample the specular component within a small lobe along around the reflective direction. 2) For geometry, we refer to the SDF-based methods [41, 17, 6] and replace the common positional encoding with the Sphere Gaussian encoding to get an accurate initial geometry for reflective objects. We use this geometry to initialize Flxicubes [35] that optimizes surface meshes based on the gradient, and further fine-tunes the Flexicubes in a differentiable pipeline. To evaluate our framework, we construct a dataset containing difficult scenes which contain strong inter-reflections between multiple surfaces. Extensive experiments demonstrate that our framework can successfully decompose indirect illumination and materials. In summary, our contributions are:• We propose a full inverse rendering method that employs multi-times Monte Carlo sampling to correctly decompose indirect illumination and materials. • We reduce the computational consumption when tracing indirect illumination by selfsupervising the diffuse map based on the Lambert model. • We refine the SDF-based architecture with Spherical Gaussian encoding to obtain a high-quality initial geometry which further releases the accumulated error during multi-times sampling. • We construct a dataset to evaluate the performance on indirect illumination.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our abstract and introduction accurately reflect the contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
implicit neural representations: 
 Neural implicit representations [23, 26, 32, 55, 44, 34, 10, 3, 4, 58, 36, 7] have achieved impressive success in many computer vision and computer graphics tasks. These methods use neural radiance fields to capture color and volume density, generating photo-realistic novel views through volume rendering [15] . However, the unconstrained volumetric representation of the original NeRF method leads to low-quality geometry. Recent 3D Gaussian Splatting (3DGS) [16, 11, 12, 20, 53] has gained popularity in novel view synthesis. Different from NeRF, it is an explicit representation that involves the optimization of multiple Gaussians to reconstruct 3D objects. 3DGS learns color and density in a volumetric point cloud, but it also fails to produce accurate geometry due to its discrete representation. Besides, these methods are all entangled learning, integrating all inherent materials and the environment map into the appearance. This limits the downstream applications in the current industrial pipeline. To produce a high-quality geometry, follow-up works [48, 43, 51, 30] use a function to associate signed distance field (SDF) and volume density. The surface mesh can be extracted from neural implicit surfaces by Marching Cubes [19] , and this 3D asset can be further applied in other applications. However, they usually perform badly in reconstructing the reflective objects due to the ambiguity of reflective appearance.Although neural implicit surfaces have achieved impressive performance in geometry reconstructing and novel views synthesis, they do not obtain the fundamental materials of PBR which limits their flexibility in downstream tasks. Neural inverse rendering methods [17, 46, 8, 2, 54, 57, 13, 38, 21] introduce the physical rendering equation to estimate the disentangled diffuse and specular component from RGB images. They approximate the rendering equation based on neural networks or basis functions, e.g., Spherical Gaussians [42, 47, 49, 54] and Spherical Harmonics [5, 1, 37, 52] . Nvdiffrec [28] introduces the full inverse rendering that estimates shape, materials, and environmental light into gradient-based optimization. However, it does not consider the shadows, leading to the entanglement of materials. Recent works [6, 41] extend SDF-based architectures with an additional appearance branch to model the reflections on the object. RefNeuS [6] introduces a reparametrization method to distinguish the reflective appearance from the diffuse appearance by a direction-relative process, but it fails to faithfully reconstruct non-reflective objects. UniSDF [41] proposes to use a weight-MLP to balance the reflective and non-reflective branches for different objects. However, these methods still face challenges in scenes with complex inter-reflections. Nvdiffrecmc [9] extends Nvdiffrec with Monte Carlo sampling to trace the light path but still ignores indirect illumination between objects. Further methods [17, 18, 45] consider indirect illumination in their design. ENVIDR [17] employs a neural renderer to learn the physical light interaction, without explicitly formulating the rendering equation. NeRO [18] applies the split-sum approximation to approximate the shading effects of both direct and indirect lights. Nefii [45] introduces ray tracing to the radiance field to model indirect illuminations. Neural Microfacet Fields [22] employs two-bounce sampling to accurately calculate indirect illumination. However, NMF represents geometry by a density field, which limits the scalability and flexibility for downstream tasks. In addition, these aforementioned methods do not fully disentangle the materials from RGB images, but decompose the appearance into reflective and diffuse color. It is hard to apply to the current industrial pipeline directly.eq: rendering equation In contrast, our Ref-MC 2 is a full inverse rendering method that also considers indirect illumination.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have discussed the limitations of the work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 The rendering equation [14] is commonly used to compute the outgoing radiance L o (p, ω o ) from the point p in outgoing direction ω o :EQUATIONwhere L i (p, ω i ) is the incoming radiance into p from the direction ω i , n is the normal of the point p, Ω is the hemisphere of directions above p, f (p, ω i , ω o ) is the BSDF evaluated for ω i and the current incoming direction ω i . The GGX [24] physics-based BSDF function is proposed to decompose the function into several physical terms. The function is described as:EQUATIONwhere f d is the diffuse term and f s is the specular term. D, F , and G are the microfacet distribution function, the Fresnel reflection coefficient, and the geometric attenuation, respectively.Previous works [28] use the split sum function [25] to approximate the rendering equation, but it inevitably omits the shadow and indirect illumination. In contrast, Monte Carlo integration [27] is a simple but unbiased estimation method that comprehensively considers all the physical terms for the outgoing radiance. The Monte Carlo integration rendering equation is:EQUATIONFigure 1 : We perform Monte Carlo sampling at the viewpoint. When the sampling ray from the point is not blocked, it is the direct illumination from environmental lighting. When the sampling ray hits an object, we divide this indirect illumination from the object into diffuse light and specular light. We sample the diffuse light from a diffuse map that is optimized through self-supervision. For specular light, we only need to partially trace the rays in a small specular lobe along the reflective direction.The gradients are backward along the tracing path, and are passed to optimize k d , k orm , normals, and environment maps.where ω i is the ith sample drawn from density p. As the number of samples grows, the estimation variance reduces, but the computation increases. Multiple importance sampling [39] (MIS) is proposed to inhibit the computational consumption. When a function can be expressed as a multiplication of n functions, it draws n i samples ω i,j from n sampling distributions p i in turn. The MIS Monte Carlo estimator for the rendering equation is:EQUATIONwhere p i ∝ the i-th multiplication function of the integrated function of the rendering equation and W i is the the balance heuristic weighting function.As discussed in Sec. 3.1, the split sum approximation ignores the shadows and the indirect illumination. Nvdiffrecmc notes this and accounts for shadows using the Monte Carlo integration method. However, they give up continuously tracing the light rays attacking the surface of objects and treat them as zero illumination. It leads to bad performance in scenes with inter-reflections. Therefore, we propose the Ref-MC 2 method to trace the light rays continuously. When taking into account indirect illumination, based on [14] , the rendering equation can be expressed in this version:EQUATIONIn this equation, r (p, ω i ) represents the location of a surface point on the object surface hit by a ray cast from p in direction ω i for the first time. The corresponding Monte Carlo integration that considers the indirect illumination can then be expressed as:EQUATIONIt can be interpreted as that when the sampling ray is not blocked, it can be seen as direct illumination from environmental lighting. When the surface point of an object blocks the sampling ray, it needs to be continuously traced from this surface point. This continuous tracing is an iterated process of Monte Carlo integration. It is noteworthy that when continuous sampling at the blocking surface point it is unnecessary to consider the indirect illumination like calculating the outgoing radiance at initial point p. This is because the energy of light gradually degrades and the impact of light after twice reflections is negligible compared to the explosively increased computational load. However, the additional computational load of ray tracing at a depth of two is still huge. To reduce our computational load, we further propose to approximate the diffuse light and transfer the computations.In Disney PBR equation [24] , the diffuse term f d is:EQUATIONEQUATIONwhere c diff is the diffuse albedo of the material. In this equation, f is related to the direction of the incoming radiance, which can yield more realistic results. However, using this equation to calculate the diffuse light for indirect illumination creates unnecessary extra computation. Because the term f d is related to the normal n and the direction of rays ω o , we cannot get the diffuse map before deep ray tracing. The following term is to introduce the roughness for diffuse light to avoid too dark edges in extremely low grazing angles. However, using this equation to compute the diffuse light for indirect illumination introduces unnecessary extra computation. To reduce the burden, we can follow H and approximate it using Lambertian diffuse lighting.:EQUATIONWhere f ind d is the diffuse light of the indirect illumination. In this approximation, f ind d is independent on the incoming direction ω i and the outgoing direction ω o . Based on this, the diffuse part of indirect light can be simplified as:EQUATIONwhere L ind diff is the diffuse part of the indirect illumination. It no longer relates to the direction of the sampling ray, so we can present it via an MLP M . Compared to the diffuse part, the specular part is strongly related to the ray direction. It is a disadvantage that we cannot approximate the specular part like the diffuse part, but it is also an advantage that we only need to sample minor rays in a small specular lobe along the reflective directions. Therefore, we significantly reduce the computations of both the diffuse part and the specular part during the multi-times Monte Carlo Sampling.As seen in the equation 5, our indirect lighting highly depends on the geometry. However, the strong ambiguity of reflections makes it hard to directly learn high-quality disentangled shapes. Therefore, we flatten the joint learning process of both geometry and materials into separate learning of them and thus reduce the number of physical terms to be disentangled in each learning stage.We first utilize SDF-based architectures [48, 6, 18] to learn an initial geometry. The two branches of the diffuse and reflective networks well disambiguate the appearance with reflections and empower the SDF network to produce a high-quality geometry, as shown in Fig. 2 (a). However, this architecture encounters challenges of indirect illumination due to the expressive capacity of Integrated Positional Encoding (IPE). It performs well in general scenes but shows limitations in representing interreflections. Due to the capacity of Spheical Gaussians (SG) [50, 33] to represent radiance directions, we introduce SG encoding instead of IPE to enhance the expressive capacity of reflective MLP, as shown in Fig. 2 (b ). Besides, we directly parameterize diffuse appearance as SG coefficients which is suitable for objects with multiple reflective surfaces. The geometry comparison is shown in Fig. 2 .The learned SDF can be then converted into a surface mesh using the Marching Cubes [19] , which supports further tuning in the following inverse rendering pipeline. Different from Nvdiffrecmc which adopts DMTET for differentiable mesh optimization, we introduce Flexicubes [35] into our inverse rendering pipeline. Flexicubes use the SDF, weight, and the deformation for vertexes of the grid cells to extract the surface mesh using the DMC [29] . The mesh is converted into a differentiable representation that can be optimized based on gradients.The main objective is to minimize the photometric loss between rendered images and ground truth:EQUATION)where C is the rendering result, and C gt is the corresponding ground truth. Following previous work [56, 28, 9] , we adopt a smoothness loss for diffuse and material as regularization:EQUATIONEQUATIONwhere x surf represents the world coordinates of points on the object's surface. k d (x surf ) and k orm (x surf ) is the material of this point. ϵ is a randomly distributed vector of tiny deformations. A self-supervised loss is used to regulate the learned diffuse color:EQUATIONwhere C diff is the diffuse light from the object surface obtained during rendering. k diff (x surf ) is the diffuse light obtained by the MLP. Overall, the full loss function is:EQUATIONwhere ω 1 , ω 2 and ω 3 are three predefined scalars.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
implementation details: 
 Dataset. We construct a dataset of multiple reflective objects based on the existing single objects to evaluate the performance. Our dataset consists of 16 groups of object compositions, most of which contain indirect illumination between reflective objects. We render the composed objects with various environmental lighting in the Blender engine. Each group contains 300 images, with 200 for the training set and 100 for the test set.Experiment setup. We optimize the 3D model on 1 RTX 3090 GPU with 24G memory. We use the Adam optimizer for the material and the environment map with an initial learning rate of 0.03. The coefficients of loss function ω 1 , ω 2 , and ω 3 are set to 0.1, 0.05, and 1, respectively. The rate of Monte Carlo sampling is commonly set to 128 consistent with the setting in Nvdiffrecmc [9] .In this section, we compare our method with Nvdiffrec [28] , Nvdiffrecmc [9] , and Nefii [45] on our constructed reflective dataset, and the results are shown in Fig. 3 . Nvdiffrec achieves photo-realistic results in the metal balls but fails on the smooth and glossy table + horse. Besides, Nvdiffrec cannot Figure 3 : Qualitative comparison. The results of renderings, materials, and environment maps are presented. Note that, the material of Nefii contains only roughness without metalness. Our method achieves the best renderings with clear reflections, compared to other inverse rendering methods. Our method is also superior to others in the disentanglement of materials and environment maps. well disentangle the materials because it does not consider the shading. In contrast, Nvdiffrecmc performs well in material learning, while its rendering results are bad. These two methods also suffer from indirect illumination from the reflection of inner objects, leading to low-quality environment maps. Nefii is the recent work that considers the indirect illumination in the radiance field, but it tends to produce low-reflective results and performs badly in these high specular objects. Compared to these methods, our method can achieve both photo-realistic rendering results and well-disentangled material learning. We can handle highly specular objects, e.g., the metal table. The environment maps learned by our methods are also superior to others. In this section, we conduct an ablation study on the times of Monte Carlo Sampling, i.e.the depth of ray tracing. Multi-times sampling considers the indirect illumination, and thus successfully disentangles the environment light and inner reflective light. As shown in Fig. 4 , the environment maps learned by single-time sampling contain noise and shades that are actually the inner reflection.In contrast, multi-times sampling significantly releases the problem, producing a clearer environment map. Single-time sampling also undermines the rendering results, with darker inverted reflections.When the sampling ray is sheltered by objects during path tracing, the ray returns no light, resulting in a dark point. Multi-times sampling returns the reflective light of the sheltering points containing both diffuse and specular light. As a consequence, the rendering results of multi-times sampling show realistic appearance and reflections. Besides, it can be seen that results with 2-time sampling are comparable to results with 3-time sampling. Tab. 1 also quantitatively shows minimal improvement by deeper tracing. However, each additional sampling time induces a 4-times computation increment. Therefore, a 2-times sampling is a more cost-effective setting. Additionally, we also compare the efficiency between the sampling with and without acceleration. As shown in Tab. 1, our method can make the training speed twice as fast while keeping a comparable PSNR. The degree of acceleration is related to the complexity of geometry, and here we adopt the median in our dataset.In this section, we further explore the necessity of the initial geometry discussed in Sec. 3.3 and present our geometry reconstruction results. As shown in Fig. 5 , without initial geometry, the rendering results show bumpy surfaces with badly learned materials in the area of hollow shape.Because multi-times sampling amplifies the errors introduced by geometry, the learned materials are even worse than the single-time sampling. When applying a good-quality initialization, the network bypasses the ambiguity brought by the geometry. It helps our method learn well-disentangled materials and finally produces high-quality rendering results. Quantitative comparison in Tab. 1 also demonstrate the necessity of geometry initialization. In addition, we showcase a real-scene geometry reconstruction result of the NeRO [18] dataset in Fig. 6 . We further use Chamfer Distance to quantitatively evaluate our geometric quality and present the results in Tab. 2. The disentangled environment map and material empower our methods to relight and edit reconstructed objects in the downstream application. Our method can also be seamlessly plugged into the industrial pipeline. As shown in Fig. 7 , our method enables the arbitrary combination of reconstructed objects. We can easily edit their metalness, roughness, and albedo color by manipulating the learned materials. Due to the well-disentangled shading, our rendering results are natural and realistic in all five environment maps, even the point light in a dark environment. They also perform well after editing materials thanks to the well-learned material map. For example, after we increase the metalness and reduce the roughness of the teapot, the teapot clearly reflects the neighboring objects on its surface. In another case, where we extremely increase the reflectance and change the base color of the toaster, it accomplishes to reflect the scene and other objects. To showcase further applications of our method, we conduct experiments on real-world datasets captured by NeRO [18] and present the comparison results in Fig. 8 . It can be seen that Ref-MC 2 achieves more realistic results under different lighting conditions. 
conclusion and limitations: 
 In conclusion, our Ref-MC 2 introduces multi-times Monte Carlo sampling into the inverse rendering pipeline to model the indirect illumination. It improves the performance in scenes with complex interreflections. However, increment of sampling times significantly increases computational consumption and makes the pipeline highly geometry-sensitive. To solve the challenge of computational efficiency, based on the Lambert model, we simplify the BRDF for indirect lighting, which allows us to reduce the number of ray traces. To improve the geometry quality, we adopt SDF-based architecture to get an initial geometry and refine a design with Spherical Gaussian encoding for reflective objects. We further use Flexicubes to take the initial mesh into the differentiable rendering pipeline that learns disentangled materials. Our Ref-MC 2 still has several limitations. The major limitation is that the 2-time sampling cannot handle extremely reflective objects, e.g., mirrors, as the specular energy hardly degrades after reflections. Besides, the training time needs to further reduce in future time.NeurIPS Paper Checklist 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: NA. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have cited them. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix:  


Decision: This paper received four positive leaning reviews --- one 5 (borderline accept), one 6 (weak accept), and two 7s (accept). 

There was broad appreciation for the technical soundness of the work, clean presentation, high quality of the results, and the overall importance of the problem addressed herein (modeling interreflections in inverse rendering). 

There were some concerns raised around limited real experiments, and other technical issues which the authors addressed in a comprehensive rebuttal. There was substantial post-rebuttal discussion, following which, an accept consensus was reached. Hence, the decision to accept the paper. 

The authors are strongly urged to incorporate the reviewer feedback and include the additional comparisons / validation promised in the rebuttal when preparing the final camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper presents a method for learning disentangled scene representations from images. The proposed pipeline has two stages: the first recovers geometry using an SDF-based model to learn geometry from the images through differentiable rendering. The second applies differentiable ray tracing to predict material parameters and environment lighting. The proposed method differs from previous approaches in that it renders inter-reflections between objects which create indirect lighting rather than only reflections of the environment and shadows. This is shown to produce better results than existing methods in scenes where these inter-reflections are present.
strength_weakness: strength_weakness: It would have been nice to see some results on real data. The synthetic examples seem to show a clear improvement, but it would help to see the difference in a more practical setting.
questions: questions: Would it be possible to run on the real capture dataset from NeRD, similar to nvdiffrec?
limitations: limitations: The authors are clear about limitations, and I think they cover them well.
paper_summary: paper_summary: The authors propose an inverse rendering method that reconstructs the geometry, materials, and lighting of 3D objects from 2D images, effectively handling scenes with multiple reflections and inter-reflections. To address the high computation cost of Monte Carlo sampling, the authors propose a specularity-adaptive sampling strategy, significantly reducing the computational complexity. The authors also introduce a reflection-aware surface model to initialize the geometry and refine it during inverse rendering, to improve the inverse rendering reconstruction quality. Experiments show the proposed method outperforms other inverse rendering methods.
strength_weakness: strength_weakness: 1. The validation on real data is missing. The paper only shows its results on synthetic data. It would be better to test this work on challenging real data, such as the specular real data captured by NeRO.

2. The selected baselines are incomplete. There exists a strong baseline candidate, NeRO, which can also reconstruct the geometry and materials of specular objects. (Although its main focus is geometry reconstruction). Since it has released all of its code, it would be better if the authors could also compare this work with NeRO.

3. Need more analysis on the SG encoding. SG encoding looks like a very important part of the final quality(because it plays an important role in the geometry reconstruction). However, the authors only use a small part of Fig. 2 to showcase its effectiveness. More visual comparison and quantitative comparison would be appreciated. In the meanwhile, SG encoding seems to be very similar to [1], which was released on Arxiv last year. The authors should explain the difference between its SG encoding and [1]'s encoding. Since the authors didn't cite [1], I assume that SG encoding should be the author's original contribution.

4. The quantitative comparison is insufficient. More quantitative comparisons on the relighting results and reconstruction of BRDF and geometry should be included.

[1] SpecNeRF: Gaussian Directional Encoding for Specular Reflections
questions: questions: 1. Please show some results on real data with some baseline comparison.
2. Please provide more baseline comparisons and quantitative comparisons as discussed in the weakness section.
3. Please answer the question related to SG encoding.
limitations: limitations: The limitations have been discussed in the paper.
paper_summary: paper_summary: This paper proposes an inverse rendering method that handles multi-time Mante Carlo integration which models indirect illumination. It reduces the computational cost by pre-computing the diffuse map based on a Lambertian model. It also proposes to use spherical Gaussian encoding to improve the initial SDF reconstruction of scenes.
strength_weakness: strength_weakness: 1. Overclaim of novelty: multi-bounce MC integration has already been employed in earlier inverse rendering work [1]. On the other hand, the diffuse Fresnel term (Eq. 7) is not commonly used in recent inverse rendering works. Most related works (TensoIR, nvdiffrec/nvdiffrecmc) use exactly Eq 9, so the claim of transferring from Eq. 7 to Eq. 9 should not be counted as a contribution.

2. Continuing from (1), using a pre-computed diffuse map also prevents correct modeling of diffuse shadows, this is not discussed nor mentioned in the paper.

3. Inaccurate/incomplete description of the geometry module: IPE is used for encoding input positions to MLPs, while judging by the text (line 181) spherical Gaussian should be used for encoding viewing/reflecting directions. However, in this work, spherical Gaussian is actually used to replace IPE, which is very confusing. Also in Figure (2b) the entire Diffuse MLP is replaced with "SG" and there is no explanation of what this stands for. Also in the same figure, the term "SGE" is not explained (maybe spherical Gaussian encoding? Then what is the difference between SGE and SG?)

4. Weak evaluation: there is only a single table in the entire paper. Only PSNR and training time are reported. It is not clear how the PNSR metric is computed (is it novel-view synthesis? Is it relighting?). There is no evaluation of other intrinsic properties that are commonly evaluated in other inverse render papers, e.g. albedo, normal error, etc. The proposed spherical Gaussian encoding is not properly ablated - only qualitative comparison is presented.

5. Missing datasets: it would make the evaluation more complete if the method could evaluate more standard datasets (NeRFactor dataset, TensoIR dataset) where many other works have been evaluated. Also, there is no real-world dataset tested, making it questionable whether the proposed algorithm can work in real-world scenarios.

Factual errors: GGX has nothing to do with the diffuse term used in Disney BRDF. GGX is a microfacet normal distribution function that models specifically the specular part of the Disney BRDF. The diffuse Fresnel term in Disney BRDF is invented by the original Disney BRDF paper and is often omitted in real-time rendering engines and replaced with the Lambertian model.

[1] Mai et. al. Neural Microfacet Fields for Inverse Rendering, ICCV 2023
questions: questions: Please see the weaknesses.
limitations: limitations: A crucial limitation, i.e. diffuse shadows are not handled by the proposed method, are omitted and not properly discussed.
paper_summary: paper_summary: This paper proposes a comprehensive inverse rendering method that incorporates indirect illumination to enhance the decomposition quality of environment lighting and BRDF materials. Specifically, this work uses multi-time Monte Carlo integration to model light transport and devises algorithms to accelerate computation by precomputing the diffuse term and leveraging an SDF-based geometry for initialization. Both qualitative and quantitative results are promising, demonstrating realistic shadows on glossy surfaces and higher PSNR values.
strength_weakness: strength_weakness: 1. The optimization time is relatively long compared to other methods. 
2. Due to computational limitations, the number of bounces is restricted, thus limiting the ability to model shiny surfaces and confining the results to glossy surfaces.
questions: questions: In general I enjoyed reading this paper. The content is well written and clear to me. Therefore, I’m on the positive side of this submission. A few questions regarding the technical details:
1. Will the geometry obtained from Sec. 3.3 be optimized in inverse rendering? The writeup suggests this (Lines 189-190), but I didn't find any experiments addressing it. I am curious about potential geometry improvements.
2. Regarding the material editing mentioned in Section 4.5, I have some questions about its implementation. It seems that material properties are represented as MLP. How do you intuitively edit different material properties, such as roughness? Or are these properties explicitly exported and stored as UV maps?
3. What does “orm” stand for in k_orm?
limitations: limitations: Limitations and potential negative social impact are well discussed.

Comments: title: Description of The Geometry
Analysis on the SG encoding
Evaluation of the albedo and the k_orm
Details about the representation of the material
comment: Thanks for your reply and valuable comments.  
1. To better capture the geometry of reflective objects, we employed the structure shown in Fig. 2(b), which differs from the **representation** used by SpecNeRF. The Gaussian Directional Encoding in SpecNeRF utilizes 3D Gaussians, and its representation is as follows:
$$\mathcal{G}_i(\mathbf{x})=\exp \left(-\left\|\mathcal{Q}\left(\mathbf{x}-\boldsymbol{\mu}_i ; \mathbf{q}_i\right) \odot \boldsymbol{\sigma}_i^{-1}\right\|_2^2\right),$$ 
where $\boldsymbol{\mu}_i $ represents position, $\boldsymbol{\sigma}_i $ represents  scale, $\mathbf{q}_i $ represents quaternion rotation.  
In contrast, our SG encoding uses spherical Gaussians, and its representation is as follows:
$$G(\boldsymbol{\nu} ; \boldsymbol{\xi}, \lambda, \boldsymbol{\mu})=\boldsymbol{\mu} e^{\lambda(\boldsymbol{\nu} \cdot \boldsymbol{\xi}-1)},$$
where  $\boldsymbol{\xi}$ is the lobe axis, $ \lambda$ is the lobe sharpness, and $\boldsymbol{\mu}$ is the lobe amplitude.   

2. We used two environment lights Golden Bay and Limpopo Golf Course from Poly Haven [1] and Materials as ground truth to generate the relighting dataset. We report the  PSNR results as follows:
|   | Golden Bay  | Limpopo Golf Poly  |   |
|---|---|---|---|
|  Ours |  __19.38__ |  __19.72__ |    |
|   nvdiffrecmc| 16.12  | 16.47  |   |
|  nvdiffrec |   15.90 |  16.04 |  |


We apologize for overlooking your comments on quantitative results. We present the quantitative metrics results (discussion with reviewer UfeL) as follows for you convenience:

**Evaluation of the albedo and the $k_{orm}$**  

|  PSNR ($\uparrow$) | albedo  | $k_{orm}$  | normal  |   |
|---|---|---|---|---|
|  Ours |  __21.50__ |  18.56 |  __28.58__ |   |
|   nvdiffrecmc| 20.04  | **18.67**  | 26.09  |   |
|  nvdiffrec |   17.58 |  18.64 |  18.31 | |

**Evaluation of the reconstructed geometry**  
|  Chamfer Distance ($\downarrow$) | coral  | bear  |  materials |   |
|---|---|---|---|---|
|  NeRO |  0.13 | 0.11  | 0.0057  |   |
| Ours  |  **0.13** |  **0.10** |  **0.0030** |   |

[1] Poly Haven. https://polyhaven.com/
Thanks for the authors' replies, which solve some of my questions, but some questions still remain:

1. As I said in the w3, it seems that SG encoding is very important to the result quality. Fig.5 of the rebuttal file indicates the same thing. I said in my review that I wanted to know the difference between the SG encoding used in the [1] and the authors' methods. I am not sure if the SG encoding can be considered as the original contribution of this work. But the authors did not fully address this in their response.

[1] SpecNeRF: Gaussian Directional Encoding for Specular Reflections

2. I mentioned in the weakness section that the quantitative comparison of this paper is insufficient. I am not sure why the author didn't respond to this concern in the rebuttal reply for me.  I noticed that Reviewer UfeL had similar concerns, which the authors addressed in their rebuttal. However, it seems that a quantitative comparison for relighting is still missing.



Given these unresolved issues, I don't plan to raise my score.
Thank you for the instant reply. The "precompute" term here indeed may cause confusion and we will consider removing it in the final version. We will modify the corresponding part in the final version based on the discussion to make the sentences clearer. Thanks again for the instructive comments.
I am mainly confused about the statement "we can pre-compute it via an MLP" (line 165). Judging by lines 199-200, it seems that this diffuse light MLP is supervised solely by the diffuse color prediction from the model. So it seems that it has nothing to do with "pre-computation". If this is the case, then the MLP is just self-supervised and the term "pre-compute" is causing unnecessary confusion.

On the other hand, precomputation of the light integral in Eq. 10 can be done by convolving the stored environment map tensor with the clamped cosine lobe - this operation is quite efficient during training. I was thinking along this way as this is also used in nvdiffrec.

Overall, I would recommend the authors to make the following changes in the final version:

1. The diffuse light MLP is used to predict only light beyond one bounce. This is hinted in Fig. 1, however, the transition around lines 153-154 is a bit hard to follow. It should be stated more clearly by changing the notation in equations to denote that diffuse light in this section is for indirect illumination only. Lines 154-162 are also unnecessary as the diffuse term is not really a part of GGX model, and most existing works (nvdiffrecmc, for example) already use the Lambertian model for the diffuse appearance

2. Describe accurately that the diffuse light MLP is self-supervised by the diffuse appearance prediction of primary rays, and remove the "pre-compute" term that might causes confusion for readers.
Thank you for your reply and appreciation. We will include the disscussion in the final paper.
Dear authors,

Thanks for clarifying my questions! I have no further questions and overall I really enjoyed reading this paper! I will keep my score as a Weak accept.
1.
Thank you very much for your positive reply. We sincerely appreciate you pointing out that we overlooked an excellent work, NMF. In the final version of our paper, we will include the comparison between our work and NMF.

2.
Your statement is correct. Indeed, for diffuse color, we only trace one bounce, while for specular color, we trace more than one bounce. The light that reaches the eyes is accounted for by ray tracing, including shadows.

3.
Similar to what is described in equation 10, the diffuse light at point $p$ can be represented as $f(p)$, which is a function dependent solely on the position of the object. Empirically, we can represent it in the **same** form as $k_{d}$ or $k_{orm}$. It is the **same** as in nvdiffrec/nvdiffrecmc and is stored as a **mipmap**.

4. As for the precomputation in Eq.10, we initially had similar concerns, which led us to conduct the following three sets of experiments：

+  At the beginning of the optimization, we set depth=1 to quickly learn a rough diffuse light map.
+ Set depth=2, and at the beginning of the experiment, do not use the diffuse light map structure. Once the optimization reaches a certain level, then applying the diffuse light map.
+  Directly optimize using an unknown diffuse light.

Our results indicate that directly using the diffuse map can achieve the same effectiveness as the previous two approaches, and the unknown diffuse light map can be optimized quickly. We would include this in our final paper.

Once again, thank you for your feedback and valuable comments.
Thanks for the prompt reply. 
1. I am convinced by the argument on the differences between NMF and the proposed method. Please include such discussion in the final version of the paper.

2-3. Regarding diffuse light, now I see there was a misunderstanding on my side. I thought the diffuse light (Eq. 7-10) refers to the outgoing radiance towards the camera/eye, but it actually describes the incoming radiance towards the intersection between the primary (camera) ray and the object surface. Eq. 7-10 does not consider the shadowing effect, but Eq. 6 considers the shadowing effect for the diffuse part of the primary ray via ray tracing. So for diffuse color, you only trace one bounce, while for specular color, you trace more than one bounce (up to 3 in the experiments). Please correct me if I am wrong in any of the above statements.

4. The results look good, I am also convinced of this part

I am now inclined to accept the paper, though I still have some additional questions regarding the diffuse light:

1. How is the environment map represented? Is it the same as nvdiffrec/nvdiffrecmc, i.e. environment map is stored as a mipmap/tensor?

2. How do you "precompute" Eq. 10 exactly? You mentioned that the whole term (or just the light integration part?) is represented as an MLP. However, the environment light is not known prior. You need to optimize the environment map during the training process as I understand. Then how do you precompute Eq. 10 and store it in an MLP, as the environment map is changing during the training process?
Thank you very much for your comments.  
1. 
Based on Section 4.5 of NMF, it is indeed that our rendering method is quite similar when our depth is set to 2. However, there are still significant differences in our work：

+ NMF uses a density field to represent geometry, whereas we use a triangle surface mesh. Since our representation method is consistent with the widely adopted approach in the industry, it allows us to easily leverage hardware-accelerated ray tracing.

+ NMF mentions in Section 6, "It also does not handle interreflections very well, since the number of secondary bounces is limited." However, continuing from (1), the number of secondary bounces in our approach is not limited.

+ Additionally, our method allows for easily increasing the ray tracing depth. A higher tracing depth is more effective in scenes where light repeatedly bounces between two shiny objects.

Thank you very much for pointing out that we overlooked this excellent work. We will conduct more experiments in the future to compare the effectiveness of our approaches. Additionally, we will validate our method in challenging scenes where light repeatedly bounces between two shiny objects.

2. 
We use the term *secondary* to denote a ray bounced from a surface position $p$, similar to NMF. For the secondary color, we use equation 10 to calculate the diffuse color. Furthermore, the diffuse light at point $p$ can be expressed using an equation $f(p)$, which is why we can represent it using an MLP. However, for light directly entering the eye, we still use Monte Carlo sampling to compute the diffuse light rather than relying on this MLP.

3. 
Our directly viewed diffuse light includes shadows, and we use it to supervise the diffuse light of our secondary rays. This essentially means that we have baked the shadows into the diffuse light, which is a key difference from the diffuse light used in nvdiffrec. Although our focus is on reflective objects, we will also conduct additional experiments on non-reflective datasets, similar to TensoIR, in the future. We greatly appreciate your feedback regarding the datasets.

4. 
We evaluated the Chamfer distance, and the results are shown in the table below:
|  Chamfer Distance ($\downarrow$) | coral  | bear  |  materials |   |
|---|---|---|---|---|
|  NeRO |  0.13 | 0.11  | 0.0057  |   |
| Ours  |  0.13 |  0.10 |  0.0030 |   |

It can be seen that on the bear and coral datasets, our geometry performs just as well as NeRO's, but on the materials dataset, our results are better.

5. 
Our "acc" means the diffuse map, and "w/o acc" indicates that for the diffuse color of the *secondary* rays, we still use ray tracing directly, similar to how it's done for the *primary* rays.
Thanks for the reply. I have additional questions regarding the author's arguments:

1. The author argues that in Neural Microfacet Fields (NMF for short), Eq. 15 uses the environment light map which is different from your design. I believe that here you mean $E(\mathbf{p})$ which is termed **irradiance environment map**. The term might be misleading but it essentially records pre-integrated cosine-weighted incoming light over the hemisphere for each possible normal direction. This is (almost) equivalent to the pre-convolved diffuse light map in nvdiffrec (as well as this work if I understand correctly). The only difference would be that they further compress the irradiance map into SH representation, while in nvdiffrec the pre-convolved diffuse light map is stored in a 6x512x512 cubemap. On the other hand, NMF uses MC sampling to compute the rendering equation, and they trace 2 bounces for indirect illumination. Please check Sec. 4.5 and Tab. 1 of the paper. Fundamentally it uses the same MC integration with mult-bounce (2) sampling to model indirect illumination.

2. Regarding the diffuse map, the paper states that "we can precompute it via an MLP" (line 165), but how? What is the difference between your diffuse map and that of nvdiffrec, except for MLP vs. explicit tensor?

3. Ignoring diffuse shadow would not impact the final results, most likely because the tested scenes are specular. This is why I asked the author to test more datasets, especially the TensoIR dataset where objects are not that specular (but also not fully diffuse)

4. Also, NeRO's glossy-real dataset contains ground-truth geometry. It would make the evaluation more complete if the authors can provide a quantitative comparison on this glossy-real dataset against NeRO.

5. I do not find anything related to diffuse map ablation in Tab. 1, can you elaborate on which entry in the table specifically reflects the effectiveness of diffuse map? Also, what does the "w.o Acc." entry do?
Our SG(spherical Gaussian) lobes were not used for encoding viewing/reflecting directions. Similar to the method in [1], the feature output by the sdf network is combined with a set of SG lobes to compute the color values. [2] also demonstrated the effectiveness of this approach. 
However, since we are dealing with more complex reflective objects, we adopted two different structures to better distinguish between diffuse light and specular light. The SGE used for specular light is similar to SG but operates in a higher dimension. We used higher dimensions to represent specular reflection because specular reflections are view-dependent. 

[1] Lior et. al. Bakedsdf: Meshing neural sdfs for real-time view synthesis, SIGGRAPH 2023

[2] Christian et. al. Binary opacity grids: Capturing fine geometric detail for mesh-based view synthesis, TOG 2024
Inspired by bakedSDF[1], we replaced the MLP with a set of SGs (spherical Gaussians) to directly obtain diffuse light. [2] also demonstrated the effectiveness of this approach. Given our focus on reflective objects, we employed different structures to more effectively distinguish between diffuse light and specular light. The SGE used for specular light is similar to SG but operates in a higher dimensionality. Since specular light is significantly influenced by the viewing angle and geometry, it is more complex than diffuse light, requiring more information for its representation.

[1] Lior et. al. Bakedsdf: Meshing neural sdfs for real-time view synthesis, SIGGRAPH 2023

[2] Christian et. al. Binary opacity grids: Capturing fine geometric detail for mesh-based view synthesis, TOG 2024
We computed the PSNR metric on our __table horse dataset__. The results are shown in the table below:
|  PSNR | albedo  | $k_{orm}$  | normal  |   |
|---|---|---|---|---|
|  Ours |  __21.50__ |  18.56 |  __28.58__ |   |
|   nvdiffrecmc| 20.04  | 18.67  | 26.09  |   |
|  nvdiffrec |   17.58 |  18.64 |  18.31 | |
Inspired by nvdiffrec's excellent work, our material learning is divided into two stage. The first stage, we use an MLP to represent the material, and the second stage, we use xatlas to generate  texture coordinates automaticly. Then we sample the MLP to initialize 2D textures. Subsequently, the textures are optimized based on the gradients provided by the nvdiffrast. We experimentally compared  two materials representations.  Using mlp to represent materials can quickly converge，and the subsequent optimization with 2D textures can continue to improve the rendering and relighting results.