Inverse Rendering using Multi-Bounce Path Tracing and Reservoir Sampling
Authors: Yuxin Dai
Qi Wang
Jingsen Zhu
Dianbing Xi
Yuchi Huo
Chen Qian
Ying He
Venue: ICLR 2025 Poster (2025)
Keywords: Inverse Rendering
Material Decomposition
3D reconstruction
relighting
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present a novel two-stage inverse rendering framework that jointly reconstructs and optimizes explicit geometry, materials, and light- ing from multi-view images.

Abstract: We introduce MIRReS, a novel two-stage inverse rendering framework that
jointly reconstructs and optimizes explicit geometry, materials, and lighting
from multi-view images. Unlike previous methods that rely on implicit irradiance fields or oversimplified ray tracing, our method begins with an initial
stage that extracts an explicit triangular mesh. In the second stage, we refine this representation using a physically-based inverse rendering model
with multi-bounce path tracing and Monte Carlo integration. This enables our method to accurately estimate indirect illumination effects, including self-shadowing and internal reflections, leading to a more precise
intrinsic decomposition of shape, material, and lighting. To address the
noise issue in Monte Carlo integration, we incorporate reservoir sampling,
improving convergence and enabling efficient gradient-based optimization
with low sample counts. Through both qualitative and quantitative assessments across various scenarios, especially those with complex shadows,
we demonstrate that our method achieves state-of-the-art decomposition
performance. Furthermore, our optimized explicit geometry seamlessly
integrates with modern graphics engines supporting downstream applications such as scene editing, relighting, and material editing.

Published as a conference paper at ICLR 2025
INVERSE RENDERING USING MULTI-BOUNCE PATH TRACING AND RESERVOIR SAMPLING
Yuxin Dail; : Qi Wang?; Jingsen Zhu²; Dianbing Xi?, Yuchi Huo?,
Chen Qian Ying Helt
LS-Lab, Nanyang Technological University SSate Key Lab of CAD&CG, Zhejiang University CCornell University "SenseTime Research and Aot
ABSTRACT
We introduce MIRReS, novel two-stage inverse rendering framework that jointly reconstructs and optimizes explicit geometry, materials, and lighting from multi-view images. Unlike previous methods that rely on implicit irra- diance fields or oversimplified ray tracing, our method begins with an initial stage that extracts an explicit triangular mesh. In the second stage, re- fine this representation using a physically-based inverse rendering model with multi-bounce path tracing and Monte Carlo integration. This en- ables our method to accurately estimate indirect illumination effects, in- cluding self-shadowing and internal reflections, leading to more precise intrinsic decomposition of shape, material, and lighting. To address the noise issue in Monte Carlo integration, we incorporate reservoir sampling, improving convergence and enabling efficient gradient-based based optimization with low sample counts. Through both qualitative and quantitative as- sessments across various scenarios, especially those with complex shadows, we demonstrate that our method achieves state-of-the-ate decomposition performance. Furthermore our oplimized explicit geomelry geamlessly integrates with modern graphics engines supporting downstream applica- tions such scene editing, relighting, and material editing. Project page: httts://brabbitdouse github. io/MIRReS/
INTRODUCTION
Inverse rendering, the process of decomposing multi-view images into goometry, material and illumination, is a long-standing challenge in computer graphics and computer vision. The task is particularly ill-posed due to the inherent ambiguity in finding solutions to repro- duce the observed image, especially when illumination conditions are unconstrained. Recent advancements in neural radiance fields (NeRFs) Mildenhall et al. (2021) and neural implicit surfaces (such as signed distance fields (SDFs)) Wang et al. (2021 0: Yariv et al. (2021) have inspired several works [oss et al. (2021a b Srinivasan et al. (2021): : ZZhng et al. (2021b 2022 2021a) that employ NeRF or SDF for scene representation. These methods often utilize auxiliary MLPs to predict materials or illumination. However, these ML.P-based methods often suffer from limited network capacity and slow convergence, resulting in dis- torted geometries and inaccurate materials. In contrast, TensolR Jin et al. (2023) employs compact
representation Chen et al. 2022 with explicit second-bounce ray marching for more accurate indirect illumination. Despite their remarkable results, these implicit methods still have two inherent disadvantages: Firstly, they represent geom- etry as implicit density fields rather than triangular meshes, restricting their application in the graphics industry where triangular meshes are the most widely accepted digital as- sets. Secondly, while some methods do sample secondary rays for indirect illumination, they rely on radiance fields instead of physically-basse based rendering (PBR) to obtain second- second-bounce radiance, thus lacking the physical constraints necessary for precise material optimization.
"quual contribution. "Corresponding author.

---

Published as a conference paper at ICLR 2025
MILKRS
Applicatonn
Input views
Albedo, roughmess and geometry
Novel view, relighting and scmee editing
Figure 1: Capabilities of MIRRes. Given multi-view images 3D scene, our method jointly optimizes geometry, materials and lighting to achieve high-quality reconstructions. This facilitates applications including novel view synthesis, relighting, and scene editing,
To address these challenges, we propose M mesh-based two-stage inverse render- ing framework that decomposes geometry, PBR materials, and illumination from multi-view images using physically-based mulli-bounce path tracing. Our framework directly optimizes triangle meshes to enable applications such scene editing, relighting, and material editing that are compatible with modern graphics engines and CAD software (see Fig. D The ex- plicit. use of triangular mesh representation also facilitates efficient path tracing with modern graphics hardware, making it possible to compute multi-bounce path tracing within accept- able timeframes. However, existing mesh-based neural inverse rendering methods, such as NVdiffree- MC Hasselgren et al. (2022), often suffer from unstable geometry optimizations. These instabilities lead artifacts such as holes and self-intersecting faces, which compro- mise the accuracy of ray-mesh interscctions and make path tracing intractable, especially in multi-bounce cases where errors will accumulate recursively.
In particular, our paper consists of three key technologies: (a) Mesh optimization. Our approach incorporates a two-stage geometry optimization and refinement process (Sec- tion 3). (b) Indirect illumination estimation. We explicitly conduct physically-based multi-bounce path tracing with Monte Carlo integration, which enforces strict physical con- straints materials, thereby improving the accuracy of indirect illumination and relighting results, (c) Convergence acceleration. Recognizing that the Monte Carlo estimator ne- cessitates substantial sample count to maintain optimization accuracy, which inherently slows convergence, we leverage reservoir sampling Bitterli et al. (2020) for direct illumina- tion, which reduces the required sample count while maintaining low noise levels. Meanwhile, coupled with a denoiser inspired by NVdiffrec-MC Hasselgren et al. (2022). our framework achieves a considerable convergence acceleration. To summarize, our contributions include:
1. We propose MIRReS, physically-based inverse rendering framework that jointly optimizes the geometry, materials and lighting from multi-view input images, achieving stato-of-the-art results in both decomposition and relighting.
2. Our method utilizes multi-boure path tracing to provide more accurate estimation indirect illumination, successfully achieving promising decomposition results in the challenging highly-shadowed scenes.
3. Our method utilizes Reservotr-base based Spatio- Temporal Importance Resampling for direct illumination, which can greatly reduce the required sample counts and accel- erate the rendering process.
RELATED WORK
2.1
NEURAL SCENE REPRESENTATIONS
As an alternative to traditional representations (e-g. mesh, point clouds, volumes, elc. ), neural representations have achieved great success in novel view synthesis and 3D modeling. Neural radiance fields (NeRF) Mildenhall et al. (2021) uses MLPs to implicitly encode
Name taken from "Multi-bounce Inverse Rendering using Reservoir Sampling".
2

---

Published as a conference paper at ICLR 2025
scene neural field of volumetric density and RGB radiance values, and uses volume rendering to produce promising novel view synthesis results. To address the limited ex- pression ability and slow speed of the vanilla MLP representation, follow-up works leverage voxels Fridovich-Keil et al. (2022); ; Sun et al.] (2022), hashgrids Müller et al. (2022), ten- sors Chen et al. (2022). polygon rasterization Chen et al. (2023), adaptive shells Wang et al. (2023b), etc. to achieve high-fidelity rendering result and real-time rendering speed. In ad- dition to NeRF-based methods, 3D Gaussian Splatting (3DGS) Kerbl et al. (2023) proposes to use point-based 3D Gaussians to represent a seene, enabling fast rendering speed due to the utilization of rasterization pipeline. stimulating follow-up works on quality improvement or many other applications Yu el. al. (2024); Tang et al. (2023a): Liang el. al. (2023). Some
other works also seek to combine neural and Traditional representations, leveraging both strengths. For example, NeRF2Mesh Tang et. al. (2023b) designs two-stage reconstruction pipeline, which refines the textured mesh surface extracted from the NeRF density field to obtain delicate textured mesh recovery. In this work, we also employ two-stage geometry optimization stratogy combining neural implicit representation and triangle meshes.
2.2 INVERSE RENDERING
The task of inverse rendering aims to estimate the underlying geometry, material and light- ing from single or multi-view input images. Due to inherent ambiguity between the decom- posed properties and the input images, inverse rendering is an extremely ill-posed problem. Some methods simply the problem under constrained assumptions,, such as controllable
lights Bi ct al. (2020): Luan al. (2021); Nam al. (2018). Physically-base basod mothods Li ct al. [2018: Zhang et al.] (2020); Jakob et al. (2022) Loubet et al. (2019) account for global illumnation effects via differentiable light transports and Monte- Carlo path tracing.
The emergence of neural representations has stimulated abundant neural inverse rendering frameworks Jin et al. (2023): : Zhang et al. (2021b; 2023), which utilize neural fields as the positional functions of material and geometry properties, along with lighting as trainable parameters such as Spherical Harmonics (SH), Spherical Gaussian (SG), environment maps, etc., and then jointly optimize them by rendering loss via diffcrentiable rendering. However, neural fields also face challenges, such as low expressive capacity and high computational overhead caused by ray marching. Meanwhile, other methods also utilize explicit geome- try representations, such as mesh Munkberg et al.] (2022); Hasselgren et al. (2022) or 3D Gaussian Kerbl et al. (2023); ; Liang al.] (2023 Table lists representative recent in- verse rendering methods and compares their settings with our method. Our method is the first inverse rendering framework that supports multi-bounce raytracing estimate indirect lighting more accurately.
Table 1: Comparison between cxisting inverse rendering methods and our method.
Method
Geometry
Lighting
Indirect Lighting
Sampling
NeRFactor
Implicit
Enviromment
N/A
TensoIR
Implicit
Ray tracing
Importance sampling Importance sampling
NVdiffre-MMC
Mesh
Ray tracing
NeILF- +++
Implicit
Implicit
Stratified sampling N/A
GS-IR
3DGS
Split-sum
Ours
Mesh
Multi-bounce Path tracing
Reservoir sampling
3 METHOD OVERVIEW
This section outlines our proposed inverse rendering framework which utilizes multi-bounce raytracing and reservoir sampling. Given multi-view image captures of an object under unknown environment lighting conditions, along with their corresponding camera poses, our method jointly reconstruets the geometry, spatially-waryingg materials, and enviromment lighting. Unlike most recent approaches that use neural implicit geometry representations (e.g. NeRF or neural SDF), our method adopts triangle mesh for an explicit geometry representation. This choice is crucial because optimizing mesh topology directly from multi- view images requires robust initialization. Inspired by NeRF2Mesh Tang et al. (023b)) we
3

---

Published as a conference paper at ICLR 2025
Taiishle Parsmnoo
Coloe Baalance Field
RP Renderel image
PBR Benazzl irage
RIP Render
cake
Blakk Exanding
Mat Net
- saepligg
Offer
Intwary Rie
Irpus
Nex52
EarMap
Lyex
Tesenictinn y
rampãez
Ply Plysical Based Reader
Cyek
Figure 2: Overview of our inverse rendering pipeline. Our two-stage process starts with the extraction of coarse mesh from radiance field, followed by joint optimization of geomelry, material, and lighting using physically-based rendering techniques. Key compo- nents such as multi-bounce path tracing, Monte Carlo integration, and reservoir sampling are highlighted to showcase their roles in enhancing the accuracy and efficiency of the re- construction process.
employ a two-stage training process: initially, we train neural radiance and signed distance field extract coarse mesh from the input images. The second stage, which forms the core of our pipeline, simultaneously optimizes scene material and lighting through our physically- based rendering and refines the mesh geometry. Fig. provides an overview of our pipeline.
STAGE RADIANCE FIELD AND COARSE MESH ACQUISITION
The main purpose of this stage is to initialize radiance field and geometry that will facilitate optimization stage 2. We use an efficient off-the-shelf NeRF-baased method (InstantNGP Müller et al. 2022 to train neural radiance field and neural density field, with position x and view direction as inputs, density a and radiance color с as outputs:
o, f = (x)x,,
Fx(x,
mesh from the density field F, using the Marching Cubes algorithm, the NeRF-based volumetric representation inherently lacks geometric constraints. This limitation leads to the extraction of geometry with imperfections such as holes and sawtooth patterns, which adversely affect optimization in stage 2. Therefore, we additionally use NeuS2 Wang al. (2023a) & SOTA SDF-based reconstruction method extract the coarse mesh Mcoarse {V.F} (where V denotes vertices and F denotes faces). The density field F, is then discarded, while F, continues be optimized in stage for geometry refinement.
Although
extract
3.2 STAGE 2: MESH REFINEMENT AND INTRINSIC DECOMPOSITION
The goal of this stage is to refine the coarse mesh Meoarse obtained from stage 1 and to decompose material and environment lighting parameters into a fine mesh Mäne-
Rendering: Starting with the extracted mesh M and camera ray r(t) = + td from origin o in direction d, we firstly use nvdiffrast Laine et al. (2020) to compute the ray-mesh intersection:
intersect(r,M)
(2)
This differentiable process allows for gradient descent optimization and differentiable ren- dering. Our pipcline jointly employs two rendering methods: radiance field rendering and physically-base surface rendering, which will be used by the tasks of mesh refinement and intrinsic decomposition, respectively.
Radiance field rendering: With the appearance field F, from the NeRF network in first stage, produce the rendering result directly from the intersected surface point x.
the

---

Published as a conference paper at ICLR 2025
Unlike NeRF which uses ray marching and volume rendering, directly feed the intersected surface point x from Eq. (2) into the appearance field to produce the ray color CRF(T):
CRE(F) = Fx d.
Physically-baas basellybbbdddda surface rendering: Given the surface shading point x, we render the shading color by the rendering equation Kajiya (1986), which is an integral over the upper hemisphere at x:
CPBR(r) = / (xxx, w. d. m) n)dwi,
(4)
where Lx(x, wi) denotes the incident lighting from direction m denotes the spatially- varying material parameters x, fr denotes the bidirectional reflectance distribution func- tion (BRDF), and n denotes the surface normal at X.
In the context of multi-bounce path tracing, the incident light (x(x, wa) is the composition of direct light, which is the environment illumination in this work, and the indirect light:
Direct environment illumination (Section
4.1
Indirect light (Section 4.2
L,(x, wi) = V(x,w;) wi) (wn)(wr)
+ Lind wi)
(5)
Direct Light Visibility
Mesh refinement: As mentioned in Section [ implicit geometry representation may introduce bias and inaccuracy in indirect illumination estimation. Therefore, we opt to optimize a triangular mesh to represent scene geometry. DMTet Shen et al. (2021) used by NVdiffrec-MC Hassclgren et al. (2022) is existing approach for direct mesh optimization, but it suffers from topological inconsistencies and geometric instability, which in turn affects the accuracy of path tracing. Instead, we opt for stable and continuous optimization approach. Inspired by NeRF2Mesh Tang et al. (2023b), we assign a trainable offset Av; to each mesh vertex V, E V to refine the geometry, and optimize them along with the appearance fields F. by minimizing the loss of radiance field rendering. Specifically, given a camera with known intrinsic and extrinsic parameters and its reference image Teet (s), we use radiance field rendering (Fq. (3) to produce an image (R)(s), and then optimize Av; and the parameters of Fo by minimizing the L2 loss between the rendering result and reference image:
CrF R| [s)
(6)
Similarly, we also use physically-based surface rendering (Eq. to produce an image (s). Owing to the differentiability of nvdiffrast's ray-mesh intersection, the gradient of the L2 loss between the PBR rendering result and the reference image can be back- propagated to Av;
(7) In summary, Av; is jointly optimized by LRF and LPEr in stage 2, which refines the geometry from Meoarse to Mane. Since Av; is continually changing during the optimization and does not change the face topology of the mesh, our mesh refinement approach ensures geometry stability and ensures the feasibility of multi-bounce path tracing.
LPBR [
Intrinsic decomposition: Based on the mesh geometry, we now describe how we repre- sent and optimize the spatially-varying material and enviromment lighting.
We adopt the physically-based BRDF model from Disney Burley & Studios (012)), which requires material parameters: diffuse albedo and roughness. We encode the spatially- varying material parameters of the scene using neural field Fm. which predicts the material parameters m given an input position x: m = Fm (x). We implement Fm as a small MLP with multi-resolution hashgrid Müller et al. (2022) The predicted m is a 4-channel vector, which will be further split by channel into the diffuse albedo (3) and roughness (1).
Following NVdiffrec-MC Hasselgren et al. (2022) represent the environment lighting as an HDR environment map with 256x512 pixels, where all pixel colors are trainable parameters.
5

---

Published as a conference paper at ICLR 2025
4 DIRECT AND INDIRECT LIGHTING
As described in Eq. the incident lighting in the rendering equation is divided into two components: direct and indirect. These estimated through reservoir sampling (Sec- tion 4.1 and multi-bounce raytracing (Section 4.2 respectively.
4.1 DIRECT LIGHTING USING RESERVOIR SAMPLING
Substituting the direct light part of Eq. into Eq. direct light:
(4 gives the rendering equation of
f(ar)
N
f(w))
CPER(T) = / V(x, wi) Leno(ws) for (x, wi d, m)(w; n) dws
(8)
N
Pdsr (wi)
In the second part of the equation, we estimate the integral with Monte Carlo integration, with samples drawn from a particular distribution Pai (wi) where only V(x, w)) and (ws) unknown, while the remaining terms are either analytically determined or train- able parameters. Therefore, determining V(x, w;) and finding an appropriate pair (w;) is key to the estimation.
Visibility estimation: Unlike implicit-based methods like TensoIR Jin et al. 2023 which estimate the visibility function V(x, w;) by the transmittance function in volume render- ing, our method can directly estimate ((x,w) by a ray-mesh intersection. We implement our ray-mesh intersection algorithm with customized CUDA kernels, which signifieantly improves computational efficiency and enables us to increase the sample count for more precise and low-variance estimation. Please refer to the appendix for implementation details.
Reservoir sampling: To reduce the variance of direct lighting estimation (i.e. to reduce rendcring noise, see Fig. 3), we utilize reservoir sam nampling Bitterli al. (2020), an advanced resampled importance sampling (RIS) technique Talbot (2005) to determine the appropriate (w:) According to the multiple importance sampling (MIS) theory Veach & Guibas (1995), the variance of the Monte Carlo estimator will reduce when (dir( is closer to the integrand [(w).. A distribution Pai (w:) x (ws fr (x, wis d, m) will be an ideal solution, but it is impossible to analytically sample from such a probability distribution as it does not have a closed-form expression.
Reservoir
w/o Reservoir To address this, Resampled Importance Sampling (RIS) Tal- bot (2005) provides more advanced technique to approximate the target distribution Pair (w;) x L ff. Firstly, an easy-to- sample proposal distribution (w;) is chosen, from which samples (wi,..., Wm are generated as candidates. In our implementation, we choose qair (w;) Lenu (w;). which can be directly sampled from our environment map. Then, we evalu- ale pai (w:) L(w:) f(xx) for each candidate sample wi S,
Figure Comparison on rendering noise with or without reservoir sam- pling with sample count
to it. Finally, we resam- ple from S according to the weight 7/1 Weighted-averagiin the sampled results after repeating for N times forms an N N-sample RIS estimator of
and assign a weight
%
peir(ws) qas-(w:)
RIS sample weight (i.e. mx)
N
I (wi)
m
pair (wa)
OPER (r) RV
Das (w:)
m Ydir (wis
We note that the hat notation pair (w;) means it is not necessarily a normalized PDF, since the normalization term can be reduced in Eq. (9). Intuitively, this sampler behaves as weighting the RIS sample weight 7/ to adjust for the difference between the proposal distribution qair and the target distribution Pair-

---

Published as a conference paper at ICLR 2025
In addition, also exploit spatial reuse and temporal reuse, incorporating samples from neighboring pixels and previous frames as candidates in Eq. 9 Please refer to our appendix for details. As in Fig. our reservoir sampling strategy significantly reduces the rendering noise under the same sample count compared to the standard Monte Carlo estimator.
4.2 INDIRECT LIGHTING USING MULTI-BOUNCE PATH TRACING
also give the rendering equation of indirect light and estimate it by Monte Carlo integration:
Similar to Eq. 8
CPBn(r) (r) = Lina(xx fr wa d, m) - n)dw,,
(10)
N
Lind w (x, wt d. m) (wf n)
(11)
Pind (w/)
In Eq. 111. after sampling second-bourn rays (t(() = x + tu from an appropriate probability distribution (ind(w;), we need to estimate their radiance values Lina wx) to complete the Monte Carlo estimation. The (ind(ws) of indirect lighting is straightforward: apply multiple importance sampling combining light importance sampling and GGX ma- terial importance sampling Heitz 2018), similar to NVdiffrec-MC Hasselgren el. al. 2022
Estimating [xnd(x,w;) wi) is relatively complicated. Most existing inverse rendering methods consider- ing indirect lighting adopt an implicit strategy to es- timale indirect illumination, using neural radiance fields to cache the outgoing radiance values rą- The radiance values of ry are estimated by the stan- dard volume rendering process in NeRF. The disad- vantage of this strategy obvious: the accuracy of Figure 4: Our rendering results of di- indirect lighting is determined by the neural radi- rect (b), indirect (c), and full (a) light- ance field without any physical constraints. NeRF ing in the Lego scene. Note that models inevitably contain estimation errors (espe- the sharp light visibility in (d) demon- cially in scenes with high-frequency details), so that strates the accuracy of our path-tracing the indirect lighting estimation will be biased and rendering model and our reconstruction inaccurate.
geometry.
In contrast, physically- based rendering methods conduct molti-bounce path tracing to pro- duce an unbiased estimation of (x, w;). However, due to the low performance implicit representations, recursively performing multi-bounce path tracing leads to intractable com- putation. Therefore, the aforementionod implicit strategy can be regarded as a compromise on the computation costs.
Benefiting from our efficient mesh-based representation, we can directly perform path tracing shown in Fig. 4 We first sample new ray Fina(t) x + Idynd starting from x as the second bounce, and then we trace Tim(tt) and intersect it with the mesh at point & The indirect lighting estimated by:
to estimate Lina(X,u))
Lind (x, w;) (PeR(Find) = -. L.( w., dind m) fidd..
(12)
Note that Eq. 12 is a recursive computation. In practice, we only consider the first three bounces, which balances the compulation costs and accuracy. It is also worth mentioning that we detach the gradients of indirect rays due to the limited GPU memory.
EXPERIMENTS
5.1 OVERVIEW
Implementation and training details: We implement MIRReS using Pytorch frame- work Paszke et. al. (2019) with CUDA extensions in SLANG.D Bangaru al. (2023). We
7

---

Published as a conference paper at ICLR 2025
customize CUDA kernels in our rendering layer to perform efficient reservoir sampling and multi-bounce path tracing. We also utilize nvdiffrast Laine et al. (2020) for differentiable ray-mesh intersection. We GPU, with the entire two-stage training process taking approximately 4.5 hours.
training and inference
single NVIDIA RTX 4090
Training follows the structure described in Section The first stage' is identical to the training of InstantNGP Müller et al. (2022) and NeuS2 Wang et al. (2023a), for which we refer their papers for details such training losses. Our model is trained by rendering loss (Eqs. (6) and along with several regularization terms, which are fully specified in the appendix.
Datasets: We evaluate our method on two benchmark datasets for inverse rendering: (1) TensolR synthetic dataset Jin et (2023), which provides ground-truth geometry, material parameters, and relighted images, and (2) Dbjeets-with-Lighting (OWL) real dataset [Um- menhofer et al. (2024), from which we select four scenes ( (Antman, Tpiece, Gamepad, and Porcelann Mug) containing ground-truth relighted images and environment maps. All ob- jects exhibit spatially-varying materials and complex global illumination effects such as dilfuse inter-rellections and specular highlights, making inverse rendering particularly chal- lenging.
Metrics: To assess geometry reconstruction, use the Mean Angular Error (MAE) for normal estimation and the Chamfer Distance (CD) for mesh accuracy. For intrinsic decom- position, evaluate albedo reconstruction quality, novel view synthesis of physically-based surface rendering, and relighting performance. We use the widely-used Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) as evaluation metrics. It is worth mentioning that due to the in- herent ambiguity between the scale of albedo and illumination, we apply scaling strategy similar to TensoIR Jin et al. (2023). For the TensolR. dataset, each RGB channel of all albedo results is scaled by global scalar; for the OWL dataset, the exposure level for each relighting result is also scaled.
NVD-MC TensoIR
Ours
GT
NVD-MC TensoIR
Ours
GT
C
Figure 5: Qualitative comparison of the reconstructed mesh on the TensolR
datasct. Zoom in for details.
5.2 COMPARISONS
We compare our method against statc-of-the-art, inverse rendering techniques, including TensolR Jin et al. (2023) (implicit-based) GS-IR Li-IRLiang et al. (2023) (3D Gaussian Splatting- based), NVdilfree- MC Hasselgren et al. (2022) (mesh-based, denoted as NVD-MC in figures and tables). We demonstrate that MIRRes outperforms these baselines in both qualitative and quantitative evaluations.
Table 2: Quantitative comparison of reconstructed geometry, albedo and relight- ing, novel view synthesis on the TensoIR dataset. "CD" denotes Chamfer distance, while "N-MAE" denotes normal MAE. Metrics are averaged over all testing images in all
dataset scenes. We highlight the best second-best third-best results, accordingly.
Geometry
Albedo
Relighting
Method
CD1
N-MAEJ
PSNRT
SSIM
LPIPS.
PSNRT
SSIMT
LPIPS
NVD-MC
0.073
5.050
28,875
0.957
0.082
27.810
0.907
0.110
TensoIR
0.083
4.100
29.275
0.950
0.085
28.580
0.944
0.081
GS-IR
N/A
4.948
30.286
0.941
0.084
24.374
0.885
0.096
Ours
0.056
3.305
32.348
0.970
0.054
32.363
0.965
0.055

---

Published as a conference paper at ICLR 2025
Albedo
Relighting
Normal
Albedo
Relighting
Normal
GT
Ours
TensoIR
GS-IR NVD-MC
Figure 6: Qualitative comparison of albedo, relighting and normal results on the TensoIR dataset. The environment map used for relighting is shown in the inset. Images are rendered high resolution to facilitate detailed examination when zoomed in.
Comparison on geometry reconstruction: The first two colurnns of Table report the quantitative comparisons of the normal MAE and the chamfer distance between the reconstructed mesh and the ground truth mesh. On average, our method achieves 23.3% lower mesh error and 19.4% lower normal error compared the second-best baseline. We also provide qualitative results of reconstructed meshes in Fig. aand normal maps in the last row of Fig. [ Note that GS-IR lacks mesh ex extraction, making Chamier distance inapplicable. TensoIR struggles with high- frequrrre details and sharp edges due to implicit density field limitations. NVdiffree-MMr suffers from artifacts such as holes and uneven surfaces, especially in reflective objects (c.g., the dish in the Hotdog scene). MIRRes, leveraging two-stage approach, produces refined moshes with higher quality.
Comparison on decomposition and relighting results: We perform a comprehensive comparison of the decomposed albedo, environment lighting and novel view synthesis results by PBR rendering. In this paragraph, we'll report both qualitative and quantitative results on TensolR dataset and OWL dataset, respectively.
TensolR dalasel: Qualitatively, as shown in Fig. 6. our method produces superior decompo- sition results all baselines. TensoIR and GS-IR have difficulties in correctly decomposing the lighting effects from the materials, leading to artifacts such as baked-in shadows on the albedo texture. NVdiffrec-MC produces suboptimal results due to their unstable geometry reconstruction, resulting in hole-like artifacts. On the other hand, thanks to our physically- based design to capture direct and indirect lighting, we successfully produce high-quality recovered material, eliminating the highly challenging shadow-like artifacts the material textures. We also produce more realistic relighting results, including more accurate shadows, specular highlights and inter-reflections. In contrast, due to the baked-in shadows, base- line methods (like TensoIR) suffer from incorrect shadows and highlights in their relighting results. Please refer to the appendix for comprehensive per-scene decomposition results.

---

Published as a conference paper at ICLR 2025
Further, also report quantitative comparisons in Table significant advantages over baselines in all results.
2 where our method achieves
Objects- with Lighting (OWL) dataset: We report quantitative comparisons relighting and novel view syuthesis Table 3 Note that we use the NeuS) Wang et al. recoonsreeet mesh provided by the dataset the initial coarse mesh in stage 1, rather than using NeuS2, which we empirically find better quality. We also incorporate metallic as an additional learnable channel in the output of the material network for this dataset. Our method achieves the best scores in all metrics, outperforming all the baselines by large margin. We also provide per-scene qualitative comparisons the appendix.
Table Quantitative comparison of novel view synthesis and relighting on the dataset. Metrics are averaged across all testing images from selected scenes: Antman, Tpiece, Gamepad, Porcelain Mug.
Relighting
Novel View Synthosis
Method
PSNR1
SSIM
LPIPS!
PSNR1
SSIM
LPIPS!
NVD-MC
21.110
0.970
0.066
34.409
0.967
0.059
TensoIR
26.382
0.966
0.038
37.122
0.985
0.045
GS-IR
18.761
0.101
0.314
30.527
0.793
0.096
Ours
28.827
0.977
0.031
38.223
0.986
0.030
5.3 ABLATION STUDIES
We conduct ablation studies to validate the effectiveness of our two key components: reser- voir sampling and multi-bounce ray tracing. Specifically, quantitatively compare the albedo PSNR Table ! using models with and without these components. The results show that both contribute to increase in PSNR, with the full model achieving the best overall performance.
Table 4: Ablation studies on reservoir sampling and multi-bounce raytracing.
Reservoir Multi-bounce
Albedo PSNR
Relighting PSNR
31.950
28.992
X
32.529
31.239
33.752
31.694
34.348
33.788
We also perform additional ablation studies on Indirect illumination, Number of SPPs, and Neural rudiance field rendering, with detailed results provided in the appendix.
CONCLUSION
We introduce a two-stage, physically-based inverse rendering framework that jointly recon- structs and optimizes explicit geometry, materials, and illumination from multi-view images. In the first stage, we train a neural radiance field and extract a coarse mesh as the initial geometry. In the second stage, we refine this mesh geometry using trainable offsets while optimizing materials and illumination through physically-based inverse rendering model that leverages multi-bounce path tracing and Monte Carlo integration. To improve con- vergence in Monte Carlo rendering, integrate sampling with multi-imporrance sampling, reducing variance and maintaining low rendering noise even at low sample counts. Our experiments, particularly in scenes with complex shadows, demonstrate that our method achieves -tate-of-the-art material, and lighting.
the-art performance scene decomposition, effectively recovering shape,
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the Ministry of Education, Singapore, under its Aca- demic Research Fund Grants (M01-T2EF2022000005 & RT19//2) and the RIE2020 Indus- try Alignment Fund-Industry Collaboration Projects (IAF-ICP) Funding Initiative, well as cash and in-kind contribution from the industry partner(s).
REFERENCES
Sai Bangaru, Lifan Wu, Tau-Mao Li, Jacob Munkberg, Gilbert Bernstein, Jonathan Ragan- Kelley, Fredo Durand, Aaron Lefohn, and Yong He. Slang.:: Fast, modular and differen- tiable shader programming. ACM Transactions on Graphics (SIGGRAPH Asia), 42(6): 18, December 2023. doi: 10.1145/3618353.
Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick HolddGGeffinn David Krieg- man, and Ravi Ramamoorthi. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 294-311. Springer, 2020.
Benedikt Bitterli, Chris Wyman, Matt PharT, Peter Shirley, Aaron Lefohn, and Wojeiech Jarosz. Spatiotemporal reservoir resampling for real-time ray tracing with dynamic direct lighting. ACM Transactions on Graphics (TOG), 39(4):148-1, 2020.
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 12684-12690 2021a.
Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural-pil: Neural pre-integrated lighting for reflectance decomposition. Advances in Neural Information Processing Systems, 34:10691-10704, 2021b.
Brent Burley and Walt Disney Animation Studios. Physically-basee shading at disney. In Acm Siggraph, volume 2012, pp. 1-7. vol. 2012, 2012.
Min-Te Chao. A general purpose unequal probability sampling plan. Biometrika, 69(3): 653-656, 1982.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision, pp. 333 350. Springer, 2022.
Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16569-16578, 2023.
Holger Dammertz, Daniel Sewtz, Johannes Hanika, and Hendrik PA Lensch. Edge-avoiding a-trous wavelet transform for fast global illumination filtering. In Proceedings of the Conference on High Performance Graphics, pp. 67-75, 2010.
Sara Fridovich-Keil Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angioo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5501-5510, 2022.
Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, and Ying-Cong Chen. Ref-neus: Ambiguity- reduced neural implicit surface learning for multi-view reconstruction with reflection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4251- 4260, 2023.
11

---

Published as a conference paper at ICLR 2025
Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, light, and material de- composition from images using monte carlo rendering and denoising. Advances in Neural Information Processing Systems, 55:22856-22960 2022.
Eric Hcitz. Sampling the ggx distribution of visible normals. Journal of Computer Graphics Techniques (JCGT), 7(4):1 13, 2018.
Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, and Delio Vicini. Dr. jit: A just-in-time compiler for differentiable rendering, ACM Tronsactions on Graphics (TOG), 41(4):1 19, 2022.
Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, and Hao Su. Tensoir: Tensorial inverse rendering. In Proceedings of the TEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2023.
James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Compuler graphies and interactive techniques, pp. 143 113-150, 1986.
Tero Karras. Maximizing parallelism in the construction of bvli, octrees, and k-d Irees. In Proceedings of the Fourth ACM COGGRAPH/Burogupphee Conference on High- Performance Graphics, pp. 33 37, 2012.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4):
1-, 2023.
Samuli Laine, Janne Hellsten, Tero Karras, Yoongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performaane differentiable rendering. ACM Transactions on Graphics, (6), 2020.
Tzu-Mao Li, Mika Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. ACM Transactions on Graphics (TOG), 371:)11-1,, 2018.
Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering- Xix preprint erXiv:2811.100 164779 2023.
Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. Reparameterizing discontinuous integrands for differentiable rendering. ACM Transaclions on Gruphics (TOG), 38(6): 1-14, 2019.
Fujun Luan, Shuang Zhao, Kavita Bala, and Zhao Dong. Unified shape and svbrdf recovery using differentiable monte carlo rendering. In Computer Graphics Forum, volume 40, pp. 113. Wiley Online Library, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ra- mamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99 106, 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller, Instant neural graphics primitives with multiresolution hash encoding, ACM transactions on graphics (TOG), 41(4):1-15, 2022.
Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8280 8290, 2022.
Murtagh. Multiscale transform methods in data analysis. University of Ulster, Coleraine, pp. 1-8, 1998.
Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. Practical svbrdf acquisition of 3d objects with unstructured flash photography. ACM Transactions on Grophics (TOG), 1-) 2018.
12

---

Published as a conference paper at ICLR 2025
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, perfomaceo deep learning library. Advances in neural information processing systems, 32, 2019.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-61011 2021.
Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 7504, 2021.
Cheng Sun, Min Sun, and Hwamn- Tzong Chen. Direct voxel grid optimization: Super- fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 5459-5469, 2022.
Justin F Talbot. Importance resampling for global Ulumination. Brigham Young University, 2005.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Gen- crative gaussian splatting for efficient 3d content creation. In The Twelfth International Conference on Learning Represenlations, 2023a.
Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang, and Gang Zeng. Delicate textured mesh recovery from nerf via adaptive surface refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17739-
17749, 2023b.
Benjamin Ummenhofer, Sanskar Agrawal, Rene Sepúlveda, Yixing Lao, Kai Zhang, Tian- hang Cheng, Stephan R. Richter, Shenlong Wang, and Germán Ros. Objects with lighting: A real-world dataset for evaluating reconstruction and rendering for object relighting. In 3DV. IEEE, 2024.
Eric Veach and Leonidas J Guibas. Optimally combining sampling techniques for monte carlo rendering. In Proceedings of the 22nd annaal conference on Computer graphics and interactive techniques, pp. 419-428, 1995.
Haoyuan Wang, Wenbo Hu, Lei Zhu, and Rynson WH Lau. Inverse rendering of glossy objects via. the neural plenoptic function and radiance fields. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. -9999-20008, 2024.
Peng Wang, Lingje Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruc- tion. Advances in Neural Information Processing Systems, 34:27171-27188 2021.
Yiming Wang. Qin Han, Mare Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Proceedings of the IEEE/CVE /CV International Conference on Computer Vision, pp. 3295 3306, 2023a.
Zian Wang, Tianchang Shen, Merlin Nimier-David, Nicholas Sharp, Jun Gao, Alexander Keller, Sanja Fidler, Thomas Müller, and Zan Gojeic. Adaptive shells for efficient neural radiance field rendering. ACM Transactions on Graphics (TOG), 42(6): 15, 2023b.
Liwen Wu, Sai Bi, Zexiang Xu, Fujun Luan, Kai Zhang, Tlyaan Georgiev, Kalyan Sunkavalli, and Ravi Ramamoorthi. Neural directional encoding for efficient and accurate view- dependent appearance modeling. In CVPR, 2024.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4811 2021.
13

---

Published as a conference paper at ICLR 2025
Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free free 3d gaussian splatting. 2024.
Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shnang Zhao. Path-space differentiable rendering. ACM Transactions on Graphics (TOG), 39 (9)433143 1, 2020.
Jingyang Zhang, Yao Yao, Shiwei Li, Jingbo Liu, Tian Fang, David McKinnon, Yanghai Tsin, and Long Quan. +: Inter-reflcctable light fields for geometry and material estimation. In Proceedings of the IEEE/CV /FF International Conference on Computer Vision, pp. 3601-3610, 2023.
Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for daysics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. -453-5462, 2021a.
Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (ToG), 40(6):1 18, 2021b.
Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Mod- cling indirect illumination for inverse rendering. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pottern Recognition, pp. 18643 18652, 2022.
APPENDIX
A IMPLEMENTATION AND TRAINING DETAILS
Visibility estimation. We here describe our visibility estimation based on an explicit ray-mesh intersoction computation (as described in Section 4.1p. Based on our mesh-based geometry, our method can directly determine V(x,w;) by ray-mesh intersection in our path tracing Iramework. With the intersection point x obtained from nvdiffrast, we sample an outgoing direction W; construct the visibility test ray r;(()= = + lue, Then, we conduct a ray-mesh intersection lest. determine whether the ray occluded. V(r, wi) will be 0 if (t) is occluded, otherwise 1. Benefiting from
mesh-based representation, we implement a linear BVH (LBVH Karras (2012) using CUDA kernels to significantly accelerate the ray-mesh interscction calculation. Our LBVH is updated in each iteration to match the mesh refinement.
Network details. In stage the density and appearance fields Fay Fe follow the stan- dard Instant-NGP configuration Müller et al. (2022), using a hash grid with 16 levels, feature dimensions per entry, coarsest resolution of Nmin 16, and finest resolution of Nmax = 2048, followed by 4-layer MLP with rial network Fn uses the same hash grid configuration, followed by a 2-layer MIP with 32 hidden channels.
64 hidden channels. In stage 2, the mate-
Training details. Apart from the rendering losses (Fqs. 6 and 77) mentioned in the main text, we also add several additional regularizations to stabilize the training, described as follows:
To prevent drastic changes in vertex offset Av during optimization, we apply the Laplacian smooth loss and vertices offset regularization loss from NeR12MMsh Tang et al. (023b):
Lamooth =
=Σ>
1
[[(v: Av.) - (V3 + Av,)11
(13)
JEX;
Collset
Σ
(14)
where X; is the set of adjacent vertex indices of v;.
14

---

Published as a conference paper at ICLR 2025
We also apply the smoothness regularizers for albedo ka, roughness p. and normal pro- posed by NVdiffrec-MC Hasselgren et al. (2022) for better intrinsic decomposition:
Cx
(x,) k (x: (xx+ee) +e),, & p,
(15)
X
xEX
where X is the set of world space positions on the surface, and small random offset vector.
Additionally, for better disentangling material parameters and light. we adopt the same monochrome regularization term of NVdiffrec-MC Hasselgren et al. (2022):
Lightt = YY (Ca + c.) V (Iret)
(16)
where Cd and Cs the demodulated diffuue and specular lighting terms, Y(x)
(x, + Xg + x6) /3 is a luminance operator, V(x) = max (x,, Xg: x,) is the HSV value compo- nent. For more details and discussions of this loss, please refer NVdiffrec-MC et al. (2022).
Hasselgren
Denoiser. We adopt the Edgo- À- A-Trous Wavelet Transform (EAWT) Dammertz et al. (210)) as an efficient and stable denoiser. This method uses a wavelet decomposition, where the input cp(p) is iteratively smoothed using By-spline kernel h Murtagh 1998 At each level i, the signal decomposed into a residual C++1 as follows:
c++( cp(p)) * h4,
(17)
where h; expands its support by inserting 2'-1 zeros between coefficients at each step, ensuring computational efficiency.
To preserve edges during smoothing, data-dependenl weighting function w(p, g) is intro- duced. This function incorporates information from the ray-traced input image (It), normal buffer (n), and position buffer (z)):
w(p, 4) = Wre Wp Wxy
(18)
where
I1 1,1
wer (p g) = exp
(19)
oft
based on color differences between pixels p and q. The o-parameters control the sensitivity to variations. These weights ensure that the filter adapts to the scene structure, preventing excessive edge blurring.
The computation of (p) involves a normalization factor k defined as:
Σ
hz(g) w(p.g).
(20)
EE
Using k, (p) is then computed as:
C++1 (p) = Σ 14(4)) w(p.g) c((g)-
(21)
9EN
We perform three iterations of this process, balancing accuracy and computational efficiency.
Spatial-temporal Reuse. Eq. 19 can be implemented in a straightforward way by gen- erating and storing all m candidate samples before selecting a final sample. However, this approach is computationally demanding. To address this, we utilize weighted reservoir sampling (WRS) Chao (1982) which transforms RIS into streaming way. That is, we maintain a reservoir structure r {y", 7 M"} for each pixel, where y" is the selected sample, T is the sum of the weights, and M is the number of samples seen far. When a new candidate (wa, 7x) comes in, we update Yomm M and decide whether to select the new sample based the ratio of
15

---

Published as a conference paper at ICLR 2025
After the final sample is selected for each primary ray (or pixel), exploit both spatial reuse and temporal reuse. Specifically, we merge reservoirs from neighboring pixels (spatial reuse) and previous frames (temporal reuse).
As i increases, the influence of temporal reuse gradually extends to encompass contributions from all past frames. Similarly, spatial reuse progressively incorporates information from larger region of the screen. This expansion occurs because, in each itcration, a pixcl's reser- voir merges with information from its neighboring pixels, and in subsequent frames, those neighboring pixels' reservoirs have already integrated data from their own surroundings. Consequently, both spatial and temporal reuse effectively propagates information across increasingly broader spatial and temporal domains.
B MORE EXPERIMENTAL RESULTS
B.1 PER-SCENE QUALITATIVE RESULTS IN TENSOIR DATASET
We provide per-scene qualitative comparisons of all scenes in the TensolR synthetic datasel in Figs. 11 and 12 We compare the results material (albedo, roughness), normal, novel view synthesis (NV)), and relighting results under 2 different environment maps per scene. Owing to our more accurate material estimation, we produce more realistic relighting results, including more accurate shadows, specular highlights and inter-reflections
B.2 PER-SCENE QUALITATIVE RESULTS IN OWL DATASET
We provide per-scene qualitative comparisons on the real-captured OWL dataset in Fig. 13 and Fig. 14 Our method demonstrates superior quality in the relighting appearances, while baseline methods suffer from color bias (e.g. Tpiece scene in Fig. 13 incorrect lighting effects (e.g. highlights in Gamepad scene Fig. [4 or missing details (e.g. textures Antman scene in Fig. 13 Although it is impossible to obtain ground truth material parameters in the real dataset, it can be intuitively obscrved that our method produces more reasonable material estimations.
B.3 MORE ABLATION STUDIES
Indirect illmination: Fig. 7 fillstrates the reconstructed albedo using our full model and an ablation model without indirect lighting, verifying that introducing indirect illumination can significantly improve the quality of material estimation.
Ours
w/o Indirect
GT
Figure 7: Ablation studies on indirect illumination. Due to the lack indirect illumi- nation, specular reflections near the plate are heavily embedded in the reconstructed albedo. In contrast, with indirect illumination, our method achieves a cleaner albedo recovery.
Number of SPPs: We analyze the novel view synthesis PSNR of the Hotdog scene across four different configurations with varying SPPs (from 4 to 64) as shown in Fig. g The configurations include path tracing without indirect illumination (PT), path tracing with indirect illumination (PT full), path tracing with reservoir sampling but without indirect illumination (ReSTIR), and path tracing with both reservoir sampling and indirect illumina- tion (ReSTIR full). The results demonstrate that the "ReSTIR full" configuration achieves the highest PSNR across all SPPs, confirming the efficacy of both reservoir sampling and indirect illumination. While the PSNR for "PT" is higher than "PT full", this does not mean the indirect illumination has a negative effect. Due to the lack indirect illumina- tion, "PT" bakes the specular reflection into the albedo, while "PT full" cannot capture
16

---

Published as a conference paper at ICLR 2025
38,00
35.75
PSNR (d18)
33.50
16
32
64
Samples per pisel
Figure 8: Novel view synthesis PSNR of hotdog across varions SPPs.
the specular reflections due to the severe rendering noisc at low SPP. As SPP increases, the specular reflections reconstructed by "PT full" become more accurate, while lower render- ing variance leads to more specular ambiguily in "PT", causing a decline in PSNR. At the same time, Fig. [ shows that a configuration with SPP higher than 32 does not necessarily improve the reconstruction accuracy. Thus, use 32 SPP as the default configuration for all our experiments in this section.
Neural radiance field rendering: As described in Section 3.2 employ two rendering methods: neural radiance field rendcring and physically-basck surface rendoring, to jointly optimize the roconstructed geometry. Here we demonstrate the necessity of the neural radiance field rendering. As illustrated in Fig. [ relying solely on surface rendering (the third column) leads to inaccuracies in the geometry, which subsequently affects the reconstruction of materials.
GT
w/ radiance
w/o radiance
Figure 9: Effect of neural radiance field rendering. Top: Results of novel view syn- thesis. Bottom: Reconstructed normals.
C LIMITATIONS AND FUTURE WORK
The performance of our optimization relies on the initial coarse geometry obtained in stage 1. While NeuS2 generally reconstructs plausible geometries in most cases, the extracted mesh still contains noticcable errors, particularly in areas with high specularity or fine de- tails. These inaccuracies lead to incorrect material estimation and compromised novel view synthesis the affected areas (see Fig. 10). Enhancing the geometry optimization capabilities remains area for future research. It would also be possible to combine our
17

---

Published as a conference paper at ICLR 2025
Normal
Albedo
NVS
NVS-GT
Figure 10: Limitations: incorrect material estimation on the NeRF Synthetic dataset. The primary challenges are due to high-frequency details and highly specular regions, as shown the Lego scene and the Hotdog scene, respectively.
method with recent methods targeting specular reflections Ge et al. 2023 Wu et al. 2024
Wang et al. (2024) to achieve higher-quality reconstruction and rendering of highly specu- lar scenes. Furthermore, our plysically-based inverse rendering framework currently does not account for the gradients of non-primary rays, which could potentially improve recon- struction accuracy. However, inchuding these gradients would significantly increase memory demands and computational costs. Exploring efficient methods for gradient computation is another avenue for future work.
18

---

Published as a conference paper at ICLR 2025
Albedo
Roughness
Normal
NVS
Relighting
Relighting
N/A
Ous
TersolR
NVD-MC
N/A
GT
Ouss
TersoR
NVD-MC
Figure 11: Qualitative results on Lego and Ficus scenes in TensoIR dataset. The corresponding environment map for relighting is placed on the bottom right of the GT relighting result.
19

---

Published as a conference paper at ICLR 2025
Albedo
Roughness
Normal
NVS
Relighting
Relighting 2
N/A
GT
Ours
Tensol
GSIR
N/A
GT
Oues
TescoR
GSIR
Figure 12: Qualitative results on the Hotdog and Armadillo scenes from the TensoIR dataset. The corresponding environment map used for relighting is displayed at the bottom right of each GT relighting result.
20

---

Published as a conference paper at ICLR 2025
Albedo
Roughness
Metallic
Relighting 1
Relighting
N/A N/A
N/A
GT
OuIS
TensoIR
N/A
NVD-MC
N/A N/A N/A
GT
F I
F
Owss
TersoIR
N/A
NVD-MC
Figure 13: Qualitative comparison on the boihet-with-Lighting dataset (part 1).
Chosen from Antman and Tpiece scenes.
21

---

Published as a conference paper at ICLR 2025
Albedo
Roughness
Metallic
Relighting 1
Relighting
N/A N/A
N/A
GT
Ows
TensoIR
N/A
GS-IR
N/A
N/A N/A N/A
GT
Ouis
TensoIR
N/A
GS-IR
N/A
Figure 14: Qualitative comparison on the Siteet-with-Lighteee dataset (part 2).
Chosen from Gamepad and Porcelain Mug scenes.
22


Reviews:
[{"review_id": "dEDwx9VKzV", "reviewer": "Reviewer_Cqns", "summary": "This paper proposes a two-stage inverse rendering framework to jointly reconstruct geometry, materials, and lighting from multi-view images. In this framework, geometries are represented as triangular meshes with trainable vertex offsets, and this representation enables explicit path tracing technique similar to the classic PBR pipeline, which is sped up with reservoir sampling for direct illumination. With these contributions, the reported results showed better reconstruction quality for geometry, materials, and lighting both quantitatively and qualitatively.", "questions": "1. In Lines 084 -- 086, you mention the effectiveness of a denoiser inspired by NVdiffrec-MC; however, I was unable to find further discussion or detailed follow-up content about it in the manuscript. Could you elaborate on the design of this denoiser and how effective it is for the final rendering?\n2. Since an explicit environment map is also being optimized, it would be helpful to include visualizations of the optimized environment maps for some scenes to better understand the impact of this optimization.\n3. In Lines 401 -- 403, you mentioned some adjustments made to the albedo and rendered images. Could you provide more details about the specific adjustments used?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This work adapts well-studied PBR techniques, such as path tracing and reservior sampling, to the domain of inverse rendering. I personally appreciate the adaptation of these computer graphics knowledge to address a more computer vision problem, and the results do show an improvement with these contributions.\n2. The most related work to this submission is NVdiffrec-MC. From the reported results, this work achieves a better performance both quantitatively and qualitatively.\n3. This paper is well written and easy to follow. The entire formulation is physically sound and all the theoretical details are explained in an intuitive way for readers with background in physics-based rendering and inverse rendering. Sufficient ablation studies are provided to verfiy the effectiveness of each component.\n4. Given the long-standing and inherently ill-posed nature of inverse rendering, I believe this work makes a valuable contribution to this field. It can inspire further research and solutions aimed at advancing the understanding and solving of this challenging problem.", "weaknesses": "1. This work relies on a high-quality intialization of geometry, for which the authors chose NeuS2. However, the authors acknowledged that some artifacts may still persist in certain areas, which cannot be fully corrected by the subsequent refinement step. Additionally, they discussed the instability of DMTet for geometry representation, which is used by NVdiffrec-MC. I would be interested to see some visual comparisons between these two geometry representations.\n2. As the formulation only considers primary rays, it cannot handle objects with transparent materials (e.g., a glass sphere exhibiting Fresnel effects). This might be an interesting direction to explore for future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y7uPO92j39", "reviewer": "Reviewer_QNEc", "summary": "This paper proposes a framework for joint reconstruction of shape, material and lighting. The framework consists of two stages. In the first stage, it uses neural SDF optimization to reconstruction the initial object geometry and radiance field. In the second stage, it uses differentiable rendering combined with differentiable marching cube to optimize geometry, materials and lighting, with indirect illumination being properly handled. Experiments on synthetic and real object datasets show that the proposed framework achieves compelling results compared to several strong baselines.", "questions": "I would like to see more arguments about technical novelties in the paper and how this paper is different from prior work. In addition, more technical details of reservoir sampling will be welcome.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. A novel inverse rendering framework that combines the advantages of neural SDF reconstruction and physically-based differentiable rendering. \n2. The paper overall is easy to follow and well organized. \n3. High-quality inverse rendering results on both synthetic and real datasets compared to several prior strong baselines.", "weaknesses": "1. Technical novelties of the paper is very limited.\n\nThis paper's major contributions are covered by two previous works that are not properly cited. Neural-PBIR, Cheng et al.. ICCV 2023 adopts a similar pipeline to first train a neural SDF to get high-quality geometry and then run PBDR to further optimize geometry, materials and lighting. Neural-PBIR also models multiple bounces and uses unbiased gradient for geometry optimization in the PBDR stage. \"Parameter-space ReSTIR for Differentiable and Inverse Rendering\", Chang et al., SIGGRAPH 2023 adopts Reservoir sampling into inverse rendering by exploring the temporal coherence across gradient iterations. It provides a much deeper theoretical analysis and more detailed comparisons. \n\nIt will significantly enhance the paper if we can explicitly discuss how the proposed method differs from or improves upon Neural-PBIR and Parameter-space ReSTIR. We may consider adding a paragraph comparing the propoesd approach to these prior works, highlighting any key differences or advancements.\n\n2. Design choices of Reservoir sampling may not be completely technically sound. \n\nIt is reasonable to explore temporal coherence to reduce the noise and hence reduce the sample. as demonstrated in Chang et al. SIGGRAPH 2023. However, the motivations described in the Eq. (9) emphasizes that there is no analytical integral of $L(\\omega_i)f(\\omega_i, \\omega_o)$ so that we need reservoir sampling to approximate the distribution of $L(\\omega_i)$. This makes much less sense as there is a clear solution to sample the environment map lighting and we can even consider MIS to sample both BRDF and lighting. The last paragraph of Section 4.1 mentioned temporal reuse and spatial reuse but we may need more detailed discussion. \n\nIt will be beneficial to have a deeper discussion to why we need reservoir sampling to sample environment map and also more analysis on the impact of temporal reuse and spatial reuse. \n\n3. Optimization time is too long.\n\nThe total optimization time is 4.5 hours, which is 4 times longer than Neural-PBIR. This is contradictory to the motivation of using reservoir sampling to reduce the number of sample. One thing to note is that the running speed of PBDR heavily depends on system engineering. This paper may consider using more advanced PBDR framework, such as Mitsuba 3, to see if the optimization time can be significantly reduced. A breakdown of the current optimization will also help us identify the issue. \n\n4. Other minor issues:\n1. Line 196 - 202: Why not directly use Neus2 for shape reconstruction? Why run instant-NGP first? Please provide more discussion and analysis. \n2. Line 242: the notation $\\Delta \\mathbf{v} $ is in consistent with the notation in Figure 2 $\\Delta \\mathbf{x}$. Please ensure notation consistency to avoid confusion. \n3. Line 263: Why skip metallic? Please provide reasoning on this design choice and the potential impacts. \n4. Experiments: Can we include experiments on Stanford-ORB to have a more complete comparison with prior methods? Can we show visualization of the environment maps to help reader understand the lighting reconstruction quality?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UYPUG7dDcv", "reviewer": "Reviewer_fqgJ", "summary": "The novel mesh-based reconstruction method, Multi-bounce Inverse Rendering using Reservoir Sampling (MIRReS), introduced in this paper, decomposes geometry, material parameters (diffuse albedo and roughness), and illumination from multi-view images using multi-bounce path tracing with Monte Carlo integration. \nThe method produces reconstruction results of unparalleled quality due to the authors' use of global illumination (Monte Carlo ray tracing) to explicitly and physically account for indirect lighting. This is only possible because, for the first time in Inverse Rendering, ReSTIR (Reservoir-based Spatio-Temporal Importance Resampling) is employed to accelerate the costly global rendering.\nAn evaluation of the new method and a comparison with the latest and best reconstruction methods for geometry, material and lighting (NVDiffRecMC 2022, TensorIR 2023 and GS-IR 2023) shows the comparatively very good reconstruction quality, both of geometry and albedo and with regard to relighting the reconstructed scenes.", "questions": "As there is currently no code from which further details about the method can be obtained. This leaves a few questions unanswered. \n1.) How does the rough estimate of the geometry in stage 1 of the algorithm affect the rendering, and how does the algorithm determine when the geometry is sufficiently accurate to begin material reconstruction and global illumination simulation in stage 2? \n2.) In this context, it would be interesting to know how the inaccurate initial estimate of the geometry affects the estimate of the specular reflections. \n3.) The paper mentions an important feature: the reconstruction result as a triangle mesh is compatible with current graphics engines and CAD software. However, the texture is modeled implicitly using Instant NGP in the method. How is it transferred to the triangle mesh?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.) The paper is very well written and easy to follow.\n2.) It contains a very good summary of the current state of the art on Inverse Rendering for Shape, Light and Material reconstruction.\n3.) The idea to use Reservoir-based Spatio-Temporal Importance Resampling to reduce the computational load of a global illumination calculation is novel and should definitely be published.", "weaknesses": "There is no code available to evaluate the method  independently. \n\nThe evaluation and the (very good) comparison with other state-of-the-art methods lack a comparison of the results of the specular reflection. Is this estimate much worse than for the other methods? In the appendix, this is mentioned as a limitation of the procedure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5UX10DKqEA", "reviewer": "Reviewer_Kcec", "summary": "The paper presents MIRReS to jointly optimize geometry, materials, and lighting. The authors propose to extract an explicit triangular mesh in the initial stage. Subsequently, they employ physically-based rendering and utilize multi-bounce path tracing and Monte Carlo integration for indirect illumination. They also incorporate a novel technique called reservoir sampling to address noise in Monte Carlo integration.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper propose to use path tracing for indirect illumination, which use PBR to compute the indirect illumination. This can be more accurate than using radiance field cache.\n2. The paper propose to use reservoir sampling in the integral of rendering equation (only for direct light).", "weaknesses": "1. Compare to using implicit field as geometry, the use of triangle mesh may harm the rendering equality. This is common as the mesh extraction can lead to an degration in geometry quality.\n2. The paper proposes to refine mesh with fixed face topology, only an offset of each vertice can be optimized, which I think the influence of the refinement is limited. \n3. A derivation of Equation 9 can improve the clarity. And $p_{dir}$ in $\\gamma_i$ remains unkown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)