Inverse Rendering using Multi-Bounce Path Tracing and Reservoir Sampling

Authors: Yuxin Dai, Qi Wang, Jingsen Zhu, Dianbing Xi, Yuchi Huo, Chen Qian, Ying He
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: inverse rendering, material decomposition, 3d reconstruction, relighting

Abstract:
We introduce MIRReS, a novel two-stage inverse rendering framework that
jointly reconstructs and optimizes explicit geometry, materials, and lighting
from multi-view images. Unlike previous methods that rely on implicit irradiance fields or oversimplified ray tracing, our method begins with an initial
stage that extracts an explicit triangular mesh. In the second stage, we refine this representation using a physically-based inverse rendering model
with multi-bounce path tracing and Monte Carlo integration. This enables our method to accurately estimate indirect illumination effects, including self-shadowing and internal reflections, leading to a more precise
intrinsic decomposition of shape, material, and lighting. To address the
noise issue in Monte Carlo integration, we incorporate reservoir sampling,
improving convergence and enabling efficient gradient-based optimization
with low sample counts. Through both qualitative and quantitative assessments across various scenarios, especially those with complex shadows,
we demonstrate that our method achieves state-of-the-art decomposition
performance. Furthermore, our optimized explicit geometry seamlessly
integrates with modern graphics engines supporting downstream applications such as scene editing, relighting, and material editing.

Decision: accept (poster)
Reviewer scores: 8, 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1730376575225

thread_type:
official_review

<thread_object>
creator: reviewer_kcec

<content>
summary:
The paper presents MIRReS to jointly optimize geometry, materials, and lighting. The authors propose to extract an explicit triangular mesh in the initial stage. Subsequently, they employ physically-based rendering and utilize multi-bounce path tracing and Monte Carlo integration for indirect illumination. They also incorporate a novel technique called reservoir sampling to address noise in Monte Carlo integration.

strengths:
1. The paper propose to use path tracing for indirect illumination, which use PBR to compute the indirect illumination. This can be more accurate than using radiance field cache.
2. The paper propose to use reservoir sampling in the integral of rendering equation (only for direct light).

weaknesses:
1. Compare to using implicit field as geometry, the use of triangle mesh may harm the rendering equality. This is common as the mesh extraction can lead to an degration in geometry quality.
2. The paper proposes to refine mesh with fixed face topology, only an offset of each vertice can be optimized, which I think the influence of the refinement is limited. 
3. A derivation of Equation 9 can improve the clarity. And $p_{dir}$ in $\gamma_i$ remains unkown.

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730545438363

thread_type:
official_review

<thread_object>
creator: reviewer_fqgj

<content>
summary:
The novel mesh-based reconstruction method, Multi-bounce Inverse Rendering using Reservoir Sampling (MIRReS), introduced in this paper, decomposes geometry, material parameters (diffuse albedo and roughness), and illumination from multi-view images using multi-bounce path tracing with Monte Carlo integration. 
The method produces reconstruction results of unparalleled quality due to the authors' use of global illumination (Monte Carlo ray tracing) to explicitly and physically account for indirect lighting. This is only possible because, for the first time in Inverse Rendering, ReSTIR (Reservoir-based Spatio-Temporal Importance Resampling) is employed to accelerate the costly global rendering.
An evaluation of the new method and a comparison with the latest and best reconstruction methods for geometry, material and lighting (NVDiffRecMC 2022, TensorIR 2023 and GS-IR 2023) shows the comparatively very good reconstruction quality, both of geometry and albedo and with regard to relighting the reconstructed scenes.

strengths:
1.) The paper is very well written and easy to follow.
2.) It contains a very good summary of the current state of the art on Inverse Rendering for Shape, Light and Material reconstruction.
3.) The idea to use Reservoir-based Spatio-Temporal Importance Resampling to reduce the computational load of a global illumination calculation is novel and should definitely be published.

weaknesses:
There is no code available to evaluate the method  independently. 

The evaluation and the (very good) comparison with other state-of-the-art methods lack a comparison of the results of the specular reflection. Is this estimate much worse than for the other methods? In the appendix, this is mentioned as a limitation of the procedure.

questions:
As there is currently no code from which further details about the method can be obtained. This leaves a few questions unanswered. 
1.) How does the rough estimate of the geometry in stage 1 of the algorithm affect the rendering, and how does the algorithm determine when the geometry is sufficiently accurate to begin material reconstruction and global illumination simulation in stage 2? 
2.) In this context, it would be interesting to know how the inaccurate initial estimate of the geometry affects the estimate of the specular reflections. 
3.) The paper mentions an important feature: the reconstruction result as a triangle mesh is compatible with current graphics engines and CAD software. However, the texture is modeled implicitly using Instant NGP in the method. How is it transferred to the triangle mesh?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730611710959

thread_type:
official_review

<thread_object>
creator: reviewer_qnec

<content>
summary:
This paper proposes a framework for joint reconstruction of shape, material and lighting. The framework consists of two stages. In the first stage, it uses neural SDF optimization to reconstruction the initial object geometry and radiance field. In the second stage, it uses differentiable rendering combined with differentiable marching cube to optimize geometry, materials and lighting, with indirect illumination being properly handled. Experiments on synthetic and real object datasets show that the proposed framework achieves compelling results compared to several strong baselines.

strengths:
1. A novel inverse rendering framework that combines the advantages of neural SDF reconstruction and physically-based differentiable rendering. 
2. The paper overall is easy to follow and well organized. 
3. High-quality inverse rendering results on both synthetic and real datasets compared to several prior strong baselines.

weaknesses:
1. Technical novelties of the paper is very limited.

This paper's major contributions are covered by two previous works that are not properly cited. Neural-PBIR, Cheng et al.. ICCV 2023 adopts a similar pipeline to first train a neural SDF to get high-quality geometry and then run PBDR to further optimize geometry, materials and lighting. Neural-PBIR also models multiple bounces and uses unbiased gradient for geometry optimization in the PBDR stage. "Parameter-space ReSTIR for Differentiable and Inverse Rendering", Chang et al., SIGGRAPH 2023 adopts Reservoir sampling into inverse rendering by exploring the temporal coherence across gradient iterations. It provides a much deeper theoretical analysis and more detailed comparisons. 

It will significantly enhance the paper if we can explicitly discuss how the proposed method differs from or improves upon Neural-PBIR and Parameter-space ReSTIR. We may consider adding a paragraph comparing the propoesd approach to these prior works, highlighting any key differences or advancements.

2. Design choices of Reservoir sampling may not be completely technically sound. 

It is reasonable to explore temporal coherence to reduce the noise and hence reduce the sample. as demonstrated in Chang et al. SIGGRAPH 2023. However, the motivations described in the Eq. (9) emphasizes that there is no analytical integral of $L(\omega_i)f(\omega_i, \omega_o)$ so that we need reservoir sampling to approximate the distribution of $L(\omega_i)$. This makes much less sense as there is a clear solution to sample the environment map lighting and we can even consider MIS to sample both BRDF and lighting. The last paragraph of Section 4.1 mentioned temporal reuse and spatial reuse but we may need more detailed discussion. 

It will be beneficial to have a deeper discussion to why we need reservoir sampling to sample environment map and also more analysis on the impact of temporal reuse and spatial reuse. 

3. Optimization time is too long.

The total optimization time is 4.5 hours, which is 4 times longer than Neural-PBIR. This is contradictory to the motivation of using reservoir sampling to reduce the number of sample. One thing to note is that the running speed of PBDR heavily depends on system engineering. This paper may consider using more advanced PBDR framework, such as Mitsuba 3, to see if the optimization time can be significantly reduced. A breakdown of the current optimization will also help us identify the issue. 

4. Other minor issues:
1. Line 196 - 202: Why not directly use Neus2 for shape reconstruction? Why run instant-NGP first? Please provide more discussion and analysis. 
2. Line 242: the notation $\Delta \mathbf{v} $ is in consistent with the notation in Figure 2 $\Delta \mathbf{x}$. Please ensure notation consistency to avoid confusion. 
3. Line 263: Why skip metallic? Please provide reasoning on this design choice and the potential impacts. 
4. Experiments: Can we include experiments on Stanford-ORB to have a more complete comparison with prior methods? Can we show visualization of the environment maps to help reader understand the lighting reconstruction quality?

questions:
I would like to see more arguments about technical novelties in the paper and how this paper is different from prior work. In addition, more technical details of reservoir sampling will be welcome.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680506886

thread_type:
official_review

<thread_object>
creator: reviewer_cqns

<content>
summary:
This paper proposes a two-stage inverse rendering framework to jointly reconstruct geometry, materials, and lighting from multi-view images. In this framework, geometries are represented as triangular meshes with trainable vertex offsets, and this representation enables explicit path tracing technique similar to the classic PBR pipeline, which is sped up with reservoir sampling for direct illumination. With these contributions, the reported results showed better reconstruction quality for geometry, materials, and lighting both quantitatively and qualitatively.

strengths:
1. This work adapts well-studied PBR techniques, such as path tracing and reservior sampling, to the domain of inverse rendering. I personally appreciate the adaptation of these computer graphics knowledge to address a more computer vision problem, and the results do show an improvement with these contributions.
2. The most related work to this submission is NVdiffrec-MC. From the reported results, this work achieves a better performance both quantitatively and qualitatively.
3. This paper is well written and easy to follow. The entire formulation is physically sound and all the theoretical details are explained in an intuitive way for readers with background in physics-based rendering and inverse rendering. Sufficient ablation studies are provided to verfiy the effectiveness of each component.
4. Given the long-standing and inherently ill-posed nature of inverse rendering, I believe this work makes a valuable contribution to this field. It can inspire further research and solutions aimed at advancing the understanding and solving of this challenging problem.

weaknesses:
1. This work relies on a high-quality intialization of geometry, for which the authors chose NeuS2. However, the authors acknowledged that some artifacts may still persist in certain areas, which cannot be fully corrected by the subsequent refinement step. Additionally, they discussed the instability of DMTet for geometry representation, which is used by NVdiffrec-MC. I would be interested to see some visual comparisons between these two geometry representations.
2. As the formulation only considers primary rays, it cannot handle objects with transparent materials (e.g., a glass sphere exhibiting Fresnel effects). This might be an interesting direction to explore for future work.

questions:
1. In Lines 084 -- 086, you mention the effectiveness of a denoiser inspired by NVdiffrec-MC; however, I was unable to find further discussion or detailed follow-up content about it in the manuscript. Could you elaborate on the design of this denoiser and how effective it is for the final rendering?
2. Since an explicit environment map is also being optimized, it would be helpful to include visualizations of the optimized environment maps for some scenes to better understand the impact of this optimization.
3. In Lines 401 -- 403, you mentioned some adjustments made to the albedo and rendered images. Could you provide more details about the specific adjustments used?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732090895649

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We extend our sincere thanks to the reviewers for their detailed and insightful feedback on our manuscript. We are grateful for their positive remarks on our results, which recognize the superior performance of our method compared to established baseline methods.

### Descriptions of Reservoir Sampling

We apologize for any confusion caused by the unclear description of reservoir sampling in Sec. 4.1, which may have been misleading. To clarify, we are including a more detailed explanation of reservoir sampling. 

Our description consists of 2 parts, we firstly describe RIS, and then describe spatial-temporal reuse.

---
**Resampled Importance Sampling (RIS)**

According to the multiple importance sampling (MIS) theory [Veach & Guibas (1995)], the variance of the Monte Carlo estimator decreases when $p_{\text{dir}}(\omega_i)$ closely approximates the integrand $f(\omega_i)$. When $p_{\text{dir}}(\omega_i)$ is proportional to $L_{\text{env}}(\omega_i) f_r(x, \omega_i, d, m)$, the Monte Carlo estimator becomes highly efficient. However, directly sampling from such a distribution is infeasible due to its lack of a closed-form expression.

To address this, Resampled Importance Sampling (RIS) [1] provides a more advanced technique to approximate the distribution $p_\mathrm{dir}(\omega_i) \propto L \cdot f$, which is referred to as the **target distribution**. In RIS, we first choose an easy-to-sample distribution $q_{\text{dir}}(\omega)$ (referred to as the **proposal distribution**) and generate $m$ candidate samples $\mathcal{S} = \{\omega_1, ..., \omega_m\}$ from it. In our implementation, we use $L_{\text{env}}(\omega)$ as the proposal distribution. Then, we evaluate 
$\hat{p}_{\text{dir}}(\omega_i) = L(\omega_i)\cdot f(\omega_i)$ 

for each candidate sample, and assign a weight

$$
\gamma_i = \frac{\hat{p_{\text{dir}}}(\omega_i)}{q_{\text{dir}}(\omega_i)}
$$

to each 
$\omega_i \in \mathcal{S}$. Here, the hat in $\hat{p}$ means that it is *not necessarily a normalized PDF* (recall the fact that the normalized $p\propto\hat{p}$ is unknown).
Finally, we resample from the $\mathcal{S}$ according to the weight $\gamma_i$. Weighted-averaging the sampled results after repeating for $N$ times forms an $N$-sample RIS estimator of (which is Eq. 9 in our paper):


$$
C_{\text{PBR}}^{\text{dir}}(\mathbf{r}) \approx \frac{1}{N} \sum_{i=1}^N \left( \frac{f(\omega_i)}{\hat{p_{\text{dir}}}(\omega_i)} \frac{1}{m} \sum_{s=1}^m \frac{\hat{p_{\text{dir}}}(\omega_s)}{q_{\text{dir}}(\omega_s)} \right)
$$


Intuitively, the estimator behaves as if sampling directly from $p_{\text{dir}}$, weighted by the RIS sample weight to adjust for the difference between $p_{\text{dir}}$ and $q_{\text{dir}}$.

[1] Justin F Talbot. 2005. Importance resampling for global illumination. Brigham Young University.

---

**Spatial-temporal Reuse**

This equation can be implemented in a straightforward way by generating and storing all $m$ candidate samples before selecting a final sample. However, this approach is computationally demanding. To address this, we  utilize weighted reservoir sampling (WRS) [Chao 1982], which transforms RIS into a streaming way. That is, we maintain a reservoir structure $r$ for each pixel, with $r.y$ serving as the selected sample, $r.\gamma_{\text{sum}}$ serving as the sum of the weights, and $r.M$ representing the number of samples seen so far. When a new candidate $(w_s, \gamma_s)$ comes in, we update $r.\gamma_{\text{sum}}$ and $r.M$, then we decide whether to select it based on the ratio:

$$
\frac{\gamma_s}{r.\gamma_{\text{sum}}}.
$$

After the final sample is selected for each primary ray (or pixel), we exploit both spatial reuse and temporal reuse. Specifically, we merge reservoirs from neighboring pixels (spatial reuse) and previous frames (temporal reuse).

As $i$ increases, the influence of temporal reuse gradually extends to encompass contributions from all past $i$ frames. Similarly, spatial reuse progressively incorporates information from a larger region of the screen. This expansion occurs because, in each iteration, a pixel’s reservoir merges with information from its neighboring pixels, and in subsequent frames, those neighboring pixels' reservoirs have already integrated data from their own surroundings. Consequently, both spatial and temporal reuse effectively propagate information across increasingly broader spatial and temporal domains.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732602198412

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised PDF

comment:
Dear reviewers,

We sincerely thank you for your detailed and insightful feedback on our manuscript. As suggested by Reviewer Cqns, we have uploaded a revised version of our manuscript. To summarize, our revisions include:
1. **Updated description of reservoir sampling in Sec 4.1**: We acknowledge that the previous description of reservoir sampling is not clear enough, so we updated the description, mostly based on our top-level comment. We believe that it is more detailed and readable compared to the previous version.
2. **Description of denoiser in Appendix Sec A**: We apologize for not providing a detailed explanation of our adopted denoiser before, so we have added a paragraph in the appendix for it.
3. **Description of spatial-temporal reuse in Appendix Sec A**: Reviewer QNEc requires a more detailed description of our spatial-temporal reuse, so we have added a paragraph in the appendix for it.

Thank you again for your time and support!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces an inverse rendering framework for the joint reconstruction of shape, material, and lighting. In the initial stage, it extracts a rough shape, followed by the second stage, where multi-bounce path tracing is employed to estimate indirect illumination more effectively. The use of path tracing combined with reservoir sampling results in notable improvements in reconstruction quality.

Reviews raised some issues, and the rebuttal has effectively addressed most issues. The remaining major issues primarily revolve around the contributions of the proposed method, the need for comparison with Neural-PBIR, and additional experiments on more datasets.

Regarding the experiments, more comprehensive evaluations on the Stanford-ORB dataset and a thorough comparison with Neural-PBIR would strengthen the paper. During the rebuttal period, the authors provided comparisons with Neural-PBIR on a limited number of scenes. The results suggest that the proposed method demonstrates advantages over Neural-PBIR. While the comparison is not exhaustive, given the limited rebuttal timeframe and the unavailability of Neural-PBIR's code, the results are convincing enough to support the claims.

As for the contributions, directly adopting ReSTIR does not constitute a major contribution. However, the authors emphasize that their primary contribution lies in proposing a holistic inverse rendering framework that cohesively integrates all components. The experiments seem to support the effectiveness of this framework compared to prior methods.