Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising
Authors: Jon Hasselgren
Nikolai Hofmann
Jacob Munkberg
Venue: NeurIPS 2022 Accept (2022)
Keywords: Applications
Optimization
Deep Learning

Abstract: Recent advances in differentiable rendering have enabled high-quality reconstruction of 3D scenes from multi-view images. Most methods rely on simple rendering algorithms: pre-filtered direct lighting or learned representations of irradiance. We show that a more realistic shading model, incorporating ray tracing and Monte Carlo integration, substantially improves decomposition into shape, materials & lighting. Unfortunately, Monte Carlo integration provides estimates with significant noise, even at large sample counts, which makes gradient-based inverse rendering very challenging. To address this, we incorporate multiple importance sampling and denoising in a novel inverse rendering pipeline. This improves convergence and enables gradient-based optimization at low sample counts. We present an efficient method to jointly reconstruct geometry (explicit triangle meshes), materials, and lighting, which substantially improves material and light separation compared to previous work. We argue that denoising can become an integral part of high quality inverse rendering pipelines.

Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising
Jon Hasselgren NVIDIA
Nikolai Hofmann NVIDIA
Jacob Munkberg NVIDIA
Abstract
Recent advances in differentiable rendering have enabled high-quality reconstruc- tion of 3D scenes from multi-view images. Most methods rely on simple rendering algorithms: pre-filtered direct lighting or learned representations of irradiance. We show that more realistic shading model, incorporating ray tracing and Monte Carlo integration, substantially improves decomposition into shape, materials & lighting. Unfortunately, Monte Carlo integration provides estimates with significant noise, large sample counts, which makes gradient-based inverse rendering very challenging. To address this, we incorporate multiple importance sampling and denoising novel inverse rendering pipeline. This improves convergence and enables gradient-based optimization at low sample counts. We present an efficient method to jointly reconstruct geometry (explicit triangle meshes), materials, and lighting, which substantially improves material and light separation compared to previous work. We argue that denoising can become integral part of high quality inverse rendering pipelines.
Introduction
Differentiable rendering shows great promise for accurate multi-view 3D reconstruction from image observations. NeRF [39] differentiable volume rendering create high quality view interpolation through neural, density-based light-fields. Surface-based methods apply signed distance fields
43
capture high quality geometry. Recent work 6 further decompose these representations into geometry, material, and environment light.
68 62
or triangle meshes 41
75
Most aforementioned methods rely appearance baked into neural light fields or apply simple shading models. Typically, direct lighting without shadows is considered, combined with pre-filtered representations of environment lighting [4 73.. Some methods account for shadowing and indirect illumination 6 75 77), but often lock the geometry optimization when sampling the shadow term, rely learned representations of irradiance. While results impressive, the deviations from physically-basee shading models makes it harder for these methods to plausibly disentangle shape, material and lighting, as shown in Figure !
In theory, is straightforward to replace the rendering engines of these 3D reconstruction methods with photorealistic differentiable renderers 31 36 46 45 71 72) and optimize in setting with more accurate light simulation, including global illumination effects. In practice, however, the noise in multi-bounce Monte Carlo rendering makes gradient-basee optimization challenging. Very high sample counts are required, which result intractable iteration times.
In this paper, we bridge the gap between current multi-vien 3D reconstruction and physically- based differentiable rendering, and demonstrate high-quality reconstructions at competitive runtime performance. We attack the challenging case of extracting explicit triangle meshes, PBR materials
Project page: https://nvlabe waabbbb..gttttt github. ho/.vdiffreeccc////////
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

NVDIFFREC
Our
NVDIFFREC
Our
Reference
Figure 1: NVDIFFREC [41 successfully reconstructs complex geometry from multi-view images, but struggles with the material & light separation. In the top row, visualize split-screens of the rendered reconstruction and the diffuse albedo texture. Note that NVDIFFREC bakes most of the lighting in the albedo texture, which hurts quality in relighting scenarios (shown in the bottom row). In contrast, by leveraging more advanced renderer, we successfully disentangle material and lighting (note the lack of shading in the albedo texture), and improve relighting quality. The dataset consists of 200 views of the Rollercoaster from LDraw resources 22 (CC BY-2.0).
and environment lighting from a set of multi-view images, in a format directly compatible with current DCC tools and game engines. For improved visual fidelity, compute direct illumination using Monte Carlo integration with ray tracing, and add several techniques to combat the increased noise levels. By carefully trading variance for bias, enable efficient gradient based optimization physically-based inverse rendering pipeline. Compared to previous 3D reconstruction methods, our formulation primarily improves material and light separation.
Concretely, we reduce variance by combining multiple importance sampling [58] and differentiable denoisers. We evaluate both neural denoisers [0 22) and cross-bilateral filters [52) in pipeline. Furthermore, we decompose the rendering equation into albedo, demodulated diffuse lighting and specular lighting, which enable precise regularization to improve light and material separation.
2 Previous Work
Neural methods for multi-view reconstruction These methods fall in two categories: implicit or explicit scene representations. NeRF [39] and follow-ups 38
64
15
40
50
69
66)
use volumetric representations and compute radiance by ray marching through neurally encoded 5D light field. While achieving impressive results on novel view synthesis, geometric quality suffers from the ambiguity of volume rendering (74). Surface-based rendering methods [43 47
68 62]
optimizing the underlying surface directly using implicit differentiation, or gradually morph from volumetric representation into surface representation. Methods with explicit representation estimate
3D meshes from images, where most approaches assume given mesh topology [35 recent work also include topology optimization
12 but
(34 54
BRDF and lighting estimation To estimate surface radiometric properties from images, previous work on BTF and SVBRDF estimation rely on special viewing configurations, lighting patterns or
complex capturing setups 30
[6
17 I8 65 5 g 53 20). Recent methods exploit neural networks
to predict BRDF's from images B3 19
32
33 33
44 37 Differentiable rendering methods [35 76
12 21
leam predict geometry, SVBRDF and, in some cases, lighting via photometric loss.
Most related our work are neural reconstruction methods with intrinsic decomposition of shape, materials, and lighting from images 6
73 75 77). Ilmmination is represented using

---

Topology (DMIee)
Gradients
PBR
materials
Environment lighting
Differentiable MC renderer
Noisy rendering
Denoised rendering
Reference image
Denoiser
Figure 2: We extend NVDIFFREC [41] with a differentiable Monte Carlo renderer for direct illumi- nation. Additionally, reduce variance, we add differentiable denoiser. These novel steps highlighted in green. Following NVDIFFREC, the topology is parameterized using an SDF, and trian- gular surface mesh extracted in each iteration using DMTet [54). combined with spatially-varyin PBR materials and HDR environment lighting. The system is supervised using only photometric loss on the rendered, denoised image compared to a reference, and gradients are back pack-propagated the denoiser, shape, materials, and lighting parameters. All parameters are optimized jointly.
771. pre-filtered approximations 41 or low resolution environment maps [75] When the shadowing term is accounted for [75], optimization is split into two passes where geometry is locked before the shadow term is sampled. Other approaches represent indirect illumination with neural networks 63 77..
mixtures of spherical Gaussians 6 41
Image denoisers Denoisers are essential tools in both real-time- and production renderers. Tra- ditionally, variants of cross-bilateral filters are used (78). which require scene-specific manual adjustments. More recently, neural denoisers 2 10 trained on large datasets have shown im- pressive quality without the need for manual tuning, and are now incorporated in most production renderers. We directly incorporate differentiable versions of these denoisers in our pipeline. We are currently unaware of image denoisers applied in differentiable rendering, but we see lot of potential for denoisers in physically-be bhysee inverse rendering going forward.
3 System
We target the challenging task of joint optimization shape, material and environment lighting from of multi-view images with known foreground segmentation masks and camera poses. Our goal is to use physically-based rendering techniques to improve the intrinsic decomposition of lighting and materials, producing assets that can be relit, edited, animated, or used in simulation. As proof- of-concept, we extend a recent approach, NVDIFFREC [41), which targets the same optimization task (shape, materials and environment lighting). Notably, they directly optimize triangular 3D model, which has obvious benefits: it is casy to import and modify the reconstructed models in cxisting DCC tools, and a triangular representation can exploit bardware-accelerate differentiable rasterization (28). In our setting, triangular 3D models also means we can leverage for efficient shadow tests. NVDIFFREC reports competitive results on view interpolation, material reconstruction, and relighting, and we will use their pipeline as a baseline in our evaluations.
ray-tracing
Our system is summarized in Figure 2 triangular mesh with arbitrary topology is optimized from set of images through 2D supervision. Geometry is represented by signed distance field defined on three-dimensional grid and reduced to triangular surface mesh through deep marching tetrahedra (DMTTtt [54). Next, the extracted surface mesh is rendered in a differentiable renderer, using the chysically-basee (PBR) material model from Disney
g This material model combines a diffuse with an isotropic, specular GGX lobe [61]. A tangent space normal map is also included to capture high frequency shading detail. Finally, the rendered image evaluated against reference image using photometric loss. In contrast to NVDIFFREC, which uses simple renderer with deferred shading and the split-sum approximation for direct lighting (without shadows), we instead leverage renderer which evaluates the direct lighting integral using Monte Carlo integration and ray tracing (shadow rays). We represent the scene lighting using a high dynamic range light probe stored as floating point texture, typically at a resolution of x56x256 256 texels. Finally, to combat the inherent variance that comes with Monte Carlo integration, we leverage differentiable image denoising and multiple importance sampling.
3

---

10
100
1000
5000
Reference
Figure 3: Visualization of the optimization process. Note that the initial guess for topology are randomized SDF values the grid. After 1000 iterations, already have high quality topology and plausible materials and lighting for this complicated asset. Synthetic dataset with 200 frames, generated from part of the Apollo capsule, courtesy of the Smithsonian (55) (CCO-1.0).
Optimization task Let & denote the optimization parameters (shape, spatially varying materials and light probe). For given camera pose, c, our differentiable renderer produces an image (c)). Given that we use Monte Carlo integration during rendering, this image inherently includes noise, and we apply a differentiable image denoiser, Dg, with parameters, 8, to reduce the variance, 6 Idenoised (c) = Dn(Is(c)) The reference image (c) is view from the same camera. Given photometric loss function L, we minimize the empirical risk
argmin E, (DD((ooo())) 0.0
using Adam (26 based on gradients w.r.t the optimization parameters, aL/0ø, and aL/00, which are obtained through differentiable rendering. We use the same loss function as NVDIFFREC. An example of the optimization process is illustrated in Figure
3.1 Direct Illumination
The outgoing radiance I(w,) in direction wo can be expressed using the rendering equation 24 as:
Ilwo) = -/o (i(w)) (wi, (ws n)dwi.
(2)
This is integral of the product of the incident radiance, Li(w;) from direction and the BSDF ful, wo). The integration domain is the hemisphere Q around the surface normal, n.
Spherical Harmonics (SH) [1 or Spherical Gaussians (SG) 6 73 are often used as efficient lighting. In
approximations direct illumination, but only work well for low- to medium-frequeed contrast, the split sum approximation [25 41 captures all-frequency image based lighting, but does incorporate shadows. Our goal is all-frequency lighting including shadows, which we tackle by evaluating the rendering equation using Monte Carlo integration:
N Li(wi)(wws (wi, wo) n) p(ui,)
1 N
L(wo)
with samples drawn from some distribution p(w:). Note that L;(w;) includes a visibility test, which be evaluated by tracing shadow ray in direction w;. Unfortunately, the variance levels in Monte Carlo integration with low number of samples makes gradient- optimization hard, particularly with complex lighting. In Section 4 we propose several variance reduction techniques, which enable an inverse rendering pipeline that efficiently reconstructs complex geometry, a wide range of lighting conditions and spatially-varying BSDFs
Shadow gradients In single view optimization 60 shape from shadows 57
direct optimiza-
tion of the position/direction of analytical light sources, shadow ray visibility gradients [36] 3 are highly beneficial. However, in our mill setting (50+ views), similar to Loubet et al. [6 we observed that gradients of diffuse scattering are negligible compared to gradients of primary visibility. Hence, for performance reasons, in the experiment presented in this paper, the shadow ray visibility gradients are detached when evaluating the hemisphere integral, and shape optimization driven by primary visibility gradients, obtained from avdiffrast [28)

---

4 Variance Reduction
We evaluate direct illumination with high frequency environment map lighting combined with a wide range of materials (diffuse, dielectrics, and metals). Strong directional sunlight, highly specular, mirror-like materials, and the visibility component can all introduce significant levels of noise. To enable optimization in an inverse rendering setting at practical sample counts, we carefully sample each of these contributing factors to obtain a signal with low variance. Below, we describe how we combat noise by using multiple importance sampling and denoising.
4.1 Multiple Importance Sampling
We leverage multiple importance sampling 158] (MIS), a framework to weigh set of different sam- pling techniques to reduce variance in Monte Carlo integration. Given set of sampling techniques, each with a sampling distribution ps. the Monte Carlo estimator for an integral Jq (())da given by MIS is
((x.)) (X
(x)
(x)x))
(4)
Σn n&Pa (2)
The weighting functions w; are chosen using the balance heuristic. Please refer to Veach's thesis (58) or the excellent PBRT book (88) for further details.
In our case, we apply MIS with three sampling techniques: light importance sampling, Plight (w), using a piecewise-constant 2D distribution sampling technique [48), cosine sampling, Paiffuse (w), for the diffuse lobe, and GGX importance sampling [23). Pspecular (w), for the specular lobe. Unlike in forward rendering with known materials and lights, material and light parameters are optimization variables. Thus, the sampling distributions, Pi. are recomputed in each optimization iteration. Following the taxonomy of differentiable Monte Carlo estimators of Zeltner et al. (71], importance sampling is detached, i.e., we do not back-propage propaaaee gradients to the scene parameters in the sampling step, only in the material evaluation. Please refer to Zeltner et al. for careful analysis of Monte Carlo estimators for differentiable light transport.
MIS is unbiased, but chaining multiple iterations of our algorithm exhibits bias, as the current, importance sampled, iteration dictates the sampling distributions used in the next iteration. Light probe intensities, for example, are optimized based to construct the sampling distribution for the subsequent pass. For unbiased rendering, the sampling distribution must be estimated using second set of uncorrelated samples instead. Furthermore, we explicitly re-use the random seed from the forward pass during gradient backpropagation to scatter gradients to the exact same set of parameters that contributed to the forward rendering. This approach is clearly biased [59], but we empirically note that this is very effective in reducing variance, and in our setting this variance-bias trade-off works in our favor.
importance sampled image, which are then used
4.2 Denoising
For differentiable rendering, the benefits of denoising are twofold. First, it improves the image quality of the rendering in the forward pass, reducing the optimization error and the gradient noise introduced in the image loss. Second, as gradients back-propagate through the denoiser's 's spatial filter kernel, gradient sharing between neighboring pixels is enabled. To see this, let's consider simple denoiser, O X & F, where the noisy rendered image X is filtered by a low-pass filter, (# represents an image space 2D convolution). Given a loss gradient, aL 80 the gradients propagated back to the renderer are AL aL @ FT, which applies the same low-pass filter in case the filter is rotationally symmetric (F F²). In other words, the renderer sees filtered loss gradients in local spatial footprint. While denoisers inherently trade a reduction in variance for increased bias, we empirically note that denoising significantly helps convergence at lower sample counts, and help to reconstruct higher frequency environment lighting. We show an illustrative example in Figure
0x 80
Following previous work in denoising for production rendering (2] Figure shows how we separate lighting into diffuse, Ca. and specular, c, terms. This lets us denoise each term separately, creating denoised buffers, Do (C.),, and (c,)... More importantly, we can use demodulated diffuse lighting, which means that the lighting term has not yet been multiplied by material diffuse albedo, k,. In forward rendering, this is important it decorrelates the noisy lighting from material textures, thus
5

---

+
Figure 4: We separate lighting into diffuse lighting, Cd. diffuse reflectance, ka and specular lighting,
Cz. This enables fine-grained regularization and denoising without smearing texture detail.
NeRF Fuctor Hotdog
NeRF Hotdog
Porsche
30
30
30
PSNR (dB)
25
SANE (dB)
25
PSNR (dB3)
25
20
15
Bilateral demoiser
15
15
deneiser (OIDN)
10
8
16
32
64
128
10
8
16
32
128
10
8
16
32
64
128
Samples per pixel
Samples per pixel
Samples per pixel
Figure 5: Ablation study on the effect of using different denoising algorithms during optimization at low sample counts on three different scenes of increasing complexity (from left to right). We plot averaged PSNR scores over 200 novel views, rendered without denoising, using high sample counts. In this experiment, we used decorrelated samples in the backward pass to highlight the effect of denoising. The most complex scene (Porsche) failed to converge 8 spp without denoising.
selectively denoising the noisy Monte-Cate estimates and avoiding to low-pass filter figh-freueent texture information. In inverse rendering, we additionally decomposition by adding regularization on the lighting terms, as disucussed in Section 5 We compose the final image because of the view dependent Fresnel term, but expect this to be improved in future work.
improve material and light
k. (ca) + Do(c.)- We currently do not demodulate specular lighting
Cross-bilateras bilateral filters bross-bilateral filters are commonly used to remove noise in rendered images (78). To evaluate this family of denoisers in inverse pipeline, we adapted Spatio-temporal Suriance-Guided Filtering (52 (SVGF), which is a popular cross-bilatera filter using surface normals and per-pixel depth as edge-stopping guides. The denoiser is applied demodulated diffuse lighting, to avoid smearing texture detail. We disabled the temporal component of SVGF, as we focus on single frame rendering, and implemented the filter differentiable module to allow for loss gradients to propagate back to our scene parameters.
Neural denoisers As a representative neural denoiser, we deploy the Open Image Denoiser (OIDN) [1, which is U-Net [51] pre-trained a large corpus of rendered images. The de- noiser is applied to the rendered image before computing the image space loss. As the network model is fully convolutional, it is trivially differentiable, and can propagate gradients from the image space loss, through the denoiser back to the renderer.
In Figure 5 we provide an overview the effect of denoisers during 3D scene reconstruction. We observe that denoising is especially helpful at low sample counts, where we obtain similar reconstruction results at 8 spp with denoising, compared to 32 spp without a denoiser. At higher sample counts, however, the benefit from denoising diminishes, as variance in the Monte-Carlo estimates decreases. Given that denoisers enable significantly faster iteration times, we consider it valuable tool for saving computational resources when fine-tuning model parameters for subsequent runs with high sample counts. Additionally, we empirically found that using denoiser during reconstruction yields higher quality light probes, as can be seen in Figure both at low and high sample counts.
We can also jointly optimize the denoiser parameters, 0, with the 3D scene reconstruction task. i.e., the denoiser is fine-tuned for the current scene. Unfortunately, this approach has undesirable side-effects: Features tend to get baked into the denoiser network weights instead of the materials or light probe. This is especially apparent with the OIDN [] denoiser, which produced color shifts

---

due to lack of regularization on the output. We got notably better results with the hierarchical kernel prediction architecture from Hasselgren al. [22), which is more constrained. However, the results still lagged behind denoisers with locked weights. We refer to the supplemental material for details.
5 Priors
In our setting: 3D reconstruction from multi-view images with constant lighting, regularization is essential in order to disentangle lighting and materials. Following previous work [75 41), apply smoothness priors for albedo. specular, and normal map textures. Taking the albedo as an example, ka (x) denotes the diffuse albedo at world space position, and is a small random displacement vector, we define the smoothness prior for albedo as:
Lea
Ik (Xaurt) ka (Xaurf + e)1,
(5)
where Xauf are the world space positions at the primary hit point on the object. We note that the smoothness prior is not sufficient to disentangle material parameters and light, especially for scenes with high frequency lighting and sharp shadows. Optimization tends to bake shadows into the albedo texture (easy) rather than reconstruct a high intensity, small area in the environment map (hard). To enable high quality relighting, explicitly want to enforce shading detail represented by lighting, and only bake remaining details into the material textures. We propose a novel regularizer term that is surprisingly effective. We compute monochrome image loss between the demodulated lighting terms and the reference image:
Light YY (ca + C (
(6)
Here, Y (x) = (x, +x, + xb) / is a simple luminance operator, and V (x) max .g,, xx) is the HSV value component. The rationale for using HSV-value for the reference image that the max operation approximates demodulation, e.g., red and white pixel have identical values. We assume that the demodulated lighting is mostly monochrome, in which case Y (x) (x), and given that we need to propagate gradients to C2 and C., Y avoids discontinuities. This regularizer is limited by our inability to demodulate the reference image. The HSV-value ignores chrominance, but we cannot separate a shadow from darker material. This has not been problem in our tests, but could interfere with optimization if the regularizer is given too much weight. Please refer to the supplemental materials for complete regularizer details.
Xg.
6 Experiments
In our experiments, we use NVDIFFREC [41] as a baseline, and refer to their work for thorough comparisons against related work. We focus the evaluation the quality of material and light separation. At test time, all view interpolation results are generated without denoising at 2k spp. All relighting results are rendered in Blender Cycles 64 spp with denoising [1. [ quantitative comparison with NYDIFFREC and NeRFactor (75) on the NeRFactor relighting setup. Note that there is an indeterminate scale factor between material reflectance (e.g., albedo) and the
Table 1:
Summarized relighting results for NeRFactor (CC-BY-3.0), NeRF (CC-BY-3.0) and our synthetic datasets. The NeRFactor dataset contains four scenes, each scene has eight validation views and eight different light probes (256 validation images). For the NeRF dataset (which contain higher frequency lighting), we use the Chair, Hotdog, Lego, Materials and Mic scenes, with eight validation views and four light probe configurations (160 validation images). Our dataset contains a variation of high and low frequency lighting with geometrically complex objects. The image metric scores are arithmetic means over all images.
NeRFactor synthetic
Nerf synthetic
Our synthetic
PSNRT
SSIM
LPIPS
PSNR1
SSIMt LPIPSJ. PSNR SSIM LPIPS
Our
26.0 dB
0.924
0.060
26.5 dB
0.932
0.055
27.1 dB
0.950 0.027 0.049
NVDIFFREC
24.8 dB
0.910
0.063
23.3 dB
0.889
0.076
23.7 dB
0.925
NERFACTOR
22.2 dB
0.896
0.087
7

---

Our NVDIFFREC Ref
Drums
Hotdog
Lego
Ficus
Chair
Hotdog
Lego
Materials
NeRFactor dataset
NeRF synthetic dataset
Figure 6: Relighting examples from the NeRFactor and NeRF synthetic datasets. The NeRF dataset contains high frequency lighting and global illumination, and is substantially more challenging than the NeRFactor version, which uses downsampled probes. Our results contain visible artifacts, but outperform the material separation of previous work.
Our reconstruction
Material edits
Scene edit
Relighting
Figure 7: Manipulations of our extracted 3D model of the Family dataset in Blender. This scene is part of the Tanks& Temples (27 dataset (CC BY-NC-SA 3.0). Tree and bird models from TurboSquid.
light intensity. This is accounted for by scaling each image to match the average luminance of the reference for the corresponding scene. The same methodology is applied for all algorithms in our comparisons. We outperform previous work, providing better material reconstruction. Figure6Jshove visual examples.
We additionally perform relighting on the synthetic NeRF dataset, which substantially more challenging than the NeRFactor variant, due to high frequency lighting and global illumination effects. NVDIFFREC produces severe artifacts, as exemplified by the Hotdog scene Figure 6 Table shows significant increase in image quality for approach. The visual examples show that our

---

Family
NYDIRREC
Character
Our
NUDBRRSC
Gold Cape
Ref
Our
Geometry
Korm
Normal
Probe
Figure 8: We show explicit decomposition of shape, materials and lighting, directly from photos with known poses. Character is part of the BlendedMVS 667 dataset (CC BY-4.0) and Gold Cape is part of the NeRD dataset (CC BY-NC-SA 4.0).
PSNR
24.4 dB
25.7 dB 8 spp denoised
25.6 dB
26.0 dB
Reference
8 spp
128 spp
128 spp denoised
Figure 9: We show the benefits of denoising on the Porsche scene from LDraw resources (29) (CC BY-2.0). At low sample counts, denoising helps both with geometric reconstruction (in the cockpit) and to capture specular highlights. Even at 128 spp, denoising improves specular highlight and high frequency lighting details.
results are plausible, though not without artifacts. Finally, we constructed novel synthetic dataset
with three scenes with highly complex geometry stress-test the system. Each scene contains 200
training views and 200 novel views for evaluation. The three scenes are shown in Figures 3 and 9 Quantitatively we outperform previous work by larger margin, and Figure shows very little shading in the albedo textures.
In Figures and g
apply method to datasets with real photos. These sets more difficult, due to inaccuracies in foreground segmentation masks and camera poses. We extract triangle meshes that can be trivially edited in 3D modeling software, and in Figure 7 we Blender perform scene editing, material editing, and relighting. Note that the results look plausible with the inserted object being properly shaded and casting shadows on the statue, material editing works well with the learned environment lighting and relighting interacts properly with the learned materials. Figure 8 shows breakdown of geometry, material parameters and environment light.
To study the impact of demoising, we optimized the Porsche scene w/ and w/o denoising. As shown in Figure 9 denoising improves both visual quality and environment lighting detail equal sample counts. The noise levels varies throughout the scene, and we note that denoising is particularly helpful in regions with complex lighting occlusion, such the specular highlight and cockpit.
Neural light-fields, e.g. Mip-NeRF excel at view interpolation. We enforce material/light separation through additional regularization, which slightly degrades view interpolation results, as

---

Table 2: View interpolation results. For reference, the NERFACTOR scores are 26.9 dB PSNR and SSIM of 0.930 on the NerFactor synthetic dataset, and Mip-NeRF has 35.0 dB PSNR and SSIM 0.978 on the Nerf synthetic dataset. The image metric scores are arithmetic means over all test images.
NeRFactor synthetie
Nerf synthetic
Our synthetic
Real-world
PSNR
SSIM
PSNR1
SSIM
PSNR
SSIM
PSNR1
SSIM
Our
29.6 dB
0.951
28.4 dB
0.938
25.6 dB
0.934
25.29 dB 0.899
NVDIFFREC 31.7 dB
0.967
30.4 dB
0.958
25.8 dB
0.944
26.58 dB
0.918
shown in Table 2
Our scores are slightly below NVDIFFREC, but, as shown above, we provide considerably better material and lighting separation.
Compute resources Tracing rays to evaluate the direct illumination is considerably more expensive than pre-filtered environment light approaches. We leverage hardware-accelerated ray intersections, but note that our implementation is far from fully optimized. Our method scales lincarly with sample count, which gives us simple way to trade quality for performance. With batch size 8 at a rendering resolution 512, we get the following itcration times on a single NVIDIA A6000 GPU.
NVDIFFREC
Our 2 spp
Our 8 spp
Our 32 spp
Our 128 spp
Our 288 spp
Iteration time
340 ms
280 ms
285 ms
300 ms
360 ms
450 ms
Our denoising strategies enable optimization at low sample counts. Unless otherwise mentioned, for the results presented in the paper, we use high quality settings of 128+ rays per pixel with 5000 x iterations (second pass with fixed topology and 2D textures), which takes -4 hours (A6000).
7 Conclusions
Our restriction to direct illumination shows up in scenes with global illumination. Similarly, we do not handle specular chains (geometry seen through a glass window). For this, we need to integrate multi- bounce path tracing, which is a clear avenue for future work, but comes with additional challenges in increased noise-levels, visibility gradients through specular chains, and drastically increased iteration times. Our renderer is intentionally biased to improve optimization times, but unbiased rendering could expect to generate better results for very high sample counts. Other limitations include lack of efficient regularization of material specular parameters and reliance on a foreground segmentation mask. Our approach is computationally intense, requiring high-end GPU for optimization runs.
To summarize, we have shown that differentiable Monte-Carlo rendering combined with variance- reduction techniques is practical and applicable to multi-view 3D object reconstruction of explicit triangular 3D models. Our physically-base renderer clearly improves material and light reconstruc- tion over previous work. By leveraging hardware accelerated ray-tracing and differentiable image denoisers, we remain competitive to previous work in terms of optimization time.
References
[1] Attila T. Áfra. Open Image Denoise, 2022. https::/www. httpss:/www.opeennnee opwnimagedenoise.coo org/
[2] Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle,
Convolutional Networks for Denoising Monte Carlo Renderings. ACM Trans. Graph., (6644):7:1197:111 2017.
[3] Sai Bangaru, Tzu-Mao Li, and Frédo Durand. Unbiased Warped-Area Sampling for Differentiable Rendering. ACM Trans. Graph., :966):245:1-245: 2020.
[4] Jomathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Bortin--rualla, and Pratul P. Srinivasan. Mip-NeRF: A Multiscale Representation for Anti- Anti-Aliasing Neural Radiance Fields. In International Conference on Computer Vision (ICCV), pages 5855 5864, 2021.
[5] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold- Geoffroy. David Kriegman, and Ravi Ramamoorthi. Neural Reflectance Fields for Appearance Acquisition.
arXiv:2008.0 033824 2020.
[6] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. NeRD: Neural Reflectance Decomposition from Image Collections. In International Conference on Computer 2021.
Vision (ICCV), pages
10

---

[7] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan T. Barron, and Hendrik P.A. Lensch. Neural-
PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition. In Advances in Neural Information Processing Systems (NeurlPS), pages 10691-10704, 2021.
[8] Mark Boss. Varun Jampani, Kihwan Kim, Hendrik Lensch, and Jan Kautz. Two-shot spatially-varying BRDF and shape estimation. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), pages 3982-3991, 2020.
[9] Brent Burley. Physically Based Shading at Disney. In SIGGRAPH Courses: Practical Physically Based Shading in Film and Game Production, 2012.
[10] Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder. ACM Trans. Gruph., (944):981--881111 19:11,,22 2017.
[11] Wenzheng Chen. Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation based differentiable renderer. In Advances In Neural Information Processing Systems (NewrIPS), 2019.
[12] Wenzheng Chen, Joey Litalien, Jun Gao. Zian Wang, Clement Fuji Tsang, Sameh Khalis, Or Litany, and Sanja Fidler. DIB-Re R+++:++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer. In Advances in Neural Information Processing Systems (NeurIPS), pages 22834-22848, 2021.
[13] Duan Gao, Xiao Li, Yue Dong, Pieter Peers, Kun Xu, and Xin Tong. Deep Inverse Rendering for High- Resolution SVBRDF Estimation from an Arbitrary Number of Images. ACM Trans. Graph., 38(4): 134:1- 134:15, 2019.
[14] Jun Gao, Wenzheng Chen, Tommy Xiang, Clement Fuji Tsang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Learning Deformable Tetrahedral Meshes for 3D Reconstruction. In Advances in Neural Information Processing Systems (NeurIPS), pages 9936 9947, 2020.
[15] Stephan J Garbin, Marck Kowalski, Matthew Johnson, Jamic Shotton, and Julien Valentin. FastNeRF: Neural Rendering at 200FPS, arXiv:2103. 10380, 2021.
[16] Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. Linear Light Source Reflectometry, ACM Trans. Graph., 22(3):749--7) 77774997772227 2003.
(17] Abhijeet Ghosh, Tongbo Chen, Pieter Peers, Cyrus A. Wilson, and Paul Debevec. Estimating Specular Roughness and Anisotropy from Second Order Spherical Gradient Illumination. Computer Graphics Forum, 28(4):1161-))
2009.
[18] D. Guamera, G. C. Guarnera, A. Ghosh, C. Denk, and M. Gleneross. BRDF Representation and Acquisition. In Proceedings of the 37th Annual Conference of the European Association for Computer Graphics: State of the Art Reports. pages 625-650, 2016.
[19] Guo, Cameron Smith, Miloš Hašan, Kalyan Sunkavalli, and Shuang Zhao. MaterialGAN: Reflectance Capture Using a Generative SVBRDF Model. ACM Trans. Graph., (9())254:11 1225::13, 2020.
[20] Haindl and J. Filip. Visual Texture. Springer-Vette Verlag, 2013.
[21] Jon Hasselgren, Jacob Munkberg, Jaakko Lehtinen, Miika Aittala, and Samuli Laine. Appearance automatic model simplification. Eurographics Symposium on Rendering, 2021.
[22] Jon Hasselgren, Jacob Munkberg, Anjul Patney, Marco Salvi, and Aaron Lefohn. Neural Temporal
Adaptive Sampling and Denoising. Computer Graphics Forum, 39(2):147-15 155, 2020.
[23] Eric Heitz. Sampling the GGX Distribution of Visible Normals. Journal of Computer Graphics Techniques
(JCGT), 1-3) 2018
[24] James T. Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques- SIGGRAPH 86. ACM Press, 1986.
[25] Brian Karis. Real shading in unreal engine 4. SIGGRAPH 2013 Course: Physically Based Shading in Theory and Practice, 2013.
[26] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference for Learning Representations, 2015.
[27] Amo Knapitsch, Jacsik Park, Qian Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Trans. Graph., 36(4))78:1-7811)) 2017.
2017.
[28] Samuli Laine, Janne Hellsten. Tero Karras, Ycongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for tigh-eerformancc differentiable rendering, ACM Trans. Graph., 19:1:194 2020.
[29] Gerald Lasser. LDraw.org, 2022. https:/ http::///wwww. 1draw. org/
[30] Hendrik P.A. Lensch, Jochen Lang, Asla M. Sa, and Hans-Peter Seidel. Planned sampling of spatially
varying BRDFs. Computer Graphics Forum, 2233)443-482, 2003.
[31] Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Trans. Gruph. (726))222:1-22:::1. 2018.
[32] Zhenggin Li. Mohammad Shafici, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2020.
11

---

[33] Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to Reconstruct Shape and Spatially- Varying Reflectance from Single Image. ACM Trans. Gruph., 37(6):229:1-269:11 2018.
[34] Yiyi Liao, Simon Donné, and Andreas Geiger. Deep Marching Cubes: Learning Explicit Surface Rep- resentations. In Conference Computer Vision and Pattern Recognition (CVPR), pages 2916-2925, 2018.
[35] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li, Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. In International Conference on Computer Vision (ICCV), pages 7707-7716, 2019.
[36] Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob, Reparameterizing Discontinuous Integrands
for Differentiable Rendering. ACM Trans. Graph. 38(6):228: 1 1-228:14, 2019.
[37]
Fujun Luan, Shuang Zhao, Kavita Bala, and Zhao Dong Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering. arXiv:2103, /5208, 2021.
[38] Ricardo Martin-Brualle Noha Radwan, Mchdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210-7219,
2021.
[39] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren
Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 2020.
[40] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 11:41:100:11122:555 2022.
[41] Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao. Wenzheng Chen, Alex Evans. Thomas Mueller, and Sanja Fidler. Extracting Triangular 3D Models, Materials, and Lighting From Images. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2270-8280) 2022.
[42] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 11453 1 1464, 2021.
[43] Michacl Niemeyer, Lars Mescheder, Michael Occhsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2020.
[44] Merlin Nimier-David, Zhao Dong, Wenzel Jakob, and Anton Kaplanyan. Material and Lighting Reconstruc- tion for Complex Indoor Scenes with Texturesppace Differentiable Rendering. In Eurographics Symposium on Rendering- - DL-only Track, 2021.
[45] Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. Radiative Backpropagation: An Adjoint Method for Lightning-Fas -Fast Differentiable Rendering. ACM Trans. Graph., 2020.
[46] Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: A retargetable forward
and inverse renderer. ACM Trans. Graph. 3866):003:12003111 2019.
[47] Michael Occhsle, Songyou Peng, and Andreas Geiger. UNISURF: Unitying Neural Implicit Surfaces and Radiance Fields for Multi- View Reconstruction. In International Conference on Computer Vision (ICCV),
pages 5589 5599,5999992 2021.
[48] Matt Pharr and Greg Humphreys. Physically Based Rendering. Second Edition: From Theory To Imple- mentation. Morgan Kaufmann Publishers Inc., 2nd edition, 2010.
[49] Albert Pumarola, Enric Corona, Gerard Fons-Moll, and Francese Moreuo-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318- 1-32277, 2021.
[50] Christian Reiser, Songyou Peng. Yiyi Liao, and Andreas Geiger. KiloNeRF: Speeding up Neural Radiance
Fields with Thousands of Tiny ML.Ps arXiv:2 103.13744, 13744, 2021.
[51]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention CatiaattooMMMCCAAA 2015,
pages 234-241, 2015.
[52] Christoph Schied, Anton Kaplanyan, Chris Wyman, Anjul Patney, Chakravarty R. Alla Chaitanya, John Burgess, Shiqiu Liu, Carsten Dachsbacher, Aaron Lefohn, and Marco Salvi. Spatiotemporal Variance- guided Filtering: Real-time Reconstruction for Path-traces Global Illmination. Proceedings of High Performance Graphics. pages 2:1-2:12, 2017.
[53] Carolin Schmitt, Simon Donne, Gernot Riegler, Vladlen Koltun, and Andreas Geiger. On Joint Estimation of Pose. Geometry and svBRDF from Handheld Scanner. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[54] Tianchang Shen, Jun Gao, Kangxue Yin. Ming- Yu Liu, and Sanja Fidler. Deep Marching Tetrahedra: a Hybrid Representation for Righlteeoluttin 3D Shape Synthesis. In Advances Neural Information Processing Systems (NeurIPS), pages 6087-6101, 2021.
[55] Smithsonian. Smithsonian 3D Digitization, 2018. https///d..i..edu.
12

---

[56] Michael Stokes, Matthew Anderson, Srinivasan Chandrasekar, and Ricardo Motta. A Standard Default Color Space for the Internet sRGB, 1996.
[57] Kushagra Tiwary. Tzofi Klinghoffer, and Ramesh Raskar. Towards learning neural representations from shadows. urXiv:2203.15940
2022.
[58] Eric Veach and Leonidas J. Guibas. Optimally Combining Sampling Techniques for Monte Carlo Rendering.
In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, page 419-428, 1995.
[59] Delio Vicini, Sébastien Speierer, and Wenzel Jakob, Path replay backpropagation: differentiating light paths using constant memory and linear time. ACM Trans. Graph, (W:4)::08:1-18::: 2021.
[60]
Delio Vicini, Sébastien Speierer, and Wenzel Jakob. Differentiable Signed Distance Function Rendering. 2022.
ACM Trans. Gruph., (1(4):125:1-125::
[61] Bruce Walter, Stephen R. Marschner, Hongsong Li, and Kenneth E. Torrance. Microfacet Models for Refraction through Rough Surfaces. In Proceedings of the 18th Eurographics Conference on Rendering
Techniques, page 195-206, 2007.
[62] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Advances in Neural Information Processing Systems (NeurIPS), pages 2717 1-27132,,,, 2021.
[63] Zian Wang. Jonah Philion, Sanja Fidler, and Jan Kautz. Learning indoor inverse rendering with 3d
spatially-varying lighting. In Proceedings of International Conference on Computer Vision (ICCV), 2021.
[64]
Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF Neural radiance fields without known camera parameters. arXiv:2102. 102007064, 2021.
[65]
Michael Weinmann and Reinhard Klein. Advances in Geometry and Reflectance Acquisition (Course Notes). In SIGGRAPH Asia 2015 Courses, 2015.
[66]
Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakom. NeX: Real-time View Synthesis with Neural Basis Expansion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 8534-8543, 2021.
[67] Yao Yao, Zixin Luio, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks. Computer Vision and Pattern Recognition (CVPR), 2020.
[68] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by discntangling geometry and appearance. Advances in Neural Information Processing Systems, 33, 2020.
[69] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for Real- Rendering of Neural Radiance Fields. In International Conference on Computer Vision (ICCV), pages
5752-5761, 2021.
[70] Greg Zaal. polyhaven, 2021. https//polyhaven.com/hoe cowm///rr.......ii..............
[71] Tizian Zeltner, Sébastien Speierer, Iliyan Georgiev, and Wenzel Jakob. Monte Carlo Estimators for
Differential Light Transport. ACM Trans. Graph. 40(4):78:1 11:8::117::16 16, 2021.
[72] Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulckas, and Shuang Zhao. Path-Space Differentiable
Rendering. ACM Trans. Graph. 39(4); 143:1 14::1114::31 2020.
[73] Kai Zhang, Fujum Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse Rendering with Spherical Gaussians Physics- Material Editing and Relighting. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[74] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++; Analyzing and improving neural radiance fields. arXiv:2010.07492, 2020.
[75] Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron. NeRFactor: Neural Factorization of Shape and Reflectance under an Unknown Illumination. ACM
Trans. Graph., 40(6):237:1 (116):237:12237:18. 2021.
[76] Yuxuan Zhang. Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torrala, and Sanja Fidler. Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering. In International Conference Learning Representations, 2021.
[77] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei Jia, and Xiaowei Zhou. Modeling Indirect Illmmiaaioon for Inverse Rendering. arXiv:2204.0688 00004.0667,,280 2022.
[78] Matthis Zwicker, Wojeiech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rous- selle, Pradeep Sen, Cyril Soler, and Sung-Eui Yoon. Recent Advances Adaptive Sampling and Recon- struction for Monte Carlo Rendering. Computer Graphics Forum, 34(2):667-66 681, 2015.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
13

---

(b) Did you describe the limitations of your work? [Yes] See Section 7
(c) Did you discuss any potential negative socictal impacts of your work? [No] There is no negative societal impacts particular to this work. While it shares the problem of deepfakes with all other multi-view 3D object reconstruction methods, do not think warrants a discussion in particular to this paper.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] While it is impossible to detail everything in large system, we describe the significant updates from NVDIFFREC [1]. Details about regularization and optimization scheduling is included in supplemental work due to limited space.
(c) Did you report error bars (e.g., with respect the random seed after running periments multiple times)? [No] While interesting, the optimization time of 3D reconstruction algorithms (our and previous work) is too significant for such detailed analysis.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Optimization performance compute resources are described in Section
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] (b) Did you mention the license of the assets? [Yes] The synthetic NeRFactor dataset does not specify license. is tightly based on the synthetic NeRF dataset (CC-BY-3.0) and based on the No additional restrictions clause we assume CC-BY-3.0 for NeRFactor as well.
(c) Did you include any new assets either supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "PAv7OQgQ1Sh", "reviewer": "Reviewer_JQ7r", "summary": "This paper presents an algorithm for the inverse rendering based on the neural fields. Unlike existing neural-fields-based inverse rendering relies on the simple shading model, the paper introduced the ray tracing with Monte Carlo integration that enables the inverse rendering more general and practical. To make the optimization tractable, the paper proposed to introduce multiple importance sampling and denoising for the differentiable ray tracing. The extensive evaluation shows that the proposed method can accurately recover physical attributes in more physically interpretable manner.", "questions": "- The paper claimed that the proposed method scales linearly with sample count but the Table at line 275 shows the iteration time doesn’t linear to the sample count. What does “scale linearly” mean?\n\n- Can the proposed method be applied to the recovery of translucent or transparent objects with some realistic constraints (e.g., known background)?\n", "limitations": "The limitation is properly described.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The introduction of this work is very clear which is supported by an extensive survey of previous studies. \n\n- Utilizing the denoiser and importance sampling for stable and efficient differentiable rendering makes a lot of sense. I expect that this research will lead the computer vision community the active utilization of real-time rendering technics, which are being actively researched and developed in the field of computer graphics.\n\n- Though most of the techniques for forward-rendering in this paper came from previous works, but their application to inverse rendering is not obvious. I think the beauty of this paper is that in every technical component, the reader can learn why it was introduced, including its surroundings thanks to the proper citations.\n\n- The evaluation successfully demonstrated the benefit of the proposed denoising.\n\n- The extensive ablation study justifies the algorithmic choice. It is really helpful to me to see the comparison of denoisers with detailed discussions in the supplementary material.\n\nWeakness:\n\n- As described in the paper, the method still considers the single bounce of light, therefore essentially inter-reflection is still a problem.\n\n- Though qualitatively demonstrated, the reconstruction accuracy of individual attributes (such as normal or specular components) was not quantitatively evaluated.\n\n- As the authors repeatedly mentioned, the training time is not as fast as expected even with the small sample count per pixel.\n\n- Experiments show that the proposed method does a better job of recovering materials and lighting than the existing methods, but on the other hand, the increase in computational cost is an issue. Therefore, it is very helpful to clarify which kinds of scenes the existing method is not good at and the proposed method is good at handling so that a user can choose the proper method depending on the scene.\n\n- The writing of the paper is very good, and the proposed method is undoubtedly very effective in recent inverse rendering based on Neural Fields. On the other hand, however, I had the impression that the paper uses existing techniques at a very high level rather than making innovative proposals, so I have slightly downgraded it from the highest evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IkiMWlRRDvO", "reviewer": "Reviewer_Hmte", "summary": "The paper proposes a method to estimate the geometry, material and lighting \nof an object from a set of multi-view images. It builds on top of previous work \nnvdiffrec. Different from nvdiffrec that applies split-sum approximation for \ndirect lighting, the proposed method applies Monte Carlo ray tracing  and make the rendering more physically accurate. To reduce the \nnoise caused by Monte Carlo integration when\nthere are few samples, the paper proposed to use denoisers to reduce the\nvariance and make the training more efficient. The experiment results show that \nthe proposed method generates better results than baseline methods such as\nnvdiffrec and nerfactor.\n", "questions": "See above.", "limitations": "The limitations are well discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper combines neural inverse rendering and traditional Monte Carlo ray\ntracing, which enjoys the benefits of both world.\nThe idea of using denoisers to reduce variance in Monte Carlo integration \nis interesting and helps make the training more efficient. The paper did\nmultiple experiments to show that it helps improve material, geometry and \nlighting estimation.\n\nWeakness/Questions:\n1. As mentioned in the paper, while using Monte Carlo integration, the paper \ndidn't consider multi-bounces or shadow gradients, which makes the method fail\nto handle indirect illumination. As a stress test, it would be interesting to\nsee how it works when increasing the number of bounces.\n\n2. For shadow gradients, how the light leakage term is set? Is it a fixed\nparameter for all experiments? \n\n3. In Section 3 of the supp, it's not clear to me how the visibility gradients\nare calculated?  Is the paper using the methods based on warped area/edge sampling? \nIt's worth adding a more detailed discussion here to differentiate the referred\nmethods and the method used by the paper.\n\n4. How robust is the denoiser in the optimization? At the early stage of the\noptimization, when the rendering images don't make too much sense, will denoiser\nmake the optimization worse? The paper talks about ramping up the spatial\nfootprint and linearly blending the noisy and denoised images in supp (Line 60), \nhow robust are those operations? Do they need to be fine-tuned for each scene?\n\n5. During optimization, since it minimizes the difference between the denoised\nimages and the GT images, I am wondering whether it's possible that the noisy \nimages have some artifacts that are not caused by low samples but incorrect\nestimations. Such artifacts will be covered by the denoiser, and will be visible\nduring testing. Is such a case possible? During testing, can we get high-quality\nnoise-free results by increasing the number of samples without using denoisers? \n\n\n6. In Line 266, the paper says view interpolation degrades with improved\nmaterial and lighting estimations. More materials/evidence are needed to support this\nclaim. \n\nOverall, I think the idea of combining neural inverse rendering, traditional \nMonte Carlo integration and denoisers is interesting and novel. The paper did \nthorough evaluations on the method and performs detailed ablation study. \nTherefore, I vote for accepting the paper.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3tewi_Yke7p", "reviewer": "Reviewer_EEfj", "summary": "The paper tackles the challenge of learning the shape, light & material properties of a scene from multiview images. The main idea is to consider a general lighting model for the rendering equation, which in turn is approximated using Monte Carlo (MC). As MC introduces large variance gradient estimates for optimization, the paper suggests importance sampling techniques. In addition, an image denoising model is incorporated into the system. The method is evaluated on standard scene datasets.", "questions": "Please address the weakness stated above.", "limitations": "The paper does not include such a discussion.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-------------\nThe paper is well written and easy to follow. The introduction is concisely informative.\nThe suggested system seems to improve previous work in learning scene properties decomposition.\nThe evaluation of synthetic data and real data is adequate.\n\nWeaknesses\n----------------\n\nThe paper focuses on the different solutions the proposed system consists of. However, I am missing a discussion on some of the challenges or design choices made in designing such a system. For example, the choice of the importance sampling techniques is given with any discussion. Some questions that arise are: Can the variance reduction be somehow quantified (bounded above)? Are there other existing techniques?\nAnother example of a missed discussion is how much the work is coupled with NVDIFFREC. Could the suggested techniques be incorporated into methods like [1] as well? \n\nIn addition, the discussion on the bias introduced (154-162) is unclear. The seems like an important issue that can also promote further research but is very shortly discussed. \n\nThe paper could be improved by being more self-contained. For example, the sampling techniques used are only referenced.\n\nThe effect of the new regularizer is not tested in an ablation study. \n\n[1]: PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting, CVPR 2021.\n[2]: ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1nSmiGmD5m", "reviewer": "Reviewer_1u76", "summary": "The paper proposes an inverse rendering pipeline for recovering geometry, reflectance and environment map from a set of multi-view images. The key contributions is a denoising step that significantly reduces variance of Monte Carlo light integration at lower sampling rates. The method achieved superior results compared to a direct lighting baseline (nvdiffrec) and Nerfactor.", "questions": "I wonder to what extent the proposed method could be used to handle inter-reflections and light refraction in e.g. translucent objects. To me this seems to be one of the biggest advantage of Monte Carlo integration versus SH/SG but I could not find relevant discussions in paper.", "limitations": "Yes limitations have been well discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The idea to denoise Monte Carlo rendering is well motivated and well implemented. To the best of my knowledge this idea is original.\n- Results are convincing. Particularly the comparison with nvdiffrec baseline shows considerable improvement in novel rendering and relighting. Compared to SG and SH representations, the recovered environment map contains high fidelity details, evident in Figure 2 in the appendix.\n- Limitations have been discussed in detail. I appreciate authors honesty in this.\n\n\nWeakness:\n- Evaluations primarily focus on novel rendering/relighting but lacks in individual intrinsic components such as 3D geometry (e.g. depth, chamfer, normal errors etc) and environment maps. \n- Comparison with recent Monte Carlo inverse rendering method [31,37] especially [37] would strengthen this paper. (Although it should be noted [31] does not solve for geometry, while [37] assumes known lighting and does not have full code release).\n- I think the paper is better suited for CV or graphics venues than NeurIPS. There are machine learning components in this paper but the main components (denoising/Monte Carlo rendering) do not seem very well aligned with NeurIPS.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept