DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer
Authors: Wenzheng Chen
Joey Litalien
Jun Gao
Zian Wang
Clement Fuji Tsang
Sameh Khamis
Or Litany
Sanja Fidler
Venue: NeurIPS 2021 Poster (2021)
Keywords: differentiable rendering
inverse graphics
material estimation
lighting estimation

Abstract: We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable renderers. Many previous learning-based approaches for inverse graphics adopt rasterization-based renderers and assume naive lighting and material models, which often fail to account for non-Lambertian, specular reflections commonly observed in the wild. In this work, we propose DIBR++, a hybrid differentiable renderer which supports these photorealistic  effects by combining rasterization and ray-tracing, taking the advantage of their respective strengths---speed and realism. Our renderer incorporates environmental lighting and spatially-varying material models to efficiently approximate light transport, either through direct estimation or via spherical basis functions. Compared to more advanced physics-based differentiable renderers leveraging path tracing, DIBR++ is highly performant due to its compact and expressive shading model, which enables easy integration with learning frameworks for geometry, reflectance and lighting prediction from a single image without requiring any ground-truth. We experimentally demonstrate that our approach achieves superior material and lighting disentanglement on synthetic and real data compared to existing rasterization-based approaches and showcase several artistic applications including material editing and relighting. 

DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer
Wenzheng Chen 4,2,3
Joey Litalien
Jun Gao 1,2,3
Zian Wang" 1,2,3
Clement Fuji Tsang
Sameh Khamis
Or Litany
Sanja Fidler1.2.5
NVIDIA
University of Toronto
Vector Institute"
McGill University"
(zenzchen, jung, ziane, cfujitaang, akhania, olitany, stidier)eniidia.cmm coay.litalienn mogiil. ung
Abstract
We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable renderers. Many previous learning Ceuning-bassed approaches for inverse graphics adopt musterization-baseee renderers and assume naive lighting and material models, which often fail to account for non-Lambertian, specular reflections commonly observed in the wild. In this work, we propose DIB- R++, a hybrid differentiable renderer which supports these photorealistic effects by combining rasterization and ray-tracing, taking the advantage of their respective strengths- speed and realism. Our renderer incorporates environmental lighting and spatially-varying material models to efficiently approximate light transport, either through direct estimation or via spherical basis functions. Compared to more advanced physics-based differentiable renderers leveraging path tracing, DIB-R++ is highly performant due to its compact and expressive shading model, which enables easy integration with learning frameworks for geometry. reflectance and lighting prediction from a single image without requiring any ground-trus -tutth We ex- perimentally demonstrate that approach achieves superior material and lighting disentanglement lemett on synthetic and real data compared to existing rasterization based approaches and showcase several artistic applications including material editing and relighting.
1 Introduction
Inferring intrinsic 3D properties from 2D images is long-standing goal of computer vision [3]. In recent years, differentiable rendering has shown great promise in estimating shape, reflectance and illumination from real photographs. Differentiable renderers have become natural candidates for learning-based inverse rendering applications, where image synthesis algorithms and neural networks can be jointly optimized to model physical aspects of objects from posed images, either by leveraging strong data priors or by directly modeling the interactions between light and surfaces.
Not all differentiable renderers rendering techniques [31, 45, 2, 44, 59] try to model the full light transport with proper visibility gradients, but they tend to require careful initialization of scene parameters and typically exhibit high computational cost which limits their usage in larger end-to-end learning pipelines. On the other hand, performance-oriees differentiable renderers [10, 27, 36, 25] trade physical accuracy for scalability and speed by approximating scene elements through neural representations or by employing simpler shading models. While the latter line of work has proven be successful in 3D scene reconstruction.
made equal. On the hand, recent physics-basee based differentiable
"Wrk done during internship at NVIDIA. Project page: https: //tp::://///eeee tlabs. gi thub wtt../..//BDRBD /DDIBRPlus.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

the frequent assumptions of Lambertian-on oobll surfaces and low- wwwfrrquenccy lighting prevent these works from modeling more complex specular transport commonly observed in the real world.
In this work, we consider the problem of single-view 3D object reconstruction without any 3D supervision. To this end, propose DIB-R++, hybrid differentiable renderer that combines rasterization and ray-tracing through an efficient deferred rendering framework. Our framework builds on top of DIB-R [10] and integrates thysics-based lighting and material models capture challenging non-Lambertian reflectance under unknown poses and illumination. Our method is versatile and supports both sunle-bouned ray- ray-tracing and spherical Gaussian representation for a compact approximation of direct illumination, allowing us to adapt and tune the shading model based on the radiometric complexity of the scene.
We validate our technique on both synthetic and real images and demonstrate superior performance on reconstructing realistic materials BRDFs and lighting configurations over prior rasterization-baase methods. We then follow the setting proposed in Zhang et al. [62] to show that DIB-R++ can reconstruct scene intrinsics also from real images without any 3D supervision. We further apply our framework to single-image appearance manipulation such as material editing and scene relighting.
2 Related Work
Differentiable Rendering, Research on differentiable rendering can be divided into two categories: poysics-basedi methods focusing on photorealistic image quality, and approximation methods aiming at higher performance. The former differentiates the forward light transport simulation [31, 45, 2, 44, 59] with careful handling of geometric discontinuities. While capable of supporting global illumination, these techniques tend to be relatively slow to optimize or require detailed initial description of the input in terms of geometry, materials, lighting and camera, which prevents their deployment in the wild. The latter line of works leverages simpler local shading models. Along this axis, rasterization-Be based differentiable renderers [10, 27, 36, 25, 16] approximate gradients by generating derivatives from projected pixels to 3D parameters. These methods are restricted to primary visibility and ignore indirect lighting effects by construction, but their simplicity and efficiency offer an attractive trade-off for 3D reconstruction. We follow this line of work and build atop DIB-R [10, 22] by augmenting its shading models with physics-based ones.
dearning-based Inverse Graphies. Recent research on inverse graphics targets the ill ill-posed prob- lem of jointly estimating geometry, reflectance and illumination from image observations using neural networks. For single image inverse rendering, one dominant approach is employ 2D CNNs to learn data-driven features and use synthetic data as supervision [42, 49, 35, 33, 52], but these methods do not always generalize complex real-world images [4]. To overcome the data issue, recent body of work investigates the use of self-supervised learning to recover scene intrinsics [1], including domain adaptation from synthetic reflectance dataset [37], object symmetry [54, 53], or multi-illumination images depicting the same scene [30, 32, 39, 58]. However, these methods either rely on specific priors or require data sources tedious to capture in practice. Some works tackle the subtask of lighting estimation only [18, 17, 20]. but still need to carcfully utilize training data that are hard to capture. Most similar to us, DIB-R R [10] tackles unsupervised inverse rendering in the context of differentiable rendering. Zhang et al. [62] further combines DIB-R with StyleGAN [24] generated images to extract and disentangle 3D knowledge. These works perform inverse rendering from real image collections without supervision, but may fail to capture complex material and lighting effects-in contrast, our method models these directly. Several techniques also try to handle more photorealistic effects but typically require complex capturing settings, such as controllable lighting (28, 29], a co-located camera- -faashighhh setup [41, 13, 34, 5, 6, 8, 48, 38], and densely captured multi-view images [14, 55, 7, 60] with additional known lighting [19] or hand-crafted inductive labels [43]. In our work, we propose a hybrid diffcrentiable renderer and learn to disentangle complex specular effects given single image. Similar to the recent NeRD [7] and PhySG [60] which recover non-Lambertian reflectance and illumination with a spherical Gaussian (SG) basis [51], we also employ SGs to model the SV-BRDF and incident lighting, but apply this representation mesh-based differentiable rendering with direct access to the surface.
3 Differentiable Deferred Rendering
In this section, we introduce DIB-R++, our differentiable rendering framework based on deferred shading [12]. DIB-R++ is hybrid differentiable renderer that can efficiently approximate direct illumination and synthesize high-quality images. Concretely, our renderer leverages the differentiable
2

---

M
Massi
Albedo
LA
Wo
(a)
(b)
(c)
(d)
renderer to obtain diffuse alhedo, surface normals and mask maps. In the shading pass (b), we then use these buffers to compute the incident radiance by sampling by representing lighting and the specular BRDF using spherical Gaussian basis. Depending the representation used in (c),
Figure 1: Overview. Given a 3D mesh M, we employ (a) a rasterization-
recover wide gamut of specular/gloss, appearances (d).
rasterization framework of DIB-R [10] to recover shape attributes and further employs physics-based material and lighting models estimate appearance.
3.1 Overview
We provide an overview of our technique in Fig. 1. We first rasterize 3D mesh to obtain diffuse albedo and material maps, surface normals, and silhouette mask. This information is deferred to the shading pass, where outgoing radiance is either estimated stochastically approximated using spherical Gaussian basis. The rasterizer and shader are differentiable by design, allowing gradients to be propagated to lighting, material and shape parameters for downstream learning tasks.
3.2 Background
Our goal is to provide a differentiable formulation of the rendering process to enable fast inverse rendering from 2D images. Let M be 3D object in a virtual scene. We start from the (non-emissive) rendering equation (RE) [23], which states that the outgoing radiance Lo at any surface point x E M in the camera direction Wo given by
Lo(x, = / fi Wi, Lx wi)|n w//dw, H22
(I)
where L; is the incident radiance, fr is the (spatially-varying) bidirectional reflectance distribution function (SV-BRDF) and n is the surface normal at x. The domain of integration the unit hemisphere 72 of incoming light directions illumination from different directions and is modulated by the cosine foreshortening term In w| Intuitively, Eq. (1) captures energy balance and computes how much light is received and scattered at a shading point in a particular direction.
The BRDF characterizes the surface's 's response to
Estimating the RE typically requires Monte Carlo (MC) integration [46], which involves tracing rays from the camera into the scene. Albeit physically correct, this process is computationally expensive and does not generally admit closed-form solution. MC estimators can exhibit high variance and may produce noisy pixel gradients at low sample count, which may significantly impact performance and convergence. To keep the problem tractable, we thus make several approximations of Eq. (1), which we detail in the next section.
3.3 Two-stage Deferred Rendering
We now describe rendering framework (Fig. 1). We start by defining three families of parameters, where π E Id encodes the shape attributes (e.g.. vertex positions), E Rdd describes the material properties, and y E Rdd captures the illumination in the scene. In what follows, shall only consider a single pixel, indexed by p. within an RGB image I E R 3x for notational simplicity.
Stage 1: Rasterization Pass. We first employ differentiable rasterizer R [10] to generate primary rays w. E S from camera and render our scene M into geometry buffers (commonly called G-buffers) containing the surface intersection point Xp R3, the surface normal np S and the apatially-varying matcrial parameters 0, (c.g.. diffuse albedo). This rendering pass also returns a visibility mask up e {0, 1} indicating whether pixel p is occupied by the rendered object, separating the foreground object Iç from its background environment T, so that / = I + Ib. We have:
R(M, p. w.) (Xp, Dp, Op, Up).
Stage 2: Shading Pass. Given surface properties and outgoing direction w., we then approximate the outgoing radiance [x(Xp, w.) through several key assumptions. First, we restrict ourselves to direct illumination only (i.e. single-bounce scattering) and assume that the incoming radiance given by distant environment map L, 2 + R². Therefore, we do not model self-occlusion and

---

L((x, (1xp:wi)== L((w) y). Such simplification largely reduces computation and memory costs and is trivially differentiable. Second, we assume that the material parameters 0 can model both diffuse and specular view-dependent effects. At high level, we define our shading model S so that:
(xp: np, wo: Pp. 7) F Lo (xp, wo).
Importantly, a differentiable parameterization of S enables the computation of pixel gradients with respect all scene parameters θ (,T, 0. 7) by differentiating ()) (S R) (M, p. w.). Given a scalar objective function defined on the rendered output I, 0I θπ is computed using DIB-R [10]. In what follows, we thus mainly focus on formulating ai ( 7 so that all gradients can be computed using the chain rule, allowing for joint optimization of geometry, material and lighting parameters. We assume henceforth a fixed pixel p for conciseness, and remove the subscript.
3.4 Shading Models
Since primary goal is to capture wide range of appearances, we provide two simple techniques to approximate Eq. (1): Monte Carlo (MC) and spherical Gaussians (SG). The former targets more mirror-like objects and can better approximate higher frequencies in the integrand, but is more expensive to compute. The latter is more robust to roughness variations but is limited by the number of basis elements. To model reflectance, we choose to use a simplified version of the isotropic Disney BRDF [9, 15] based on the Cook-TForracee model [11], which includes diffuse albedo a E [0, 1Js, specular albedo E [0, 1. surface roughness 3 E [0, 1] and metalness E [0, 1]. Metalness allows us to model both metals and plastics in a unified framework. We let the diffuse albedo vary spatially (a a(x) and globally define all other attributes to restrict the number of learnable parameters.
Monte Carlo Shading. Given a surface point x E .M to shade, we importance sample the BRDF to obtain N light directions w and compute the BRDF value. We represent the incident lighting L (MC) as a high-dynamic range image 7 E 2 R3xhu x using an equirectangular projection, which can be queried for any direction via interpolation between nearby pixels. The final pixel color is then computed as the average over all samples, divided by the probability of sampling wt::
N fr(x, w woi 0) I T.MO) (w 7) In w/|
S(MC) (x n. Wo: 0o: 0,7)
(4)
N
p(w²)
When the surface is near-specular (e.g., a mirror), one can efficiently estimate the RE as reflected rays are concentrated in bundles (e.g., to satisfy the law of reflection). However, this estimator can suffer from high variance for rougher surfaces; a higher number of samples may be necessary to produce usable gradients. While this can be partially improved with multiple importance sampling [50], emitter sampling would add a significant overhead due to the environment map being updated at every optimization step. This motivates the use of more compact representation.
Spherical Gaussian Shading. To further accelerate rendering while preserving expressivity in shading model, we use a spherical Gaussian (SG) [51] representation. Projecting both the cosine- weighted BRDF and incident radiance into an SG basis allows for fast, analytic integration within our differentiable shader, at the cost of some high frequency features in the integrand. Concretely, an SG kernel has the form G(w:E. A, 44) = fs e (E where w E S2 is the input spherical direction to evaluate, 8 is the axis, À E RR., is the sharpness, and μa E R.. is the amplitude of the lobe. We represent our environment map using a mixture of K lighting SGs Gr, so that:
K
L (SG) (wi 7) R Σ 9 (wis Af. pt),
(5)
where 7 : ( 1 14( Ja. For the BRDF, we follow Wang et al. [51] and fit single, monochromatic SG to the specular lobe so that (S))) is sum of diffuse and specular lobes. The full derivation can be found in our supplementary material (Sec. A). Finally, we approximate the cosine foreshortening term using a single SG [n wi| Ge(wi: n, 2.133, 1.17) [40]. Regrouping all terms, the final pixel color can be computed as:
S(SG) (x. n. wo: 0,,) {
n.
/ f( (x, w., wo: 0) (5()) (wi) y) Ge(we) dui,
(6)
which has an analytic form that can be automatically differentiated inside our renderer. All parameters of the SGs, as well as the BRDF parameters, are learnable.

---

Env. map (dense)
0.05
0.10
3 0.15 8
0.20
3
0.25
3
0.30
0.35
0.40
3 0.45
MC Bin.
K-
128 SGs
Figure 2: Visual comparisons between MC and SG shading. On the left, we show an environment map and its SG-ressesentttion (K 128). While losing sharp details, SGs only needs 1% parameters compared to the dense pixel HDR map. On the right, we increase roughness left-to-right, revealing that our spherical basis can correctly approximate direct illumination at maddraae-tt-hiige roughness.
Figure 3: Optimization. We optimize complex lighting and material for both MC shading (left) and SG shad- ing (right). In cach case, first optimize lighting and show the light and rendered image in first block. Next we optimize the surface mate- rial and show the surface ma- terial value and rendered im- age in second block.
Optimization of MC Shading Opt. Env. map
Optimization of SG Shading Opt. SG Light
Opt. |
Opt. 3,
05'05 oee-
orden
ornze
Opt.
Comparison. To visually compare our two shading techniques and understand their limitations, we render a unit sphere under the same lighting (represented differently) in Fig. 2. To do so, we first fit a HDR environment map with K 128 SGs using equirectangular projection. As shown the left, SGs smooth out high frequency details and sharp corners but require much fewer parameters to reconstruct incident lighting (896 vs. 98 304). On the right, visualize the effect of increasing the surface roughness / under the corresponding light representation.
Intuitively, this point of diminishing return indicates that MC is only so useful when the surface reflects most of the incoming light (e.g., a mirror). Indeed, when 3 is small enough (3 - 0) and we deal with a highly non-Lambertian surface, small number of MC samples are enough to estimate direct illumination, which in turn implies faster render speed and low memory cost. On the other end of the spectrum (B + 1), significantly more samples (c.g., g., N > 1000) needed to accurately integrate incident light, resulting in longer inference times. In such a case, SGs should be favored since they offer significant improvement. In the absence of any prior knowledge the material type, SG shading is preferred. This reflected our experiments Sec. 5-6.
Optimization. We perform sanity check on our renderer in Fig. 3 by optimizing for lighting and reflectance properties from a multi-view image Li-loss with fixed geometry. We show the ground-truth (GT) parameters and rendered images in the first row, along with initial parameters (Init.) in the second row. Here, we optimize parameters separately using gradient-descent while the others are kept fixed to validate each component of our shading model. DIB-R++ can successfully estimate material and lighting parameters, including the environmental lighting, surface roughness and specular albedo of the object. We find that the converged material parameters closely match GT, while the optimized environment map loses some details due to gradients coming entirely from surface reflections (foreground supervision only). In particular, surface highlights are well captured by our technique. For more optimization results, please refer to the supplementary (Sec. C).
4 Application: Single Image 3D Reconstruction
We demonstrate the effectiveness of our hybrid framework through the learning-based problem of single image 3D reconstruction without supervision. While previous works [10, 62] generally focus on diffuse illumination only, our goal is to jointly infer geometry, reflectance, and lighting from a single image I containing strong specular transport. To this end, we employ convolutional neural network F, parameterized by learnable weights 0, to predict 3D attributes of a mesh M with pre-determined topology (sphere in our case). We adopt the U-Net [47] architecture
the original DIB-R [10, 62] and modify its output to also predict the appropriate BRDF attributes 0 and light parameters 7 (pixel colors or SG coefficients) so that F(: 9) (πr, 0, 7))) We then render these parameters back to image I using our differentiable renderer and apply loss L on the RGB output
5

---

to compare the input image I and the rendered image I, where:
[()) Oim [1, I) + Cmk (V, V) OperCqer T) + Chpphup (T).
Similar to DIB-R [10], we combine multiple consisteney losses with regularization terms: Zim is an image loss computing the d'istancce between the rendered image and the imput image. Lmsk is an Intersection-over-Unnii Union- (loU) loss of the rendered silhouette and the input mask V of the object [25], Eper is a perceptual loss [21, 61] computing the L -'iitttncee between the pre-trained AlexNet [26] feature maps of rendered image and input image, and Lusp is a Laplacian loss [36, 25] to penalize the change in relative positions of neighboring vertices. We set O'm = 20, mmssk 5.
Oper 0.5, Clap = 5, which we empirically found worked best.
5 Evaluation on Synthetic Datasets
We conduct extensive experiments to evaluate the performance of DIB-R++, We first quantitatively evaluate on synthetic data where we have access to ground-trute geometry. material and lighting. Since MC ad SG shading have individual pros and cons, we validate them under different settings. In particular, we generate separate datasets with two different surface materials: purely metallic surfaces with no roughness, and glossy surfaces with random positive roughness. We compare the performance of both shading models against the baseline method [10].
Synthetic Datasets. We chose 485 different car models from TurboSquid to prepare data for metallic and glossy surfaces. We also collected 438 freely available high-dynamic range (HDR) environment maps from HDRI Haven' use as reference lighting. which contain a wide variety of illumination configurations for both indoor and outdoor scenes. To render all 3D models, we use Blender's 's Cycles* path tracer with the Principled BRDF model [9]. We create two datasets. Metallic-Surfacee and Glossy-Surfacee For metallic surfaces, we set B = 0 and m = 1. Conversely, we set m = 0, s = 1 and randomly pick 3 E [0, 0.4] to generate images for glossy surfaces.
Baseline. We compare our method with the rasterization-basse baseline DIB-R [10], which sup- ports spherical harmonies (SH) lighting. While the original lighting implementation in [10] is monochromatic, we extend it to RGB for a fairer comparison. For quantitative evaluation, we first report the common L pixel loss between the re-rendered image using our predictions and ground- truth (GT) image( T. || 1),,), and 2D IoU loss between rendered silhouettes and ground-truth masks(L= = Voy, We experimentally find that these numbers are very close in different methods. Thus, we further evaluate the quality of diffuse albedo and lighting predictions using normalized cross correlation (NCC. L
Σ yormm where Ypred is the predicted albedo and light while 7 ll is GT). We provide more details of these metrics in the supplementary material (Sec. E).
5.1 Metallic Surfaces
Experimental Settings. We first ap- ply all methods to the metallic car dataset. Since this surface property is known a priori, we relax the task for MC shading by setting 3 0 and only predict geometry, diffuse albedo and lighting from the input image. This allows us to render MC at low sam- ple count (N = 4), achieving higher rendering speed and a lower memory cost. particular, we predict the relative offset for all |M| = 642 vertices in a mesh and a 256 x 256 texture map, following the choices in [10]. We also predict a 256 x 256 RGB environment map. For SG shading, we predict all parameters. While shape and texture are the same as MC shading, we adopt K = 32 for SG and predict two global parameters 3 and for the specular BRDF This keeps the number of parameters relatively low while providing enough flexibility to capture different radiometric configurations.
Shading
Metallic surfaces (1)
Glossy surfaces (L)
Image
2D IoU
Light
Tex.
Image
2D JoU
Light
Tox.
MC
0.019
0.062
0.074
0.152
0.024
0.061
0.106
0.142
SG SH [10]
0.019
0.069
0.085
0.218
0.024
0,057
0,091
0,140
0.019
0.056
0.220
0.206
0.024
0.062
0.131
0.152
Table 1: Quantitative results of single image 3D Reconstruction on synthetic data. While all the methods achieve comparable per- formance on re-rendered images and 2D IoUS, both MC and SG achieve better results on lighting and texture. MC is particularly better for metallic surfaces, and SG works best for glossy surfaces.
"nctps://turbosquid. com. We obtain consent via agreement with TurboSquid. following their license at https://bleg wtw.loog.urbbouuid.co com/ http://uubuuuuuddeee d-dd-model -iieelllleeneeeee 'nttpe:/hhdrihavenn com. We follow the CCO license at https: h/tt///aveennnn htteddiiiaenn.com/mmo wttp:///icess..ppppppp 'https://blender.orr org

---

RGB Iopur
GT albedo
GT lighting (envnup)
MC (Cn()
SG (Ours)
SH
Texture mup
Difluse albedo
Re-rendered (wiin Fight)
Texture map
Difluse Abedo
Re-rendered (witk light)
NIA
NIA
MC
50
[00
Figure 4; Prediction results on the metallic car dataset (3 = 0). While the re-rendered images look similar, the underlying components (material and lighting) are different. A SH basis [10] cannot recover the high frequency details of the sky light maps. In this case, MC performs best due to low variance the estimator for mirror-like BRDFs. SGs can recover the overall form and contrast of the light map, tend to predict incorrect texture maps incorporating the ground dominant color (e.g.. brown). Note that GT texture maps are not available as they cannot be compared due to different ae-pparameterization/ / texture atlases.
Experimental Results. Quantitative and qualitative results are shown in Fig. and Table (Left), respectively. Since the main loss function comes from the difference between GT and re-rendered images, we find the re-rendered images (with light) from the predictions are all close to the GT image for different methods, and quantitatively, the image loss and 2D IoU loss are also similar across different models. However, we observe significant differences on the predicted albedo and lighting. Specifically, in Fig. 4, MC shading successfully predicts cleaner diffuse albedo maps and more accurate lighting, while Chen et al. [10] "bakes in" high specular effects into the texture. Quantitatively, we outperform [10] with 3x improvement in terms of NCC loss for lighting, demonstrating the effectiveness of our DIB-R++ We further compare MC shading with SG shading. While SG shading achieves reasonable lighting predictions, it fails reconstruct the high frequency details in the lighting and has circular spot effects caused by the isotropic SGs. Finally, we note that due the ambiguity of the learning task, the overall intensities of all predicted texture maps can largely vary. Still, we observe that MC can better recover fine details, such as the wheels' rims.
5.2 Glossy Surfaces
Experimental Settings. We further apply model to synthetic images rendered with positive roughness (glossy). To apply MC shading in such a case, we assume no prior knowledge for material and use high sample count (N 1024) account for possibly low roughness images in the dataset. Due to high rendering time and memory cost, we subsample 4% of the pixels and apply Cim to those pixels only in each training iteration. As such, we do not the perceptual loss Lpe it relies the whole image. We predict 32 x 16 environment map for lighting and predict global /3 and m for the specular BRDF For SG shading, we use the same settings as for the metallic surfaces.
Experimental Results. We apply both MC and SG shading and compare with [10]. Results are shown in Fig. 5 and Table 1 (Right). Qualitatively, SG shading has better lighting predictions with correct high-luminance regions. The specular highlights in the images successfully guide SG shading, while the bright reflection the car window and front cover are fused with texture map in [10]. MC also has reasonable lighting predictions, but the predicted light map lacks structure due to weak surface reflections. Without perceptual loss term, the predicted textures also tend to be blurrier.
Quantitatively, SG shading significantly outperforms [10] on lighting prediction terms NCC (0.078 vs. 0.127) and improves on texture prediction in terms of BRDF/ighting disentanglement, We also compare with MC shading, where MC achieves slightly worse results on lighting predictions compared to SG, but is still much better than [10].
7

---

RGB
GT albedo
GT lighting (enenup)
MC (Cu()
SG (Ours)
SH [HO
Texture mup
Diflue albedo
Re-andred (with light)
Texture map
Diffuse albedo
Re-reddred (with light
NIA
NIA
MC
Figure 5: Prediction results on the glossy car dataset (8 > 0). When the objects are glossy but not perfectly specular, SGs correctly disentangle reflectance from lighting, as evidenced by the absence of white highlights the predict albedos. Chen al. [10] cannot capture these bright regions due a diffuse-only shading model, while MC oversmooths the predictions due to noisier pixel gradients. While methods cannot completely reconstruct the environment map, our method can predict the correct dominant light location and sky color.
5.3 Discussion
As shown in our previous two experiments, Monte Carlo shading works best under a metallic assumption (3 = 0), which case the rendered images can have rich details low sample count (N < 4). However, when the surface is more Lambertian (i.e., when 3 is becoming larger), we have to compensate with a larger to produce noise-free renderings, which impacts learning both time- and mesory-wise. As a consequence, we recommend applying MC shading metallic surfaces only, and default to SGs otherwise.
Our spherical Gaussian shading pipeline provides an analytic formulation for estimating the rendering equation, which avoids the need of tracing ray samples, largely accelerating the rendering process. While SGs can be blurry on metallic surfaces, in most case (e.g., when / 2 0.2) it can model similar rendering effects at a fraction of the cost, achieving better results than MC shading and [10].
After inspecting the predicted surface material properties (8, 8, m) and diffuse albedo with the ground-truth parameters in Blender, find the materials contain little correlation and the intensities of diffuse albedo might change. As for SG, we are using only 32 basis elements to simulate a complex, high definition environment map (2K). Since SGs can only represent finite amount of details, we find the predicted global / tends to be too small. One hypothesis for this is that the optimizer artificially prefers more reflections (and thus lower roughness) be able to estimate at least some portions of the environment map. On the other hand, in MC, due to the absence of perceptual loss, the predicted texture too blurry and cannot represent GT high detail. We find that the predicted 3 and m do not have strong correlation with the GT material. Lastly, we note that the predicted texture map has to change ensure the re-rendered images are correct, which leads to some differences with GT. More analysis can be found in supplementary (Sec. Cnd F).
overall intensity accommodate for other parameters to
In summary, when we have no prior knowledge about the material, our re-rendered images can be very close the input images but the predicted material parameters are not always aligned with the GT materials. We believe this problem can be relieved by incorporating additional local constraints, e.g., part-based material priors, or by leveraging anisotropic SGs [56]. For instance, car body is metallic while its wheels are typically diffuse; predicting different parameters for each region has the potential of improving disentanglement and interpretability. We leave this future work.
6 Evaluation on a Real-world Dataset
We further qualitatively evaluate method via training StyleGAN generated data and testing real imagery. following the pipeline in [62] in Sec. 6.1, and use our predictions to perform artistic manipulation in Sec. 6.2.

---

RGB Inpur
Decomposttion
Texture map
Envmup
aridia
Re-rendered (with light)
Phol thas on (Spant SKDF Ligo
50
SG
Figure 6: Results on real imagery from the SeeeGG--geeeetted dataset (cars and white female faces). Our method can recover meaningful decomposition opposed to [62]. as shown by cleaner texture maps and directional highlights (e.g.. windshield). Even when using monochromatic lighting faces, method correctly predict the specular highlights on forehead and none in hair, while SH produces dark artifacts.
6.1 Realistic Imagery
Experimental Settings. Our DIB-R++ can also be applied to learn 3D) properties from realistic imagery. Following [62],
StyleGAN [24] generate multi-view images of cars and faces, which the data we need to train our model. The generated objects contains cars under various lighting conditions, ranging from high specular paint to nearly diffuse. Thus, we only apply SG shading and adopt the same setting, where we predict [M] 642 vertex movements, a 256 x 256 diffuse texture map, K 32 SG bases and two global and We also compare [62] the baseline on the same dataset by using the same training procedure.
Experimental Results on StyleGAN Dataset. In the absence of ground-truie truth the StyleGAN generated data, we qualitatively evaluate our results and compare with [62] in Fig. 6. Our DIB-R++ reconstructs more faithful material and lighting components, producing interpretable decomposi- tion. Specifically, our model can represent the dominant light direction more accurately, while naive shading tend to merge reflectance with lighting. We also provide example with monochromatic lighting on a face example to reduce the degrees of freedom for the SH representation, yet it cannot correctly model light.
Extension to Real Imagery from LSUN Dataset. Our model trained on synthetic data [62] generated by StyleGAN [24]. Thanks to this powerful generative model, the distribution of GAN images is similar to the distribution of real images, allowing our model to generalize well. We show reconstruction results real images from LSUN [57] in Fig. 7. We provide more results in the supplementary (Sec. E), we also provide additional turntable videos on our project webpage.
During inference, we do not need any camera pose and predict shapes in canonical view. However, camera poses are needed to re-render the shape. Since ground-truth camera poses are not available for real images, we manually adjust the camera poses in Fig. 7. As result, the re-rendered images are slightly misaligned with GT. However, DIB-R++ still accounts for specularities and predicts correct predominant lighting directions and clean textures.
6.2 Material Editing and Relighting
Finally, we demonstrate some applications of DIB-R++ to artistic manipulation in Fig. 8. On the left, we show examples of editing the diffuse albedo, where we can insert text, decals or modify the base tint. Since our textures are not contaminated by lighting, clean texture maps can be easily edited by hand and the re-rendered images look natural. On the right, we show examples of editing lighting and surface materials, where rotate the light (top) or increase glossiness (bottom). We also showcase results where we change lighting orientation or modify the object's 's glossiness with consistent shading, which is not feasible with naive, Lambertian-only shading model.

---

RGB Input
Decomposition Pid abol Speasar REDEA Ligho
Texture
Fnymup
Pred. albedo
Re-rendered (with light)
Figure 7: Prediction on LSUN Dataset (Cars). DIB-R++, trained on StyleGAN dataset, can generalize well to
real images. Moreover, it also predicts correct high specular lighting directions and usable, clean textures.
RGB Input
Re-rendered
Edited diffuse albedo
RGB Input
Rotating enymap
WeuriPs
RGB Input
Re-rendered
Edited diffiuse albedo
RGB Input
Increusing specular albedo
Figure 8: Our SG method allows for artistic manipulation of appearance, such as novel view synthesis, material editing (both diffuse and specular components), and relighting, thanks to our effective disentanglement.
7 Conclusion
We presented DIB-R++, a hybrid differentiable renderer that can effectively disentangle material and lighting. When embedded in a learning framework for single image 3D reconstruction, our method produces state-of-the-an results, and enables applications such as material editing and relighting. One limitation of our method is that the predicted base color may sometimes "bleed" into the lighting predictions. Combining our technique with segmentation methods like DatasetGAN [63] could alleviate this issue for more practical, artist- frtist-friendly disentanglement. Moreover, the predicted reflections are sometimes blurrier than ground-truth; this is mainly due to a limited number SG components for lighting and could potentially be improved with larger mixture. Finally, on some occasions, the diffuse albedo in our synthetic dataset have baked-in reflections (e.g.. GT red car in Fig. 5) instead of uniform base color, which obfuscates the learning process. This could be mitigated by using more advanced physics-based materials such as those modeling clear coats.
Broader Impact. Our work focuses on disentangling geometry from appearance using differen- tiable renderer, a relatively nascent research area. We show that augmenting a rasterization based renderer with physics-based shading models improves reconstruction and allows for easier integration within larger machine learning pipelines. DIB-R++ relies on simple topology and strong data prior assumptions produce useful decompositions: therefore cannot generalize to the complexity and multi-modality of real-world scenes in its current form. Nonetheless, we believe that our work takes an important step in the joint estimation of shape, material and environmental lighting from single image and we hope that it can advance applications in performance-oriented settings such as AR/VR, simulation technology and robotics. For instance, autonomous vehicles need correctly assess their surroundings from limited signals; directly modeling light-surface interactions (e.g., specular highlights) may provide important cues this end.
Like any ML model, DJB-R++HR is prone to biases imparted through training data which requires an abundance of caution when applied to sensitive applications. For example, it needs to be carefully inspected when it used to recover the 3D parameters of human faces and bodies as it is not tailored for them. It is not recommended in off-the-shelf settings where privacy erroneous recognition can lead to potential misuse or any harmful application. For purposes of real deployment, one would need to carefully inspect and de- bias the dataset to depict the target distribution of a wide range of possible lighting conditions, skin tones, or at the intersection of race and gender.
Disclosure of Funding. This work was funded by NVIDIA. Wenzheng Chen, Jun Gao and Zian Wang acknowledge additional indirect revenue in the form of student scholarships from University of Toronto and the Vector Institute. Joey Litalien acknowledges indirect funding from McGill University and the Natural Sciences and Engineering Research Council of Canada (NSERC).
10

---

References
[1] Hassan Alhaija, Siva Mustikovela, Varun Jampani, Justus Thies, Matthias Niessner, Andreas Geiger, and Carsten Rother. Intrinsic autoencoders for joint neural rendering and intrinsic image decomposition. In International Conference on 3D Vision (3DV), 2020.
[2] Sai Bangaru, Tzu-Mao Li, and Frédo Durand. Unbiased warped-area sampling for differentiable rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 3966)245:11245:116 2020.
[3] Harry Barrow, J. Tenenbaum, A. Hanson, and E, Riseman. Recovering intrinsic scene character- istics. Comput. Vis. Syst, 2:3-26, 1978.
[4] Sai Bi, Nima Khademi Kalantari, and Ravi Ramamoorthi. Deep hybrid real and synthetic training for intrinsic decomposition. In Eurographics Symposium on Rendering: Experimental Ideas & Implementations (EGSR). 2018.
[5] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Gos Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition, 2020.
[6] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, and Ravi Ra- mamoorthi David Kriegman. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images, 2020.
[7] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik PA Lensch. NeRD: Neural reflectance decomposition from image collections, 2020.
[8] Mark Boss, Varun Jampani, Kihwan Kim, Hendrik Lensch, and Jan Kautz. Two-shot spatially- varying BRDF and shape estimation. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3982-3991, 2020.
[9] Brent Burley. Physically-based shading at Disney. In Pructical Physically- Shading in Film and Game Production (SIGGRAPH Course), pages 10:1-10:7, 2012.
[10] Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3D objects with an interpolation-based differentiable renderer. In Advances In Neural Information Processing Systems (NeurIPS), 2019.
[11] R. L. Cook and K. Torrance. reflectance model for computer graphics. ACM Trans. Graph., 1(1):7-24, January 1982.
[12] Michael Deering, Stephanie Winner, Bic Schediwy, Chris Duffy, and Neil Hunt. The triangle processor and normal vector shader: A vlsi system for high performance graphics. In Proc. of the 15th Anmual Conference Computer Graphics and Interactive Techniques, SIGGRAPH '88, page 21-30, 1988.
[13] Valentin Deschaintre, Miika Aittala, Frédo Durand, George Drettakis, and Adrien Bousseau. Single-image svbrdf capture with a rendering-aware deep network. ACM Transactions on Graphics (SIGGRAPH Conference Proceedings), 37(128): aug 2018.
[14] Yue Dong, Guojun Chen, Pieter Peers, Jiawan Zhang, and Xin Tong. Appearance-front frotimo Recovering spatially varying surface reflectance under unknown lighting. ACM Trans. Graph. (Proc. SIGGRAPII Asia), 33(6), November 2014.
[15] Brian Karis (Epic Games). Real shading in Unreal Engine 4. SIGGRAPH 2013 Courses, 2013.
[16] Jun Gao, Wenzheng Chen, Tommy Xiang, Clement Fuji Tsang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Learning deformable tetrahedral meshes for 3d reconstruction. In Advances In Neural Information Processing Systems, 2020.
[17] Marc-André Gardner, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Christian Gagné, and Jean- François Lalonde. Deep parametric indoor lighting estimation. Proceedings of the IEFE International Conference Computer Vision, pages 7175-7183, 2019.

---

[18] Mare-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and -ean-François Lalonde. Learning predict indoor illumination from single image. arXiv preprint arXiv: / 7700.000900 2017.
[19] Purvi Goel, Loudon Cohen, James Guesman, Vikas Thamizharasan, James Tompkin, and Daniel Ritchie. Shape from tracing: Towards reconstructing 3d object geometry and svbrdf material from images via differentiable path tracing. 2020 International Conference on 3D Vision
(3DV), pages 1 188-1195. IEEE, 2020.
[20] Yannick Hold-Geoffroy, Akshaya Athawale, and Jean-François Lalonde. Deep sky modeling for single image outdoor lighting estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6927-6953 2019.
[21] Phillip Isola, Jan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 1125- 1134,134, 2017. 2017.
[22] Krishna Murthy J., Edward Smith, Cann-Fracciis Lafleche, Clement Fuji Tsang, Artem Rozant- sev, Wenzheng Chen, Tommy Xiang, Rev Lebaredian, and Sanja Fidler. Kaolin: A pytorch library for accelerating 3d deep learning research. arXiv:1911.0506 2019.
[23] James T. Kajiya. The rendering equation. SIGGRAPH 866, page 143-150, 1986.
[24] Tero Karras, Samuli Laine, and Timo Aila. style-based generator architecture for generative adversarial networks. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4401 04000,44000000 2019.
[25] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3D mesh renderer. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[26] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Pereira, C. J. C. Burges, Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
[27] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for performrffee differentiable rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 39(6), 2020.
[28] Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. Image-based reconstruction of spatially varying materials. In Proc. of the 12th Eurographics Conference on Rendering, page 103-1 14, 2001.
[29] Hendrik P.A. Lensch, Jochen Lang, Asla M. Sa, and Hans-Peter Seidel. Planned sampling of spatially varying BRDFs. Computer Gruphics Forum, 2003.
[30] Louis Lettry, Kenneth Vanhoey, and Lue Van Gool. Unsupervised deep single-image intrinsic decomposition using illumination- -vryynn image sequences. In Computer Graphics Forum, volume 37, pages 409-419, 2018.
[31] Tzu-Mao Li, Miia Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable Monte Carlo ray tracing through edge sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia), (7(6):222:1 222:11, 1, 2018
[32] Zhengqi Li and Noah Snavely. Learning intrinsic image decomposition from watching the world. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9039-9048, 2018.
[33] Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2475-2484, 2020.
12

---

[34] Zhengqin Li, Kalyan Sunkavalli, and Manmohan Chandraker. Materials for masses: Svbrdf acquisition with a single mobile phone image. In Proceedings of the European Conference on Computer Vision (ECCV), pages 72-87, 2018.
[35] Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 37(6): 2018.
[36] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: differentiable renderer for amage-based 3D reasoning. The IEEE International Conference on Computer Vision (ICCV), Oct 2019.
[37] Yunfei Liu, Yu Li, Shaodi You, and Feng Lu. Unsupervised learning for intrinsic image decomposition from single image. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[38] Fujun Luan, Shuang Zhao, Kavita Bala, and Zhao Dong. Unified shape and svbrdf recovery using differentiable monte carlo rendering. ArXiv, 2021.
[39] Wei-Chiu Ma. Hang Chu, Bolei Zhou, Raquel Urtasun, and Antonio Torralba. Single image intrinsic decomposition without a single intrinsic image. In Proc. of the European Conference on Computer Vision (ECCV), pages 201-217, 2018
[40] Julian Meder and Beat Bruderlin. Hemispherical Gaussians for accurate light integration. Computer Vision and Graphics, 2018
[41] Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. Practical SVBRDF acquisition of 3D objects with unstructured flash photography. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 37 2018.
[42] Takuya Narihira, Michael Maire, and Stella X Yu. Direct intrinsics: Learning albedo-shading decomposition by convolutional regression. In Proc. of the IEEE International Conference on Computer Vision (ICCV), pages 2992-2992, 2015.
[43] Merlin Nimier-David, Zhao Dong, Wenzel Jakob, and Anton Kaplanyan. Material and Lighting Reconstruction for Complex Indoor Scenes with Texture-space Differentiable Rendering. In Adrien Bousseau and Morgan McGuire, editors, Eurographics Symposium on Rendering DL-only Track. The Eurographics Association, 2021.
[44] Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. Radiative backpropa- gation: An adjoint method for lightning-fant differentiable rendering. ACM Trans. Graph. (Proc. SIGGRAPH). 39(4,, July 2020.
[45] Merlin Nimier-David, Delio Vicini, Tizian Zelterr, and Wenzel Jakob. Mitsuba 2: A retargetable forward and inverse renderer. ACM Trans. Graph. (Proc. SIGGRAPH). 38(6), November 2019.
[46] Matt Pharr, Greg Humphreys, and Wenzel Jakob. Physically Based Rendering, Third Edition: From Theory To Implementation. Morgan Kaufmann Publishers Inc., 3rd edition, 2016.
[47] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical Image Computing and Computer-assiee assiiste Intervention, pages 234-241, 2015.
[48] Shen Sang and M. Chandraker. Single-shot neural relighting and svbrdf estimation. In ECCV, 2020.
[49] Jian Shi, Yue Dong, Hao Su, and Stella X. Yu. Learning non-Lambertian object intrinsics across ShapeNet categories. In Proc. of the IEFF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1685-1694, 2017.
[50] Eric Veach and Leonidas J. Guibas. Optimally combining sampling techniques for Monte Carlo rendering. Proceedings of the 22nd Annual Conference on Computer Gruphics and Interactive Techniques, SIGGRAPH '95, pages 419-428, 1995.
13

---

[51] Jiaping Wang, Peiran Ren, Minmin Gong, John Snyder. and Baining Guo. All-frequeney rendering of dynamic, contialll-varyinng reflectance. ACM Trans. Graph. (Proc. SIGGRAPH), 28(5):1-10, December 2009.
[52] Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz. Learning indoor inverse rendering with 3d spatially-varying lighting. In Proceedings of International Conference on Computer Vision (ICCV), 2021.
[53] Shangzhe Wu, Ameesh Makadia, Jiajun Wu, Noah Snavely, Richard Tucker, and Angjoo Kanazawa. De-rendering the world's revolutionary artefacts. arXiv preprint arXiv:2104.03954, 2021.
[54] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Unsupervised learning of probably symmetric deformable 3d objects from images in the wild. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-10, 2020.
[55] Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. Recovering shape and maatialy-vvayying surface reflectance under unknown illumination. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 35(6), 2016.
[56] Kun Xu, Wei-Lun Sun, Zhao Dong, Dan-Yong Zhao, Run-Dong Wu, and Shi-Min Hu. Anisotropic spherical gaussians. ACM Trans. Graph. (Proc. SIGGRAPH), 32(6), November 2013.
[57] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop, 2016.
[58] Ye Yu and William AP Smith. Inverserendernet: Learning single image inverse rendering. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[59] Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. Path-space differentiable rendering. ACM Truns. Graph. (Proc. SIGGRAPHI), 39(4): 13:11-143:9, 2020.
[60] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical gaussians forphysics-based material editing and relighting. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),202 2021.
[61] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[62] Yuxuan Zhang, Wenzheng Chen, Jun Gao, Huan Ling, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image GANs meet differentiable rendering for inverse graphics and interpretable 3D neural rendering. International Conference on Learning Representations (ICLR), 2020.
[63] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean- Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. DatasetGAN: Efficient labeled data factory with minimal human effort. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] We provided extensive experiment in Sec. 5, 6. and
6.2.
(b) Did you describe the limitations of your work? [Yes] See Sec. 7 and Broader Impact.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We provided the discussion in the Broader Impact section with further discussion in Supplementary Material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [No] The current implementation requires complex dependencies. We plan to release the code after refactoring.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See corresponding sections in main paper (Sec. 5, 6.1 and 6.2) and Supplementary Material.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] According to our experiments, our model converged well, and random seeds had little effect on performance.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] This is provided in the Supplementary Material.
4. If you are using existing assets (e.g., code, data, models) or curating/rrelasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We provided the references to the datasets in Sec. 5 and 6.1.
(b) Did you mention the license of the assets? [Yes] We provided the license of TarroSqqid and HDRI Haven datasets in Sec. 5.
(c) Did you include any new assets either in the supplemental material or as URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [Yes] We discussed this in Sec. 6.
(e) Did you discuss whether the data you are using/curating contains personally identi- fiable information offensive content? [Yes] We provided discussion this in the Supplementary Material.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "nv5s9Wj9YlJ", "reviewer": "Reviewer_FLZ6", "summary": "DIB-R++ proposes an extending DIB-R with a physics based shading model. The method predicts an environment map for lighting and uses a simplified version of isotropic Disney BRDF model. The shading model is single bounce and is estimated using either Monte Carlo methods or using spherical basis functions. With these modifications, the authors demonstrate that they are able to decompose the lighting and materials of single objects (cars, faces) from a single image better than DIB-R. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and well explained. The addition of more physically based shading to DIB-R seems like a logical and well motivated extension. The paper presents an interesting analysis of the pros and cons of using Monte Carlo versus Spherical Gaussians. From a computational complexity standpoint, the spherical gaussians approach makes a lot of sense and the qualitative results confirm that the impact on quality is minimal. I have listed some suggestions/questions/concerns below, overall I think this paper is marginally above the acceptance threshold.\n\na) It is noted in LN 251 that DIB-R can accurately re-render the image (albeit with incorrect albedo and lighting). From the quantitative and qualitative results it is fairly easy to conclude that DIB-R++ produces more physically plausible results, however, the usefulness of this improvement is never demonstrated to the reader. It would be informative to see the modifications in Fig 7 applied to DIB-R to hopefully demonstrate why this more complex method should be considered.  \n\nb) Topics like glossy/metallic and MC/SG are well analyzed in the paper, but other topics like geometry are barely discussed. It would be nice to know how the proposed modifications to DIB-R affect the predicted geometry. Additionally, it would be good to include metrics computed against the ground truth geometry. Table 1 is also missing a number of details, specifically what are these values actually measuring (Mean squared error?), please specify. Also consider reporting other common image metrics such as PSNR, SSIM, and LPIPS. \n\nc) The paper relies heavily on the qualitative results. The way they are currently presented makes it difficult to determine the effectiveness of the method. It would be beneficial to include examples of the ground truth albedo in the same poses as the predicted albedo example. Additionally it would be good to include ground truth examples of the scene re-rendered (with light).\n\nOther:\n+ Why is approximating the cosine foreshortening with a SG useful?\n+ LN 291. Can you provide these errors, I think it would be interesting to the reader to understand how far off these are.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do a good job of explaining the limitations throughout the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "hreu56Nf7k", "reviewer": "Reviewer_SWGM", "summary": "This paper proposes a single image inverse rendering method, named DIB-R++ that extends DIB-R [7] with MC/SG-based ray-tracing to accelerate training/inference. This method works in two-stage, first it uses DIB-R [7] to infer the mesh (and associated geometric attributes) from a single input image, then in the second stage, the network infers albedo and environment light jointly, and a direct light only shader renders a photorealistic image from the inferred geometry, material and lighting.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n\n1. The proposed DIB-R++ is unsupervised and does not require ground truth of geometry, material or environment light.\n   \n2. In experiments, the proposed DIB-R++ outperforms DIB-R [7] on both metallic and glossy surfaces.\n\n\nWeaknesses:\n\n1. My major concern about this paper is novelty. Using SG to accelerate training and inference is not a new idea, and it has been studied in NeRD [33] and PhySG [34], the difference is that the proposed DIB-R++ uses DIB-R [7] to recover a mesh from the input image, then infers material and illumination (represented by MC or SG, which is not novel). So I think the novelty is rather incremental compared with both NeRD [33] and PhySG [34] and DIB-R [7]. \n\n2. The paper mentioned single image 3D reconstruction in Section 4, but there were no such experimental evaluations in the paper. It is hard to tell the 3D reconstruction quality without quantitative evaluations on reconstructed mesh, depth or normal.\n\n3. This paper also lacks experimental comparisons with a state-of-the-art method, e.g., [ref1] Li et al. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. CVPR 2020. Moreover, the proposed DIB-R++ can only infer direct illumination, while [ref1] can also learn global illumination. I'd like to hear more discussion on this in rebuttal.\n\n4. The paper is not easy to follow, and it would be good to show a system diagram that describes network architecture, network input/output, data flow and training/inference steps.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See Weaknesses above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "SbVK88HB6Rm", "reviewer": "Reviewer_ShTU", "summary": "This paper proposes an improved pipeline for self-supervised training of a neural network that recovers the 3D properties (geometry, brdf, light) of an image containing a single object. This is achieved by proposing DIB-R++, a differentiable renderer which offers improved rendering speed at the cost of less accurate renderings. The DIB-R++ builds on top of the differentiable rasterization framework of DIB-R in order to add physically-based material and lighting models. DIB-R++ comes in 2 variations: A) MC -  relies on Monte Carlo sampling of incident light directions, which is better suited for reflective surfaces. B) SG - relies on using Spherical Gaussian for fast analytic integrations, which is better for rough surfaces. It is important to note that DIB-R++ assumes direct illumination only.\n\nExperimentations include analyzing the effects of forwards and backward passes through the DIB-R++ renderer in isolation (figures 2 and 3), and training neural networks on metal synthetic (section 5.1), glossy synthetic (section 5.2), and StyleGAN generated (section 6.1) datasets. This paper also contains interesting experiments to demonstrate the effects of using different variations and parameters of this renderer in terms of speed and rendering quality in the supplementary materials.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ This paper is well presented and experiments were nicely detailed.\n+ The experiment comparing using SG vs MC for the DIB-R++ rendering (figure 2) offers interesting insights on the suitability of each renderer based on the material roughness.\n+ The optimization task (figure 3) is reassuring that differentiating through DIB-R++ yields meaningful signals with respect to the material and lighting conditions\n+ Models trained with DIB-R++ seem to recover better light and material parameters compared to previous works.\n\nCons:\n- Contribution of paper should be stated more clearly - initially the renderer formulation (DIB-R++) seems to be the contribution, but I can not pinpoint the novelty from a graphics/rendering point of view as all of the used formulations were pre-existing and cited. After further reading, the main contribution seems to be in evaluating how the formulation of DIB-R++ fits into the context of training neural networks.\n- There is a claim of disentangling lighting and materials in the title and abstract of this paper, but I'm not sure what in the proposed formulation of DIB-R++ would cause for better disentanglement? Seems to be a better renderer for training neural networks in terms of less computations for better performance, but there isn't really anything in the formulation to encourage disentanglement. This is also hinted at by the authors in supplementary materials section C.1. Therefore I find the disentanglement term misleading.\n\nQuestions:\n- How does DIB-R++ compare to other differentiable renderers, e.g. the one from Mitsuba 2?\n- How does the training time of the models using DIB-R++ renderer compare to the methods of \"Learning to predict 3D objects with an interpolation-based differentiable renderer\" by Wenzheng Chen et al. and \"Image GANs meet differentiable rendering for inverse graphics and interpretable 3D neural rendering\" by Yuxuan Zhang et al.? Is it slower or faster to train? Does the model converge in less or more iterations? \n\nOverall, I like the experimentations of this paper and they demonstrate improved performance compared to previous methods. However, the contribution in the renderer does not seem significant enough to me for a NeurIPS paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would prefer if the fact that the training phase relies on multi-view images was stated more clearly, especially in the conclusion/limitations. This was briefly mentioned in the experimental settings of section 6.1. Also might want to mention that this approach is only suitable when we want to recover the geometry and material of a single 3D object captured in an image.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Discrimination / Bias / Fairness Concerns"], "time_spent_reviewing": "10"}, "extra_text": {"ethical_concerns": "One of the evaluations of this paper is on inferring the 3D properties (geometry and reflective parameters) of human faces. The authors have done a good job of acknowledging the biases that may be imposed on their models from the training data (section 7, broader impact). Since the face images from the dataset shown in the original paper/supplementary materials do not represent diversity of ethnicities very well, the decomposition might not produce correct results for other ethnicities (I have witnessed such failure cases in the past). However, it should be noted that recovering particularly human face 3D parameters is not the primary concern of this paper."}}, {"review_id": "9mh8q_N6hCo", "reviewer": "Reviewer_8HSK", "summary": "The paper proposes a method to reconstruct geometry, material and lighting from \na single image. They apply differentiable rasterizers to calculate the rendering loss, \nwhich is back-propagated to supervise the training of the desired components.\nThey compare different lighting representations include HDR envmaps as well as \nSpherical Gaussians.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n1. My major concern over the paper is the technical contribution of the paper. The \npaper builds on an existing rasterization-based inverse-rendering framework DIB-R.\nThe difference is that DIB-R assumes a Lambertian model, while the proposed method \nhandles specular materials. However, the overall framework is almost the same.       \nJointly optimizing lighting, materials, and geometry are also common, for example, \n[14].  The paper says [14] does not generalize to real images, and however [14] does \nshow results on real images and the paper does not compare to [14]. It is also not clear to \nme why the proposed method would generalize, since it is also trained on synthetic dataset.\nFor me, it seems the major contribution of the paper is that it adds the lighting, material in \nthe inverse-rendering framework of DIB-R, which I think is incremental. \n\n2. The paper has a lot of text on the shading model and lighting representation. While \nI appreciate the efforts, I have to say that both representations are pretty standard in \ncomputer graphics and such discussions do not increase the technical contributions. \n\n3. In terms of differentiable rasterizers, I think [8] has better performance \nthan [7] in terms of better handling occlusions and efficiency, why not use [8]?  \n\n4. The loss function only consists of losses on images and vertices. Since the model\nis trained on a synthetic dataset, why not also render ground truth material and lighting \nand use them as direct supervisions? Involving those GT in training would better resolve \nthe ambiguity in inverse rendering.  \n\n5. The inverse-rendering problem itself is highly ambiguous. In the case that the object \nis purely lambertian, there is no way to infer the lighting accurately, and such a problem \nexists for both SGs and envmaps. Therefore the proposed method is mostly limited to glossy \nsurfaces. \n\n6. In the results section, the paper provides no comparison against previous single-image lighting \nand SVBRDF estimation method, for example [14]. For lighting estimation, the following paper \nshould be cited and compared: Deep Parametric Indoor Lighting Estimation, ICCV 2019.\n\n7. In all results on synthetic data, the corresponding ground truth relighting images should \nbe included as references. Also there should be videos showing renderings under different lighting \nconditions and viewpoints with comparison to ground truth. \n\n\nOverall I think the technical contribution of the proposed method is not enough to be accepted. \nMost of the components are from previous works and the rasterizer-based optimization framework itself is also standard. Therefore, I vote for rejection. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See main reviews.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found this paper generally well written and interesting, but they had concerns related to novelty, clarity of presentation, and ease of understanding the technical underpinnings of how the model works. The authors addressed many of the concerns and misunderstandings in the rebuttal, but how it looks is that improving upon these points would require a second round of reviews, and thus rejecting the paper in its current form is the right thing to do.
Decision: Accept (Poster)