DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer
Authors: Wenzheng Chen
Joey Litalien
Jun Gao
Zian Wang
Clement Fuji Tsang
Sameh Khamis
Or Litany
Sanja Fidler
Venue: NeurIPS 2021 Poster (2021)
Keywords: differentiable rendering
inverse graphics
material estimation
lighting estimation

Abstract: We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable renderers. Many previous learning-based approaches for inverse graphics adopt rasterization-based renderers and assume naive lighting and material models, which often fail to account for non-Lambertian, specular reflections commonly observed in the wild. In this work, we propose DIBR++, a hybrid differentiable renderer which supports these photorealistic  effects by combining rasterization and ray-tracing, taking the advantage of their respective strengths---speed and realism. Our renderer incorporates environmental lighting and spatially-varying material models to efficiently approximate light transport, either through direct estimation or via spherical basis functions. Compared to more advanced physics-based differentiable renderers leveraging path tracing, DIBR++ is highly performant due to its compact and expressive shading model, which enables easy integration with learning frameworks for geometry, reflectance and lighting prediction from a single image without requiring any ground-truth. We experimentally demonstrate that our approach achieves superior material and lighting disentanglement on synthetic and real data compared to existing rasterization-based approaches and showcase several artistic applications including material editing and relighting. 

DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer
Wenzheng Chen 4,2,3
Joey Litalien
Jun Gao 1,2,3
Zian Wang" 1,2,3
Clement Fuji Tsang
Sameh Khamis
Or Litany
Sanja Fidler1.2.5
NVIDIA
University of Toronto
Vector Institute"
McGill University"
(zenzchen, jung, ziane, cfujitaang, akhania, olitany, stidier)eniidia.cmm coay.litalienn mogiil. ungl
Abstract
We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable renderers. Many previous learning Ceuning-bassed approaches for inverse graphics adopt musterization-baseee renderers and assume naive lighting and material models, which often fail to account for non-Lambertian, specular reflections commonly observed in the wild. In this work, we propose DIB- R++, a hybrid differentiable renderer which supports these photorealistic effects by combining rasterization and ray-tracing, taking the advantage of their respective strengths- speed and realism. Our renderer incorporates environmental lighting and spatially-varying material models to efficiently approximate light transport, either through direct estimation or via spherical basis functions. Compared to more advanced physics-based differentiable renderers leveraging path tracing, DIB-R++ is highly performant due to its compact and expressive shading model, which enables easy integration with learning frameworks for geometry. reflectance and lighting prediction from a single image without requiring any ground-trus -tutth We ex- perimentally demonstrate that approach achieves superior material and lighting disentanglement lemett on synthetic and real data compared to existing rasterization based approaches and showcase several artistic applications including material editing and relighting.
1 Introduction
Inferring intrinsic 3D properties from 2D images is long-standing goal of computer vision [3]. In recent years, differentiable rendering has shown great promise in estimating shape, reflectance and illumination from real photographs. Differentiable renderers have become natural candidates for learning-based inverse rendering applications, where image synthesis algorithms and neural networks can be jointly optimized to model physical aspects of objects from posed images, either by leveraging strong data priors or by directly modeling the interactions between light and surfaces.
Not all differentiable renderers rendering techniques [31, 45, 2, 44, 59] try to model the full light transport with proper visibility gradients, but they tend to require careful initialization of scene parameters and typically exhibit high computational cost which limits their usage in larger end-to-end learning pipelines. On the other hand, performance-oriees differentiable renderers [10, 27, 36, 25] trade physical accuracy for scalability and speed by approximating scene elements through neural representations or by employing simpler shading models. While the latter line of work has proven be successful in 3D scene reconstruction.
made equal. On the hand, recent physics-basee based differentiable
"Wrk done during internship at NVIDIA. Project page: https: //tp::://///eeee tlabs. gi thub wtt../..//BDRBD /DDIBRPlus.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

the frequent assumptions of Lambertian-on oobll surfaces and low- wwwfrrquenccy lighting prevent these works from modeling more complex specular transport commonly observed in the real world.
In this work, we consider the problem of single-view 3D object reconstruction without any 3D supervision. To this end, propose DIB-R++, hybrid differentiable renderer that combines rasterization and ray-tracing through an efficient deferred rendering framework. Our framework builds on top of DIB-R [10] and integrates thysics-based lighting and material models capture challenging non-Lambertian reflectance under unknown poses and illumination. Our method is versatile and supports both sunle-bouned ray- ray-tracing and spherical Gaussian representation for a compact approximation of direct illumination, allowing us to adapt and tune the shading model based on the radiometric complexity of the scene.
We validate our technique on both synthetic and real images and demonstrate superior performance on reconstructing realistic materials BRDFs and lighting configurations over prior rasterization-baase methods. We then follow the setting proposed in Zhang et al. [62] to show that DIB-R++ can reconstruct scene intrinsics also from real images without any 3D supervision. We further apply our framework to single-image appearance manipulation such as material editing and scene relighting.
2 Related Work
Differentiable Rendering, Research on differentiable rendering can be divided into two categories: poysics-basedi methods focusing on photorealistic image quality, and approximation methods aiming at higher performance. The former differentiates the forward light transport simulation [31, 45, 2, 44, 59] with careful handling of geometric discontinuities. While capable of supporting global illumination, these techniques tend to be relatively slow to optimize or require detailed initial description of the input in terms of geometry, materials, lighting and camera, which prevents their deployment in the wild. The latter line of works leverages simpler local shading models. Along this axis, rasterization-Be based differentiable renderers [10, 27, 36, 25, 16] approximate gradients by generating derivatives from projected pixels to 3D parameters. These methods are restricted to primary visibility and ignore indirect lighting effects by construction, but their simplicity and efficiency offer an attractive trade-off for 3D reconstruction. We follow this line of work and build atop DIB-R [10, 22] by augmenting its shading models with physics-based ones.
dearning-based Inverse Graphies. Recent research on inverse graphics targets the ill ill-posed prob- lem of jointly estimating geometry, reflectance and illumination from image observations using neural networks. For single image inverse rendering, one dominant approach is employ 2D CNNs to learn data-driven features and use synthetic data as supervision [42, 49, 35, 33, 52], but these methods do not always generalize complex real-world images [4]. To overcome the data issue, recent body of work investigates the use of self-supervised learning to recover scene intrinsics [1], including domain adaptation from synthetic reflectance dataset [37], object symmetry [54, 53], or multi-illumination images depicting the same scene [30, 32, 39, 58]. However, these methods either rely on specific priors or require data sources tedious to capture in practice. Some works tackle the subtask of lighting estimation only [18, 17, 20]. but still need to carcfully utilize training data that are hard to capture. Most similar to us, DIB-R R [10] tackles unsupervised inverse rendering in the context of differentiable rendering. Zhang et al. [62] further combines DIB-R with StyleGAN [24] generated images to extract and disentangle 3D knowledge. These works perform inverse rendering from real image collections without supervision, but may fail to capture complex material and lighting effects-in contrast, our method models these directly. Several techniques also try to handle more photorealistic effects but typically require complex capturing settings, such as controllable lighting (28, 29], a co-located camera- -faashighhh setup [41, 13, 34, 5, 6, 8, 48, 38], and densely captured multi-view images [14, 55, 7, 60] with additional known lighting [19] or hand-crafted inductive labels [43]. In our work, we propose a hybrid diffcrentiable renderer and learn to disentangle complex specular effects given single image. Similar to the recent NeRD [7] and PhySG [60] which recover non-Lambertian reflectance and illumination with a spherical Gaussian (SG) basis [51], we also employ SGs to model the SV-BRDF and incident lighting, but apply this representation mesh-based differentiable rendering with direct access to the surface.
3 Differentiable Deferred Rendering
In this section, we introduce DIB-R++, our differentiable rendering framework based on deferred shading [12]. DIB-R++ is hybrid differentiable renderer that can efficiently approximate direct illumination and synthesize high-quality images. Concretely, our renderer leverages the differentiable
2

---

M
Massi
Albedo
LA
Wo
(a)
(b)
(c)
(d)
renderer to obtain diffuse alhedo, surface normals and mask maps. In the shading pass (b), we then use these buffers to compute the incident radiance by sampling by representing lighting and the specular BRDF using spherical Gaussian basis. Depending the representation used in (c),
Figure 1: Overview. Given a 3D mesh M, we employ (a) a rasterization-
recover wide gamut of specular/glosy, appearances (d).
rasterization framework of DIB-R [10] to recover shape attributes and further employs physics-based material and lighting models estimate appearance.
3.1 Overview
We provide an overview of our technique in Fig. 1. We first rasterize 3D mesh to obtain diffuse albedo and material maps, surface normals, and silhouette mask. This information is deferred to the shading pass, where outgoing radiance is either estimated stochastically approximated using spherical Gaussian basis. The rasterizer and shader are differentiable by design, allowing gradients to be propagated to lighting, material and shape parameters for downstream learning tasks.
3.2 Background
Our goal is to provide a differentiable formulation of the rendering process to enable fast inverse rendering from 2D images. Let M be 3D object in a virtual scene. We start from the (non-emissive) rendering equation (RE) [23], which states that the outgoing radiance Lo at any surface point x E M in the camera direction Wo given by
Lo(x, = / fi Wi, Lx wi)|n w//dw, H22
(I)
where L; is the incident radiance, fr is the (spatially-varying) bidirectional reflectance distribution function (SV-BRDF) and n is the surface normal at x. The domain of integration the unit hemisphere 72 of incoming light directions illumination from different directions and is modulated by the cosine foreshortening term In w| Intuitively, Eq. (1) captures energy balance and computes how much light is received and scattered at a shading point in a particular direction.
The BRDF characterizes the surface's 's response to
Estimating the RE typically requires Monte Carlo (MC) integration [46], which involves tracing rays from the camera into the scene. Albeit physically correct, this process is computationally expensive and does not generally admit closed-form solution. MC estimators can exhibit high variance and may produce noisy pixel gradients at low sample count, which may significantly impact performance and convergence. To keep the problem tractable, we thus make several approximations of Eq. (1), which we detail in the next section.
3.3 Two-stage Deferred Rendering
We now describe rendering framework (Fig. 1). We start by defining three families of parameters, where π E Id encodes the shape attributes (e.g.. vertex positions), E Rdd describes the material properties, and y E Rdd captures the illumination in the scene. In what follows, shall only consider a single pixel, indexed by p. within an RGB image I E R 3x for notational simplicity.
Stage 1: Rasterization Pass. We first employ differentiable rasterizer R [10] to generate primary rays w. E S from camera and render our scene M into geometry buffers (commonly called G-buffers) containing the surface intersection point Xp R3, the surface normal np S and the apatially-varying matcrial parameters 0, (c.g.. diffuse albedo). This rendering pass also returns a visibility mask up e {0, 1} indicating whether pixel p is occupied by the rendered object, separating the foreground object Iç from its background environment T, so that / = I + Ib. We have:
R(M, p. w.) (Xp, Dp, Op, Up).
Stage 2: Shading Pass. Given surface properties and outgoing direction w., we then approximate the outgoing radiance [x(Xp, w.) through several key assumptions. First, we restrict ourselves to direct illumination only (i.e. single-bounce scattering) and assume that the incoming radiance given by distant environment map L, 2 + R². Therefore, we do not model self-occlusion and

---

L((x, (ixp:wi)== L((w) y). Such simplification largely reduces computation and memory costs and is trivially differentiable. Second, we assume that the material parameters 0 can model both diffuse and specular view-dependent effects. At high level, we define our shading model S so that:
(xp: np, wo: Pp. 7) F Lo (xp, wo).
Importantly, a differentiable parameterization of S enables the computation of pixel gradients with respect all scene parameters θ (,T, 0. 7) by differentiating ()) (S R) (M, p. w.). Given a scalar objective function defined on the rendered output I, 0I θπ is computed using DIB-R [10]. In what follows, we thus mainly focus on formulating ai ( 7 so that all gradients can be computed using the chain rule, allowing for joint optimization of geometry, material and lighting parameters. We assume henceforth a fixed pixel p for conciseness, and remove the subscript.
3.4 Shading Models
Since primary goal is to capture wide range of appearances, we provide two simple techniques to approximate Eq. (1): Monte Carlo (MC) and spherical Gaussians (SG). The former targets more mirror-like objects and can better approximate higher frequencies in the integrand, but is more expensive to compute. The latter is more robust to roughness variations but is limited by the number of basis elements. To model reflectance, we choose to use a simplified version of the isotropic Disney BRDF [9, 15] based on the Cook-TForracee model [11], which includes diffuse albedo a E [0, 1Js, specular albedo E [0, 1. surface roughness 3 E [0, 1] and metalness E [0, 1]. Metalness allows us to model both metals and plastics in a unified framework. We let the diffuse albedo vary spatially (a a(x) and globally define all other attributes to restrict the number of learnable parameters.
Monte Carlo Shading. Given a surface point x E .M to shade, we importance sample the BRDF to obtain N light directions w and compute the BRDF value. We represent the incident lighting L (MC) as a high-dynamic range image 7 E 2 R3xhu x using an equirectangular projection, which can be queried for any direction via interpolation between nearby pixels. The final pixel color is then computed as the average over all samples, divided by the probability of sampling wt::
N fr(x, w woi 0) I T.MO) (w 7) In w/|
S(MC) (x n. Wo: 0o: 0,7)
(4)
N
p(w²)
When the surface is near-specular (e.g., a mirror), one can efficiently estimate the RE as reflected rays are concentrated in bundles (e.g., to satisfy the law of reflection). However, this estimator can suffer from high variance for rougher surfaces; a higher number of samples may be necessary to produce usable gradients. While this can be partially improved with multiple importance sampling [50], emitter sampling would add a significant overhead due to the environment map being updated at every optimization step. This motivates the use of more compact representation.
Spherical Gaussian Shading. To further accelerate rendering while preserving expressivity in shading model, we use a spherical Gaussian (SG) [51] representation. Projecting both the cosine- weighted BRDF and incident radiance into an SG basis allows for fast, analytic integration within our differentiable shader, at the cost of some high frequency features in the integrand. Concretely, an SG kernel has the form G(w:E. A, 44) = fs e (E where w E S2 is the input spherical direction to evaluate, 8 is the axis, À E RR., is the sharpness, and μa E R.. is the amplitude of the lobe. We represent our environment map using a mixture of K lighting SGs Gr, so that:
K
L (SG) (wi 7) R Σ 9 (wis Af. pt),
(5)
where 7 : ( 1 14( Ja. For the BRDF, we follow Wang et al. [51] and fit single, monochromatic SG to the specular lobe so that (S))) is sum of diffuse and specular lobes. The full derivation can be found in our supplementary material (Sec. A). Finally, we approximate the cosine foreshortening term using a single SG [n wi| Ge(wi: n, 2.133, 1.17) [40]. Regrouping all terms, the final pixel color can be computed as:
S(SG) (x. n. wo: 0,,) = / f( (x, w., wo: 0) (5()) (wi) y) Ge(we) dui,
n.
(6)
which has an analytic form that can be automatically differentiated inside our renderer. All parameters of the SGs, as well as the BRDF parameters, are learnable.

---

Env. map (dense)
0.05
0.10
3 0.15 8
0.20
3
0.25
3
0.30
0.35
0.40
3 0.45
MC Bin.
K-
128 SGs
Figure 2: Visual comparisons between MC and SG shading. On the left, we show an environment map and its SG-ressesentttion (K 128). While losing sharp details, SGs only needs 1% parameters compared to the dense pixel HDR map. On the right, we increase roughness left-to-right, revealing that our spherical basis can correctly approximate direct illumination at maddraae-tt-hiige roughness.
Figure 3: Optimization. We optimize complex lighting and material for both MC shading (left) and SG shad- ing (right). In cach case, first optimize lighting and show the light and rendered image in first block. Next we optimize the surface mate- rial and show the surface ma- terial value and rendered im- age in second block.
Optimization of MC Shading Opt. Env. map
Optimization of SG Shading Opt. SG Light
Opt. |
Opt. 3,
05'05 oee-
orden
ornze
Opt.
Comparison. To visually compare our two shading techniques and understand their limitations, we render a unit sphere under the same lighting (represented differently) in Fig. 2. To do so, we first fit a HDR environment map with K 128 SGs using equirectangular projection. As shown the left, SGs smooth out high frequency details and sharp corners but require much fewer parameters to reconstruct incident lighting (896 vs. 98 304). On the right, visualize the effect of increasing the surface roughness / under the corresponding light representation.
Intuitively, this point of diminishing return indicates that MC is only so useful when the surface reflects most of the incoming light (e.g., a mirror). Indeed, when 3 is small enough (3 - 0) and we deal with a highly non-Lambertian surface, small number of MC samples are enough to estimate direct illumination, which in turn implies faster render speed and low memory cost. On the other end of the spectrum (B + 1), significantly more samples (c.g., g., N > 1000) needed to accurately integrate incident light, resulting in longer inference times. In such a case, SGs should be favored since they offer significant improvement. In the absence of any prior knowledge the material type, SG shading is preferred. This reflected our experiments Sec. 5-6.
Optimization. We perform sanity check on our renderer in Fig. 3 by optimizing for lighting and reflectance properties from a multi-view image Li-loss with fixed geometry. We show the ground-truth (GT) parameters and rendered images in the first row, along with initial parameters (Init.) in the second row. Here, we optimize parameters separately using gradient-descent while the others are kept fixed to validate each component of our shading model. DIB-R++ can successfully estimate material and lighting parameters, including the environmental lighting, surface roughness and specular albedo of the object. We find that the converged material parameters closely match GT, while the optimized environment map loses some details due to gradients coming entirely from surface reflections (foreground supervision only). In particular, surface highlights are well captured by our technique. For more optimization results, please refer to the supplementary (Sec. C).
4 Application: Single Image 3D Reconstruction
We demonstrate the effectiveness of our hybrid framework through the learning-based problem of single image 3D reconstruction without supervision. While previous works [10, 62] generally focus on diffuse illumination only, our goal is to jointly infer geometry, reflectance, and lighting from a single image I containing strong specular transport. To this end, we employ convolutional neural network F, parameterized by learnable weights 0, to predict 3D attributes of a mesh M with pre-determined topology (sphere in our case). We adopt the U-Net [47] architecture
the original DIB-R [10, 62] and modify its output to also predict the appropriate BRDF attributes 0 and light parameters 7 (pixel colors or SG coefficients) so that F(: 9) (πr, 0, 7))) We then render these parameters back to image I using our differentiable renderer and apply loss L on the RGB output
5

---

to compare the input image I and the rendered image I, where:
[()) Oim [1, I) + Cmk (V, V) OperCqer T) + Chpphup (T).
Similar to DIB-R [10], we combine multiple consisteney losses with regularization terms: Zim is an image loss computing the d'istancce between the rendered image and the imput image. Lmsk is an Intersection-over-Unnii Union-Union (loU) loss of the rendered silhouette and the input mask V of the object [25], Eper is a perceptual loss [21, 61] computing the L -'iitttncee between the pre-trained AlexNet [26] feature maps of rendered image and input image, and Lusp is a Laplacian loss [36, 25] to penalize the change in relative positions of neighboring vertices. We set O'm = 20, mmssk 5.
Oper 0.5, Clap = 5, which we empirically found worked best.
5 Evaluation on Synthetic Datasets
We conduct extensive experiments to evaluate the performance of DIB-R++, We first quantitatively evaluate on synthetic data where we have access to ground-trute geometry. material and lighting. Since MC ad SG shading have individual pros and cons, we validate them under different settings. In particular, we generate separate datasets with two different surface materials: purely metallic surfaces with no roughness, and glossy surfaces with random positive roughness. We compare the performance of both shading models against the baseline method [10].
Synthetic Datasets. We chose 485 different car models from TurboSquid to prepare data for metallic and glossy surfaces. We also collected 438 freely available high-dynamic range (HDR) environment maps from HDRI Haven' use as reference lighting. which contain a wide variety of illumination configurations for both indoor and outdoor scenes. To render all 3D models, we use Blender's 's Cycles* path tracer with the Principled BRDF model [9]. We create two datasets. Metallic-Surfacee and Glossy-Surfacee For metallic surfaces, we set B = 0 and m = 1. Conversely, we set m = 0, s = 1 and randomly pick 3 E [0, 0.4] to generate images for glossy surfaces.
Baseline. We compare our method with the rasterization-basse baseline DIB-R [10], which sup- ports spherical harmonies (SH) lighting. While the original lighting implementation in [10] is monochromatic, we extend it to RGB for a fairer comparison. For quantitative evaluation, we first report the common L pixel loss between the re-rendered image using our predictions and ground- truth (GT) image( T. || 1),,), and 2D IoU loss between rendered silhouettes and ground-truth masks(L= = Voy, We experimentally find that these numbers are very close in different methods. Thus, we further evaluate the quality of diffuse albedo and lighting predictions using normalized cross correlation (NCC. L
Σ yormm where Ypred is the predicted albedo and light while 7 ll is GT). We provide more details of these metrics in the supplementary material (Sec. E).
5.1 Metallic Surfaces
Experimental Settings. We first ap- ply all methods to the metallic car dataset. Since this surface property is known a priori, we relax the task for MC shading by setting 3 0 and only predict geometry, diffuse albedo and lighting from the input image. This allows us to render MC at low sam- ple count (N = 4), achieving higher rendering speed and a lower memory cost. particular, we predict the relative offset for all |M| = 642 vertices in a mesh and a 256 x 256 texture map, following the choices in [10]. We also predict a 256 x 256 RGB environment map. For SG shading, we predict all parameters. While shape and texture are the same as MC shading, we adopt K = 32 for SG and predict two global parameters 3 and for the specular BRDF This keeps the number of parameters relatively low while providing enough flexibility to capture different radiometric configurations.
Shading
Metallic surfaces (1)
Glossy surfaces (L)
Image
2D IoU
Light
Tex.
Image
2D JoU
Light
Tox.
MC
0.019
0.062
0.074
0.152
0.024
0.061
0.106
0.142
SG SH [10]
0.019
0.069
0.085
0.218
0.024
0,057
0,091
0,140
0.019
0.056
0.220
0.206
0.024
0.062
0.131
0.152
Table 1: Quantitative results of single image 3D Reconstruction on synthetic data. While all the methods achieve comparable per- formance on re-rendered images and 2D IoUS, both MC and SG achieve better results on lighting and texture. MC is particularly better for metallic surfaces, and SG works best for glossy surfaces.
"nctps://turbosquid. com. We obtain consent via agreement with TurboSquid. following their license at https://bleg wtw.loog.urbbouuid.co com/ http://uubuuuuuddeee d-dd-model -iieelllleeneeeee 'nttpe:/hhdrihavenn com. We follow the CCO license at https: h/ttri/avennnnn htteddiiiaenn.com/mmo wttp:///icess..ppppppp 'https://blender.orr org

---

RGB Iopur
GT albedo
GT lighting (envnup)
MC (Cn()
SG (Ours)
SH
Texture mup
Diflus albedo
Re-rendered (wiin Fight)
Texture map
Difluse Abedo
Re-rendered (witk light)
NIA
NIA
MC
50
[00
Figure 4; Prediction results on the metallic car dataset (3 = 0). While the re-rendered images look similar, the underlying components (material and lighting) are different. A SH basis [10] cannot recover the high frequency details of the sky light maps. In this case, MC performs best due to low variance the estimator for mirror-like BRDFs. SGs can recover the overall form and contrast of the light map, tend to predict incorrect texture maps incorporating the ground dominant color (e.g.. brown). Note that GT texture maps are not available as they cannot be compared due to different ae-pparameterization/ / texture atlases.
Experimental Results. Quantitative and qualitative results are shown in Fig. and Table (Left), respectively. Since the main loss function comes from the difference between GT and re-rendered images, we find the re-rendered images (with light) from the predictions are all close to the GT image for different methods, and quantitatively, the image loss and 2D IoU loss are also similar across different models. However, we observe significant differences on the predicted albedo and lighting. Specifically, in Fig. 4, MC shading successfully predicts cleaner diffuse albedo maps and more accurate lighting, while Chen et al. [10] "bakes in" high specular effects into the texture. Quantitatively, we outperform [10] with 3x improvement in terms of NCC loss for lighting, demonstrating the effectiveness of our
We further compare MC shading with SG shading. While SG shading achieves reasonable lighting predictions, it fails reconstruct the high frequency details in the lighting and has circular spot effects caused by the isotropic SGs. Finally, we note that due the ambiguity of the learning task, the overall intensities of all predicted texture maps can largely vary. Still, we observe that MC can better recover fine details, such as the wheels' rims.
5.2 Glossy Surfaces
Experimental Settings. We further apply model to synthetic images rendered with positive roughness (glossy). To apply MC shading in such a case, we assume no prior knowledge for material and use high sample count (N 1024) account for possibly low roughness images in the dataset. Due to high rendering time and memory cost, we subsample 4% of the pixels and apply Cim to those pixels only in each training iteration. As such, we do not the perceptual loss Lpe it relies the whole image. We predict 32 x 16 environment map for lighting and predict global /3 and m for the specular BRDF For SG shading, we use the same settings as for the metallic surfaces.
Experimental Results. We apply both MC and SG shading and compare with [10]. Results are shown in Fig. 5 and Table 1 (Right). Qualitatively, SG shading has better lighting predictions with correct high-luminance regions. The specular highlights in the images successfully guide SG shading, while the bright reflection the car window and front cover are fused with texture map in [10]. MC also has reasonable lighting predictions, but the predicted light map lacks structure due to weak surface reflections. Without perceptual loss term, the predicted textures also tend to be blurrier.
Quantitatively, SG shading significantly outperforms [10] on lighting prediction terms NCC (0.078 vs. 0.127) and improves on texture prediction in terms of BRDF/ighting disentanglement, We also compare with MC shading, where MC achieves slightly worse results on lighting predictions compared to SG, but is still much better than [10].
7

---

RGB
GT albedo
GT lighting (enenup)
MC (Cu()
SG (Ours)
SH [HO
Texture mup
Diflue albedo
Re-andred (with light)
Texture map
Diffuse albedo
Re-reddred (with light
NIA
NIA
MC
Figure 5: Prediction results on the glossy car dataset (8 > 0). When the objects are glossy but not perfectly specular, SGs correctly disentangle reflectance from lighting, as evidenced by the absence of white highlights the predict albedos. Chen al. [10] cannot capture these bright regions due a diffuse-only shading model, while MC oversmooths the predictions due to noisier pixel gradients. While methods cannot completely reconstruct the environment map, our method can predict the correct dominant light location and sky color.
5.3 Discussion
As shown in our previous two experiments, Monte Carlo shading works best under a metallic assumption (3 = 0), which case the rendered images can have rich details low sample count (N < 4). However, when the surface is more Lambertian (i.e., when 3 is becoming larger), we have to compensate with a larger to produce noise-free renderings, which impacts learning both time- and mesory-wise. As a consequence, we recommend applying MC shading metallic surfaces only, and default to SGs otherwise.
Our spherical Gaussian shading pipeline provides an analytic formulation for estimating the rendering equation, which avoids the need of tracing ray samples, largely accelerating the rendering process. While SGs can be blurry on metallic surfaces, in most case (e.g., when / 2 0.2) it can model similar rendering effects at a fraction of the cost, achieving better results than MC shading and [10].
After inspecting the predicted surface material properties (8, 8, m) and diffuse albedo with the ground-truth parameters in Blender, find the materials contain little correlation and the intensities of diffuse albedo might change. As for SG, we are using only 32 basis elements to simulate a complex, high definition environment map (2K). Since SGs can only represent finite amount of details, we find the predicted global / tends to be too small. One hypothesis for this is that the optimizer artificially prefers more reflections (and thus lower roughness) be able to estimate at least some portions of the environment map. On the other hand, in MC, due to the absence of perceptual loss, the predicted texture too blurry and cannot represent GT high detail. We find that the predicted 3 and m do not have strong correlation with the GT material. Lastly, we note that the predicted texture map has to change ensure the re-rendered images are correct, which leads to some differences with GT. More analysis can be found in supplementary (Sec. Cnd F).
overall intensity accommodate for other parameters to
In summary, when we have no prior knowledge about the material, our re-rendered images can be very close the input images but the predicted material parameters are not always aligned with the GT materials. We believe this problem can be relieved by incorporating additional local constraints, e.g., part-based material priors, or by leveraging anisotropic SGs [56]. For instance, car body is metallic while its wheels are typically diffuse; predicting different parameters for each region has the potential of improving disentanglement and interpretability. We leave this future work.
6 Evaluation on a Real-world Dataset
We further qualitatively evaluate method via training StyleGAN generated data and testing real imagery. following the pipeline in [62] in Sec. 6.1, and use our predictions to perform artistic manipulation in Sec. 6.2.

---

RGB Inpur
Decomposttion
Texture map
Envmup
aridia
Re-rendered (with light)
Phol thas on (Spant SKDF Ligo
50
SG
Figure 6: Results on real imagery from the SeeeGG--geeeetted dataset (cars and white female faces). Our method can recover meaningful decomposition opposed to [62]. as shown by cleaner texture maps and directional highlights (e.g.. windshield). Even when using monochromatic lighting faces, method correctly predict the specular highlights on forehead and none in hair, while SH produces dark artifacts.
6.1 Realistic Imagery
Experimental Settings. Our DIB-R++ can also be applied to learn 3D) properties from realistic imagery. Following [62],
StyleGAN [24] generate multi-view images of cars and faces, which the data we need to train our model. The generated objects contains cars under various lighting conditions, ranging from high specular paint to nearly diffuse. Thus, we only apply SG shading and adopt the same setting, where we predict [M] 642 vertex movements, a 256 x 256 diffuse texture map, K 32 SG bases and two global and We also compare [62] the baseline on the same dataset by using the same training procedure.
Experimental Results on StyleGAN Dataset. In the absence of ground-truie truth the StyleGAN generated data, we qualitatively evaluate our results and compare with [62] in Fig. 6. Our DIB-R++ reconstructs more faithful material and lighting components, producing interpretable decomposi- tion. Specifically, our model can represent the dominant light direction more accurately, while naive shading tend to merge reflectance with lighting. We also provide example with monochromatic lighting on a face example to reduce the degrees of freedom for the SH representation, yet it cannot correctly model light.
Extension to Real Imagery from LSUN Dataset. Our model trained on synthetic data [62] generated by StyleGAN [24]. Thanks to this powerful generative model, the distribution of GAN images is similar to the distribution of real images, allowing our model to generalize well. We show reconstruction results real images from LSUN [57] in Fig. 7. We provide more results in the supplementary (Sec. E), we also provide additional turntable videos on our project webpage.
During inference, we do not need any camera pose and predict shapes in canonical view. However, camera poses are needed to re-render the shape. Since ground-truth camera poses are not available for real images, we manually adjust the camera poses in Fig. 7. As result, the re-rendered images are slightly misaligned with GT. However, DIB-R++ still accounts for specularities and predicts correct predominant lighting directions and clean textures.
6.2 Material Editing and Relighting
Finally, we demonstrate some applications of DIB-R++ to artistic manipulation in Fig. 8. On the left, we show examples of editing the diffuse albedo, where we can insert text, decals or modify the base tint. Since our textures are not contaminated by lighting, clean texture maps can be easily edited by hand and the re-rendered images look natural. On the right, we show examples of editing lighting and surface materials, where rotate the light (top) or increase glossiness (bottom). We also showcase results where we change lighting orientation or modify the object's 's glossiness with consistent shading, which is not feasible with naive, Lambertian-only shading model.

---

RGB Input
Decomposition Pid abol Speasar REDEA Ligho
Texture
Fnymup
Pred. albedo
Re-rendered (with light)
Figure 7: Prediction on LSUN Dataset (Cars). DIB-R++, trained on StyleGAN dataset, can generalize well to
real images. Moreover, it also predicts correct high specular lighting directions and usable, clean textures.
RGB Input
Re-rendered
Edited diffuse albedo
RGB Input
Rotating enymap
WeuriPs
RGB Input
Re-rendered
Edited diffiuse albedo
RGB Input
Increusing specular albedo
Figure 8: Our SG method allows for artistic manipulation of appearance, such as novel view synthesis, material editing (both diffuse and specular components), and relighting, thanks to our effective disentanglement.
7 Conclusion
We presented DIB-R++, a hybrid differentiable renderer that can effectively disentangle material and lighting. When embedded in a learning framework for single image 3D reconstruction, our method produces state-of-the-an results, and enables applications such as material editing and relighting. One limitation of our method is that the predicted base color may sometimes "bleed" into the lighting predictions. Combining our technique with segmentation methods like DatasetGAN [63] could alleviate this issue for more practical, artist- frtist-friendly disentanglement. Moreover, the predicted reflections are sometimes blurrier than ground-truth; this is mainly due to a limited number SG components for lighting and could potentially be improved with larger mixture. Finally, on some occasions, the diffuse albedo in our synthetic dataset have baked-in reflections (e.g.. GT red car in Fig. 5) instead of uniform base color, which obfuscates the learning process. This could be mitigated by using more advanced physics-based materials such as those modeling clear coats.
Broader Impact. Our work focuses on disentangling geometry from appearance using differen- tiable renderer, a relatively nascent research area. We show that augmenting a rasterization based renderer with physics-based shading models improves reconstruction and allows for easier integration within larger machine learning pipelines. DIB-R++ relies on simple topology and strong data prior assumptions produce useful decompositions: therefore cannot generalize to the complexity and multi-modality of real-world scenes in its current form. Nonetheless, we believe that our work takes an important step in the joint estimation of shape, material and environmental lighting from single image and we hope that it can advance applications in performance-oriented settings such as AR/VR, simulation technology and robotics. For instance, autonomous vehicles need correctly assess their surroundings from limited signals; directly modeling light-surface interactions (e.g., specular highlights) may provide important cues this end.
Like any ML model, DJB-R++HR is prone to biases imparted through training data which requires an abundance of caution when applied to sensitive applications. For example, it needs to be carefully inspected when it used to recover the 3D parameters of human faces and bodies as it is not tailored for them. It is not recommended in off-the-shelf settings where privacy erroneous recognition can lead to potential misuse or any harmful application. For purposes of real deployment, one would need to carefully inspect and de- bias the dataset to depict the target distribution of a wide range of possible lighting conditions, skin tones, or at the intersection of race and gender.
Disclosure of Funding. This work was funded by NVIDIA. Wenzheng Chen, Jun Gao and Zian Wang acknowledge additional indirect revenue in the form of student scholarships from University of Toronto and the Vector Institute. Joey Litalien acknowledges indirect funding from McGill University and the Natural Sciences and Engineering Research Council of Canada (NSERC).
10

---

References
[1] Hassan Alhaija, Siva Mustikovela, Varun Jampani, Justus Thies, Matthias Niessner, Andreas Geiger, and Carsten Rother. Intrinsic autoencoders for joint neural rendering and intrinsic image decomposition. In International Conference on 3D Vision (3DV), 2020.
[2] Sai Bangaru, Tzu-Mao Li, and Frédo Durand. Unbiased warped-area sampling for differentiable rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 3966)245:11245:116 2020.
[3] Harry Barrow, J. Tenenbaum, A. Hanson, and E, Riseman. Recovering intrinsic scene character- isties. Comput. Vis. Syst, 2:3-26, 1978.
[4] Sai Bi, Nima Khademi Kalantari, and Ravi Ramamoorthi. Deep hybrid real and synthetic training for intrinsic decomposition. In Eurographics Symposium on Rendering: Experimental Ideas & Implementations (EGSR). 2018.
[5] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Gos Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition, 2020.
[6] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, and Ravi Ra- mamoorthi David Kriegman. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images, 2020.
[7] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik PA Lensch. NeRD: Neural reflectance decomposition from image collections, 2020.
[8] Mark Boss, Varun Jampani, Kihwan Kim, Hendrik Lensch, and Jan Kautz. Two-shot spatially- varying BRDF and shape estimation. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3982-3991, 2020.
[9] Brent Burley. Physically-based shading at Disney. In Pructical Physically- Shading in Film and Game Production (SIGGRAPH Course), pages 10:1-10:7, 2012.
[10] Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3D objects with an interpolation-based differentiable renderer. In Advances In Neural Information Processing Systems (NeurIPS), 2019.
[11] R. L. Cook and K. Torrance. reflectance model for computer graphics. ACM Trans. Graph., 1(1):7-24, January 1982.
[12] Michael Deering, Stephanie Winner, Bic Schediwy, Chris Duffy, and Neil Hunt. The triangle processor and normal vector shader: A vlsi system for high performance graphics. In Proc. of the 15th Anmual Conference Computer Graphics and Interactive Techniques, SIGGRAPH '88, page 21-30, 1988.
[13] Valentin Deschaintre, Miika Aittala, Frédo Durand, George Drettakis, and Adrien Bousseau. Single-image svbrdf capture with a rendering-aware deep network. ACM Transactions on Graphics (SIGGRAPH Conference Proceedings), 37(128): aug 2018.
[14] Yue Dong, Guojun Chen, Pieter Peers, Jiawan Zhang, and Xin Tong. Appearance-front frotimo Recovering spatially varying surface reflectance under unknown lighting. ACM Trans. Graph. (Proc. SIGGRAPII Asia), 33(6), November 2014.
[15] Brian Karis (Epic Games). Real shading in Unreal Engine 4. SIGGRAPH 2013 Courses, 2013.
[16] Jun Gao, Wenzheng Chen, Tommy Xiang, Clement Fuji Tsang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Learning deformable tetrahedral meshes for 3d reconstruction. In Advances In Neural Information Processing Systems, 2020.
[17] Marc-André Gardner, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Christian Gagné, and Jean- François Lalonde. Deep parametric indoor lighting estimation. Proceedings of the IEFE International Conference Computer Vision, pages 7175-7183, 2019.

---

[18] Mare-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and -ean-François Lalonde. Learning predict indoor illumination from single image. arXiv preprint arXiv: / 7700.000900 2017.
[19] Purvi Goel, Loudon Cohen, James Guesman, Vikas Thamizharasan, James Tompkin, and Daniel Ritchie. Shape from tracing: Towards reconstructing 3d object geometry and svbrdf material from images via differentiable path tracing. 2020 International Conference on 3D Vision
(3DV), pages 1 189-1195. IEEE, 2020.
[20] Yannick Hold-Geoffroy, Akshaya Athawale, and Jean-François Lalonde. Deep sky modeling for single image outdoor lighting estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6927-6953 2019.
[21] Phillip Isola, Jan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 1125-11344 2017.
2017.
[22] Krishna Murthy J., Edward Smith, Cann-Fracciis Lafleche, Clement Fuji Tsang, Artem Rozant- sev, Wenzheng Chen, Tommy Xiang, Rev Lebaredian, and Sanja Fidler. Kaolin: A pytorch library for accelerating 3d deep learning research. arXiv:1911.0506 2019.
[23] James T. Kajiya. The rendering equation. SIGGRAPH 866, page 143-150, 1986.
[24] Tero Karras, Samuli Laine, and Timo Aila. style-based generator architecture for generative adversarial networks. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4401 04000,44000000 2019.
[25] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3D mesh renderer. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018
[26] Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Pereira, C. J. C. Burges, Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012.
[27] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for performrffee differentiable rendering. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 39(6), 2020.
[28] Hendrik P. A. Lensch, Jan Kautz, Michael Goesele, Wolfgang Heidrich, and Hans-Peter Seidel. Image-based reconstruction of spatially varying materials. In Proc. of the 12th Eurographics Conference on Rendering, page 103-1 14, 2001.
[29] Hendrik P.A. Lensch, Jochen Lang, Asla M. Sa, and Hans-Peter Seidel. Planned sampling of spatially varying BRDFs. Computer Gruphics Forum, 2003.
[30] Louis Lettry, Kenneth Vanhoey, and Lue Van Gool. Unsupervised deep single-image intrinsic decomposition using illumination- -vryynn image sequences. In Computer Graphics Forum, volume 37, pages 409-419, 2018.
[31] Tzu-Mao Li, Miia Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable Monte Carlo ray tracing through edge sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia), (7(6):222:1 222:11, 1, 2018
[32] Zhengqi Li and Noah Snavely. Learning intrinsic image decomposition from watching the world. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9039-9048, 2018.
[33] Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2475-2484, 2020.
12

---

[34] Zhengqin Li, Kalyan Sunkavalli, and Manmohan Chandraker. Materials for masses: Svbrdf acquisition with a single mobile phone image. In Proceedings of the European Conference on Computer Vision (ECCV), pages 72-87, 2018.
[35] Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 37(6): 2018.
[36] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: differentiable renderer for amage-based 3D reasoning. The IEEE International Conference on Computer Vision (ICCV), Oct 2019.
[37] Yunfei Liu, Yu Li, Shaodi You, and Feng Lu. Unsupervised learning for intrinsic image decomposition from single image. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[38] Fujun Luan, Shuang Zhao, Kavita Bala, and Zhao Dong. Unified shape and svbrdf recovery using differentiable monte carlo rendering. ArXiv, 2021.
[39] Wei-Chiu Ma. Hang Chu, Bolei Zhou, Raquel Urtasun, and Antonio Torralba. Single image intrinsic decomposition without a single intrinsic image. In Proc. of the European Conference on Computer Vision (ECCV), pages 201-217, 2018
[40] Julian Meder and Beat Bruderlin. Hemispherical Gaussians for accurate light integration. Computer Vision and Graphics, 2018
[41] Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. Practical SVBRDF acquisition of 3D objects with unstructured flash photography. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 37 2018.
[42] Takuya Narihira, Michael Maire, and Stella X Yu. Direct intrinsics: Learning albedo-shading decomposition by convolutional regression. In Proc. of the IEEE International Conference on Computer Vision (ICCV), pages 2992-2992, 2015.
[43] Merlin Nimier-David, Zhao Dong, Wenzel Jakob, and Anton Kaplanyan. Material and Lighting Reconstruction for Complex Indoor Scenes with Texture-space Differentiable Rendering. In Adrien Bousseau and Morgan McGuire, editors, Eurographics Symposium on Rendering DL-only Track. The Eurographics Association, 2021.
[44] Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. Radiative backpropa- gation: An adjoint method for lightning-fant differentiable rendering. ACM Trans. Graph. (Proc. SIGGRAPH). 39(4,, July 2020.
[45] Merlin Nimier-David, Delio Vicini, Tizian Zelterr, and Wenzel Jakob. Mitsuba 2: A retargetable forward and inverse renderer. ACM Trans. Graph. (Proc. SIGGRAPH). 38(6), November 2019.
[46] Matt Pharr, Greg Humphreys, and Wenzel Jakob. Physically Based Rendering, Third Edition: From Theory To Implementation. Morgan Kaufmann Publishers Inc., 3rd edition, 2016.
[47] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical Image Computing and Computer-assiee assiiste Intervention, pages 234-241, 2015.
[48] Shen Sang and M. Chandraker. Single-shot neural relighting and svbrdf estimation. In ECCV, 2020.
[49] Jian Shi, Yue Dong, Hao Su, and Stella X. Yu. Learning non-Lambertian object intrinsics across ShapeNet categories. In Proc. of the IEFF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1685-1694, 2017.
[50] Eric Veach and Leonidas J. Guibas. Optimally combining sampling techniques for Monte Carlo rendering. Proceedings of the 22nd Annual Conference on Computer Gruphics and Interactive Techniques, SIGGRAPH '95, pages 419-428, 1995.
13

---

[51] Jiaping Wang, Peiran Ren, Minmin Gong, John Snyder. and Baining Guo. All-frequeney rendering of dynamic, contialll-varyinng reflectance. ACM Trans. Graph. (Proc. SIGGRAPH), 28(5):1-10, December 2009.
[52] Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz. Learning indoor inverse rendering with 3d spatially-varying lighting. In Proceedings of International Conference on Computer Vision (ICCV), 2021.
[53] Shangzhe Wu, Ameesh Makadia, Jiajun Wu, Noah Snavely, Richard Tucker, and Angjoo Kanazawa. De-rendering the world's revolutionary artefacts. arXiv preprint arXiv:2104.03954, 2021.
[54] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Unsupervised learning of probably symmetric deformable 3d objects from images in the wild. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-10, 2020.
[55] Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. Recovering shape and maatialy-vvayying surface reflectance under unknown illumination. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 35(6), 2016.
[56] Kun Xu, Wei-Lun Sun, Zhao Dong, Dan-Yong Zhao, Run-Dong Wu, and Shi-Min Hu. Anisotropic spherical gaussians. ACM Trans. Graph. (Proc. SIGGRAPH), 32(6), November 2013.
[57] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop, 2016.
[58] Ye Yu and William AP Smith. Inverserendernet: Learning single image inverse rendering. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[59] Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. Path-space differentiable rendering. ACM Truns. Graph. (Proc. SIGGRAPHI), 39(4): 13:11-143:9, 2020.
[60] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical gaussians forphysics-based material editing and relighting. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),202 2021.
[61] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[62] Yuxuan Zhang, Wenzheng Chen, Jun Gao, Huan Ling, Yinan Zhang, Antonio Torralba, and Sanja Fidler. Image GANs meet differentiable rendering for inverse graphics and interpretable 3D neural rendering. International Conference on Learning Representations (ICLR), 2020.
[63] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean- Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. DatasetGAN: Efficient labeled data factory with minimal human effort. In Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] We provided extensive experiment in Sec. 5, 6. and
6.2.
(b) Did you describe the limitations of your work? [Yes] See Sec. 7 and Broader Impact.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We provided the discussion in the Broader Impact section with further discussion in Supplementary Material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [No] The current implementation requires complex dependencies. We plan to release the code after refactoring.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See corresponding sections in main paper (Sec. 5, 6.1 and 6.2) and Supplementary Material.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] According to our experiments, our model converged well, and random seeds had little effect on performance.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] This is provided in the Supplementary Material.
4. If you are using existing assets (e.g., code, data, models) or curating/rrelasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We provided the references to the datasets in Sec. 5 and 6.1.
(b) Did you mention the license of the assets? [Yes] We provided the license of TarroSqqid and HDRI Haven datasets in Sec. 5.
(c) Did you include any new assets either in the supplemental material or as URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [Yes] We discussed this in Sec. 6.
(e) Did you discuss whether the data you are using/curating contains personally identi- fiable information offensive content? [Yes] We provided discussion this in the Supplementary Material.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "wsFZ70LrNv9", "reviewer": "Reviewer_FzMA", "summary": "This paper tackles the problem of learning to recover 3D shape, surface materials and environment lighting from a single image. It extends a recent learning-based single-image 3D reconstruction framework with realistic BRDF and shading modules that allows it to automatically disentangle material and lighting, using the same analysis-by-synthesis pipeline without relying on ground-truth data for training.\n\nThe paper presents extensive analysis and evaluations on both synthetic and realistic GAN-generated images, demonstrating impressive results on the disentanglement. This result presents a significant step forward in the field of 3D reconstruction and inverse graphics, by putting realistic material and lighting models in an unsupervised learning framework for general objects (if my understanding of training and testing setup is correct, see W2).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n### S1. Method and results\nThis paper presents very impressive results on single-image inverse graphics of general objects. Existing work can be grouped into three categories:\n1. unsupervised 3D reconstruction assuming simplistic material and lighting;\n2. supervised methods that also recovers sophisticated material and lighting on synthetic scenes;\n3. optimization-based photometric stereo methods.\n\nThis paper is the first one (to my best knowledge) to show that sophisticated material and lighting can be learned from only image collections of relatively realistic scenes in an analysis-by-synthesis framework, similar to those in unsupervised 3D reconstruction. It goes beyond existing methods, and presents an exciting path towards the goal of general inverse graphics in the real world.\n\n### S2. Extensive evaluation\nThe paper presents a large collection of analysis and evaluation experiments, including experiments on toy examples showcasing the difference of the different variants, and extensive comparisons with previous methods on various datasets. In particular, it shows the proposed framework can be easily adapted and produce robust results in various scenarios.\n\n### S3. Writing\nThe paper is well written. I enjoyed reading it. Some technical details are missing in the main paper due to the lack of space, but the supplementary material resolved many of my questions.\n\n\n## Weaknesses\n### W1. (minor) Limitations\nThe model does not deal with spatially-varying materials (except diffuse albedo). But that has even more ambiguity and would be super challenging to disentangle from a single RGB image.\n\n### W2. Clarifications required\nThere is no figure for the training pipeline, and some training details are missing. Below are some details that may require further clarifications.\n- Could the authors clarify what kind of data is required during training (multi-views, masks, known camera?) and what is the input during inference (only one single image, without masks, unknown camera?)?\n- Does the model require known camera viewpoint (or object pose), at training or at both training and inference? I did not find it explicitly mentioned in the paper. But the predicted environment maps seem to perfectly align with the ground truth ones, which is very impressive, as it would have so much ambiguity to recover both the object pose and orientation of the environment lighting at the same time. I did not find clear descriptions in its predecessor DIB-R either.\n- Decomposing albedo and lighting from a single image is fundamentally ambiguous. The recovered albedo images look highly plausible even when some regions are saturated by specular reflections, eg Fig 4, 5, 6. Why is this possible? The paper describes the method very well, but I'm still missing the key insight in the success of the method. For example, why wouldn't the model put all shading in the albedo image? Are there any additional constraints or regularization? Or is it the \"learning\" part that is the key here, as the model learns category-specific priors over a collection of training images?\n- I might have more follow-up questions after these are clarified.\n\n\n## Additional questions / comments\n- I would highly recommend releasing the code, which will hugely facilitate future work in inverse graphics.\n- The paper probably should not stress \"spatially-varying BRDF\" as the roughness, metalness and specular albedo terms are all global, and only the diffuse albedo is spatially varying.\n- I suppose the paper does not consider no spatially varying lighting. I was confused because in line 135, it states \"We assume henceforth a fixed pixel p, and remove the subscript\", which seems to imply that the environment lighting described in the following is also spatially varying (ie, each pixel has its own environment lighting). However, this does not seem to be the case, otherwise the task would have been almost impossible without additional constraints.\n- What is the resolution of the environment map for experiments on the metallic objects with MC shading in Sec 5.1? The predicted environment map in Fig 4 appear to be sharper than those in Fig 5.\n- A few recent papers that might be relevant:\n1. Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image. Li et al. CVPR 2020.\n2. Single-Shot Neural Relighting and SVBRDF Estimation. Sang et al. ECCV 2020.\n\n\n---\n## Post-Rebuttal\nI appreciate the authors' effort in the rebuttal and further responses. The authors clarified an important detail of the multi-view training scheme, which was missing in the original manuscript. There is a consistency regularization between two views by swapping the predictions in the reconstruction loss. I think this multi-view constraint is key to the successful disentanglement of the albedo and illumination. Otherwise, everything could go into the albedo. \n\nI was overly impressed by the results in my original review without knowing this training detail. Nevertheless, the decomposition results are still quite interesting. I will decrease my rating to 6, but still borderline accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A few failure cases are provided in the supplementary material, which are mainly about disentangling albedo and environment lighting and recovering texture details.\nI think it will also be interesting to see how incorporating such sophisticated material and lighting models impacts the shape learning. The numerical evaluations in the supplementary material shows some brief comparisons, but on 2D image and masks, which suggest similar performance on shape recovery. Are there any cases where the shape or viewpoint is wrong, hence resulting in poor appearance decomposition?\n\nThe discussion on social impact sounds reasonable to me, and I do not see immediate concerns on potential negative impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6.5"}, "extra_text": {"ethical_concerns": "I do not see major ethical concerns of the proposed method. But I do wonder how the method performs on faces of black people, especially since the specular effects are more apparent. This is not a major concern as this paper is not solely focused on faces."}}, {"review_id": "dAuplYD2YJ1", "reviewer": "Reviewer_2BAr", "summary": "The authors address the task of predicting scene intrinsics in terms of spatially varying reflectance behavior and lighting characteristics from a single image without any 3D supervision. For this purpose, they leverage the potential offered by differentiable (graphics) renderers in their DIB-R++ approach. Given a 3D mesh as input, the authors use the renderer to get diffuse albedo, surface normal and masks. These buffers are then used to compute the incident radiance by sampling lighting and specular BRDF using Monte Carlo Shading or spherical Gaussian Shading depending on the material type. The proposed deferred, image-based renderer incorporates environment lighting and spatially varying reflectance models to efficiently approximate the light transport and exploits the combination of rasterization and ray-tracing to leverage the individual advantages regarding speed and realism. The authors exemplarily use their inverse rendering framework for the single image 3D reconstruction and material editing and relighting.\n\nThe approach offers a clear potential over learning-based inverse rendering approaches based on rasterization based renderers that rely on assumptions like naive lighting and material models. However, more evaluation (including comparisons to more different approaches) would be required to support the statements that ‘Compared to more advanced physics-based differentiable renderers leveraging path tracing, DIB-R++ is highly performant due to its compact and expressive shading model’ and that their ‘approach achieves superior material and lighting disentanglement on synthetic and real data compared to existing rasterization-based approaches’ (as mentioned in the abstract).\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Exposition\n- The paper is well-structured and easy to follow. Figures/tables and captions are informative.\n- The approach is well-motivated.\n\n\nEvaluation\n- The authors provide quantitative and qualitative results for single image based 3D reconstruction where they evaluate the benefits/disadvantage of Monte Carlo shading and spherical Gaussians for metallic or glossy surfaces.\n- Comparisons to other approaches have not been provided and, hence, the potential of the proposed approach seems to be not fully demonstrated. Besides additional insights regarding quality, the comparison to such approaches in terms of the required processing times and memory requirements would be interesting.\n- There are many different coefficients for the individual loss terms in equation 7. The choice of these weights has been heuristically determined without providing a detailed discussion/evaluation. More insights seem to be relevant in this regards. Also, the evaluation could benefit from showing the effect of the individual terms in the scope of an ablation study.\n- Why is StyleGAN used to generate multi-view images of cars and faces, i.e. the data we needed for model training, and not some real-world dataset? The generated data as shown in the supplemental seems to contain several artifacts.\n- For the StyleGAN based input data, a visual comparison to the composition results (albedo, specular component, texture as well as relighting) from reference [11] has been provided. Similar comparisons could be performed on synthetic data and complemented with an quantitative error evaluation.\n- The authors provide a discussion of limitations and the broader impact.\n\n\nReproducibility\nThe authors mentioned that they ‘plan to release the code after refactoring’, i.e. reducing complex dependencies. Otherwise, the details in the paper could be sufficient to allow the implementation by a more experienced person.\n\n\nReferences\nGoel et al., Shape from Tracing: Towards Reconstructing 3D Object Geometry and SVBRDF Material from Images via Differentiable Path Tracing\nLi et al., Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image\nTewari et al., State of the Art on Neural Rendering\nLuan et al., Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering\nNimier-David et al., Material and Lighting Reconstruction for Complex Indoor Scenes with Texture-space Differentiable Rendering (might have been published before the NeuRIPS deadline, but I am not sure)\n\n\nPost-rebuttal comments:\n\nThe rebuttal addresses my main concerns regarding the ablation of loss terms, the need for multi-view images and their creation using StyleGAN (though I have to admit that I would also still expect tests from data without synthesis artefacts to better see the performance of the proposed approach), the processing times and memory requirements (here, numeric values have been provided), and comparisons on synthetic data, as well as other reviewers’ comments.\n\nDespite these aspects, the results of the ablations and comparisons have not yet been demonstrated in terms of qualitative or quantitative evaluations and several of the intended revisions of discussions have not been provided in a detailed form. Also, the discussion of other references has been kept short without details on the novelty. In addition, I agree with other reviewers that the novelty is limited.\n\nAs I have the feeling that the revision of the manuscript would be quite significant and, hence,  require another review cycle, I remain at a borderline recommendation (very slightly tending towards reject, as the direct borderline recommendation is not available). \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors provide a discussion of limitations and the broader impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "Q6UasSmn7VI", "reviewer": "Reviewer_AV84", "summary": "This paper aims for single-view 3D object reconstruction without 3D supervision. It jointly infers geometry, reflectance, and lighting given a single image. The authors use a deferred image-based renderer and apply two standard techniques for shading. The proposed method is evaluated on synthetic datasets of metallic and glossy surfaces, as well as StyleGAN generated images.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n1. In general the paper is well-written.\n\n1. The method is able to learn 3D geometry, reflectance and lighting without 3D supervision.\n\n1. The authors provide thorough comparisons between MC and SG shading models.\n\n1. Experimental results show that the method can disentangle material and lighting without supervision.\n\n### Weaknesses\n\n1. This paper is a fairly straightforward extension of DIB-R [7] by modeling the image generation using SV-BRDF adopting standard techniques (MC and SG) for shading. The technical novelty seems to be relatively limited.\n\n1. As the authors discussed in L291, the predicted material has little correlation with the ground truth material. I understand that this is due to the ambiguity of the task without 3D supervision. Does this misalignment hamper the performance of material editing/relighting or not? It would be good if the authors can elaborate more on this. \n\n1. Compared to [11], this paper does not consider the modeling of background. It would be good if the authors can explain why that background is not considered in this paper.\n\n1. It is not easy to evaluate the performance of the proposed method from the qualitative results. More specifically,\n    a. In Fig.4 and Fig.5, it might be good to show the GT albedo and the GT images rendered at the same viewpoint as the predictions for comparison.\n    b. In Fig. 7 (left), why does the car in the bottom row has two different colors from different viewpoints?\n    c. In Fig. 7 (right), it is not easy to evaluate the performance given the few number of images. A video would be great to demonstrate the editing.\n\n1. The ablation study of the loss functions is missing. \n\n### Detailed Comments\n\n1. While it is nice that the authors point out the difference compared to recent works NeRD and PhySG, it is unclear what is the advantage of the proposed method by modeling objects using the mesh. It may instead be considered as a disadvantage as the proposed method is limited to a fixed topology (same as the initialized shape, i.e. sphere as considered in the paper).\n\n1. Some technical details are missing. \n- While the experimental setting following [7] and [11]. It would be better to explain some details to be self-contained. For example, I assume that the perceptual loss $L_{per}$ is defined on the masked images following [7]. \n- It would be good if the authors can explain how the masks are obtained for the StyleGAN generated data.\n\n1. There are some minor language issues. For example:\n- L146: we “importance sample” the BRDF… \n- L286: and default to SGs otherwise \n- L314: We also compare [11] .… -> We also compare with [11] ...\n- L315:  by using … -> using\n- L316: groundtruth -> ground-truth\n\n---\n\nThanks for the responses. I keep my original rating and vote for accepting this paper. Please incorporate all the promised changes in the final version.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "4wgOCEVCdZr", "reviewer": "Reviewer_LsqZ", "summary": "This paper proposes an extension of prior DIB-R method to reconstruct material, lighting and geometry from a single image. There are two major improvements. One is that authors replace phong BRDF with more realistic physically-based microfacet BRDF. The other is that instead of using spherical harmonics to approximate environment lighting, authors use spherical Gaussian or a relatively high resolution panorama so they can recover more high-frequency lighting. Experiments on synthetic dataset and a gan-generating dataset show that the proposed method can achieve similar performances on image reconstruction loss and albedo loss but more accurate lighting estimation. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will list the strengths of the paper and my questions. When listing my questions, I will separate them into major questions that may change my evaluation and minor questions. Authors can focus on major questions in the rebuttal.\n\nStrengths:\n\n1. This paper solves an extremely challenging problem: simultaneously reconstructing lighting, geometry, and SVBRDF from a single image captured under the unconstrained environment through a single forward pass of a deep network. Moreover, it does not rely on any ground-truth supervisions but only uses the image reconstruction loss and mask segmentation loss. Given the challenging nature of the problem, the qualities of the predictions are probably reasonable. In fact, I think authors can probably emphasize more how challenging the problem is. \n\n2. Compared to the prior baseline, this paper demonstrates clear improvements in lighting estimation, possibly due to the new lighting and material representations. Both qualitative and quantitative comparisons show that the proposed method can better capture high-frequency lighting with far more details. \n\n3. The paper is well written and is easy to follow. \n\nMajor questions:\n\n1. The novelty of the paper may be somewhat limited. This paper can be considered as a direct extension of prior baseline DIB-R but with better lighting and material representations and rendering methods. However, both representations (microfacet BRDFs, SG for lighting approximation) and rendering methods (MC sampling and spherical Gaussian integration) are well known in vision and graphics communities and are expected to better handle high-frequency lighting. \n\n2. The quality of some predictions may not be impressive enough. More visualization and explanation can be added. \n\n    a. As also suggested by authors in Section 7. In all examples, it is easy to see the color bleeding from lighting to diffuse albedo. But since this is a very ill-posed problem, maybe such results are reasonable. \n\n    b. There are not many discussions and comparisons on the specularity reconstruction. Authors show specularity reconstruction in Figure 6. But it is hard to see from a single view whether the specularity is realistic or not. One good visualization is to render the object from different views and show the difference. \n\n    c. The quantitative comparison in Table 1 is a little confusing. For both metallic and glossy surfaces, the proposed method achieves much more accurate lighting reconstruction and similar diffuse albedo prediction. However, the image rendering quality is still similar to the prior baseline, which makes me wonder if it is caused by a worse specularity prediction. But there is no discussion on the specular component of the SVBRDF. Authors may explain the results in more detail. \n\n   d. In Figure 4, the overall intensities of the diffuse albedo are very different. For better visualization, authors may consider rescaling the diffuse albedo and the lighting accordingly. \n\n3. There are no real examples in the paper. Authors only show the reconstruction results on examples generated by style gan, which I think cannot be considered as real examples. I will be glad to see more real examples to demonstrate how the proposed method can perform in practice. \n\n4. The BRDF definition is confusing. Authors use Disney BRDF, which uses metallic to control the specular albedo. But in the paper, authors predict specular albedo s as well, which cannot be found in the BRDF definition in the supplementary material. Besides, Eq. (4) in the supplementary material is wrong. Please correct it. Also why Eq. (4) is an SG approximation? \n\n5. The title of the paper can probably be modified. The proposed renderer may not be an image-based renderer since it essentially takes the material, lighting, and geometry as inputs and renders an image, just like a traditional renderer does. Besides, I am not sure if it is necessary to emphasize this is a deferred renderer. While it is a deferred renderer, this is unrelated to the major contributions and may just cause confusion. \n\n6. The loss functions can probably be improved. Given the ill-posed nature of the problem, I feel authors may consider using ground-truth materials and lighting for supervisions or try to optimize with multiple-view inputs. This may help improve the quality and potentially show clearer improvements on specularity reconstruction with the new material representation. \n\nMinor questions:\n1. Line 90: Our method first rasterizes a 3D mesh to obtain *material parameters* (not only diffuse albedo), surface normals.\n\n2. The MC sampling can be improved. Authors mention that MC sampling has difficulties in rendering diffuse objects. However, in computer graphics, diffuse objects are considered to be easy to render. The standard technique is to use multiple importance sampling (MIS), which jointly samples the light source and BRDF. Authors may at least mention that in the paper or try to include MIS  into the comparison. \n \n3. Some citations can be included: \n\nWhen discussing camera-flashlight set up on line 77. Two papers can be added:\n\nDeschaintre, Valentin, et al. \"Single-image svbrdf capture with a rendering-aware deep network.\" ToG, 2018\n\nLi, Zhengqin, Kalyan Sunkavalli, and Manmohan Chandraker. \"Materials for masses: SVBRDF acquisition with a single mobile phone image.\" ECCV, 2018.\n\nWhen discussing synthetic datasets for training. One paper can be added:\n\nLi, Zhengqin, et al. \"Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image.\" CVPR 2020. \n\nBased on the above questions, my current evaluation of this paper is leaning towards negative. I would like to see authors' answers to my questions and update my review. \n\n\nAfter rebuttal: authors' rebuttal solved most of my questions. Therefore I updated my rating to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors may miss two more limitations:\n1. The specularity predicted by the proposed method is much more blurry compared to the ground-truth. As shown in Figure 6, while the car in the input image have sharp reflection, I cannot see similar reflection in the specular BRDF visualization. \n2. For very diffuse object, the lighting estimation may fail since there are too many ambiguities. But I may not see such comparisons and analysis in the paper. \n\nI cannot see any negative societal impact of the work. ", "review_text": "", "extra_scores": {"ethical_concerns": "I cannot see any ethical issues. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission was thoroughly reviewed and discussed. The ratings are borderline, but three of the four reviewers support acceptance. Reviewers are concerned about the magnitude of the contribution, but generally see the work as a useful addition to the literature. The AC supports the accept recommendation. The authors are encouraged to thoroughly address the reviewers' concerns and recommendations in the revision.
Decision: Accept (Poster)