DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer

Abstract: We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable renderers. Many previous learning-based approaches for inverse graphics adopt rasterization-based renderers and assume naive lighting and material models, which often fail to account for non-Lambertian, specular reflections commonly observed in the wild. In this work, we propose DIBR++, a hybrid differentiable renderer which supports these photorealistic  effects by combining rasterization and ray-tracing, taking the advantage of their respective strengths---speed and realism. Our renderer incorporates environmental lighting and spatially-varying material models to efficiently approximate light transport, either through direct estimation or via spherical basis functions. Compared to more advanced physics-based differentiable renderers leveraging path tracing, DIBR++ is highly performant due to its compact and expressive shading model, which enables easy integration with learning frameworks for geometry, reflectance and lighting prediction from a single image without requiring any ground-truth. We experimentally demonstrate that our approach achieves superior material and lighting disentanglement on synthetic and real data compared to existing rasterization-based approaches and showcase several artistic applications including material editing and relighting. 


Reviews and author responses:

This paper aims for single-view 3D object reconstruction without 3D supervision. It jointly infers geometry, reflectance, and lighting given a single image. The authors use a deferred image-based renderer and apply two standard techniques for shading. The proposed method is evaluated on synthetic datasets of metallic and glossy surfaces, as well as StyleGAN generated images.  ### Strengths
1. In general the paper is well-written.

1. The method is able to learn 3D geometry, reflectance and lighting without 3D supervision.

1. The authors provide thorough comparisons between MC and SG shading models.

1. Experimental results show that the method can disentangle material and lighting without supervision.

### Weaknesses

1. This paper is a fairly straightforward extension of DIB-R [7] by modeling the image generation using SV-BRDF adopting standard techniques (MC and SG) for shading. The technical novelty seems to be relatively limited.

1. As the authors discussed in L291, the predicted material has little correlation with the ground truth material. I understand that this is due to the ambiguity of the task without 3D supervision. Does this misalignment hamper the performance of material editing/relighting or not? It would be good if the authors can elaborate more on this. 

1. Compared to [11], this paper does not consider the modeling of background. It would be good if the authors can explain why that background is not considered in this paper.

1. It is not easy to evaluate the performance of the proposed method from the qualitative results. More specifically,
    a. In Fig.4 and Fig.5, it might be good to show the GT albedo and the GT images rendered at the same viewpoint as the predictions for comparison.
    b. In Fig. 7 (left), why does the car in the bottom row has two different colors from different viewpoints?
    c. In Fig. 7 (right), it is not easy to evaluate the performance given the few number of images. A video would be great to demonstrate the editing.

1. The ablation study of the loss functions is missing. 

### Detailed Comments

1. While it is nice that the authors point out the difference compared to recent works NeRD and PhySG, it is unclear what is the advantage of the proposed method by modeling objects using the mesh. It may instead be considered as a disadvantage as the proposed method is limited to a fixed topology (same as the initialized shape, i.e. sphere as considered in the paper).

1. Some technical details are missing. 
- While the experimental setting following [7] and [11]. It would be better to explain some details to be self-contained. For example, I assume that the perceptual loss $L_{per}$ is defined on the masked images following [7]. 
- It would be good if the authors can explain how the masks are obtained for the StyleGAN generated data.

1. There are some minor language issues. For example:
- L146: we “importance sample” the BRDF… 
- L286: and default to SGs otherwise 
- L314: We also compare [11] .… -> We also compare with [11] ...
- L315:  by using … -> using
- L316: groundtruth -> ground-truth

---

Thanks for the responses. I keep my original rating and vote for accepting this paper. Please incorporate all the promised changes in the final version.
 Limitations and societal impact are discussed.

The authors address the task of predicting scene intrinsics in terms of spatially varying reflectance behavior and lighting characteristics from a single image without any 3D supervision. For this purpose, they leverage the potential offered by differentiable (graphics) renderers in their DIB-R++ approach. Given a 3D mesh as input, the authors use the renderer to get diffuse albedo, surface normal and masks. These buffers are then used to compute the incident radiance by sampling lighting and specular BRDF using Monte Carlo Shading or spherical Gaussian Shading depending on the material type. The proposed deferred, image-based renderer incorporates environment lighting and spatially varying reflectance models to efficiently approximate the light transport and exploits the combination of rasterization and ray-tracing to leverage the individual advantages regarding speed and realism. The authors exemplarily use their inverse rendering framework for the single image 3D reconstruction and material editing and relighting.

The approach offers a clear potential over learning-based inverse rendering approaches based on rasterization based renderers that rely on assumptions like naive lighting and material models. However, more evaluation (including comparisons to more different approaches) would be required to support the statements that ‘Compared to more advanced physics-based differentiable renderers leveraging path tracing, DIB-R++ is highly performant due to its compact and expressive shading model’ and that their ‘approach achieves superior material and lighting disentanglement on synthetic and real data compared to existing rasterization-based approaches’ (as mentioned in the abstract).
  Exposition
- The paper is well-structured and easy to follow. Figures/tables and captions are informative.
- The approach is well-motivated.


Evaluation
- The authors provide quantitative and qualitative results for single image based 3D reconstruction where they evaluate the benefits/disadvantage of Monte Carlo shading and spherical Gaussians for metallic or glossy surfaces.
- Comparisons to other approaches have not been provided and, hence, the potential of the proposed approach seems to be not fully demonstrated. Besides additional insights regarding quality, the comparison to such approaches in terms of the required processing times and memory requirements would be interesting.
- There are many different coefficients for the individual loss terms in equation 7. The choice of these weights has been heuristically determined without providing a detailed discussion/evaluation. More insights seem to be relevant in this regards. Also, the evaluation could benefit from showing the effect of the individual terms in the scope of an ablation study.
- Why is StyleGAN used to generate multi-view images of cars and faces, i.e. the data we needed for model training, and not some real-world dataset? The generated data as shown in the supplemental seems to contain several artifacts.
- For the StyleGAN based input data, a visual comparison to the composition results (albedo, specular component, texture as well as relighting) from reference [11] has been provided. Similar comparisons could be performed on synthetic data and complemented with an quantitative error evaluation.
- The authors provide a discussion of limitations and the broader impact.


Reproducibility
The authors mentioned that they ‘plan to release the code after refactoring’, i.e. reducing complex dependencies. Otherwise, the details in the paper could be sufficient to allow the implementation by a more experienced person.


References
Goel et al., Shape from Tracing: Towards Reconstructing 3D Object Geometry and SVBRDF Material from Images via Differentiable Path Tracing
Li et al., Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image
Tewari et al., State of the Art on Neural Rendering
Luan et al., Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering
Nimier-David et al., Material and Lighting Reconstruction for Complex Indoor Scenes with Texture-space Differentiable Rendering (might have been published before the NeuRIPS deadline, but I am not sure)


Post-rebuttal comments:

The rebuttal addresses my main concerns regarding the ablation of loss terms, the need for multi-view images and their creation using StyleGAN (though I have to admit that I would also still expect tests from data without synthesis artefacts to better see the performance of the proposed approach), the processing times and memory requirements (here, numeric values have been provided), and comparisons on synthetic data, as well as other reviewers’ comments.

Despite these aspects, the results of the ablations and comparisons have not yet been demonstrated in terms of qualitative or quantitative evaluations and several of the intended revisions of discussions have not been provided in a detailed form. Also, the discussion of other references has been kept short without details on the novelty. In addition, I agree with other reviewers that the novelty is limited.

As I have the feeling that the revision of the manuscript would be quite significant and, hence,  require another review cycle, I remain at a borderline recommendation (very slightly tending towards reject, as the direct borderline recommendation is not available). 
 The authors provide a discussion of limitations and the broader impact.

This paper proposes an extension of prior DIB-R method to reconstruct material, lighting and geometry from a single image. There are two major improvements. One is that authors replace phong BRDF with more realistic physically-based microfacet BRDF. The other is that instead of using spherical harmonics to approximate environment lighting, authors use spherical Gaussian or a relatively high resolution panorama so they can recover more high-frequency lighting. Experiments on synthetic dataset and a gan-generating dataset show that the proposed method can achieve similar performances on image reconstruction loss and albedo loss but more accurate lighting estimation.   I will list the strengths of the paper and my questions. When listing my questions, I will separate them into major questions that may change my evaluation and minor questions. Authors can focus on major questions in the rebuttal.

Strengths:

1. This paper solves an extremely challenging problem: simultaneously reconstructing lighting, geometry, and SVBRDF from a single image captured under the unconstrained environment through a single forward pass of a deep network. Moreover, it does not rely on any ground-truth supervisions but only uses the image reconstruction loss and mask segmentation loss. Given the challenging nature of the problem, the qualities of the predictions are probably reasonable. In fact, I think authors can probably emphasize more how challenging the problem is. 

2. Compared to the prior baseline, this paper demonstrates clear improvements in lighting estimation, possibly due to the new lighting and material representations. Both qualitative and quantitative comparisons show that the proposed method can better capture high-frequency lighting with far more details. 

3. The paper is well written and is easy to follow. 

Major questions:

1. The novelty of the paper may be somewhat limited. This paper can be considered as a direct extension of prior baseline DIB-R but with better lighting and material representations and rendering methods. However, both representations (microfacet BRDFs, SG for lighting approximation) and rendering methods (MC sampling and spherical Gaussian integration) are well known in vision and graphics communities and are expected to better handle high-frequency lighting. 

2. The quality of some predictions may not be impressive enough. More visualization and explanation can be added. 

    a. As also suggested by authors in Section 7. In all examples, it is easy to see the color bleeding from lighting to diffuse albedo. But since this is a very ill-posed problem, maybe such results are reasonable. 

    b. There are not many discussions and comparisons on the specularity reconstruction. Authors show specularity reconstruction in Figure 6. But it is hard to see from a single view whether the specularity is realistic or not. One good visualization is to render the object from different views and show the difference. 

    c. The quantitative comparison in Table 1 is a little confusing. For both metallic and glossy surfaces, the proposed method achieves much more accurate lighting reconstruction and similar diffuse albedo prediction. However, the image rendering quality is still similar to the prior baseline, which makes me wonder if it is caused by a worse specularity prediction. But there is no discussion on the specular component of the SVBRDF. Authors may explain the results in more detail. 

   d. In Figure 4, the overall intensities of the diffuse albedo are very different. For better visualization, authors may consider rescaling the diffuse albedo and the lighting accordingly. 

3. There are no real examples in the paper. Authors only show the reconstruction results on examples generated by style gan, which I think cannot be considered as real examples. I will be glad to see more real examples to demonstrate how the proposed method can perform in practice. 

4. The BRDF definition is confusing. Authors use Disney BRDF, which uses metallic to control the specular albedo. But in the paper, authors predict specular albedo s as well, which cannot be found in the BRDF definition in the supplementary material. Besides, Eq. (4) in the supplementary material is wrong. Please correct it. Also why Eq. (4) is an SG approximation? 

5. The title of the paper can probably be modified. The proposed renderer may not be an image-based renderer since it essentially takes the material, lighting, and geometry as inputs and renders an image, just like a traditional renderer does. Besides, I am not sure if it is necessary to emphasize this is a deferred renderer. While it is a deferred renderer, this is unrelated to the major contributions and may just cause confusion. 

6. The loss functions can probably be improved. Given the ill-posed nature of the problem, I feel authors may consider using ground-truth materials and lighting for supervisions or try to optimize with multiple-view inputs. This may help improve the quality and potentially show clearer improvements on specularity reconstruction with the new material representation. 

Minor questions:
1. Line 90: Our method first rasterizes a 3D mesh to obtain *material parameters* (not only diffuse albedo), surface normals.

2. The MC sampling can be improved. Authors mention that MC sampling has difficulties in rendering diffuse objects. However, in computer graphics, diffuse objects are considered to be easy to render. The standard technique is to use multiple importance sampling (MIS), which jointly samples the light source and BRDF. Authors may at least mention that in the paper or try to include MIS  into the comparison. 
 
3. Some citations can be included: 

When discussing camera-flashlight set up on line 77. Two papers can be added:

Deschaintre, Valentin, et al. "Single-image svbrdf capture with a rendering-aware deep network." ToG, 2018

Li, Zhengqin, Kalyan Sunkavalli, and Manmohan Chandraker. "Materials for masses: SVBRDF acquisition with a single mobile phone image." ECCV, 2018.

When discussing synthetic datasets for training. One paper can be added:

Li, Zhengqin, et al. "Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image." CVPR 2020. 

Based on the above questions, my current evaluation of this paper is leaning towards negative. I would like to see authors' answers to my questions and update my review. 


After rebuttal: authors' rebuttal solved most of my questions. Therefore I updated my rating to 6. Authors may miss two more limitations:
1. The specularity predicted by the proposed method is much more blurry compared to the ground-truth. As shown in Figure 6, while the car in the input image have sharp reflection, I cannot see similar reflection in the specular BRDF visualization. 
2. For very diffuse object, the lighting estimation may fail since there are too many ambiguities. But I may not see such comparisons and analysis in the paper. 

I cannot see any negative societal impact of the work. 

This paper tackles the problem of learning to recover 3D shape, surface materials and environment lighting from a single image. It extends a recent learning-based single-image 3D reconstruction framework with realistic BRDF and shading modules that allows it to automatically disentangle material and lighting, using the same analysis-by-synthesis pipeline without relying on ground-truth data for training.

The paper presents extensive analysis and evaluations on both synthetic and realistic GAN-generated images, demonstrating impressive results on the disentanglement. This result presents a significant step forward in the field of 3D reconstruction and inverse graphics, by putting realistic material and lighting models in an unsupervised learning framework for general objects (if my understanding of training and testing setup is correct, see W2).  ## Strengths
### S1. Method and results
This paper presents very impressive results on single-image inverse graphics of general objects. Existing work can be grouped into three categories:
1. unsupervised 3D reconstruction assuming simplistic material and lighting;
2. supervised methods that also recovers sophisticated material and lighting on synthetic scenes;
3. optimization-based photometric stereo methods.

This paper is the first one (to my best knowledge) to show that sophisticated material and lighting can be learned from only image collections of relatively realistic scenes in an analysis-by-synthesis framework, similar to those in unsupervised 3D reconstruction. It goes beyond existing methods, and presents an exciting path towards the goal of general inverse graphics in the real world.

### S2. Extensive evaluation
The paper presents a large collection of analysis and evaluation experiments, including experiments on toy examples showcasing the difference of the different variants, and extensive comparisons with previous methods on various datasets. In particular, it shows the proposed framework can be easily adapted and produce robust results in various scenarios.

### S3. Writing
The paper is well written. I enjoyed reading it. Some technical details are missing in the main paper due to the lack of space, but the supplementary material resolved many of my questions.


## Weaknesses
### W1. (minor) Limitations
The model does not deal with spatially-varying materials (except diffuse albedo). But that has even more ambiguity and would be super challenging to disentangle from a single RGB image.

### W2. Clarifications required
There is no figure for the training pipeline, and some training details are missing. Below are some details that may require further clarifications.
- Could the authors clarify what kind of data is required during training (multi-views, masks, known camera?) and what is the input during inference (only one single image, without masks, unknown camera?)?
- Does the model require known camera viewpoint (or object pose), at training or at both training and inference? I did not find it explicitly mentioned in the paper. But the predicted environment maps seem to perfectly align with the ground truth ones, which is very impressive, as it would have so much ambiguity to recover both the object pose and orientation of the environment lighting at the same time. I did not find clear descriptions in its predecessor DIB-R either.
- Decomposing albedo and lighting from a single image is fundamentally ambiguous. The recovered albedo images look highly plausible even when some regions are saturated by specular reflections, eg Fig 4, 5, 6. Why is this possible? The paper describes the method very well, but I'm still missing the key insight in the success of the method. For example, why wouldn't the model put all shading in the albedo image? Are there any additional constraints or regularization? Or is it the "learning" part that is the key here, as the model learns category-specific priors over a collection of training images?
- I might have more follow-up questions after these are clarified.


## Additional questions / comments
- I would highly recommend releasing the code, which will hugely facilitate future work in inverse graphics.
- The paper probably should not stress "spatially-varying BRDF" as the roughness, metalness and specular albedo terms are all global, and only the diffuse albedo is spatially varying.
- I suppose the paper does not consider no spatially varying lighting. I was confused because in line 135, it states "We assume henceforth a fixed pixel p, and remove the subscript", which seems to imply that the environment lighting described in the following is also spatially varying (ie, each pixel has its own environment lighting). However, this does not seem to be the case, otherwise the task would have been almost impossible without additional constraints.
- What is the resolution of the environment map for experiments on the metallic objects with MC shading in Sec 5.1? The predicted environment map in Fig 4 appear to be sharper than those in Fig 5.
- A few recent papers that might be relevant:
1. Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image. Li et al. CVPR 2020.
2. Single-Shot Neural Relighting and SVBRDF Estimation. Sang et al. ECCV 2020.


---
## Post-Rebuttal
I appreciate the authors' effort in the rebuttal and further responses. The authors clarified an important detail of the multi-view training scheme, which was missing in the original manuscript. There is a consistency regularization between two views by swapping the predictions in the reconstruction loss. I think this multi-view constraint is key to the successful disentanglement of the albedo and illumination. Otherwise, everything could go into the albedo. 

I was overly impressed by the results in my original review without knowing this training detail. Nevertheless, the decomposition results are still quite interesting. I will decrease my rating to 6, but still borderline accept. A few failure cases are provided in the supplementary material, which are mainly about disentangling albedo and environment lighting and recovering texture details.
I think it will also be interesting to see how incorporating such sophisticated material and lighting models impacts the shape learning. The numerical evaluations in the supplementary material shows some brief comparisons, but on 2D image and masks, which suggest similar performance on shape recovery. Are there any cases where the shape or viewpoint is wrong, hence resulting in poor appearance decomposition?

The discussion on social impact sounds reasonable to me, and I do not see immediate concerns on potential negative impacts.

 Thanks authors for the new results! These new results solve my questions. I would like to change my rating to marginally above the acceptance threshold since I am not very sure if the contribution is very significant but overall I will be happy to see this paper accepted. 

 We are glad that most questions of the reviewer have been addressed. We address the unsolved questions below.

- Novelty

Simultaneously reconstructing 3D geometry, material and lighting from a single image without any supervision is a highly challenging task. Our work firstly introduces PBR materials and lighting into a learning framework to accomplish this goal. We believe none of the previous works demonstrated this without additional constraints. As such, we think our method is an important step towards the design of a practical and general-purpose unsupervised inverse graphics framework, which we see as a significant contribution.

- Real Images

As requested by the reviewer, we provide real image reconstruction results with [anymonous link](https://storageneurips2021.s3.ca-central-1.amazonaws.com/real+data+%26+specular.zip). 

Our model is trained on synthetic dataset generated by StyleGAN. Thanks to this powerful generative model, the distribution of GAN images is similar to the distribution of real images. Thus, our model can be well generalized to real images. The file `lsun_reconstuction.png` shows a table of 10 reconstructions on real car instances.

Note that during inference, we do not need any camera pose and thus predict shapes in a canonical view. However, the camera pose is needed to re-render the object. Since ground truth camera poses are not available for real images, we manually adjust the camera poses. As a result, the re-rendered images exhibit a slight mismatch with GT images. However, our model still accounts for high specular lighting and predicts correct lighting directions along with clean diffuse albedo maps.

- Specularity

We agree that rotating the environment map can lead to intensity changes for diffuse objects. To better understand the specular reflections, we have created several turntable videos to showcase how the specularities change as a function of camera position. In particular, in each video we first show input image and re-rendered image in the top row. We then fix the object and lighting and rotate the camera around the object to show an animation of specular reflections. To better compare the effect of lighting, in the second row we present rendered image without lighting on the left, and with lighting on the right.  

We show 11 examples in total, where the green car (`gan_example1.mp4`) is from Fig.7 of the main paper. The remaining 10 examples (`lsun_example{1..10}.mp4`) are from the LSUN dataset. Note that when we rotate the camera, the specular reflections also change. Moreover, if the car is rotated to poses similar to the ones in GT images, the specular effect well coincide. 



 Thank authors for the detailed rebuttal! I will list the questions that have been solved and have not been solved below. 

Questions that have not been solved:
1. Novelty: I may still suggest that the novelty of the paper is somewhat incremental. While the problem is new and challenging, it is essentially combining DIB-R with PBR materials and better lighting approximation. 
2. Specularity: It is difficult to say if the rotating environment map example can show specularity. For a diffuse object, a rotating environment map may also cause intensity changes. Increasing the specular albedo may not show specularity clearly either. I may still suggest that a multi-view rendering is necessary. 
3.  No real examples: I feel real images are probably necessary to demonstrate the generalizability of the proposed method. I hope authors can add them in the final version. Unfortunately, we cannot submit new images in the rebuttal. 

Questions that have been solved:
1. Color bleeding: authors' explanation makes sense. Given that this problem is very hard, the current results may be reasonable. 
2. Comparison in table 1: authors' explanation is correct. Better lighting and diffuse albedo prediction does not necessarily mean lower rendering loss. Sorry I missed that. 
3. Intensities in Figure 4: this can be easily modified in the final version. 
4. BRDF definition: authors' explanation makes sense and it can be easily fixed in the final version. 
5. Title: this can be easily fixed too. 
6. Using ground-truth for supervision: authors' explanation makes sense but I feel authors can probably consider emphasizing more that this is an unsupervised method in the introduction and related work section, which may highlight the challenging nature of the problem. 
7. MIS: this is a minor issue. I am satisfied if authors will add some discussions. 
8. Very diffuse object: I did not mean the proposed method will perform worse for diffuse objects. I only assumed that diffuse objects may be harder for any object-based lighting estimation method. But still, thank authors for the explanation. 

Given that I may still have some unsolved questions, I may lean towards keeping my original rating.  My major concern is the specularity experiment and the real images experiment. 

 We didn’t have “explicit consistency regularization between the predictions across multiple views”, we only regularize the prediction from a single view to be consistent on multi-view ground truth data. 

To be specific, assume we have two images $I_1$ and $I_2$ which are two views(cameras noted as $V_1$ and $V_2$) of the same car, we can predict shape, texture, material and lighting for each image, noted as  $S_1$,  $T_1$,  $M_1$,  $L_1$ and $S_2$ , $T_2$ , $M_2$, $L_2$ .  The multiview consistency loss would be:
$$
L = || I_1 - Render(V_1, S_1, T_1, M_1, L_1) ||_1 + || I_2 - Render(V_2, S_1, T_1, M_1, L_2) ||_1 + || I_1 - Render(V_1, S_2, T_2, M_2, L_1) ||_1  + || I_2 - Render(V_2, S_2, T_2, M_2, L_2) ||_1
$$

Besides that, there are no more consistency regulations, e.g. we didn’t add any loss to force $S_1,S_2$ or $T_1,T_2$ to be the same. Moreover, while more views would provide more constraints, we find 2 views are enough and in all the experiments we only use 2 views. 


 Thanks for the detailed response. I have a follow-up question for the authors regarding the training on multi-view images. Is there any explicit consistency regularization between the predictions across multiple views, including optimized in the same batch? Or is each view treated independently during training?

 We thank the reviewer for detailed comments. We are very happy that the reviewer recognizes our method as “first one to show that sophisticated material and lighting can be learned from only image collections” and our result “presents a significant step forward in the field of 3D reconstruction and inverse graphics”. We will release all the code and address all the concerns below.

>There is no figure for the training pipeline.

Thanks for the suggestion. We will add such a visualization figure in the revision.

>...what kind of data is required during training..and what is the input during inference...?

During training, we need multiview images with camera poses and object masks for the same object. The multiview images are either rendered by 3D CAD models, or generated by StyleGAN [11]. During inference, we only need a single image, no camera, no mask is required.

> Does the model require known camera viewpoint (or object pose), at training or at both training and inference?... the predicted environment maps seem to perfectly align with the ground truth ones,...

Camera poses are only required in training, since we need to render the predicted geometry, texture, lighting and albedo back to the image with the corresponding camera pose. It is not required in the inference stage (we always predict a 3D model in canonical pose). Specifically, since we have multiview images for the same object, we train the model with the multiview consistency loss (Eq. 2 in ImageGAN[11]). We believe this is the key reason why we have excellent lighting predictions. We will clarify it in the revision.


>Decomposing albedo and lighting from a single image is fundamentally ambiguous. The recovered albedo images look highly plausible even when some regions are saturated by specular reflections, eg Fig 4, 5, 6. Why is this possible?

We believe the main reason why albedo and shading can be disentangled is that we have multiview images for supervision and apply multiview consistency loss during training. It forces the lighting to be separated from the texture. For example, we predict geometry/lighting/material in one view and supervise it in another view. When the input image has lighting effects, the rendering formula knows it should contribute to lighting instead of texture. if not, it would be wrong in another view. More examples are shown in Fig D and Fig E in the supplementary material, where we successfully optimize albedo and lighting by multiview images.

>The paper probably should not stress "spatially-varying BRDF" as the roughness, metalness and specular albedo terms are all global, and only the diffuse albedo is spatially varying.

When we talk about spatially varying BRDF, we mostly focus on spatially varying texture maps. We will clarify it in the revision. 

>I suppose the paper does not consider no spatially varying lighting

The removal of index p is simply for notational convenience. A single envmap is learned for each scene, and so it does not vary spatially. We can clarify this in the revision.

>What is the resolution of the environment map for experiments on the metallic objects with MC shading in Sec 5.1? The predicted environment map in Fig 4 appear to be sharper than those in Fig 5.

In the case of metallic surfaces, we predict envmap with 256x256 resolution as the image reflections contain high-frequency lighting cues (Fig.4). On the other hand, in the glossy surface case the lighting would be more blurry since the lighting mainly comes from a small portion of specular point, we predict 16x32 envmap. We mentioned the resolution in L241 and L268, and will make it more clear in the revision.

>...interesting to see how incorporating such sophisticated material and lighting models impacts the shape learning...Are there any cases where the shape or viewpoint is wrong, hence resulting in poor appearance decomposition?

Thanks for the suggestion. We use multiview images in the training which covers 360 degrees of the car. We empirically find the shape prediction is stable for current testing sets. We will try cars from more viewpoints or with more sophisticated materials and see how it influences the shape predictions.

>how the method performs on faces of black people

As noted in the paper, our face dataset mostly contains Caucasian faces. We will examine black people predictions in the revision.


 We thank the reviewer for detailed comments. We will incorporate all the editorial suggestions and address all the concerns below.

>This paper is a fairly straightforward extension of DIB-R [7] by modeling the image generation using SV-BRDF adopting standard techniques (MC and SG) for shading. The technical novelty seems to be relatively limited. 

While MC and SG themselves are considered standard in graphics, integrating them within a deep learning framework to support advanced lighting and material predictions is not trivial. DIBR only supports Lambertian surfaces and naive lighting, while our method supports a larger set of appearances and lighting configurations. In addition, our method simultaneously reconstructs 3D geometry/material/lighting from a single image without any supervision. To the best of our knowledge, none of the previous methods demonstrated such capability. As such, we believe that this should be considered a significant contribution.

>...the predicted material has little correlation with the ground truth material... Does this misalignment hamper the performance of material editing/relighting or not?

We believe it will not strongly influence the editing applications.  As mentioned in the paper, the network tries to compensate by predicting low roughness and using the specular albedo to control the intensity of specular highlights.  While the parameters are not aligned with GT, the re-rendered images are still aligned with GT. As shown in Fig. 7, one can still easily edit the predicted parameters since they are meaningful and could generate realistic effects. We will add more analysis in the revision.

>Compared to [11], this paper does not consider the modeling of background

[11] generates background by employing the generative ability of StyleGAN (Fig7 in [11]). Here, we focus more on the 3D reconstruction without merging with StyleGAN. It could be an interesting direction and we leave it for future work.

>It is not easy to evaluate the performance of the proposed method from the qualitative results...b. In Fig. 7 (left), why does the car in the bottom row has two different colors from different viewpoints? c. In Fig. 7 (right), it is not easy to evaluate the performance given the few number of images. A video would be great to demonstrate the editing.

Thanks for the suggestion. We will add GT albedo in new views in the revision.  b) Here we show we can edit the image by changing the albedo color (orange to pink) or adding patterns (2 lines) in albedo. We will add more explanations. c) We will also make a video to better show the editing effects.

>The ablation study of the loss functions is missing.

Thanks for the suggestion. Please see the ablation study part in general comments.

>...it is unclear what is the advantage of the proposed method by modeling objects using the mesh.

We believe mesh representation has more advantages than the implicit-based methods in terms of the appearance editing. Implicit-based methods generally need post-processing (e.g. marching cube) to reconstruct the geometry. Moreover, it is not clear how to generate proper texture map parameterization for them. Instead, as an explicit representation, mesh can be naturally incorporated with standard rendering pipelines and is editable by artists. For instance, an artist could directly employ our methods to predict geometry, texture, lighting and material from a single image and then edit it, which can hardly be done by other representations.

>Some technical details are missing.

The perceptual loss L_per is defined on the masked images following [7] and we compute the mask using MaskRCNN. We will incorporate all the editorial suggestions and add more explanations in the revision.


 We thank the reviewer for detailed comments. As pointed out by the reviewer, “simultaneously reconstructing lighting, geometry, and SVBRDF from a single image” is extremely challenging and our method “can better capture high-frequency lighting with far more details”. We will incorporate all the editorial suggestions and discuss missing references in the revision. We address all the concerns below.

>The novelty of the paper may be somewhat limited

While the microfacet BRDFs and SG themselves are not novel in graphics, introducing them in learning tasks is an open research topic. Previous works (e.g. PhySG [34], NeRD [33]) only adopted SG in multiview, optimization-based tasks. Our method not only extends SG to the ill-posed, single view reconstruction task, but also presents MC for metallic surfaces. As a more general method for various surfaces, our framework moves one step further in the field of learning-based single image 3D reconstruction using inverse graphics, which is extremely challenging due to the lack of 3D ground truth.

Our method simultaneously reconstructs 3D geometry/albedo/material/lighting from a single image without any supervision. To the best of our knowledge, none of the previous methods demonstrated such capability. We believe that this should be considered a significant contribution.

>...easy to see the color bleeding from lighting to diffuse albedo...

Indeed the task is ill-posed and as discussed in Section 7, the predicted albedo may sometimes “bleed” into the lighting prediction. In future work, it may be interesting to tackle this by e.g. leveraging part segmentation. 

>There are not many discussions and comparisons on the specularity reconstruction...it is hard to see from a single view whether the specularity is realistic or not...

Thanks for the suggestion. We have shown two specular examples in Fig.7, “Rotating envmap” and “Increasing specular albedo”. We will also add GT renderings in the corresponding views as reference in the revision.

>The quantitative comparison in Table 1 is a little confusing. For both metallic and glossy surfaces, the proposed method achieves much more accurate lighting reconstruction and similar diffuse albedo prediction. However, the image rendering quality is still similar to the prior baseline...

We apologize for the confusion and want to clarify that our method achieves better diffuse albedo predictions as well. We reported NCC score (normalized cross correlation, e.g. 8 in supp) on the predicted albedo vs ground truth albedo in Table 1 (4-th & 8-th column). For metallic surface, the score is 0.145 (MC, ours) vs. 0.203 (SH, baseline)  while for the glossy surface, it is 0.123 (SG, ours) vs. 0.136 (SH, baseline). This is also reflected in Fig. 5 & 6, where the baseline [7]  often blends the specular lighting into the texture (as naive lighting cannot handle strong reflections) while our predictions clearly disentangle specularity from the texture, thanks to our DIBR++. Since the rendered image is a combination of lighting and texture, and the baseline merges lighting into the texture, we get similar performance in terms of image rendering quality and both of them are very close to the ground truth.  This is not a sign that we get worse predictions but a sign that the baseline [7] merges lighting into texture. We will add more discussions in the revision.

>In Figure 4, the overall intensities of the diffuse albedo are very different

In all the figures, we show the raw predictions. We have considered the scaling factor of predicted images when computing the loss. When we calculate the losses of albedo and lighting, we use normalized cross correlation (Eq 8 in supplementary) which gets rid of the influences of scaling. We will also add a new visualization of scaled albedos in the revision.
 
>There are no real examples in the paper

Thanks for the suggestion. Following [11], we use StyleGAN to generate multi-view images. [11] has shown the model trained on the StyleGAN dataset can be well generalized to real images. This is also observed in our experiments, where we get plausible reconstructions for images in LSUN dataset. We will add real image reconstruction in the revision.

>The BRDF definition is confusing

The specular albedo term s is used in the Fresnel term, which is pulled out of the integral along with the geometry term G, since they cannot be represented as SGs. We do not make use of the metallic term for SG shading for simplicity, but we do use it for MC. The approximation for the Fresnel reflection coefficient fits the (1-cos(x))^5 term found in Schlick’s approximation using an SG for slightly better efficiency. This is a different kind of approximation; we will make it clear in the revision. There is indeed a typo in the equation, we will also correct it.

>The title of the paper can probably be modified

Thanks for the suggestion. We will think of a better name that emphasizes more our learning tasks (i.e. single image), and less the deferred aspect of our pipeline. 

>...consider using ground-truth materials and lighting for supervisions or try to optimize with multiple-view inputs.

Thanks for the suggestion. We agree that adding lighting and material supervision or using multiview inputs would definitely improve the performance. However, this paper mainly focuses on utilizing differentiable rendering to infer geometry, texture, material and lighting from a single image in an unsupervised way. While some methods to exploit supervision for these properties, they are mainly limited to synthetic, renderer datasets -- in the real world, these properties are typically hard to acquire. Alternatively, as mentioned in Sec 7, adding more priors, e.g. part information, to constrain materials to improve the performance can hopefully alleviate the problem, and we leave it for future work.

>...use multiple importance sampling (MIS)

We do not adopt MIS for two reasons. First, it requires sampling two rays (BRDF lobe + envmap) instead of one, which increases the runtime during optimization. Moreover, importance sampling the light map requires computing marginal and conditional distributions over its pixels at every optimization step, which can incur an additional overhead. While only sampling the BRDF can indeed reduce variance, MIS still leaves some amount of noise even at low sample count for environment maps. We will mention MIS in the revision and discuss its shortcomings for our applications.

>The specularity predicted by the proposed method is much more blurry

Thanks for pointing it out. We note that it is challenging to recover sharp and accurate specular reflections, since the predictions mostly come from thin highlight portions in the image. We do find that the predicted lighting can be blurry as we only use 32 SG components to represent it. However, the direction of dominant lighting is typically well handled by our technique, and similar specularities can be observed compared to the ground truth image. We will discuss it in the limitation section.

>For very diffuse object, the lighting estimation may fail...

SG and MC have more advantages in specular cases. However, they will not fail for the diffuse objects. We have shown the results of diffuse objects in Tbl.B and Fig N in supp, where MC, SG and SH have similar performances. It indicates we can still apply SG to diffuse objects. 


 We thank the reviewer for detailed comments. We appreciate that the reviewer recognizes our method as “offers a clear potential over learning-based inverse rendering approaches based on rasterization based renderers that rely on assumptions like naive lighting and material models”. We address all the concerns below.

>...Comparisons to other approaches have not been provided… the comparison to such approaches in terms of the required processing times and memory requirements would be interesting.

Thanks for the suggestion. For the differentiable rendering performance evaluation, we provide the rendering time in Tbl.A in supp comparing MC with SG. Furthermore, since both our method and Mitusba2 [3] supports ray tracing, we also evaluate the running times and memory for the same ray tracing optimization task. We restrict the ray samples to be the same (4 samples) and run 300 iterations to optimize the environment map while fixing everything else.  Our method performs faster (5.4sec v.s. 9.8sec) and requires less memory(1475MB v.s. 3287MB), which demonstrates the “highly performant” claim. We will add the new comparison with Mitsuba2 [3] in the revision.

On the other hand, for the learning task, we provided quantitative and qualitative results and compared with state of the art methods [7,11] for the single image based 3D reconstruction task, where we are the only method to predict geometry, lighting and SVBRDF from a single image in an unsupervised way. Moreover, we achieve much better lighting and albedo results than [7,11].

>There are many different coefficients for the individual loss terms in equation 7.

Thanks for the suggestion. Please see the ablation study of loss terms part in the general comments.

>Why is StyleGAN... not some real-world dataset?

To perform the single image 3D reconstruction task, multiview images with camera poses for the same object are needed. While one could utilize rendered CAD models, a domain gap with real images exists. To the best of our knowledge, there is no such a real dataset which offers a large number of multiview images for the same object with calibrated camera pose.This is also stated in [11]. Thus, following [11], we use StyleGAN to generate multi-view images. [11] shows the model trained on the StyleGAN dataset can generalize well to real images. We tried real images in LSUN dataset and have similar observations. We will add such real image reconstructions in the revision.

>Similar comparisons could be performed on synthetic data and complemented with an quantitative error evaluation.

Thanks for the suggestion. We will add the decomposition figures on synthetic data and compare the predicted specular light with GT.

> Reproducibility & References

We will release the code and add the reference in the revision.


 We thank all the reviewers for their detailed comments. As pointed out by the reviewers, “simultaneously reconstructing lighting, geometry, and SVBRDF from a single image” is “extremely challenging” (R2), while DIBR++ is “ the first one to show that sophisticated material and lighting can be learned from only image collections” (R4), “can disentangle material and lighting without supervision” (R3) and “offers a clear potential over learning-based inverse rendering approaches based on rasterization based renderers” (R1). We first address shared concerns here and reply to individual concerns of each reviewer later.

## Editorial and Visualization Suggestions (R1,R2,R3,R4)
We thank reviewers for their helpful discussions. We will incorporate the editorial suggestions and missing references. We will also add the following visualization figures: a pipeline figure (R4), decomposition figures on synthetic data (R1), GT reference images in different views (R2,R3). We will also prepare videos for Fig.7 (R3). 

## Ablation Study for Loss Terms, (R1,R3)
We directly follow the experimental settings of N3MR[10] and ImageGAN[11]. The ablation study of loss terms has been addressed in previous works, e.g. smoothness loss L_sm is ablated in N3MR (Fig 5) while perceptual loss L_per is ablated in ImageGAN[11] (Fig P in supp). We ablate the loss terms in our method and get similar observations, e.g. L_sm regularizes the geometry to be smooth while L_per helps create high-frequency, detailed textures. We will add the ablation study experiment in the revision.


Meta-review: The submission was thoroughly reviewed and discussed. The ratings are borderline, but three of the four reviewers support acceptance. Reviewers are concerned about the magnitude of the contribution, but generally see the work as a useful addition to the literature. The AC supports the accept recommendation. The authors are encouraged to thoroughly address the reviewers' concerns and recommendations in the revision.

Decision: accept