Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a single image inverse rendering method, named DIB-R++ that extends DIB-R [7] with MC/SG-based ray-tracing to accelerate training/inference. This method works in two-stage, first it uses DIB-R [7] to infer the mesh (and associated geometric attributes) from a single input image, then in the second stage, the network infers albedo and environment light jointly, and a direct light only shader renders a photorealistic image from the inferred geometry, material and lighting.
main_review: 
Strengths:

1. The proposed DIB-R++ is unsupervised and does not require ground truth of geometry, material or environment light.
   
2. In experiments, the proposed DIB-R++ outperforms DIB-R [7] on both metallic and glossy surfaces.


Weaknesses:

1. My major concern about this paper is novelty. Using SG to accelerate training and inference is not a new idea, and it has been studied in NeRD [33] and PhySG [34], the difference is that the proposed DIB-R++ uses DIB-R [7] to recover a mesh from the input image, then infers material and illumination (represented by MC or SG, which is not novel). So I think the novelty is rather incremental compared with both NeRD [33] and PhySG [34] and DIB-R [7]. 

2. The paper mentioned single image 3D reconstruction in Section 4, but there were no such experimental evaluations in the paper. It is hard to tell the 3D reconstruction quality without quantitative evaluations on reconstructed mesh, depth or normal.

3. This paper also lacks experimental comparisons with a state-of-the-art method, e.g., [ref1] Li et al. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. CVPR 2020. Moreover, the proposed DIB-R++ can only infer direct illumination, while [ref1] can also learn global illumination. I'd like to hear more discussion on this in rebuttal.

4. The paper is not easy to follow, and it would be good to show a system diagram that describes network architecture, network input/output, data flow and training/inference steps.

limitations_and_societal_impact: See Weaknesses above.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a method to reconstruct geometry, material and lighting from 
a single image. They apply differentiable rasterizers to calculate the rendering loss, 
which is back-propagated to supervise the training of the desired components.
They compare different lighting representations include HDR envmaps as well as 
Spherical Gaussians.

main_review: 
1. My major concern over the paper is the technical contribution of the paper. The 
paper builds on an existing rasterization-based inverse-rendering framework DIB-R.
The difference is that DIB-R assumes a Lambertian model, while the proposed method 
handles specular materials. However, the overall framework is almost the same.       
Jointly optimizing lighting, materials, and geometry are also common, for example, 
[14].  The paper says [14] does not generalize to real images, and however [14] does 
show results on real images and the paper does not compare to [14]. It is also not clear to 
me why the proposed method would generalize, since it is also trained on synthetic dataset.
For me, it seems the major contribution of the paper is that it adds the lighting, material in 
the inverse-rendering framework of DIB-R, which I think is incremental. 

2. The paper has a lot of text on the shading model and lighting representation. While 
I appreciate the efforts, I have to say that both representations are pretty standard in 
computer graphics and such discussions do not increase the technical contributions. 

3. In terms of differentiable rasterizers, I think [8] has better performance 
than [7] in terms of better handling occlusions and efficiency, why not use [8]?  

4. The loss function only consists of losses on images and vertices. Since the model
is trained on a synthetic dataset, why not also render ground truth material and lighting 
and use them as direct supervisions? Involving those GT in training would better resolve 
the ambiguity in inverse rendering.  

5. The inverse-rendering problem itself is highly ambiguous. In the case that the object 
is purely lambertian, there is no way to infer the lighting accurately, and such a problem 
exists for both SGs and envmaps. Therefore the proposed method is mostly limited to glossy 
surfaces. 

6. In the results section, the paper provides no comparison against previous single-image lighting 
and SVBRDF estimation method, for example [14]. For lighting estimation, the following paper 
should be cited and compared: Deep Parametric Indoor Lighting Estimation, ICCV 2019.

7. In all results on synthetic data, the corresponding ground truth relighting images should 
be included as references. Also there should be videos showing renderings under different lighting 
conditions and viewpoints with comparison to ground truth. 


Overall I think the technical contribution of the proposed method is not enough to be accepted. 
Most of the components are from previous works and the rasterizer-based optimization framework itself is also standard. Therefore, I vote for rejection. 


limitations_and_societal_impact: See main reviews.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: DIB-R++ proposes an extending DIB-R with a physics based shading model. The method predicts an environment map for lighting and uses a simplified version of isotropic Disney BRDF model. The shading model is single bounce and is estimated using either Monte Carlo methods or using spherical basis functions. With these modifications, the authors demonstrate that they are able to decompose the lighting and materials of single objects (cars, faces) from a single image better than DIB-R. 
main_review: The paper is well written and well explained. The addition of more physically based shading to DIB-R seems like a logical and well motivated extension. The paper presents an interesting analysis of the pros and cons of using Monte Carlo versus Spherical Gaussians. From a computational complexity standpoint, the spherical gaussians approach makes a lot of sense and the qualitative results confirm that the impact on quality is minimal. I have listed some suggestions/questions/concerns below, overall I think this paper is marginally above the acceptance threshold.

a) It is noted in LN 251 that DIB-R can accurately re-render the image (albeit with incorrect albedo and lighting). From the quantitative and qualitative results it is fairly easy to conclude that DIB-R++ produces more physically plausible results, however, the usefulness of this improvement is never demonstrated to the reader. It would be informative to see the modifications in Fig 7 applied to DIB-R to hopefully demonstrate why this more complex method should be considered.  

b) Topics like glossy/metallic and MC/SG are well analyzed in the paper, but other topics like geometry are barely discussed. It would be nice to know how the proposed modifications to DIB-R affect the predicted geometry. Additionally, it would be good to include metrics computed against the ground truth geometry. Table 1 is also missing a number of details, specifically what are these values actually measuring (Mean squared error?), please specify. Also consider reporting other common image metrics such as PSNR, SSIM, and LPIPS. 

c) The paper relies heavily on the qualitative results. The way they are currently presented makes it difficult to determine the effectiveness of the method. It would be beneficial to include examples of the ground truth albedo in the same poses as the predicted albedo example. Additionally it would be good to include ground truth examples of the scene re-rendered (with light).

Other:
+ Why is approximating the cosine foreshortening with a SG useful?
+ LN 291. Can you provide these errors, I think it would be interesting to the reader to understand how far off these are.
limitations_and_societal_impact: The authors do a good job of explaining the limitations throughout the paper.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes an improved pipeline for self-supervised training of a neural network that recovers the 3D properties (geometry, brdf, light) of an image containing a single object. This is achieved by proposing DIB-R++, a differentiable renderer which offers improved rendering speed at the cost of less accurate renderings. The DIB-R++ builds on top of the differentiable rasterization framework of DIB-R in order to add physically-based material and lighting models. DIB-R++ comes in 2 variations: A) MC -  relies on Monte Carlo sampling of incident light directions, which is better suited for reflective surfaces. B) SG - relies on using Spherical Gaussian for fast analytic integrations, which is better for rough surfaces. It is important to note that DIB-R++ assumes direct illumination only.

Experimentations include analyzing the effects of forwards and backward passes through the DIB-R++ renderer in isolation (figures 2 and 3), and training neural networks on metal synthetic (section 5.1), glossy synthetic (section 5.2), and StyleGAN generated (section 6.1) datasets. This paper also contains interesting experiments to demonstrate the effects of using different variations and parameters of this renderer in terms of speed and rendering quality in the supplementary materials.
main_review: Pros:
+ This paper is well presented and experiments were nicely detailed.
+ The experiment comparing using SG vs MC for the DIB-R++ rendering (figure 2) offers interesting insights on the suitability of each renderer based on the material roughness.
+ The optimization task (figure 3) is reassuring that differentiating through DIB-R++ yields meaningful signals with respect to the material and lighting conditions
+ Models trained with DIB-R++ seem to recover better light and material parameters compared to previous works.

Cons:
- Contribution of paper should be stated more clearly - initially the renderer formulation (DIB-R++) seems to be the contribution, but I can not pinpoint the novelty from a graphics/rendering point of view as all of the used formulations were pre-existing and cited. After further reading, the main contribution seems to be in evaluating how the formulation of DIB-R++ fits into the context of training neural networks.
- There is a claim of disentangling lighting and materials in the title and abstract of this paper, but I'm not sure what in the proposed formulation of DIB-R++ would cause for better disentanglement? Seems to be a better renderer for training neural networks in terms of less computations for better performance, but there isn't really anything in the formulation to encourage disentanglement. This is also hinted at by the authors in supplementary materials section C.1. Therefore I find the disentanglement term misleading.

Questions:
- How does DIB-R++ compare to other differentiable renderers, e.g. the one from Mitsuba 2?
- How does the training time of the models using DIB-R++ renderer compare to the methods of "Learning to predict 3D objects with an interpolation-based differentiable renderer" by Wenzheng Chen et al. and "Image GANs meet differentiable rendering for inverse graphics and interpretable 3D neural rendering" by Yuxuan Zhang et al.? Is it slower or faster to train? Does the model converge in less or more iterations? 

Overall, I like the experimentations of this paper and they demonstrate improved performance compared to previous methods. However, the contribution in the renderer does not seem significant enough to me for a NeurIPS paper.
limitations_and_societal_impact: I would prefer if the fact that the training phase relies on multi-view images was stated more clearly, especially in the conclusion/limitations. This was briefly mentioned in the experimental settings of section 6.1. Also might want to mention that this approach is only suitable when we want to recover the geometry and material of a single 3D object captured in an image.
review_title: null

Meta-review: The reviewers found this paper generally well written and interesting, but they had concerns related to novelty, clarity of presentation, and ease of understanding the technical underpinnings of how the model works. The authors addressed many of the concerns and misunderstandings in the rebuttal, but how it looks is that improving upon these points would require a second round of reviews, and thus rejecting the paper in its current form is the right thing to do.
Decision: Accept (Poster)