Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: I thank the authors for the rebuttal and the additional experiments. The additions do partially address my concerns, although not entirely. For instance, the experiments on non-face classes are very preliminary and it is unclear if they work at all (no other views shown). I hope the authors are right that the method will work on other classes after some tuning, but this is not demonstrated in the paper. Overall, I am quite in a borderline mode. I think the paper looks promising and after further improving the experimental evaluation it can become a great publication. But for now the experiments, especially the new ones, look somewhat incomplete and rushed, more suitable for a workshop paper. Therefore, I still lean towards rejection.

---

The paper proposes an approach to learning the 3D structure of images without explicit supervision. The proposed model is a Generative Adversarial Network (GAN) with an appropriate task-specific structure: instead of generating an image directly with a deep network, three intermediate outputs are generated first and then processed by a differentiable renderer. The three outputs are the 3D geometry of the object (represented by a mesh in this work), the texture of the object, and the background image. The final output of the model is produced by rendering the geometry with the texture and overlaying on top of the background. The whole system can be trained end-to-end with a standard GAN objective. The method is applied to the FFHQ dataset of face images, where it produces qualitatively reasonable results.

I am in the borderline mode about this paper. On one hand, I believe the task of unsupervised learning 3D from 2D is interesting and important, and the paper makes an interesting contribution in this direction. On the other hand, the experimental evaluation is quite limited: the results are purely qualitative, on a single dataset, and do not contain much analysis of the method. It would be great if the authors could add more experiments to the paper during the discussion phase.

More detailed comments:
Pros:
1) The paper is presented well, is easy to read. I like the detailed table with comparison to related works, and a good discussion of the limitations of the method and the tricks involved in making it work. I also like section 4 clearly discussing the assumptions of the work, although I think it could be shortened quite a bit.
2) The proposed method is reasonable and seems to work in practice, judging from the qualitative results.

cons:
1) The experiments are limited. 
1a) There are no quantitative results. I understand it is non-trivial to evaluate the method on 3D reconstruction, although one could either train a network inverting the generator, or, perhaps simpler, apply a pre-trained image-to-3D network to the generated images. But at least some image quality measures (FID, IS) could be reported. 
1b) The method is only trained on one dataset of faces. It would be great to apply the method to several other datasets as well, for instance, cars, bedrooms, animal faces, ShapeNet objects. This would showcase the generality of the approach. Otherwise, I am worried the method is fragile and only applies to very clean and simple data. Also, if the method is only applied to faces, it makes sense to mention faces in the title. 
1c) It would be very helpful to have more analysis of the different variants of the method, ideally with quantitative results (again, at least some image quality results). Figure 3 goes in this direction, but it is very small and does not give a clear understanding of the relative performance of diferent variants.

2) A missing very relevant citation of HoloGAN by Nguyen-Phuoc et al.  [1]. It is not yet published, but has been on arXiv for some time. I am a bit unsure about the ICLR policy in this case (this page https://iclr.cc/Conferences/2019/Reviewer_Guidelines suggests that arXiv paper may be formally considered prior work, in which case it should be discussed in full detail), but at least a brief mention would definitely be good.

[1] HoloGAN: Unsupervised learning of 3D representations from natural images. Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, Yong-Liang Yang. arXiv 2019.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: SUMMARY: A new modification to the rendering mechanism in a differentiable renderer  to generate 3D images, trained in an unsupervised manner on 2D images of faces

CLAIMS:
- train a generative model in an unsupervised way on 2D face images,
- by generating 3D representation of [shape, texture, background] and feeding that to a differentiable renderer
- modify the rendering mechanism to be differentiable wrt vertices of the triangular mesh (in addition to the texture)
- curb the problems of training by using shape image pyramid, object size constraint


LIT REVIEW:
Well done, sufficient summarization of past work. But not "first":
it would be pertinent to mention the ICCV 2019 paper "HoloGAN: UNSUPERVISED LEARNING OF 3D REPRESENTATIONS FROM NATURAL IMAGES" (https://arxiv.org/abs/1904.01326), which tackles the exact same problem. It does not use a triangular mesh representation, or a differentiable renderer, instead it uses a 3D feature representation and a neural renderer. However, the problems tackled are very close to avoid mentioning this paper.

Hence, it might not be good to claim
- in the abstract: "the first method to learn a generative model of 3D shapes from natural images in a fully unsupervised way",
- in introduction: "For the first time, to the best of our knowledge, we provide a procedure to build a generative model that learns explicit 3D representations in an unsupervised way from natural images",
- and similar claims in other places.

Also, Pix2Scene (https://openreview.net/forum?id=BJeem3C9F7) also has similar ideas, although they tackled primitive shapes and not faces.

DECISION: This paper has very promising results.
Although it is limited to faces, which the community knows is something GANs are good at modeling because of the inherent structure, it is nevertheless a relevant piece of work in modelling 3D scenes in a graphics way and then training using adversarial learning. I am particularly impressed by the renderings of the depth and texture, and would be interested to explore that area further.

However, it is more pertinent to check how the model performs objects more complicated than faces. A very simple experiment is to try this on ImageNet images, which are also centered and aligned. This would help investigate the possibility of extending this method to more complicated objects than faces.

I would suggest to maybe put more focus on the fact that you have used the traditional graphics pipeline and integrated that into adversarial learning, as opposed to dealing with just weights and biases. That is indeed significant (in my opinion).

Knowing that most GAN training time is spent in overcoming a lot of failures, it would be great if the authors can summarize the failure cases and elaborate on the experiments performed to overcome those failures. This was briefly touched upon in Section 7, but it would be great if they could elaborate more on them possibly in the appendix.

It would be great if the authors can share their code, there was no mention of any possibility of this.

ADDITIONAL FEEDBACK:
Page 5:
...an added constrain*T* on m in the optimization...
...fooled by an inverted dept*H* image...
page 6:
...rendered <remove>attribute and</remove> depth, attribute and alpha map...
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper tries to solve the problem of recovering the 3D structure from 2D images. To this end, it describes a GAN-type model for generating realistic images, where the generator disentangles shape, texture, and background. Most notably, the shape is represented in three dimensions as a mesh made of triangles. The final image is rendered by a fixed differentiable renderer from a randomly-sampled viewpoint. This allows the model to learn to generate realistic 3D shapes, even though it is trained only using 2D images.

The authors introduce a novel renderer based on the Lambertian image model, which is differentiable not only with respect to the texture but also with respect to position on mesh vertices, which allows better shape learning compared to prior art. Authors also identify some learning ambiguities: objects can be represented by the background layers, and sometimes object surface contains high-frequency errors. These are addressed by generating a bigger background and randomly selecting its part as the actual background, and by averaging shapes generated at different scales to smoothen the surface of generated objects, respectively. Authors do mention pitfalls of the model in the conclusion: fixed-topology mesh, the background is not modelled as a mesh, the model works only with images containing a single centred object, the image model is Lambertian.

I think that the approach is extremely interesting, addresses an important problem, and shows promising results. However, I vote to REJECT this paper, because the evaluation is insufficient, and the paper lacks clarity.

The approach is evaluated only on a single dataset and is not compared to any baselines. While results from ablations of the model are provided, they are only qualitative, consisting of a single example per ablation, and are hard to read and interpret---in particular, the provided description of the ablations and corresponding results is unclear. There are no quantitative results in the paper, and it is difficult for me to judge how good the method is given only qualitative examples from a single dataset.

As for clarity, I think that the distinction between supervised, unsupervised and weakly supervised learning in section 2 in unnecessary, does not add value to the paper, and can confuse the reader. Section 4 contains some unnecessary assumptions and incorrect claims. For example, the renderer R doesn't need to be able to generate perfect images for the approach to work; I think Theorem 1 is also incorrect since it does not take e.g. mode collapse into account, which prevents the learned distribution from being the same as the data distribution. Section 5 is very unclear, with practically no explanation for equations (6-11), which makes them very difficult to decipher.

The related works section is quite thorough, but the authors missed two extremely relevant papers: [1] and [2], which do a very similar thing and contain some of the ideas used in this paper.

I think the paper would be very valuable if the differentiable renderer was clearly explained and more evaluation and comparisons with baselines were provided.

[1] Rezende et. al., "Unsupervised Learning of 3D Structure from Images", NIPS 2016.
[2] Nugyen-Phuoc et. al., "HoloGAN: Unsupervised learning of 3D representations from natural images", ICCV 2019.


=== UPDATE ===
I appreciate adding more examples for ablations, FID scores and the LSUN dataset experiments. However, I still think that the exposition could be significantly improved, as eg. the description of the differentiable renderer is difficult to follow -- the equations should be better explained. Also, Figure 3. is difficult to understand; and the caption saying that "one rectangle overlaps another" is not helpful. 

I think this is really cool work, but due to the lack of clarity, I think it shouldn't be accepted at this conference. Having said that, I am increasing my score to "weak reject" because of the improvements.

review_title: Official Blind Review #1

Meta-review: The paper proposes a GAN approach for unsupervised learning of 3d object shapes from natural images. The key idea is a two-stage generative process where the 3d shape is first generated and then rendered to pixel-level images. While the experimental results are promising, the experimental results are mostly focused on faces (that are well aligned and share roughly similar 3d structures across the dataset). Results on other categories are preliminary and limited, so it's unclear how well the proposed method will work for more general domains. In addition, comparison to the existing baselines (e.g., HoloGAN; Pix2Scene; Rezende et al., 2016) is missing. Overall, further improvements are needed to be acceptable for ICLR. 


Extra note: Missing citation to a relevant work
Wang and Gupta, Generative Image Modeling using Style and Structure Adversarial Networks
https://arxiv.org/abs/1603.05631
Decision: Reject