Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors provided a framework for inverse graphics, i.e., infer 3D mesh, light, and texture from a 2D image. They first use StyleGAN to generate realistic multi-view images, and then trained their model with a differentiable graphics renderer.

Pros
- Training on data generated by StyleGAN is novel. It is easy to control the view of rendered images, but they usually do not look real. Real images look real, but we usually do not know the viewpoint (most annotated datasets are either small or have limited annotation quality). The authors proposed a method to generate images of the same category of objects with the same viewpoint (Figure 2), which addresses this issue -- the viewpoints are known, and the images look real.
- The experimental part is impressive. 3D reconstructions look realistic, and the authors demonstrated the effectiveness to train on StyleGAN by comparing their model to a neural network trained on PASCAL3D+.
- The way the authors generate data by StyleGAN is also novel. They empirically realized some latent code in StyleGAN controls the camera viewpoint, so that they only need to choose some latent codes that well spanned over the space. It is a wise and elegant way to control the viewpoints of an object in StyleGAN.

In summary, compared to previous methods, generating multi-view images from StyleGAN solves the unrealistic and unknown viewpoint issue, and the results look impressive.
review_title: Inverse graphics work with impressive results

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
This paper proposes to couple a GAN, an inverse graphics network, and a differentiable renderer. The authors base their work on StyleGAN, and use the observation that a specific part of the latent code corresponds to camera view-point to rapidly annotate a large amount of synthetic images with approximate camera pose. They then use these images and rough annotations to train the inverse graphics network to provide 3D and texture data. The differentiable renderer is used to synthesize 2D images from 3D, which can be compared to the input for consistency. In a second step, the authors use the inferred 3D data to disentangle the latent space of StyleGAN to allow to use it as a controllable renderer.

--- Strengths ---

The presented results look great. The inverse graphics network, seems to get both shape and textures mostly right. This is especially remarkable, given the small amount of supervision that was used. The controllable StyleGAN provides very plausible results. Most importantly, results  on real images are shown, and indicate the the presented approach does make progress in brining inverse graphics networks to real images.

While the individual parts of this work are taken from other prior works, the specific combination of different components is novel and quite creative. Using a GAN to train an inverse graphics network is to the best of my knowledge novel. So is using an inverse graphics network to turn a GAN into a controllable renderer. There are also various key-insights that make the approach work, such as the that StyleGAN is partially  disentangled with respect to view-point, or the use of multi-view consistency when training the inverse graphics network.

--- Weaknesses ---

The paper is overall well written, but leaves out some technical descriptions which make it not self-contained and hard to reproduce. Specifically, I'd like to ask the authors to elaborate more on the individual loss terms in Equation (1). Especially, the 3D-related losses $L_{lap}$ and $L_{mov}$ are unclear. 

While many results are shown in the supplement, I would have loved to see a video that shows more results. For example, rotating the obtained 3D models, or showing interpolations of various factors for the controllable renderer. Even if the results are not perfect, a video would help to judge the overall quality of the results.

--- Summary ---

This paper shows an interesting pipeline with good results. It presents some non-trivial observations and effectively leverages them into a complete system that looks quite impressive.

Typos:

Page 3, last sentence: "conten"

--- Post rebuttal ---
After reading the other reviewers comments and the rebuttal, I'm keeping my initial score.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes to use StyleGAN images to train a differential renderer, then in turn to use this differential renderer to train a more controllable GAN (dubbed StyleGAN-R) which allows to perform independent manipulation of shape, background, and texture. There are many different things in this paper, and I am struggling a bit to decide what's the main contribution, here are the ones that seem the main one to me (note I am neither a GAN nor a neural renderer expert):

1. training a neural renderer with GAN images: to the best of my knowledge, that has never been demonstrated and is nice in itself. However, the way this is achieved is not completely satisfying: the entire idea is based on the empirical observation that the first part  of styleGAN latent codes correspond to viewpoint; the training requires manual selection of "good" latent codes and annotation of the viewpoints (once, this is fast, but remains very unsatisfying and strongly limits the number of viewpoints); further image filtering is required using a trained MaskRCNN and masks are necessary for training. Additionally, while this is nice to see, there seem to be very little/no technical contribution in this part which is mainly engineering and empirical results

2. Another way to see the first contribution is to say that the paper demonstrate that using GAN generated images allows to improve over using datasets of manually annotated images. This has been an important hope as an application of GAN, and the beginning of the results section (4.1) seems to emphasize this aspect: less annotation time and higher quality than training a differential renderer on Pascal3D. Again, while this would make a very nice story, this is not related to any technical contribution. Additionally, it's hard to really trust the results on this, since the training of the networks are complicated and the results are hard to evaluate, it might be that the hyperparameters are simply better adapted to training with the StyleGAN training set than with Pascal3D (the presented AMT experiment is good for evaluation, but it is too expensive to do to use it to choose training hyperparameters)

3. StyleGAN-R for decorelated image manipulations. This is again very appealing, but done in an unsatisfying way and with limited results (only on cars). There are many tricks involved in making this work, from the choice of 144 (!?) dimension among the 2048 for the latent vector to the complicated stage-wise training. The results of this step look good, but are only shown for cars, since reconstruction was also done for horses and birds, I can only assume that this doesn't work for other categories.

To summarize, I think there are several cool stories in this paper and some appealing results. While the methods remain unsatisfying, the paper shows interesting proof of concepts, I would thus tend to accept it, but I could be convinced otherwise especially if other reviewers who know the field better than I do point to other papers demonstrating similar points. 

I list bellow more specific issues/requests:

a. whether it works or not, I want to see the equivalent of figures 8-9-10 with other categories. This should also be discussed in limitations (or the paper should demonstrate it works). I am annoyed that this is not discussed more visibly in the paper and kind of hidden. No clear answer on this point would lead me to recommend rejection (but I could still recommend accept even if the method doesn't work very well on other categories)

b. I would like to see a comparison between StyleGAN (using the annotated viewpoints) and styleGAN-R for camera controler (figure 8, but using the code predicted by the approach before styleGAN finetuning - not the optimized code as in fig 7), that seems the natural baseline

c. I don't agree at all with the claim in 4.1 that the approach works with articulated classes: horses are indeed articulated, but looking at the horse dataset used (figure C in appendix) reveals that the selected images from styleGAN do *not* present any articulation (on the contrary to most real horse datasets). Thus the approach works simply because styleGAN do not present the same diversity as natural images, and would not work for actual articulated horses, this is more a limitation that a strength.

d. fig 7 comparison with optimization: to me the natural approach/baseline would be to use a much simpler CNN to predict the latent code (and potentially do some local optimization afterward). This type of learning for optimization approach often regularize the problem well, and lead to good results. This would also be more similar in spirit to the proposed approach.

review_title: review before discussion: nice stories, unsatisfactory solutions, and nice proof of concept results

Meta-review: The paper proposes to bring together a GAN, a differentiable renderer, and an inverse graphics model. This combined model learns 3D-aware image analysis and synthesis with very limited annotation effort (order of minutes). The results look impressive, even compared to training on a labeled dataset annotation of which took several orders of magnitude more time.

The reviewers point out the novelty of the proposed system and the very high quality of the results. On the downside, R2 mentions that the model appears over-engineered and some important experimental results are missing. The authors’ response addresses these concerns quite well.

Overall, this is a really strong work with compelling results, taking an important step towards employing generative models and neural renderers “in the wild”. I think it can make for a good oral. 

Decision: Accept (Oral)