Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: #### Summary ####
This paper studies an interesting inverse-graphics problem. It proposed a novel method to learn 3D shape reconstruction using pre-trained 2D image generative adversarial networks. Given an image containing one single object of interest, it first predicts the graphics code (e.g., viewpoint, lighting, depth, and albedo) by minimizing the reconstruction error using a differentiable renderer. The next step is to render many pseudo samples by randomization in the viewpoint and lighting space, while keeping the predicted depth and albedo fixed. A pre-trained 2D image GAN is further used to project the pseudo samples to the learned data manifold through GAN-Inversion. Finally, these projected samples are added to the set for the next round optimization. Experimental evaluations have been conducted on several categories including face, car, building, and horse.


#### Comments ####
Overall, this is a very interesting paper with good presentations, promising experimental results, and solid quantitative comparisons with the previous work. Reviewer would like to point out the potential weakness of the paper as follows.

W1: Though impressed by the results (especially the proposed method works for horse and building), reviewer suspects the paper only works in a very simplified setting: (1) the GAN was previously trained on a large amount of 2D images of a single category with many variations in identity, viewpoint, and lighting; (2) the initialization (or step 1 in Section 3.1) step seems very critical to the overall performance; and (3) viewpoint and lightning randomization seems have to be hand-tuned. Reviewer would like to see the discussions on the underlying assumptions more explicitly. In addition, reviewer would like to know how does the method generalize to “dirty” data: people with sunglasses, people with noticeable earrings, people partially occluded by wavy long hair, and people with a side view (looks like the input has to be a frontal face image). Same question applies to those non-convex shapes: a convertible car or a car with the window open. Reviewer suspects the method in the current form cannot handle them well.

W2: Some important experimental settings are neither presented nor clarified. For example, it is not clear what is the difference between “Ours (3D)” and “Ours (GAN)”, which should be clarified. For image editing (see Figure 6 and Figure 8), reviewer sees a noticeable change in background color sometimes (e.g., second row in Figure 6). It would be good to give a very detailed explanation of the image editing process (e.g., what’s the input and output format in each stage). As ellipsoid was used to initialize the face shape, reviewer would like to know what was the initialization for other categories such as building and car (see Figure 10). 

W3: It would be good to report the time spent on the computation and optimization and how it is compared to the baselines in Table 1 and Table 2. This is very important metric to report as a fair comparison to the previous work.

== Post-rebuttal Comments ==

I am raising my score from 7 to 8, as author responses addressed my comments well (especially answer to W1 and the Figure 13) than expected.
review_title: [Official Review] 

final_score: rating: 7: Good paper, accept
review_content: Pros:
1. This is the first work that attempts to reconstruct 3D shape from 2D image in an unsupervised way using GANs. The idea is neat: Use networks to predict four 3D parameters and use GAN to generate / synthesize the images corresponding to a set of parameters. Then these synthesized images can be used as pseudo ground truth to train the 3D parameter network.
2. The experiments are comprehensive to support the effectiveness of the proposed pipeline. 2 tasks are evaluated: 3D shape reconstruction and object-aware image manipulation. On 3D shape reconstruction, performances are reported on two datasets and demonstrated it outperforms SoTA method by a large margin. On image manipulation, the visualization results look reasonable and visually better than previous method.

Cons / Questions:
1. The authors claim in the introduction section that this proposed method has advantage over previous method as it doesn't assume symmetry of the instance. But in this proposed method, a symmetrical ellipsoid is used as the shape prior. Is this a stronger implicit assumption than the symmetry assumption? Is there any experiment to explore how the shape prior affects the models' training results? E.g. what if a non-convex shape prior is provided, what if asymmetrical prior is provided?
2. In the introduction section, the authors use `'building' as an example to show that previous method's symmetry assumption cannot work. But in the experiments and comparison with previous works, all data used are symmetrical: human face, animal face, cars, etc. Visualizations on buildings are shown in appendix but there is no quantitative analysis or comparing with current SoTA method.
3. How is the generator initialized? As the generator is always fixed during training, I assume the generator is using some pre-trained network? If so, can we still call this network fully unsupervised?
review_title: This paper firstly proposes an unsupervised method to recover 3D shape from 2D image using GANs. The results are state-of-the-art and the experiments are comprehensive.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes an iterative method that jointly estimates viewpoints, light directions, depth, and albedo from single images, by projecting intermediate renderings to the nautral image manifold. Intuitively, the method works by generating, with pre-trained GANs, multiple views of the same object under different lightings, and then inferring 3D shapes from those variants. The key idea is to use pre-trained 2D GANs to make such data generation photorealistic. The authors also demonstrate 3D edits, such as 3D rotation and relighting, that one can perform after running their model.

I like this paper because
(1) it presents the novel idea of "generating", by GAN inversion, photorealistic multi-view, multi-light data of the given real object, from which the 3D shape can then be estimated;
(2) extensive evaluations were performed to demonstrate the high quality achieved; and
(3) one can perform 3D edits, such as 3D rotation and relighting, on top of the model outputs. Having explicit 3D understanding for relighting makes a lot of sense to me, and this paper presents a new angle of doing so by GAN inversion. 

In terms of drawbacks, this paper would benefit from the following experiments or clarifications:
(1) How crucial is the size of the dataset for GAN pretraining? For example, if the dataset is small, not covering many of the face poses, I imagine the GAN projections may not always look realistic, thereby causing the shape estimation to degrade. Such failure cases or studies should be shown so that the reader understands what impact the dataset bias or size has on the final results. An example would be a plot of shape reconstruction error w.r.t. the dataset size or face pose coverage.
(2) How robust is the algorithm to the shape initialization (ellipsoid shape)? More importantly, what about its location -- what if the "off-the-shelf scene parsing model" fails so that the ellipsoid is placed off the main object? If the model fails because of bad initializations, what do the failure modes look like? Are the shapes completely garbage, or something that looks like the initialization? 
(3) Why is the viewing direction parameterized in R6 (I presume the start and end XYZs)? Shouldn't it be in S2, just like the light direction? If they are in R6, then results on zooming in/out should be included. Otherwise, there seems to be no point in defining them in R6.
review_title: Good work that estimates 3D shapes w/o supervision with the help of pre-trained GANs

Meta-review: The paper proposes to use pre-trained 2D (i.e., image) GANs as a mechanism for recovering 3D shape from a single 2D image. The work demonstrates impressive results on not only human and cat faces, but also cars and buildings. The method is demonstrated with qualitative results and quantitative results on multiple datasets and tasks.

The reviewers were persuaded by the novelty and "neatness" of the idea (and the AC is in agreement) as well as the results. At submission time, there were some concerns with experimental details. For instance, there was a question of how carefully the settings have to be tuned (always a concern with unsupervised methods) as well as an overarching concern about the initialization and whether the method will work on less clean data. The reviewers (and the AC) seem to think that these have been sorted out in discussion. 

All three reviewers were in favor of acceptance and the area chair is inclined to agree with the reviewers. In particular, the AC finds the work interesting and compelling. While there is an updated version already uploaded during the discussion, the AC encourages the reviewers to double check all the questions from the reviewers and include the answers from the discussion into the camera ready (even these results are in the appendix).
Decision: Accept (Oral)