Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes to let the discriminator estimate the geometry of the scene from the 2D rendered version of the generated 3D scenes to improve the geometry of the generated scenes.
- The discriminator estimates six geometry factors from a 2D rendered scene.
- The six geometry factors = a depth map, a normal map, an albedo map, a global lighting direction, a viewpoint, and confidence maps.
- The geometry factors supervise the generator to match them.
- The GT depth map and its normal map for the discriminator are given by the volume rendering of depths.
- The proposed method improves various 3D-aware GANs including pi-GAN, StyleNeRF and VolumeGAN.
  - 3D shapes are more accurate in the qualitative results.
- Experiments are conducted on FFHQ, AFHQ cat, LSUN bedroom.

strengths_and_weaknesses: ### Strengths
- Letting the discriminator learn the geometry is novel.
- Some sources of supervision are self-supervision.

### Weaknesses
- Confusions
  - L137-139 The D estimates six factors: depth, normal, albedo, lighting, viewpoint, and confidence.
  - L93 The G produces color and density.
  - L163-170 describes the procedure for producing depth and normal from the density of a generated scene.
  - Eq(6) The G tries to match the geometries estimated by the D.
  - **How do the other four geometries guide the generator?**
  - L161 The normal of the generated scene supervises the D.
  - L121-127 The D tries to match the geometries extracted by the G.
  - **How do the other four geometries from the D are supervised? The problem is under-constrained by the below objectives.**
  - L140 The geometries reconstruct the image.
  - L143 The symmetry assumption

- Writing
  - Redundant descriptions
    - D estimates six factors: L95, L137, L151
  - L150-159 is difficult to understand.
  - The description of the proposed method can be largely improved. Current structure is very confusing because the description for G/D architecture and G/D objectives are entangled.

- Evaluation
  - Reprojection error is not sound because there are corner cases where the error can be small with wrong shapes: planar shape.
  - I am not sure  whether the same ablation study and GAN inversion on other backbones will lead to the same conclusion. Inferiority of pi-GAN compared to other 3D-aware GANs is well-known in the research community.

questions: Please refer to the above section.

limitations: Yes. Necessity of pretrained networks for extra supervision and its domain gap.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an improved discriminator architecture for 3D-aware image synthesis. The authors advocate to enforce consistency between the geometric features of images (depth, surface normals etc) rendered from the generative NeRF as well as predicted by off-the-shelf methods (Unsup3D). Such context can make discrimination between real and fake images easier for the network. Experiments show that the proposed geometry-aware discriminator as an additional building block helps improve several baseline methods on human/cat faces and LSUN bedroom datasets.

strengths_and_weaknesses: 
Strengths:
- I like the motivation of making the discriminator geometry-aware, which is what prevalent 2D image discriminators are largely lacking. The idea of utilizing off-the-shelf methods is also simple, and elegantly bridges with the framework of generative neural radiance fields.
- The results clearly show that add geometry awareness improves the 3D geometry interpretability of image generation. Although not strictly improving on the FID score for every metric on every backbone/dataset, overall the results show significant improvements both qualitatively and quantitatively. 
- The paper is nicely written and the presentation is very clear.

Weaknesses:
- It would be good if the authors could provide some failure case analysis to better understand the limitation of the proposed method.
- I think Section 4.4 seem somewhat extraneous to the paper, as it's not central to the paper, and (in addition to [18] and [33]) introduces yet another building block IBRNet [31] to the system. The authors should also elaborate why such novel view synthesis component is necessary to improve the synthesis quality, as the original proposed model is already a 3D-aware image synthesis method based on NeRF.
- Why is GRAF [29] called G-NeRF throughout the entire paper? I believe G-NeRF is more commonly referred to another paper [A]. If the authors meant a GAN + NeRF framework in general, I would recommend not inventing an abbreviation to avoid confusion.
- The loss function notation in Eq 2 and 5 seems to be inconsistent.

[A] Meng et al. "GNeRF: GAN-based Neural Radiance Field without Posed Camera." ICCV 2021.

questions: It would be great if the authors could address all the concerns listed above.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work addresses the task of 3D-aware image synthesis. In particular, the authors present a general formulation called GeoD that can be used to improve the performance of various existing 3D-aware GAN variants wrt to the plausibility of the generated images and the reconstruction quality. Specifically, the authors propose to use the output of the discriminator to supervise the extracted geometry of the generator. They evaluate their framework on various GAN variants such as StyleNeRF and VolumeGAN on 3 datasets: FFHQ, AFHQ cat and LSUN bedrooms and show that their method outperforms existing methods wrt to the FID score, the Scale-Invariant Depth Error (SIDE) and the Reprojection Error (RE).

strengths_and_weaknesses: # Strengths:
------------

1. I think that the idea of having a geometry-aware discriminator that is used to supervise the predicted geometry of the generator is very interesting. Moreover, the authors demonstrate that their model can be used in combination with various existing approaches such as StyleNeRF and $pi$-GAN and significantly boost their performance. This clearly shows the contribution of this work.

2. Based on the provided results it seems that the proposed method outperforms methods with non geometry aware discriminators wrt FID, SIDE and RE scores.

# Weaknesses:
-------------

1. I am wondering why didn't the authors consider EG3D (Efficient Geometry-aware 3D Generative Adversarial Networks by Chen at al.) as a baseline? This method has demonstrated superior results in comparison to $\pi$-GAN for example. Hence, I think EG3D would have been a stronger baseline. In particular for the case of EG3D I am wondering whether replacing their StyleGANV2 discriminator with GeoD would still lead to improved results.

2. A more powerful baseline in terms of reconstruction fidelity that the authors could consider in  their evaluation is StyleSDF. I am wondering whether any of the existing 3D-aware GAN variants with the proposed discriminator would still be better than StyleSDF.

3. I think that another important aspect of the model that is not properly discussed is its efficiency. Without a doubt using any of the existing 3D-aware GAN models where only the generator is 3D-aware would probably require less time to train and probably would be able to also generate scenes in various resolutions faster, since they do not have to account for the geometry-aware discriminator. However, I am wondering how much slower is the proposed method in terms of training epochs and how much time is needed for rendering various resolution size images? I believe this analysis is valuable since it will allow us to see what are the limitations of the proposed method.

4. Another experiment that the authors could consider including is showing synthesized images, normal maps and 3D meshes from camera poses that significantly differ from the training poses. It would be interesting to see whether this formulation yields smoother reconstructions even under these more challenging scenarios.

5. Since the output of the discriminator is used as pseudo-labels to supervise the geometry produced by the generator, I am wondering how prone is the proposed model to local minima. For example, during the first stages of training when the pseudo-labels are quite noisy, is it possible for the model to get stuck? From Table 2, it seems that the "Trained from scratch" variant is the performing the best in terms of image plausibility (FID) and reprojection error (RE). However, this result is a bit unexpected for me, since I would have expected the Pretrained variant to be the best performing one. Any idea why this is the case?
questions: 1. Information regarding the parametrization of the normal map $\mathbf{n}$, the albedo map $\mathbf{a}$ and the global lighting direction $\mathbf{l}$ are not provided  neither in the main paper nor in the supplementary. This needs to be clarified to ensure that reproducibility. Similarly, how is the confidence maps $\bc$ parametrized?

2. In L151-152, the authors state "One encoder will output albedo $\mathbf{a}$, normal $\mathbf{n}$, roughness $\mathbf{r}$ ...". What is roughness and how is it used? This is not explained in the text.

3. I find it a bit weird that for the case of Figure 2 while the normal maps look noisy but have captured the 3D geometry to some extent the authors did not show the 3D meshes. How do the 3D meshes look like? 

4. To generate the results of Figure 5, what generator did the authors use?

5. I think that a pictorial representation of the geometry branch in GeoD both for the case of objects and scenes would significantly facilitate understanding the various details presented in section 3.3.

6. Instead of using the proposed geometry parametrization, I think that using an off-the-shelf depth estimator to extract depth maps. My intuition is that this variant would be easier to train and would lead to better results. Have the authors considered this setup?

limitations: I think that the limitations of this work are adequately discussed in the main paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The current discriminators of 3D-aware GANs only discriminate between real and fake. This paper proposes learning a geometry-aware discriminator to improve 3D-aware GANs. In particular, the discriminator is additionally asked to derive the normal from the inputs, which is then applied as supervision for the generator. The author also introduces a consistency branch for the discriminator and claims such a design could improve multi-view consistency. Experimental results show the effectiveness of the proposed discriminator in both image and geometry quality.
strengths_and_weaknesses: Strengths:

The problem studied in the paper is important and the method is a possible direction for improving the view consistency of 3D GANs.

The proposed method shows an improvement in geometry quality. 


Weaknesses:

1. The discriminator learns to regress 3D attributes with an autoencoding training objective for real images. The discriminator then estimates the normals of fake images, which are used as pseudo labels to supervise the generator. Would it be better to use a separate network as the 3D attributes regressor? 

2. In equation 12, the author should provide enough details to understand the equation. What do all the terms mean? For example, explain the meaning of d_{u+1} and d_{u-1}, and detail the elements of the intrinsic matrix K. 

3. The authors did not verify the effectiveness of GeoD on high-resolution experiments (e.g., at 1024x1024 resolution). The qualitative results are not very impressive. As shown in the demo video, the generated images are blurry. The ablation study was limited to 64x64 resolution images, and the results were not very convincing. The same goes for the GAN inversion experiment. I recommend the author demonstrate the effectiveness of GeoD on high-resolution images. 

4. The paper structure needs to be adjusted. For example, Section 4.4 proposes a consistency branch, which should be moved to the methods section. 

5. In Figures 2 and 3, why is the bedroom mesh not shown? 

6. The approach seems to be ad-hoc. For example, the author employs different architectures for the geometry branch for objects and scenes, limiting the scalability of the method.

questions: Please see the weaknesses. 
limitations: Yes, they did.
review_title: null

Meta-review: metareview: This paper presents a new 3d-aware image generative model. Compared to previous methods such as pi-GAN, the paper introduces a new geometry-aware discriminator, which helps the model learn better 3D shapes. Many reviewers found the paper well-written, the idea novel/interesting, and the results convincing. They also raised several concerns regarding the evaluation, baselines, and high-res synthesis. The rebuttal has addressed most of the concerns. The AC agreed with most of the reviewers and recommended accepting the paper. 
confidence: Certain
Decision: Accept