Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator
Authors: Zifan Shi
Yinghao Xu
Yujun Shen
Deli Zhao
Qifeng Chen
Dit-Yan Yeung
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D-aware image synthesis
generative adversarial network

Abstract: 3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. A popular solution is to adopt the generative adversarial network (GAN) and replace the generator with a 3D renderer, where volume rendering with neural radiance field (NeRF) is commonly used. Despite the advancement of synthesis quality, existing methods fail to obtain moderate 3D shapes. We argue that, considering the two-player game in the formulation of GANs, only making the generator 3D-aware is not enough. In other words, displacing the generative mechanism only offers the capability, but not the guarantee, of producing 3D-aware images, because the supervision of the generator primarily comes from the discriminator. To address this issue, we propose GeoD through learning a geometry-aware discriminator to improve 3D-aware GANs. Concretely, besides differentiating real and fake samples from the 2D image space, the discriminator is additionally asked to derive the geometry information from the inputs, which is then applied as the guidance of the generator. Such a simple yet effective design facilitates learning substantially more accurate 3D shapes. Extensive experiments on various generator architectures and training datasets verify the superiority of GeoD over state-of-the-art alternatives. Moreover, our approach is registered as a general framework such that a more capable discriminator (i.e., with a third task of novel view synthesis beyond domain classification and geometry extraction) can further assist the generator with a better multi-view consistency. Project page can be found at https://vivianszf.github.io/geod.

Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator
Zifan Shi!*
Yinghao Xu
Yujun Shen
Deli Zhao
Qifeng Chen
Din Yeung'
HHUST
CCUHK
3 Ant Group
Abstract
3D-aware image synthesis aims at learning a generative model that can render photo-realistic 2D images while capturing decent underlying 3D shapes. A popular solution is to adopt the generative adversarial network (GAN) and replace the generator with 3D renderer, where volume rendering with neural radiance field (NeRF) is commonly used. Despite the advancement of synthesis quality, existing methods fail obtain moderate 3D shapes. We argue that, considering the two- player game in the formulation of GANS, only making the generator 3D-aware is not enough. In other words, displacing the generative mechanism only offers the capability, but not the guarantee, of producing 3D-aware images, because the supervision of the generator primarily comes from the discriminator. To address this issue, we propose Geol) through learning a geometry-ave aware discriminator to improve 3D-aware GANs. Concretely, besides differentiating real and fake samples from the 2D image space, the discriminator is additionally asked to derive the geometry information from the inputs, which is then applied as the guidance of the generator. Such simple yet effective design facilitates learning substantially more accurate 3D shapes. Extensive experiments on various generator architectures and training datasets verify the superiority of GeoD over state-of-the-art alternatives. Morcover, our approach is registered as a general framework such that a more capable discriminator (i.e., with third task of novel view synthesis beyond domain classification and geometry extraction) can further assist the generator with a better multi-view consistency. Project page be found here.
Introduction
3D-aware image synthesis has received growing attention due to its potential in modeling the 3D visual world. Compared to 2D generation [1, 13-16, 28], which primarily focuses the image quality and diversity, 3D-aware generation expected to also learn an accurate 3D shape underlying the synthesized image. However, existing generative models, such the popular generative adversarial network (GAN) [8], fail to capture the 3D information from 2D images, as they produce an image based on 2D representations only. To bridge this gap, recent studies [2, 3, 9, 23-25, 29, 30, 34, 36] seek help from 3D rendering techniques, like neural radiance field (NeRF) [19], and propose to replace the generator in GAN with 3D renderer. Such replacement enables explicit control of viewpoints when generating an image, benefiting from the 3D awareness of the generator.
Many attempts have been made to improve 3D-aware GANs from the generator side, with either better representations [3, 5, 24, 25, 30, 34] or novel architectures [2, 9, 23, 29, 36]. Despite the advancement of 2D synthesis quality, they still suffer from unsatisfactory 3D shapes, which appear flatten, noisy, or uneven surfaces as shown Fig. la. We argue that, given the imnerato-discriminnaoo competition in training GANs, only making the generator 3D-aware is not enough. Specifically, the generator
Work was done during internship under Ant Group.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

G-NeRR
G-NeRF
Z
Fike insage
Z
Goomesrz
Fole
5
Glomeay
Mesh
Real
Mesh
Real
GeoD
R/F
(a) G-NeRF +D
(b) G-NeRF GeoD
Figure 1: Concept comparison between (a) existing aw-rw GANs where only the generator is made 3D-aware with the help of NeRF [19], and (b) our GeoD where the discriminator supervises the generator with the extracted geometry. Such an additional task (i.e., geometry extraction) substantially improved 3D-aware GANs with more accurate 3D shapes (see meshes). Here, z and denote a sampled latent code and camera pose, respectively, while "VR" stands for volume rendering [19].
obtains supervision (i.e., gradients) mostly from the discriminator, however, the discriminator remains differentiating real and fake images from the 2D image space, shown in Fig. la. Under such a case, even though the generator is potentially capable of learning 3D shapes, it is still exposed to the risk of appearaape ambiguity (i.e., well rendered 2D appearance does not necessarily imply decent 3D shape) because of the lack of adequate 3D guidance.
To tackle this problem, we propose geometry- 3Daaware GANs from the discriminator side. As shown in Fig. Ib, besides the basic task of real/fake domain classification, we assign the discriminator an additional task of geometry extraction. For one thing, the classification branch remains the same as that in conventional GANs, and helps the generator improve the realness of the syntheses from the 2D space. For another, the geometry branch targets deriving the shape-related information, such as depth and normal, from a given image. Such information is employed geometry branch is unsupervisedly learned on real data only and jointly updated with the generator, forming co-evolution manner.
discriminator, termed GeoD, which advances
extra signal supervise the generator for fair 3D shape. Here, the
We evaluate our approach on various 3D-aware GAN variants, including r-GAN [2], StyleNeRF [9], and VolumeGAN [34], to verify its generalization ability. Both qualitative and quantitative results on a range of datasets, including FFHQ (human faces) [14], AFHQ cat (animal faces) [4], and LSUN bedroom (indoor scenes) [35], validate that GeoD outperforms the baselines with far more accurate 3D shape withoul sacrificing the image quality. Furthermore, we show that our framework is general such that it can be applied for other 3D-related improvements. In particular, when assigned with third task of novel view synthesis, the discriminator can urge the generator to learn better multi-view consistency, demonstrating motivation of making the discriminator 3D-aware.
2
Related work
3D-aware image synthesis. 3D-aware image synthesis has received lots of attention recently. The key difference from 2D GANs is the use of 3D representation in the generator. VON [37] and HoloGAN [22] use the voxelized 3D representations to perform 3D-aware image synthesis, but these methods suffer from the 3D inconsistency due to the lack of underlying geometry. [2, 5, 24, 29, 30] introduce NeRF [19] to synthesize 3D-aware images, and [3, 9, 23, 34, 36] propose render low- resolution feature maps with NeRF and then leverage powerful 2D CNNs to generate high-fidelity images. The underlying shapes of these approaches, however, are noisy and imprecise, To tackle the inaccurate shape, ShadeGAN [25] proposes to introduce the albedo field instead of the radiance field, and adds a lighting constraint by explicitly modeling the illumination process. However, the discriminators of these approaches still distinguish images from 2D space, which is risky to bring the shape-appearance ambiguity due to the lack of 3D supervision. Our main goal is to improve 3D geometry from the discriminator's perspective by introducing geometry-aware discrimination to supervise the 3D-a generator. It is worth noting that our geometry-aware discriminator can be casily applied to the approaches built upon the neural radiance field.

---

Geometry extraction. Extracting geometries from images has been a most cases only monocular data is available train a generative model, we only discuss geometry extraction from monocular images here. A traditional way is to train a network with multi-view
problem. As in
data or labeled geometries in supervised manner, such as depth estimation [7, 26, 27] and inverse rendering [11, 17, 88, 32]. By introducing differentiable renderers, recent work [33] starts extracting geometries from monocular images unsupervisedly. Concretely, the network first decomposes an image into several geometry factors (e.g., albedoo, depth, and lighting), and a differentiable renderer then reconstructs the image following physical models. However, such methods require assumptions, such symmetry and shading, and thus they are only applicable on object images. All these tasks and models can be integrated into the discriminator for geometry extraction.
3 Method
3.1 Preliminaries
Generative neural radiance field. The neural radiance field (NeRF) [19] adopts a continuous function, F(x,d) d) = (c, rT), to learn the RGB color с E IR and volume density a E R given the 3D coordinate R$ and the viewing direction d S F(..)) -) is typically parameterized with a multi-layer perceptron (MLP). To render an image under the given camera pose, the pixel color is obtained via volume rendering along the corresponding ray with near and far bounds. Owing to NeRF's strong performance in 3D reconstruction and novel view synthesis, recent attempts on 3D-aware image synthesis propose introduce the generative neural radiance field (G-NeRF) [29], G(x, d, 2) = (c, to the generator. It improves the 3D consistency of synthesized images across different camera views, due to the underlying geometry encoded in G-NeRF.
Geometry extraction from monocular images. Geometry extraction from single view attempts to predict 3D information, such as depth, normal, and albedo, from monocular photographs. Unsupervised extraction is made possible by diffcrentiable rendering, which enables the gradient calculation of 3D objects through images during the rendering process. It can be formulated into an autoencoding process based on differentiable rendering to learn the geometry factors in an unsupervised manner. The geometry encoder ((-) maps the image into geometry factors, e-g., depth, normal, albedo, and lighting. The decomposed factors are then used to reconstruct the input image through a differentiable renderer.
3.2 Geometry-aware discrinination
Recall that we introduce a geometry branch to the discriminator to provide 3D supervision signals for the generator. In this part, we present how the geometry branch is incorporated build geometry-aware discriminator for improving 3D-aware image synthesis.
Learning geometry-awe discriminator with real images. Recent attempts 3D-aware GANs inherit 2D discriminator which only distinguishes images in 2D space, suffering from the limited 3D supervision for the generator. To make the maximum use of the 3D information encoded in the 2D images and prevent the discriminator from distinguishing images without awareness of geometry, assign geometry extraction task beyond bi-class domain classification, which is derive the geometry information from the given real image I,. To achieve this, we introduce geometry branch (-)) and supervise the discriminator with an extra autoencoding training objective:
I, (),()))
(1)
Sy = (I,, i,),
(2)
where ((() represents a differentiable renderer, and 4, is a function that measures image distance.
Performing geometry-aware discrimination on fake images. As discussed above, the 3D-aware generator built upon G-NeRF [2. 3, 5, 9, 23-25, 29, 30, 34, 36] can synthesize consistent 3D-aware images owing to the underlying geometry encoded in G-NeRF rather than the supervision signal from the discriminator. In such a case, the underlying geometry cannot be accurate despite the good quality of the synthesized 2D images. To alleviate the issue of the imperfect shape encoded in G-NeRF, we leverage the intrinsic geometry characteristics of a single image and give geometry-awant supervision to the generator. Concretely, we first ask the discriminator to factorize the geometry information
3

---

of synthesized images. And then the geometry k extracted by (--) is regarded as a pseudo label to supervise the corresponding geometry k encoded in G-NeRF in loop-back manner. The image generation process is formulated as , = Giz. E), where z and ç are the sampled latent code and camera pose. An extra training objective included perform geometry discrimination:
k = +[T)),
(3)
k (z,
(4)
geo -11 (k,k,)
(5)
where y is the fake image synthesized by the generator, and l denotes a loss function that measures the difference between the geometries. G2 represents geometry extraction from G-NeRR
Loss functions. To summarize, with the purpose of making the discriminator geometry-aware, the generator and the discriminator are jointly trained with:
min La P & (I(P(G(2, ())) Nyoo Nyoofgoo 8a
min Ln = EI,~p, [ (D(I,) + E L 0p
AR Vi, [I
X Xgeofi
where f(() - - log(1 + exp(-()) [20] is the negated logistic loss, and the third term in Eq. (7) is the gradient penalty. X, Ngea and Nooo represent weights for different loss terms. 0g and OD denote the parameters of the generator and GeoD, respectively.
3.3 Implementation details
Geometry branch in GeoD. Considering the complexity of contents in the image, we employ different architectures of geometry branch for objects and scenes. The geometry branch for objects is built upon Unsup3d [33] and trained unsupervisedly, with the assumption that objects are symmetric. In this branch, the encoder decomposes the input image I into six factors (I) (d, n, a, 1. v, c), including a depth map d, a normal map n, an albedo map a, a global lighting direction I, a viewpoint v, and confidence maps All factors except the viewpoint During the reconstruction process, an image I under the canonical view will first be reconstructed from d, n, a, and using Lambertian shading model. Then, a reprojection operator warps I. from the canonical view to viewpoint given the depth map d, which gives reconstructed image i. Considering the symmetry assumption, a second reconstructed image i' is obtained from flipped depth and albedo. To weaken the influence of asymmetric parts, confidence maps, which indicate the pixel-wise symmetry score, are used the weighting factors of pixel-wise reconstruction loss. The reconstruction loss for geometry branch of objects on real data is thus given by:
estimated under the canonical view.
Cr.obj geo
= (I, AIip (I, + (,,,ic; (I,
where 4 and 4p denote the pixel-wise 4 loss and the perceptual loss [12], respectively. Ayuip represents the weighting factor loss term. The geometry (e.g., normal) used for generator supervision is obtained by warping the geometry under the canonical view the one under viewpoint v.
Geometry branch for scenes is built on Li al.'s [18]. The network is constructed in cascade manner with two stages. Each stage contains two encoders. One encoder will output albedo normal n, roughness r and depth d, while the other one aims at predicting spatially-varyine lighting The following differentiable rendering layer then renders the diffuse image I2 and the specular image I, based on these outputs. a, n, r, d, I,, I, along with the original image will become the inputs of the encoders in the next stage. Because of the large diversity of geometries, materials, and lighting, ground-trui data for each decomposed component available for training. Therefore, the training loss for geometry branch of scenes is the sum of l2 loss between each predicted factor q and its ground-trum data q
CT.sce
Σ
Agl2(g. 9),
geo
O (E(............)
where l2 and Ay denote the wixel-wise l2 loss and the weighting factor for each loss term.

---

Geometry extraction from G-NeRE. As discussed above, the underlying geometry encoding in G-NeRF can be leveraged as an implicit constraint for synthesizing 3D-aware images. Our approach aims at supervising the underlying geometry by Geol) and the normal is chosen as the geometry representation to pass the 3D information during the discrimination process. We first obtain the depth map d regarding different camera rays by performing volume rendering [19] on the depth-axis:
NN
d(r) = Σ Ti(1 - eex--eep(ppp00 -x))))p-
(10)
1=1
T. exp(
Σ [x)
(11)
where 6, = ||x+1 - x;ll2 is the distance between adjacent sampled points, and 2 2[x;] denotes the depth value of each point x;. Then the tangent map, t, is derived from the depth map along u, directions [33]. Here, we take the u direction as an example:
U,e d K
Pu+1, K
d Pu-1,0: P.
(12)
where Pa.y = (Ir, y/, 1)2 is the homogeneous coordinate of image pixel (ar, y). and K is the camera intrinsic matrix. And then the normal map á can be obtained by performing the outer product on the tangent map ń t, t u, followed by the point-wise normalization operation.
4 Experiments
4.1 Settings
Datasets. We evaluate the proposed GeoD on three real-world unstructured datasets, including FFHQ [14], AFHQ cat [4], and LSUN bedroom [35]. FFHQ contains unique 70K high-resolution real images of human faces. All images are aligned and cropped following [14]. AFHQ cat includes around 5K images of various cat faces in different poses. We use the cat face detector to get the landmarks and align images following [14]. Images are then cropped to keep the face in the center. There are about 3M images in the LSUN bedroom. The images are captured in various camera views. We use center- center-cropping to preprocess the images.
Baselines. We compare against three state-of-the-ar methods in 3D-aware image synthesis, including T-GAN [2]. StyleNeRF [9] and VolumeGAN [34]. Baselines are either released by the authors or trained with official implementations.
Training. We inherit the generators from the baselines and build GeoD on baselines' discriminator. We follow the training protocol of the baselines. For human and cat faces, GeoD is trained from scratch along with the original GAN pipeline. As a result of the complexity of indoor scenes, is difficult Iearn reasonable geometries from monocular images only. Therefore, for GeoD of scenes, we pretrain the geometry branch on synthetic data [18] and NYU dataset [21] following Li et al. [18]. The resolution of the geometry branch in GeoD is 64 64 for faces and 256x256 for scenes. Images are resized to satisfy this requirement. The entire training ensures the discriminator to see 25000K real images. Due to the expensive rendering process, π -GA is trained with images of resolution 64x64, and VolumeGAN on LSUN bedroom is trained on 128x 128 images. Other experiments are conducted on the resolution of 256x256. More details are available in Supplementary Material.
Evaluation metrics. We use three metrics to evaluate the performance, including Fréchet Inception Distance (FID) [10], Scale-Invariant Depth Error (SIDE) [6], and Reprojection Error (RE) [34). FID is calculated on 50K real images and 50K fake images rendered with random latent codes and camera views, reflecting the quality and diversity of the generated images. SIDE evaluates the generated geometries' accuracy on 10K images by comparing the ground-truth depth and the depth map rendered from the generator. We adopt the pretrained models, LeMul [11] for faces and DPT [27] for scenes, as the depth estimators to extract the depth map from 2D images as ground truths. Since the resolution of NeRF different from the image resolution in StyleNeRF and VolumeGAN, we resize the ground- ground-truth truth depth to NeRF's 's resolution for comparison. RE evaluates the consistency of synthesized images on 10K pairs. We randomly sample two views, and render the image and its depth map under each view given the same latent code. The depth map will be resized to image size if the resolutions of two do not match. Then, we warp the image from one view to the other view using its depth, and calculate the error between the warped image and the image rendered under that view.
5

---

FFHQ
AFHQ Cat
LSUN Bedroom
TT-GAN
N/A
N/A
N/A
TT-AAN GeaD
N/A
N/A
N/A
Figure 2: Qualitative results with r-GAN [2] as the base model. From top bottom: synthesized results, normal maps derived from the generative neural radiance field, and 3D meshes.
Table I: Quantitative comparisons different GAN architectures and datasets. FID [10] used to evaluate the 2D synthesis quality. while SIDE [6] and RE [34] helps evaluate the 3D shape. Numbers with # obtained by pulling the extracted depths the mean depth plane, because the baselines tend to learn a flaten depth either too near or too far, as shown in Figs. 3 and 4.
FFHQ [14]
AFHQ Cat [4]
LSUN Bedroom [35]
FID
SIDE|
REJ
FID4
SIDE
REJ
FID
SIDE
REJ
T-GAN [2] T-AAN + GeoD
8.698
0.133
0.023
9.320
0.062
0.019
15.712
0.355
0.050
6.876
0.066
0.011
9.353
0.045
0.019
8.535
0.153
0.025
StyleNeRF [9]
8.513
0.165
0.085
5.117
0.130
0.111
8.327
0.155
0.240*
StyleNeRF+ + GeoD
7.860
0.044
0.074
4.547
0.045
0.104
9.148
0.094
0.227
VolumeGAN [34]
9.598
0.084
0. 168
5.136
0.053
0.128
18.107
0.451
0.098*
VolumeGAN + GeoD
8.391
0.039
0.061
6.475
0.038
0.043
18.796
0.113
0.050
4.2
Main results
2D image quality and 3D shape. Tab. 1 reports the FID scores evaluated on different models and datasets. With our GeoD, the FID scores are better than or on par with the baselines' in most cases, indicating better image quality and diversity. This is also reflected in the qualitative results in Figs. 2 to 4. For StyleNeRF on bedrooms and VolumeGAN on cats and bedrooms, the FID scores are slightly worse than the baselines'. We conjecture the reason is that these basclines tend to learn flat shape as shown in Figs. 3 and 4, and thus they reduce the 3D-aware image generation problem to a 2D image generation task, leading to lower FID scores. Our method, however, keeps the image quality comparable with or even better than the baselines while producing correct underlying geometries.
Analysis on geometries. We analyze the underlying geometry of the generated images from two perspectives, i.e., the accuracy of the geometry and the consistency across different views, which are reflected by SIDE and RE in Tab. 1. Here, for StyleNeRF [9], SIDE and RE are evaluated only on the foreground objects of FFHQ and AFHQ cat. Lower SIDE and RE values on all datasets and architectures indicate the achievement of better geometries with GeoD. This is also reflected

---

FFHQ
AFHQ Cat
LSUN Bedroom
StyleNeRF
N/A
N/A
N/A
StyleNeRF GeoD
N/A
N/A
N/A
Figure 3: Qualitative results with StyleNeRE [9] the base model. From top to bottom:
synthesized results, normal maps derived from the generative neural radiance field, and 3D meshes.
in the qualitative comparisons in Figs. to 4. For each sample, we uniformly sample three views and show the normal map and the mesh each view. и-GAN generates noisy shapes three datasets. With our GcoD, the pits are all gone and the generated shapes are more vivid, resulting in better accuracy and consistency scores. StyleNeRR and VolumeGAN apply NeRF in lower resolution than the image resolution, and tend to generate flat shapes as shown in Figs. 3 and 4. Especially on AFHQ cat and LSUN bedroom datasets, their generated geometries are nearly or completely flat planes. By using GeoD as the discriminator, the resulting geometry is much more dinensinal...... VolumeGAN combines the volume representation and neural radiance fields for rendering, and therefore the generated shape is finer than that of StyleNeRF. Fig, 5 visualizes the geometries extracted from the 3D-aware generator and GeoD. GeoD successfully extracts vivid geometries and thus is able to guide the generator to produce fine geometries.
4.3 Ablation study
We conduct ablation studies to analyze the effectiveness of GeoD under three settings: trained from scratch, pretrained, and fine- tuned. T-GAN is chosen as the backbone and all experiments are done on FFHQ 64x64. In the first setting, GeoD is trained together with the generator from scratch. In the second setting, we first pretrain the geometry branch of GeoD on FFHQ dataset until converges. Then, the geometry branch of GeoD will be freezed during the entire training of GAN. In the third setting, the geometry branch of GeoD will first be trained offline on FFHQ dataset for half of the iterations that it converges. After that, the weights will be used initialize the geometry branch of GeoD in the training of the whole framework. Results are presented in Tab. 2. Pretrained geometry branch GeoD provides better supervision geometry at the beginning, hence the accuracy of the geometry is better. However, too strong geometry supervision from the start may dominate the supervision signal and thus leads to a sub-optimal solution for RGB synthesis. Therefore, the setting that GeoD is trained from scratch achieves better image quality and consistency.
Table 2: Ablation study conducted FFHQ [14] under 64x64 64 resolution using 7-GAN [2].
FID
SIDEJ
RF.
Trained from scratch
6.876
0.066
0.011
Pretrained
7.221
0.057
0.015
Fine-tuned
7.136
0.058
0.014
7

---

FFHQ
AFHQ Cat
LSUN Bedroom
VolumeGAN
59
N/A
N/A
N/A
VolumeGAA+ GeoD
N/A
N/A
N/A
Figure 4: Qualitative results with VolumeGAN [34] as the base model. From top bottom: synthesized results, normal maps derived from the generative neural radiance field, and 3D meshes.
4.4 Extension of GeoD on consistency improvement
Besides the underlying geometry, multi-view consistency is also important for 3D-aware image synthesis. this section, we show that GeoD is general framework such that it is also applicable to improve the multi- view consistency with a third novel view synthesis task. Spe- cially, we ask the generator to synthesize N multi-view images (4))--1 where (4)--22 are treated as the source images. These source images then used to reconstruct an image, IN=, which shares the same view as IN-1- The difference between [== and TN reflects the degree of consistency and in return guides the generator to be more consistent across different views. We instantiate the consistency branch with IBRNet [31], a 3D reconstruction model pretrained on multiple scene reconstruction datasets. In the experiment, we set N to be 6 and work on LSUN bedroom dataset. We choose StyleNeRF and VolumeGAN baselines, because they face more severe inconsistency issues as convolutional layers and upsampling operators are introduced to compensate the low-resolution NeRF inside for high-resolution synthesis. Tab. reports the evaluation on models with or without the consistency branch. With consistency branch, FID scores are on par with baselines' or experience a slightly degradation. While the geometries become better given the supervision from the consistency branch, which is indicated by the SIDE metric. The consistency of the generated images also improved. The results demonstrate the necessity of introducing consistency branch in the discriminator to enhance the multi-view consistency of the generator.
Table Extension of GeoD on improving multi-view consistency. For this purpose, we equip the discriminator with a third branch, learned with the task of novel view synthesis. Such design improves the multi-view consistency (RE) without sacrificing 2D image quality (FID) and 3D shape (SIDE).
FID
SIDEL
REP
StyleNeRF StyleNeRF + GeoD con
8.327
0.155
0.240
8,800
0.153
0.217
StyleNeRF + GeoD geo
9.148
0.094
0.227
StyleNeRF GeoD
9.222
0.090
0.204
geo-Foon
VolumeGAN
18.107
0.451
0.098
VolumeGAN GeoD con
18.140
0.382
0.093
VolumeGAN GeoD VolumeGAN + GeoD geo-t-con
18.796
0.113
0.050
18.861
0.112
0.042
8

---

Image
Generator
GeoD
Figure Qualitative analysis on the geometries learned by GeoD. From top to bottom: synthesized images, normal maps extracted from the generator, and normal maps predicted by the geometry branch of our GeoD. We can tell that our geometry-aware discriminator indeed supervises the generator with adequate 3D knowledge, resulting in more accurate 3D shapes.
Reconstruction
Novel View
Novel View
w/ GeoD
Real Image
GeoD
Figure 6: 3D reconstruction and novel view synthesis on real data with GAN inversion. Thanks to our geometry-awar discriminator, the generator in GeoD is capable of learning more accurate 3D shape and hence reversing achieves far better shape reconstruction. Meanwhile, on the task of novel view synthesis, which largely relies on the underlying shape, our GeoD presents results with higher fidelity as well higher similarity to the source image (e.g., novel view 1).
4.5 Application of GeoD in GAN inversion
One of the potential applications of GeoD is to help better geometry reconstruction from real image. To extract the underlying shape from real image, GAN inversion is performed on the 3D-aware generator following [2]. As shown in Fig. 6, the generator of 7-GAN trained with 2D discriminator reconstructs geometries with bumpy surfaces, resulting in inconsistent novel view synthesis. With our geometry-awar discriminator, the reconstructed shape becomes smooth and realistic, and the images synthesized under different viewpoints more consistent with each other.
5 Conclusion
In this paper, we propose new geometry-aware discriminator, GeoD, to improve 3D-aware image synthesis. GeoD not only provides supervision on 2D images but also guides the learning of the underlying geometry. With such discriminator, 3D-aware generators produce high-fidelity images with better underlying geometry. Besides, we further extend our GeoD to improve the mult-view consistency of the generator. Experimental results demonstrate the effectiveness of our proposed discriminator from both the image quality and the geometry perspectives. We hope our work can bring more attention to the research effective discriminators.
Discussion. Despite the high quality achieved in both the 2D image and the geometry, the quality of the generated geometries is influenced by the performance of the geometry branch in GeoD. For example, geometry branch for scenes requires labeled data for training. However, these labels are not available in the current dataset for synthesis. Consequently, training the geometry branch on the same dataset as the generator impossible, and one pretrained on other datasets adopted. The domain gap between the two datasets degrades the quality of supervision signals on geometries, and thus leads sub-optimal solution for the generator. A more efficient geometry branch or novel fine-tuning method can be proposed to alleviate the problem.

---

Acknowledgments and Disclosure of Funding
This work has been made possible by Research Impact Fund project (R6003-21) and Innovation and Technology Fund project (ITS/004/2IFP) funded by the Hong Kong Government.
References
[1] A. Brock, Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Int. Conf. Learn. Represent. 2019.
[2] E. Chan, M. Montciro, P. Kellnhofer, J. Wu, and G. Wetzstein, pi-GAN: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In JEEE Conf. Comput. Vis. Puttern Recog,, 2021.
[3] E. R. Chan, C. Z. Lin, M. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. Guibas, J. Tremblay, S. Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog,, 2022.
[4] Y. Choi, Y. Uh, J. Yoo, J. J. Ha. Stargan v2: Diverse image synthesis for multiple domains. In JEEE Conf. Comput. Vis. Pattern Recog., 2020.
[5] Y. Deng, J. Yang, J. Xiang, and X. Tong. Gram: Generative radiance manifolds for 3d-aware image generation. In IEEE Conf. Compur. Vis. Pattern Recog., 2022.
[6] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from single image using a multi-scale deep network. In Adv. Neural Inform. Process. Syst., 2014.
[7] C. Godard, O. Mac Aodha, and G. J. Brostow. Unsupervised monocular depth estimation with left-right consistency. In IEEE Conf. Comput. Vis. Patlern Recug., 2017.
[8] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Adv. Neural Inform. Process. Syst., 2014.
[9] J. Gu, L. Liu, P. Wang, and C. Thcobalt. Stylenerf: A style-based 3d-aware generator for high-resolution image synthesis. In Int. Conf. Learn. Represent. 2022.
[10] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by two time-scale update rule converge to local nash equilibrium. Adv. Neural Inform. Process. Syst., 2017.
[11] L.-N. Ho, A. T. Tran, Q. Phung, and M. Hoai. Toward realistic single-view 3d object reconstruction with unsupervised learning from multiple images. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[12] J. Johnson, A. Alahi, and I. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Eur. Conf. Comput. Vis., 2016.
[13] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Int. Conf. Learn. Represent., 2018
[14] T. Karras, S. Laine, and T. Aila. A bayee generator architecture for generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[15] T. Karras, S. Laine, M. Aittala, Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. In IEEE Conf. Comput. Vis. Puttern Recog., 2020.
[16] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias-free generative adversarial networks. Adv. Neural Inform. Process. Syst., 2021.
[17] 7. Li, 7. Xu, R. Ramamoorth,, K. Sunkavalli, and M. Chandraker. Learning to reconstruct shape and spatially-varying reflectance from single image. ACM Trans. Graph., 2018.
[18] Z. Li, M. Shafiei, R. Ramamoorthi, K. Sunkavalli, and M. Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
[19] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoonthi, and R. Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In Eur. Conf. Comput. Vis., 2020.
[20] V. Nagarajan and Z. Kolter. Gradient descent gan optimization is locally stable. In Adv. Neural Inform. Process. Syst., 2017.
10

---

[21] P. K. Nathan Silberman, Derek Hoiem and R. Fergus. Indoor segmentation and support inference from rgbd images. In Eur. Conf. Comput. Vis., 2012.
[22] T. Nguyen- Phuoc, C. Li, L. Theis, C. Richardt, and Y.-L. Yang. HoloGAN: Unsupervised learning of 3d representations from natural images. In Int. Conf. Comput. Vis., 2019.
[23] M. Niemeyer and A. Geiger. GIRAFFE; Representing scenes compositional generative neural feature fields. In IEEE Conf. Compur. Vis. Pattern Recog., 2021.
[24] R. Or-EI. X. Luo, M. Shan, E. Shechtman, J. Park, and I. Kemelmacher- Shiiaccmer--Siizzma.. Stylesdf: High- resolution 3d-consistent image and geometry generation. In IEEE Conf. Comput. Vis. Pottern Recog., 2022.
[25] X. Pan, X. Xu, C. C. Loy, C. Theobalt, and B. Dai. A sonding-guided generative implicit model for shape-aceurate 3d-aware image synthesis. In Ady. Neural Inform. Process. Syst. 2021.
[26] R. Ranfil, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Trans. Pattern Anal. Mach. Intell., 2020.
[27] R. Ranftl, A. Bochkovskiy, and V. Koltun. Vision transformers for dense prediction. ArXiv preprint, 2021.
[28] A. Sauer, K. Schwarz, and A. Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In SIGGRAPH, 2022.
[29] K. Schwarz, Y. Liao, M. Niemeyor, and A. Geiger, GRAF: Generative radiance fields for 3d-aware image synthesis. In Adv. Neural Inform. Process. Syst., 2020.
[30] A. Tewari, X. Pan, O. Fried, M. Agrawala, C. Theobalt, et al. Disentangled3d: Learning a 3d generative model with disentangled geometry and appearance from monocular images. In IEEE Conf.' Comput. Vis. Pattern Recog,, 2022.
[31] Q. Wang. Z. Wang, K. Genova, P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser. Ibrnet: Learning multi-view baaee rendering. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[32] Z. Wang, J. Philion, S. Fidler, and J. Kautz. Learning indoor inverse rendering with 3d spatially-varying lighting. In JEEE Conf. Comput. Vis. Pattern Recog., 2021.
[33] S. Wu, C. Rupprecht, and A. Vedaldi. Unsupervised learning of probably symmetric deformable 3d objects from images in the wild, In IEEF Conf. Comput. Vis. Pattern Recog., 2020.
[34] Y. Xu, S. Peng, C. Yang. Y. Shen, and B. Zhou. 3d-aware image synthesis via learning structural and textural representations. In JEEE Conf. Comput. Vis. Pattern Recog., 2022.
[35] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. LSUN: Construction of a large-scale image dataset using deep lcarning with humans in the loop. arXiv preprint 500.0:650 2015.
[36] P. Zhou, I.. Xie, B. Ni, and Q. Tian. Cips-3d: A 3d-aware generator of gans hased on conditionally- independent pixel synthesis. arXiv preprint arXiv:21 10.09788, 2021.
[37] J.-Y. Zhu, Z. Zhang, C. Zhang, J. Wu, A. Torralba, J. B. Tenenbaum, and W. T. Freeman. Visual object networks: image generation with disentangled 3D representations. In Adv. Neural Inform. Process. Syst., 2018
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 5.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Available in the supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...

---

(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof's of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or URLY? [No] Code will be made publicly available later. Data and implementation details are available for reproduction.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Available in Sec. 4.1 and supplementary material.
(c) Did you report eтror bars (e.g., with respect the random seed after running experiments multiple times)? [No] We fix the random seed for all experiments.
(d) Did you include the total amount of compute and the type of resources used (c.g. type of GPUs, internal cluster, or cloud provider)? [Yes] Available in the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "n7lekSOxBtw", "reviewer": "Reviewer_PMt3", "summary": "The current discriminators of 3D-aware GANs only discriminate between real and fake. This paper proposes learning a geometry-aware discriminator to improve 3D-aware GANs. In particular, the discriminator is additionally asked to derive the normal from the inputs, which is then applied as supervision for the generator. The author also introduces a consistency branch for the discriminator and claims such a design could improve multi-view consistency. Experimental results show the effectiveness of the proposed discriminator in both image and geometry quality.", "questions": "Please see the weaknesses. ", "limitations": "Yes, they did.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe problem studied in the paper is important and the method is a possible direction for improving the view consistency of 3D GANs.\n\nThe proposed method shows an improvement in geometry quality. \n\n\nWeaknesses:\n\n1. The discriminator learns to regress 3D attributes with an autoencoding training objective for real images. The discriminator then estimates the normals of fake images, which are used as pseudo labels to supervise the generator. Would it be better to use a separate network as the 3D attributes regressor? \n\n2. In equation 12, the author should provide enough details to understand the equation. What do all the terms mean? For example, explain the meaning of d_{u+1} and d_{u-1}, and detail the elements of the intrinsic matrix K. \n\n3. The authors did not verify the effectiveness of GeoD on high-resolution experiments (e.g., at 1024x1024 resolution). The qualitative results are not very impressive. As shown in the demo video, the generated images are blurry. The ablation study was limited to 64x64 resolution images, and the results were not very convincing. The same goes for the GAN inversion experiment. I recommend the author demonstrate the effectiveness of GeoD on high-resolution images. \n\n4. The paper structure needs to be adjusted. For example, Section 4.4 proposes a consistency branch, which should be moved to the methods section. \n\n5. In Figures 2 and 3, why is the bedroom mesh not shown? \n\n6. The approach seems to be ad-hoc. For example, the author employs different architectures for the geometry branch for objects and scenes, limiting the scalability of the method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WcOBvNsoBtK", "reviewer": "Reviewer_mehJ", "summary": "This work addresses the task of 3D-aware image synthesis. In particular, the authors present a general formulation called GeoD that can be used to improve the performance of various existing 3D-aware GAN variants wrt to the plausibility of the generated images and the reconstruction quality. Specifically, the authors propose to use the output of the discriminator to supervise the extracted geometry of the generator. They evaluate their framework on various GAN variants such as StyleNeRF and VolumeGAN on 3 datasets: FFHQ, AFHQ cat and LSUN bedrooms and show that their method outperforms existing methods wrt to the FID score, the Scale-Invariant Depth Error (SIDE) and the Reprojection Error (RE).\n", "questions": "1. Information regarding the parametrization of the normal map $\\mathbf{n}$, the albedo map $\\mathbf{a}$ and the global lighting direction $\\mathbf{l}$ are not provided  neither in the main paper nor in the supplementary. This needs to be clarified to ensure that reproducibility. Similarly, how is the confidence maps $\\bc$ parametrized?\n\n2. In L151-152, the authors state \"One encoder will output albedo $\\mathbf{a}$, normal $\\mathbf{n}$, roughness $\\mathbf{r}$ ...\". What is roughness and how is it used? This is not explained in the text.\n\n3. I find it a bit weird that for the case of Figure 2 while the normal maps look noisy but have captured the 3D geometry to some extent the authors did not show the 3D meshes. How do the 3D meshes look like? \n\n4. To generate the results of Figure 5, what generator did the authors use?\n\n5. I think that a pictorial representation of the geometry branch in GeoD both for the case of objects and scenes would significantly facilitate understanding the various details presented in section 3.3.\n\n6. Instead of using the proposed geometry parametrization, I think that using an off-the-shelf depth estimator to extract depth maps. My intuition is that this variant would be easier to train and would lead to better results. Have the authors considered this setup?\n", "limitations": "I think that the limitations of this work are adequately discussed in the main paper.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n------------\n\n1. I think that the idea of having a geometry-aware discriminator that is used to supervise the predicted geometry of the generator is very interesting. Moreover, the authors demonstrate that their model can be used in combination with various existing approaches such as StyleNeRF and $pi$-GAN and significantly boost their performance. This clearly shows the contribution of this work.\n\n2. Based on the provided results it seems that the proposed method outperforms methods with non geometry aware discriminators wrt FID, SIDE and RE scores.\n\n# Weaknesses:\n-------------\n\n1. I am wondering why didn't the authors consider EG3D (Efficient Geometry-aware 3D Generative Adversarial Networks by Chen at al.) as a baseline? This method has demonstrated superior results in comparison to $\\pi$-GAN for example. Hence, I think EG3D would have been a stronger baseline. In particular for the case of EG3D I am wondering whether replacing their StyleGANV2 discriminator with GeoD would still lead to improved results.\n\n2. A more powerful baseline in terms of reconstruction fidelity that the authors could consider in  their evaluation is StyleSDF. I am wondering whether any of the existing 3D-aware GAN variants with the proposed discriminator would still be better than StyleSDF.\n\n3. I think that another important aspect of the model that is not properly discussed is its efficiency. Without a doubt using any of the existing 3D-aware GAN models where only the generator is 3D-aware would probably require less time to train and probably would be able to also generate scenes in various resolutions faster, since they do not have to account for the geometry-aware discriminator. However, I am wondering how much slower is the proposed method in terms of training epochs and how much time is needed for rendering various resolution size images? I believe this analysis is valuable since it will allow us to see what are the limitations of the proposed method.\n\n4. Another experiment that the authors could consider including is showing synthesized images, normal maps and 3D meshes from camera poses that significantly differ from the training poses. It would be interesting to see whether this formulation yields smoother reconstructions even under these more challenging scenarios.\n\n5. Since the output of the discriminator is used as pseudo-labels to supervise the geometry produced by the generator, I am wondering how prone is the proposed model to local minima. For example, during the first stages of training when the pseudo-labels are quite noisy, is it possible for the model to get stuck? From Table 2, it seems that the \"Trained from scratch\" variant is the performing the best in terms of image plausibility (FID) and reprojection error (RE). However, this result is a bit unexpected for me, since I would have expected the Pretrained variant to be the best performing one. Any idea why this is the case?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5cy1huCfBM", "reviewer": "Reviewer_QrG6", "summary": "This paper proposes to let the discriminator estimate the geometry of the scene from the 2D rendered version of the generated 3D scenes to improve the geometry of the generated scenes.\n- The discriminator estimates six geometry factors from a 2D rendered scene.\n- The six geometry factors = a depth map, a normal map, an albedo map, a global lighting direction, a viewpoint, and confidence maps.\n- The geometry factors supervise the generator to match them.\n- The GT depth map and its normal map for the discriminator are given by the volume rendering of depths.\n- The proposed method improves various 3D-aware GANs including pi-GAN, StyleNeRF and VolumeGAN.\n  - 3D shapes are more accurate in the qualitative results.\n- Experiments are conducted on FFHQ, AFHQ cat, LSUN bedroom.\n", "questions": "Please refer to the above section.\n", "limitations": "Yes. Necessity of pretrained networks for extra supervision and its domain gap.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Letting the discriminator learn the geometry is novel.\n- Some sources of supervision are self-supervision.\n\n### Weaknesses\n- Confusions\n  - L137-139 The D estimates six factors: depth, normal, albedo, lighting, viewpoint, and confidence.\n  - L93 The G produces color and density.\n  - L163-170 describes the procedure for producing depth and normal from the density of a generated scene.\n  - Eq(6) The G tries to match the geometries estimated by the D.\n  - **How do the other four geometries guide the generator?**\n  - L161 The normal of the generated scene supervises the D.\n  - L121-127 The D tries to match the geometries extracted by the G.\n  - **How do the other four geometries from the D are supervised? The problem is under-constrained by the below objectives.**\n  - L140 The geometries reconstruct the image.\n  - L143 The symmetry assumption\n\n- Writing\n  - Redundant descriptions\n    - D estimates six factors: L95, L137, L151\n  - L150-159 is difficult to understand.\n  - The description of the proposed method can be largely improved. Current structure is very confusing because the description for G/D architecture and G/D objectives are entangled.\n\n- Evaluation\n  - Reprojection error is not sound because there are corner cases where the error can be small with wrong shapes: planar shape.\n  - I am not sure  whether the same ablation study and GAN inversion on other backbones will lead to the same conclusion. Inferiority of pi-GAN compared to other 3D-aware GANs is well-known in the research community.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4nlDilxdmC8", "reviewer": "Reviewer_7Jaj", "summary": "This paper presents an improved discriminator architecture for 3D-aware image synthesis. The authors advocate to enforce consistency between the geometric features of images (depth, surface normals etc) rendered from the generative NeRF as well as predicted by off-the-shelf methods (Unsup3D). Such context can make discrimination between real and fake images easier for the network. Experiments show that the proposed geometry-aware discriminator as an additional building block helps improve several baseline methods on human/cat faces and LSUN bedroom datasets.\n", "questions": "It would be great if the authors could address all the concerns listed above.", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n- I like the motivation of making the discriminator geometry-aware, which is what prevalent 2D image discriminators are largely lacking. The idea of utilizing off-the-shelf methods is also simple, and elegantly bridges with the framework of generative neural radiance fields.\n- The results clearly show that add geometry awareness improves the 3D geometry interpretability of image generation. Although not strictly improving on the FID score for every metric on every backbone/dataset, overall the results show significant improvements both qualitatively and quantitatively. \n- The paper is nicely written and the presentation is very clear.\n\nWeaknesses:\n- It would be good if the authors could provide some failure case analysis to better understand the limitation of the proposed method.\n- I think Section 4.4 seem somewhat extraneous to the paper, as it's not central to the paper, and (in addition to [18] and [33]) introduces yet another building block IBRNet [31] to the system. The authors should also elaborate why such novel view synthesis component is necessary to improve the synthesis quality, as the original proposed model is already a 3D-aware image synthesis method based on NeRF.\n- Why is GRAF [29] called G-NeRF throughout the entire paper? I believe G-NeRF is more commonly referred to another paper [A]. If the authors meant a GAN + NeRF framework in general, I would recommend not inventing an abbreviation to avoid confusion.\n- The loss function notation in Eq 2 and 5 seems to be inconsistent.\n\n[A] Meng et al. \"GNeRF: GAN-based Neural Radiance Field without Posed Camera.\" ICCV 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept