Benchmarking and Analyzing 3D-aware Image Synthesis with a Modularized Codebase
Authors: Qiuyu Wang
Zifan Shi
Kecheng Zheng
Yinghao Xu
Sida Peng
Yujun Shen
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: 3D-aware image synthesis

Abstract: Despite the rapid advance of 3D-aware image synthesis, existing studies usually adopt a mixture of techniques and tricks, leaving it unclear how each part contributes to the final performance in terms of generality. Following the most popular and effective paradigm in this field, which incorporates a neural radiance field (NeRF) into the generator of a generative adversarial network (GAN), we build
a well-structured codebase through modularizing the generation process. Such a design allows researchers to develop and replace each module independently, and hence offers an opportunity to fairly compare various approaches and recognize their contributions from the module perspective. The reproduction of a range of cutting-edge algorithms demonstrates the availability of our modularized codebase. We also perform a variety of in-depth analyses, such as the comparison across different types of point feature, the necessity of the tailing upsampler in the generator, the reliance on the camera pose prior, etc., which deepen our understanding of existing methods and point out some further directions of the research work. Code and models will be made publicly available to facilitate the development and evaluation of this field.

Benchmarking and Analyzing 3D-aware Image Synthesis with a Modularized Codebase
Qiuyu Wang'
Zifan Shi
Kecheng Zheng'
Yinghao Xu
Sida Peng* *ZJU
Yujun Shen
Ant Group
HKUST
CUHK
Abstract
Despite the rapid advance of 3D-aware image synthesis, existing studies usually adopt mixture of techniques and tricks, leaving unclear how each part contributes to the final performance in terms of generality. Following the most popular and effective paradigm in this field, which incorporates neural radiance field (NeRF) into the generator of a generative adversarial network (GAN), we build a well-structured codebase, dubbed Carver, through modularizing the generation process. Such a design allows researchers to develop and replace each module independently, and hence offers an opportunity fairly compare various approaches and recognize their contributions from the module perspective. The reproduction of a range of cutting-edge algorithms demonstrates the availability of our modularized codebase. We also perform variety of in-depth analyses, such as the comparison across different types of point feature, the necessity of the tailing upsampler in the generator, the reliance on the camera pose prior, etc., which deepen our understanding of existing methods and point out some further directions of the research work. We release code and models here to facilitate the development and evaluation of this field.
Introduction
Learning a 3D-aware generative model has received growing attention considering its practical applications, such digital avatar and virtual reality. In addition to image quality and diversity, which are widely pursued by 2D generation, 3D-aware image synthesis also requires the output images to be spatially consistent across different viewing directions. Due the lack of large-scale 3D data, previous attempts [46, 39, 20, 8, 56, 40, 58] propose to learn the 3D-aware model with 2D images as the only supervision. To accomplish such challenging task, the most popular solution is to introduce neural radiance fields (NeRFs) into generative adversarial networks (GANs) as the inductive bias. In this way, the GAN generator acquires the awareness of the underlying geometry when rendering an image, whose fidelity is promised by the competition with the discriminator [19].
Towards effective incorporation between NeRFs and GANs, many techniques have been proposed, such as SIREN activation [6] and tri-plane representation [8]. However, the popularity of this field unintentionally leads to some problems. (1) Existing algorithms
usually developed with different codebases [7, 21, 57, 9], which adopt different 3D coordinate systems and rendering pipelines, making it hard to transfer well-trained models from one codebase another. (2) State-of-the-an performance is usually achieved through an adequate combination of many techniques and tricks, where some are novel while some are inherited from prior arts. However, existing codebases typically hold an entangled implementation, making it hard to recognize the contribution of each part. (3) follow-up problem of an entangled implementation is the inconvenience of drawing merits from different approaches, causing additional burden to the advance of this field.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Table 1: Analyses of 3D-aware image synthesis performed with our modularized codebase.
Module
Analysis
Observation
MI.P Volume Tri Tri-plane
Different point features exhibit competitive eupacities.
Point Embedder
Combination of multiple types of point feature
The contribation is marginal compared to a single type of point feature.
Numher of planes
Bi planes performs on par with tri planes.
Decoder depth (f.e., number of layens)
The depth only matters for M-badd point embedder.
Feature Decoder
Activation function
SIREN is better than ReLU when upsampler module is abseat.
Volume Renderer
Density- hased v.r SDF hased
SDF based representation currently lags behind the density based one.
Upsampler
Effects on the generation quality and consistency Upsamplers benedit the quality but harm multi-viow consistency.
Pose Sampler
Eiffects of the pre defined pose priors
The more accurate the poses are, the hetter the generation quality is.
This work fills in this gap with modularized codebase for 3D-aware image synthesis. In particular, we reformulate the generation process into a bunch of modules, as shown in Fig. 1, including pose sampler, stochasticity mapper, point sampler, point embedder, feature decoder, volume renderer [36], and an upsampler. Besides, we also integrate a visualizer and an evaluator into our codebase to facilitate online evaluation. With such a design, we are able share the 3D coordinate system and the rendering pipeline for all methods, and hence leave the research effort to the improvement of each individual module. In summary. our contributions are three-fold.
We build hidhly-modularized easy-to-use codebase for 3D-aware image synthesis, and also use it to re-implement a range of classic algorithms in this field. The on-par or even better reproduction results suggest that existing approaches can be easily reformulated into a module combination following our pipeline, demonstrating the availability of our toolkit.
Our codebase allows users to replace a particular module (e.g., from the function perspective) arbitrarily and independently, facilitating the per-module evaluation as well as the design integration from various methods. We believe our codebase could help the community with a more convenient algorithm development.
Thanks to the modularity, we perform a variety of in-depth analyses regarding different modules, which is beyond the capacity of previous functionally entangled codebases. The studies and observations are summarized in Tab. 1, Besides deepening our understanding of this field, these analyses also help point out some further directions of the research work, as discussed in Sec. 4.3.
2 Background
Task setting. 3D-aware image synthesis aims at generating multi-view images only from 2D image collections. Basically, always incorporates 3D inductive bias into 2D generative models, enabling the generation of 3D models from 2D images without any 3D data. Previous efforts always concentrate on 3D shape generation [53, 5, 29, 54], which typically demands extensively annotated 3D datasets for training models. Thanks to the progress of neural implicit fields [36, 42, 35, 32, 18, 2, 44, 51] and generative models [19, 28, 3, 45, 24] especially GANs [25-27]. 3D-aware image synthesis has been advanced significantly in terms of the 3D consistency and visual quality.
Common solution. Neural Radiance Field (NeRF) [36] F(x, v) - (c, regresses color E R3 and volume density a E IR from coordinate x E R33 and viewing direction v E S?, parameterized with multi-layer perceptron (MLP) networks. Recent attempts on 3D-aware image synthesis propose to condition NeRF with a latent code z, resulting in their generative forms like GRAF [46], G(æ, v) - (c, (), to generate multi-view images of the object. Many recent attempts have been made to improve generative NeRF, including the incorporation of convolutional upsamplers [39, 20, 8, 56, 40, 58], hybrid representations(55. 8, 4, 61], other implicit fields [41, 40, 55], patch-wis, training [50], and MPI-based rendering [63, 14]. Some works have leveraged alternative 3D representations, such as meshes [17, 30J, voxel grids [47, 64, 37, 38, 16, 22], and depth [48], to achieve 3D-aware image synthesis, which is not the primary focus of our paper.
Datasets. We benchmark 3D-aware image synthesis on three datasets, including FFHQ [26], ShapeNet Cars [10] and Cats [62]. FFHQ [26] is 2D dataset with 70K unique high-quality face images of resolution 1024x 1024. A variety of accessories are also covered in the dataset, including eyeglasses, earrings, eíc. The face images are shot from frontal views and near-frontal views. ShapeNet Cars [10] is the car subset of ShapeNet [10], which consists of 8K CAD model.
Both the geometry and texture are stored for each model. We random sample camera poses that span
2

---

Pose Sampler
Stochasticity Mapper
Upsampler
Visualizer
Detemmiiistic Pose Sampling
Uncertainty Pose Sampling
Color
t A7to, 1)
Lauent Code
Geometry
Point
Feature
Volume
Point Sampler
Evaluator
Embedder
Decoder
Renderer
Uniform
Tri-paaee Volume MLP
Le Somplus
Occupaney
Inception Distance (FII)
Normal
Density
Face iDastily Consisiency (ID)
Fixed
SIREN
Feature
Reprojection Error (RE)
MPI
Color
Depth Error (DE)
SDF
Poso Error (PE)
Vieving Divecton
Figure I: Overview of our modularized pipeline for 3D-aware image synthesis, which modularizes the generation process in a universal way. Each module can be improved independently, facilitating algorithm development. Note that the discriminator omitted for simplicity.
the entire 360" camera azimuth and 180º camera elevation distributions to render CAD model into 2D images. Cats [62] contains 6,444 cat face images of resolution 256x256 The camera poses are generally on the front and rear-front of the cat. Discussion. Typically, the datasets used for 3D-aware image synthesis comprise single-object datasets with an easy-to-define camera pose prior, as opposed to compositional scene datasets. The objects in these datasets have similar zoom, scale, and geometry, making them suitable for learning 3D representation from 2D observations. Previous studies [6, 46, 20, 56, 8] commonly use these datasets for evaluation, while other datasets, such as CARLA [15], LSUN Bedroom [60], and CelebA [33], are not covered in this work.
Metrics. We include Fréchet Inception Distance, reprojection error, face identity consistency, pose error and depth error to evaluate the methods. Fréchet Inception Distance (FID) [23] is adopted to evaluate the quality and the diversity of the synthesized images from 3D-aware image synthesis model. Reprojection Error (RE) [56] measures the distance between two adjacent views by warping them to each other based on the rendered depth maps. Identity Consistency (ID) [8] is designed for facial identity consistency evaluation. For each generated identity, mean Areface [12] cosine similarity score is calculated between pairs of views rendered from random camera poses. Pose Error (PE) [8. 56] measures the accuracy of the rendered objects' poses. A pre-trained pose estimator (e.g., head pose estimator) is leveraged to estimate the pose from the rendered image. Pose error calculates the distance between the estimated pose and the given camera pose for rendering. Depth Error (DE) [8] is used to evaluate the quality of the underlying shape in 3D-aware image synthesis, where a pre-trained depth estimator is adopted to predict the depth for the rendered 2D image. The predicted depth is then compared with the rendered depth to indicate the shape quality.
3 Modularized pipeline for 3D-aware image synthesis
We propose a modularized framework that employs GANs to generate 3D representations from single-view images. The overall pipeline can be seen in Fig. 1. Our framework aims to achieve modularity in the design, and thus enables easy integration of different components. By providing a modularized platform that grants easy usage and flexible configurations, our framework can foster collaborative efforts toward improving the codebase over the long term. The following sections will provide details of each module.
Pose sampler. 3D-aware generative models rely on camera pose to regulate the synthesized view of an object. As a result, pose sampler is proposed to sample poses as the model input during training. The pose sampler in our framework supports two kinds of pose sampling: stochastic pose sampling and deterministic pose sampling. Stochastic pose sampling enables sampling pose from random or pre-defined pose distribution, i.e., Gaussian distribution or uniform distribution. Deterministic pose sampling allows for sampling poses for each sample with its ground-truth pose. Ground-truth poses are easily accessible for some datasets, such as ShapeNetCars [10]. Although the FFHQ [26] and Cats [62] datasets do not include ground-truie poses, we utilize readily available face pose estimator [13] to approximate these poses, treating the results as our ground-truth. By offering both types of pose sampling, our framework provides flexibility in pose selection for users.
Point sampler. Sampling points is an essential step for NeRF because requires a set of points along single ray for rendering. However, various methods use different coordinate systems to sample
3

---

points, which can make it challenging to study components from different methods. To alleviate this issue, we develop a unified point sampler that allows users to sample points a consistent coordinate system. Initially, we sample coordinates in pixel space and then transform them into camera space using camera intrinsies. We then use the sampled camera-tor to-aattooorrddd matrix (i.e., camera extrinsic) to transform the coordinates into world space. Our point sampler facilitates the combination of components from different methods by providing a simple workflow for point sampling in a unified coordinate system. This simplifies the sampling process, making it easier for users to sample points and integrate components from various methods.
Stochasticity mapper. Following the StyleGAN family [26, 271, we learn a stochasticity mapper that takes a random noise z N (0, 1) as input, and outputs an intermediate latent code w E W to modulate the styles of 3D-aware synthesis modules (e.g., feature decoder, and upsampler). The learnable latent space W can better simulate the native distribution of real data. Moreover, we can incorporate camera parameters 0 as conditions into the stochasticity mapper to enhance the 3D) consistency of synthesized images. This allows the target view influence the scene synthesis process, leading to more realistic and accurate 3D-aware synthesized images.
Point embedder. The point embedder is responsible for transforming raw point coordinates into point features. Our framework provides several options for this transformation, including extracting from multi-layer perceptron (MLP), querying from feature volume, tri- tri-plane or multiplane image (MPI) representation, or their combinations. By leveraging explicit structure information encoded volume or tri-plane representation, the point embedder can provide a more detailed description, leading to significant impact on the quality of the final rendering. Therefore, the design of the point embedder is of great importance. Our framework offers unified interface for extracting MLP, volume, tri-plane, and MPI-based point features, providing users with convenient development experience.
Feature decoder. The feature decoder is a module that converts the extracted point features from the point embedder into color, density, or SDF values. Typically, the feature decoder consists of multi-layer perceptron (MLP) that uses ReLU as an activation function. However, recent research has demonstrated that SIREN [49]. which employs periodic activation functions, exhibits greater capability in modeling fine details than ReLU-based representations. Our framework supports both ReI.U and SIREN- -ased MIP decoders, as well as some other choices, providing users with the flexibility to choose the most suitable decoder for their specific application needs.
Volume renderer. The volume renderer is critical component that transforms decoded colors, densities, or other properties into 2D images, making it easy to receive supervision from 2D training datasets. Our framework includes support for the basic integration formula in NeRF [36] and offers a range of clamping modes for color and density values, as well as additional options for volume rendering. These features provide users with the flexibility to customize their rendering process according to their specific needs, enabling them to achieve high-quality results that meet the demands of their particular application.
Upsampler. Volumetric rendering can result in a large memory footprint and slow rendering speeds when generating high-resolution images. To maintain efficiency, many recent approaches [40, 20, 56, 8] employ convolutional upsamplers to render high-resolution images. These methods first generate a low-resolution feature map and then use an upsampler to progressively add appearance information and increase the resolution of the rendering. The typical upsampler consists of upsampling layers with 1 x 1 or 3 3 convolutional layers. Our framework provides users with range of upsamplers [40, 20, 56, 8] to choose from, allowing them to customize their upsampling process to meet their specific needs.
Evaluator. The current codebase for 3D GANs lacks systematic evaluation metrics. To address this issue, our codebase includes support for various evaluation metrics for 3D generation, such as FID [23], reprojection error [56], face identity consistency [8], depth error [8], and pose error [8]. We have integrated widely used 3D face reconstruction model [13] and face recognition model [12] into our framework, making it convenient to test various metrics. The inclusion of these metrics enables quantitative evaluation and allows the community to gain a better understanding of the effects of the key components from various methods.

---

Table 2: Overview of methods supported by codebase. We provide outline of the modules in our supported methods and present the reproduced results with our codebase on FFHQ [26], with the FID [23] used the evaluation metric.
Method
Pose
Point
Feature
Volume
Upsampler
Resolution
Official
Reproduction
Sampler
Embedder
Decoder
Renderer
GRAP [46]
Stochastic
MT.P
ReLU
Density, Color
No
128 128
46.30
45.50
a-GAN [6]
Stochastic
ML.P
SIREN
Density, Color
No
128 128
29.90
27.81
11.50
StyleDDF [40]
Stochastic
MLP
SIREN
SDE Color, Feature
Yes
512x512
10.07
10.71
1024 1024
10.01
10.14
256x
8.00
8.31
StyleNeRF [20]
Stochastic
Density.
7.37
1024x 1024
8.10
8.08
VolumeGAN [56]
Stochustic
Volume
LeakyReLU
Density, Cllar,, Featurre
Yes
256
9.10
10.37
GRAM [14]
Deterministic
MPI
SIREN
Occupancy, Color
No
256x256
14.50
13.83
EpiGRAF [50]
Deterministic
Tri plane
LeakyReLUU
Demsity, Color
No
512x512
9.92
9.19
256 256
4.80
4.72
EG3D [8]
Deterministic
Tri plane
SoMplus
Densily, Color, Feature
Yes
512x512
4.70
4.63
Visualizer. In addition to quantitative metrics, our framework also supports qualitative visualization of generated images and extraction of the underlying geometry. With our developed codebase, users can easily generate multi-view images, as well as obtain 3D shapes of each generated sample.
Experiments
This section commences with concise overview of the implementation details of experiments. Following this, we review the methods that are supported by our codebase. We then present our observations and analyses of the primary modules in our framework, highlighting which components are essential in 3D GANs. Finally, based on our experimental settings, we discuss promising future directions for 3D-aware image synthesis.
Implementation details. We employ our developed codebase to conduct extensive experiments on the main modules in 3D GANs. To ensure systematic evaluation, we use the state-of-ther of-the-art 3D-aware GAN, EG3D [8], as our backbone model and substitute each module with alternative choices. For instance, to investigate the point embedder, we substitute its tri-plane with feature volume or other representations. We benchmark 3D-aware image synthesis on FFHQ [26]. Cats [62], and ShapeNet Cars [10] datasets. The models trained on the FFHQ and Cats datasets a resolution of 256, and the ShapeNet Cars dataset at a resolution of 128. We train all models on 8 NVIDIA A 100 GPUs for 25 million images, with batch size of 32. Following the approach in EG3D [8], set the generator learning rate to 0.0025 and the discriminator learning rate to 0.002. More details of our experimental settings can be found in Appendix A.
4.1 Supported methods and reproduced results
We support all highly representative models the field of 3D-aware image synthesis, as they encompass almost all mainstream point embedders, including MLP [46, 6, 40, 20, 14], volume [56]. tri-plane [50, 8] and MPI [14]. As for feature decoder activation, they include both SIREN [6, 40, 14). ReL.U [46, 20] some other activations [56, 8, 50]. Additionally, some of these methods include an upsampler [40, 20, 56, 8), while others do not [46, 6, 14, 50]. We then provide brief introduction to each of these methods, which are supported by our codebase. GRAF [46] is the first work that learns a generative model for implicit radiance fields in 3D-aware image synthesis. --GAN [6] introduces mapping network condition layers in the SIREN [49] using feature-wist linear modulation (FiLM) [43], novel architecture in 3D GANs. StyleSDF [40] is another method that incorporates signed distance functions (SDFs) into 3D generative models and achieves impressive results in terms of visual and geometric quality. StyleNeRE [20] integrates the neural radiance field (NeRF) [36] into a style-based generator to improve rendering efficiency and consistency for high-uesolution image generation. VolumeGAN [56] uses feature volume to represent the underlying geometry, enabling high-fidelity 3D-aware image synthesis. GRAM [14] adopts the multiplane image (MPI) representation to constrain point sampling and radiance field learning on 2D manifolds, facilitating
5

---

MLP- -basdd
Volume based
Tri-plane- based
FFHQ
Cats
Cars
Figure 2: Qualitative comparison across various single point embedders FFHQ [26), Cats [62] and ShapeNet Cars [10), where the MI.P-based, volume-based, and tripplane-hased point features exhibit on-par performance in generating multi-view consistent images and high-quality geometries.
fine-detail learning. EpiGRAF [50] proposes patch sampling strategy stabilize training and accelerate convergence. Finally, EG3D [8] proposes a tri-plane-basee 3D GAN framework that is efficient and expressive for high-resolution geometry-aware image synthesis.
Meanwhile, to evaluate performance our implementation, we conduct experiments on the FFHQ dataset, and the results are presented in Tab. 2. Upon comparison of our reproduction with the original implementation, we have observed a similar level of performance terms of the officially reported metrics. The marginal differences observed in the FID for all methods indicate that codebase capable of precisely replicating the official results. The convenience afforded by the shared coordinate system and the versatility of each module has allowed to retrain various models a unified and optimized setting, leading to generally improved results in our reproduction compared to the official implementation.
4.2 Analyses
Types of different point embedders. Based on Tab. 3 and Fig. 2, we can conclude that point features extracted by different point embedders exhibit competitive capacities when combined with the upsampler. The reason may be that the upsampler enhances the modeling ability of scene appearance and relieves the burden of encoding appearance with neural fields in 3D space. Note that tripplane-based point embedder is more computationally efficient than MLP-based and volume-based point embedders.
Combination of multiple point embedders. The results in Tab. 3 and Fig. 3 show that combining the outputs of multiple embedders has negligible impact on the final outcome. When training model using multiple embedders, the network tends to find the simplest way to obtain the desired outcome,
Table 3: Analysis of different types of point features.
Point Embedder
FFIQ
[26]
Cats [62]
Cars [10]
MLP
Volume
Tri-plane
FIDJ
ID1
DE.
REJ
FID
FID
X
5.15
0.777
0.470
5.0e
0.091
4.05
2.42
5.1e-1
2.25
X
/
4,72
0.743
0.547
4.5e
0.111
3.99
2.75
5.1e
X
v
4.69
0.748
0.465
3.3e
0.104
4.42
2.59
4.6c
2.78
v
4.62
0.769
0.467
4.7e
0.091
4.70
2.65

---

MLP Volume
Tri-plane
MLP
Volume Tri-plane
MLP Volume
Tri- Tri-plane
FFHQ
Cats
Cars
Figure 3: Qualitative comparison across various composite point embedders FFHQ [26], Cats [62]
and ShapeNet Cars [10]. where these compound point features exhibit on-par performance in generating multi-view consistent images and high-quality geometries.
which may result in the output features of certain embedders being overlooked. Therefore, using single type of point feature is sufficient for model training.
Number of planes. We also inves- tigate the plane-based point embed- ders by examining the necessity of using three planes, as [8]. To accomplish this, we substitute the tri- plane with both single-plane and bi- plane representations. As expected, the singlepplane representation per- forms poorly, as the model cannot
Table 4: Analysis of plane-based point features.
FFHQ [26]
Cats [62]
Cars [10]
Point Embedder
FID.
IDT
DEL
PET
REI
FID.
HDJ
Bi Blaplane (XY X7)
4.55
0.738
0.398
4.6
0.083
4,33
3:21
Bi-plane (XY ZY)
4.38
0.746
0.379
4.4e
0.104
3.95
Bi plane (XZ++ ZY)
4.52
0.754
0.385
5.5c.
0.095
4.77
2.50
Tri plame
4.72
0.743
0.547
4.5c 0.111
3.99
2.75
accurately determine a point's 's location in 3D space with only one plane. However, the bi-plane representation slightly outperforms the tri-plane representation, shown in Tab. 4. The bi-plane model accurately determines 3D point positions without ambiguities and has fewer parameters. Thus the bi-plane representations this does not necessarily imply that fewer planes are always superior. The datasets we trained on are object-level and relatively simple, making the bi-plane representation sufficient. In more complex tasks, additional planes may be required.
par with the tri-plane representations in 3D GANs. Nonetheless,
Depth of feature decoder. As illustrated in Tab. 5, the depth of the feature decoder plays crucial role the accuracy of Mased point features. However, for volume and plane-based point features, the depth of the feature decoder appears to be less significant. This be attributed to the fact that,
7

---

in the case of volume or tri-plane- plane-based point features, the feature volume or feature plane predominantly determines the point representation's 's capability, while the feature decoder serves as simple mapper to convert the point features into corresponding density color values. Conversely, for MLP-based point features, the model relies solely the feature decoder to transform the raw coordinates to the density color values, and thus the depth of feature decoder determines the model's capacity.
Table 5: Analysis of the depth of feature decoder.
FFTIQ [26]
Point Embedder
Depth
FID
ID1
DEL
PEL
REJ
17.22
0.761
0.807
0.105
MLP
7.39
0.782
0.552
7,3e
0.087
5.15
0.470
5.65
0.784
0,437
4,4g
0.095
Volume
5.18
0.787
0.381
4.0e
0.100
16
4.65
0.778
0,413
5.1e
0.085
4.72
0.743
0,547
4.5e
0.111
Tri Tri-plane
4.77
0.750
0.414
4,4e
0.101
5.58
0.750
0.566
5.6e
0.108
Activation type of feature decoder. As il- lustrated in Tab. 6, we find that without the upsampler. SIREN-based MLP outperforms the ordinary MLP while with the upsampler the result turns out to be the opposite. One plausible explanation for this observation could be the design of the upsampler module, which may have the tendency to amplify the "ripple" artifacts induced by the SIREN-based layers [6], while mitigating the blurry artifacts produced by the ordinary layers.
Table 6: Analysis of the activation type used in feature decoder.
FFHQ [26]
Activation Type
FIDJ
ID1
DEL
PEJ
REC
w/ upsampler
256
256
SIREN
11.66
0.763
9.1e
0.089
ReI.U
7.39
0.782
0,552
7.3c
0.087
upsampler
SIREN
6.58
0.741
0.340
6.6r
0.071
ReLU
7.30
0.729
0.498
4.6e
0.084
Geometric representation. The results pre- sented in Tab. indicate that the SDF-based representations generally yields inferior quanti- tative metrics compared to vanilla density- bbsee representations for all point embedders. These findings are consistent with previous works [59]. Moreover, we observe that utilizing sphere initialization and additional regularization terms, such as eikonal loss and minimal surface loss [40], can adversely affect GAN training, leading flawed results. Thus, incorporating SDF does not offer superior generation quality compared to density-based representations.
Table 7: Analysis of the underlying geometric representation.
Geometric Representation
FFIIQ [26]
FID
ID1
DEL
PEL
REJ
0.610
0.874
5.9c -1
MLP
Density
5.15
0.777
0.470
5.0e
0.091
SDF
7.27
0.676
0.938
5.0e
0.200
Volume
Density
4.65
0.778
0.413
5.1e
0,085
SDF
13.31
0.534
0.626
10.9e
0.161
Tri-plane
Density
4.72
0.743
0.547
4.5e
0.111
Pose priors. As shown in Tab. 8 and Fig. 4, the pose priors significantly impacts the quality of the generated 3D geometry. Specifically, when
Table 8: Analysis of using different pose priors.
FFIQQ
[26]
Pose Prior
FIDJ
DEJ
PEL
REL
training with the random pose distribution (RPD), the generated 3D geometry lacks normal facial structures and exhibits inconsistency in novel view synthesis. On the other hand, when trained solely with the accurate pose distribution (APD), the model tends overfit and consequently struggles with generating dif- ferent views of faces. However, when provided with ground-trutt pose (GTP) information, the model can produce photo-realistic images and generate adequate and high-quality geometry.
MLP w/ RPD
14.56
0.413
1.513
5.8c
2
0.405
MLP APD
0.788
1.659
5.9e
0.407
MLP w/ GTP
5.15
0.777
0.470
5.0e
0.091
10.47
5.5e -2
0.390
Volume w/ APD
7.34
0.731
1.125
4.8e
0.367
"flat face shape"
Volume
0.778
5.1e
0.085
Tri-plane w/ RPD
15.18
0.427
2.181
5.6c
0.379
Tri-planc w/ APD
5.45
0.764
1.502
5.4e
0.405
Tri-planc w/ GTP
4.72
0,743
0.547
4,5e
0.111
Upsampler. As evidenced in Tab. 9, the model incorporating an upsam- pler exhibits superior image quality, owing to its capacity to enhance image details. Nonetheless, the inclusion of an upsampler, typically accomplished through convolutional
Table 9: Analysis of the effect of upsampler, with tri-plane point feature as an example.
Upsampler
Resolution
FFH [26]
Training Time
Inference
FID ID1
DEL
PEJ
REJ
Speed
V
256x2266
4.72 0.743
0.547 4.5e
0.111
2.7 Days
FPS
x
256x
6.86
0.749
0.443
6.2e
0.104
6.9 Days
20 FPS
8

---

MLP
Volume
Tri-plane
Random pose distribution (RPD)
pose distribution (APD)
Ground-truth pose (GTP)
Figure 4: Qualitative comparison across different pose priors, where having ground-truff pose for each training example plays a vital role in helping the model learn adequate geometry.
layers and non-linear activation, may compromise multi-view consistency. Conversely, climinating the upsampler yields heightened view consistency, albeit at the expense of significantly increased computational cost, manifested in prolonged training and inference times.
4.3 Outlooks
Based on the developed codebase, our experimental analysis has revealed numerous unresolved challenges in 3D GANs that warrant further investigation to enhance its applicability in downstream domains. Here, we propose several promising avenues for future research in the realm of 3Dawware image synthesis, with the aim of fostering continued progress and facilitating breakthroughs in this rapidly evolving field.
Training stability. Training 3D-aware image synthesis models is often prone to mode collapse, resulting in unsatisfactory image generation and inferior geometry. This may be due mismatch between the distribution of physically meaningful factors, such as camera poses and rendering parameters, and that of real images. Therefore, is crucial to investigate the training stability of 3D-aware image synthesis models.
Efficiency. Training 3D-aware image synthesis models can take 3-10 days multiple high-end GPUs, and the models can be slow during inference, which limits their practicality in downstream applications. Therefore, the need improve the training efficiency 3D generative models is increasing, and enhancing the inference efficiency is crucial for their practical deployment in real- world applications.
Pose acquisition. The role of pose in 3D GAN training is of great significance, but obtaining accurate pose information from in tn-the-wild data is non con-trivial task. One potential solution is to incorporate pose estimator into the 3D GAN model. Further research may focus on improving pose acquisition methods for 3D-aware image synthesis models.
Hybrid representations. The current representations employed in 3D GANs are primarily implicit representations. However, in some complicated scenarios, these representations may prove insufficient. For instance, when modeling a human head, the implicit representation is suitable for hair modeling, while explicit representations such as meshes may be appropriate for face modeling. To address this issue, future research should focus on exploring hybrid representations that can handle more complex data.
Conclusion
This paper proposes a modularized codebase for 3D-aware image synthesis development, enabling researchers to independently develop and replace each module. This approach provides means to compare various methods fairly and recognize their key contributions from a module perspective. Extensive experimental analyses reveal the potential future research directions for 3D-aware image synthesis models.

---

Broader Impact
The development of 3D-aware image synthesis has significant potential for impact in fields such as entertainment, gaming, and virtual reality. This technology has the ability to revolutionize the way we create and interact with virtual environments by generating realistic 3D models and images. However, with any emerging technology, ethical considerations must be taken into account to prevent potential misuse for malicious purposes such as the creation of realistic fake images or videos. Therefore, researchers and developers must consider the social and ethical implications of this technology and work collaboratively with stakeholders to develop appropriate guidelines and best practices, ensuring that is developed and used ethically and responsibly. Open and transparent discussions about the ethical implications of this technology can help maximize its potential benefits while minimizing potential risks.
Acknowledgments
We would like to express our gratitude Kai Zhu for his valuable assistance in reproducing EpiGRAF [50] and to Jiaxin Xie for her invaluable help in reproducing GRAM [14]. We extend our thanks Hao Ouyang and Yuxi Xiao for their insightful comments and discussions, particularly their suggestion on refining this manuscript. We are also thankful to the anonymous reviewers for their valuable feedback and suggestions, which greatly contributed improving the clarity of this manuscript.
References
[1] Barron, Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and Srinivasan. Mip-nerf: multiscale representation for anti-aliasing neural radiance fields. In Int. Conf. Comput. Vis., 2021.
[2] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[3] Y. Bengio, R. Ducharme, and P. Vincent. A neural prohabilistic language model. Adv. Neural Inform. Process. Syst. 2000.
[4] A. W. Bergman, P. Kellnhofer, Y. Wang, E. R. Chan, D. B. Lindell, and G. Wetzstein. Generative neural articulated radiance fields. arXiv preprint arXiv:2206.1433 /4314, 2022.
[5] A. Brock, T. Lim, J. M. Ritchic, and N. Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv:1608, 160:1640804466 2016.
[6] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein. 7-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. In IEEE Conf. Comput. Vis. Pattern Recog. 2021.
[7] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein. n-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. sttps://ghtt 2021.
com mar www.rnteirooooooooooooo.. GAN,
[8] E. Chan, Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. Mello, O. Gallo, L. J. Guibas, Tremblay, S. Khamis, et al. Efficient geometry- 3D generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[9] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, al. Efficient geometry-aware 3D generative adversarial networks. httppp/////tttt.o.oo WVlabs/ aegd, 2022.
[10] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. ShapeNet: An information-rich 3D model repository. arXiv preprint arXiv:1512.0300 2015.
[11] Deng, W. Dong, Socher, L.-J. Li, Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog., 2009.
[12] J. Deng. J. Guo, N. Xue, and S. Zafeiriou. AreFace: Additive angular margin loss for deep face recognition. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
10

---

[13] Y. Deng, J. Yang. S. Xu. D. Chen. Y. Jia, and X. Tong. Accurate 3D face reconstruction with weakly- supervised learning: From single image image set. In IEEE Conf. Comput. Vix. Pattern Recog. Worksh., 2019.
[14] Y. Deng, J. Yang. Xiang, and X. Tong. GRAM: Generative radiance manifolds for 3D-aware image generation. In IPFE Conf. Comput. Vis. Pattern Recog., 2022.
[15] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An open urban driving simulator. In Conf. on Robot Learn., 2017.
[16] Gadelha, S. Maji, and Wang. 3D shape induction from 2D views of multiple objects. In Int. Conf. 3D Vis., 2017.
[17] Gao, Shen, Wang, W. Chen, K. Yin, D. Li, O. Litany, Z. Gojeie, and Fidler. GET3D: A generative model of high quality 3D textured shapes learned from images. In Adv. Neural Inform. Process, Syst., 2022.
[18] K. Genova, F. Cole, A. Sud, A. Sarna, and T. Funkhouser. Local deep implicit functions for 3D shape. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
[19] 1. Goodfellow, J. Pouget-Abadicc M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Adv. Neural Inform. Process. Syst. 2014.
[20] J. Gu, L. Liu, P. Wang. and C. Theobalt. StyleNeRF: A style-based 3D-aware generator for high-resolution image synthesis. In Int. Conf. Learn. Represent., 2021.
[21] J. Gu, L.. Liu, P. Wang, and C. Theobalt. StyleNeRF: A style-hased 3D-aware generator for high-resolution image synthesis. https:/ / //gtthub//ttutbb//
http:/aaabooooeeeeeeee//syllllee 2021.
[22] Z. Hao, A. Mallya, S. Belongie, and M. Liu. GANeraft: Unsupervised 3D neural rendering of minecraft worlds. Int. Conf. Comput, Vis., 2021.
[23] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by two time-scale update rule converge to local nash equilibrium. Adv. Neural Inform. Process. Syst., 2017.
[24] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Adv. Neural Inform. Process. Syst., 2020.
[25] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In Int. Conf. Learn. Represent., 2018.
[26] T. Karras, S. Laine, and T. Aila. A based generator architecture for generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[27] T. Karras, S. Laine, M. Aittala, Hellsten. J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN, In IEEE Conf. Comput. Vis. Puttern Recog,, 2020.
[28] D. Kingma and M. Welling. Auto-encoding variational bayes. Int. Conf. Learn. Represent., 2014.
[29] Li, C. Niu, and K. Xu. Learning part generation and assembly for structure-aware shape synthesis. In Assoc, Adv. Artif. Intell., 2020.
[30] Y. Liao, K. Schwarz, L. M. Mescheder, and A. Geiger. Towards unsupervised learning of generative models for 3D controllable image synthesis. In IEEE Conf. Comput. Vis. Patiern Recog. 2020.
[31] T.-Y. Lin, M. Maire, S. Belongic, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, Microsoft coco: Common objects in context. In Eur. Conf. Comput. Vis., 2014.
[32] S. Liu, Y. Zhang. S. Peng, B. Shi, M. Pollefeys, and Z. Cui. DIST: Rendering deep implicit signed distance function with differentiable sphere tracing. In JEEE Conf. Comput. Vis. Pattern Recog., 2020.
[33] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Int. Conf. Comput. Vis., 2015.
[34] L. Mescheder, A. Geiger, and S. Nowozin. Which training methods for GANs do actually converge? In Int. Conf. Mach. Learn., 2018.
[35] L. Mescheder, M. Oechsle, M. Niemeyer. S. Nowozin, and A. Geiger. Occupancy networks: Learning 3D reconstruction in function space. In IEEE Conf. Comput. Vis. Pattern Recog.. 2019.
11

---

[36] B. Mildenhall, P. P. Srinivasan. M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes neural radiance fields for view synthesis. Communi. the ACM, 2021.
[37] T. Nguyen-Phuoe, C. Li, I.. Theis, C. Richardt, and Y.-L. Yang. HoloGAN: Unsupervised learning of 3D representations from natural images. In Int. Conf. Comput. Vis., 2019.
[38] T. Nguyen-Phuoe, C. Richardt, L. Mai, Y.-.. Yang, and N. Mitra. BlockGAN: Learning 3D object-aware scene representations from unlabelled images. Adv. Neural Inform. Process. Syst., 2020.
[39] M. Niemeyer and A. Geiger. GIRAFFE: Representing scenes compositional generative neural feature fields. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[40] R. Or-EI, X. Luo, M. Shan, E. Shechtman, J. Park, and I. Kernelmacherr-Shiizemae StyleSDF: High- resolution 3D-consistent image and geometry generation. In IEEE Conf. Comput. Vis, Pattern Recog.. 2022.
[41] X. Pan, X. Xu, C. C. Loy, C. Theobalt, and B. Dai. A shading-guides generative implicit model for shape-accurate 3D-aware image synthesis. In Adv. Neural Inform. Process. Syst., 2021.
[42] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepsSDF: Learning continuous signed distance functions for shape representation. In IEEE Conf. Comput. Vis. Pattern Recog.. 2019.
[43] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. FiLM: Visual reasoning with a general conditioning layer. In Assoc. Adv. Artif. Intell., 2018.
[44] C. Reiser, S. Peng, Y. Liao, and A. Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Int. Conf. Comput. Vis., 2021.
[45] D. Rezende and S. Mohamed, Variational inference with normalizing flows. In Int. Conf. Mach. Learn., 2015.
[46] K. Schwarz, Y. Liao, M. Niemeyer. and A. Geiger. GRAF: Generative radiance fields for 3D-aware image synthesis. In Adv. Neural Inform. Process. Syst., 2020.
[47] K. Schwarz, A. Sauer, M. Niemeyer, Y. Liao, and A. Geiger. VoxGRAF: Fast 3D-aware image synthesis with sparse voxel grids. In Adv. Neural Inform. Process. Syst., 2022.
[48] Z. Shi, Y. Shen, J. Zhu, D.-Y. Yeung, and Q. Chen. 3D-aware indoor scene synthesis with depth priors. In Eur. Conf. Compur. Vis., 2022.
[49] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. In Adv. Neural Inform. Process. Syst. 2020.
[50] I. Skorokhodov, S. Tulyakov, Y. Wang, and Wonka. EpiGRAF: Rethinking training of 3D GANs. In Adv. Neural Inform. Process. Syst., 2022.
[51] H. Turki, Ramanan, and Satyanarayanan. Mega-nerf: Scalable construction large-scale nerfs for virtual fhy-throughs. In JEFE Conf. Comput. Vis. Pattern Recog., 2022.
[52] P. Wang, L. Liu, Y. Liu, Theobalt. Komura, and W. Wang. NeuS: by volume rendering for multi-view reconstruction. In Adv. Neural Inform. Process. Syst., 2021.
Learning neural implicit surfaces
[53] Wu, C. Zhang, Xue, B. Freeman, and J. Tenenbaum. Learning a probabilistic latent space of object shapes via 3D generative adversarial modeling. Adv. Neural Inform. Process. Syst., 29, 2016.
[54] R. Wu, Y. Zhuang, K. Xu, H. Zhang, and B. Chen. Pq-net: A generative part seq2seq network for 3D shapes. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
[55] X. Xu, X. Pan, D. Lin, and B. Dai. Generative occupancy fields for 3D surface-aware image synthesis. In Adv. Neural Inform. Process. Syst., 2021.
[56] Y. Xu, S. Peng, C. Yang, Y. Shen, and B. Zhou. 3D-aware image synthesis via learning structural and textural representations. In IEEE Conf. Comput. Vis. Pattern Recog.. 2022.
[57] Y. Xu, S. Peng, C. Yang, Y. Shen, and B. Zhou. 3D-aware image synthesis via learning structural and textural representations. ttttps://gitaan httettttu.ooooeeeeeeeeeeeeeeettt
2022.
[58] Y. Xue, Li. K. Singh, and Y. J. Lee. Giraffe hd: A high-resolution 3D 3D-aware generative model. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
12

---

[59] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman. Volume rendering of neural implicit surfaces. In Adv. Neural Inform. Process, Syst., 2021.
[60] F. Yu, A. Seff. Y. Zhang. S. Song, T. Funkhouser. and Xiao. LSUN: Construction a large-scale image dataset using deep learning with humans in the loop. arXiv preprint urXiv:1506.0336 2015.
[61] J. Zhang, Z. Jiang. D. Yang. H. Xu, Y. Shi, G. Song, Z. Xu, X. Wang, and J. Feng. Avatargen: 3D generative model for animatable human avatars, arXiv preprint arXiv:2208. 00561, 2022.
[62]
Tang. head detection-hov to effectively exploit shape and texture features. In Eur. Conf. Comput. Vis., 2008.
Zhang, Sun,
[63] X. Zhao, F. Ma. D. Güera, Z. Ren. A. G. Schwing. and A. Colburn. Generative multiplane images: Making 2D GAN 3D-aware. In Eur. Conf. Comput. Vis., 2022.
[64] J. Y. Zhu, Z. Zhang, C. Zhang, Wu, A. Torralba. Tenenbaum, and Freeman. Visual object networks: Image generation with disentangled 3D representations. In Adv. Neural Inform. Process. Syst., 2018.
Appendix
A Experimental details
A.1 Backbone
In this study, we adopt the state-of-the-art 3D-aware GAN, EG3D [8], as our backbone model, which we have reproduced in our modularized codebase. This codebase is utilized to perform all experiments presented in this paper. Our investigation involves substituting each module in 3D GANs with alternative choices to study their individual effects. For instance, to explore the point embedder, we replace its tri- tri-plane with feature volume or other representations. Similarly, to investigate the feature decoder, we alter the depth or activation function of the decoder. Given our modularized design, these changes can easily implemented by adjusting the parameters in the configuration files.
In our experiments, we use only one-stage training to save computational resources. Specifically, we perform the training only at neural rendering resolution of 64,,4, without stepping the resolution up 128 128.
A.2 Point embedder
In this work, we mainly explore the capacity of three point embedders: MLP, volume, and tri-plane, as well as their combinations. The MLP-based point embedder utilizes a multi-layer perceptron (MLP) to transform raw point coordinates into point features. The volume-based point embedder queries point features from the feature volume, while the tri-plane based point embedder queries point features from the tri-plane feature representation.
The experimental settings for the tri-plane based point embedder are identical to those of [8]. For the ML.P-based point embedder, we adopt an MLP network to extract point features, similar to [20]. This network employs ReLU activation, 1 1x1 convolutions modulated by style vectors, and has depth of 16, with hidden layer dimension of 128 and an output layer dimension of 64. Notably, for MLP-based point features, the point embedder and the feature decoder are actually the same. As for the volume-basee based point embedder, we generate the feature volume using generator that utilizes 3D convolutions, which is the same network architecture as [56]. The feature volume resolution is set as 64. The feature decoder for volume-base point features is an MLP network with the same architecture as the MLP-based point embedder. Other settings including geometric representation, upsampler, pose priors, etc., remain consistent with our backbone model.
We also investigate the impact of composite point embedders, which entail combining two or more point embedders. Specifically, we explore combinations of MLP and volume, MLP and tri-plane, volume and tri-plane, and MLP, volume, and tri-plane. The combination of MLP and volume involves concatenating the MLP-based point features and volume-based point features along the feature channel dimension, while the other combinations follow the same principle. To conserve computational resources during training with composite point embedders, we set the Ml.e-based point features as the point coordinates, with the MLP serving as identical mapping.
13

---

A.3 Feature decoder
Considering that the feature decoder typically comprises a multi-layer perceptron, it is crucial to investigate the impact of its depth and activation layer type the performance. To this end, we conduct an empirical study on the depth and activation type of the feature decoder. Specifically, we perform experiments experimental setup inherited from Appendix A.2, but with varying depths the MLP. In terms the activation type of feature decoder, we employ both SIREN-based layers and ordinary ReLU-based layers in two settings: with and without an upsampler. And the depth of both MLPs was set to 8, with hidden dimension of 128 and an output dimension of 64. When training the model with SIREN-based layer, similar to [6]. we set the generator learning rate to 0.00006 and the discriminator learning rate to 0.0002.
three point features, including MLP, volume, and tri-plane, utilizing the
A.4 Geometric representation
Prior works, such NeuS [52], have incorporated the signed distance function (SDF) into the volume rendering formula to enable the reconstruction of smooth surface models. Moreover, 3D GANs [40] have explored the use of SDF goal is to examine the effects of two distinct geometric representations: density and SDF. We perform experiments on three point embedders: MLP, volume, and tri- rilp.a The baseline models utilizing vanilla density are identical to the model described in Appendix A.2. For SDF-based models, the feature decoder outputs SDF values. We also incorporate sphere initialization, eikonal loss, and minimal surface loss during training, with the same loss weight as [40].
geometrie representation for consistent generation. Hence, our
A.5 Upsampler
Our study aims to analyze the influence of the upsampler in 3D GANs. To ensure fair comparison, we conduct experiments on the generation of 256x256 resolution images. When an upsampler not utilized, the neural rendering resolution is the same as the image resolution, resulting in significantly longer training time and higher computational costs. We conduct experiments on a model without an upsampler, and disable dual discrimination since the output images are only fixed resolution of 256x256. Other settings remain consistent with the backbone model.
A.6 Pose priors
We additionally explore pose priors on the FFHQ [26] dataset. To obtain the accurate pose distribution (APD) of the dataset, we adopt the approach described in [6]. where the pose distribution modeled as a Gaussian prior. Camera poses are sampled from a normal distribution with a vertical mean of T/2 radians, standard deviation of 0.155 radians; and horizontal mean of ///2 radians, a standard deviation of 0.3 radians. Regarding the random pose distribution (RPD), Gaussian distribution also assumed for the pose, with only the vertical/horizontal mean and standard deviation being randomly assigned. And we introduce the acquisition of ground -round poses in Appendix B.
B Dataset details
We conduct our experiments on three datasets, including FFHQ [26], Cats [62], and ShapeNet
Cars [10]. Since the original data of these datasets lacks pose labels, we perform preprocessing steps for each dataset. We follow [8] to align, crop and get pose matrix for each image in the FFHQ [26]
dataset. Cats [62] contains more than 6K real-world cat images, and the data is preprocessed following [14]. The ShapeNet Cars [10] dataset comprises various synthetic car models. We use the dataset rendered from [8] which is composed approximately 530K images. Unlike the forward- facing datasets, its camera poses encompass the full range of 360 horizontal and 180 vertical distributions. In our experiments, the resolution of 256 x 256 is employed for FFHQ and Cats datasets, while for the ShapeNet Cars dataset, resolution of 128 x 128 is used.
14

---

Table Al: Training time and inference speed of models with different point embedders. The tri-plane- based point embedder exhibits the highest computational efficiency.
Point Embedker
FFIQ 256x256
Cats [62] 256 256
Cars [10]
128x 128
Volume
Tri-plane
Training Time
Inference Speed
Training Time
Inference Speed
Training Time
Inference Speed
5.6 Days
29 FPS
6.1 Days
29 PPS
6,9 Days
23 FPS
6.8
FPS
7.3 Days
24
Days
20 FPS
Days
3.1 Days
Days
6.9 Days
24S
7.5 Days
24 FPS
Days
20 FPS
K
2.8 Days
49 FPS
3.3 Days
49 FPS
2.8 Days
48 IPS
X
V
3.8 Days
38 FPS
4.4 Days
38 FPS
3,9 Days
35 FPS
3.9 Days
FPS
4.5 Days
38
Days
35
Samples synthesized FFHQ [26] with truncation 0.7 using the model with MLP. based point embedder. For each generated identity, we show the underlying geometry under two views and appearance under three views.
Figure A1:
C
More results
C.1 Efficiency comparison
We report the training time and inference speed of models utilizing various point embedders in Tab. A1. The training time computed by training our models on 8 NVIDIA A100 GPUs for 25 million images. Inference speed is measured on single NVIDIA A100 GPU, where we processed IK images and calculated the average FPS. As shown in Tab. A1, tripplane-based point embedders exhibit superior computational efficiency compared to MLP-based and volume-based point embedders. MI.P-based point embedders require a larger number of layers to extract point features, leading to longer processing times. Among these, volume-based point embedder is the least efficient, as it involves 3D convolutions.
C.2 More qualitative results
We show more results of different point embedders on FFHQ [26] in Figs. Al to A3. Interestingly, we observe that models equipped with tri-plane-baser point embedders generate 3D shapes with sharper noses, while those equipped with MLP-based or volume-based point embedders do not exhibit this characteristic. This phenomenon can be observed more clearly in Fig. A4. However, the underlying reason for this remains unknown.
15

---

Samples synthesized on FFHQ [26] with truncation 0.7 using the model with volume- based point embedder. For each generated identity, we show the underlying geometry under two views and appearance under three views.
Figure A2:
Figure A3: Samples synthesized on FFHQ [26] with truncation 0.7 using the model with tri-plane- based point embedder. For each generated identity, we show the underlying geometry under two views and appearance under three views.
Tri-plane
Volume
MLP
Figure A4: Qualitative comparison across different single point embedders on FFHQ [26], zoom in for better viewing. Models equipped with tri-plane-based point embedders generate 3D shapes with sharper noses, which can appear unnatural.
16

---

D Limitations and future work
Efficiency. Training our 3D GAN models time-consuming process, especially when utilizing MLP-based and volume-based point embedders. To draw meaningful conclusions, we must conduct a plethora of experiments. Currently we are uncertain about how to improve the efficiency of our models' training. Each experiment requires careful hyper-parne parameter tuning, which is challenging task. However, due to computational limitations, multiple runs of our experiments are infeasible. Consequently, we identify the "optimal" settings using limited number of attempts.
be rather unstable. Some experiments are highly sensitive to parameerraesterss such as the value of RI regularization [34], learning rate, etc. However, our study does investigate this aspect in depth. Future research addressing hyper-parameter sensitivity and training stability may lead to significant reductions in training costs and more compelling results.
Training stability. We have observed that training 3D GANs
Universality. We trained all our 3D-aware image synthesis models on simple, object-level datasets, such FFHQ [26] for face generation, and our conclusions are based on these datasets. However, paper does not explore the extension of 3D GANs to a higher degree of universality, which represents promising research direction for the future. This universality pertains to generating diverse objects (e.g., ImageNet [11] or Microsoft CoCo [31]), dynamic objects or scenes, and large-scale scenes.
17


Reviews:
[{"review_id": "PTzTQlSUFk", "reviewer": "Reviewer_ggV7", "summary": "", "questions": "", "limitations": "1.If the propsoed codebase is not the first in this field, the authors should introduce and compare related codebases, and give comparisons between them in the experiments. This is to show the advantages of this new work.\n2. How to define \"codebase\" should be clearly given, is it a module-level library? but it also describes each single reference approach (e.g.,[7, 20, 53, 9]) as codebases, it may be confusing.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper provides a highly modularized codebase for 3D-aware image synthesis, including a pose sampler, point sampler, stochasticity mapper, point embedder, feature decoder, volume renderer, upsampler, evaluator, and visualizer. \nThe proposed codebase contributes to re-implement a range of classic algorithms, and customize algorithms by replacing particular modules from the function perspective. This codebase allows users to change the module arbitrarily and independently for convenient algorithm development. The results for reproduced methods show that the codebase is able to generate equivalent performance. And the analysis of each module demonstrates that the code base is advanced in the capacity of the functionalities.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "VkzyTsqbg1", "reviewer": "Reviewer_783m", "summary": "", "questions": "", "limitations": "Please refer to *Opportunities For Improvement* and *Documentation*", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ In general, it is a well-motivated paper and the writing is easy to follow. A modularized codebase is in need and would help boost the development of 3D-aware image synthesis.\n+  The authors evaluate several representative methods and conduct detailed experiments on FFHQ.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ABABwW008t", "reviewer": "Reviewer_M7ju", "summary": "", "questions": "", "limitations": "Yes L356", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper tries to make research clearer and easier by disentangling various factors in the implementation.\n1. This paper provides useful observations for the research community. E.g., Table 1.\n1. The background section thoroughly covers the literature.\n1. Experiments cover diverse methods and ablations.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "j94f36wzr5", "reviewer": "Reviewer_DaES", "summary": "", "questions": "", "limitations": "The authors addressed the potential negative societal impact, but did not address the limitations. \nSee \"Opportunities For Improvement\" for the corresponding suggesttions.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "> + Beneficial analyses: The analyses regarding different modules can help further work to build on top of a better baseline.\n> + On-par or even better reproduction results: it is non-trivial to achieve on-par accuracy as compared to all baselines. Thus, this codebase can be one of the top options for beginners.\n> + Modularized design space: the authors decoupled the complex design options in 3D-aware image synthesis into several modules, which enables the ability to allow users to replace any part with their proposed new modules.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: **Abstract:**
The paper addresses the advancement of 3D-aware image synthesis by creating a modularized codebase that integrates the neural radiance field (NeRF) into the generator of a generative adversarial network (GAN). The codebase aims to clarify the contribution of various techniques and offer a structured platform for researchers to fairly compare methods and develop modules independently. The codebase reproduces multiple advanced algorithms and provides in-depth analyses to deepen the understanding of existing methodologies.

**General Overview:**
Across all reviewers, there is a unanimous consensus that the paper is of value and should be accepted. The primary highlights recognized by reviewers include the establishment of a highly modularized codebase and its significant potential to accelerate the research and development in the 3D-aware image synthesis domain.

**Specific Highlights:**
  1. Modular Codebase Design: As pointed out by Reviewer ggV7, the paper's primary contribution is the development of a modularized codebase, encompassing several crucial components like pose sampler, point sampler, and volume renderer, to name a few. The design empowers users to modify or replace modules individually, making algorithm development more flexible and accessible.
  2. Reproduction and Performance: Reviewer ggV7 praises the fact that the introduced codebase is adept at re-implementing a diverse array of algorithms and maintains the performance of these reproduced methods. This showcases the codebase's capability to stand as a standard benchmark in the field.
  3. Contribution to the Research Community: Reviewer M7ju acknowledges the paper's attempt to simplify research by presenting a clearer picture of the multifaceted implementation aspects. The paper, through its observations (like those in Table 1), is regarded as a significant contribution to the wider research community.
  4. Background and Experiments: M7ju also commends the paper's comprehensive coverage of related literature, ensuring readers have a well-rounded understanding of the topic. The variety and depth of experiments, which include diverse methods and ablations, further strengthens the paper's findings.
  5. Clarity and Motivation: Reviewer 783m finds the paper's motivation clear and the writing coherent, which enhances its accessibility to a wider audience. There's also an acknowledgment of the demand for such a modularized codebase in the field, emphasizing its timeliness and relevance.

**Conclusion:**
The paper is well-received by the reviewers, with all of them recommending its acceptance. The modularized codebase's design and its potential to standardize research in 3D-aware image synthesis is particularly noteworthy. Furthermore, the paper's experiments, analysis, and literature coverage make it a valuable resource for the community. The consensus is that the paper fills a pertinent gap in the research arena, making it a commendable piece of work.
Decision: Accept (Poster)