SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which may result in the output features of certain embedders being overlooked. Therefore, using a single type of point feature is sufficient for model training.
Number of planes. We also investigate the plane-based point embedders by examining the necessity of using three planes, as in [8]. To accomplish this, we substitute the triplane with both single-plane and biplane representations. As expected, the single-plane representation performs poorly, as the model cannot
Table 4: Analysis of plane-based point features.
Point Embedder	FFHQ [26]	Cats [62]	Cars [10]
FID↓	ID↑	DE↓	PE↓	RE↓	FID↓	FID↓
Bi-plane (XY + XZ)	4.55	0.738	0.398	4.6e-4	0.083	4.33	3.21
Bi-plane (XY + ZY)	4.38	0.746	0.379	4.4e-4	0.104	3.95	2.81
Bi-plane (XZ + ZY)	4.52	0.754	0.385	5.5e-4	0.095	4.77	2.50
Tri-plane	4.72	0.743	0.547	4.5e-4	0.111	3.99	2.75
accurately determine a point's location in 3D space with only one plane. However, the bi-plane representation slightly outperforms the tri-plane representation, as shown in Tab. 4. The bi-plane model accurately determines 3D point positions without ambiguities and has fewer parameters. Thus the bi-plane representations are on par with the tri-plane representations in 3D GANs. Nonetheless, this does not necessarily imply that fewer planes are always superior. The datasets we trained on are object-level and relatively simple, making the bi-plane representation sufficient. In more complex tasks, additional planes may be required.
Depth of feature decoder. As illustrated in Tab. 5, the depth of the feature decoder plays a crucial role in the accuracy of MLP-based point features. However, for volume and plane-based point features, the depth of the feature decoder appears to be less significant. This can be attributed to the fact that,
in the case of volume or tri-plane-based point features, the feature volume or feature plane predominantly determines the point representation's capability, while the feature decoder serves as a simple mapper to convert the point features into corresponding density or color values. Conversely, for MLP-based point features, the model relies solely on the feature decoder to transform the raw coordinates to the density or color values, and thus the depth of feature decoder determines the model's capacity.
Activation type of feature decoder. As illustrated in Tab. 6, we find that without the upsampler, SIREN-based MLP outperforms the ordinary MLP while with the upsampler the result turns out to be the opposite. One plausible explanation for this observation could be the design of the upsampler module, which may have the tendency to amplify the "ripple" artifacts induced by the SIREN-based layers [6], while mitigating the blurry artifacts produced by the ordinary layers.
Geometric representation. The results presented in Tab. 7 indicate that the SDF-based representations generally yields inferior quantitative metrics compared to vanilla density-based representations for all point embeddings. These findings are consistent with previous works [59]. Moreover, we observe that utilizing sphere initialization and additional regularization terms, such as eikonal loss and minimal surface loss [40], can adversely affect GAN training, leading to flawed results. Thus, incorporating
SDF does not offer superior generation quality compared to density-based representations.
Pose priors. As shown in Tab. 8 and Fig. 4, the pose priors significantly impacts the quality of the generated 3D geometry. Specifically, when training with the random pose distribution (RPD), the generated 3D geometry lacks normal facial structures and exhibits inconsistency in novel view synthesis. On the other hand, when trained solely with the accurate pose distribution (APD), the model tends to overfit to a "flat face shape" and consequently struggles with generating different views of faces. However, when provided with ground-truth pose (GTP) information, the model can produce photo-realistic images and generate adequate and high-quality geometry.
Upsampler. As evidenced in Tab. 9, the model incorporating an upsampler exhibits superior image quality, owing to its capacity to enhance image details. Nonetheless, the inclusion of an upsampler, typically accomplished through convolutional
Table 5: Analysis of the depth of feature decoder.
Point Embedder	Depth	FFHQ [26]
FID↓	ID↑	DE↓	PE↓	RE↓
MLP	4	17.22	0.761	0.807	12.2e-4	0.105
8	7.39	0.782	0.552	7.3e-4	0.087
16	5.15	0.777	0.470	5.0e-4	0.091
Volume	4	5.65	0.784	0.437	4.4e-4	0.095
8	5.18	0.787	0.381	4.0e-4	0.100
16	4.65	0.778	0.413	5.1e-4	0.085
Tri-plane	2	4.72	0.743	0.547	4.5e-4	0.111
4	4.77	0.750	0.414	4.4e-4	0.101
8	5.58	0.750	0.566	5.6e-4	0.108
Table 6: Analysis of the activation type used in feature decoder.
Activation Type	FFHQ [26]
FID↓	ID↑	DE↓	PE↓	RE↓
- w/ upsampler	256 × 256
SIREN	11.66	0.763	0.352	9.1e-4	0.089
ReLU	7.39	0.782	0.552	7.3e-4	0.087
- w/o upsampler	64 × 64
SIREN	6.58	0.741	0.340	6.6e-4	0.071
ReLU	7.30	0.729	0.498	4.6e-4	0.084
Table 7: Analysis of the underlying geometric representation.
Geometric Representation	FFHQ [26]
FID↓	ID↑	DE↓	PE↓	RE↓
MLP	SDF	8.87	0.610	0.874	5.9e-4	0.184
Density	5.15	0.777	0.470	5.0e-4	0.091
Volume	SDF	7.27	0.676	0.938	5.0e-4	0.200
Density	4.65	0.778	0.413	5.1e-4	0.085
Tri-plane	SDF	13.31	0.534	0.626	10.9e-4	0.161
Density	4.72	0.743	0.547	4.5e-4	0.111
mpared to density-based representations.
Table 8: Analysis of using different pose priors.
Pose Prior	FFHQ [26]
FID↓	ID↑	DE↓	PE↓	RE↓
MLP w/ RPD	14.56	0.413	1.513	5.8e-2	0.405
MLP w/ APD	9.96	0.788	1.659	5.9e-2	0.407
MLP w/ GTP	5.15	0.777	0.470	5.0e-4	0.091
Volume w/ RPD	10.47	0.429	1.562	5.5e-2	0.390
Volume w/ APD	7.34	0.731	1.125	4.8e-2	0.367
Volume w/ GTP	4.65	0.778	0.413	5.1e-4	0.085
Tri-plane w/ RPD	15.18	0.427	2.181	5.6e-2	0.379
Tri-plane w/ APD	5.45	0.764	1.502	5.4e-2	0.405
Tri-plane w/ GTP	4.72	0.743	0.547	4.5e-4	0.111
Table 9: Analysis of the effect of upsampler, with tri-plane point feature as an example.
Upsampler	Resolution	FFHQ [26]	Training Time	Inference Speed
FID↓	ID↑	DE↓	PE↓	RE↓
✓	256×256	4.72	0.743	0.547	4.5e-4	0.111	2.7 Days	49 FPS
X	256×256	6.86	0.749	0.443	6.2e-4	0.104	6.9 Days	20 FPS
Figure 4: Qualitative comparison across different pose priors, where having ground-truth pose for each training example plays a vital role in helping the model learn adequate geometry.
layers and non-linear activation, may compromise multi-view consistency. Conversely, eliminating the upsampler yields heightened view consistency, albeit at the expense of significantly increased computational cost, manifested in prolonged training and inference times.
4.3 Outlooks
Based on the developed codebase, our experimental analysis has revealed numerous unresolved challenges in 3D GANs that warrant further investigation to enhance its applicability in downstream domains. Here, we propose several promising avenues for future research in the realm of 3D-aware image synthesis, with the aim of fostering continued progress and facilitating breakthroughs in this rapidly evolving field.
Training stability. Training 3D-aware image synthesis models is often prone to mode collapse, resulting in unsatisfactory image generation and inferior geometry. This may be due to a mismatch between the distribution of physically meaningful factors, such as camera poses and rendering parameters, and that of real images. Therefore, it is crucial to investigate the training stability of 3D-aware image synthesis models.
Efficiency. Training 3D-aware image synthesis models can take 3-10 days on multiple high-end GPUs, and the models can be slow during inference, which limits their practicality in downstream applications. Therefore, the need to improve the training efficiency of 3D generative models is increasing, and enhancing the inference efficiency is crucial for their practical deployment in real-world applications.
Pose acquisition. The role of pose in 3D GAN training is of great significance, but obtaining accurate pose information from in-the-wild data is a non-trivial task. One potential solution is to incorporate a pose estimator into the 3D GAN model. Further research may focus on improving pose acquisition methods for 3D-aware image synthesis models.
Hybrid representations. The current representations employed in 3D GANs are primarily implicit representations. However, in some complicated scenarios, these representations may prove insufficient. For instance, when modeling a human head, the implicit representation is suitable for hair modeling, while explicit representations such as meshes may be more appropriate for face modeling. To address this issue, future research should focus on exploring hybrid representations that can handle more complex data.
This paper proposes a modularized codebase for 3D-aware image synthesis development, enabling researchers to independently develop and replace each module. This approach provides a means to compare various methods fairly and recognize their key contributions from a module perspective. Extensive experimental analyses reveal the potential future research directions for 3D-aware image synthesis models.
The development of 3D-aware image synthesis has significant potential for impact in fields such as entertainment, gaming, and virtual reality. This technology has the ability to revolutionize the way we create and interact with virtual environments by generating realistic 3D models and images. However, as with any emerging technology, ethical considerations must be taken into account to prevent potential misuse for malicious purposes such as the creation of realistic fake images or videos. Therefore, researchers and developers must consider the social and ethical implications of this technology and work collaboratively with stakeholders to develop appropriate guidelines and best practices, ensuring that it is developed and used ethically and responsibly. Open and transparent discussions about the ethical implications of this technology can help maximize its potential benefits while minimizing potential risks.
We would like to express our gratitude to Kai Zhu for his valuable assistance in reproducing EpiGRAF [50] and to Jiaxin Xie for her invaluable help in reproducing GRAM [14]. We extend our thanks to Hao Ouyang and Yuxi Xiao for their insightful comments and discussions, particularly their suggestion on refining this manuscript. We are also thankful to the anonymous reviewers for their valuable feedback and suggestions, which greatly contributed to improving the clarity of this manuscript.
[1] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Int. Conf. Comput. Vis., 2021.
[2] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[3] Y. Bengio, R. Ducharme, and P. Vincent. A neural probabilistic language model. In Adv. Neural Inform. Process. Syst., 2000.
[4] A. W. Bergman, P. Kellnhofer, Y. Wang, E. R. Chan, D. B. Lindell, and G. Wetzstein. Generative neural articulated radiance fields. arXiv preprint arXiv:2206.14314, 2022.
[5] A. Brock, T. Lim, J. M. Ritchie, and N. Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv:1608.04236, 2016.
[6] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein.  $\pi$ -GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[7] E. R. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein.  $\pi$ -GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. https://github.com/marcoamonteiro/pi-GAN, 2021.
[8] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, et al. Efficient geometry-aware 3D generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[9] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, et al. Efficient geometry-aware 3D generative adversarial networks. https://github.com/NVlabs/eg3d, 2022.
[10] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. ShapeNet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012, 2015.
[11] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conf. Comput. Vis. Pattern Recog., 2009.
[12] J. Deng, J. Guo, N. Xue, and S. Zafeiriou. ArcFace: Additive angular margin loss for deep face recognition. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[59] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman. Volume rendering of neural implicit surfaces. In Adv. Neural Inform. Process. Syst., 2021.
[60] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
[61] J. Zhang, Z. Jiang, D. Yang, H. Xu, Y. Shi, G. Song, Z. Xu, X. Wang, and J. Feng. Avatargen: a 3D generative model for animatable human avatars. arXiv preprint arXiv:2208.00561, 2022.
[62] W. Zhang, J. Sun, and X. Tang. Cat head detection-how to effectively exploit shape and texture features. In Eur. Conf. Comput. Vis., 2008.
[63] X. Zhao, F. Ma, D. Guera, Z. Ren, A. G. Schwing, and A. Colburn. Generative multiplane images: Making a 2D GAN 3D-aware. In Eur. Conf. Comput. Vis., 2022.
[64] J.-Y. Zhu, Z. Zhang, C. Zhang, J. Wu, A. Torralba, J. Tenenbaum, and B. Freeman. Visual object networks: Image generation with disentangled 3D representations. In Adv. Neural Inform. Process. Syst., 2018.
A Experimental details
A.1 Backbone
In this study, we adopt the state-of-the-art 3D-aware GAN, EG3D [8], as our backbone model, which we have reproduced in our modularized codebase. This codebase is utilized to perform all experiments presented in this paper. Our investigation involves substituting each module in 3D GANs with alternative choices to study their individual effects. For instance, to explore the point embedder, we replace its tri-plane with feature volume or other representations. Similarly, to investigate the feature decoder, we alter the depth or activation function of the decoder. Given our modularized design, these changes can be easily implemented by adjusting the parameters in the configuration files.
In our experiments, we use only one-stage training to save computational resources. Specifically, we perform the training only at a neural rendering resolution of  $64 \times 64$ , without stepping the resolution up to  $128 \times 128$ .
A.2 Point embedder
In this work, we mainly explore the capacity of three point embeddings: MLP, volume, and tri-plane, as well as their combinations. The MLP-based point embedder utilizes a multi-layer perceptron (MLP) to transform raw point coordinates into point features. The volume-based point embedder queries point features from the feature volume, while the tri-plane based point embedder queries point features from the tri-plane feature representation.
The experimental settings for the tri-plane based point embedder are identical to those of [8]. For the MLP-based point embedder, we adopt an MLP network to extract point features, similar to [20]. This network employs ReLU activation,  $1 \times 1$  convolutions modulated by style vectors, and has a depth of 16, with a hidden layer dimension of 128 and an output layer dimension of 64. Notably, for MLP-based point features, the point embedder and the feature decoder are actually the same. As for the volume-based point embedder, we generate the feature volume using a generator that utilizes 3D convolutions, which is the same network architecture as [56]. The feature volume resolution is set as  $64 \times 64$ . The feature decoder for volume-based point features is an MLP network with the same architecture as the MLP-based point embedder. Other settings including geometric representation, upsampler, pose priors, etc., remain consistent with our backbone model.
We also investigate the impact of composite point emitters, which entail combining two or more point emitters. Specifically, we explore combinations of MLP and volume, MLP and tri-plane, volume and tri-plane, and MLP, volume, and tri-plane. The combination of MLP and volume involves concatenating the MLP-based point features and volume-based point features along the feature channel dimension, while the other combinations follow the same principle. To conserve computational resources during training with composite point emitters, we set the MLP-based point features as the raw point coordinates, with the MLP serving as an identical mapping.
A.3 Feature decoder
Considering that the feature decoder typically comprises a multi-layer perceptron, it is crucial to investigate the impact of its depth and activation layer type on the performance. To this end, we conduct an empirical study on the depth and activation type of the feature decoder. Specifically, we perform experiments on three point features, including MLP, volume, and tri-plane, utilizing the experimental setup inherited from Appendix A.2, but with varying depths of the MLP. In terms of the activation type of feature decoder, we employ both SIREN-based layers and ordinary ReLU-based layers in two settings: with and without an upsampler. And the depth of both MLPs was set to 8, with a hidden dimension of 128 and an output dimension of 64. When training the model with a SIREN-based layer, similar to [6], we set the generator learning rate to 0.00006 and the discriminator learning rate to 0.0002.
A.4 Geometric representation
Prior works, such as NeuS [52], have incorporated the signed distance function (SDF) into the volume rendering formula to enable the reconstruction of smooth surface models. Moreover, 3D GANs [40] have explored the use of SDF as a geometric representation for consistent generation. Hence, our goal is to examine the effects of two distinct geometric representations: density and SDF. We perform experiments on three point embeddings: MLP, volume, and tri-plane. The baseline models utilizing vanilla density are identical to the model described in Appendix A.2. For SDF-based models, the feature decoder outputs SDF values. We also incorporate sphere initialization, eikonal loss, and minimal surface loss during training, with the same loss weight as [40].
A.5 Upsampler
Our study aims to analyze the influence of the upsampler in 3D GANs. To ensure a fair comparison, we conduct experiments on the generation of  $256 \times 256$  resolution images. When an upsampler is not utilized, the neural rendering resolution is the same as the image resolution, resulting in significantly longer training time and higher computational costs. We conduct experiments on a model without an upsampler, and disable dual discrimination since the output images are only at a fixed resolution of  $256 \times 256$ . Other settings remain consistent with the backbone model.
A.6 Pose priors
We additionally explore pose priors on the FFHQ [26] dataset. To obtain the accurate pose distribution (APD) of the dataset, we adopt the approach described in [6], where the pose distribution is modeled as a Gaussian prior. Camera poses are sampled from a normal distribution with a vertical mean of  $\pi / 2$  radians, standard deviation of 0.155 radians; and a horizontal mean of  $\pi / 2$  radians, a standard deviation of 0.3 radians. Regarding the random pose distribution (RPD), a Gaussian distribution is also assumed for the pose, with only the vertical/horizontal mean and standard deviation being randomly assigned. And we introduce the acquisition of ground-truth poses in Appendix B.
B Dataset details
We conduct our experiments on three datasets, including FFHQ [26], Cats [62], and ShapeNet Cars [10]. Since the original data of these datasets lacks pose labels, we perform preprocessing steps for each dataset. We follow [8] to align, crop and get pose matrix for each image in the FFHQ [26] dataset. Cats [62] contains more than  $6K$  real-world cat images, and the data is preprocessed following [14]. The ShapeNet Cars [10] dataset comprises various synthetic car models. We use the dataset rendered from [8] which is composed of approximately  $530K$  images. Unlike the forward-facing datasets, its camera poses encompass the full range of  $360^{\circ}$  horizontal and  $180^{\circ}$  vertical distributions. In our experiments, the resolution of  $256 \times 256$  is employed for FFHQ and Cats datasets, while for the ShapeNet Cars dataset, a resolution of  $128 \times 128$  is used.
Table A1: Training time and inference speed of models with different point embeddings. The tri-plane-based point embedder exhibits the highest computational efficiency.
Point Embedder	FFHQ [26] 256×256	Cats [62] 256×256	Cars [10] 128×128
MLP	Volume	Tri-plane	Training Time	Inference Speed	Training Time	Inference Speed	Training Time	Inference Speed
✓	X	X	5.6 Days	29 FPS	6.1 Days	29 FPS	6.9 Days	23 FPS
X	✓	X	6.8 Days	24 FPS	7.3 Days	24 FPS	8.3 Days	20 FPS
X	X	✓	2.7 Days	49 FPS	3.1 Days	49 FPS	2.7 Days	48 FPS
✓	✓	X	6.9 Days	24 FPS	7.5 Days	24 FPS	8.4 Days	20 FPS
✓	X	✓	2.8 Days	49 FPS	3.3 Days	49 FPS	2.8 Days	48 FPS
X	✓	✓	3.8 Days	38 FPS	4.4 Days	38 FPS	3.9 Days	35 FPS
✓	✓	✓	3.9 Days	38 FPS	4.5 Days	38 FPS	4.0 Days	35 FPS
Figure A1: Samples synthesized on FFHQ [26] with truncation 0.7 using the model with an MLP-based point embedder. For each generated identity, we show the underlying geometry under two views and appearance under three views.
C More results
C.1 Efficiency comparison
We report the training time and inference speed of models utilizing various point embeddings in Tab. A1. The training time is computed by training our models on 8 NVIDIA A100 GPUs for 25 million images. Inference speed is measured on a single NVIDIA A100 GPU, where we processed 1K images and calculated the average FPS. As shown in Tab. A1, tri-plane-based point embeddings exhibit superior computational efficiency compared to MLP-based and volume-based point embeddings. MLP-based point embeddings require a larger number of layers to extract point features, leading to longer processing times. Among these, volume-based point embedding is the least efficient, as it involves 3D convolutions.
C.2 More qualitative results
We show more results of different point embeddings on FFHQ [26] in Figs. A1 to A3. Interestingly, we observe that models equipped with tri-plane-based point embeddings generate 3D shapes with sharper noses, while those equipped with MLP-based or volume-based point embeddings do not exhibit this characteristic. This phenomenon can be observed more clearly in Fig. A4. However, the underlying reason for this remains unknown.
Figure A2: Samples synthesized on FFHQ [26] with truncation 0.7 using the model with a volume-based point embedder. For each generated identity, we show the underlying geometry under two views and appearance under three views.
Figure A3: Samples synthesized on FFHQ [26] with truncation 0.7 using the model with a tri-plane-based point embedder. For each generated identity, we show the underlying geometry under two views and appearance under three views.
Figure A4: Qualitative comparison across different single point embeddings on FFHQ [26], zoom in for better viewing. Models equipped with tri-plane-based point embeddings generate 3D shapes with sharper noses, which can appear unnatural.
D Limitations and future work
Efficiency. Training our 3D GAN models is a time-consuming process, especially when utilizing MLP-based and volume-based point embeddings. To draw meaningful conclusions, we must conduct a plethora of experiments. Currently we are uncertain about how to improve the efficiency of our models' training. Each experiment requires careful hyper-parameter tuning, which is a challenging task. However, due to computational limitations, multiple runs of our experiments are infeasible. Consequently, we identify the "optimal" settings using a limited number of attempts.