StyleNeRF: A Style-based 3D Aware Generator for High-resolution Image Synthesis
Authors: Jiatao Gu
Lingjie Liu
Peng Wang
Christian Theobalt
Venue: ICLR 2022 Poster (2022)
Keywords: Neural Radiance Field
StyleGAN
high resolution image generation

Abstract: We propose StyleNeRF, a 3D-aware generative model for photo-realistic high-resolution image synthesis with high multi-view  consistency, which can be trained on unstructured 2D images. Existing approaches either cannot synthesize high-resolution images with fine details or yield clearly noticeable 3D-inconsistent artifacts. In addition, many of them lack control on style attributes and explicit 3D camera poses. To address these issues, StyleNeRF integrates the neural radiance field (NeRF) into a style-based generator to tackle the aforementioned challenges, i.e., improving rendering efficiency and 3D consistency for high-resolution image generation. To address the first issue, we perform volume rendering only to produce a low-resolution feature map, and progressively apply upsampling in 2D. To mitigate the inconsistencies caused by 2D upsampling, we propose multiple designs including a better upsampler choice and a new regularization loss to enforce 3D consistency. With these designs, StyleNeRF is able to synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality. StyleNeRF also enables control of camera poses and different levels of styles, which can generalize to unseen views. It also supports challenging tasks such as style mixing, inversion and simple semantic edits. 


Published as a conference paper at ICLR 2022
STYLENERF: A STYLE-BASED 3D-AWARE GENERA- TOR FOR HIGH-RESOLUTIO IMAGE SYNTHESIS
Jiatao Gu Lingjie Liu Peng Wang', Christian Theobalt"
Meta AI Max Planck Institute for Informatics Thhe University of Hong Kong
jigu Eb. com *{1liu, theobalt]empi -nnf mpg. de "pwang38cs hku .kk
ABSTRACT
We propose StyleNeRF, 3D-aware generative model for photo-realistic high- resolution image synthesis with high multi-view consistency, which can be trained unstructured 2D images. Existing approaches either cannot synthesize high- resolution images with fine details or yield noticeable 3D-inconsistent artifacts. addition, many of them lack control over style attributes and explicit 3D camera poses. StyleNeRF integrates the neural radiance field (NeRF) into a style-based generator tackle the aforementioned challenges, i.e., improving rendering ef- ficiency and 3D consistency for high-resolution image generation. We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the first issue. To mitigate the inconsistencies caused by 2D upsampling, we propose multiple designs, including a better up- sampler and a new regularization loss. With these designs, StyleNeRF can synthe- size high-resolution images at interactive rates while preserving 3D consistency at high quality. StyleNeRR also enables control of camera poses and different lev- els of styles, which can generalize to unseen views. It also supports challenging tasks, including style mixing and semantic editing. Code and pre-trained models are available at:
INTRODUCTION
Figure 1: Synthesized 1024 images by StyleNeRF, with the low-resolution feature maps.
Photo-realistic free-view image synthesis is a long-standing problem in computer vision and com- puter graphics. Traditional graphics pipeline requires production-qualie 3D models, computation- ally expensive rendering and manual work, making it challenging to apply to large- scale real-world scenes. In the meantime, Generative Adversarial Networks (GANs, Goodfellow et al.) (014) be trained on a large number of unstructured images to generate high-quality images. However, most GAN models operate in 2D space and lack the 3D understanding of the input, which results in their inability to synthesize images of the same scene with 3D consistency and direct camera control.
Natural images the 2D projection of the 3D world. Hence, recent works generative mod- els (Schwarz et al. 2020) Chan et al. 202T) enforce 3D structures by incorporating a neural radiance field (NeRF, Mildenhall et al. 2020). However, these methods cannot synthesize high- high-resolution images with delicate details due to the computationally expensive rendering process of NeRF. Fur- thermore, the slow rendering process leads to inefficient training and makes these models unsuitable for interactive applications. GIRAFFE (Niemeyer & Geiger 2021b) combines NeRF with a CNN-
corresponding author.

---

Published as a conference paper at ICLR 2022
based renderer, which has the potential to synthesize high-resolution images. However, this method falls short of 3D-consistent image generation and so far has not shown high-resolution results.
We propose StyleNeRF, new 3D-aware generative model for high-resolution 3D consistent image synthesis at interactive rates. It also allows control of the 3D camera pose and enables control of specific style attributes. StyleNeRF incorporates 3D scene representations into a style-based genera- tive model. To prevent the expensive direct color image rendering from the original NeRF approach, only use NeRF to produce low-resolution feature map and upsample it progressively high resolution. To improve 3D consistency, we propose several designs, including a desirable upsampler that achieves high consistency while mitigating artifacts in the outputs, a novel regularization term that forces the output to match the rendering result of the original NeRF and fixing the issues of view direction condition and noise injection. StyleNeRF is trained using unstructured real-world images. A progressive training strategy significantly improves the stability of learning real geometry.
We evaluate StyleNeRF on various challenging datasets. StyleNeRF can synthesize photo-realistic 1024 images at interactive rates while achieving high multi-view consistency. None of the existing methods can achieve both characteristics. Additionally, StyleNeRF enables direct control on styles.
2 RELATED WORK
Neural Implicit Fields Representing 3D scenes as neural implicit fields has increasingly gained much attention. Michalkiewicz et al.) (2019); Mescheder et (2019); Paar et (2019)) Peng et al. (2020) predict neural implicit fields with 3D supervision. Some of them (Sitzmann et al.) 2019 Niemeyer et al. 2019) assume that the ray color only lies on the geometry surface and propose differentiable renderers to learn a neural implicit surface representation. NeRF and and similar works (Lombardi et al., 2019 Mildenhall et al. 2020 Liu et al. 2020; Zhang et al., (220) utilize volume rendering technique to render neural implicit volume representations for novel view synthe- sis. In this work, we focus generative NeRF Unlike the discussed methods, which require posed multi-view images, our approach only needs unstructured single-view images for training.
Image Synthesis with GANs Starting from Goodfellow et al. (2014). GANs have demonstrated high-quality results (Durukaar et al., 2017 Mordido et al., 2018 Doan et al., 2019 Zhang et al, 2019 Brock et al.) 2018: Karras et al.) 2018). StyleGANs (Karras et al. 2019 2020b) achieve
SOTA quality and support different levels of style control. Karras et al. (2021) solve the "texture sticking" problem of 2D GANs in generating animations with 2D transformations. Some meth-
ods (Harkönen et al., 2020 Tewari et al., 2020a Shen et al. 2020 Abdal et al., 2020 Tewari et al.) 2020b Leimkühler & Drettakis |021 Shoshan et al. 2021 leverage disentangled properties in the latent space to enable explicit controls, most of which focus on faces. While these methods can syn- thesize face poses parameterized by angles, extending them to general objects and controlling 3D cameras is not easy. [Cen et al. (2021a) proposed to generate segmentation maps from implicit fields enable 3D control. However, it requires 3D meshes for pre-training. In contrast, our work can synthesize images for general objects, enabling explicit 3D camera control.
3D-Aware GANs Recently, 3D representations have been integrated into 2D generative models to enable camera control. Voxel-based GANs (Henzler et al. 2019 Nguyen- Phuoc et al.) 2019 2020) lack fine details in the output due to resolution restriction. Radiance fields-based methods (Schwarz) ctt al. 2020; CCan et al.] 2021; Niemeyyrr & Geiger 2021a) achieve higher quality but have dif- ficulties in training on high-resolution images (512 and beyond) due to the expensive rendering process. GIRAFFE (Niemeyer & Geiger 2021b) improves the training and rendering efficiency by combining NeRF with CNN-based renderer; GSN VDies et al. (02T) models a locally con- ditional NeRF with a similar renderer for unconstrained indoor scene generation. However, they both produce severe view-inconsistent artifacts due to their network designs (e.g., 3 x Conv and upsampler). In contrast, our method can effectively preserve view consistency in image synthesis.
3 METHOD
3.1 IMAGE SYNTHESIS AS NEURAL IMPLICIT FIELD RENDERING
Style- Generative Neural Radiance Field We start by modeling a 3D scene as neural ra- diance field (NeRF, Mildenhall al. 2020). It is typically parameterized as multilayer perceptrons
2

---

Published as a conference paper at ICLR 2022
(MLPs), which takes the position E R and viewing direction d S as input, and predicts the density (xx) E R+ and view dependent color c(æ, d) E R To model high-frequency details, follwing NeRF (Mildenhall et al. 2020), map each dimension of and d with Fourier features:
(x)x) [sin (2n x), cos( ......, sin(2"_1z)) cos cos((2=11))
We formalize StyleNeRF representations by conditioning NeRF with style vectors w follows: 6 (x)
Fer Fw gw- o...o gu, O (æ). where w f
Similar as StyleGAN2 (Krrrs et al. 2020b), J is a mapping network that maps noise vectors from the spherical Gaussian space 2 to the style space W: (-) is the 7th layer MLP whose weight matrix is modulated by the input style vector w. @ (x) is the n-th layer feature of that point. We then use the extracted features to predict the density and color, respectively:
ow ho 2 (x); Cw d) he 1 ( (d)).
(3)
where h and he can be a linear projection or 2-layer MLPs. Different from the original NeRF, we assume ne > no for Equation (3) as the visual appearance generally needs more capacity to model than the geometry. The first min no. layers are shared in the network.
Volume Rendering Image synthesis is modeled as volume rendering from given camera pose P E P. For simplicity, we assume a camera is located on the unit sphere pointing to the origin with fixed field of view (FOV). We sample the camera's 's pitch & yaw from uniform Gaussian distribution. To render an image E R #x we shoot a camera ray r(t) = o + td (o is the camera origin) for each pixel, and then calculate the color using the volume rendering equation:
|NeRF (r) = / Pw (t) d)dt, where (w) cxp (- (w(((t)) (4)
In practice, the above equation is discretized by accumulating sampled points along the ray. Fol- lowing NeRF (Mildenhall et al. 20000), stratified and hierarchical sampling are applied for more accurate discrete approximation the continuous implicit function.
Challenges Compared 2D generative models (c.g., StyleGANs (Karras et al. 2019 2020b
the images generated by NeRF-based models have 3D consistency, which is guaranteed by mod- eling the image synthesis as a physics process, and the neural 3D scene representation is invariant across different viewpoints. However, the drawbacks are apparent: these models cost much more computation to render an image at the exact resolution. For example, 2D GANs are 100 ~ 1000 times more efficient to generate 10242 image than NeRF-based models. Furthermore, NeRF con- sumes much more memory to cache the intermediate results for gradient back pock-propagttion during training. making it difficult to train on high-resolution images. Both of these restrict the scope of applying NeRF-based models in high-quality image synthesis, especially at the training stage when calculating the objective function over the whole image is crucial.
3.2 APPROXIMATION FOR REGHLRESOOUTION IMAGE GENERATION
In this section, we propose how to improve the efficiency of Style by taking inspiration from 2D GANs. We observe that the image generation of 2D GANs (e.g., StyleGANs) is fast due to two main reasons: (1) each pixel only takes single forward pass through the network; (2) image features generated progressively from coarse to fine, and the feature maps with higher resolutions typically have a smaller number of channels to save memory.
In SryleNeRF, the first point can be partially achieved by early aggregating the features into the 2D space before the final colors computed. In this way, each pixel is assigned with feature vector, Furthermore, it only needs to pass through a network once rather than calling the network multiple times for all sampled points the ray as NeRF does. We approximate Equation as:
I Approx (r) = 100 (1) he 1 (r(1)), (d)) di ~ he o " (A(r)),<(d)),
(5)
where L (A(r)) It Iit The definitions of A() and (.) can be extended the operations set of rays, each ray
Iu +1 o A(r) and A(r) So Pw (t) Φ (rt
3

---

Published as a conference paper at ICLR 2022
LIF
Bi--lneea
Proposed
Figure 2: Internal representations and the outputs from StyleNeRR trained with different upsampling operators. With LII,, patterns stick to the same pixel coordinates when the viewpoint changes; With bilinear interpolation. bubble-shape artifacts can be seen on the feature maps and images. Our proposed upsampler faithfully preserves 3D consistency while getting rid of bubble-shape artifacts.
processed independently. Next, instead of using volume rendering to render a high-resolution feature map directly, we can employ NeRF generate downsampled feature map a low resolution and then employ upsampling in 2D space to progressively increase into the required high resolution. We take two adjacent resolutions as an example. Suppose R2 E TR. H/2xW/222 and RH E TR. x are the corresponding rays of the pixels in the low- and high-resolution images, respectively. To approximate the high-resolution feature map, we can up-sample in the low-resolution feature space: (6)
th (A(RH)) Upsample (ø" (A(Rt)))
Recursively inserting Upsample e operators enables efficient high-resolution image synthesis as the computationally expensive volume rendering only needs to generate low-resolution feature map. The efficiency is further improved when using fewer channels for higher resolution.
While early aggregation and upsampling operations can accelerate the rendering process for high- resolution image synthesis, they would come with scarification to the inherent consistency of NeRF. There are two reasons why they introduce inconsistency. First, the resulting model contains non- linear transformations to capture spurious correlations in 2D observation, mainly when substantial ambiguity exists. For example, our training data are unstructured single-view images without suffi- cient multi-view supervision. Second, such a pixel-space operation like up-sampling would compro- mise 3I) consistency. Therefore, naïve model designs would lead to severe multi-view inconsistent outputs (c.g., when moving the camera to render images, hairs are constantly changing). In the following, we propose several designs and choices to alleviate the inconsistency in the outputs.
3.3 PRESERVING 3D CONSISTENCY
Upsampler design Up-sampling in 2D space causes multi-view inconsistency in general; how- ever, the specific design choice of the upsampler determines how much such inconsistency is in- troduced. As our model is directly derived from NeRF, MLP (1 x 1 Conv) is the basic building block. With MLPs, however, pixel-wise learnable upsamplers such as pixelshuffle (Shi et al. 2016)
or LIIF (Chen et al. 2021b) produce "chessboard" or "texture sticking" artifacts due to its tendency of relying on the image coordinates implicitly. the meanwhile, Karras et al. (2199 2020b 2021
proposed to use non-learnable upsamplers that interpolate the feature map with pre- pre-ee low- pass filters (e-g. bilinear interpolation). While these upsamplers can produce smoother outputs, we observed non-removable "bubble" artifacts in both the feature maps and output images. We conjec- ture it is due to the lack of local variations when combining MLPs with fixed low-pass filters. We achieve the balance between consistency and image quality by combining these two approaches (sec Figure 2). For any input feature map X E x
Upsampled e(X) Conv2d (P i sixxexehee ssuuffllff fle (Repeat(X, (x),)) K),
4

---

Published as a conference paper at ICLR 2022
Camerap
Camerap
Sole oood
Affina
Lotenta E 2
Pounter
Latenta 2
Ponsler
Poutler
Mod
Po piod
IKEX
Demood
Mapping
Stapping
PG
Stanand
PG
11
breat ipongee
Marbbe
corples
Apprepation
B.IKK
Moal
Kig
Marber
BILK
(t")
BLK
BL.K
Demoad
Siput
BLKK (T))
Upeampple
RGB
BIIK
Styles
styles
Aggiegation
RGB
(optional)
AN KAN
Figure 3: Example architecture. (Left) Original NeRF path; (right) Main path of StyleNeRF.
where tig RD RR1 is learnable 2-layer MIP, and K is fixed blur kernel Zhang 2019
NeRF path regularization We propose a new regularization term to enforce 3D consistency, which regularizes the model output to match the original path (Equation 4.. In this way, the final outputs can be closer the NeRF results, which have multi-view consistency. This implemented by sub-sampling pixels on the output and comparing them against those generated by NeRF:
[S] Σ E
(s))S (I
(IApprux (R.)) |NeRF (Roon (t.))
where S is the set of randomly sampled pixels; Rin and Row are the corresponding rays of the pixels in the low-resolution image generated via NeRF and high-resolution output of StyleNeRF.
Predicting colors with view direction condition was suggested by the original NeRF for modeling view-dependent effects and was by default applied most follow-up works (Chan et al. 2021 Niemeyer & Geiger 2021b). However, this design would give the model additional freedom to capture spurious correlations and dataset bias, especially only a single-view target is provided. We, therefore, remove the view direction in color prediction, which improves the synthesis consistency (See Figure g
Remove view direction condition
Fix 2D noise injection Existing studies (Karras et al. 2019 2020b Feng et al. 2020) have showed that injecting per-pixel noise increase the model's capability of modeling stochastic variation (e.g. hairs, stubble). Nevertheless, such 2D noise only exists in the image plane, which will not change in 3D consistent way when the camera moves. To preserve 3D) consistency, solution is to trade the model's capability of capturing variation by removing the noise injection as Karras et al. al. (202)) does. Optionally, we also propose novel geometry-awat noise injection based on the estimated surface from StyleNeRF. See Appendix A.4 for more details.
default
3.4 STYLENERF ARCHITECTURE
In this section, we describe the network architecture and the learning procedure of StyleNeRF.
Mapping Network Following StyleGAN2, latent codes are sampled from standard Gaussian and processed by mapping network. Finally, the output vectors are broadcast to the synthesis networks.
Synthesis Network Considering that our training images generally have unbounded background, we choose NeRF++ (Zhang et al. 2020), a variant of NeRF, as the StyleNeRF backbone. NeRF++ consists of foreground NeRF in unit sphere and background NeRF represented with inverted
5

---

Published as a conference paper at ICLR 2022
AFRI
FTHQ
Comporrr
HoloGAN
GRAF
piGAN
GIRAFFE
Ours
Figure 4: Qualitative comparisons at 256 StyleNeRR achieves the best quality and 3D consistency.
sphere parameterization. As shown in Figure two MLPs are used to predict the density where the background network has fewer parameters than the foreground one. Then shared MLP is employed for color prediction. Each style-conditioned block consists of an affine transformation layer and convolution layer (Conv). The Conv weights are modulated with the affne-transformed styles, and then demodulated for computation. leaky ReLU is used non-linear activation. The number blocks depends on the input and target image resolutions.
Discriminator & Objectives We use the same discriminator as StyleGAN2. Following previous works (Chan et al. 2021 Niemeyer & Geiger 2021b), StyleNeRF adopts non-saturating GAN ob- jective with regularization (Mescheder et al. to enforce 3D consistency. The final loss function is defined as follows (D is the discriminator and is the generator including the mapping and synthesis networks):
| (2018). new NeRF path regularization is employed
C(D, G) = E3~zZ (f(D(G(z, + Ei~pan ((T-((T) + A VIVD + LNeRF-path (9)
where f(u) = - log(1 + exp(-u)), and Pdata is the data distribution. We set B = 0.2 and = 0.5.
Progressive Training We train StyleNeRF progressively from low high resolution, which makes the training more stable and efficient. We observed in the experiments that were directly training for the highest resolution easily makes the model fail capture the object geometry. We suspect is because both Equations (5) and (6) are just approximations the original NeRF. Therefore, inspired by Karras et al. (017). we propose new three-stage progressive training strategy:
first images, we train StyleNeRF without approximation at low-resolution; then, during T; TV images, both the generator and discriminator linearly increase the output resolutions until reaching the target resolution; At last, fix the architecture and continue training the model the highest resolution until T3 images. Please refer to the Appendix A.5 for more details.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets We evaluate StyleNeRF four high-resolution unstructured real datasets: FFHQ Kar- ras et al. 2019), MetFaces (Karras et al. 2020a). AFHQ (Choi et al.) 2020) and CompCars (Yagg al. 2015). The dataset details are described in Appendix

---

Published as a conference paper at ICLR 2022
Baselines We compare approach with voxel-based method, HoloGAN ( Nguyen-Phuoc Phuoc al. 2019), and three radiance field- based methods: GRAF (Schwarz et al., 2020 T GAN (Chan et al,
2021 and GIRAFFE (Niemeyer & Geiger 2021b). As most of the baselines
restricted to low
resolutions, we made the comparison at 250 pixels for fairness. We also report the results of the
state-of-the-art 2D GAN (Karras al. 2020b) for reference. See more details in Appendix C
Configurations All datasets except MetFaces are trained progressively. However, the MetFaces dataset is too small to train stably, so we finetune from the pretrained model for FFHQ at the highest resolution. By default, we train 64 images per batch, and set Ti 500k, T2 5000k and T3 25000k images, respectively. The input resolution is fixed 32 for all experiments.
4.2 RESULTS
Qualitative comparison We evaluate approach and the baselines three datasets: FFHQ, AFHQ, and CompCars. Target images resized to 256 We render each object a sequence and sam- ple four viewpoints shown counterclock- wise in Figure 4 While all baselines can generate images under direct camera control, HoloGAN, GARF, and T-GAN fail to learn geometry correctly and thus produce severe artifacts. GIRAFFE syn- thesizes images in better quality; how- it produces 3D inconsistent arti- facts: the shape and appearance in the output change constantly when the era moves. We believe it due to the wrong choice of 3 3 Conv layers. Compared to the baselines, StyleNeRF achieves the best visual quality with high 3D consistency across views. Please see more results in the supplemental video.
Quantitative comparison We measure the visual quality of image generation by the Frechet Inception Distance (FID,
Heusel et al.] 2017) and Kernal Inception Distance (KID, Binkkowsii et al. 2018)) in Table M Across all three datasets, StyleNeRF consistently outperforms the baselines by significant gains in terms of FID and KID and largely reduces the performance gap between the 3D-aware GANs and the SOTA 2D GAN (i.e., StyleGAN2). Note that while 2D GANs achieve high quality for each image, they cannot synthesize images of the same consistency. Table [ also shows the speed comparison over different image resolutions on FFHQ. StyleNeRF enables rendering interactive rates and achieves significant speed-up over voxel and pure NeRF-based methods, and is comparable to GIRAFFE and StyleGAN2. In addition, we include comparison on the consistency of synthesis quantitatively Appendix [
Figure 5: Uncurated set of images at 512² produced by StyleNeRF from three datasets. Each example is ren- dered from randomly sampled camera.
with 3D
High-resolution synthesis Unlike the baseline models, our method can generate high-resolution images (5122 and beyond). Figure shows uncurated set of images rendered by StyleNeRF. We also show more results of StyleNeRE and report the quantitative results of the SOTA 2D GAN (StyleGAN2 (Karras et al. 2020b)) for reference in the Appendix D
4.3 CONTROLLABLE IMAGE SYNTHESIS
Explicit camera control Our method can synthesize novel views with direct camera control and generalize extreme camera poses, which starkly differs from the training camera pose distribution. Figure [ shows our results with extreme camera poses, such as zoom-in and -out, and steep view angles. The rendered images maintain good consistency given different camera poses.
7

---

Published as a conference paper at ICLR 2022
Table Quantitative comparisons at 2562. We calculate FID, KIDx103, and the average rendering time (batch size 1). The 2D GAN (StyleGAN2 (Karras al. 2020b results are for reference.
FFHQ 256
AFHQ 256"
CompCars 256²
Rendering time (ms image)
Models
FID
KID
FID
KID
FID
KID
64
128
256
512
1024
2D GAN
1.6
51
53
HoloGAN
75
68.0
78
59.4
48
39.6
213
215
222
GRAF
57.2
121
83.8
86.7
61
246
990
3852
15475
x-GAN
90.0
47
29.3
295
328.9
58
198
766
3063
12310
GIRAFFE
35
23.7
31
13.9
32
23.8
8
9
Ours
3.7
3.5
4.3
74
98
Figure 6: Images synthesized from camera poses which starkly differ from training camera poses.
Style mixing and interpolation Figure 7 shows the results of style mixing and interpolation. As shown in style mixing experiments, copying styles before 2D aggregation affects geometry aspects (shape of noses, glasses, etc.), while copying those after 2D aggregation brings changes in appear- ance (colors of skins, eyes, hairs, etc.), which indicates clear disentangled styles of geometry and appearance. In the style interpolation results, the smooth interpolation between two different styles without visual artifacts further demonstrates that the style space is semantically learned.
Style inversion and editing We can also do inverse rendering tasks with a learned StyleNeRF model. We first pre-train a camera pose predictor in self- supervised manner. Then given an input image, we use this predictor estimate the camera pose and op-
Neponnssseee raining
NoB 2ith repolactaatin
timize the styles via back propagation to find the best styles that match the target image. We can further perform semantic editing. For instance, we can opti- mize the styles according to text snippet using CLIP loss (Radford et al. [2021] Patashnik et al.) (2021). An example is shown Figure
Traine with WIP Biiatinn
6
Eusebduch Inn dantiyy
4.4 ABLATION STUDIES
Figure 8 (a) shows typical result without progressive training. Despite not directly affecting the quality of each single image, the model fails to learn the correct shape, e.g. the face should be in convex shape but is predicted as concave surfaces. This leads to severe 3D inconsistent artifacts when the camera moves (see the red eircles in Figure (a))
Figure 8: Failure results.
As shown in Figure 8 (b), without NeRF-path regularization, the model sometimes gets stuck learn- ing geometry 3D and produces a "flat shape" output.
Figure (c) demonstrates example of the result with view direction as a condition. The modeling of view-dependent effects introduces an ambiguity between 3D shape and radiance and thus leads to degenerate solution without multi-view consistency (see the red circles in Figure 8 (c)).
4.5 LIMITATIONS AND FUTURE WORK
While StyleNeRF can efficiently synthesize photo-realistic high-resolution images in high multi- view consistency with explicit camera control, it sacrifices some properties that pure NeRs-b
8

---

Published as a conference paper at ICLR 2022
Soutce
Sourer
batoro 21 Aggrogition
Copy styles from
Copy spyles from after ZD sagngitton
Style intrpalation with Canes moving
Style inversion and
ediing
Target Image
Estimating the camera
GAN Inversion
Rotating the camera
Editing the sryles with CLTP
Figure 7: Style mixing (Top): images were generated by copying the specified styles from source B to source A. All images rendered from the same camera pose. Style interpolation (Middle): lin- early interpolate two sets of style vectors (leftmost and rightmost images) while rotating the camera. Style inversion and editing (Below): the target image is selected from the DFDC dataset (Dolhansky et al. 2019). To edit with CLIP scores, we input "a person with green hair" as the target text.
methods have. An example is shown in Figure g (d). where extract the underlying geometry with marching cube from the learned density. Although StyleNeRR is able to recover a coarse geometry, it captures less details compared to the pure NeRF-based models such 7-GAN (Chan jet al.] (2021)). Additionally, StyleNeRF only empirically preserves multi-view consistency, but does not guarantee strict 3D consistency. Further exploration and theoretical analysis needed.
CONCLUSION
We proposed a 3D-aware generative model, StyleNeRF, for efficient high-resolution image gen- eration with high 3D consistency, which allows control over explicit 3D camera poses and style attributes. Our experiments have demonstrated that StyleNeRF can synthesize photo-realistic 1024 images interactive rates and outperforms previous 3w-a generative methods.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
Our work focuses on technical development, i.e., synthesizing high-quality images with user control. Our approach can be used for movie post-production, gaming, helping artists reduce workload, generating synthetic data to develop machine learning techniques, etc. Note that our approach is not biased towards any specific gender, race, region, social class. It works equally well irrespective of the difference in subjects.
However, the ability of generative models, including our approach, to synthesize images a quality that some might find difficult to differentiate from source images raises essential concerns about dif- ferent forms of disinformation, such as generating fake images or videos. Therefore, we believe the image synthesized using our approach must present itself as synthetic. We also believe it is essential to develop appropriate privacy-preserving techniques and large-scale authenticity assessment, such as fingerprinting, forensics, and other verification techniques to identify synthesized images, Such safeguarding measures would reduce the potential for misuse.
We also hope that the high-quality images produced by our approach could foster the development of the forgery mentioned above detection and verification systems. Finally, we believe that a robust public conversation is essential to creating set of appropriate regulations and laws that would alleviate the risks of misusing these techniques while promoting their positive effects on technology development.
REPRODUCIBILIIT STATEMENT
We assure that all the results shown in the paper and supplemental materials can be reproduced. Furthermore, we will open-source our code together with pre-trained checkpoints. To reproduce our results, provide the preprocessing procedures of each dataset and implementation details in the main paper and Appendix.
ACKNOWLEDGEMENTS
Christian Theobalt was supported by ERC Consolidator Grant 4DReply (770784). Lingjie Liu was supported Lise Meitner Postdoctoral Fellowship.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit the embedded images? In CVPR, pp. 8296-8305, 2020.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mndd gans. arXiv preprint arXiv: 00780011111,,, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. CoRR, abs/1809. 1096, 2018. URL http:://axiv.org//
18C9.11099
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 5799-5809, 2021.
Anpei Chen, Ruiyang Liu, Ling Xie, Zhang Chen, Hao Su, and Jingyi Yu. Sofgan: portrait image generator with dynamic styling. ACM transactions graphics, 2021a.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2021b.
Yunjey Choi, Youngjung Uh. Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3188-8197, 2020.
10

---

Published as a conference paper at ICLR 2022
Terrance Deies, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind. Unconstrained scene generation with locally conditioned radiance fields. arXiv, 2021.
Thang Doan, J. Monteiro, Isabela Albuquerque, Bogdan Mazoure, A. Durand, Joelle Pineau, and R. Devon Hjelm. Online adaptative curriculum learning for gans. In AAAI, 2019.
Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, and Cristian Canton Ferrer. The deep- fake detection challenge (dfdc) preview dataset. arXiv preprint VtXiv:/900.08880 2019.
Ishan Durugkar, lan Gemp, and Sridhar Mahadevan. Generative multi-adversarial networks, 2017.
Ruili Feng, Deli Zhao, and Zhengjun Zha. On noise injection in generative adversarial networks. arXiv preprint (xXiv:2006.0589) 2020.
lan J. Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial
nets. In NIPS, pp. 2672-2680, 2014. URL http:/ http://papera. wwps. cttppperrcpppr -nts
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: interpretable gan controls. In Proc. NeurIPS, 2020.
Discovering
Philipp Henzler, Niloy Mitra, and Tobias Ritschel. Escaping plato's cave: 3d shape from adver- sarial rendering. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. Proceedings of the IEEE International Conference on Computer Vision, pp. 1501- 1510, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018. URL https: / http:///pppwwepwwp//t//p net htterrrwwwweeeetttoooraaaaa----- www.ouummmiiHh999900
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06676 2020a.
Tero Karras, Samuli Laine, Miia Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 10 8119, 2020b.
Tero Karras, Miik Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. arXiv preprint arXiv:2106. /2423, 2021.
Thomas Leimkühher and George Drettakis. Freestylegan: Free-view editable portrait rendering with the camera manifold. 40(6), 2021. doi: 10.1145/3478513.48883
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. NeurlPS, 2020.
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann. and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions Graphics (TOG), 38(4):65, 2019.
11

---

Published as a conference paper at ICLR 2022
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning. pp. 1491-3490. PMLR, 2018.
Lars Mescheder, Michael Oechsle. Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings IEEE Conf. Computer Vision and Pattemn Recognition (CVPR), 2019.
Mateusz Michalkiewicz, Jhony K. Pontes, Dominic Jack, Mahsa Baktashmottagh, and Anders Eriks- son. Implicit surface representations as layers in neural networks. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv preprint N.XV:20033.8934.. 2020.
Gonçalo Mordido, Haojin Yang. and C. Meinel. Dropout-gan, Learning from a dynamic ensemble of discriminators. ArXiv, abs/1 1346, 2018.
Thu Nguyen-Phuoe, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Holo- gan: Unsupervised learning of 3d representations from natural images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7588-7591 2019.
Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra. Block- gan: Learning 3d object-aware scene representations from unlabelled images. arXiv preprint arXiv:2002.00 08988, 2020.
Michael Niemeyer and Andreas Geiger. Campari: Camera-aware decomposed generative neural radiance fields. In Proc. of the International Conf. on 3D Vision (3DV), 2021a.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 11453-11464 2021b.
Michael Niemeyer, Lars Mescheder, Michael Occhsle, and Andreas Geiger. Differentiable volu- metric rendering: Learning implicit 3d representations without 3d supervision. arXiv preprint 1rXi0: 2019.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdff: Learning continuous signed distance functions for shape representation. International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. arXiv preprint arXiv:2 2021.
Songyou Peng, Michael Niemeyer, Lars M. Mescheder, Mare Pollefeys, and Andreas Geiger. Con- volutional occupancy networks. ArXiv, 0bs/2003.004138 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2 100.00020.00..0 2021.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure- ence on Computer Vision and Pattern Recognition (CVPR), 2016.
revisited, In Confer-
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. arXiv preprint wrXiv:2007.02442, 2020.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, pp. 9243-9252, 2020.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video suser-resolution using efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874-1883, 2016.
12

---

Published as a conference paper at ICLR 2022
Alon Shoshan, Naday Bhonker, Igor Kviatkovsky, and Gerard Medioni. controllable gans, 2021.
Gan-control: Explicitly
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-aware neural scene representations. In Advances in Neural Information Pro- cessing Systems, pp. 1119-1130, 2019.
Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj. Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zöllhofer, and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait images. In CVPR.june 2020a.
Ayush Tewari, Mohamed Elgharib, Mallikarjun BR, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zöllhofer, and Christian Theobalt. Pie: Portrait image embedding for semantic control. volume 39, December 2020b.
Qianqian Wang, Zhicheng Wang. Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo
Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi- view image-basse based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4690 4699, 2021.
Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang. A large-scale dataset for fine- grained categorization and verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3973-3981 2015.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. volume 97 of Proceedings of Machine Learning Research, pp. 7354-7363, Long Beach, California, USA, 09-15 Jun 2019. PMLR. |ttp:tt::
mlr.
press/ /0tes//99/// 21772hang1199 html
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
Richard Zhang. Making convolutional networks shift-invariant again. In International conference machine learning, pp. 7324-7334. PMLR, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
13

---

Published as a conference paper at ICLR 2022
A ADDITIONAL MODEL DETAILS
A.1 IMPLICIT FIELDS MODELING
Modeling backgournd with NeRF++ Most unstructured images used in this work have a complex and unbounded background, especially for the images taken from 360 degree directions (c.g., images in the CompCars dataset). Therefore, it is inefficient and difficult to model the entire embedded scene in the image within a fixed bounding box. Instead, following NeRF++ (Zhang et al.) 2020), partition the whole scene into foreground and background where the background is modeled with an additional network that takes as inputs the background latent codes and a 3D point x: = (2; y, 2) transformed using inverted sphere parameterization:
x' = (x/r, 1/Tr, where r r= v +
(10)
Then, we uniformly sample G background points in an inverse depth range: (1/R,0) 0) where R = 2.0 in our experiments represent where the background starts.
Hierarchical volume sampling Similar to NeRF (Mildenhall et al. 2020). we adopt an efficient hierarchical sampling strategy for the foreground network learning. We first uniformly sample N points between the near and far planes and then perform importance sampling of M points based on the estimated density distribution in the coarse sampling stage. Different from NeRF (Mildenhall et al. 2020), we use a single network for coarse and fine sampling to predict the color and density.
A.2 CAMERA
Camera setup We assume that each image is rendered by camera with fixed intrinsic matrix in a pose sampled from a pre-defined camera pose distribution. The intrinsics are determined by field- of-view (FOV) and image resolution, normalized to [-1, 1] regardless of the actual resolution. The camera is located on the unit sphere, pointing at the world origin where the target object is located. Then the camera pose p(0, 6) is the function of pitch (0) and yaw (Ø). Typically, the cameras in real data do not follow the simple distributions that we assumed, which is the main problem for all 31>-aware GANs (including the proposed model) learn the correct geometry. To tackle this issue, Niemeyer& & Geiger (2021a) proposed to learn a neural network to generate camera simultancously poses jointly with scene representations. However, our initial attempts of applying similar methods to StyleNeRF did not work, and the camera generator quickly diverged. Instead, we manually set the parameters @ and from either a Gaussian or uniform distribution depending on the datasets.
Camera predictor Once StyleNeRF is trained, we can optionally train ad- ditional camera predictor in self- supervised manner. More specifically, we first randomly sample camera pose and the latent codes to render the output im- age. Then, CNN-based encoder (back- bone initialized with a pretrained ResNet- 18) reads the image and predicts the input camera parameters based on a smoothed LI loss function. The camera predictor can be directly used on natural images, in particular for style inversion tasks (see Figure 7 our early exploration shows that it is non-trivial to train the camera pose from random through Tatent optimization.
Smazzhed
Sykhhett
Prescor
Figure 9: An illustration of self-supervised training the camera predictor.
A.3 DERIVATION OF APPROXIMATION
The following shows the detailed derivation of Equation (6) For simplicity, we omit the condition on style vectors in the equation.
14

---

Published as a conference paper at ICLR 2022
Per-pixe
Notse-
Geemetryy soorr Novee Inpecton
Figure 10: Example of sticking texture because of the injection of 2D noise at the test time. Note that the texture cropped inside the red box does not change across different views.
/ p(t) he (rt) dt
Apppox (r)
Er (he 10 ( (r(t),,(d)))
the E. (r())
hh ((())))))
th (
0
~ he f o E. (0 (r(t))),((d)
ha o
p(t) p" (r(t)) C(d)
ha o Ro (A(r)).((d)
In the above derivation, treat the volume rendering integral expectation. Therefore, if p(t) close to an impulse function which is near zero everywhere outside a small region near the object surface, locally he and &"c,"t can be seen as linear functions. Because the expectation of linear function in an interval is equal to the value of the function evaluated at the middle point of the interval, the accuracy of the proposed approximation is well justified under this assumption.
A.4 DETAILS ABOUT THE NOISE INJECTION
As mentioned in the paper, injecting per-pixel noise improves the generation quality of GANs. How- ever, naively adding such noise as in Karras et al. (2019) is feasible for our purpose because such noise exists in the pixel space, and it does not move when the viewpoint changes. On the other hand, the output gets noisy such inconsistency. the default setup in our experiments is to remove the noise injection as similarly done in (Krrras et al., (02T) which works reasonably but has less ability to model local stochastic variations, resulting in lower visual quality (in terms of metrics like FID, KID, etc).
inject random noise for each frame separately. To avoid
As an alternative, to improve synthesizing quality and keep multi-view consistency, we design a geometry-aware method for noise injection. Precisely, for each feature layer at the resolution of N2, we extract the underlying geometry using marching-cube based on the predicted density. Next, set the volume resolution 3 accordingly. Then, assign independent Gaussian noise each vertex of the extracted mesh and render the noise map via rasterization from the same viewpoint. Finally, we inject the resulting noise to each layer before non-linear activation. Since this process
15

---

Published as a conference paper at ICLR 2022
SE
Aggnokke
BLX
BL.
slust
(1) Move apglogiian
Glast
in BiB
5002
XD A.gagggaa
(b)
22632
Gla
Figure 11: Illustrations of the proposed variants of progressive training for StyleNeRF. (a) is for progressive growing of the generator; (b) is the same generator by progressively inserting upsamplers; (c) is the corresponding discriminator.
requires a proper density function to start with, we keep the standard per-pixel noise unchanged during training and only apply such geometry-awan noise inference time. Compared to models without noise injection, although the proposed approach can capture more stochastic details, it is relatively slower (due to multi-scale marching-cute and rasterization), and the quality of the learned geometry constrains the final output.
A.5 PROGRESSIVE TRAINING
As described in the main paper, it is essential to train StyleNeRF progressively from low to high resolutions learn good geometry. We consider two types of progressive training:
Progressive Growing Following Karras et al. (017)) we start from a shallow network for low- resolution, and progressively fade in layers. Since the parameters of the newly added layers are random, it essential to have a linear interpolation with low-resolution output to stabilize training.
Progressive Up-sampling Instead of growing new layers, another option is to train StyleNeRF by progressively inserting upsample le operations during training as similarly done in Chan et al. 2021. In this case, we do not need to fade in layers linearly.
In both cases, the discriminator correspondingly grows from low high resolutions progressively as described in Karras et al. (2017). Figure []] illustrates progressive training for the generator and the discriminator. In practice, we found that both methods worked similarly terms of visual quality, while progressive growing achieves a better speed advantage for low-resolution images. Therefore, our results are reported by training with progressive growing.
A.6 HYPERPARAMETERS
We reuse the same architecture and parameters of StyleGAN2 for the mapping network (8 fully connected layers, 100x lower learning rate) and discriminator. In addition, both the latent and style dimensions are
512. For the foreground and background fields, before predicting the density, we use the style-based MLPs with 256 and 128 hidden units, respectively. The Fourier feature dimension set to L 10 (Equation (T) for both fields. For layers after 2D aggregation, the flexible layer configuration used by StyleGAN2, which decreases the feature sizes from 512 (for 322) until 32 (for 10242). We keep most of the training configurations unchanged from StyleGAN2.
follow
A.7 IMPLEMENTATION
We implement our model based on the official Pytorch implementation of StyleGAN2-ADA | By default, we train the StyleNeRF model by going through 25000k images with a minibatch 64. All models are trained on 8 Tesla V100 GPUs for about three days.
16

---

Published as a conference paper at ICLR 2022
Table 2: (Left) FID, KID x x10 on images with 5122 and 1024 pixels. 2D GAN represents the StyleGAN2 (Karras al. 2020b). (Right) We also compare the results of replacing 3 x 3 Conv with 1 and/or removing per-pixel noise injection for 2D models. We use bold font to represent the default setting for both model.
FFHQ 5122
AFHQ 512²
MetFace 512?
FFHQ 1024
1
Noise
2D GAN
Ours
Models
FID
KID
FID
KID
FID
KID
FID
KID
Yes
11.4
7.8
2D GAN
3.1
0.7
8.6
1.7
18.9
2.7
2.7
0.5
Yes
Yes
5,9
6.0
Ours
7.8
2.2
13.2
3.6
20.4
3.3
8.1
2.4
No
No.
4.6
B DATASET DETAILS
FFHQ hnttps://gite http://///////fitiit/.oa aag.a///gfhgggassssiii contains 70k images of real human faces in resolution of 1024'. We assume the human face to be captured at the origin. In the training stage, we sample the pitch and yaw of the camera from the Gaussian distribution.
AFHQ (Attps:
com hwtp///oooaaaaaaaaeeaeeeee v2#
contains in total 15k images of animal faces in- cluding cat, dog and wild three categories in resolution of 512². We directly merge all training images without using the label information. Similar to FFHQ setting, we sample the pitch and yaw from the Gaussian distribution.
MetFaces
contains 1336 images
of faces extracted from artworks. The original resolution is 1024 and we resize it to 512" for our main experiments. As mentioned in the paper, we finetune StyleNeRF from the FFHQ checkpoint due to the small size of MetFaces.
CompCars
ie. cuhk. edu, cct.ceeeeee..//aaaattttt/cctoopp
contains
136726 images capturing the entire cars with different styles. The original dataset contains images with different aspect ratios. We preprocess the dataset by center cropping and resizing them into 2562. In the training stage, sample the camera 360 degree uniformly.
C BASELINE DETAILS
al. 2019) We train HoloGAN on top of the official implementation Specifically. HoloGAN is trained in resolution 256 with Adam optimizer with an initial learning rate of 5e-5. We train each HoloGAN model for epochs. The learning rate linearly decays between epoch 25 and 50. As original HoloGAN only supports training images of resolution 642 and 1282, synthesize images of resolution 2562 we follow the same adaptation scheme as used in GIRAFFE (Niemeyer & Geiger 2021b) that add one convolution layer with AdaIn (Huang & Belongie 2017) and leaky ReLU activation on top the official synthesis network.
HoloGAN Nguyen-Phuun
GRAF (Schwarz al., 2020) We adopt the official implementation 3 and retrain GRAF models on FFHQ, AFHQ and preprocessed CompCars dataset in resolution 256 Following the default setting, all models are trained at the target resolution using the patch- basch discriminator.
T-GA (Chan et al. 2021) Same as GRAF, we use the official implementation and retrain TAAN models these three datasets: FFHQ, AFHQ and CompCars. Due to the high cost of running --GAN in high-resolution, we follow the same pipeline, which progressively increases the resolution from 322 until 128?, and render the final outputs in 256 by sampling more pixels.
GIRAFFE Niemeyer & Geiger 2021b) For FFHQ dataset, directly their pretrained mod- els of resolution 250 For AFHQ and CompCars dataset, we retrain GIRAFFE with the official Gi- raffe implementation [] We change the default random crop preprocessing to center crop compare with other approaches.
17

---

Published as a conference paper at ICLR 2022
Table 3: PSNR, SSIM and LPIPS scores between the images synthesized by 3D-aware generative models, and the corresponding pseudo targets generated by IBRNet (Wang et al. 202I.
Models
FFHQ 256
AFHQ 256
PSNR f
SSIM 1
LPIPS +
PSNR
SSIM
LPIPS +
T-GAN
29.5
0.92
0.04
28.5
0.86
0.12
GIRAFFE
25.8
0.81
0.13
26.2
0.75
0.25
Ours
29.0
0.89
0.08
26.8
0.80
0.13
Table 4: FID on the outputs from StyleNeRF trained with different upsampling operators.
Metric
Bilinear
LIIF
PixelShuffle w/o blur
Proposed
FID
12.4
10.6
32.4
7.8
D ADDITIONAL RESULTS
Comparison to StyleGAN2 We also provide quantitative comparisons on high-resolution image synthesis against StyleGAN2 and variant that adopts comparable settings with our default model. As shown in Table there is still gap to the best 2D-GAN models. Nevertheless, the proposed StyleNeRF reaches similar or even better results as 2D models in similar architectures. This implies the quality drops are mainly from less powerful architecture (e.g., noise injection, no 3 convolutions). As future work, more exploration can done to close this gap.
Quantitative comparison on 3D Consistency We conduct quantitative comparisons to measure the multi-view consistency of the synthesized images from StyleNeRI and baselines (7-GAN, GIRAFFE). We do not employ the metric used in GRAF (Schwarz et al. 2020) because the caleu- lation of that metric requires 3D ground-truth geometry, and thus it can only be applied synthetic data, not real data where the ground-trute geometry is not available.
Instead, propose new method measure multi-view consistency without the requirement of 3D shapes. First for each model, we randomly sample 1000 seeds, and for each seed, we generate sequence of 9 images following the same trajectory. Next, we take four images as conditioning, and predict pseudo ground truth images given the remaining five camera poses using pre- pre-trained IBRNet (Wang et al., 2021). T), recently proposed image-based rendering method for generic view interpolation and can generalize to novel scenes. Then we measure the typical image reconstruction scores (PSNR, SSIM, and LPIPS (Zhang et al. 2018)) between the pseudo targets and the remaining images. Since the IBRNet model is fixed for all the methods, the reconstruction scores can be used as a metric to measure the multi-view consistency of different methods (the better the reconstruction scores are, the higher consistency the synthesized images are).
We tested on the FFHQ 256 and AFHQ 256² and reported the results in Table StyleNeRF achieves much better consistency than GIRAFFE, while is comparable to r-GAN, the SoTA pure NeRF-hased model, in terms of multi-view consistency. Meanwhile, StyleNeRF can generate high- resolution images efficiently, which cannot be achieved by existing pure NeRF-based models.
3D Reconstruction We also include the COLMAP (Schönberger Frahm 2016) reconstruction example of the proposed StyleNeRF on FFHQ to validate the 3D consistency of the model's 's output. As shown in Figure [12] given a style, we sample 36 camera poses generate images and obtain the reconstructed point clouds from COLMAP with default parameters and no known camera poses.
Ablation upsamplers test test different upsampling operators quantitatively on FFHQ 5122 and report the FID scores in Table We also include the additional result of using pixelshuffle without applying the blur kernel as described in Equation (6). The FID results show that the proposed upsampler achieves the best quality compared to other operators.
Additional Visual Results We show additional results of generating images with different camera views as follows. Please also refer to the supplemental video, which shows more results.
18

---

Published as a conference paper at ICLR 2022
Guaenied images StyleNeR with
differont comeras
Reconstacted clouds with COLMAP
Figure 12: COLMAP reconstructions for models trained on FFHQ 512?
Figure 13: Example on FFHQ 1 displayed from different viewing angles.
19

---

Published as a conference paper at ICLR 2022
Figure 14: Example on AFHQ (512?), MetFaces (512?) and CompCars (2562) displayed from dif- ferent viewing angles.
20

---

Published as a conference paper at ICLR 2022
Figure 15: High resolution samples with explicit camera control on FFHQ 1024
21

---

Published as a conference paper at ICLR 2022
i
E
Figure 16: Random samples with random camera poses FFHQ 5122.
22

---

Published as a conference paper at ICLR 2022
Figure 17: Random samples with random camera poses on MetFaces 5122.
23

---

Published as a conference paper at ICLR 2022
Figure 18: Random samples with random camera poses on AFHQ 5122
24

---

Published as a conference paper at ICLR 2022
Figure 19: Random samples with random camera poses on CompCars 2562
25


Reviews:
[{"review_id": "y_A13k7GINt", "reviewer": "Reviewer_Qm5b", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n(1) The proposed StyleNerF Model can synthesize high resolution, photo-realistic images with strong multi-view consistency, while achieving interactive rendering rate. Furthermore, it allows controls of camera poses and style attributes. \n(2) To tackle the problem of efficiency, authors first use NeRF to volume render low resolution feature maps, and then feed into stylegan 2D synthesis blocks to synthesize high resolution images. To avoid multi-view inconsistencies, authors proposed to use NeRF path regularization along with other design choices.\n(3) Extensive experiments on several benchmarks as well as ablation study validate the effectiveness of proposed component and demonstrate state of the art results on 3D generative view synthesis.\n\nWeakness:\n(1) The proposed method remove viewing direction from the NeRF to avoid ambiguity learned from single images, which is fine, but strictly speaking it’s no longer a “radiance field”, but instead a “irradiance fields”.\n(2) In Figure 3, authors need to add clarification for each notation (such as FG,BG, BLK) to make reader easier for readers to understand.\n(3) From the demo video, I saw a “gaze following” phenomena, where the eye looking direction always follows the camera path, is there any explanation and way to fix this problem?\n(4) I am curious if the proposed method can also synthesize 3D scenes from unstructured photos. For example, StyleGAN is able to train on photos of nature landscapes to generate beautiful 2D images. I am wondering what would happen if the proposed model was train on such data.\n", "summary_of_the_paper": "In this manuscript, authors proposed a novel 3D-aware generative model for photo-realistic high resolution image synthesis. The proposed method combines Neural Radiance Fields (NerF) and StyleGAN, and tackles the challenges of efficiency, multi-view consistency and rendering quality. Specifically, authors propose to use NeRF to render low resolution images before feeding them into 2D StyleGAN network to upsample the feature maps in order to obtain images with high resolution and therefore the proposed model can render images at interactive rates. Extensive experiments on numerous datasets demonstrate superior performance than prior state of art 3D generative view synthesis approaches and largely close the gap between 2D and 3D view synthesis methods.", "summary_of_the_review": "Based on the strength and weakness I mentioned above, I strongly recommend this paper to be accepted because its novelty on closing the gap between 3D GAN and 2D GAN problem and this is the first 3D generative view synthesis work that can render high resolution images with interactive speed while preserving very high rendering quality. Its proposed component can be very useful to push the boundaries of this area in the future.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "yWgtEovaKCn", "reviewer": "Reviewer_TLAR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The proposed method is able to synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality. \n+ It enables control of camera poses and different levels of styles. \n+ It also supports new tasks such as style mixing, inversion, and simple semantic edits.\n\nWeaknesses:\n- The paper lacks novelty. The method is mostly a simple combination of Pi-GAN which uses a StyleGAN-alike MLP for NeRF, and GIRAFFE which uses volume rendering to generate low-resolution features that are sequentially upsampled by CNN. Thus, the contribution of this work seems not particularly strong. \n- The approximation relationship in Eq. 5 is not quite obvious. Please provide more detailed proof and explanations for it.\n- In Figure 2, it is not clear how the internal representations are visualized? What do different colors mean? In addition, the mentioned artifacts can hardly be noticed in the final image output. \n- The upsampler design is not quite clear. It is hard to understand how such a design solves the “chessboard”, “texture sticking” and \"bubble\" artifacts.\n- What is $\\varphi_\\theta$ in Eq. 7?\n- For evaluation, in Figure 4, the results of the baselines, e.g pi-GAN, are surprisingly bad. This is not consistent with the results in the original paper. This requires some explanations.\n- In Figure 3, what is $K_{fg}$ and $K_{bg}$, and how to obtain them?", "summary_of_the_paper": "The paper presented StyleNeRF, a 3D-aware generative model for high-resolution image synthesis with high multi-view consistency. \nStyleNeRF integrates the neural radiance field (NeRF) into a style-based generator to improve rendering efficiency and 3D consistency. \nIt performs volume rendering only to produce a low-resolution feature map, and progressively applies upsampling in 2D.\nIt also presents a new upsampling module and a new regularization loss to enforce 3D consistency. \n", "summary_of_the_review": "This is a borderline paper.  It achieves very interesting results. However, it has quite a few issues in terms of novelty and evaluation, and many algorithm designs and technical details are not well explained.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "suz3Mu6Nkl6", "reviewer": "Reviewer_gKfo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "With the great success that has been achieved by StyleGAN in 2D, it is natural to extend the key idea behind StyleGAN to 3D for a 3D-aware generative model. It is clearly shown in the paper that this work has strengths including:\n1. a very good empirical effort to extend StyleGAN to 3D.\n2. the proposed method produces SOTA results, the generated images are impressive.\n3. the proposed method is fast and supports high-resolution renderings.\n\nMeanwhile, there are also several weaknesses:\n1. lack of technical novelty, the proposed combines several existing techniques that are well explored in the GAN field.\n2. the key contribution that makes this work conceptually different may be unclear. While it is true that all the tricks as a whole deliver impressive results, it is not easy to identify the unique contribution in this work.\n3. 3D-aware generative models have a focus on multi-view consistency, in contrast to 2D GANs. However, the multi-view consistency (particularly at fine levels) is compromised when resorting to the CNN-based renderer to produce the final image. The proposed pipeline is a good workaround but this compromise is absolutely nonnegligible, it does not fundamentally solve the key issue existing in this 3D-aware generative method. This deviates from the original course of the 3D-aware generative model.\n\nMore that need clarification:\n1. what is the geometric meaning of the equation (5)? Is it that only the feature at the distance of the depth value is hit for the subsequent rendering? If this is the case, the rendering process still has to evaluate the density of all the samples along the ray for obtaining the depth value. I am not sure about the gain of this approximation in terms of improving computation efficiency. Would appreciate it if this can be explained.\n2. how general is the upsampler design in this work? It seems to work very well in the paper. Would be exciting to see if this design choice can also be used in other GANs to improve the synthesis and thus show the generalizability of this design choice.\n3. this work uses separate networks for the background and the foreground (based on NeRF++), I am wondering if this is critical, as this design is not employed in other baseline methods, in which the generated images have their bg and fg twisted together.\n4. it is a bit confusing that GRAF and pi-GAN perform so much worse than in their original paper. The datasets used in this paper are different from theirs, comparisons on their datasets may be a better option to rule out those irrelevant factors (e.g., hyper-parameters) and thus offer a fairly understandable comparison.\n5. in Table 1, I understand that FID and KID do not penalize the 2D GAN for having no multi-view consistency, but still, there should be proper highlights in this table... We can have a metric for quantitatively measuring the multi-view consistency (e.g., like in GRAF, measuring the multi-view reconstruction quality of generated multi-view images of the same 3D), on which the proposed method would dominate.\n\nOthers:\n1. section 3.2, around equation (5), some of the Phi may have incorrect superscripts, causing some troubles in following this part.\n2. section 3.4: \"an mapping network\" -> \"a mapping network\".\n3. it may be better to not expand the equation (2) and not expose many g, the g functions are not used elsewhere, so it may only make it lengthy and a bit more unreadable.\n4. section 4.2: \"GARF\" -> \"GRAF\".", "summary_of_the_paper": "This paper proposes a 3D-aware generative model for photo-realistic high-resolution image synthesis. The core of this work is a set of tricks that are employed to speed up the generation and enforce the multi-view consistency while trying to synthesize higher resolution images in 2D with a CNN-based renderer.", "summary_of_the_review": "Overall, I appreciate very much the engineering effort behind this work. Although the lack of significant conceptual novelty, it has truly brought up the 3D-aware generative model onto a new level with great quality. Please check the concerns expressed above for not giving a higher rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EHYc9Ub5aik", "reviewer": "Reviewer_MWb6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n(1) StyleNeRF gives better qualitative and quantitative results than the baseline in high resolution situations, which is particularly evident in the fewer artifacts. Also, this method adopts rendering at interactive rates. (2) The paper solves the problem of high-resolution synthesis that cannot be solved by traditional NeRF-based methods. (3) The proposed pioneering approach of fusing 2D GAN tricks and NeRF-based tricks is a meaningful and groundbreaking work.\n\nWeaknesses:\n(1) There are some typos. In Figure 3. The 'r' in 'right' should be 'R'. The 'an mapping network' should be 'a mapping network' in the third row of Section 3.4. The pi-GAN and π-GAN(in Baseline details of Appendix C) should be written in the same way. (2) Figure 3 is quite confusing, the blocks in figure should be mentioned in paper, like FG and BLK. (3) It is suggested to conduct more quantitative experiments about different upsampling operators to show the strengths of the proposed method. (4) There is an interesting phenomenon where the pupil position in the human eyes of the synthesized images (Figure 6 and 15)  are different so that the human eyes seem to toward the direction of the camera observation, while the pupil position in the human eyes of the synthesized images (Figure 12 and 13)  are nearly same. Could this be explained?\n", "summary_of_the_paper": "This paper proposes StyleNeRF which combines NeRF and a style-based generator to improve rendering efficiency and 3D consistency in high-resolution image synthesis. In this paper, (1) The NeRF is used to produce a low-resolution feature map and upsample it progressively to high resolution. (2) Several designs are proposed to improve 3D consistency, including a desirable upsampler, a novel regularization term. (3) A progressive training strategy is adopted to significantly improves the stability of learning the real geometry.", "summary_of_the_review": " I think the contributions of this paper far outweigh its shortcomings and this paper can be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: An interesting paper on combining NerFs with StyleGAN to get high-quality and high-resolution 3d aware generative models. The results are very good visually and also allow interactive speeds.  The technique is natural and concurrent papers are proposing variations

The reviewers identified a few limitations including that the nerf does not have a viewing direction and also seems limited to aligned objects with a common structure, like faces. Still the results are very interesting and suitable for publication.
Decision: Accept (Poster)