StyleNeRF: A Style-based 3D Aware Generator for High-resolution Image Synthesis

1 INTRODUCTION . Photo-realistic free-view image synthesis of real-world scenes is a long-standing problem in computer vision and computer graphics . Traditional graphics pipeline requires production-quality 3D models , computationally expensive rendering , and manual work , making it challenging to apply to large-scale image synthesis for a wide range of real-world scenes . In the meantime , Generative Adversarial Networks ( GANs , Goodfellow et al. , 2014 ) can be trained on a large number of unstructured images to synthesize high-quality images . However , most GAN models operate in 2D space and lack the 3D understanding of the training images , which results in their inability to synthesize images of the same scene with multi-view consistency and direct camera control . Natural images are the 2D projection of the 3D world . Hence , recent works on generative models ( Schwarz et al. , 2020 ; Chan et al. , 2021 ) enforce 3D structures by incorporating a neural radiance field ( NeRF , Mildenhall et al. , 2020 ) . However , these methods can not synthesize high-resolution images with delicate details due to the computationally expensive rendering process of NeRF . Furthermore , the slow rendering process leads to inefficient training and makes these models unsuitable for interactive applications . GIRAFFE ( Niemeyer & Geiger , 2021b ) combines NeRF with a CNN- based renderer , which has the potential to synthesize high-resolution images . However , this method falls short of 3D-consistent image generation and so far has not shown high-resolution results . We propose StyleNeRF , a new 3D-aware generative model for high-resolution 3D consistent image synthesis at interactive rates . It also allows control of the 3D camera pose and enables control of specific style attributes . StyleNeRF incorporates 3D scene representations into a style-based generative model . To prevent the expensive direct color image rendering from the original NeRF approach , we only use NeRF to produce a low-resolution feature map and upsample it progressively to high resolution . To improve 3D consistency , we propose several designs , including a desirable upsampler that achieves high consistency while mitigating artifacts in the outputs , a novel regularization term that forces the output to match the rendering result of the original NeRF and fixing the issues of view direction condition and noise injection . StyleNeRF is trained using unstructured real-world images . A progressive training strategy significantly improves the stability of learning real geometry . We evaluate StyleNeRF on various challenging datasets . StyleNeRF can synthesize photo-realistic 10242 images at interactive rates while achieving high multi-view consistency . None of the existing methods can achieve both characteristics . Additionally , StyleNeRF enables direct control on styles , and 3D camera poses even for the poses starkly different from training . 2 RELATED WORK . Neural Implicit Fields Representing 3D scenes as neural implicit fields has increasingly gained much attention . Michalkiewicz et al . ( 2019 ) ; Mescheder et al . ( 2019 ) ; Park et al . ( 2019 ) ; Peng et al . ( 2020 ) predict neural implicit fields with 3D supervision . Some of them ( Sitzmann et al. , 2019 ; Niemeyer et al. , 2019 ) assume that the ray color only lies on the geometry surface and propose differentiable renderers to learn a neural implicit surface representation . NeRF and and similar works ( Lombardi et al. , 2019 ; Mildenhall et al. , 2020 ; Liu et al. , 2020 ; Zhang et al. , 2020 ) utilize a volume rendering technique to render neural implicit volume representations for novel view synthesis . In this work , we focus on generative NeRF . Unlike the discussed methods , which require posed multi-view images , our approach only needs unstructured single-view images for training . Image Synthesis with GANs Starting from Goodfellow et al . ( 2014 ) , GANs have demonstrated high-quality results ( Durugkar et al. , 2017 ; Mordido et al. , 2018 ; Doan et al. , 2019 ; Zhang et al. , 2019 ; Brock et al. , 2018 ; Karras et al. , 2018 ) . StyleGANs ( Karras et al. , 2019 ; 2020b ) achieve SOTA quality and support different levels of style control . Karras et al . ( 2021 ) solve the “ texture sticking ” problem of 2D GANs in generating animations with 2D transformations . Some methods ( Härkönen et al. , 2020 ; Tewari et al. , 2020a ; Shen et al. , 2020 ; Abdal et al. , 2020 ; Tewari et al. , 2020b ; Leimkühler & Drettakis , 2021 ; Shoshan et al. , 2021 ) leverage disentangled properties in the latent space to enable explicit controls , most of which focus on faces . While these methods can synthesize face poses parameterized by two angles , extending them to general objects and controlling 3D cameras is not easy . Chen et al . ( 2021a ) proposed to generate segmentation maps from implicit fields to enable 3D control . However , it requires 3D meshes for pre-training . In contrast , our work can synthesize images for general objects , enabling explicit 3D camera control . 3D-Aware GANs Recently , 3D representations have been integrated into 2D generative models to enable camera control . Voxel-based GANs ( Henzler et al. , 2019 ; Nguyen-Phuoc et al. , 2019 ; 2020 ) lack fine details in the output due to resolution restriction . Radiance fields-based methods ( Schwarz et al. , 2020 ; Chan et al. , 2021 ; Niemeyer & Geiger , 2021a ) achieve higher quality but have difficulties in training on high-resolution images ( 5122 and beyond ) due to the expensive rendering process . GIRAFFE ( Niemeyer & Geiger , 2021b ) improves the training and rendering efficiency by combining NeRF with a CNN-based renderer ; GSN ( DeVries et al. , 2021 ) models a locally conditional NeRF with a similar renderer for unconstrained indoor scene generation . However , they both produce severe view-inconsistent artifacts due to their network designs ( e.g. , 3 × 3 Conv and upsampler ) . In contrast , our method can effectively preserve view consistency in image synthesis . 3 METHOD . 3.1 IMAGE SYNTHESIS AS NEURAL IMPLICIT FIELD RENDERING . Style-based Generative Neural Radiance Field We start by modeling a 3D scene as neural radiance field ( NeRF , Mildenhall et al. , 2020 ) . It is typically parameterized as multilayer perceptrons ( MLPs ) , which takes the position x ∈ R3 and viewing direction d ∈ S2 as input , and predicts the density σ ( x ) ∈ R+ and view-dependent color c ( x , d ) ∈ R3 . To model high-frequency details , follwing NeRF ( Mildenhall et al. , 2020 ) , we map each dimension of x and d with Fourier features : ζL ( x ) = [ sin ( 20x ) , cos ( 20x ) , . . . , sin ( 2L−1x ) , cos ( 2L−1x ) ] ( 1 ) We formalize StyleNeRF representations by conditioning NeRF with style vectors w as follows : φnw ( x ) = g n w ◦ gn−1w ◦ . . . ◦ g1w ◦ ζ ( x ) , where w = f ( z ) , z ∈ Z ( 2 ) Similar as StyleGAN2 ( Karras et al. , 2020b ) , f is a mapping network that maps noise vectors from the spherical Gaussian spaceZ to the style spaceW ; giw ( . ) is the ith layer MLP whose weight matrix is modulated by the input style vectorw . φnw ( x ) is the n-th layer feature of that point . We then use the extracted features to predict the density and color , respectively : σw ( x ) = hσ ◦ φnσw ( x ) , cw ( x , d ) = hc ◦ [ φncw ( x ) , ζ ( d ) ] , ( 3 ) where hσ and hc can be a linear projection or 2-layer MLPs . Different from the original NeRF , we assume nc > nσ for Equation ( 3 ) as the visual appearance generally needs more capacity to model than the geometry . The first min ( nσ , nc ) layers are shared in the network . Volume Rendering Image synthesis is modeled as volume rendering from a given camera pose p ∈ P . For simplicity , we assume a camera is located on the unit sphere pointing to the origin with a fixed field of view ( FOV ) . We sample the camera ’ s pitch & yaw from a uniform or Gaussian distribution . To render an image I ∈ RH×W×3 , we shoot a camera ray r ( t ) = o + td ( o is the camera origin ) for each pixel , and then calculate the color using the volume rendering equation : INeRFw ( r ) = ∫ ∞ 0 pw ( t ) cw ( r ( t ) , d ) dt , where pw ( t ) = exp ( − ∫ t 0 σw ( r ( s ) ) ds ) · σw ( r ( t ) ) ( 4 ) In practice , the above equation is discretized by accumulating sampled points along the ray . Following NeRF ( Mildenhall et al. , 2020 ) , stratified and hierarchical sampling are applied for more accurate discrete approximation to the continuous implicit function . Challenges Compared to 2D generative models ( e.g. , StyleGANs ( Karras et al. , 2019 ; 2020b ) ) , the images generated by NeRF-based models have 3D consistency , which is guaranteed by modeling the image synthesis as a physics process , and the neural 3D scene representation is invariant across different viewpoints . However , the drawbacks are apparent : these models cost much more computation to render an image at the exact resolution . For example , 2D GANs are 100 ∼ 1000 times more efficient to generate a 10242 image than NeRF-based models . Furthermore , NeRF consumes much more memory to cache the intermediate results for gradient back-propagation during training , making it difficult to train on high-resolution images . Both of these restrict the scope of applying NeRF-based models in high-quality image synthesis , especially at the training stage when calculating the objective function over the whole image is crucial . 3.2 APPROXIMATION FOR HIGH-RESOLUTION IMAGE GENERATION . In this section , we propose how to improve the efficiency of StyleNeRF by taking inspiration from 2D GANs . We observe that the image generation of 2D GANs ( e.g. , StyleGANs ) is fast due to two main reasons : ( 1 ) each pixel only takes single forward pass through the network ; ( 2 ) image features are generated progressively from coarse to fine , and the feature maps with higher resolutions typically have a smaller number of channels to save memory . In StyleNeRF , the first point can be partially achieved by early aggregating the features into the 2D space before the final colors are computed . In this way , each pixel is assigned with a feature vector , Furthermore , it only needs to pass through a network once rather than calling the network multiple times for all sampled points on the ray as NeRF does . We approximate Equation ( 4 ) as : IApproxw ( r ) = ∫ ∞ 0 pw ( t ) · hc ◦ [ φncw ( r ( t ) ) , ζ ( d ) ] dt ≈ hc ◦ [ φnc , nσw ( A ( r ) ) , ζ ( d ) ] , ( 5 ) where φn , nσw ( A ( r ) ) = gnw ◦ gn−1w ◦ . . . ◦ gnσ+1w ◦ A ( r ) and A ( r ) = ∫∞ 0 pw ( t ) · φnσw ( r ( t ) ) dt . The definitions of A ( . ) and φn , nσw ( . ) can be extended to the operations on a set of rays , each ray processed independently . Next , instead of using volume rendering to render a high-resolution feature map directly , we can employ NeRF to generate a downsampled feature map at a low resolution and then employ upsampling in 2D space to progressively increase into the required high resolution . We take two adjacent resolutions as an example . Suppose RL ∈ RH/2×W/2 and RH ∈ RH×W are the corresponding rays of the pixels in the low- and high-resolution images , respectively . To approximate the high-resolution feature map , we can up-sample in the low-resolution feature space : φn , nσw ( A ( RH ) ) ≈ Upsample ( φn , nσw ( A ( RL ) ) ) ( 6 ) Recursively inserting Upsample operators enables efficient high-resolution image synthesis as the computationally expensive volume rendering only needs to generate a low-resolution feature map . The efficiency is further improved when using fewer channels for higher resolution . While early aggregation and upsampling operations can accelerate the rendering process for highresolution image synthesis , they would come with scarification to the inherent consistency of NeRF . There are two reasons why they introduce inconsistency . First , the resulting model contains nonlinear transformations to capture spurious correlations in 2D observation , mainly when substantial ambiguity exists . For example , our training data are unstructured single-view images without sufficient multi-view supervision . Second , such a pixel-space operation like up-sampling would compromise 3D consistency . Therefore , naı̈ve model designs would lead to severe multi-view inconsistent outputs ( e.g. , when moving the camera to render images , hairs are constantly changing ) . In the following , we propose several designs and choices to alleviate the inconsistency in the outputs .

Summary: This paper proposes StyleNeRF which combines NeRF and a style-based generator to improve rendering efficiency and 3D consistency in high-resolution image synthesis. In this paper, (1) The NeRF is used to produce a low-resolution feature map and upsample it progressively to high resolution. (2) Several designs are proposed to improve 3D consistency, including a desirable upsampler, a novel regularization term. (3) A progressive training strategy is adopted to significantly improves the stability of learning the real geometry.