StyleNeRF: A Style-based 3D Aware Generator for High-resolution Image Synthesis

1 INTRODUCTION . Photo-realistic free-view image synthesis of real-world scenes is a long-standing problem in computer vision and computer graphics . Traditional graphics pipeline requires production-quality 3D models , computationally expensive rendering , and manual work , making it challenging to apply to large-scale image synthesis for a wide range of real-world scenes . In the meantime , Generative Adversarial Networks ( GANs , Goodfellow et al. , 2014 ) can be trained on a large number of unstructured images to synthesize high-quality images . However , most GAN models operate in 2D space and lack the 3D understanding of the training images , which results in their inability to synthesize images of the same scene with multi-view consistency and direct camera control . Natural images are the 2D projection of the 3D world . Hence , recent works on generative models ( Schwarz et al. , 2020 ; Chan et al. , 2021 ) enforce 3D structures by incorporating a neural radiance field ( NeRF , Mildenhall et al. , 2020 ) . However , these methods can not synthesize high-resolution images with delicate details due to the computationally expensive rendering process of NeRF . Furthermore , the slow rendering process leads to inefficient training and makes these models unsuitable for interactive applications . GIRAFFE ( Niemeyer & Geiger , 2021b ) combines NeRF with a CNN- based renderer , which has the potential to synthesize high-resolution images . However , this method falls short of 3D-consistent image generation and so far has not shown high-resolution results . We propose StyleNeRF , a new 3D-aware generative model for high-resolution 3D consistent image synthesis at interactive rates . It also allows control of the 3D camera pose and enables control of specific style attributes . StyleNeRF incorporates 3D scene representations into a style-based generative model . To prevent the expensive direct color image rendering from the original NeRF approach , we only use NeRF to produce a low-resolution feature map and upsample it progressively to high resolution . To improve 3D consistency , we propose several designs , including a desirable upsampler that achieves high consistency while mitigating artifacts in the outputs , a novel regularization term that forces the output to match the rendering result of the original NeRF and fixing the issues of view direction condition and noise injection . StyleNeRF is trained using unstructured real-world images . A progressive training strategy significantly improves the stability of learning real geometry . We evaluate StyleNeRF on various challenging datasets . StyleNeRF can synthesize photo-realistic 10242 images at interactive rates while achieving high multi-view consistency . None of the existing methods can achieve both characteristics . Additionally , StyleNeRF enables direct control on styles , and 3D camera poses even for the poses starkly different from training . 2 RELATED WORK . Neural Implicit Fields Representing 3D scenes as neural implicit fields has increasingly gained much attention . Michalkiewicz et al . ( 2019 ) ; Mescheder et al . ( 2019 ) ; Park et al . ( 2019 ) ; Peng et al . ( 2020 ) predict neural implicit fields with 3D supervision . Some of them ( Sitzmann et al. , 2019 ; Niemeyer et al. , 2019 ) assume that the ray color only lies on the geometry surface and propose differentiable renderers to learn a neural implicit surface representation . NeRF and and similar works ( Lombardi et al. , 2019 ; Mildenhall et al. , 2020 ; Liu et al. , 2020 ; Zhang et al. , 2020 ) utilize a volume rendering technique to render neural implicit volume representations for novel view synthesis . In this work , we focus on generative NeRF . Unlike the discussed methods , which require posed multi-view images , our approach only needs unstructured single-view images for training . Image Synthesis with GANs Starting from Goodfellow et al . ( 2014 ) , GANs have demonstrated high-quality results ( Durugkar et al. , 2017 ; Mordido et al. , 2018 ; Doan et al. , 2019 ; Zhang et al. , 2019 ; Brock et al. , 2018 ; Karras et al. , 2018 ) . StyleGANs ( Karras et al. , 2019 ; 2020b ) achieve SOTA quality and support different levels of style control . Karras et al . ( 2021 ) solve the “ texture sticking ” problem of 2D GANs in generating animations with 2D transformations . Some methods ( Härkönen et al. , 2020 ; Tewari et al. , 2020a ; Shen et al. , 2020 ; Abdal et al. , 2020 ; Tewari et al. , 2020b ; Leimkühler & Drettakis , 2021 ; Shoshan et al. , 2021 ) leverage disentangled properties in the latent space to enable explicit controls , most of which focus on faces . While these methods can synthesize face poses parameterized by two angles , extending them to general objects and controlling 3D cameras is not easy . Chen et al . ( 2021a ) proposed to generate segmentation maps from implicit fields to enable 3D control . However , it requires 3D meshes for pre-training . In contrast , our work can synthesize images for general objects , enabling explicit 3D camera control . 3D-Aware GANs Recently , 3D representations have been integrated into 2D generative models to enable camera control . Voxel-based GANs ( Henzler et al. , 2019 ; Nguyen-Phuoc et al. , 2019 ; 2020 ) lack fine details in the output due to resolution restriction . Radiance fields-based methods ( Schwarz et al. , 2020 ; Chan et al. , 2021 ; Niemeyer & Geiger , 2021a ) achieve higher quality but have difficulties in training on high-resolution images ( 5122 and beyond ) due to the expensive rendering process . GIRAFFE ( Niemeyer & Geiger , 2021b ) improves the training and rendering efficiency by combining NeRF with a CNN-based renderer ; GSN ( DeVries et al. , 2021 ) models a locally conditional NeRF with a similar renderer for unconstrained indoor scene generation . However , they both produce severe view-inconsistent artifacts due to their network designs ( e.g. , 3 × 3 Conv and upsampler ) . In contrast , our method can effectively preserve view consistency in image synthesis . 3 METHOD . 3.1 IMAGE SYNTHESIS AS NEURAL IMPLICIT FIELD RENDERING . Style-based Generative Neural Radiance Field We start by modeling a 3D scene as neural radiance field ( NeRF , Mildenhall et al. , 2020 ) . It is typically parameterized as multilayer perceptrons ( MLPs ) , which takes the position x ∈ R3 and viewing direction d ∈ S2 as input , and predicts the density σ ( x ) ∈ R+ and view-dependent color c ( x , d ) ∈ R3 . To model high-frequency details , follwing NeRF ( Mildenhall et al. , 2020 ) , we map each dimension of x and d with Fourier features : ζL ( x ) = [ sin ( 20x ) , cos ( 20x ) , . . . , sin ( 2L−1x ) , cos ( 2L−1x ) ] ( 1 ) We formalize StyleNeRF representations by conditioning NeRF with style vectors w as follows : φnw ( x ) = g n w ◦ gn−1w ◦ . . . ◦ g1w ◦ ζ ( x ) , where w = f ( z ) , z ∈ Z ( 2 ) Similar as StyleGAN2 ( Karras et al. , 2020b ) , f is a mapping network that maps noise vectors from the spherical Gaussian spaceZ to the style spaceW ; giw ( . ) is the ith layer MLP whose weight matrix is modulated by the input style vectorw . φnw ( x ) is the n-th layer feature of that point . We then use the extracted features to predict the density and color , respectively : σw ( x ) = hσ ◦ φnσw ( x ) , cw ( x , d ) = hc ◦ [ φncw ( x ) , ζ ( d ) ] , ( 3 ) where hσ and hc can be a linear projection or 2-layer MLPs . Different from the original NeRF , we assume nc > nσ for Equation ( 3 ) as the visual appearance generally needs more capacity to model than the geometry . The first min ( nσ , nc ) layers are shared in the network . Volume Rendering Image synthesis is modeled as volume rendering from a given camera pose p ∈ P . For simplicity , we assume a camera is located on the unit sphere pointing to the origin with a fixed field of view ( FOV ) . We sample the camera ’ s pitch & yaw from a uniform or Gaussian distribution . To render an image I ∈ RH×W×3 , we shoot a camera ray r ( t ) = o + td ( o is the camera origin ) for each pixel , and then calculate the color using the volume rendering equation : INeRFw ( r ) = ∫ ∞ 0 pw ( t ) cw ( r ( t ) , d ) dt , where pw ( t ) = exp ( − ∫ t 0 σw ( r ( s ) ) ds ) · σw ( r ( t ) ) ( 4 ) In practice , the above equation is discretized by accumulating sampled points along the ray . Following NeRF ( Mildenhall et al. , 2020 ) , stratified and hierarchical sampling are applied for more accurate discrete approximation to the continuous implicit function . Challenges Compared to 2D generative models ( e.g. , StyleGANs ( Karras et al. , 2019 ; 2020b ) ) , the images generated by NeRF-based models have 3D consistency , which is guaranteed by modeling the image synthesis as a physics process , and the neural 3D scene representation is invariant across different viewpoints . However , the drawbacks are apparent : these models cost much more computation to render an image at the exact resolution . For example , 2D GANs are 100 ∼ 1000 times more efficient to generate a 10242 image than NeRF-based models . Furthermore , NeRF consumes much more memory to cache the intermediate results for gradient back-propagation during training , making it difficult to train on high-resolution images . Both of these restrict the scope of applying NeRF-based models in high-quality image synthesis , especially at the training stage when calculating the objective function over the whole image is crucial . 3.2 APPROXIMATION FOR HIGH-RESOLUTION IMAGE GENERATION . In this section , we propose how to improve the efficiency of StyleNeRF by taking inspiration from 2D GANs . We observe that the image generation of 2D GANs ( e.g. , StyleGANs ) is fast due to two main reasons : ( 1 ) each pixel only takes single forward pass through the network ; ( 2 ) image features are generated progressively from coarse to fine , and the feature maps with higher resolutions typically have a smaller number of channels to save memory . In StyleNeRF , the first point can be partially achieved by early aggregating the features into the 2D space before the final colors are computed . In this way , each pixel is assigned with a feature vector , Furthermore , it only needs to pass through a network once rather than calling the network multiple times for all sampled points on the ray as NeRF does . We approximate Equation ( 4 ) as : IApproxw ( r ) = ∫ ∞ 0 pw ( t ) · hc ◦ [ φncw ( r ( t ) ) , ζ ( d ) ] dt ≈ hc ◦ [ φnc , nσw ( A ( r ) ) , ζ ( d ) ] , ( 5 ) where φn , nσw ( A ( r ) ) = gnw ◦ gn−1w ◦ . . . ◦ gnσ+1w ◦ A ( r ) and A ( r ) = ∫∞ 0 pw ( t ) · φnσw ( r ( t ) ) dt . The definitions of A ( . ) and φn , nσw ( . ) can be extended to the operations on a set of rays , each ray processed independently . Next , instead of using volume rendering to render a high-resolution feature map directly , we can employ NeRF to generate a downsampled feature map at a low resolution and then employ upsampling in 2D space to progressively increase into the required high resolution . We take two adjacent resolutions as an example . Suppose RL ∈ RH/2×W/2 and RH ∈ RH×W are the corresponding rays of the pixels in the low- and high-resolution images , respectively . To approximate the high-resolution feature map , we can up-sample in the low-resolution feature space : φn , nσw ( A ( RH ) ) ≈ Upsample ( φn , nσw ( A ( RL ) ) ) ( 6 ) Recursively inserting Upsample operators enables efficient high-resolution image synthesis as the computationally expensive volume rendering only needs to generate a low-resolution feature map . The efficiency is further improved when using fewer channels for higher resolution . While early aggregation and upsampling operations can accelerate the rendering process for highresolution image synthesis , they would come with scarification to the inherent consistency of NeRF . There are two reasons why they introduce inconsistency . First , the resulting model contains nonlinear transformations to capture spurious correlations in 2D observation , mainly when substantial ambiguity exists . For example , our training data are unstructured single-view images without sufficient multi-view supervision . Second , such a pixel-space operation like up-sampling would compromise 3D consistency . Therefore , naı̈ve model designs would lead to severe multi-view inconsistent outputs ( e.g. , when moving the camera to render images , hairs are constantly changing ) . In the following , we propose several designs and choices to alleviate the inconsistency in the outputs .

Summary: In this manuscript, authors proposed a novel 3D-aware generative model for photo-realistic high resolution image synthesis. The proposed method combines Neural Radiance Fields (NerF) and StyleGAN, and tackles the challenges of efficiency, multi-view consistency and rendering quality. Specifically, authors propose to use NeRF to render low resolution images before feeding them into 2D StyleGAN network to upsample the feature maps in order to obtain images with high resolution and therefore the proposed model can render images at interactive rates. Extensive experiments on numerous datasets demonstrate superior performance than prior state of art 3D generative view synthesis approaches and largely close the gap between 2D and 3D view synthesis methods.