Generative Occupancy Fields for 3D Surface-Aware Image Synthesis

1 Introduction . Deep generative adversarial networks [ 1–4 ] have demonstrated their superiority in synthesizing photorealistic and striking images . However , these models are often constrained in the 2D domain , struggling to generate 3D consistent images , let alone grasping the underlying 3D object shapes . 3D-aware image synthesis thus becomes an appealing and promising choice as it learns a 3D representation explicitly from a collection of unposed images . Consequently , it can not only synthesize 3D consistent images by manually controlling the rendering camera poses , but also pave the way for various downstream tasks such as shape editing and relighting . Inspired by the success of neural radiance fields ( NeRF ) [ 5 ] in 3D scene modeling , recent 3D-aware generative models , referred to as generative radiance fields ( GRAFs ) , have applied NeRF as the explicit 3D representation for image synthesis [ 6 , 7 ] . With the help of NeRF , they are capable of hallucinating photorealistic images in a 3D consistent manner . Moreover , since NeRF holds the superior ability for rendering translucent objects by compositing colored densities along each ray in its volume rendering process , it also significantly facilitates the training of GRAFs as gradients are naturally distributed over the entire volume . However , they still incur an inevitable incapacity of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . capturing an accurate and compact object surface . As shown in Fig . 1 ( a ) , the state-of-the-art GRAF model pi-GAN is prone to predict diffused object surfaces , as the volume densities are smoothly spread around the surfaces . Such diffused surfaces could significantly hamper the applications of GRAF models in downstream tasks such as shape recovery . Moreover , under different light conditions , the artifacts of surfaces could be amplified and inherited through the rendering process , resulting in synthesized images that are messy and faulty . In this work , we propose Generative Occupancy Fields ( GOF ) , a novel GRAF-like image synthesis model that can learn compact object surfaces . GOF is inspired by the design of occupancy networks [ 8 ] that implicitly represents a 3D surface with the continuous decision boundary of a neural classifier . In this way , occupancy networks are capable of effectively locating surfaces via root-finding and encouraging the compactness of modeled surfaces inherently . However , GOF avoids directly applying such a design to 3D-aware image synthesis . While occupancy networks require precise object masks to train [ 9 , 10 ] , a more crucial factor is that they rely on the surface points for differentiable rendering [ 11 , 12 , 9 , 13 ] . A generative model equipped with occupancy representations will thus meet severe convergence problems during training due to the sparsity of gradients . To unify the merits of both NeRF and occupancy networks for 3D-aware image synthesis , GOF adopts the design of GRAFs and at the same time leverages a nontrivial transition from the cumulative rendering to rendering with only the surface points , i.e . start from radiance fields and march to occupancy representations . Specifically , GOF will reinterpret the alpha values in the cumulative rendering process as occupancy values , so that it can locate the learned surface via root-finding . Subsequently , it can naturally encourage the compactness of learned surfaces by gradually shrinking the sampling region in the rendering process from the entire volume to a minimal neighboring region around the surface . Thanks to the unified integration of radiance fields and occupancy representations , GOF can benefit from the representation effectiveness of radiance fields while ensuring the compactness of learned object surfaces through the shrinking process . As presented in Fig . 1 ( a ) , in GOF the distribution of cumulative rendering weights concentrates more closely around object surfaces compared to that of pi-GAN , eventually resulting in a compact and smooth surface . Moreover , GOF can thus alternatively render an image only with points on the learned surfaces like occupancy networks as illustrated in Fig . 1 ( b ) . And during inference such a rendering scheme has the potential to alleviate the burden of sampling a large number of points along each ray for synthesizing a single image . Through exhaustive experiments on synthetic and real-world datasets , we demonstrate that GOF can achieve state-of-the-art performance on 3D-aware image synthesis . Meanwhile , it is capable of capturing compact and accurate 3D shapes that empower its applications in various downstream tasks such as 3D shape reconstruction . We validate this point by quantitative results of 3D shape reconstruction on the Synface dataset . Finally , we have also verified the ability of GOF in rendering high-quality images with only the surface points , which is hardly achievable in previous approaches . 2 Related Work . Neural implicit function for 3D representations . A plethora of works [ 14 , 8 , 15–21 ] has exploited neural implicit functions for 3D geometry modeling . Among these works , neural radiance fields ( NeRF ) [ 5 ] has attracted growing attention due to its compelling results on novel view synthesis . It leverages an MLP network to approximate the radiance fields of static 3D scenes . And by learning to reconstruct existing views , it is capable of capturing 3D geometric details from only 2D supervision . A series of succeeding variants of NeRF have been proposed to improve it , including utilizing the spatial sparsity to reduce its computational complexity [ 22 , 23 ] , refining the rendering process to improve its efficiency [ 24 , 25 ] , as well as adopting reflectance decompositions to enhance its modeling capacity [ 26 , 27 ] . There are also works that capitalize on the differentiable rendering of neural implicit functions for 3D reconstruction [ 11 , 12 , 9 , 13 , 28 , 29 ] . Specifically , SDFDiff [ 12 ] relies on the interpolation of eight neighboring SDF samples around the surface intersection to obtain the derivatives , while Atzmon et al [ 28 ] use a sample network to relate samples ’ positions to network parameters and thus achieve an improved generalization ability . More interestingly , by adopting occupancy representations , DVR [ 11 ] and IDR [ 9 ] show volumetric rendering is inherently differentiable so that network parameters can be optimized directly with derived analytic gradients . Different from methods aforementioned above , GOF is a generative model for 3D-aware image synthesis that can learn 3D representations from a set of 2D images with unknown camera poses . Generative 3D-aware image synthesis . In order to synthesize 3D consistent images , researchers have explored a lot on how to incorporate 3D representations into the classical GAN model [ 1 ] . Some methods [ 30–32 ] resort to learning from 3D data directly , yet the requirement of 3D supervision limits their practical applicability . A more appealing alternative is thus learn from unposed 2D images in an unsupervised manner . Preceding works along this line of research adopt voxels as their intermediate 3D representations [ 33–35 ] and achieve explicit control over the pose of synthesized images . Inspired by the superior representation capacity of radiance fields over voxels , recent attempts [ 6 , 7 , 36 ] have replaced voxels with neural radiance fields [ 5 ] to improve the fidelity of synthesized 3D consistent images . Despite the striking performance , these models , referred to as generative radiance fields ( GRAFs ) , tend to predict diffused object surfaces , which impedes its applicability in various downstream tasks . In this work , GOF aims at resolving this problem of GRAFs by combining them with the perspective of occupancy networks [ 8 ] and recent successes of recovering smooth and accurate shapes from natural images [ 37–40 ] . Recently , three concurrent works , UNISURF [ 10 ] , NeuS [ 41 ] and VolSDF [ 42 ] , also combine implicit surfaces and radiance fields in a unified framework , sharing similar spirits with our proposed GOF but different in tasks and focuses . Specifically , They focus on multi-view 3D reconstruction and attempt to alleviate the requirement of training-time precise masks through the integration of radiance fields and occupancy representations . Nevertheless , they still require images with ground-truth poses for training . By contrast , GOF targets on the challenging task of 3D-aware image synthesis , where the synthesized images should be not only natural and vivid , but also consistent in the 3D space . By integrating radiance fields and occupancy representations , GOF is able to facilitate the convergence of GRAFs and ensure the compactness of learned object surfaces . Compared to existing GRAFs , the applicability of GOF is thus significantly broadened . 3 Methodology . We propose generative occupancy fields ( GOF ) , a novel synthesis model , belonging to generative radiance fields and aiming to learn from unposed images . Conditioned on a latent code z ∼ pz , our generator gθ can generate a 3D radiance field R , from which we can render a realistic image with a sampled camera pose ξ ∼ pξ and simultaneously recover smooth and compact object surfaces . In the following , we first present the background of neural radiance fields , and then introduce our proposed GOF model in detail . 3.1 Neural Radiance Fields . We adopt neural radiance fields ( NeRF ) as our explicit 3D representation for image synthesis , owing to its strong performance in novel view synthesis on complex scenes . NeRF represents a static scene as per-point volume densities and view-dependent RGB colors . Given a 3D point x ∈ R3 in space and a view direction d ∈ R3 , NeRF capitalizes on a multi-layer perceptron ( MLP ) to predict the volume density σ ( x ) ∈ R and the emitted color c ( x , d ) ∈ R3 . To render a novel view for the scene , NeRF leverages the classic volume rendering technique [ 43 ] to estimate the color of each pixel . It starts by accumulating the colored densities of N points { xi = o + tid } sampled within near and far bounds [ tn , tf ] along the camera ray r ( t ) = o + td , where o stands for the camera origin . The integrated color is then estimated via alpha composition as follows : Ĉ ( r ) = N∑ i=1 Ti ( 1− exp ( − σ ( xi ) δi ) ) c ( xi , d ) , where Ti = exp ( − i−1∑ j=1 σ ( xj ) δj ) , ( 1 ) where δi = |xi+1 − xi| is the distance between adjacent points . Note that , equation ( 1 ) is naturally differentiable and NeRF can be directly optimized through the reconstruction error of existing views . 3.2 3D Surface-Aware Image Synthesis via Generative Occupancy Fields . To apply NeRF as the 3D representation , the proposed generative occupancy fields ( GOF ) incorporates an additional latent code z ∼ pz into NeRF , such that synthesizing an image follows a reformulated cumulative rendering process : Ĉ ( r , z ) = N∑ i=1 Ti ( 1−exp ( −σθ ( xi , z ) δi ) ) cθ ( xi , d , z ) , where Ti = exp ( − i−1∑ j=1 σθ ( xj , z ) δj ) . ( 2 ) However , directly training GOF according to Eq . ( 2 ) fails to maintain the surface compactness as reported in previous approaches [ 6 , 7 ] . Actually , such a defect arises from an inevitable “ shape-color ambiguity ” of the cumulative rendering process , i.e. , small perturbations on surfaces still lead to realistic RGB images which are enough to fool the discriminator . Owing to the constrained range of poses seen at training , the discriminator is less motivated to further concentrate the color weights wi = Ti ( 1− exp ( −σθ ( xi , z ) δi ) ) aforementioned in Fig . 1 ( a ) on the exact object surface . On the other hand , we observe that although leading to diffused surfaces at the end , color weights wi gradually concentrate around the object surface as the training proceeds . Inspired by this observation , in GOF we propose a training-time operation to facilitate the concentration of color weights wi . The basic idea is gradually shrinking the sample region in the cumulative rendering process from the entire volume to a narrow interval around the surface , so that color weights are enforced to continuously move towards the exact surface . To enable the proposed training-time shrinking process , GOF is required to locate the surface by thresholding the predicted densities σθ ( x , z ) , assuming points on the surface have the largest densities , However , values of the densities predicted in generative radiance fields could range from 0 to 50 , making it hard to determine an effective threshold τ during the whole training period . On the other hand , in the cumulative rendering process shown in Eq . ( 2 ) , we found that the intermediate alpha values used for numerical stability inherently fall in a fixed value range as : αθ ( xi , z ) = 1− exp ( −σθ ( xi , z ) δi ) ∈ [ 0 , 1 ] . ( 3 ) More importantly , these alpha values αθ ( x , z ) come close to 1 for points in the occupied space while approaching 0 for points in the free space , making them resemble the occupancy values [ 8 ] in both quantity and semantics . Inspired by the similarity , we thus propose to reformulate generative radiance fields by predicting alpha values αθ ( x , z ) directly instead of volume densities σθ ( x , z ) . In the mean time , we reinterpret the alpha values as occupancy values , and subsequently locate surfaces with root-finding , a more effective strategy originated in occupancy networks [ 11 ] . According to the above reformulation and reinterpretation , we thus dub our method as Generative Occupancy Fields . As GOF estimates alpha values instead of volume densities , the original volume rendering process in Eq . ( 2 ) conditioned on the latent code z is reformulated as Ĉ ( r , z ) = N∑ i=1 αθ ( xi , z ) ∏ j < i ( 1− αθ ( xj , z ) ) cθ ( xi , d , z ) , ( 4 ) where the value range of αθ ( xi , z ) is guaranteed with a sigmoid function . And to locate the surface via root-finding , for a specific ray r ( t ) = o+td we will evenly sampleM points { xk = o+tkd ; k = 1 , ... , M } that partition the entire volume [ tn , tf ] into M equally-spaced bins . After obtaining the corresponding alpha values { αθ ( xk , z ) ; k = 1 , ... , M } by querying the generator gθ , the surface S is located in the kS-th bin where αθ changes for the first time from free space ( αθ < τ ) to occupied space ( αθ < τ ) : kS = argmin k ( αθ ( xk , z ) < τ ≤ αθ ( xk+1 , z ) ) , ( 5 ) where τ is a pre-defined threshold . In practice , we empirically set τ as 0.5 . In order to find the surface point xs = o + tsd more precisely , we further apply the above secant method iteratively for ms times , resulting in a fine-grained bin [ xkS , xkS+1 ] . It ’ s worth noting that the M sampled points are only used for root-finding . Thus they do not require the computation of gradients in the implementation . Based on the located surface S , we can thus successfully conduct the proposed shrinking process , which is schematically elaborated in Fig . 2 . Specifically , when sampling N points for Eq . ( 4 ) at each training step , we will only sample within a region neighboring the surface [ ts −∆ , ts + ∆ ] : ti ∼ U [ ts −∆ + 2i− 2 N ∆ , ts −∆ + 2i N ∆ ] , where i = 1 , 2 , ... , N. ( 6 ) ∆ is the sampling interval , which is set to ∆init = ( tf − tn ) /2 at the beginning , a half of the distance between near tn and far bounds tf . And it will decrease monotonically with an exponential decay rate γ until it drops to a pre-defined minimal value ∆min . Formally , ∆n = max ( ∆init exp ( −γn ) , ∆min ) for n-th decaying step . Additionally , during training when the estimated ts is too close to the near or the far bound so that the sampling region [ ts −∆ , ts + ∆ ] exceeds the original range [ tn , tf ] , we will shift the region [ ts −∆ , ts + ∆ ] back to within [ tn , tf ] . As shown in Fig . 2 , at the beginning of training , points sampled for Eq . ( 4 ) will cover the entire volume , leading to dispersed gradients which facilitate the convergence of GOF . And as the training goes , the predicted surface will become more and more accurate , which is the outcome of gradually refining the sampling region , and in turn also makes the above shrinking operation valid . Thanks to the dedicated shrinking process , the color weights wi can successfully concentrate on the object surface as illustrated in Fig . 1 ( a ) . As a result , GOF is capable of synthesizing highfidelity images in a 3D-consistent manner and simultaneously capturing compact object surfaces . During inference , to synthesize an image under a random camera pose ξ ∼ pξ , the generator gθ will fetch a truncated latent code ẑ and sample N points { xi } on each ray within the minimal region [ ts−∆min , ts+∆min ] for the rendering as in Eq. ( 4 ) . An important benefit of learning a compact object surface is that we can effectively reduce the number of sampled points for rendering , even using only one point on each ray , i.e . the surface point . As shown in Fig . 1 ( b ) , the image rendered with only the surface point is almost indistinguishable from that with multiple points . Such equivalence can be guaranteed theoretically when ∆min → 0 , and we include the proof in the supplementary material .

Summary: The paper suggests a new generative model for 3D-aware image synthesis by generating radiance fields with an underlying 3D occupancy representation. This choice of representations allows the generative model to learn surfaces, which are rendered to view consistent images from arbitrary poses using volume rendering. During training, the rendering is performed using samples from a gradually shrinking sampling region around the surface, which encourages the surface to be more compact. The paper demonstrates the method's ability to synthesize images with 3D consistency and further presents their matching shape and normal.