Generative Occupancy Fields for 3D Surface-Aware Image Synthesis

1 Introduction . Deep generative adversarial networks [ 1–4 ] have demonstrated their superiority in synthesizing photorealistic and striking images . However , these models are often constrained in the 2D domain , struggling to generate 3D consistent images , let alone grasping the underlying 3D object shapes . 3D-aware image synthesis thus becomes an appealing and promising choice as it learns a 3D representation explicitly from a collection of unposed images . Consequently , it can not only synthesize 3D consistent images by manually controlling the rendering camera poses , but also pave the way for various downstream tasks such as shape editing and relighting . Inspired by the success of neural radiance fields ( NeRF ) [ 5 ] in 3D scene modeling , recent 3D-aware generative models , referred to as generative radiance fields ( GRAFs ) , have applied NeRF as the explicit 3D representation for image synthesis [ 6 , 7 ] . With the help of NeRF , they are capable of hallucinating photorealistic images in a 3D consistent manner . Moreover , since NeRF holds the superior ability for rendering translucent objects by compositing colored densities along each ray in its volume rendering process , it also significantly facilitates the training of GRAFs as gradients are naturally distributed over the entire volume . However , they still incur an inevitable incapacity of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . capturing an accurate and compact object surface . As shown in Fig . 1 ( a ) , the state-of-the-art GRAF model pi-GAN is prone to predict diffused object surfaces , as the volume densities are smoothly spread around the surfaces . Such diffused surfaces could significantly hamper the applications of GRAF models in downstream tasks such as shape recovery . Moreover , under different light conditions , the artifacts of surfaces could be amplified and inherited through the rendering process , resulting in synthesized images that are messy and faulty . In this work , we propose Generative Occupancy Fields ( GOF ) , a novel GRAF-like image synthesis model that can learn compact object surfaces . GOF is inspired by the design of occupancy networks [ 8 ] that implicitly represents a 3D surface with the continuous decision boundary of a neural classifier . In this way , occupancy networks are capable of effectively locating surfaces via root-finding and encouraging the compactness of modeled surfaces inherently . However , GOF avoids directly applying such a design to 3D-aware image synthesis . While occupancy networks require precise object masks to train [ 9 , 10 ] , a more crucial factor is that they rely on the surface points for differentiable rendering [ 11 , 12 , 9 , 13 ] . A generative model equipped with occupancy representations will thus meet severe convergence problems during training due to the sparsity of gradients . To unify the merits of both NeRF and occupancy networks for 3D-aware image synthesis , GOF adopts the design of GRAFs and at the same time leverages a nontrivial transition from the cumulative rendering to rendering with only the surface points , i.e . start from radiance fields and march to occupancy representations . Specifically , GOF will reinterpret the alpha values in the cumulative rendering process as occupancy values , so that it can locate the learned surface via root-finding . Subsequently , it can naturally encourage the compactness of learned surfaces by gradually shrinking the sampling region in the rendering process from the entire volume to a minimal neighboring region around the surface . Thanks to the unified integration of radiance fields and occupancy representations , GOF can benefit from the representation effectiveness of radiance fields while ensuring the compactness of learned object surfaces through the shrinking process . As presented in Fig . 1 ( a ) , in GOF the distribution of cumulative rendering weights concentrates more closely around object surfaces compared to that of pi-GAN , eventually resulting in a compact and smooth surface . Moreover , GOF can thus alternatively render an image only with points on the learned surfaces like occupancy networks as illustrated in Fig . 1 ( b ) . And during inference such a rendering scheme has the potential to alleviate the burden of sampling a large number of points along each ray for synthesizing a single image . Through exhaustive experiments on synthetic and real-world datasets , we demonstrate that GOF can achieve state-of-the-art performance on 3D-aware image synthesis . Meanwhile , it is capable of capturing compact and accurate 3D shapes that empower its applications in various downstream tasks such as 3D shape reconstruction . We validate this point by quantitative results of 3D shape reconstruction on the Synface dataset . Finally , we have also verified the ability of GOF in rendering high-quality images with only the surface points , which is hardly achievable in previous approaches . 2 Related Work . Neural implicit function for 3D representations . A plethora of works [ 14 , 8 , 15–21 ] has exploited neural implicit functions for 3D geometry modeling . Among these works , neural radiance fields ( NeRF ) [ 5 ] has attracted growing attention due to its compelling results on novel view synthesis . It leverages an MLP network to approximate the radiance fields of static 3D scenes . And by learning to reconstruct existing views , it is capable of capturing 3D geometric details from only 2D supervision . A series of succeeding variants of NeRF have been proposed to improve it , including utilizing the spatial sparsity to reduce its computational complexity [ 22 , 23 ] , refining the rendering process to improve its efficiency [ 24 , 25 ] , as well as adopting reflectance decompositions to enhance its modeling capacity [ 26 , 27 ] . There are also works that capitalize on the differentiable rendering of neural implicit functions for 3D reconstruction [ 11 , 12 , 9 , 13 , 28 , 29 ] . Specifically , SDFDiff [ 12 ] relies on the interpolation of eight neighboring SDF samples around the surface intersection to obtain the derivatives , while Atzmon et al [ 28 ] use a sample network to relate samples ’ positions to network parameters and thus achieve an improved generalization ability . More interestingly , by adopting occupancy representations , DVR [ 11 ] and IDR [ 9 ] show volumetric rendering is inherently differentiable so that network parameters can be optimized directly with derived analytic gradients . Different from methods aforementioned above , GOF is a generative model for 3D-aware image synthesis that can learn 3D representations from a set of 2D images with unknown camera poses . Generative 3D-aware image synthesis . In order to synthesize 3D consistent images , researchers have explored a lot on how to incorporate 3D representations into the classical GAN model [ 1 ] . Some methods [ 30–32 ] resort to learning from 3D data directly , yet the requirement of 3D supervision limits their practical applicability . A more appealing alternative is thus learn from unposed 2D images in an unsupervised manner . Preceding works along this line of research adopt voxels as their intermediate 3D representations [ 33–35 ] and achieve explicit control over the pose of synthesized images . Inspired by the superior representation capacity of radiance fields over voxels , recent attempts [ 6 , 7 , 36 ] have replaced voxels with neural radiance fields [ 5 ] to improve the fidelity of synthesized 3D consistent images . Despite the striking performance , these models , referred to as generative radiance fields ( GRAFs ) , tend to predict diffused object surfaces , which impedes its applicability in various downstream tasks . In this work , GOF aims at resolving this problem of GRAFs by combining them with the perspective of occupancy networks [ 8 ] and recent successes of recovering smooth and accurate shapes from natural images [ 37–40 ] . Recently , three concurrent works , UNISURF [ 10 ] , NeuS [ 41 ] and VolSDF [ 42 ] , also combine implicit surfaces and radiance fields in a unified framework , sharing similar spirits with our proposed GOF but different in tasks and focuses . Specifically , They focus on multi-view 3D reconstruction and attempt to alleviate the requirement of training-time precise masks through the integration of radiance fields and occupancy representations . Nevertheless , they still require images with ground-truth poses for training . By contrast , GOF targets on the challenging task of 3D-aware image synthesis , where the synthesized images should be not only natural and vivid , but also consistent in the 3D space . By integrating radiance fields and occupancy representations , GOF is able to facilitate the convergence of GRAFs and ensure the compactness of learned object surfaces . Compared to existing GRAFs , the applicability of GOF is thus significantly broadened . 3 Methodology . We propose generative occupancy fields ( GOF ) , a novel synthesis model , belonging to generative radiance fields and aiming to learn from unposed images . Conditioned on a latent code z ∼ pz , our generator gθ can generate a 3D radiance field R , from which we can render a realistic image with a sampled camera pose ξ ∼ pξ and simultaneously recover smooth and compact object surfaces . In the following , we first present the background of neural radiance fields , and then introduce our proposed GOF model in detail . 3.1 Neural Radiance Fields . We adopt neural radiance fields ( NeRF ) as our explicit 3D representation for image synthesis , owing to its strong performance in novel view synthesis on complex scenes . NeRF represents a static scene as per-point volume densities and view-dependent RGB colors . Given a 3D point x ∈ R3 in space and a view direction d ∈ R3 , NeRF capitalizes on a multi-layer perceptron ( MLP ) to predict the volume density σ ( x ) ∈ R and the emitted color c ( x , d ) ∈ R3 . To render a novel view for the scene , NeRF leverages the classic volume rendering technique [ 43 ] to estimate the color of each pixel . It starts by accumulating the colored densities of N points { xi = o + tid } sampled within near and far bounds [ tn , tf ] along the camera ray r ( t ) = o + td , where o stands for the camera origin . The integrated color is then estimated via alpha composition as follows : Ĉ ( r ) = N∑ i=1 Ti ( 1− exp ( − σ ( xi ) δi ) ) c ( xi , d ) , where Ti = exp ( − i−1∑ j=1 σ ( xj ) δj ) , ( 1 ) where δi = |xi+1 − xi| is the distance between adjacent points . Note that , equation ( 1 ) is naturally differentiable and NeRF can be directly optimized through the reconstruction error of existing views . 3.2 3D Surface-Aware Image Synthesis via Generative Occupancy Fields . To apply NeRF as the 3D representation , the proposed generative occupancy fields ( GOF ) incorporates an additional latent code z ∼ pz into NeRF , such that synthesizing an image follows a reformulated cumulative rendering process : Ĉ ( r , z ) = N∑ i=1 Ti ( 1−exp ( −σθ ( xi , z ) δi ) ) cθ ( xi , d , z ) , where Ti = exp ( − i−1∑ j=1 σθ ( xj , z ) δj ) . ( 2 ) However , directly training GOF according to Eq . ( 2 ) fails to maintain the surface compactness as reported in previous approaches [ 6 , 7 ] . Actually , such a defect arises from an inevitable “ shape-color ambiguity ” of the cumulative rendering process , i.e. , small perturbations on surfaces still lead to realistic RGB images which are enough to fool the discriminator . Owing to the constrained range of poses seen at training , the discriminator is less motivated to further concentrate the color weights wi = Ti ( 1− exp ( −σθ ( xi , z ) δi ) ) aforementioned in Fig . 1 ( a ) on the exact object surface . On the other hand , we observe that although leading to diffused surfaces at the end , color weights wi gradually concentrate around the object surface as the training proceeds . Inspired by this observation , in GOF we propose a training-time operation to facilitate the concentration of color weights wi . The basic idea is gradually shrinking the sample region in the cumulative rendering process from the entire volume to a narrow interval around the surface , so that color weights are enforced to continuously move towards the exact surface . To enable the proposed training-time shrinking process , GOF is required to locate the surface by thresholding the predicted densities σθ ( x , z ) , assuming points on the surface have the largest densities , However , values of the densities predicted in generative radiance fields could range from 0 to 50 , making it hard to determine an effective threshold τ during the whole training period . On the other hand , in the cumulative rendering process shown in Eq . ( 2 ) , we found that the intermediate alpha values used for numerical stability inherently fall in a fixed value range as : αθ ( xi , z ) = 1− exp ( −σθ ( xi , z ) δi ) ∈ [ 0 , 1 ] . ( 3 ) More importantly , these alpha values αθ ( x , z ) come close to 1 for points in the occupied space while approaching 0 for points in the free space , making them resemble the occupancy values [ 8 ] in both quantity and semantics . Inspired by the similarity , we thus propose to reformulate generative radiance fields by predicting alpha values αθ ( x , z ) directly instead of volume densities σθ ( x , z ) . In the mean time , we reinterpret the alpha values as occupancy values , and subsequently locate surfaces with root-finding , a more effective strategy originated in occupancy networks [ 11 ] . According to the above reformulation and reinterpretation , we thus dub our method as Generative Occupancy Fields . As GOF estimates alpha values instead of volume densities , the original volume rendering process in Eq . ( 2 ) conditioned on the latent code z is reformulated as Ĉ ( r , z ) = N∑ i=1 αθ ( xi , z ) ∏ j < i ( 1− αθ ( xj , z ) ) cθ ( xi , d , z ) , ( 4 ) where the value range of αθ ( xi , z ) is guaranteed with a sigmoid function . And to locate the surface via root-finding , for a specific ray r ( t ) = o+td we will evenly sampleM points { xk = o+tkd ; k = 1 , ... , M } that partition the entire volume [ tn , tf ] into M equally-spaced bins . After obtaining the corresponding alpha values { αθ ( xk , z ) ; k = 1 , ... , M } by querying the generator gθ , the surface S is located in the kS-th bin where αθ changes for the first time from free space ( αθ < τ ) to occupied space ( αθ < τ ) : kS = argmin k ( αθ ( xk , z ) < τ ≤ αθ ( xk+1 , z ) ) , ( 5 ) where τ is a pre-defined threshold . In practice , we empirically set τ as 0.5 . In order to find the surface point xs = o + tsd more precisely , we further apply the above secant method iteratively for ms times , resulting in a fine-grained bin [ xkS , xkS+1 ] . It ’ s worth noting that the M sampled points are only used for root-finding . Thus they do not require the computation of gradients in the implementation . Based on the located surface S , we can thus successfully conduct the proposed shrinking process , which is schematically elaborated in Fig . 2 . Specifically , when sampling N points for Eq . ( 4 ) at each training step , we will only sample within a region neighboring the surface [ ts −∆ , ts + ∆ ] : ti ∼ U [ ts −∆ + 2i− 2 N ∆ , ts −∆ + 2i N ∆ ] , where i = 1 , 2 , ... , N. ( 6 ) ∆ is the sampling interval , which is set to ∆init = ( tf − tn ) /2 at the beginning , a half of the distance between near tn and far bounds tf . And it will decrease monotonically with an exponential decay rate γ until it drops to a pre-defined minimal value ∆min . Formally , ∆n = max ( ∆init exp ( −γn ) , ∆min ) for n-th decaying step . Additionally , during training when the estimated ts is too close to the near or the far bound so that the sampling region [ ts −∆ , ts + ∆ ] exceeds the original range [ tn , tf ] , we will shift the region [ ts −∆ , ts + ∆ ] back to within [ tn , tf ] . As shown in Fig . 2 , at the beginning of training , points sampled for Eq . ( 4 ) will cover the entire volume , leading to dispersed gradients which facilitate the convergence of GOF . And as the training goes , the predicted surface will become more and more accurate , which is the outcome of gradually refining the sampling region , and in turn also makes the above shrinking operation valid . Thanks to the dedicated shrinking process , the color weights wi can successfully concentrate on the object surface as illustrated in Fig . 1 ( a ) . As a result , GOF is capable of synthesizing highfidelity images in a 3D-consistent manner and simultaneously capturing compact object surfaces . During inference , to synthesize an image under a random camera pose ξ ∼ pξ , the generator gθ will fetch a truncated latent code ẑ and sample N points { xi } on each ray within the minimal region [ ts−∆min , ts+∆min ] for the rendering as in Eq. ( 4 ) . An important benefit of learning a compact object surface is that we can effectively reduce the number of sampled points for rendering , even using only one point on each ray , i.e . the surface point . As shown in Fig . 1 ( b ) , the image rendered with only the surface point is almost indistinguishable from that with multiple points . Such equivalence can be guaranteed theoretically when ∆min → 0 , and we include the proof in the supplementary material .

Summary: The authors propose a novel formulation of the 3D GAN problem using hybrid volume-surface approach. The method leads to a more accurate shape reconstruction allowing for more efficient rendering. The reconstructed image quality is also comparable or better.