A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis

1 Introduction . Advanced deep generative models , e.g. , StyleGAN [ 1 , 2 ] and BigGAN [ 3 ] , have achieved great successes in natural image synthesis . While producing impressive results , these 2D representationbased models can not synthesize novel views of an instance in a 3D-consistent manner . They also fall short of representing an explicit 3D object shape . To overcome such limitations , researchers have proposed new deep generative models that represent 3D scenes as neural radiance fields [ 4 , 5 ] . Such 3D-aware generative models allow explicit control of viewpoint while preserving 3D consistency during image synthesis . Perhaps a more fascinating merit is that they have shown the great potential of learning 3D shapes in an unsupervised manner from just a collection of unconstrained 2D images . If we could train a 3D-aware generative model that learns accurate 3D object shapes , it would broaden various downstream applications such as 3D shape reconstruction and image relighting . Existing attempts for 3D-aware image synthesis [ 4 , 5 ] tend to learn coarse 3D shapes that are inaccurate and noisy , as shown in Fig.1 ( a ) . We found that such inaccuracy arises from an inevitable ambiguity inherent in the training strategy adopted by these methods . In particular , a form of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . regularization , which we refer to as `` multi-view constraint '' , is used to enforce the 3D representation to look realistic from different viewpoints . The constraint is commonly implemented by first projecting the generator ’ s outputs ( e.g. , radiance fields [ 6 ] ) to randomly sampled viewpoints , and then feeding them to a discriminator as fake images for training . While such a constraint enables these models to synthesize images in a 3D-aware manner , it suffers from the shape-color ambiguity , i.e. , small variations of shape could lead to similar RGB images that look equally plausible to the discriminator , as the color of many objects is locally smooth . Consequently , inaccurate shapes are concealed under this constraint . In this work , we propose a novel shading-guided generative implicit model ( ShadeGAN ) to address the aforementioned ambiguity . In particular , ShadeGAN learns more accurate 3D shapes by explicitly modeling shading , i.e. , the interaction of illumination and shape . We believe that an accurate 3D shape should look realistic not only from different viewpoints , but also under different lighting conditions , i.e. , satisfying the `` multi-lighting constraint '' . This idea shares similar intuition with photometric stereo [ 7 ] , which shows that accurate surface normal could be recovered from images taken under different lighting conditions . Note that the multi-lighting constraint is feasible as real-world images used for training are often taken under various lighting conditions . To fulfill this constraint , ShadeGAN takes a relightable color field as the intermediate representation , which approximates the albedo but does not necessarily satisfy viewpoint independence . The color field is shaded under a randomly sampled lighting condition during rendering . Since image appearance via such a shading process is strongly dependent on surface normals , inaccurate 3D shape representations will be much more clearly revealed than in earlier shading-agnostic generative models . Hence , by satisfying the multi-lighting constraint , ShadeGAN is encouraged to infer more accurate 3D shapes as shown in Fig.1 ( b ) . The above shading process requires the calculation of the normal direction via back-propagation through the generator , and such calculation needs to be repeated dozens of times for a pixel in volume rendering [ 4 , 5 ] , introducing additional computational overhead . Existing efficient volume rendering techniques [ 8 , 9 , 10 , 11 , 12 ] mainly target static scenes , and could not be directly applied to generative models due to their dynamic nature . Therefore , to improve the rendering speed of ShadeGAN , we formulate an efficient surface tracking network to estimate the rendered object surface conditioned on the latent code . This enables us to save rendering computations by just querying points near the predicted surface , leading to 24 % and 48 % reduction of training and inference time without affecting the quality of rendered images . Comprehensive experiments are conducted across multiple datasets to verify the effectiveness of ShadeGAN . The results show that our approach is capable of synthesizing photorealistic images while capturing more accurate underlying 3D shapes than previous generative methods . The learned distribution of 3D shapes enables various downstream tasks like 3D shape reconstruction , where our approach significantly outperforms other baselines on the BFM dataset [ 13 ] . Besides , modeling the shading process enables explicit control over lighting conditions , achieving image relighting effect . Our contributions can be summarized as follows : 1 ) We address the shape-color ambiguity in existing 3D-aware image synthesis methods with a shading-guided generative model that satisfies the proposed multi-lighting constraint . In this way , ShadeGAN is able to learn more accurate 3D shapes for better image synthesis . 2 ) We devise an efficient rendering technique via surface tracking , which significantly saves training and inference time for volume rendering-based generative models . 3 ) We show that ShadeGAN learns to disentangle shading and color that well approximates the albedo , achieving natural relighting effects in image synthesis . 2 Related Work . Neural volume rendering . Starting from the seminal work of neural radiance fields ( NeRF ) [ 6 ] , neural volume rendering has gained much popularity in representing 3D scenes and synthesizing novel views . By integrating coordinate-based neural networks with volume rendering , NeRF performs high-fidelity view synthesis in a 3D consistent manner . Several attempts have been proposed to extend or improve NeRF . For instance , [ 14 , 15 , 16 ] further model illumination , and learn to disentangle reflectance with shading given well-aligned multi-view and multi-lighting images . Besides , many studies accelerate the rendering of static scenes from the perspective of spatial sparsity [ 8 , 9 ] , architectural design [ 10 , 11 ] , or efficient rendering [ 17 , 12 ] . However , it is not trivial to apply these illumination and acceleration techniques to volume rendering-based generative models [ 5 , 4 ] , as they typically learn from unposed and unpaired images , and represent dynamic scenes that change with respect to the input latent codes . In this work , we take the first attempt to model illumination in volume rendering-based generative models , which serves as a regularization for accurate 3D shape learning . We further devise an efficient rendering technique for our approach , which shares similar insight with [ 12 ] , but does not rely on ground truth depth for training and it is not limited to a small viewpoint range . Generative 3D-aware image synthesis . Generative adversarial networks ( GANs ) [ 18 ] are capable of generating photorealistic images of high-resolution , but lack explicit control over camera viewpoint . In order to enable them to synthesis images in a 3D-aware manner , many recent approaches investigate how 3D representations could be incorporated into GANs [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 5 , 4 , 28 , 29 , 30 ] . While some works directly learn from 3D data [ 19 , 20 , 21 , 22 , 30 ] , in this work we focus on approaches that only have access to unconstrained 2D images , which is a more practical setting . Several attempts [ 23 , 24 , 25 ] adopt 3D voxel features with learned neural rendering . These methods produce realistic 3D-aware synthesis , but the 3D voxels are not interpretable , i.e. , they can not be transferred to 3D shapes . By leveraging differentiable renderer , [ 26 ] and [ 27 ] learn interpretable 3D voxels and meshes respectively , but [ 26 ] suffers from limited visual quality due to low voxel resolution while the learned 3D shapes of [ 27 ] exhibit noticeable distortions . The success of NeRF has motivated researchers to use radiance fields as the intermediate 3D representation in GANs [ 5 , 4 , 28 ] . While achieving impressive 3D-aware image synthesis with multi-view consistency , the extracted 3D shapes of these approaches are often imprecise and noisy . Our main goal in this work is to address the inaccurate shape by explicitly modeling illumination in the rendering process . This innovation helps achieve better 3D-aware image synthesis with broader applications . Unsupervised 3D shape learning from 2D images . Our work is also related to unsupervised approaches that learn 3D object shapes from unconstrained , monocular view 2D images . While several approaches use external 3D shape templates or 2D key-points as weak supervisions to facilitate learning [ 31 , 32 , 33 , 34 , 35 , 36 , 37 ] , in this work we consider the harder setting where only 2D images are available . To tackle this problem , most approaches adopt an “ analysis-by-synthesis ” paradigm [ 38 , 39 , 40 ] . Specifically , they design photo-geometric autoencoders to infer the 3D shape and viewpoint of each image with a reconstruction loss . While succeed in learning the 3D shapes for some object categories , these approaches typically rely on certain regularization to prevent trivial solutions , like the commonly used symmetry assumption on object shapes [ 39 , 40 , 31 , 32 ] . Such assumption tends to produce symmetric results that may overlook the asymmetric aspects of objects . Recently , GAN2Shape [ 41 ] shows that it is possible to recover 3D shapes for images generated by 2D GANs . This method , however , requires inefficient instance-specific training , and recovers depth maps instead of full 3D representations . The proposed 3D-aware generative model also serves as a powerful approach for unsupervised 3D shape learning . Compared with aforementioned autoencoder-based methods , our GAN-based approach avoids the need to infer the viewpoint of each image , and does not rely on strong regularizations . In experiments , we demonstrate superior performance over recent state-of-the-art approaches Unsup3d [ 39 ] and GAN2Shape [ 41 ] . 3 Methodology . We consider the problem of 3D-aware image synthesis by learning from a collection of unconstrained and unlabeled 2D images . We argue that modeling shading , i.e. , the interaction of illumination and shape , in a generative implicit model enables unsupervised learning of more accurate 3D object shapes . In the following , we first provide some preliminaries on neural radiance fields ( NeRF ) [ 6 ] , and then introduce our shading-guided generative implicit model . 3.1 Preliminaries on Neural Radiance Fields . As a deep implicit model , NeRF [ 6 ] uses an MLP network to represent a 3D scene as a radiance field . The MLP fθ : ( x , d ) → ( σ , c ) takes a 3D coordinate x ∈ R3 and a viewing direction d ∈ S2 as inputs , and outputs a volume density σ ∈ R+ and a color c ∈ R3 . To render an image under a given camera pose , each pixel color C of the image is obtained via volume rendering along its corresponding camera ray r ( t ) = o+ td with near and far bounds tn and tf as below : C ( r ) = ∫ tf tn T ( t ) σ ( r ( t ) ) c ( r ( t ) , d ) dt , where T ( t ) = exp ( − ∫ t tn σ ( r ( s ) ) ds ) . ( 1 ) In practice , this volume rendering is implemented with a discretized form using stratified and hierarchical sampling . As this rendering process is differentiable , NeRF could be directly optimized via posed images of a static scene . After training , NeRF allows the rendering of images under new camera poses , achieving high-quality novel view synthesis .

Summary:  This paper proposes a GAN capable of generating a relightable radiance field when trained on an unlabeled dataset of face images. The relightable radiance field generator is conditioned on the 3D location and a latent code and outputs a volume density field (à la NeRF) and an "albedo" field (I strongly advise against the use of this word; see below) that allows explicit control of the light location. The training is conducted in the GAN style, where the training signal comes from only a discriminator loss. The intuition is that the generator is better off generating a meaningful 3D volume that, when rendered from a random camera view, generates a photorealistic face image that falls into the training data distribution, than generating some intricately-designed volume that gives faces of different identities when viewed from different angles.  The authors demonstrate that the model is able to learn 3D shapes in an unsupervised fashion from just 2D images. The work is mainly compared against pi-GAN, which is shown to suffer from what the authors call "color-shape ambiguities." For efficient rendering, the authors also propose an auxiliary network that predicts the surface location given the latent code and a viewing direction, such that the network can sample around the predicted surface to avoid expensive sampling of unoccupied space.