Do 2D GANs Know 3D Shape? Unsupervised 3D Shape Reconstruction from 2D Image GANs

1 INTRODUCTION . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) are capable of modeling the 2D natural image manifold ( Zhu et al. , 2016 ) of diverse object categories with high fidelity . Recall the fact that natural images are actually the projections of 3D objects to the 2D plane , an ideal 2D image manifold should be able to reflect some underlying 3D geometrical properties . For example , it is shown that a GAN could shift the object in its generated images ( e.g. , human faces ) in a 3D rotation manner , as a direction in the GAN image manifold may correspond to viewpoint change ( Shen et al. , 2020 ) . This phenomenon motivates us to ask - “ Is it possible to reconstruct the 3D shape of a single 2D image by exploiting the 3D-alike image manipulation effects produced by GANs ? ” Despite its potential to serve as a powerful method to learn 3D shape from unconstrained RGB images , this problem remains much less explored . Some previous attempts ( Lunz et al. , 2020 ; Henzler et al. , 2019 ; Szabó et al. , 2019 ) also adopt GANs to learn 3D shapes from images , but they rely on explicitly modeling 3D representation and rendering during training ( e.g . 3D voxels , 3D models ) . Due to either heavy memory consumption or additional training difficulty brought by the rendering process , the qualities of their generated samples notably lag behind their 2D GAN counterparts . Another line of works ( Wu et al. , 2020 ; Goel et al. , 2020 ; Tulsiani et al. , 2020 ; Li et al. , 2020 ) for unsupervised 3D shape learning generally learns to infer the viewpoint and shape for each image in an ‘ analysis by synthesis ’ manner . Despite their impressive results , these methods often assume object shapes are symmetric ( symmetry assumption ) to prevent trivial solutions , which is hard to generalize to asymmetric objects such as ‘ building ’ . We believe that existing pre-trained 2D GANs , without above specific designs , already capture sufficient knowledge for us to recover the 3D shapes of objects from 2D images . Since the 3D structure of an instance could be inferred from images of the same instance with multiple viewpoint and lighting variations , our insight is that we may create these variations by exploiting the image manifold captured by 2D GANs . However , the main challenge is to discover well-disentangled semantic directions in the image manifold that control viewpoint and lighting in an unsupervised manner , as manually inspect and annotate the samples in the image manifold is laborious and time-consuming . To tackle the above challenge , we observe that for many objects such as faces and cars , a convex shape prior like ellipsoid could provide a hint on the change of their viewpoints and lighting conditions . Inspired by this , given an image generated by GAN , we employ an ellipsoid as its initial 3D object shape , and render a number of unnatural images , called ‘ pseudo samples ’ , with various randomly-sampled viewpoints and lighting conditions as shown in Fig . 2 . By reconstructing them using the GAN , these pseudo samples could guide the original image towards the sampled viewpoints and lighting conditions in the GAN manifold , producing a number of natural-looking images , called ‘ projected samples ’ . These projected samples could be adopted as the ground truth of the differentiable rendering process to refine the prior 3D shape ( i.e . an ellipsoid ) . To achieve more precise results , we further regard the refined shape as the initial shape and repeat the above steps to progressively refine the 3D shape . With the proposed approach , namely GAN2Shape , we show that existing 2D GANs trained on images only are sufficient to accurately reconstruct the 3D shape of a single image for many object categories such as human faces , cars , buildings , etc . Our method thus is an effective approach for unsupervised 3D shape reconstruction from unconstrained 2D images without any 2D keypoint or 3D annotations . With an improved GAN inversion strategy , our method works not only for GAN samples , but also for real natural images . On the BFM benchmark ( Paysan et al. , 2009 ) , our method outperforms a recent strong baseline designed specifically for 3D shape learning ( Wu et al. , 2020 ) . We also show high-quality 3D-aware image manipulations using the semantic latent directions discovered by our approach , which achieves more accurate human face rotation than other competitors . Our contributions are summarized as follows . 1 ) We present the first attempt to reconstruct the 3D object shapes using GANs that are pre-trained on 2D images only . Our work shows that 2D GANs inherently capture rich 3D knowledge for different object categories , and provides a new perspective for 3D shape generation . 2 ) Our work also provides an alternative unsupervised 3D shape learning method , and does not rely on the symmetry assumption of object shapes . 3 ) We achieve highly photo-realistic 3D-aware image manipulations including rotation and relighting without using any external 3D models . 2 RELATED WORK . Generative Adversarial Networks . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ; Xiangli et al. , 2020 ) have achieved great success in 2D natural image modeling . After a series of improvements , the state-of-the-art GANs like StyleGAN ( Karras et al. , 2019 ; 2020b ) and BigGAN ( Brock et al. , 2019 ) could synthesis images with high fidelity and resolution . These GANs are composed of 2D operations , like 2D convolutions . Recently , there is a surge of interest to model 3D-aware image distributions with 3D GAN architectures and 3D representations ( Nguyen-Phuoc et al. , 2019 ; Wu et al. , 2016 ; Szabó et al. , 2019 ; Henzler et al. , 2019 ; Lunz et al. , 2020 ) . However , due to either heavy memory consumption or increased difficulty in training , there are still gaps between their image qualities and those of 2D GANs . Therefore , in this work , we are interested in investigating whether we can recover the 3D object shape by directly mining the knowledge of 2D GANs . Our resulting method could be viewed as a memory-efficient way for 3D shape and texture generation , which shares the benefits of both 2D and 3D generative models , e.g. , high quality and resolution samples , and a 3D interpretation of contents . 3D-aware Generative Manipulation . Given a pre-trained GAN , there has been works trying to discover the latent space directions that manipulate the image content in a 3D-controllable manner . InterFaceGAN ( Shen et al. , 2020 ) learns to rotate human faces by learning latent direction from manual annotations . In contrast , SeFa ( Shen & Zhou , 2020 ) and GANSpace ( Härkönen et al. , 2020 ) discover meaningful latent directions without supervision , where some of them is coupled with content pose . Recently , StyleRig ( Tewari et al. , 2020 ) , DiscoFaceGAN ( Deng et al. , 2020 ) , and RotateRender ( Zhou et al. , 2020 ) achieves 3D-aware manipulation on human face by using an external 3D human face model , i.e. , 3DMM ( Blanz & Vetter , 1999 ) , as a strong 3D shape prior . Unlike these methods , our method could accurately manipulate on both pose and lighting factors without borrowing external 3D models or supervisions . This allows our method to generalize beyond human faces , like cats and cars . Unsupervised 3D Shape Learning . Unsupervised learning of 3D shapes from raw , monocular view images has attracted increasing attention recently , as it relieves the heavy burden on manual annotations . The key challenge of this problem is the lack of multiple view or lighting images for a single instance . To tackle this problem , a number of works leverage external 3D shape models or weak supervisions like 2D key-points . For example , Gecer et al . ( 2019 ) ; Kanazawa et al . ( 2018a ) ; Sanyal et al . ( 2019 ) ; Shang et al . ( 2020 ) reconstruct the 3D shape of human faces or bodies using 3DMM ( Blanz & Vetter , 1999 ) or SMPL ( Loper et al. , 2015 ) as a prior knowledge . And Tran & Liu ( 2018 ) ; Kanazawa et al . ( 2018b ) adopt the guidance of 2D key-points to learn the 3D shape of face or objects . The works of Tulsiani et al . ( 2020 ) ; Goel et al . ( 2020 ) further discard the need of former supervisions , but still rely on initial category-specific shape templates . Some representative works that rely only on natural 2D images include Sahasrabudhe et al . ( 2019 ) ; Wu et al . ( 2020 ) ; Li et al . ( 2020 ) 1 , which learn to disentangle the viewpoint and shape for each image via autoencoders that explicitly model the rendering process . However , these methods still rely on certain regularization to prevent trivial solutions . For example , Sahasrabudhe et al . ( 2019 ) adopts an autoencoder with a small bottleneck embedding to constrain the deformation field , which tend to produce blur and imprecise results . Wu et al . ( 2020 ) ; Li et al . ( 2020 ) adopt the assumption that many object shapes are symmetric , thus their inferred shapes sometimes miss the unsymmetric aspects of the objects in images . Our method in this work provides an alternative solution for the fore-mentioned ‘ single view ’ challenge , which is to obtain multiple view and lighting images by exploiting the image manifold modeled by 2D GANs . Our method does not necessarily rely on the symmetry assumption on object 1We include methods that require silhouettes in this part . Random sample . shape , thus better captures the asymmetric aspects of objects . In experiments , we demonstrate superior performance over a recent state-of-the-art method Unsup3d ( Wu et al. , 2020 ) . 3 METHODOLOGY . Preliminaries on Generative Adversarial Networks . We first provide some preliminaries on GANs before discussing how we exploit GANs for 3D shape recovery . GANs ( Goodfellow et al. , 2014 ) have achieved great successes in modeling natural images . A GAN consists of a generator G that maps a latent vector z to an image x ∈ R3×H×W , and a discriminator D that distinguishes between generated images and real ones . z is drawn from a prior distribution such as the multivariate normal distribution N ( 0 , I ) d. GANs trained on natural images are able to continuously sample realistic images , thus could be used as an approximation of the natural image manifold ( Zhu et al. , 2016 ) . In this work , our study is mainly based on a representative GAN , StyleGAN2 ( Karras et al. , 2020b ) . The generator of StyleGAN2 consists of two parts , a mapping network F : Z 7→ W that maps the latent vector z in the input latent space Z to an intermediate latent vector w ∈ W , and a synthesis network that mapsw to the output image . In the following sections , we refer to the synthesis network as G. The image manifold of StyleGAN2 thus could be defined as M = { G ( w ) |w ∈ W } . Such a two-stage structure makes it easier to perform GAN-inverson on top of StyleGAN2 . i.e. , searching for a w that best reconstructs a real target image ( Abdal et al. , 2019 ) .

Summary: This paper studies an interesting inverse-graphics problem. It proposed a novel method to learn 3D shape reconstruction using pre-trained 2D image generative adversarial networks. Given an image containing one single object of interest, it first predicts the graphics code (e.g., viewpoint, lighting, depth, and albedo) by minimizing the reconstruction error using a differentiable renderer. The next step is to render many pseudo samples by randomization in the viewpoint and lighting space, while keeping the predicted depth and albedo fixed. A pre-trained 2D image GAN is further used to project the pseudo samples to the learned data manifold through GAN-Inversion. Finally, these projected samples are added to the set for the next round optimization. Experimental evaluations have been conducted on several categories including face, car, building, and horse.