Do 2D GANs Know 3D Shape? Unsupervised 3D Shape Reconstruction from 2D Image GANs

1 INTRODUCTION . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) are capable of modeling the 2D natural image manifold ( Zhu et al. , 2016 ) of diverse object categories with high fidelity . Recall the fact that natural images are actually the projections of 3D objects to the 2D plane , an ideal 2D image manifold should be able to reflect some underlying 3D geometrical properties . For example , it is shown that a GAN could shift the object in its generated images ( e.g. , human faces ) in a 3D rotation manner , as a direction in the GAN image manifold may correspond to viewpoint change ( Shen et al. , 2020 ) . This phenomenon motivates us to ask - “ Is it possible to reconstruct the 3D shape of a single 2D image by exploiting the 3D-alike image manipulation effects produced by GANs ? ” Despite its potential to serve as a powerful method to learn 3D shape from unconstrained RGB images , this problem remains much less explored . Some previous attempts ( Lunz et al. , 2020 ; Henzler et al. , 2019 ; Szabó et al. , 2019 ) also adopt GANs to learn 3D shapes from images , but they rely on explicitly modeling 3D representation and rendering during training ( e.g . 3D voxels , 3D models ) . Due to either heavy memory consumption or additional training difficulty brought by the rendering process , the qualities of their generated samples notably lag behind their 2D GAN counterparts . Another line of works ( Wu et al. , 2020 ; Goel et al. , 2020 ; Tulsiani et al. , 2020 ; Li et al. , 2020 ) for unsupervised 3D shape learning generally learns to infer the viewpoint and shape for each image in an ‘ analysis by synthesis ’ manner . Despite their impressive results , these methods often assume object shapes are symmetric ( symmetry assumption ) to prevent trivial solutions , which is hard to generalize to asymmetric objects such as ‘ building ’ . We believe that existing pre-trained 2D GANs , without above specific designs , already capture sufficient knowledge for us to recover the 3D shapes of objects from 2D images . Since the 3D structure of an instance could be inferred from images of the same instance with multiple viewpoint and lighting variations , our insight is that we may create these variations by exploiting the image manifold captured by 2D GANs . However , the main challenge is to discover well-disentangled semantic directions in the image manifold that control viewpoint and lighting in an unsupervised manner , as manually inspect and annotate the samples in the image manifold is laborious and time-consuming . To tackle the above challenge , we observe that for many objects such as faces and cars , a convex shape prior like ellipsoid could provide a hint on the change of their viewpoints and lighting conditions . Inspired by this , given an image generated by GAN , we employ an ellipsoid as its initial 3D object shape , and render a number of unnatural images , called ‘ pseudo samples ’ , with various randomly-sampled viewpoints and lighting conditions as shown in Fig . 2 . By reconstructing them using the GAN , these pseudo samples could guide the original image towards the sampled viewpoints and lighting conditions in the GAN manifold , producing a number of natural-looking images , called ‘ projected samples ’ . These projected samples could be adopted as the ground truth of the differentiable rendering process to refine the prior 3D shape ( i.e . an ellipsoid ) . To achieve more precise results , we further regard the refined shape as the initial shape and repeat the above steps to progressively refine the 3D shape . With the proposed approach , namely GAN2Shape , we show that existing 2D GANs trained on images only are sufficient to accurately reconstruct the 3D shape of a single image for many object categories such as human faces , cars , buildings , etc . Our method thus is an effective approach for unsupervised 3D shape reconstruction from unconstrained 2D images without any 2D keypoint or 3D annotations . With an improved GAN inversion strategy , our method works not only for GAN samples , but also for real natural images . On the BFM benchmark ( Paysan et al. , 2009 ) , our method outperforms a recent strong baseline designed specifically for 3D shape learning ( Wu et al. , 2020 ) . We also show high-quality 3D-aware image manipulations using the semantic latent directions discovered by our approach , which achieves more accurate human face rotation than other competitors . Our contributions are summarized as follows . 1 ) We present the first attempt to reconstruct the 3D object shapes using GANs that are pre-trained on 2D images only . Our work shows that 2D GANs inherently capture rich 3D knowledge for different object categories , and provides a new perspective for 3D shape generation . 2 ) Our work also provides an alternative unsupervised 3D shape learning method , and does not rely on the symmetry assumption of object shapes . 3 ) We achieve highly photo-realistic 3D-aware image manipulations including rotation and relighting without using any external 3D models . 2 RELATED WORK . Generative Adversarial Networks . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ; Xiangli et al. , 2020 ) have achieved great success in 2D natural image modeling . After a series of improvements , the state-of-the-art GANs like StyleGAN ( Karras et al. , 2019 ; 2020b ) and BigGAN ( Brock et al. , 2019 ) could synthesis images with high fidelity and resolution . These GANs are composed of 2D operations , like 2D convolutions . Recently , there is a surge of interest to model 3D-aware image distributions with 3D GAN architectures and 3D representations ( Nguyen-Phuoc et al. , 2019 ; Wu et al. , 2016 ; Szabó et al. , 2019 ; Henzler et al. , 2019 ; Lunz et al. , 2020 ) . However , due to either heavy memory consumption or increased difficulty in training , there are still gaps between their image qualities and those of 2D GANs . Therefore , in this work , we are interested in investigating whether we can recover the 3D object shape by directly mining the knowledge of 2D GANs . Our resulting method could be viewed as a memory-efficient way for 3D shape and texture generation , which shares the benefits of both 2D and 3D generative models , e.g. , high quality and resolution samples , and a 3D interpretation of contents . 3D-aware Generative Manipulation . Given a pre-trained GAN , there has been works trying to discover the latent space directions that manipulate the image content in a 3D-controllable manner . InterFaceGAN ( Shen et al. , 2020 ) learns to rotate human faces by learning latent direction from manual annotations . In contrast , SeFa ( Shen & Zhou , 2020 ) and GANSpace ( Härkönen et al. , 2020 ) discover meaningful latent directions without supervision , where some of them is coupled with content pose . Recently , StyleRig ( Tewari et al. , 2020 ) , DiscoFaceGAN ( Deng et al. , 2020 ) , and RotateRender ( Zhou et al. , 2020 ) achieves 3D-aware manipulation on human face by using an external 3D human face model , i.e. , 3DMM ( Blanz & Vetter , 1999 ) , as a strong 3D shape prior . Unlike these methods , our method could accurately manipulate on both pose and lighting factors without borrowing external 3D models or supervisions . This allows our method to generalize beyond human faces , like cats and cars . Unsupervised 3D Shape Learning . Unsupervised learning of 3D shapes from raw , monocular view images has attracted increasing attention recently , as it relieves the heavy burden on manual annotations . The key challenge of this problem is the lack of multiple view or lighting images for a single instance . To tackle this problem , a number of works leverage external 3D shape models or weak supervisions like 2D key-points . For example , Gecer et al . ( 2019 ) ; Kanazawa et al . ( 2018a ) ; Sanyal et al . ( 2019 ) ; Shang et al . ( 2020 ) reconstruct the 3D shape of human faces or bodies using 3DMM ( Blanz & Vetter , 1999 ) or SMPL ( Loper et al. , 2015 ) as a prior knowledge . And Tran & Liu ( 2018 ) ; Kanazawa et al . ( 2018b ) adopt the guidance of 2D key-points to learn the 3D shape of face or objects . The works of Tulsiani et al . ( 2020 ) ; Goel et al . ( 2020 ) further discard the need of former supervisions , but still rely on initial category-specific shape templates . Some representative works that rely only on natural 2D images include Sahasrabudhe et al . ( 2019 ) ; Wu et al . ( 2020 ) ; Li et al . ( 2020 ) 1 , which learn to disentangle the viewpoint and shape for each image via autoencoders that explicitly model the rendering process . However , these methods still rely on certain regularization to prevent trivial solutions . For example , Sahasrabudhe et al . ( 2019 ) adopts an autoencoder with a small bottleneck embedding to constrain the deformation field , which tend to produce blur and imprecise results . Wu et al . ( 2020 ) ; Li et al . ( 2020 ) adopt the assumption that many object shapes are symmetric , thus their inferred shapes sometimes miss the unsymmetric aspects of the objects in images . Our method in this work provides an alternative solution for the fore-mentioned ‘ single view ’ challenge , which is to obtain multiple view and lighting images by exploiting the image manifold modeled by 2D GANs . Our method does not necessarily rely on the symmetry assumption on object 1We include methods that require silhouettes in this part . Random sample . shape , thus better captures the asymmetric aspects of objects . In experiments , we demonstrate superior performance over a recent state-of-the-art method Unsup3d ( Wu et al. , 2020 ) . 3 METHODOLOGY . Preliminaries on Generative Adversarial Networks . We first provide some preliminaries on GANs before discussing how we exploit GANs for 3D shape recovery . GANs ( Goodfellow et al. , 2014 ) have achieved great successes in modeling natural images . A GAN consists of a generator G that maps a latent vector z to an image x ∈ R3×H×W , and a discriminator D that distinguishes between generated images and real ones . z is drawn from a prior distribution such as the multivariate normal distribution N ( 0 , I ) d. GANs trained on natural images are able to continuously sample realistic images , thus could be used as an approximation of the natural image manifold ( Zhu et al. , 2016 ) . In this work , our study is mainly based on a representative GAN , StyleGAN2 ( Karras et al. , 2020b ) . The generator of StyleGAN2 consists of two parts , a mapping network F : Z 7→ W that maps the latent vector z in the input latent space Z to an intermediate latent vector w ∈ W , and a synthesis network that mapsw to the output image . In the following sections , we refer to the synthesis network as G. The image manifold of StyleGAN2 thus could be defined as M = { G ( w ) |w ∈ W } . Such a two-stage structure makes it easier to perform GAN-inverson on top of StyleGAN2 . i.e. , searching for a w that best reconstructs a real target image ( Abdal et al. , 2019 ) .

Summary: 1. This is the first work that attempts to reconstruct 3D shape from 2D image in an unsupervised way using GANs. The idea is neat: Use networks to predict four 3D parameters and use GAN to generate / synthesize the images corresponding to a set of parameters. Then these synthesized images can be used as pseudo ground truth to train the 3D parameter network.