Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

1 INTRODUCTION . The ability to infer 3D properties such as geometry , texture , material , and light from photographs is key in many domains such as AR/VR , robotics , architecture , and computer vision . Interest in this problem has been explosive , particularly in the past few years , as evidenced by a large body of published works and several released 3D libraries ( TensorflowGraphics by Valentin et al . ( 2019 ) , Kaolin by J. et al . ( 2019 ) , PyTorch3D by Ravi et al . ( 2020 ) ) . The process of going from images to 3D is often called “ inverse graphics ” , since the problem is inverse to the process of rendering in graphics in which a 3D scene is projected onto an image by taking into account the geometry and material properties of objects , and light sources present in the scene . Most work on inverse graphics assumes that 3D labels are available during training ( Wang et al. , 2018 ; Mescheder et al. , 2019 ; Groueix et al. , 2018 ; Wang et al. , 2019 ; Choy et al. , 2016 ) , and trains a neural network to predict these labels . To ensure high quality 3D ground-truth , synthetic datasets such as ShapeNet ( Chang et al. , 2015 ) are typically used . However , models trained on synthetic datasets often struggle on real photographs due to the domain gap with synthetic imagery . To circumvent these issues , recent work has explored an alternative way to train inverse graphics networks that sidesteps the need for 3D ground-truth during training . The main idea is to make ∗indicates equal contribution . graphics renderers differentiable which allows one to infer 3D properties directly from images using gradient based optimization , Kato et al . ( 2018 ) ; Liu et al . ( 2019b ) ; Li et al . ( 2018 ) ; Chen et al . ( 2019 ) . These methods employ a neural network to predict geometry , texture and light from images , by minimizing the difference between the input image with the image rendered from these properties . While impressive results have been obtained in Liu et al . ( 2019b ) ; Sitzmann et al . ( 2019 ) ; Liu et al . ( 2019a ) ; Henderson & Ferrari ( 2018 ) ; Chen et al . ( 2019 ) ; Yao et al . ( 2018 ) ; Kanazawa et al . ( 2018 ) , most of these works still require some form of implicit 3D supervision such as multi-view images of the same object with known cameras . Thus , most results have been reported on the synthetic ShapeNet dataset , or the large-scale CUB ( Welinder et al. , 2010 ) bird dataset annotated with keypoints from which cameras can be accurately computed using structure-from-motion techniques . On the other hand , generative models of images appear to learn 3D information implicitly , where several works have shown that manipulating the latent code can produce images of the same scene from a different viewpoint ( Karras et al. , 2019a ) . However , the learned latent space typically lacks physical interpretation and is usually not disentangled , where properties such as the 3D shape and color of the object often can not be manipulated independently . In this paper , we aim to extract and disentangle 3D knowledge learned by generative models by utilizing differentiable graphics renderers . We exploit a GAN , specifically StyleGAN ( Karras et al. , 2019a ) , as a generator of multi-view imagery to train an inverse graphics neural network using a differentiable renderer . In turn , we use the inverse graphics network to inform StyleGAN about the image formation process through the knowledge from graphics , effectively disentangling the GAN ’ s latent space . We connect StyleGAN and the inverse graphics network into a single architecture which we iteratively train using cycle-consistency losses . We demonstrate our approach to significantly outperform inverse graphics networks on existing datasets , and showcase controllable 3D generation and manipulation of imagery using the disentangled generative model . 2 RELATED WORK . 3D from 2D : Reconstructing 3D objects from 2D images is one of the mainstream problems in 3D computer vision . We here focus our review to single-image 3D reconstruction which is the domain of our work . Most of the existing approaches train neural networks to predict 3D shapes from images by utilizing 3D labels during training , Wang et al . ( 2018 ) ; Mescheder et al . ( 2019 ) ; Choy et al . ( 2016 ) ; Park et al . ( 2019 ) . However , the need for 3D training data limits these methods to the use of synthetic datasets . When tested on real imagery there is a noticeable performance gap . Newer works propose to differentiate through the traditional rendering process in the training loop of neural networks , Loper & Black ( 2014 ) ; Kato et al . ( 2018 ) ; Liu et al . ( 2019b ) ; Chen et al . ( 2019 ) ; Petersen et al . ( 2019 ) ; Gao et al . ( 2020 ) . Differentiable renderers allow one to infer 3D from 2D images without requiring 3D ground-truth . However , in order to make these methods work in practice , several additional losses are utilized in learning , such as the multi-view consistency loss whereby the cameras are assumed known . Impressive reconstruction results have been obtained on the synthetic ShapeNet dataset . While CMR by Kanazawa et al . ( 2018 ) and DIB-R by Chen et al . ( 2019 ) show real-image 3D reconstructions on CUB and Pascal3D ( Xiang et al. , 2014 ) datasets , they rely on manually annotated keypoints , while still failing to produce accurate results . A few recent works , Wu et al . ( 2020 ) ; Li et al . ( 2020 ) ; Goel et al . ( 2020 ) ; Kato & Harada ( 2019 ) , explore 3D reconstruction from 2D images in a completely unsupervised fashion . They recover both 3D shapes and camera viewpoints from 2D images by minimizing the difference between original and re-projected images with additional unsupervised constraints , e.g. , semantic information ( Li et al . ( 2020 ) ) , symmetry ( Wu et al . ( 2020 ) ) , GAN loss ( Kato & Harada ( 2019 ) ) or viewpoint distribution ( Goel et al . ( 2020 ) ) . Their reconstruction is typically limited to 2.5D ( Wu et al . ( 2020 ) ) , and produces lower quality results than when additional supervision is used ( Goel et al . ( 2020 ) ; Li et al . ( 2020 ) ; Kato & Harada ( 2019 ) ) . In contrast , we utilize GANs to generate multi-view realistic datasets that can be annotated extremely efficiently , which leads to accurate 3D results . Furthermore , our model achieves disentanglement in GANs and turns them into interpretable 3D neural renderers . Neural Rendering with GANs : GANs ( Goodfellow et al. , 2014 ; Karras et al. , 2019a ) can be regarded as neural renderers , as they take a latent code as input and “ render ” an image . However , the latent code is sampled from a predefined prior and lacks interpretability . Several works generate images with conditions : a semantic mask ( Zhu et al. , 2017 ) , scene layout Karacan et al . ( 2016 ) , or a caption ( Reed et al. , 2016 ) , and manipulate the generated images by modifying the input condition . Despite tremendous progress in this direction , there is little work on generating images through an interpretable 3D physics process . Dosovitskiy et al . ( 2016 ) synthesizes images conditioned on object style , viewpoint , and color . Most relevant work to ours is Zhu et al . ( 2018 ) , which utilizes a learnt 3D geometry prior and generates images with a given viewpoint and texture code . We differ in three important ways . First , we do not require a 3D dataset to train the 3D prior . Second , the texture in our model has 3D physical meaning , while Zhu et al . ( 2018 ) still samples from a prior . We further control background while Zhu et al . ( 2018 ) synthesizes objects onto white background . Disentangling GANs : Learning disentangled representations has been widely explored , Lee et al . ( 2020 ) ; Lin et al . ( 2019 ) ; Perarnau et al . ( 2016 ) . Representative work is InfoGAN Chen et al . ( 2016 ) , which tries to maximize the mutual information between the prior and the generated image distribution . However , the disentangled code often still lacks physical interpretability . Tewari et al . ( 2020 ) transfers face rigging information from an existing model to control face attribute disentanglement in the StyleGAN latent space . Shen et al . ( 2020 ) aims to find the latent space vectors that correspond to meaningful edits , while Härkönen et al . ( 2020 ) exploits PCA to disentangle the latent space . Parallel to our work , Zhang et al . ( 2021 ) ; Li et al . ( 2021 ) attempt to interpret the semantic meaning of StyleGAN latent space . In our work , we disentangle the latent space with knowledge from graphics . 3 OUR APPROACH . We start by providing an overview of our approach ( Fig . 1 ) , and describe the individual components in more detail in the following sections . Our approach marries two types of renderers : a GANbased neural “ renderer ” and a differentiable graphics renderer . Specifically , we leverage the fact that the recent state-of-the-art GAN architecture StyleGAN by Karras et al . ( 2019a ; b ) learns to produce highly realistic images of objects , and allows for a reliable control over the camera . We manually select a few camera views with a rough viewpoint annotation , and use StyleGAN to generate a large number of examples per view , which we explain in Sec . 3.1 . In Sec . 3.2 , we exploit this dataset to train an inverse graphics network utilizing the state-of-the-art differentiable renderer , DIBR by Chen et al . ( 2019 ) in our work , with a small modification that allows it to deal with noisy cameras during training . In Sec . 3.3 , we employ the trained inverse graphics network to disentangle StyleGAN ’ s latent code and turn StyleGAN into a 3D neural renderer , allowing for control over explicit 3D properties . We fine-tune the entire architecture , leading to significantly improved results . 3.1 STYLEGAN AS SYNTHETIC DATA GENERATOR . We first aim to utilize StyleGAN to generate multi-view imagery . StyleGAN is a 16 layers neural network that maps a latent code z ∈ Z drawn from a normal distribution into a realistic image . The code z is first mapped to an intermediate latent code w ∈ W which is transformed to w∗ = ( w∗1 , w ∗ 2 , ... , w ∗ 16 ) ∈ W ∗ through 16 learned affine transformations . We call W ∗ the transformed latent space to differentiate it from the intermediate latent space W . Transformed latent codes w∗ are then injected as the style information to the StyleGAN Synthesis network . Different layers control different image attributes . As observed in Karras et al . ( 2019a ) , styles in early layers adjust the camera viewpoint while styles in the intermediate and higher layers influence shape , texture and background . We provide a careful analysis of all layers in Appendix . We empirically find that the latent code w∗v : = ( w ∗ 1 , w ∗ 2 , w ∗ 3 , w ∗ 4 ) in the first 4 layers controls camera viewpoints . That is , if we sample a new code w∗v but keep the remaining dimensions of w ∗ fixed ( which we call the conten code ) , we generate images of the same object depicted in a different viewpoint . Examples are shown in Fig . 2 . We further observe that a sampled codew∗v in fact represents a fixed camera viewpoint . That is , if we keep w∗v fixed but sample the remaining dimensions of w ∗ , StyleGAN produces imagery of different objects in the same camera viewpoint . This is shown in columns in Fig . 2 . Notice how aligned the objects are in each of the viewpoints . This makes StyleGAN a multi-view data generator ! “ StyleGAN ” multi-view dataset : We manually select several views , which cover all the common viewpoints of an object ranging from 0-360 in azimuth and roughly 0-30 in elevation . We pay attention to choosing viewpoints in which the objects look most consistent . Since inverse graphics works require camera pose information , we annotate the chosen viewpoint codes with a rough absolute camera pose . To be specific , we classify each viewpoint code into one of 12 azimuth angles , uniformly sampled along 360 deg . We assign each code a fixed elevation ( 0◦ ) and camera distance . These camera poses provide a very coarse annotation of the actual pose – the annotation serves as the initialization of the camera which we will optimize during training . This allows us to annotate all views ( and thus the entire dataset ) in only 1 minute – making annotation effort neglible . For each viewpoint , we sample a large number of content codes to synthesize different objects in these views . Fig . 2 shows 2 cars , and a horse and a bird . Appendix provides more examples . Since DIB-R also utilizes segmentation masks during training , we further apply MaskRCNN by He et al . ( 2017 ) to get instance segmentation in our generated dataset . As StyleGAN sometimes generates unrealistic images or images with multiple objects , we filter out “ bad ” images which have more than one instance , or small masks ( less than 10 % of the whole image area ) .

Summary: This paper proposes to couple a GAN, an inverse graphics network, and a differentiable renderer. The authors base their work on StyleGAN, and use the observation that a specific part of the latent code corresponds to camera view-point to rapidly annotate a large amount of synthetic images with approximate camera pose. They then use these images and rough annotations to train the inverse graphics network to provide 3D and texture data. The differentiable renderer is used to synthesize 2D images from 3D, which can be compared to the input for consistency. In a second step, the authors use the inferred 3D data to disentangle the latent space of StyleGAN to allow to use it as a controllable renderer.