Unsupervised Generative 3D Shape Learning from Natural Images

1 INTRODUCTION . Generative Adversarial Nets ( GAN ) ( see Goodfellow et al . ( 2014 ) ) have become the gold standard generative model over the years . Their capability is demonstrated in many data sets of natural images . These generative models can create sample images that are nearly indistinguishable from real ones . GANs do not need to make assumptions on the data formation other than applying a neural network on a latent noise input . They can also operate in a fully unsupervised way . GANs have strong theoretical foundations since the beginning . Goodfellow et al . ( 2014 ) demonstrate that , under suitable conditions , GANs learn to generate samples with the same probability distribution as samples in the training data set have . However , there is one notable drawback of GANs compared to classical generative models like Gaussian Mixture Models by Xu & Jordan ( 1996 ) or Naive Bayes classifiers by McCallum et al . ( 1998 ) . Classical models are interpretable , but GANs are not . Interpretability is achieved in classical models by making strong assumptions on the data formation process and by using them on interpretable engineered features . Our work combines the best of both worlds for 3D shape learning . We keep the advantages of GANs : unsupervised training , applicability on real datasets without feature engineering , the theoretical guarantees and simplicity . We also make our representation interpretable , as our generator network provides a 3D mesh as an output . To make the learning possible , we make the assumption that natural images are formed by a differentiable renderer . This renderer produces fake images by taking the 3D mesh , its texture , a background image and a viewpoint as its input . During training a discriminator network is trained against the generator in an adversarial fashion . It tries to classify whether its input is fake or comes from the training dataset . The key observation is that a valid 3D object should look realistic from multiple viewpoints . This image realism is thus enforced by the GAN training . This idea was first applied for generating 3D shapes in Gadelha et al . ( 2017 ) . Their pioneering work had several limitations though . It was only applicable to synthetic images , where the background mask is available and produced only black and white images . Our method works on natural images and we do not need silhouettes as supervision signal . From a theoretical point of view , image realism means that the images lie inside the support of the probability distribution of natural images ( see Ledig et al . ( 2017 ) , where image realism was used for super-resolution ) . AmbientGAN of Bora et al . ( 2018 ) proves that one can recover an underlying latent data distribution , when a known image formation function is applied to the data , thus achieving data realism by enforcing image realism . Our work and that of Gadelha et al . ( 2017 ) are special cases of AmbientGAN for 3D shape learning . However this task in general does not satisfy all assumptions in Bora et al . ( 2018 ) , which results in ambiguities in the training . We resolve these issues in our paper by using suitable priors . We summarize our contributions below : • For the first time , to the best of our knowledge , we provide a procedure to build a generative model that learns explicit 3D representations in an unsupervised way from natural images . We achieve that using a generator network and a renderer trained against a discriminator in a GAN setting . Samples from our model are shown in Figure 1 . • We introduce a novel differentiable renderer , which is a fundamental component to obtain a high-quality generative model . Notably , it is differentiable with respect to the 3D vertex coordinates . The gradients are not approximated , they can be computed exactly even at the object boundaries and in the presence of self-occlusions . • We analyze our learning setup in terms of the ambiguities in the learning task . These ambiguities might derail the training to undesirable results . We show that these problems can only be solved when one uses labels or prior knowledge on the data distribution . Finally , we provide practical solutions to overcome the problems that originate from the ambiguities . 2 RELATED WORK . Before we discuss prior work , we would like to state clearly what we mean by supervised , unsupervised and weakly supervised learning in this paper . Usually , supervised learning is understood as using annotated data for training , where the annotation was provided by human experts . In some scenarios the annotation comes from the image acquisition setup . Often these approaches are considered unsupervised , because the annotation is not produced by humans . Throughout our paper we consider supervision based on the objective function and its optimization , and not based on the origin of the annotation . To that purpose , we use the notion of the target and training objective . The target objective is defined as the function that measures the performance of the trained model f , i.e. , Ltarget ( f ) = E [ lt ( f ( x ) , y ) ] , ( 1 ) 3D Method Supervision Signal 3D Representation Capabilities . where lt is the loss function , x , y are the data and labels respectively . The training objective is the function that is optimized during the training , They are defined for the supervised , weakly supervised and unsupervised case as Lsupervised ( f ) = E [ ls ( f ( x ) , y ) ] , ( 2 ) Lweakly ( f ) = E [ lw ( f ( x ) , yw ) ] , ( 3 ) Lunsupervised ( f ) = E [ lu ( f ( x ) ) ] , ( 4 ) where yw denotes a subset of labels and the loss functions ls , lw and lu may be different from lt . In the case of most supervised tasks ( e.g . in classification ) the target is the same as the training objective ( cross-entropy ) . Another example is monocular depth estimation . In this case , the inputs are monocular images and the model can be trained using stereo images . This makes it a weakly supervised method under the definitions above . The GAN training is unsupervised , it has the same target and training objectives ( the Jensen-Shannon divergence ) , and thus even the target objective does not use labels . In Table 1 , we show the most relevant prior work with a detailed list of used supervision signals . There , we consider the full training scenario from beginning to end . For example if a method uses a pre-trained network from another previous work in its setup , we consider it supervised if the pre-trained network used additional annotation during its training . A very successful 3D generative model is the Basel face model introduced by Paysan et al . ( 2009 ) , which models the 3D shape of faces as a linear combination of base shapes . To create it , classical 3D reconstruction techniques ( see Hartley & Zisserman ( 2003 ) ) and laser scans were used . This model is used in several methods ( e.g . Geng et al . ( 2019 ) ; Gecer et al . ( 2019 ) ; Sanyal et al . ( 2019 ) ; Sela et al . ( 2017 ) ; Tran et al . ( 2017 ) ; Genova et al . ( 2018 ) ; Tewari et al . ( 2017 ) ) for 3D reconstruction tasks by regressing its parameters . There are other methods based on GANs , such as those of Wu et al . ( 2016 ) ; Achlioptas et al . ( 2018 ) , and autoencoders ( e.g . Ranjan et al . ( 2018 ) ) that learn 3D representations by directly using 3D as the supervision signal . The most relevant papers similar to our work are the ones that use differentiable rendering and using randomly sampled viewpoints to enforce image realism . There are GAN based methods by Gadelha et al . ( 2017 ) ; Henzler et al . ( 2018 ) ; Rajeswar et al . ( 2019 ) and Variational autoencoder based methods by Rezende et al . ( 2016 ) ; Henderson & Ferrari ( 2019 ) ; Kato & Harada ( 2019 ) . However , these methods are only applicable on synthetic data or use weak supervision for training as shown in Table 1 . Our method can also be interpreted as a way to disentangle the 3D and the viewpoint factors from images in an unsupervised manner . Reed et al . ( 2015 ) used image triplets for the task . They utilized an autoencoder to reconstruct an image from the mixed latent encodings of other two images . Mathieu et al . ( 2016 ) and Szabó et al . ( 2018 ) only use image pairs that share the viewpoint attribute , thus reducing part of the supervision in the GAN training . Similarly , StyleGAN Karras et al . ( 2019 ) and Hu et al . ( 2018 ) use mixing latent variables for unsupervised disentangling . HoloGAN ( Nguyen-Phuoc et al . ( 2019 ) ) is a method for disentangling objects and viewpoints using latent shape representations . In contrast we learn an explicite 3D mesh representation , which can be used in traditional rendering pipelines . By using this rendering in the training , we demonstrate the disentangling of the 3D shape from the viewpoint without any labels and also guarantee interpretability and consistency across viewpoints . An important component of our model is the renderer . Differentiable renderers like Neural mesh renderer Kato et al . ( 2018 ) or OpenDR Loper & Black ( 2014 ) . have been used along with neural networks for shape learning tasks . Differentiability is essential to make use of the gradient descent algorithms commonly employed in the training of neural networks . We introduce a novel renderer , where the gradients are exactly computed and not approximated at the object boundaries . 3 METHOD . We are interested in building a mapping from a random vector to a 3D object ( texture and vertex coordinates ) , and a background image . We call these three components the scene representation . To generate a view of this scene we also need to specify a viewpoint , which we call the camera representation . The combination of the scene and camera representations is also referred to as simply the representation , and it is used by a differential renderer R to construct an image . We train a generator G in an adversarial fashion against a discriminator D by feeding zero-mean Gaussian samples z as input ( see Fig . 2 ) . The objective of the generator is to map Gaussian samples to scene representations m that result in realistic renderings xf for the viewpoint v used during training . The discriminator then receives the fake xf and real xr images as inputs . The GAN training solves the following optimization problem , min G max D Exr∼pr [ log ( D ( xr ) ) ] + Ez∼N , v∼pv [ log ( 1−D ( R ( G ( z ) , v ) ) ) ] , ( 5 ) where xf = R ( G ( z ) , v ) are the generated fake images , m = G ( z ) are the 3D shape representations and xr are the real data samples . The renderer R is a fixed function , i.e. , without trainable parameters , but differentiable . The viewpoints v are randomly sampled from a known viewpoint distribution . In practice , G and D are neural networks and the optimization is done using a variant of stochastic gradient descent ( SGD ) .

Summary: The paper tries to solve the problem of recovering the 3D structure from 2D images. To this end, it describes a GAN-type model for generating realistic images, where the generator disentangles shape, texture, and background. Most notably, the shape is represented in three dimensions as a mesh made of triangles. The final image is rendered by a fixed differentiable renderer from a randomly-sampled viewpoint. This allows the model to learn to generate realistic 3D shapes, even though it is trained only using 2D images.