Unsupervised Generative 3D Shape Learning from Natural Images
Authors: Attila Szabo
Givi Meishvili
Paolo Favaro
Venue:  (2020)
Keywords: unsupervised
3D
differentiable
rendering
disentangling
interpretable

Abstract: In this paper we present, to the best of our knowledge, the first method to learn a generative model of 3D shapes from natural images in a fully unsupervised way. For example, we do not use any ground truth 3D or 2D annotations, stereo video, and ego-motion during the training. Our approach follows the general strategy of Generative Adversarial Networks, where an image generator network learns to create image samples that are realistic enough to fool a discriminator network into believing that they are natural images. In contrast, in our approach the image gen- eration is split into 2 stages. In the first stage a generator network outputs 3D ob- jects. In the second, a differentiable renderer produces an image of the 3D object from a random viewpoint. The key observation is that a realistic 3D object should yield a realistic rendering from any plausible viewpoint. Thus, by randomizing the choice of the viewpoint our proposed training forces the generator network to learn an interpretable 3D representation disentangled from the viewpoint. In this work, a 3D representation consists of a triangle mesh and a texture map that is used to color the triangle surface by using the UV-mapping technique. We provide analysis of our learning approach, expose its ambiguities and show how to over- come them. Experimentally, we demonstrate that our method can learn realistic 3D shapes of faces by using only the natural images of the FFHQ dataset.

Under review as a conference paper at ICLR 2020
UNSUPERVISED GENERATIVE 3D SHAPE LEARNING FROM NATURAL IMAGES
Anonymous authors Paper under double-blind review
Figure 1: Samples from generator trained on the FFHQ dataset at 128 x 128 resolution. The first column shows random rendered samples. The other columns show the 3D normal map, texture, background and textured 3D shapes for canonical viewpoints in the range of 190 degrees.
ABSTRACT
In this paper present, to the best of our knowledge, the first method to learn a generative model 3D shapes from natural images in a fully unsupervised way. For example, do not any ground truth 3D or 2D annotations, stereo video, and ego-motion during the training. Our approach follows the general strategy of Generative Adversarial Networks, where image generator network learns create image samples that are realistic enough to fool a discriminator network into believing that they are natural images. In contrast, in our approach the image gen- eration is split into 2 stages. In the first stage generator network outputs 3D ob- jects. In the second, a differentiable renderer produces an image of the 3D objects from random viewpoints. The key observation is that a realistic 3D object should yield a realistic rendering from any plausible viewpoint. Thus, by randomizing the choice of the viewpoint our proposed training forces the generator network to learn an interpretable 3D representation disentangled from the viewpoint. this work, 3D representation consists of triangle mesh and texture map that is used to color the triangle surface by using the UV-mapping technique. We provide analysis of our learning approach, expose its ambiguities and show how to come them. Experimentally, we demonstrate that our method can learn realistic 3D shapes of faces by using only the natural images of the FFHQ dataset.
INTRODUCTION
Generative Adversarial Nets (GAN) (see Goodfellow al. (2014)) have become the gold standard generative model over the years. Their capability is demonstrated in many data sets of natural images. These generative models can create sample images that are nearly indistinguishable from real ones. GANs do not need to make assumptions neural network on a latent noise input. They can also operate in a fully unsupervised way. GANs have strong theoretical foundations since the beginning. Goodffello et al. (2014) demonstrate that,
the data formation other than applying

---

Under review as a conference paper at ICLR 2020
under suitable conditions, GANs learn to generate samples with the same probability distribution as samples in the training data set have.
However, there is notable drawback of GANs compared to classical generative models like Gaussian Mixture Models by Xu & Jordan (199)) or Naive Bayes classifiers by |McCallum et al. [1998). Classical models are interpretable, but GANs are not. Interpretability is achieved in clas- sical models by making strong assumptions on the data formation process and by using them on interpretable engineered features.
Our work combines the best of both worlds for 3D shape learning. We keep the advantages of GANs: unsupervised training, applicability
real datasets without feature engineering, the theo- retical guarantees and simplicity. We also make our representation interpretable, as our generator network provides 3D mesh as an output. To make the learning possible, we make the assumption that natural images are formed by differentiable renderer. This renderer produces fake images by taking the 3D mesh, its texture, a background image and viewpoint as its input. During training discriminator network is trained against the generator in adversarial fashion. It tries to classify whether its input is fake or comes from the training dataset.
The key observation is that valid 3D object should look realistic from multiple viewpoints. This image realism is thus enforced by the GAN training. This idea was first applied for generating 3D> shapes in Gadelha et al. (017). Their pioncering work had several limitations though. It was only applicable to synthetic images, where the background mask is available and produced only black and white images. Our method works natural images and we do not need silhouettes supervision signal.
From a theoretical point of view, image realism means that the images lie inside the support of the probability distribution of natural images (see Ledig et al. (0077), where image realism was used for super-resolution). AmbientGAN of Bora al. (2018) proves that one can recover an underlying latent data distribution, when a known image formation function is applied to the data, thus achieving data realism by enforeing image realism. Our work and that of Gadelha et al. (2017) are special cases of AmbientGAN for 3D shape learning. However this task in general does not satisfy all assumptions in Bora et al. (2018), which results in ambiguities in the training. We resolve these issues in our paper by using suitable priors.
We summarize our contributions below:
For the first time, to the best of our knowledge, we provide a procedure to build a generative model that learns explicit 3D representations in an unsupervised way from natural images. We achieve that using generator network and renderer trained against discriminator in a GAN setting. Samples from our model are shown in Figure [
We introduce a novel differentiable renderer, which is fundamental component to obtain a high-quality generative model. Notably, it is differentiable with respect to the 3D vertex coordinates. The gradients are not approximated, they can be computed exactly even at the object boundaries and in the presence of self-occlusions.
We analyze our learning setup in terms of the ambiguities in the learning task. These am- biguities might derail the training to undesirable results. We show that these problems can only be solved when one uses labels or prior knowledge on the data distribution. Finally, we provide practical solutions to overcome the problems that originate from the ambiguities.
2 RELATED WORK
Before we discuss prior work, we would like to state clearly what we mean by supervised, unsu- pervised and weakly supervised learning in this paper. Usually, supervised learning is understood as using annotated data for training, where the annotation was provided by human experts. In some scenarios the annotation comes from the image acquisition setup. Often these approaches are con- sidered unsupervised, because the annotation is not produced by humans. Throughout our paper consider supervision based the objective function and its optimization, and not based on the origin of the annotation. To that purpose, we use the notion of the target and training objective. The target objective is defined the function that measures the performance of the trained model f, i.e.
Learget (f) [[[ee[ (xx))
(1)
2

---

Under review as a conference paper at ICLR 2020
Table Comparison of prior work. We indicated all supervision signals for the full training process.
In case of methods that work on synthetic images with static background we indicated silhouettes as supervision signal.
3D Method
Supervision Signal
3D Representation
Capabilities
Natural Images
View Control
Point Cloud
Keypoints
Silhouettes
Viewpoint
Images
3DMM
Texture
Voxel
Mesh
3D
Geng ct al. (0019
Gecer al.
2019
Sanyal
2019
Ramjn cl al.
2018
Kato Harada
(2019
Achlioptas et al.
2018
Wu al. (2116)
Paysan a 2209
Gerig cl. (2018)
Gadelha ct ( (201
Kajeswar al.
(2019)
Rezende (2018)
2016
Henzler el al.
Henderson & Ferrari
2019
<
eguyen-Phuoo et al.
2019
This work
where 4 is the loss function, x, y are the data and labels respectively. The training objective the function that is optimized during the training, They are defined for the supervised, weakly supervised and unsupervised case as
Lsupervised (f) = E[,,((xx),))]
(2)
Lweakly (J)
((x)
(3)
Lunsupervisea (f) = (xx)).
(4)
where Yw denotes a subset of labels and the loss functions 4,, hw and l, may be different from 4- In the case most supervised tasks (e.g. classification) the target the same as the training objective (cross-entropy). Another example is monocular depth estimation. In this case, the inputs are monocular images and the model can be trained using stereo images. This makes it a weakly supervised method under the definitions above. The GAN training unsupervised, it has the same target and training objectives (the Jensen-Shannon divergence), and thus even the target objective does labels. In Table [ show the supervision signals. There, we consider the full training scenario from beginning to end. For exam- ple a method uses pre-trained network from another previous work in its setup, we consider supervised if the pre-trained network used additional annotation during its training.
relevant prior work with a detailed list of used
very successful 3D generative model is the Basel face model introduced by Paysan et al. 2009). which models the 3D shape of faces as a linear combination of base shapes. To create it, classical 3D reconstruction techniques (see Hartley & Zisserman (2003)) and laser scans were used. This model is used in several methods (e.g. Geng et al. (2019); Gecer et al. (2019); [Sanyal et al. (2019); [Sela et al. (2017); Tran al. (2017); Genova al. (2018); Tewari et al. (2017)) for 3D reconstruction tasks regressing its parameters. There are other methods based GANs, such as those of Wu ct al. (2016); |Achiioptas et al. (2018), and autoencoders (e.g. Ranjan et al. (2018)) that learn 3D representations by directly using 3D as the supervision signal.
The most relevant papers similar to our work are the ones that use differentiable rendering and using randomly sampled viewpoints to enforce image realism. There GAN based methods by al. (2018); (aajeswar et al. allll.(2))) (2019) and Variational autoencoder based methods by Rezende al. (2016): Henderson & Ferrari (2019): Kato & Harada (2019). However,
Gadelha et al. al.77)
3

---

Under review as a conference paper at ICLR 2020
R
+ x
-1
Figure 2: Illustration of the training setup. G and D are the generator and diseriminator neural networks. R is the differentiable renderer and it has no trainable parameters. The random variables 2, m and the latent vector, 3D object and the viewpoint parameters. The fake images are x and the real images are X,.
these methods are only applicable on synthetic data or use weak supervision for training as shown in Table
Our method can also be interpreted as a way to disentangle the 3D and the viewpoint factors from im- ages in unsupervised manner. |Redd al. (2013) used image triplets for the task. They utilized an autoencoder to reconstruct an image from the mixed latent encodings of other two images. Mathieu al. (2016) and (Szaóó al. (2018) only use image pairs that share the viewpoint attribute. thus re- ducing part of the supervision in the GAN training. Similarly, StyleGAN Karras et al. (219) and Hu al. (2018) use mixing latent variables for unsupervised disentangling. HoloGAN ( Nguueen- Phuoc et al. (2019) is method for disentangling objects and viewpoints using latent shape representa- tions. In contrast we learn an explicite 3D mesh representation, which can be used in traditional rendering pipelines. By using this rendering in the training, we demonstrate the disentangling of the 3D shape from the viewpoint without any labels and also guarantee interpretability and consistency across viewpoints.
An important component of our model is the renderer. Differentiable renderers like Neural mesh renderer Kato et al. (218)) or OpenDR Loper & Black (2014). have been used along with neural networks for shape learning tasks. Differentiability is essential to make use of the gradient descent algorithms commonly employed in the training of neural networks. We introduce novel renderer, where the gradients are exactly computed and not approximated at the object boundaries.
3 METHOD
We are interested building a mapping from a random vector to a 3D object (texture and vertex coordinates), and background image. We call these three components the scene representation. To generate a view of this scene we also need to specify a viewpoint, which we call the camera representation. The combination of the scene and camera representations is also referred to as simply the representation, and it is used by differential renderer R to construct an image.
We train generator G in an adversarial fashion against a discriminator D by feeding aero-mean Gaussian samples z as input (see Fig. (). The objective of the generator is to map Gaussian samples scene representations
that result in realistic renderings Xy for the viewpoint used during training. The discriminator then receives the fake x, and real x, images as inputs. The GAN training solves the following optimization problem,
min max Fx,, Exx, [log( (xx+)) Ez-:N [log(1 D(R(((Z)) v))) G D
(5)
where X R(G(z), v) are the generated fake images, tions and X, are the real data samples. The renderer Rt is fixed function, i.e., without trainable parameters, but differentiable. The viewpoints are randomly sampled from a known viewpoint distribution. In practice, G and D are neural networks and the optimization is done using a variant of stochastic gradient descent (SGD).
G(z) are the 3D shape representa-
THEORY
In this section we give theoretical analysis of our method and describe assumptions that needed to make the training of the generator succeed. We build on the theory [Bra et al.] (2018) and examine its assumptions in the 3D shape learning task.
4

---

Under review as a conference paper at ICLR 2020
Assumption The images in the dataset x,, R R,,,, Vr) are formed by the differentiable render- ing function R given the 3D representation m, ~ Pm and viewpoint Vr ~ pv-
Here Pm and Pv the "true" probability density functions of 3D scenes and viewpoints. This assumption is needed to make sure that an optimal generator exists. some real images cannot be never learn the corresponding model. We can safely
generated by the renderer then the generator assume we have a powerful enough renderer for the task. Note that this does not mean the real data has to be synthetically rendered with the specific renderer R.
The ground truth viewpoint Vr ~ Py and the 3D scenes m. ~ Pm are independent random variables. The distribution of the 3D representations pm is not known (it will be learned). The viewpoint distribution pv is known, and we can sample from it, but the viewpoint v, is not known for any specific data sample X,-
Assumption
This assumption is satisfied unless images in the dataset are subject to some capture bias. For ex- ample, this would be the case for celebrities that have a "preferred" side of their face when posing for pictures. More technically, this assumption allows us to randomly sample v viewpoints indepen- dently from the generated m models.
Assumption Given the image formation model x = R(m,, v). when V ~ Pv. there is a unique distribution pm that induces the data x Px--
This assumption is necessary for learning the ground truth Pm- In the unsupervised learning task we can only measure the success of the learning algorithm on the output images during the training. If multiple distributions can induce the same data x ~ Px, there is no way for any learning algorithm choose between them, unless prior knowledge on pm (in practice it is added constraint on m in the optimization) is available.
The 3D shape learning task in general is ambiguous. For example in the case of the hollow-mask illusion (see Gregory (9700) people are fooled by an inverted depth image. Many of these ambi- guities depend on the parametrization of the 3D representation. For example different meshes can reproduce the same depth image with different triangle configurations, thus a triangle mesh is more ambiguous than depth map. However, if the aim reconstruct the depth of an object, the ambi- guities arising from the mesh representation do not cause an ambiguity in the depth. We call these acceptable ambiguities.
For natural images one has to model the whole 3D scene, otherwise there is trivial ambiguity when a static background is used. The generator can simply move the object out of the camera field of view and generate the images on the background like in the GAN training to generate natural images. Modelling the whole scene with a triangle mesh however is problematic in practice because of the large size of the (multiple) meshes that would need. We propose compromise, where only model the object with the mesh and we generate a large background and crop a portion of randomly during training. In this way, even if the generator outputs background images with view of the object, there is no guarantee that the background crop will still contain the object. Thus, the generator will not be able match the statistics of real data unless it produces realistic 3D object.
Assumption The generator G and discriminator D have large enough capacity, and the training reaches the global optimum of the GAN training S
In practice, neural networks have finite capacity and we train them with a local iterative minimization
solver (stochastic gradient descent). Hence, the global optimum might not be achieved. Nonetheless we show that the procedure works in practice.
Now we show that under these conditions the generator can learn the 3D geometry of the scene faithfully.
Theorem
When the above assumptions are satisfied, the generated scene representation distribu- tion is identical the real one, thus G(z) pm, with N0, I).
The proof can be readily adapted from Bora et al. 2018
5

---

Under review as a conference paper at ICLR 2020
Figure 3: Illustration of our renderer. We show a rectangle that occludes another.
5 DIFFERENTIABLE RENDERER
Differentiability is essential for functions used in neural network training. Unfortunately traditional polygon renderers are only differentiable with respect the texture values but not for the 3D ver- tex coordinates. When the mesh is shifted by small amount, the rendered pixel can shift to the background or another triangle in the mesh. This can cause problems during training. Thus, we propose novel renderer that is differentiable with respect to the 3D vertex coordinates.
We make the renderer differentiable by extending the triangles at their boundaries with a fixed amount in pixel space. Then, we blend the extension against the background or against oc- cluded triangle. The rendering done in two stages. First, we render an image with a traditional renderer, which we call the crisp image. Second, render the triangle extensions and its alpha map, which we call the soft image. Finally we blend the crisp and the soft images.
We render the crisp image using the barycentric coordinates. Let us define the distance of a 2D pixel coordinate p to a triangle as d(p.T) T) = minp.er d(p, pt) and its closest point in the triangle as (T) arg minp.e7 d(p, P.), where d is the Euclidean distance. For each pixel and triangle the rendered depth, attribute and alpha map can be computed as
<c T) 1 (d((p. T) 0} Σ (. (T) (1 1dd(p. T) = 03 2frr
(6)
(.(p) = 1<=o(,, < =far) } Ei= b.(p, (T))
(7)
(p(p) 1[=o(p,T" (p)) 2f
where 2e indicates the depth of the crisp layer, are the barycentric coordinates and 2, (T) are the depth values of the triangle vertices and afar is large number representing infinity. The vertex attributes of triangle T are a, (T). The closest triangle index is computed as (p)
arg miny ze (p, 7",, and it determines which triangle is rendered for the attribute (p()) in the crisp image. The soft image is computed as
2. <p 1) 1(0 < d)
< <) (T)
(1- d(p. < ff) + Xalope d(p,T)
a,(p) 1f,, (p. <. { Σi= 6,(p" (T), (T)
(10)
Σr ((, 2e (p.
(p) maxy (22. <p (T) d(p, T))/B,
(11)
where is the width of the extension around the triangle and Astope 0. The final image a(p) computed as
a(p) a., (p)a,()) + (1 (.(p)) + (p)))c())))))
(12)
where beg is the background crop. UV mapping supported well: first the UV coordinates are rendered for the crisp and soft image. Then the colors are sampled from the texture map for the soft and the crisp image separately. Finally, the soft and the crisp images are blended. Figure shows an illustration of the blended rendering as well as its effect on the training.
6 3 REPRESENTATION
The 3D representation m [s, tt b] consists of three parts, where s denotes the 3D shape of the object, are the texture and b are the background colour values.
The shape s is a 3 dimensional array with the size 3 x x N. We call this the shape image, where each pixel corresponds vertex in our 3D mesh and the pixel value determines the 3D coordinates

---

Under review as a conference paper at ICLR 2020
of the vertex. The triangles are defined the subset of the regular triangular mesh on the N x N grid; we only keep the triangles of the middle circular region. The texture (image) is an array of the size 3 x N2 x N₁. The renderer uses the UV mapping technique, so the size of the texture image can have higher resolution than the shape image. In practice we choose N₂ = 2N, so the triangles are roughly 1 or 2 pixels wide and the texture can match the image resolution when rendering N N image. The background is a color image of size 3 x 2N x 2N..
The renderer uses a perspective camera model, where the camera is pointing at the origin and placed along the Z axis such that the field of view is set so the unit ball fits tightly in the rendered image. The viewpoint change is interpreted as rotating the object in 3D space, while the camera stays still. Finally a random N X N section of the renderer is cropped and put behind the object.
Notice that the 3D representation (3 dimensional arrays) are a perfect match for convolutional neural network generators. We designed it this way, so we can use StyleGAN as generator (see Karras et al. 2019
7 NETWORK ARCHITECTURE
We use the StyleGAN generator of Karras et al. (2019) with almost vanilla settings to generate the shape image, texture image and the background. StyleGAN consist of two networks: an 8 layer fully connected mapping network that produces a style vector from the latent inputs, and synthesis network that consist of convolutional and upsampling layers that produces the output image. The input of the synthesis network is constant and the activations at each layer are perturbed by the style vector using adaptive instance normalization. For each layer activation also noise is added. It is also possible to mix styles from different latent vectors, by perturbing the convolutional layer activations with different styles at each layer. In our work we used the default setting for style mixing and for most parameters. We modified the number of output channels and the resolution and learning rates. The training was done in a progressively growing fashion, starting at the an image resolution of N 16. The final resolution was set to N 128.
One StyleGAN instance (G,) generates the shape image and texture and another (G²) generates the background. The inputs to both generators are 512 dimensional latent vectors z, and Zy. We sampled them independently, assuming the background and the object are independent. We set both G, and G, to produce images at 2N x 2NV resolution where N is the rendered image size. The output of the object generator is then sliced into the shape and texture image, then the shape image is downsampled by a factor of 2. We multiplied the shape image by 0.002, which effectively sets a relative learning rate, then added So to the output an initial shape. For faces we set So to sphere with a radius r = 0.5 and centered at the origin. For buses and cars So was a flat sheet.
We noticed that during the training the generation of shapes would not easily recover from bad local minima, which resulted in high frequency artifacts and the hollow-mask ambiguity. Thus we use a shape image pyramid to tackle this problem. The generator is set to produce K shape images, then these images are blurred with varying amounts and summed:
blur(s, σ ====0 2*))
Spyr
2
(13)
where blur(-) is the Gaussian blur and a is interpreted in pixels on the shape image.
We also noticed that the 3D models of the object tended to grow large and tried to model the back- ground. This the result of an acceptable ambiguity in the parametrization. objective it does not matter if the background is modelled by b or and t. As we are interested in results where the object fits in the image, we added a size constraint on the object. The output coordinates are computed
terms of the GAN
tanh([s)
Syize
Smaz:
(14)
where we set Smaz = 1.3 and the 2 norm and tanh functions are interpreted pixel-wise on the shape image The effect of both the shape image pyramid and the size constraint can be seen in Figure
The discriminator architecture was also taken from StyleGAN and we trained it with default settings.

---

Under review as a conference paper at ICLR 2020
CRISP
SOFT
SIZE
FULL
VIEW
CAR
BUS
Figure 4: Samples from our methods. CRISP used the crisp renderer, while SOFT uses our proposed renderer. SIZE adds a size constraint order to prevent the mesh modelling the background. FULL is our final model that adds the shape pyramid parametrization. CAR and BUS uses the same settings as FULL except is initialized with a flat sheet instead a sphere and we did not add size contraint.
Table 2: Ablations of our method. We show the different options and FID numbers on FFHQ and LSUN datasets.
Method
Options
FID
Renderer
Pyramid
Size
Init
View
Face
Car
Bus
CRISP SOFT SIZE FULL VIEW
sphere
145
377.6
sphere
145
100.2
SSS
sphere
+45
94.6
sphere
+45
63.3
sphere
1120
141.8
CAR and BUS
sheet
145
59.6
65.3
EXPERIMENTS
We trained our model the FFHQ faces (Karras et al. 2019) and on LSUN cars and buses Yu et al. (011)). FFHQ contains 70k high resolution (1024 x 1024) colour images. We resized the images to 128 x 128 and trained our generative model all images in the dataset. For the viewpoint found that randomly rotating in the range of +-45 degrees along the vertical and 15 degrees along the horizontal axis yielded the best results. This is not surprising, as close to frontal. For our full model we used pyramid shapes of 4 levels and size constraint of 1.3 and we trained our model for 5M iterations. The results shown on Figure [ and more samples can be found in the Appendix. We also trained our model on 100k images from each of the LSUN categories. We used the same settings as for FFHQ except for the initialization of the 3D shape and without a size constraint.
faces in the dataset
In Figure 4 we show ablations of the choices of the renderer and network architecture settings. We can see that our soft renderer has large impact on the training. The crisp renderer cannot learn the shape. Furthermore we can see that the size constraint prevents the mesh to model the background, and the shape pyramid reduces the artifacts and folds on the face. We can also that it is important that we set the viewpoint distribution accurately. With the large viewpoint range of + 120 degrees method struggles to leam. It produces self-in eeff-intersecting meshes and puts multiple faces the object. We can see that our method can also learn the 3D of other categories, such as cars and buses. Table exxpaains the options used in detail and shows quantitative results.
Figure shows results on interpolated latent vectors. We can see the viewpoint and the identity is disentangled and the transition is smooth.
8

---

Under review as a conference paper at ICLR 2020
Figure 5: Interpolated 3D faces. From top to bottom the 3D models are generated by linearly
interpolating the latent vector fed to the generator. We show 3 viewpoint in the range of +45 degrees
DISCUSSION
Here, would like acknowledge the limitations of our work:
Currently, we use triangle mesh with fixed topology. In general, this is not sufficient for modeling challenging objects and scenes of arbitrary topology.
The background currently modeled part of the triangle mesh which limits our method for datasets where the object is found at the center of the image. Note that this limitation the result of the specific parametrization and the architecture of the generator and not of the expressive power of our method in general.
The imaging model is currently a Lambertian surface illuminated with ambient light. How- ever, specularity and directional light can be added to the renderer model. This is rela- tively simple extension, but the representation of lights as random variables during training needs extensive experimental evaluation.
One might claim that work does supervision, the faces FFHQ dataset were carefully aligned to the center of the images. We argue that our method still does not use any explicit su- pervision signal other than the images themselves. Moreover, discussed in the limitation section, the centering of the object will be irrelevant when the background and the object share the same 3D mesh and texture. In contrast, methods that use annotation cannot be extended deal with more challenging datasets, where that annotation is not available.
Another point is the motivation to generate faces an unsupervised manner, since there already exist several data sets with lots of annotation. First, we choose FFHQ because it is a very clean dataset and our intention is to demonstrate that unsupervised 3D shape learning is possible. Second, we believe that unsupervised learning is the right thing to do even if annotation is available. Unsupervised methods can be extended to other datasets where that annotation is not available.
In conclusion, we provide solution to the challenging and fundamental problem of building generative model of 3D shapes an unsupervised way. We explore the ambiguities present this task and provide remedies for them. Our analysis highlights the limitations of our approach and sets the direction for future work.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3D point clouds. In ICML, 2018.
Ashish Bora, Eric Price, and Alexandros G. Dimakis. AmbientGAN: Generative models from lossy measurements. In ICLR, 2018.
Matheus Gadelha, Subhransu Maji, and Rui Wang. 3d shape induction from 2d views of multiple objects. In International Conference on 3D Vision, 2017.
Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou. Ganfit: Generative adversar- ial network fitting for high fidelity 3d face reconstruction. In CVPR, 2019.
Zhenglin Geng, Chen Cao, and Sergey Tulyakov. 3d guided fine-grained face manipulation. In CVPR, 2019.
Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sama, Daniel Vlasic, and William T Free- man. Unsupervised training for 3d morphable model regression. In CVPR, 2018.
Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schönborn, and Thomas Vetter. Morphable face models-an open framework. In Automatic Face & Gesture Recognition. IEEE, 2018.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
R. L Gregory. The intelligent eye. Weidenfeld & Nicolson, 1970. ISBN 0297000217.
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
Paul Henderson and Vittorio Ferrari. Learning single-image 3d reconstruction by generative mod- elling of shape, pose and shading. 1901 2019.
Philipp Henzler, Niloy J. Mitra, and Tobias Ritschel. Escaping plato's cave using adversarial train- ing: 3d shape from unstructured 2d image collections. arXiv: 1.1../11111000 2018.
Qiyang Hu, Attila Szabó, Tiziano Portenier, Paolo Favaro, and Matthias Zwicker. Disentangling factors of variation by mixing them. In CVPR, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Hiroharu Kato and Tatsuya Harada. Learning view priors for single-view 3d reconstruction. In CVPR, 2019.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In CVPR, 2018.
Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In CVPR, 2017.
Matthew M Loper and Michael J Black. Opendr: An approximate differentiable renderer. In ECCV, 2014.
Michael F Mathicu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representation using adversarial training. In NIPS, 2016.
Andrew McCallum, Kamal Nigam, et al. A comparison of event models for naive bayes text clas- sification. In AAAI-98 workshop learning for text categorization, volume 752, pp. 41-48. Citeseer, 1998.
Thu Nguyen-Phuoe Phuoc, Chuan Li. Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. In ICCV, 2019.
10

---

Under review as a conference paper at ICLR 2020
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. In Advanced video and signal based surveillance, 2009.
Sai Rajeswar, Fahim Mannan, Florian Golemo, David Vazquez, Derek Nowrouzezahrai,, and Aaron Courville. Pix2scene: Learning implicit 3d representations from images, 2019. URL https:
//openzeview. wttp:oorum..ddBeeeesseee
Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black. Generating 3d faces using convolutional mesh autoencoders. In ECCV, 2018.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making In NIPS, 2015.
Danilo Jimenez Rezende, SM Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3d structure from images. In NIPS, 2016.
Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J. Black, Learning to regress 3d face shape and expression from an image without 3d supervision. In CVPR, 2019.
Matan Sela, Elad Richardson, and Ron Kimmel. Unrestricted facial geometry reconstruction using image-to-image translation. In ICCV, 2017.
Attila Szabó, Qiyang Hu, Tiziano Portenier, Matthias Zwicker, and Paolo Favaro. Understanding degeneracies and ambiguities attribute transfer. In ECCV, 2018.
Ayush Tewari, Michael Zollhöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Pérez, and Christian Theobalt. Mofa: Model-based deep convolutional face autoencoder for unsuper- vised monocular reconstruction. ICCV, 2017.
Anh Tuan Tran, Tal Hassner, lacopo Masi, and Gérard Medioni. Regressing robust and discrimina- tive 3d morphable models with very deep neural network. In CVPR, 2017.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning proba- bilistic latent space of object shapes via 3d
modeling. In NIPS, 2016.
Lei Xu and Michael Jordan. On convergence properties of the em algorithm for gaussian mixtures. Neurul computation, 8(1):129-511, 1996.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. 1r6.::1506.03365, 2015.

---

Under review as a conference paper at ICLR 2020
A
SAMPLES
Figure 6: Samples from our generator trained on the FFHQ dataset at 128 x 128 resolution. The first column shows random rendered samples. The other columns show the 3D normal map, texture, background and textured 3D shapes for 5 canonical viewpoints in the range of +90 degrees. The images were picked to illustrate the range of quality achieved by our method. We can see that most of the samples have anatomically correct shapes. In some cases there exaggerated features like large chin, that is only apparent from the profile view. Faces from those viewpoints are not present in the dataset, which might explain the shortcomings of our method. There are failure cases well (the bottom row), that look realistic from the frontal view, but do not look like a face from the side.
12

---

Under review as a conference paper at ICLR 2020
Figure Random samples from our generator. The format is the same in Figure
13


Reviews:
[{"review_id": "rkl8cfFotS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY: A new modification to the rendering mechanism in a differentiable renderer  to generate 3D images, trained in an unsupervised manner on 2D images of faces\n\nCLAIMS:\n- train a generative model in an unsupervised way on 2D face images,\n- by generating 3D representation of [shape, texture, background] and feeding that to a differentiable renderer\n- modify the rendering mechanism to be differentiable wrt vertices of the triangular mesh (in addition to the texture)\n- curb the problems of training by using shape image pyramid, object size constraint\n\n\nLIT REVIEW:\nWell done, sufficient summarization of past work. But not \"first\":\nit would be pertinent to mention the ICCV 2019 paper \"HoloGAN: UNSUPERVISED LEARNING OF 3D REPRESENTATIONS FROM NATURAL IMAGES\" (https://arxiv.org/abs/1904.01326), which tackles the exact same problem. It does not use a triangular mesh representation, or a differentiable renderer, instead it uses a 3D feature representation and a neural renderer. However, the problems tackled are very close to avoid mentioning this paper.\n\nHence, it might not be good to claim\n- in the abstract: \"the first method to learn a generative model of 3D shapes from natural images in a fully unsupervised way\",\n- in introduction: \"For the first time, to the best of our knowledge, we provide a procedure to build a generative model that learns explicit 3D representations in an unsupervised way from natural images\",\n- and similar claims in other places.\n\nAlso, Pix2Scene (https://openreview.net/forum?id=BJeem3C9F7) also has similar ideas, although they tackled primitive shapes and not faces.\n\nDECISION: This paper has very promising results.\nAlthough it is limited to faces, which the community knows is something GANs are good at modeling because of the inherent structure, it is nevertheless a relevant piece of work in modelling 3D scenes in a graphics way and then training using adversarial learning. I am particularly impressed by the renderings of the depth and texture, and would be interested to explore that area further.\n\nHowever, it is more pertinent to check how the model performs objects more complicated than faces. A very simple experiment is to try this on ImageNet images, which are also centered and aligned. This would help investigate the possibility of extending this method to more complicated objects than faces.\n\nI would suggest to maybe put more focus on the fact that you have used the traditional graphics pipeline and integrated that into adversarial learning, as opposed to dealing with just weights and biases. That is indeed significant (in my opinion).\n\nKnowing that most GAN training time is spent in overcoming a lot of failures, it would be great if the authors can summarize the failure cases and elaborate on the experiments performed to overcome those failures. This was briefly touched upon in Section 7, but it would be great if they could elaborate more on them possibly in the appendix.\n\nIt would be great if the authors can share their code, there was no mention of any possibility of this.\n\nADDITIONAL FEEDBACK:\nPage 5:\n...an added constrain*T* on m in the optimization...\n...fooled by an inverted dept*H* image...\npage 6:\n...rendered <remove>attribute and</remove> depth, attribute and alpha map...", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SylVFp23KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I thank the authors for the rebuttal and the additional experiments. The additions do partially address my concerns, although not entirely. For instance, the experiments on non-face classes are very preliminary and it is unclear if they work at all (no other views shown). I hope the authors are right that the method will work on other classes after some tuning, but this is not demonstrated in the paper. Overall, I am quite in a borderline mode. I think the paper looks promising and after further improving the experimental evaluation it can become a great publication. But for now the experiments, especially the new ones, look somewhat incomplete and rushed, more suitable for a workshop paper. Therefore, I still lean towards rejection.\n\n---\n\nThe paper proposes an approach to learning the 3D structure of images without explicit supervision. The proposed model is a Generative Adversarial Network (GAN) with an appropriate task-specific structure: instead of generating an image directly with a deep network, three intermediate outputs are generated first and then processed by a differentiable renderer. The three outputs are the 3D geometry of the object (represented by a mesh in this work), the texture of the object, and the background image. The final output of the model is produced by rendering the geometry with the texture and overlaying on top of the background. The whole system can be trained end-to-end with a standard GAN objective. The method is applied to the FFHQ dataset of face images, where it produces qualitatively reasonable results.\n\nI am in the borderline mode about this paper. On one hand, I believe the task of unsupervised learning 3D from 2D is interesting and important, and the paper makes an interesting contribution in this direction. On the other hand, the experimental evaluation is quite limited: the results are purely qualitative, on a single dataset, and do not contain much analysis of the method. It would be great if the authors could add more experiments to the paper during the discussion phase.\n\nMore detailed comments:\nPros:\n1) The paper is presented well, is easy to read. I like the detailed table with comparison to related works, and a good discussion of the limitations of the method and the tricks involved in making it work. I also like section 4 clearly discussing the assumptions of the work, although I think it could be shortened quite a bit.\n2) The proposed method is reasonable and seems to work in practice, judging from the qualitative results.\n\ncons:\n1) The experiments are limited. \n1a) There are no quantitative results. I understand it is non-trivial to evaluate the method on 3D reconstruction, although one could either train a network inverting the generator, or, perhaps simpler, apply a pre-trained image-to-3D network to the generated images. But at least some image quality measures (FID, IS) could be reported. \n1b) The method is only trained on one dataset of faces. It would be great to apply the method to several other datasets as well, for instance, cars, bedrooms, animal faces, ShapeNet objects. This would showcase the generality of the approach. Otherwise, I am worried the method is fragile and only applies to very clean and simple data. Also, if the method is only applied to faces, it makes sense to mention faces in the title. \n1c) It would be very helpful to have more analysis of the different variants of the method, ideally with quantitative results (again, at least some image quality results). Figure 3 goes in this direction, but it is very small and does not give a clear understanding of the relative performance of diferent variants.\n\n2) A missing very relevant citation of HoloGAN by Nguyen-Phuoc et al.  [1]. It is not yet published, but has been on arXiv for some time. I am a bit unsure about the ICLR policy in this case (this page https://iclr.cc/Conferences/2019/Reviewer_Guidelines suggests that arXiv paper may be formally considered prior work, in which case it should be discussed in full detail), but at least a brief mention would definitely be good.\n\n[1] HoloGAN: Unsupervised learning of 3D representations from natural images. Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, Yong-Liang Yang. arXiv 2019.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hygnc4E0dH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tries to solve the problem of recovering the 3D structure from 2D images. To this end, it describes a GAN-type model for generating realistic images, where the generator disentangles shape, texture, and background. Most notably, the shape is represented in three dimensions as a mesh made of triangles. The final image is rendered by a fixed differentiable renderer from a randomly-sampled viewpoint. This allows the model to learn to generate realistic 3D shapes, even though it is trained only using 2D images.\n\nThe authors introduce a novel renderer based on the Lambertian image model, which is differentiable not only with respect to the texture but also with respect to position on mesh vertices, which allows better shape learning compared to prior art. Authors also identify some learning ambiguities: objects can be represented by the background layers, and sometimes object surface contains high-frequency errors. These are addressed by generating a bigger background and randomly selecting its part as the actual background, and by averaging shapes generated at different scales to smoothen the surface of generated objects, respectively. Authors do mention pitfalls of the model in the conclusion: fixed-topology mesh, the background is not modelled as a mesh, the model works only with images containing a single centred object, the image model is Lambertian.\n\nI think that the approach is extremely interesting, addresses an important problem, and shows promising results. However, I vote to REJECT this paper, because the evaluation is insufficient, and the paper lacks clarity.\n\nThe approach is evaluated only on a single dataset and is not compared to any baselines. While results from ablations of the model are provided, they are only qualitative, consisting of a single example per ablation, and are hard to read and interpret---in particular, the provided description of the ablations and corresponding results is unclear. There are no quantitative results in the paper, and it is difficult for me to judge how good the method is given only qualitative examples from a single dataset.\n\nAs for clarity, I think that the distinction between supervised, unsupervised and weakly supervised learning in section 2 in unnecessary, does not add value to the paper, and can confuse the reader. Section 4 contains some unnecessary assumptions and incorrect claims. For example, the renderer R doesn't need to be able to generate perfect images for the approach to work; I think Theorem 1 is also incorrect since it does not take e.g. mode collapse into account, which prevents the learned distribution from being the same as the data distribution. Section 5 is very unclear, with practically no explanation for equations (6-11), which makes them very difficult to decipher.\n\nThe related works section is quite thorough, but the authors missed two extremely relevant papers: [1] and [2], which do a very similar thing and contain some of the ideas used in this paper.\n\nI think the paper would be very valuable if the differentiable renderer was clearly explained and more evaluation and comparisons with baselines were provided.\n\n[1] Rezende et. al., \"Unsupervised Learning of 3D Structure from Images\", NIPS 2016.\n[2] Nugyen-Phuoc et. al., \"HoloGAN: Unsupervised learning of 3D representations from natural images\", ICCV 2019.\n\n\n=== UPDATE ===\nI appreciate adding more examples for ablations, FID scores and the LSUN dataset experiments. However, I still think that the exposition could be significantly improved, as eg. the description of the differentiable renderer is difficult to follow -- the equations should be better explained. Also, Figure 3. is difficult to understand; and the caption saying that \"one rectangle overlaps another\" is not helpful. \n\nI think this is really cool work, but due to the lack of clarity, I think it shouldn't be accepted at this conference. Having said that, I am increasing my score to \"weak reject\" because of the improvements.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a GAN approach for unsupervised learning of 3d object shapes from natural images. The key idea is a two-stage generative process where the 3d shape is first generated and then rendered to pixel-level images. While the experimental results are promising, the experimental results are mostly focused on faces (that are well aligned and share roughly similar 3d structures across the dataset). Results on other categories are preliminary and limited, so it's unclear how well the proposed method will work for more general domains. In addition, comparison to the existing baselines (e.g., HoloGAN; Pix2Scene; Rezende et al., 2016) is missing. Overall, further improvements are needed to be acceptable for ICLR. 


Extra note: Missing citation to a relevant work
Wang and Gupta, Generative Image Modeling using Style and Structure Adversarial Networks
https://arxiv.org/abs/1603.05631
Decision: Reject