Do 2D GANs Know 3D Shape? Unsupervised 3D Shape Reconstruction from 2D Image GANs
Authors: Xingang Pan
Bo Dai
Ziwei Liu
Chen Change Loy
Ping Luo
Venue: ICLR 2021 Oral (2021)
Keywords: Generative Adversarial Network
3D Reconstruction

Abstract: Natural images are projections of 3D objects on a 2D image plane. While state-of-the-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures. And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images? To answer these questions, in this work, we present the first attempt to directly mine 3D geometric cues from an off-the-shelf 2D GAN that is trained on RGB images only. Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner. The core of our framework is an iterative strategy that explores and exploits diverse viewpoint and lighting variations in the GAN image manifold. The framework does not require 2D keypoint or 3D annotations, or strong assumptions on object shapes (e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cats, cars, and buildings. The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation. We quantitatively demonstrate the effectiveness of our approach compared to previous methods in both 3D shape reconstruction and face rotation. Our code is available at https://github.com/XingangPan/GAN2Shape.

Published as a conference paper at ICLR 2021
Do 2D GANS KNOW 3D SHAPE? UNSUPERVISED 3D SHAPE RECONSTRUCTION FROM 2D IMAGE GANS
Xingang Pan'
Bo Dai
Ziwei Liu
Chen Change Loy²
Ping Luo
'he Chinese University of Hong Kong px1170ic cubk. edu hk 'The University of Hong Kong pluc@cs. hku.hk hk
S-Lab, Nanyang Technological University (bo.da'. ziwei. ecloy)entu cdu.
2D GAN sample
3D mesh
Normal
Textured
Rocation
Relighting
Figure The first column shows images generated by off-the-shelff 2D GANs trained on RGB images only, while the rest show that our method can unsupervisedly reconstruct 3D shape (viewed in 3D mesh, surface normal, and texture) given a single 2D image by exploiting the geometric cues contained in GANs. The last two columns depicts 3D-aware image manipulation effects (rotation and relighting) enabled by our framework. More results are provided in the Appendix.
ABSTRACT
Natural images are projections of 3D objects on 2D image plane. While state-of- the-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the un- derlying 3D object structures. And so, how could we exploit such knowledge to recover the 3D shapes of objects in the images? To answer these questions, in this work, we present the first attempt to directly mine 3D geometric cues from an off- the-shelf 2D GAN that is trained on RGB images only. Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used recover 3D shape from a single 2D image in an unsupervised manner. The core of our framework is an iterative strategy that explores and ex- ploits diverse viewpoint and lighting variations the GAN image manifold. The framework does not require 2D keypoint or 3D annotations, tions on object shapes (e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cars, buildings, etc. The recov- ered 3D shapes immediately allow high-quality image editing like relighting and object rotation. We quantitatively demonstrate the effectiveness of our approach compared to previous methods both 3D shape reconstruction and face rotation. Our code is available at
strong assump-
INTRODUCTION
Generative adversarial networks (GANs) (Goodfellow al. 2014 capable of modeling the 2D natural image manifold (Zhu et al. 2016) of diverse object categories with high fidelity. Recall the fact that natural images actually the projections of 3D objects the 2D plane, ideal 2D image

---

Published as a conference paper at ICLR 2021
Render
GAN inversion
Refine
input
Inttial shape
refined shape (stage 1)
stage
stage
pseudo samples
projected samples
Figure 2: Framework outline. Starting with an initial ellipsoid 3D shape (viewed in surface nor- mal), our approach renders various 'pseudo samples' with different viewpoints and lighting condi- tions, GAN inversion is applied to these samples to obtain the 'projected samples', which are used as the ground truth of the rendering process to refine the initial 3D shape. This process is repeated until more precise results are obtained.
manifold should be able to reflect some underlying 3D geometrical properties. For example, it is shown that a GAN could shift the object in its generated images (e.g., human faces) in a 3D rotation manner, as direction in the GAN image manifold may correspond to viewpoint change (Shen et al.) 2020 This phenomenon motivates us ask "Is it possible to reconstruct the 3D shape of a single 2D image by exploiting the 3D-alike image manipulation effects produced by GANs?" Despite its potential to serve as a powerful method to learn 3D shape from unconstrained RGB images, this problem remains much less explored.
Some previous attempts (Lunz et al. 2020 Henzler et al. 2019 Szabó et al. 2019) also adopt GANs to learn 3D shapes from images, but they rely explicitly modeling 3D representation and rendering during training (e.g. 3D voxels, 3D models). Due to either heavy memory consumption or additional training difficulty brought by the rendering process, the qualities of their generated samples notably lag behind their 2D GAN counterparts. Another line of works (Wu et al. 2020 Goel eet al. 2020 Tulsiani et al.) 2020 Li al. 2020) for unsupervised 3D shape learning generally learns to infer the viewpoint and shape for each image in an 'analysis by synthesis" manner. Despite their impressive results, these methods often assume object shapes are symmetric (symmetry assumption) to prevent trivial solutions, which is hard to generalize to asymmetric objects such as 'building'.
We believe that existing pre-trained 2D GANs, without above specific designs, already capture suf- ficient knowledge for us to recover the 3D shapes of objects from 2D images. Since the 3D structure of an instance could be inferred from images of the same instance with multiple viewpoint and light- ing variations, our insight is that we may create these variations by exploiting the image manifold captured by 2D GANs. However, the main challenge to discover well-disentangled semantic di- rections in the image manifold that control viewpoint and lighting in an unsupervised manner, as manually inspect and annotate the samples the image manifold laborious and time-consuming.
To tackle the above challenge, we observe that for many objects such as faces and cars, convex shape prior like ellipsoid could provide a hint on the change of their viewpoints and lighting con- ditions. Inspired by this, given image generated by GAN, we employ 3D object shape, and render a number of unnatural images, called 'pseudo samples', with various aandomly-sampled viewpoints and lighting conditions as shown in Fig. 2 By reconstructing them using the GAN, these pseudo samples could guide the original image towards the sampled view- points and lighting conditions in the GAN manifold, producing number of natural-looking images, called 'projected samples'. These projected samples could be adopted as the ground truth of the differentiable rendering process refine the prior 3D shape (i.e. an ellipsoid). To achieve more precise results, we further regard the refined shape as the initial shape and repeat the above steps to progressively refine the 3D shape.
ellipsoid as its initial
With the proposed approach, namely GAN2Shape, we show that existing 2D GANs trained on im- ages only are sufficient to accurately reconstruct the 3D shape of a single image for many object categories such as human faces, cars, buildings, efc. Our method thus is an effective approach for unsupervised 3D shape reconstruction from unconstrained 2D images withou any 2D keypoint or 3D annotations. With improved GAN inversion strategy, our method works not only for GAN samples, but also for real natural images. On the BFM benchmark (Paysan et al. 2009). our method outperforms recent strong baseline designed specifically for 3D shape learning (Wu et al.] 2020). We also show high-quality 3D-aware image manipulations using the semantic latent directions dis- covered by approach, which achieves more accurate human face rotation than other competitors.
Our contributions are summarized as follows. 1) We present the first attempt to reconstruct the 3D object shapes using GANs that are pre-trained 2D images only. Our work shows that 2D GANs
2

---

Published as a conference paper at ICLR 2021
inherently capture rich 3D knowledge for different object categories, and provides new perspective
for 3D shape generation. 2) Our work also provides an alternative unsupervised 3D shape learning method, and does not rely on the symmetry assumption of object shapes. 3) We achieve highly photo-realistic 3D-aware image manipulations including rotatoon and relighting without using any external 3D models.
2 RELATED WORK
Generative Adversarial Networks. Generative Adversarial Networks (GANs) (Goodfellow et al.
2014 Xiangli et al. 2020) have achieved great success in 2D natural image modeling. After series of improvements, the state-of-the-art GANs like StyleGAN (Karras et al. 2019 2020b) and BigGAN (Brock ct al. 2019) could synthesis images with high fidelity and resolution. These GANs composed of 2D operations, like 2D convolutions. Recently, there is surge of interest to model 3D-aware image distributions with 3D GAN architectures and 3D representations (Nguyen-Phuoc et al. 2019 Wu et 2016, Szabó et al.) 2019 Henzler et al. 2019 Lunz et al. (020). However, due to either heavy memory consumption or icrrassed difficulty in traming, there are still gaps between their image qualities and those of 2D GANs.
Therefore, in this work, we are interested in investigating whether we can recover the 3D object shape by directly mining the knowledge of 2D GANs. Our resulting method could be viewed memory-efficient way for 3D shape and texture generation, which shares the benefits of both 2D and 3D generative models, e.g., high quality and resolution samples, and 3D interpretation of contents.
3D-aware Generative Manipulation. Given a pre-trained GAN, there has been works trying to discover the latent space directions that manipulate the image content in 3D-controllable man- ner. InterFaceGAN (Shen et al. (220)) learns to rotate human faces by learning latent direction from manual annotations. In contrast, SeFa (Shen & Zhou 2020) and GANSpace (Härkönen et al.) 2020) discover meaningful latent directions without supervision, where some of them is coupled with content pose. Recently, StyleRig (Tewari et al.] 2020), DiscoFaceGAAN (Deng et al., 20200. and RotateRender (Zhou al. 2020) achieves 3D-aware manipulation on human face by using an external 3D human face model, L.e., 3DMM (Blanz & Vetter 1999), as a strong 3D shape prior.
Unlike these methods, our method could accurately manipulate on both pose and lighting factors without borrowing external 3D models or supervisions. This allows our method to generalize beyond human faces, like cats and cars.
Unsupervised 3D Shape Learning. Unsupervised learning of 3D shapes from raw, monocular view images has attracted increasing attention recently, as it relieves the heavy burden manual annotations. The key challenge of this problem is the lack of multiple view or lighting images for single instance. To tackle this problem, a number of works leverage external 3D shape models or weak supervisions like 2D key-points. For example. Gecer et al. (2019); Kanazawa et al. (201 8a); Sanyal et al.] (2019): Shang et al. (2020) reconstruct the 3D shape of human faces or bodies using 3DMM (Blanz& & Vetter 1999) or SMPL (Loper et al.) 2015) as a prior knowledge. And Tran & Liu 2018 Kanazawa et al. (2018)) adopt the guidance of 2D key-points learn the 3D shape of face or objects. The works of Tulsiani et al. (2020); Goel et al. (2200) further discard the need of former supervisions, but still rely on initial category-specifer shape templates.
Some representative works that rely only on natural 2D images include (Sahasrabudee et al. (2019); Wu et al. (2020): Li et al. [2020 which learn to disentangle the viewpoint and shape for each image via autoencoders that explicitly model the rendering process. However, these methods still rely on certain regularization to prevent trivial solutions. For example, SShasrabudde et al. (2019) adopts an autoencoder with small bottleneck embedding to constrain the deformation field, which tend to produce blur and imprecise results. Wu et al. (2020); Li et al. (2020) adopt the assumption that many object shapes are symmetric, thus their inferred shapes sometimes miss the unsymmetric aspects of the objects in images.
Our method in this work provides alternative solution for the fore-mentioned 'single view" chal- lenge, which is to obtain multiple view and lighting images by exploiting the image manifold mod- eled by 2D GANs. Our method does not necessarily rely on the symmetry assumption on object
We include methods that require silhouettes in this part.
3

---

Published as a conference paper at ICLR 2021
Random
Reconstruttion Loss
D
light
view
light
depth
albedo
Reconstruction
Lass
Renderer
E
G
generator
Renderer
samples
projected samples (I.)
reconstruction
@a Step 1
(b)
Step
input
Reconstruction
Loss
view
light
Reconstruction
view
light
depth
albedo
Encodar
Loss
Renderer
Encoder Decoder
Foeed Network
Natwerk
(c) Step3
optimized
Figure 3: Method overview. (a) Given a single image, Step 1 initializes the depth with ellipsoid (viewed in surface normal), and optimizes the albedo network A. (b) Step 2 uses the depth and albedo to render "pseudo samples' with various random viewpoint and lighting conditions, and con- ducts GAN inversion them obtain the 'projected samples'. (c) Step 3 refines the depth map by optimizing (V, L, D, A) networks to reconstruct the projected samples. The refined depth and models used the new initialization repeat the above steps.
shape, thus better captures the asymmetric aspects of objects. In experiments, we demonstrate supe- rior performance over a recent state-of-ther method Unsup3d (Wu et al. 2020).
METHODOLOGY
Preliminaries on Generative Adversarial Networks. We first provide some preliminaries on GANs before discussing how exploit GANs for 3D shape recovery. GANs (Goodfellow et al.) 2014) have achieved great successes modeling natural images. GAN consists of generator G that maps latent vector
image E x and discriminator D that distinguishes between generated images and real ones. z is drawn from prior distribution such as the multivari- ate normal distribution (0,, z)d. GANs trained on natural images are able to continuously sample realistic images, thus could be used as an approximation of the natural image manifold (Zhu et al. 2016).
In this work, our study is mainly based on a representative GAN, StyleGAN2 Karras et al., 2020b). The generator of StyleGAN2 consists of two parts, mapping network F : 2 + W that maps the latent vector z in the input latent space Z to an intermediate latent vector w E W, and a synthesis network that maps w the output image. In the following sections, refer to the synthesis network as G. The image manifold of StyleGAN2 thus could be defined as M = =G[w)]uu e W). Such two-stage structure makes it easier to perform GAN-inverson on top of StyleGAN2. i.e. searching for w that best reconstructs real target image (Abdal et al. 2019
3.1 FROM 2D GANS TO 3D SHAPES
In order to mine the viewpoint and lighting cues in the GAN image manifold, need to explicitly model these factors. Here we follow the photo-gcometric autoencoding design in /w et al. (2020) For image I xx adopt function that is designed to predict four factors (d, a, 8,

---

Published as a conference paper at ICLR 2021
including depth map d E R"xw,, albedo image E R viewpoint e R and light direction E S This function thus is implemented with four corresponding sub-networks (D,A, A, V. L) as shown in Fig. (a). The four factors recomposed to form the original image I via a rendering process Φ, which consists of lighting A and reprojection IT steps as follows:
i= (dd, a, v.
-I(A(d, 1), d,
Here A performs shading with albedo a, depth map d, and light direction I, while II performs viewpoint change and generates the image viewed from viewpoint v. The viewpoint change is realized with a differentiable renderer (Kato et al. 2018). We put more details in the Appendix.
Step 1: Using a Weak Shape Prior. As our goal is to explore images of the same instance in the GAN image manifold that have novel viewpoints and lighting conditions, we would like to seek for a hint on the change of viewpoint and lighting to guide the exploration. To achieve this, we make the observation that many objects including faces and cars have a somewhat convex shape prior. We could thus resort to an ellipsoid as a weak prior create 'pseudo samples' with varying viewpoints and lightings as the hint.
Specifically, as shown in Fig (a), given an image sample from the GAN, we initialize the depth map d to have an ellipsoid shape. With an off-the-shelf scene parsing model (Zhao et al. 2017). we could position the ellipsoid to be roughly aligned with the object in the image. Then the albedo network A is optimized with the reconstruction objective CI, Î), where I is calculated via Eq. (. and C is a weighted combination of L1 loss and perceptual loss (Johnson et al. 2016). The viewpoint v and lighting are initialized with a canonical setting, i.e., Up = O and lighting is from the front. This allows to have an initial guess about the four factors do, ao Vo Lo. with dg being the weak shape prior.
Step 2: Sampling and Projecting to the GAN Image Manifold. With the above shape prior an initialization, we are able to create 'pseudo samples' by sampling a number of random viewpoints {v.li = 1.2....,m) and lighting directions {4,1 i 1, 2, ..., m) Specifically, we use {v:} and {4.) along with de and ag to render pseudo samples [I.} via Eq. (1. { and {4:} are randomly sampled from some prior distributions, like the multi-variate normal distribution. As shown in Fig 3 (b), although these pseudo samples have unnatural distortions and shadows, they provide cues on
how the face rotates and how the light changes.
In order to leverage such cues to guide novel viewpoint and lighting direction exploration in the GAN image manifold, we perform GAN inversion to these pseudo samples, i.e., reconstruct them with the GAN generator G. Specifically, we train an encoder E that learns to predict the intermediate latent vector w; for each sample. Unlike prior works that directly predict the latent vector, in our setting the latent vector w for the original sample I is known, thus the encoder E only need to predict an offset Aw; = EI,, that encodes the difference between the pseudo sample I, and the original sample I, which is much easier. Thus the optimization goal is:
1 m m = (I. + w)) + A1))(1
i g arg min
(2)
where m is the number of samples, Pg is the parameter of the encoder F. A is the regularization coefficient, and L' is distance metric for images. Following, Pan et al. (2020), we adopt the LI distance of the discriminator features as the distance metric L' which is shown work better for GAN samples. The regularization term [L))2 is to prevent the latent offset to grow too large during training.
As shown in Fig 3 (b), to achieve better reconstruction, the projected samples {i, G(E(L) + +w) should have simılar viewpoint and lighting changes as the pseudo samples } In the meanwhile, the generator G' could regularize the projected samples to lie in the natural image manifold, thus fixing the unnatural distortions and shadows in the pseudo samples. In the next step, we discuss how exploit these projected samples to infer the 3D shape.
Step 3: Learning the 3D Shape. The above projected samples (ī.) provide images of multiple viewpoint and lighting conditions while having nearly the same object content, egg., face identi- ty. Such 'single instance multiple view &lghting setting makes it possible to learn the underlying
5

---

Published as a conference paper at ICLR 2021
3D shape with aforementioned peotto-eommtric autoencoding model. Specifically, as illustrated in Fig (c), the viewpoint network V and the lighting network L predict instance cestance-specffic view D, and lighting 1 for each sample I, while the depth network D and albedo network A output the shared depth and albedo à with the original sample I as input. The four networks are jointly optimized with the following reconstruction objective:
0A.0v..
arg min 0D. 00.0A.01 Bv Or
Σ
(I. (DD) + XgCsmooth (D E=0
where 0D. AA, Bv, 0r are the parameters of network D, A. V. L respectively, and Lemooth (D(I) is smoothness term defined the same way in (Zhou et al., 2017). Note that also reconstruct the original sample I, i.e., use I as additional I,, to prevent the infered factors deviate from the original sample. Here we do not use the sampled {w;} and (4,) in step 2, as the viewpoints and lighting directions of the projected samples may not accurately match the pseudo samples, thus we learn to predict them with V and L instead.
The above reconstruction loss would decrease only when the underlying depth, albedo, and all sam- ples' viewpoints and lighting directions are correctly inferred. As shown in Fig (c), the object shape, parameterized by depth, does shift toward the true underlying shape, e.g., shape of a human face. And the unnatural distortions and shadows that exist in the pseudo samples are mitigated in the reconstructions.
Iterative Self-refinement. While the above three steps produce a more accurate 3D object shape than the prior shape, it may miss some details like the wrinkles on the face. This is because the projected samples {I:} may not accurately preserve all semantics of the original instance in I, as the pseudo samples are obtained from an approximated weak shape prior. Fortunately, we can repeat the cycle of these three steps multiple times by regarding the refined 3D shape and trained networks in the previous cycle as the initialized shape and networks in new cycle. This allows us to refine the 3D shape and projected samples progressively. In our experiments, we use four cycles of these three steps unless otherwise stated.
3.2 DISCUSSION
Regularizing the Latent Offset. In the Eq. 2 of Step 2, the latent offset Aw; = E(I,) is only regularized with weak L2 norm term, thus the resulting latent vector Aw; + may fall out of the intermediate latent distribution W of the GAN. This could possibly cause the projected samples {I,} to inherit some unnatural distortions in the pscudo samples (L:). thus stronger regularization regularization strategy, which is to
on Am;; is required. To address this issue, we propose restrict Aw; to be a valid offset created by the mapping network F as Aw; = F(E(L.) F(0). We provide more discussion on this design in the Appendix.
Joint Training of Multiple Instances. Note that our discussion above involves only a single target image I, i.e., our method is used in an instance-specifie manner. However, our pipeline could also be jointly trained on multiple instances to achieve a better generalization ability and faster con- vergence speed on new samples. The extension joint training is straightforward, whose details are included in the Appendix. The key difference between training with multiple instances and a single instance that, since there are multiple target instances with diverse viewpoint and lighting variations, the absolute latent offset E(I,) in Eq. 2 should be replaced with a relative latent offset E(I.) - E(I), which could generalize better across different target instances.
EXPERIMENTS
Implementation Details. We first evaluate our method on 3D shape recovery, and then show its application to 3D-aware image manipulation. The datasets used include CelebA (Liu et al. 2015). BFM (Paysan et al.) 2209). combined cat dataset (Zhang et al. 2008 Parkhi et al. 2012). LSUN Car Yu et al. 2015). and LSUN Church (Yu et al. (2015). 5). all of which are unconstrained RGB images. For BFM, on these datasets in our experiments. We recommend readers to refer to the Appendix for more implementation details and qualitative results.
the same as in Wu et al. (2020). We adopt StyleGAN2 pre- trained

---

Published as a conference paper at ICLR 2021
(a) Outs
Input:
Shape
Normal
Textured
Unsaupd
Ingut
Shape
Nommal
Textured
Sample
3D Shage
Nomal
Textured
Figure 4: Qualitative comparisons. (a) shows the reconstructed 3D mesh textured mesh of our method. (b) shows the results of Unsup3d (Wu et al. 2020 We see that results in (a) are more accurate and realistic.
surface normal, and
Input
Relighting
Unsup3d
Figure 5: Qualitative comparison on buildings. The first row shows the input image and our relighting effects described in Sec 4.2) The second row shows the recovered shape (viewed in surface normal and mesh) of our method, while the last row shows the results of Unsup3d.
4.1 UNSUPERVISED 3D SHAPE RECONSTRUCTION
Qualitative Evaluation. The qualitative results of method and Unsup3d (Wu et al. 2020
shown in Fig [ Fig[4 and Fig Our method recovers the 3D shapes with high-quality across hu- man faces, cats, cars, and buildings. For example, the wrinkles on human faces and the edges and flat surfaces on cars are well captured. The results of Unsup3d, while good, tend to miss some asym- metric aspects of human faces, e.g., its predicted eye directions always point the front. Besides, as shown in Fig (b) and Fig Unsup3d has difficulty in generalizing to datasets with asymmetric objects or large viewpoint variations like buildings and cars, while our method does not.
Quantitative comparison is conducted on the BFM dataset. Following (Wu et al. 2020). we report the scale-invariant depth error (SIDE) and mean angle deviation (MAD) scores as the evaluation metrics. In order to evaluate on the test set images instead of GAN gener- ated samples, we need to first conduct GAN inversion, i.e., finding the intermediate latent vectors {w} that best reconstruct these images. We adopt the method in Karras et al. (2020b), which achieves satisfactory reconstruction results, as shown in the first row of Fig|6] To achieve better generalization. method is pre-trus me-trrined set has 160k images in total). Due to the additional testing time brought by the GAN inversion and instance-specific training steps, report results the first 500 images of the test set.
Quantitative Evaluation.
only 200 training images before testing (the training

---

Published as a conference paper at ICLR 2021
Imput Recorstruction
Table I: Comparisons on the BFM dataset. We report SIDE and MAD errors. 'Smmmetry' indicates whether the symmetry assump-- tion on object shape is used. We outperform others both metrics.
No. Method
Symmetry
SIDE (x () MAD (deg.))
(1) Supervised
N
0.419
10.83
Ous
(2) Const, null depth
2.723
43.22
(3)
Average g.t. depth
/
1.788
22.99
(4) Unsup3d (Wu et al. 2020
Y
0.807
16,34
Unsupd
(5) Ours (w/o regularze)
0.925
16.42
(6) Ours
0.756
14.81
(7)
Unsupäd (Wu al. 2020
1.334
33.79
Figure 6: Results without symmetry assumption.
(8)
Ours
N
1.023
17.09
Table 2: Effects of different shape prior. We report results of the original ellipsoid shape, asym- metric shape with ellipsoid for the left half and sphere for the right half, shape with its position shifted by 1/6 and 1/4 image width, weaker shape prior whose height is half of the original one, and
no shape prior. Qualitative results can be found in Fig 14 in the Appendix.
Shape prior
Origin
Asymmetric
Shift w/6
Shift w/4
Weak
Flat
SIDE (x10-2)1
0.756
0.769
0.767 14.93
0.775
0.764
1.221
MAD (deg.)4
14.81
14.95
15.07
14.97
20.46
Trput
Rotation.3
BoAe
Reightigg
Reighting-GAN
Figure 7: 3D-aware image manipulation, including rotation and relighting. We show results ob- tained via both 3D mesh and GANs. The input of the first row is a real natural image. Our method achieves photo-realistic manipulation effects obeying the objects' underlying 3D structures.
Although the above GAN inversion step could introduce additional error due imperfect recon- struction, our method still noticeably outperforms Unsup3d and other trivial baselines, as shown in Tab[ is worth noting that our method does not necessarily rely on the symmetric assumption. To demonstrate this, we also report results of our method and Unsup3d when both are learned without the symmetric assumption. As (7)(7)8) in Tab and Fig 6 show, in this setting our method signif- icantly outperforms Unsup3d, which tends to produce messy results. Furthermore, No.(5) provides the results without using the latent offset regularization described in Sec S.2 where the performance decreases notably, demonstrating the effectiveness of the proposed regularization technique.
We further study the effects of using different shape prior. The setting is the same as Tab and the results are shown in Tab Some variations like asymmetric, shifted, and weaker shape prior only slightly affect the performance, showing that the result not very sensitive to the shape prior. But the results would get worse for flat shape, as it could not indicate viewpoint and lighting change.
4.2 3D-AWARE IMAGE MANIPULATION
Object Rotation and Relighting. After training, the recovered 3D shape and the encoder E im- mediately allow various 3D-aware image manipulation effects. In Fig we show two manipulation effects, including object rotation and relighting. For each effect, we show results rendered using the recovered 3D shape and albedo, and their corresponding projections in the GAN manifold via the encoder E. The results rendered from 3D shapes strictly follow the underlying structure of the object, while their projections in GAN are more photo-realistic. As mentioned before, our method is applicable to real natural images via GAN inversion, and the first row in Fig is an example, where method also works well.
8

---

Figure 8: Qualitative comparison on face ro- tation. "Ours (GAN)" and "Ours (3D)" indicate results generated by GAN and rendered from Published as a conference paper at ICLR 2021
GANSpao
Sefn
Ours (GAN) Okes (D)
Fiont
Table 3: Identity-preserving face rotation. We compare with HoloGAN, GANSpace, and SeFa. The metrics are identity distances measured as an- gles in the ArcFace feature embeddings.
Trft
Method
eTrOr moan (deg.N.
emror max (deg.).
HoloGAN
47.38
69.24
GANSpace
41.17
58.93
SeFa
41.79
60.73
Ours (3D)
28.93
43.02
Ours (GAN)
39.85
57.21
3D mesh respectively. The face identities in the baseline methods tend to drift during rotation.
mash Figure 9: Qualitative results on the LSUN Horse dataset Yuu et al. 2015).
Input
Nomta
Textured
Relightng
Identity-preserving Face Rotation. We compare our method with other unsupervised method- S that also achieve face rotation with GANs, including HoloGAN (Nguyen-Phuod GANSpace (Härkönen et al. 2020). and SeFa (Shen & Zhou, 2020). Specifically, for each method, we randomly sample 100 face images, and for each face, we shift its yaw angle from -20 degree to 20 degree, obtaining 20 samples with different poses. A face identity detection model ArcFace (Deng) al. (219) is then used to evaluate how much the face identities change during rotation. More de- tails are included in the Appendix. Since method explicitly recover the 3D face shapes, results rendered from these recovered 3D shapes significantly outperform other counterparts, as shown in Tad Fig 8 In the alternative attempt where we directly operate in the GAN manifold following the projected directions of the recovered 3D shapes, face identities are also better preserved.
al., 2019),
Discussion. We have shown that our method is applicable to many object categories. However, for objects with more sophisticated shapes like horses, a simple convex shape prior may not well reflect the viewpoint and lighting variations, and thus the 3D shapes could not be very accurately inferred. The second row of Fig provides one example. Another limitation of our method is that our 3D mesh is parameterized by depth map, which could model the back-side shape of objects, as the first row of Fig shows. This could possibly be addressed by adopting better parameterization of the 3D mesh.Despir these limitations, our method still captures some aspects of the horses' shapes, like the rough shape of the head and belly, and achieves reasonable relighting effects.
5 CONCLUSION
We have presented the first method that directly leverages off-the-shelf 2D GANs to recover 3D ob- ject shapes from images. We found that existing 2D GANs inherently capture sufficient knowledge to recover 3D shapes for many object categories, including human faces, cats, cars, and buildings. Based on a weak convex shape prior, our method could explore the viewpoint and lighting variations in the GAN image manifold, and exploit these variations refine the underlying object shape in an iterative manner. We have further shown the application of our method on 3D-aware image manip- ulations including object rotation and relighting. Our results uncover the potential of 2D GANs in modeling the underlying 3D geometry of 2D image manifold.
Acknowledgment. This research supported by NTU NAP and STAR through the Industry Alignment Fund Industry Collaboration Projects Grant. This work is also supported by the HK General Research Fund No. No.27208.20.
conducted in collaboration with SenseTime. This work is

---

Published as a conference paper at ICLR 2021
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? ICCV, pp. 4432 4441, 2019.
Volker Blanz and Thomas Vetter. morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference Computer graphics and interactive techniques, 1999.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2019.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In CVPR. pp. 4690-4699,,000
Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d imitative-contrasivve learning. In CVPR, pp. 5154-5163, 2020.
Baris Gecer, Stylianos Ploumpis, Irene Kotsia, and Stefanos Zafeiriou. Ganfit: Generative adversar- ial network fitting for high fidelity 3d face reconstruction. In CVPR, pp. 1155-1164, 2019.
Shubham Goel, Angjoo Kanazawa, and Jitendra Malik. Shape and viewpoint without keypoints. ECCV, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. arXiv preprint 0xXx:2200..02546, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Escaping plato's cave: 3d shape from adversar- ial rendering. In ICCV, pp. 9984-9993, 2019.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pp. 694-711. Springer, 2016.
Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, pp. 7122-7131, 2018a.
Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category- specific mesh reconstruction from image collections. In ECCV, pp. 371-386, 2018b.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pp. 4401-4411 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila, Training generative adversarial networks with limited data. In Proc. NeurIPS, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, Analyz- ing and improving the image quality of stylegan. In CVPR, pp. 8110-8119 19, 2020b.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In CVPR, pp- 3907-3916, 2018.
Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang, and Jan Kautz. Self-supervised single-view reconstruction via semantic consistency. ECCV, 2020.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, pp. 3730-3738, 2015.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions graphics (TOG), 34(6):1-16, 2015.
10

---

Published as a conference paper at ICLR 2021
Sebastian Lunz, Yingzhen Li, Andrew Fitzgibbon, and Nate Kushman. Inverse graphics gan: Learn- ing to generate 3d shapes from unstructured 2d data. arXiv preprint arXiv:2002.12674, 2020.
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In ICML, volume 30. pp. 2013.
Thu Nguyen-Phuoe, Chuan Li. Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. In ICCV, pp. 7588-7597, 2019.
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. ECCV, 2020.
Omkar Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, pp. 3498-3505. IEEE, 2012.
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. 2009 Sixth IEEE International Conference Advanced Video and Signal Based Surveillance, pp. 296-3996 Icee, 2009.
Mihir Sahasrabudhe, Zhixin Shu, Edward Bartrum, Riza Alp Guler, Dimitris Samaras, and lasonas Kokkinos. Lifting autoencoders: Unsupervised learning of a fully-disentangled 3d morphable model using deep non-rigid structure from motion. In ICCV Workshops, pp. 0-0, 2019.
Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J Black. Learning to regress 3d face shape and expression from an image without 3d supervision. In CVPR, pp. 7763-7772, 2019.
Jiaxiang Shang, Tianwei Shen, Shiwei Li, Lei Zhou, Mingmin Zhen, Tian Fang, and Long Quan. Self-supervised monocular 3d face reconstruction by occlusion-aware multi-view geometry con- sistency. ECCV, 2020.
Yujun Shen and Bolei Zhou. Closed-fome factorization of latent semantics in gans. arXiv preprint arXiv:2007.06600 2020.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou Interpreting the latent space of gans for semantic face editing. In CVPR, pp. 9243-9252, 2020.
Attila Szabó, Givi Meishvili, and Paolo Favaro. Unsupervised generative 3d shape learning from natural images. arXiv preprint arXiv:1910.00287, 2019.
Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj. Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhofer, and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait images. In CVPR, pp. 6142-6151, 2020.
Luan Tran and Xiaoming Liu. Nonlinear 3d face morphable model. In CVPR, pp. 7346-7355, 2018.
Shubham Tulsiani, Nilesh Kulkarni, and Abhinav Gupta. Implicit mesh reconstruction from unan- notated image collections. arXiv preprint arXiv:2007.08504 2020.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a proba- bilistic latent space of object shapes via 3d senerative-adversaraall modeling. In NIPS, 2016.
Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Unsupervised learning of probably sym- metric deformable 3d objects from images in the wild. In CVPR, 2020.
Yuxin Wu and Kaiming He. Group normalization. In ECCV, pp. 3-19, 2018.
Yuanbo Xiangli, Yubin Deng, Bo Dai, Chen Change Loy, and Dahua Lin. Real or not real, that is the question. ICLR, 2020.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 030.0335,, 2015.
Weiwei Zhang, Jian Sun, and Xiaoou Tang. Cat head detection-hov to effectively exploit shape and texture features. In ECCV, pp. 802-816. Springer, 2008

---

Published as a conference paper at ICLR 2021
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. CVPR. pp. 2881-2890, 2017.
Hang Zhou, Jihao Liu, Ziwei Liu, Yu Liu, and Xiaogang Wang. Rotate-and-render: Unsupervised photorealistic face rotation from single-view images. In CVPR, 2020.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, 2017.
Jun- Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexci A Efros. Generative visual manipu- lation on the natural image manifold. In ECCV, pp. 597-613. Springer, 2016.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
In this appendix, we provide more qualitative results and the implementation details.
A.1 QUALITATIVE EXAMPLES
more qualitative results of our method, including 3D shape reconstruction and 3D- aware image manipulations. The effects of iterative training is shown in [i where the object shapes become more precise at latter training stages. Fig [2] shows that our method works well without using the symmetry assumption. Fig 13 shows that our method achieves reasonable results for many challenging cases.
U
Input
3D mesh
Nomal
Textured
Rotation
Relghting
Figure 10: This is extension of Fig background.
The last row shows example without removing the
13

---

Published as a conference paper at ICLR 2021
Input
Initialization
Stagel
Stage2
Stage3
Input
Iritialization
Stagel
Stage2
Stage3
Stage1
Figure 11: Effects of iterative training. The reconstructed object shapes get more precise at latter training stages.
Impu
(bb
Figure 12: (a) Our results with the symmetry assumption. (b) Our results without the symmetry assumption. (c) The results of Unsup3d Wu et al. (2020) without the symmetry assumption. Our method achieves competitive results even without using the symmetry assumption.
Sunglasses
Sunglasses
Earrings
Side view
Long hair
Open window
Input
Normal
Mesh
Figure 13: Results on challenging cases. Our method achieves reasonable results for most cases.
Effeets of shape prior. Fig 14 shows the qualitative results of using different shape priors. Unlike the results in Tab here joint training is not adopted to better showcase the effects of different shape. When the ellipsoid shifted by 1/6 image width or is asymmetric, the shape could be progressively refined during the iterative training process. But the results would get worse if the shift is too large or a flat shape is used, because the viewpoint and lighting changes could not be revealed with these shapes.
14

---

Published as a conference paper at ICLR 2021
Imput
Origin
Stuge
Stuge 2
Stage 4
Asymmettri
Stage
Stage
Stage
Shift w/l
Stage
2
Stage 4
Shift w/4
Stago
Stage
Stage
Weak
Stage
Stage 2
Stage 4
Flat
Stago
Stago
Stagio
Figure 14: Effects of different shape prior. We show results of the original ellipsoid shape, asym- metric shape with ellipsoid for the half and sphere for the right half, shape with its position shifted by 1/6 and 1/4 image width, weaker shape prior whose height is half of the original one, and flat shape.
Table 4: Effects of dataset size. We report quantitative results the BFM dataset with different number of training samples.
Dataset size
160k
40k
10k
3k
lk
SIDE (x 10-
0.756
0.765
0.784
0.794
0.945
MAD (deg.)4
14.81
14.90
15.10
15.28
16.84
Effects of dataset size. Fig shows the effects of dataset size used for training the GAN. It can be seen that the performance only drops marginally even when the dataset size is reduced from 160k 3k. And the performance get notably worse when the dataset size reduces 1k, as the image quality of the GAN starts to deteriorate given insufficient data. This could potentially be addressed by recent data augmentation strategy GANs (Karras et al. 2020a).
A.2 IMPLEMENTATION DETIALS
A.2.1 PHOTO-GEOMETRIC AUTOENCODING
The photo-geometric autoencoding model we used mainly follows the work of Here we describe the and II in T
Wu et al. (2020)
J A(d, a, I) produces the illuminated texture with the albedo depth map d, and light direction Specifically, we first calculate the normal map n of the depth map d, which describes the normal direction of the surface for each pixel position (i,j). Then the illuminated texture is calculate with Jij = (k, + ka max 0, (1, niy)) aij. where k and hig are the scalar weight for the ambient and diffuse terms, and are predicted by the lighting network V. The lighting direction is described as I - (lo, by. 1)²/((²
/(12 + 1 1),.5, where l and ly are also the predictions of the lighting network.
Given the above illuminated texture J, the reprojection function II would map as i [1(J, d, v). To do this, we denote the 3D point in the reference frame of the camera as P Pa. Pg. P.) E IR3, which is projected to pixel p j. 1) via:
viewpoint
G = W
f
G
= H-1
KP, K=
[
W
tan Lrov
where FOov is the field of view of the perspective camera, and is assume to be FOO R 10 Given depth map which describes depth value di for each pixel (i.j), we would have P dej K -'
The viewpoint E S represents the rotation angles (W1:3) and translations (V1.6) along axes. v thus describes an Euclidean transform (R, T) E (3))), which transforms 3D points to the predicted viewpoint. The mapping from a pixel (i, in the original view the pixel (" in the
and
15

---

Published as a conference paper at ICLR 2021
predicted view could then be described by the warping function Nd,v (i,5) + (1',8) as:
p' (di) RK TLK--p- T)
(5)
where p' (d'.)',1), Finally, II applies this warp to the illuminated texture J to obtain the repro-
jected image [], d, v) as Lz Jes. where (i. (f', Here 7d. is the backward warping of Vdd,o which is achieved via a differentiable renderer Kato al. (2018) in the same way as Wu et al. (222)).
A.2.2 REGULARIZING THE LATENT OFFSET
As described in Section 3.2 in order to keep the projected samples {I,} in the natural image man- the predicted latent offset Aw; is needed. One possible solution
ifold, stronger regularization to let the encoder predict the offset the original latent space, i.e., the one before the mapping network F, then the output of the generator would be ([F(E( + z)), where is the original latent vector corresponding to I. However, one issue of this solution is that, compared with the in- termediate latent space W, it much more difficult to perform GAN inversion the original latent space Z to obtain the latent vector & of a real natural image. This would impede this method to be applied to real natural images other than GAN samples.
To address this issue, in this work we propose a new regularization strategy for the latent offset Aw; which is to restrict it to be valid offset created by the mapping network F as follows:
Aw; F[Az,) F(0)
= -(E(L)) - F(O)
(6)
which regards Aun; as the shift in the W space by adding perturbation around zero in the space. Note that the point zero is natural choice as it is the mean point in the prior distribution N A(, 7)d. In order to make the regularization strength adjustable, one can further view the mapping network F as two consecutive parts F' Fi F2, and regularize Aw; with only Fi as below:
Aw; (E(L)) + (o) - F(O)
The depth d of F1 thus controls the strength of regularization. As shown in Tables 10 12 for the BFM dataset, the full mapping network (i.e., d 8). For other datasets with more complex images, the regularization of a full mapping network is too strong and sometimes lead to sub-optimal results, thus set 2.
A.2.3 TRAINING DETIALS
In this section, provide the network architectures, hyper- ppper-parameters, and other details in experiments.
Model Architectures. For fair comparison the BFM dataset, the images used in the photo- geometric model are resized to 642 resolution, and the D, A, L, V network have the same architec- ture Wu et al. (2020). For other datasets, the image resolution is resized to 128?, and the network architectures are added by additional layers accordingly, as shown in Tab and Tab For the GAN encoder E, adopt ResNet-alike architecture (He al. (0166), as presented in Tab and Tab 8] The abbreviations for the network layers are described below:
Conv(cin, Cout, k. s, s,p): convolution with Cin input channels, Cout output channels, kernel size k. stride and padding p.
Deconv(cin, Cout, ki, s, p): deconvolution with em input channels, Cous output channels, kernel size k. stride s, and padding
GN(n): group normalization (Wu & He 2018) with n groups.
LReLU(a): leaky ReLU (Maas et al. 2013) with a negative slope of a.
Upsample(s): nearest-neighhor upsampling with scale of s.
Avg pool)oo(s)) average pooling with a stride of s.
ResBlock Cout): residual block defined in Tab
16

---

Published as a conference paper at ICLR 2021
Table 6: Network architecture for depth net D and albedo net A. Cout is for D and 3 for A.
Encoder
Output size
Conv(3,
64
Conv(32, 4, (N(1))
Conv(64, 4, -
32
Table 5: Network architecture for view-
16
Conv(128, 256, 2, 1) GN(64) LRcLU(00.)
point net V and lighting net L. The output
Conv(256, 512, 4, 2, 1) LReLU
channel size Cout 6 for and for
Conv(512, 256, 1, ReLU
Encoder
Output
Decoder
Output size
Conv(3, 3,, 1) ReLU
64
Decomv(25..512.41.000 ReLU
Conv(32, (evv32,,4,44,0 6, 1)++ ReLU
32
Comv(512.512.331.1) ReLU
Decomn(512..26:4222.)) (N(664) ReLU
Conv(64, 2,
ReLU
16
Comv(25..26.331..)) GN(64) ReLU
Conv(128, 256, 4, I) ReLU
Dccomv256.128442.1) ( ReLU
16
Conv(256, 512, 2, 1)++ ReLU
Comv(188.128331..) GN(32) ReLU
16
Conv(512, 512, 4, 1, +) ReLU
Decomv(188,64442..)) GN(16) ReLU
32
Conv(512, Couts 1, 1, 0) Tanh
C GN(16) ReLU
32
Decomv(64.32.4221)) GN(8) GN(()) ReLU
64
(2,,2 ReLU
64
Upsample(2)
128
Conv(32,32.3 GN(8) ReLU
128
Conv(32,32.5,122)+ GN(8) ReLU Conv(32.con.5.1.2)) Tanh
128
128
Table 7: Network architecture of GAN encoder net for 128 resolution input images. For 64 resolution input, the last ResBlock is removed and the following channels are cut down by half.
Table 8: Network architecture for the
ResBlock(Gins Cout) in Tab The output of Residual path and Identity path are added as the final output.
Encoder
Output size
Conv(3, 32, 4, 2, 1)+ ReLU
Residual path
ResBlock(32, 64)
32
ReLU Conv(Cin Couts
ResBlock(64, 128)
16
ReLU + Conv(Couts Couts
ResBlock(125 128, 256)
Identity path
ResBlock(256, 512)
Avg. pyg.pool(2)
Conv(S12, 1024, 4, 1, 0) ReLU
Conv(Cin- Cout. 1,
Conv(1024, 512, 1, 1,
Table 10: Hyper-parametes for LSUN Car and
LSUN Church datasets.
Table 9: Hyper-parameters
Parameter
Value
Symmetry
Parameter
Value/Range
Number pseudo samples
3200
Optimizer
Adam
Depth of Fi
Learning rate
x 10-1
Batchsize
32
Depth map
(0.9, 1.1)
Number stages
Step iterations (1st stage)
700
Ellipsoid
(0.91, 1.02)
Step 2 iterations (Ist stage)
700
A1 (in Eq
0.01
Step 3 iterations (Ist stage)
600
Ao (in Eq
0.01
Step 1 iterations (other stages)
200
Step 2 iterations (other stages)
500
Step 3 iterations (other stages)
400
N
The prpee-parameters of our experiments are provided in Tables 9 The batch size here batch size used in the Step and Step 3 of our method. And the number of stages indicates how many times the Step 1-3 are repeated.
Siyper-parameters.
10
12
Joint pre-training. Note that for the CelebA, Cat, and BFM dataset, we adopt joint pre-training on 200 samples to achieve better generalization when fine-tuning on new samples. Specifically, for each iteration of each step, we randomly choose a batch samples from the 200 samples, and the training is the same as that of single sample except that the gradient calculated from each sample are averaged after back-propas propaarrpa.
Symmetry assumption. In method, the symmetry assumption object shapes is optional choice, and could further improve the results for human face and cat datasets. Thus we also adopt this assumption for CelebA, Cat, and BFM dataset unless otherwise stated. Note that the results
17

---

Published as a conference paper at ICLR 2021
Table 11: Hyper-parameters for CelebA and
Cat datasets.
Table 12: Hyper- parameters for BFM dataset.
Joint Pre-train
Value
Symmetry
Joint Pre-train
Value
Symmetry
Number samples
200
Number of samples
200
Number pseudo samples m
256
Number of pscudo samples m
480
Depth dl of Fi
Batchsize
64
Depth d of Fi
8
Batchsize
96
Number of stages
Number of stages
8
Step iterations (1st stage)
1500
Step iterations (1st stage)
1500
Step iterations (Ist stage)
1500
Step iterations (1st stage)
1500
Step iterations (Ist stage)
1500
Step iterations (other stages)
700
Step iterations (Ist stage)
1500
Step 2 iterations (other stages)
1000
Step iterations (other stages)
700
Step iterations (other stages)
1000
Step iterations (other stages)
1000
Step 3 iterations (other stages)
1000
Fine-tune
Value
Symmetry
Number of pseudo samples m
1600
Fine-tune
Value
Symmetry
Depth d of Fj
2
Number of pseudo samples m
1600
Batchsize
16
Depth d Fi
8
Number of stages
4
Butchsize
16
Step I iterations (1st stage)
600
Number of stages
1
Step 2 iterations (Ist stage)
600
Step 1 iterations
500
Step 3 iterations (Ist stage)
400
Y
Step 2 iterations
500
Step 1 iterations (other stages)
200
Step 3 iterations
200
Y
Step 2 iterations (other stages)
500
Step 3 iterations (other stages)
300
N
without the symmetry assumption are also provided in Figures 6 12 Tab 12 and other datasets,
which is also satisfactory. Unlike Unsup3d (Wu et al. (020), our method does not need to predict an additional confidence map for the symmetry, we can drop the symmetry assumption after the first training stage, as shown in TT
Sampling viewpoints and lighting directions. the Step 2 of our method, we need to randomly sample a number of viewpoints and lighting directions from some prior distributions. For viewpoint, we calculate the mean and covariance statisties of viewpoints estimated from the BFM, CelebA, and Cat dataset using the viewpoint prediction network of (Wu et al. 2020). These statistics are used to sample the random viewpoints from multi-variate normal distributions for the corresponding datasets. For LSUN Car and Church datasets, we simply use the statistics estimated from the FFHQ dataset (Karras et al. 2019), and we found it works well.
For lighting directions, we sample from the following uniform distributions: lg ~ (Imin T xmax); by ~ (yymin Mmaz). ). ka ~ Ld (dmin, dmax), ), and h2- = aka, where [mmin Tmax- Iman Imazs dmin dmaz, a] = 0.6, 0.6] for other datasets. Note the sampled hig and k2 are added to the predictions of the lighting network L as the final diffuse and ambient term. This setting tends to increase the diffuse term and decrease the ambient term, as the diffuse term could better reflect the structure of the objects.
[[-0.9. [-0.9, 0.9, 0.3, 0.8, -0 0.7, -0.4] for BFM dataset and [-1, -0.2, 0.8, -0.1,
Masking out the background. In this work, we care about the main object rather than the back- ground. Thus, for GANS trained on the LSUN Car and LSUN Church datasets that contain large background area, we could use an off off-the-shelf scene parsing model PSPNet50 (Zhao et al. 2017) to parse the object areas as a pre- pro-processing for better visualization. For an input image, the scene parsing model used only once to get the object mask for the original view, thus does not provide signals on the 3D shape. For novel views, the mask is warped with the current shape accordingly, and is used to mask out the image background in Eq and Eq 3 For other datasets we do not mask out the background.
Face rotation evaluation. In the tity detection model ArcFace (Deng et al. 2019) to evaluate the shift of face identity during rotation. For each human face sample, we first rotate its yaw angle from -20 degree to 20 degrees to obtain 20 samples with different poses. The face angle is obtained with the viewpoint prediction network from Wu et al. (202)). We then calculate the identity distances between the front-view face and other faces, and report both their mean and max values in TT] For face identity prediction mod- el, the identity distances are represented as angles between the final feature embeddings. The above mean and max values are averaged across 100 samples as the final results.
face rotation experiments, we use a face iden-
18


Reviews:
[{"review_id": "oS_dMnHoC0I", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an iterative method that jointly estimates viewpoints, light directions, depth, and albedo from single images, by projecting intermediate renderings to the nautral image manifold. Intuitively, the method works by generating, with pre-trained GANs, multiple views of the same object under different lightings, and then inferring 3D shapes from those variants. The key idea is to use pre-trained 2D GANs to make such data generation photorealistic. The authors also demonstrate 3D edits, such as 3D rotation and relighting, that one can perform after running their model.\n\nI like this paper because\n(1) it presents the novel idea of \"generating\", by GAN inversion, photorealistic multi-view, multi-light data of the given real object, from which the 3D shape can then be estimated;\n(2) extensive evaluations were performed to demonstrate the high quality achieved; and\n(3) one can perform 3D edits, such as 3D rotation and relighting, on top of the model outputs. Having explicit 3D understanding for relighting makes a lot of sense to me, and this paper presents a new angle of doing so by GAN inversion. \n\nIn terms of drawbacks, this paper would benefit from the following experiments or clarifications:\n(1) How crucial is the size of the dataset for GAN pretraining? For example, if the dataset is small, not covering many of the face poses, I imagine the GAN projections may not always look realistic, thereby causing the shape estimation to degrade. Such failure cases or studies should be shown so that the reader understands what impact the dataset bias or size has on the final results. An example would be a plot of shape reconstruction error w.r.t. the dataset size or face pose coverage.\n(2) How robust is the algorithm to the shape initialization (ellipsoid shape)? More importantly, what about its location -- what if the \"off-the-shelf scene parsing model\" fails so that the ellipsoid is placed off the main object? If the model fails because of bad initializations, what do the failure modes look like? Are the shapes completely garbage, or something that looks like the initialization? \n(3) Why is the viewing direction parameterized in R6 (I presume the start and end XYZs)? Shouldn't it be in S2, just like the light direction? If they are in R6, then results on zooming in/out should be included. Otherwise, there seems to be no point in defining them in R6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "W5g80NDOPK2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary ####\nThis paper studies an interesting inverse-graphics problem. It proposed a novel method to learn 3D shape reconstruction using pre-trained 2D image generative adversarial networks. Given an image containing one single object of interest, it first predicts the graphics code (e.g., viewpoint, lighting, depth, and albedo) by minimizing the reconstruction error using a differentiable renderer. The next step is to render many pseudo samples by randomization in the viewpoint and lighting space, while keeping the predicted depth and albedo fixed. A pre-trained 2D image GAN is further used to project the pseudo samples to the learned data manifold through GAN-Inversion. Finally, these projected samples are added to the set for the next round optimization. Experimental evaluations have been conducted on several categories including face, car, building, and horse.\n\n\n#### Comments ####\nOverall, this is a very interesting paper with good presentations, promising experimental results, and solid quantitative comparisons with the previous work. Reviewer would like to point out the potential weakness of the paper as follows.\n\nW1: Though impressed by the results (especially the proposed method works for horse and building), reviewer suspects the paper only works in a very simplified setting: (1) the GAN was previously trained on a large amount of 2D images of a single category with many variations in identity, viewpoint, and lighting; (2) the initialization (or step 1 in Section 3.1) step seems very critical to the overall performance; and (3) viewpoint and lightning randomization seems have to be hand-tuned. Reviewer would like to see the discussions on the underlying assumptions more explicitly. In addition, reviewer would like to know how does the method generalize to “dirty” data: people with sunglasses, people with noticeable earrings, people partially occluded by wavy long hair, and people with a side view (looks like the input has to be a frontal face image). Same question applies to those non-convex shapes: a convertible car or a car with the window open. Reviewer suspects the method in the current form cannot handle them well.\n\nW2: Some important experimental settings are neither presented nor clarified. For example, it is not clear what is the difference between “Ours (3D)” and “Ours (GAN)”, which should be clarified. For image editing (see Figure 6 and Figure 8), reviewer sees a noticeable change in background color sometimes (e.g., second row in Figure 6). It would be good to give a very detailed explanation of the image editing process (e.g., what’s the input and output format in each stage). As ellipsoid was used to initialize the face shape, reviewer would like to know what was the initialization for other categories such as building and car (see Figure 10). \n\nW3: It would be good to report the time spent on the computation and optimization and how it is compared to the baselines in Table 1 and Table 2. This is very important metric to report as a fair comparison to the previous work.\n\n== Post-rebuttal Comments ==\n\nI am raising my score from 7 to 8, as author responses addressed my comments well (especially answer to W1 and the Figure 13) than expected.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LpyyHa7LkLP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1. This is the first work that attempts to reconstruct 3D shape from 2D image in an unsupervised way using GANs. The idea is neat: Use networks to predict four 3D parameters and use GAN to generate / synthesize the images corresponding to a set of parameters. Then these synthesized images can be used as pseudo ground truth to train the 3D parameter network.\n2. The experiments are comprehensive to support the effectiveness of the proposed pipeline. 2 tasks are evaluated: 3D shape reconstruction and object-aware image manipulation. On 3D shape reconstruction, performances are reported on two datasets and demonstrated it outperforms SoTA method by a large margin. On image manipulation, the visualization results look reasonable and visually better than previous method.\n\nCons / Questions:\n1. The authors claim in the introduction section that this proposed method has advantage over previous method as it doesn't assume symmetry of the instance. But in this proposed method, a symmetrical ellipsoid is used as the shape prior. Is this a stronger implicit assumption than the symmetry assumption? Is there any experiment to explore how the shape prior affects the models' training results? E.g. what if a non-convex shape prior is provided, what if asymmetrical prior is provided?\n2. In the introduction section, the authors use `'building' as an example to show that previous method's symmetry assumption cannot work. But in the experiments and comparison with previous works, all data used are symmetrical: human face, animal face, cars, etc. Visualizations on buildings are shown in appendix but there is no quantitative analysis or comparing with current SoTA method.\n3. How is the generator initialized? As the generator is always fixed during training, I assume the generator is using some pre-trained network? If so, can we still call this network fully unsupervised?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to use pre-trained 2D (i.e., image) GANs as a mechanism for recovering 3D shape from a single 2D image. The work demonstrates impressive results on not only human and cat faces, but also cars and buildings. The method is demonstrated with qualitative results and quantitative results on multiple datasets and tasks.

The reviewers were persuaded by the novelty and "neatness" of the idea (and the AC is in agreement) as well as the results. At submission time, there were some concerns with experimental details. For instance, there was a question of how carefully the settings have to be tuned (always a concern with unsupervised methods) as well as an overarching concern about the initialization and whether the method will work on less clean data. The reviewers (and the AC) seem to think that these have been sorted out in discussion. 

All three reviewers were in favor of acceptance and the area chair is inclined to agree with the reviewers. In particular, the AC finds the work interesting and compelling. While there is an updated version already uploaded during the discussion, the AC encourages the reviewers to double check all the questions from the reviewers and include the answers from the discussion into the camera ready (even these results are in the appendix).
Decision: Accept (Oral)