Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
Authors: Yuxuan Zhang
Wenzheng Chen
Huan Ling
Jun Gao
Yinan Zhang
Antonio Torralba
Sanja Fidler
Venue: ICLR 2021 Oral (2021)
Keywords: Differentiable rendering
inverse graphics
GANs

Abstract: Differentiable rendering has paved the way to training neural networks to perform “inverse graphics” tasks such as predicting 3D geometry from monocular photographs. To train high performing models, most of the current approaches rely on multi-view imagery which are not readily available in practice.  Recent Generative Adversarial Networks (GANs) that synthesize images, in contrast, seem to acquire 3D knowledge implicitly during training: object viewpoints can be manipulated by simply manipulating the latent codes. However, these latent codes often lack further physical interpretation and thus GANs cannot easily be inverted to perform explicit 3D reasoning. In this paper, we aim to extract and disentangle 3D knowledge learned by generative models by utilizing differentiable renderers. Key to our approach is to exploit GANs as a multi-view data generator to train an inverse graphics network using an off-the-shelf differentiable renderer, and the trained inverse graphics network as a teacher to disentangle the GAN's latent code into interpretable 3D properties. The entire architecture is trained iteratively using cycle consistency losses. We show that our approach significantly outperforms state-of-the-art inverse graphics networks trained on existing datasets, both quantitatively and via user studies. We further showcase the disentangled GAN as a controllable 3D “neural renderer", complementing traditional graphics renderers.

Published as a conference paper at ICLR 2021
IMAGE GANS MEET DIFFERENTIABLE RENDERING FOR INVERSE GRAPHICS AND INTERPRETABLE 3D NEURAL RENDERING
Yuxuan Zhangt.4x
Wenzheng 1.
Huan Ling
Jun Gao 1.2,3
Yinan Zhang²
Antonio Torralba
Sanja F.d 1.2.3
NVIDIA
University of Toronto
Vector Institute" MIT CSAIL
University of Waterloo
Stanford University"
(alexhang,
ing. jung, sfidler] Bovidii aeoiiiaa.ccm.. con, yineneyesten eurd. borsalbeEnit wdd
ABSTRACT
Differentiable rendering has paved the way to training neural networks to perform "inverse graphics" tasks such as predicting 3D geometry from monocular pho- tographs. To train high performing models, most of the current approaches rely on multi-view imagery which are not readily available in practice. Recent Gener- ative Adversarial Networks (GANs) that synthesize images, in contrast, seem acquire 3D knowledge implicitly during training: object viewpoints can be ma- nipulated by simply manipulating the latent codes. However, these latent codes often lack further physical interpretation and thus GANs cannot easily be inverted to perform explicit 3D reasoning. this paper, we aim to extract and disentangle 3D knowledge learned by generative models by utilizing differentiable renderers. Key to our approach is to exploit GANs as a multi-view data generator to train inverse graphics network using an off-the-shelf differentiable renderer, and the trained inverse graphics network as teacher to disentangle the GAN's N latent code into interpretable 3D properties. The entire architecture is trained iteratively using cycle consistency losses. We show that our approach significantly outperforms state-of-the-art inverse graphics networks trained existing datasets, both quan- titatively and via user studies. We further showcase the disentangled GAN as a controllable 3D "neural renderer", complementing traditional graphics renderers.
INTRODUCTION
The ability to infer 3D properties such as geometry, texture, material, and light from photographs key many domains such AR/VR, robotics, architecture, and computer vision. Interest in this problem has been explosive, particularly in the past few years, as evidenced by large body of published works and several released 3D libraries (TensorflowGraphics by Valentin er (2019), Kaolin by J. et al. (2019), PyTorch3D by Ravi et al. (2020)).
The process of going from images 3D is often called "inverse graphics", since the problem is inverse to the process of rendering in graphics in which 3D scene is projected onto image by taking into account the geometry and material properties objects, and light sources present in the scene. Most work inverse graphics assumes that 3D labels are available during training (Wang et al., 2018; Mescheder et al., 2019; Groueix et al., 2018; Wang et al., 2019; Choy et al., 2016), and trains neural network to predict these labels. To ensure high quality 3D ground-truth, synthetic datasets such as ShapeNet (Chang et al., 2015) are typically used. However, models trained on synthetic datasets often struggle on real photographs due the domain gap with synthetic imagery.
To circumvent these issues, recent work has explored an alternative way to train inverse graphics networks that sidesteps the need for 3D ground-truth during training. The main idea to make
indicates equal contribution.

---

Published as a conference paper at ICLR 2021
Hidl- daata
Graphics Network
Intent code
fraining g
StyleGAN
Cop
L fox
Diserlanglng Mapping Nelwork
Figure I:
We employ two "renderers": GAN (StyleGAN in our work), and a differentiable graphics renderer (DIB-R R in our work). We exploit StyleGAN as synthetic data generator. and we label this data extremely efficiently. This "dataset" is used to train inverse graphics network that predicts 3D properties from images. We use this network to disentangle StyleGAN's latent code through carefully designed mapping network.
graphics renderers differentiable which allows one to infer 3D properties directly from images us- ing gradient based optimization, Kato al. (2018); Liu et al. (2019b); Li et al. (2018); Chen et al. (2019). These methods employ a neural network to predict geometry, texture and light from images, by minimizing the difference between the input image with the image rendered from these proper- ties. While impressive results have been obtained in Liu et al. (2019b); Sitzmann et al. (2019); Liu al. (2019a); Henderson & Ferrari (2018); Chen et al. (2019); Yao et al. (2018); Kanazawa et al. (2018), most of these works still require some form of implicit 3D supervision such as multi-view images of the same object with known cameras. Thus, most results have been reported on the syn- thetic ShapeNet dataset, or the large-scale CUB (Welinder et al., 2010) bird dataset annotated with keypoints from which cameras can be accurately computed using structure-from moom-motion techniques.
On the other hand, generative models of images appear to learn 3D information implicitly, where several works have shown that manipulating the latent code can produce images of the same scene from a different viewpoint (Karras et al., 2019a). However, the learned latent space typically lacks physical interpretation and is usually not disentangled, where properties such as the 3D shape and color of the object often cannot be manipulated independently.
In this paper, we aim to extract and disentangle 3D knowledge learned by generative models by utilizing differentiable graphics renderers. We exploit a GAN, specifically StyleGAN (Karras al., 2019a), as a generator of multi-view imagery to train an inverse graphics neural network using a differentiable renderer. In turn, use the inverse graphics network to inform StyleGAN about the image formation process through the knowledge from graphics, effectively disentangling the GAN's latent space. We connect StyleGAN and the inverse graphics network into single architecture which we iteratively train using cycle-consistency losses. We demonstrate our approach to signif- icantly outperform inverse graphics networks on existing datasets, and showcase controllable 31) generation and manipulation of imagery using the disentangled generative model.
2 RELATED WORK
3D from 2D:
Reconstructing 3D objects from 2D images is one of the mainstream problems in 3D computer vision. We here focus review to single-image 3D reconstruction which is the domain of our work. Most of the existing approaches train neural networks to predict 3D shapes from images by utilizing 3D labels during training, Wang et al. (2018); Mescheder et al. (2019); Choy et al. (2016); Park et al. (2019). However, the need for 3D training data limits these methods to the use of synthetic datasets. When tested on real imagery there is noticeable performance gap.
Newer works propose to differentiate through the traditional rendering process in the training loop of neural networks, Loper & Black (2014); Kato al. (2018); Liu et al. (2019b); Chen et al. (2019); Petersen et al. (2019); Gao et al. (2020). Differentiable renderers allow one to infer 3D from 2D im- ages without requiring 3D ground-truth. However, in order to make these methods work in practice, several additional losses are utilized in learning, such as the multi-view consistency loss whereby the cameras are assumed known. Impressive reconstruction results have been obtained on the syn- thetic ShapeNet dataset. While CMR by Kanazawa et al. (2018) and DIB-R by Chen et al. (2019) show real-image 3D reconstructions on CUB and Pascal3D (Xiang et al., 2014) datasets, they rely on manually annotated kcypoints, while still failing to produce accurate results.
A few recent works, Wu et al. (2020); Li et al. (2020); Goel et al. (2020); Kato & Harada (2019), explore 3D reconstruction from 2D images in completely unsupervised fashion. They recover both 3D shapes and camera viewpoints from 2D images by minimizing the difference between orig- inal and re-projected images with additional unsupervised constraints, e.g., semantic information (Li et al. (2020),, symmetry (Wu et al. (2020), GAN loss (Kato & Harada (2019)) or viewpoint distribution (Goel et al. (2020)). Their reconstruction is typically limited to 2.5D (Wu et al. (2020)),
2

---

Published as a conference paper at ICLR 2021
Figure 2:
We show examples of cars (first two rows) synthesized in chosen viewpoints (columns). To get these, we fix the latent code wº that controls the viewpoint (one code per column) and randomly samplc the remaining dimensions of (Style)GAN's latent code (to get rows). Notice how well aligned the two cars are in each column. In the third row we show the same approach applied to horse and bird StyleGAN.
and produces lower quality results than when additional supervision is used (Goel et al. (2020); Li et al. (2020); Kato & Harada (2019)). In contrast, we utilize GANs to generate multi-view realistic datasets that can be annotated extremely efficiintly, which leads to accurate 3D results. Furthermore, our model achieves disentanglement in GANs and turns them into interpretable 3D neural renderers.
Neural Rendering with GANs: GANs (Goodfellow et al., 2014; Karras et al., 2019a) can be regarded as neural renderers, as they take a latent code as input and "'redde" an image. However, the latent code is sampled from a predefined prior and lacks interpretability. Several works generate images with conditions: a semantic mask (Zhu et al., 2017), scene layout Karacan al. (2016), or a caption (Reed et al., 2016), and manipulate the generated images by modifying the input condition. Despite tremendous progress in this direction, there little work on generating images through an interpretable 3D physics process. Dosovitskiy et al. (2016) synthesizes images conditioned on object style, viewpoint, and color. Most relevant work to ours is Zhu et al. (2018), which utilizes a learnt 3D geometry prior and generates images with given viewpoint and texture code. We differ in three important ways. First, we do not require a 3D dataset to train the 3D prior. Second, the texture in our model has 3D physical meaning, while Zhu al. (2018) still samples from prior. We further control background while Zhu et al. (2018) synthesizes objects onto white background.
Disentangling GANs: Learning disentangled representations has been widely explored, Lee et al. (2020); Lin et al. (2019); Perarnau al. (2016). Representative work is InfoGAN Chen et al. (2016), which tries to maximize the mutual information between the prior and the generated image distribution. However, the disentangled code often still lacks physical interpretability. Tewari et al. (2020) transfers face rigging information from an existing model to control face attribute disentanglement in the StyleGAN latent space. Shen et al. (2020) aims find the latent space vectors that correspond to meaningful edits, while Härkönen et al. (2020) exploits PCA to disentangle the latent space. Parallel to our work, Zhang et al. (2021); Li et al. (2021) attempt to interpret the semantic meaning of StyleGAN latent space. In our work, we disentangle the latent space with knowledge from graphics.
3
OUR APPROACH
We start by providing overview of our approach (Fig. 1), and describe the individual components in more detail in the following sections. Our approach marries two types of renderers: GAN- based neural "renderer" and differentiable graphics renderer. Specifically, we leverage the fact that the recent state-of-the-art GAN architecture StyleGAN by Karras et al. (2019abb) learns to produce highly realistic images of objects, and allows for a reliable control over the camera. We manually select a few camera views with rough viewpoint annotation, and use StyleGAN to generate a large number of examples per view, which we explain in Sec. 3.1. In Sec. 3.2, we exploit this dataset to train an inverse graphics network utilizing the state-of-the-ard differentiable renderer, DIB- by Chen et al. (2019) in our work, with a small modification that allows to deal with noisy cameras during training. In See. 3.3, we employ the trained inverse graphics network to disentangle StyleGAN's latent code and turn StyleGAN into a 3D neural renderer, allowing for control over explicit 3D properties. We fine-tune the entire architecture, leading to significantly improved results.
3.1 STYLEGAN AS SYNTHETIC DATA GENERATOR
We first aim to utilize StyleGAN to generate multi-view imagery. StyleGAN is 16 layers neural network that maps a latent code 2 E Z drawn from a normal distribution into a realistic image. The code is first mapped to intermediate latent code w W which is transformed to w"
3

---

Published as a conference paper at ICLR 2021
(wi, w2,..,, W[s) E W through 16 learned affine transformations. We call W the transformed latent space to differentiate it from the intermediate latent space W. Transformed latent codes w" are then injected as the style information to the StyleGAN Synthesis network.
Different layers control different image attributes. As observed in Karras et al. (2019a), styles in early layers adjust the camera viewpoint while styles in the intermediate and higher layers influ- ence shape, texture and background. We provide a careful analysis of all layers in Appendix. We empirically find that the latent code w :- (wi, w2, wÿ. wi) in the first 4 layers controls camera viewpoints. That is, if we sample a new code wo but keep the remaining dimensions of w" fixed (which we call the conten code). we generate images of the same object depicted in different viewpoint. Examples are shown in Fig. 2.
We further observe that sampled code w in fact represents a fixed camera viewpoint. That is, if we keep ".," fixed but sample the remaining dimensions of u², StyleGAN produces imagery of different objects in the same camera viewpoint. This is shown in columns in Fig. 2. Notice how aligned the objects are in each of the viewpoints. This makes StyleGAN a multi-view data generator!
We manually select several views, which cover all the common viewpoints of an object ranging from 0-360 in azimuth and roughly 0-30 in elevation. We pay at- tention to choosing viewpoints in which the objects look most consistent. Since inverse graphics works require camera pose information, solute camera pose. To be specific, we classify each viewpoint code into one of 12 azimuth angles, uniformly sampled along 360 deg. We assign each code fixed elevation (09) and camera distance. These camera poses provide a very coarse annotation of the actual pose - -the annotation serves as the initialization of the camera which we will optimize during training. This allows us annotate all views (and thus the entire dataset) in only minute making annotation effort neglible. For each viewpoint, we sample a large number of content codes to synthesize different objects in these views. Fig. 2 shows 2 cars, and a horse and bird. Appendix provides more examples.
"StyleGAN" multi-view dataset:
annotate the chosen viewpoint codes with a rough ab-
Since DIB-R also utilizes segmentation masks during training, we further apply MaskRCNN by He et al. (2017) to get instance segmentation in our generated dataset. As StyleGAN sometimes gener- ates unrealistic images or images with multiple objects, we filter out "bad" images which have more than one instance, or small masks (less than 10% of the whole image area).
3.2 TRAINING AN INVERSE GRAPHICS NEURAL NETWORK
Following CMR by Kanazawa et al. (2018), and DIB-R by Chen et al. (2019), we aim to train a 3D prediction network f. parameterized by 0, to infer 3D shapes (represented as meshes) along with textures from images. Let Iv denote an image in viewpoint V from our StyleGAN dataset, and M its corresponding object mask. The inverse graphics network makes prediction as follows: {5, = fo(Iv), where S denotes the predicted shape, and T texture map. Shape S is deformed from a sphere as in Chen et al. (2019). While DIB-R also supports prediction of lighting, we empirically found its performance is weak for realistic imagery and we thus omit lighting estimation in our work.
To train the network, we adopt DIB-R as the differentiable graphics renderer that takes (S, T) and V as input and produces rendered image I6- = r(S, T, V) along with rendered mask M'. Following DIB-R, the loss function then takes the following form:
S, T, V: =ocot (1, 1')= + (pscep (1, Lio
(1)
+ +smm Lmm (S)) + >tap (S) + Amoo L (S)
Here, Lest is the standard L1 image reconstruction loss defined in the RGB color space while Ipecpt is the perceptual loss that helps the predicted texture look more realistic. Note that rendered images do not have background, so Leal and ['prrcepe are calculated by utilizing the mask. TIoU computes the intersection-over-umne losses such as the Laplacian loss Lisp and flatten loss Lam are commonly used to ensure that the shape is well behaved. Finally, Lmov regularizes the shape deformation be uniform and small.
between the ground-truth mask and the rendered mask. Regularization
Since we also have access to multi-view images for each object we also include a multi-view con- sistency loss. In particular, our loss per object le is:
(x = Σ (L(I Sk. Tr, V/*, 0) + Llyyy, SE. Th. v))
where {Sk, Tk. Lx} = fo([v+t)
4

---

Published as a conference paper at ICLR 2021
3D Neural Renderer
canero
Figure 3: A mapping network maps camera. shape, texture and background into disentangled code that is passed to StyleGAN for "rendering". We refer to this network as StyleGAN-R.
Oieer teekkanik
mich
StyleGAN
bock glound
While more views provide more constraints, empirically, two views have been proven sufficient. We randomly sample view pairs (i,j) for efficiency.
We use the above loss functions to jointly train the neural network f and optimize viewpoint cameras V (which were fixed in Chen et al. (2019)). We assume that different images generated from the same w correspond to the same viewpoint V. Optimizing the camera jointly with the weights of the network allows us to effectively deal with noisy initial camera annotations.
3.3 DISENTANGLING ING STYLEGAN WITH THE INVERSE GRAPHICS MODEL.
The inverse graphics model allows us to infer a 3D mesh and texture from a given image. We now utilize these 3D properties disentangle StyleGAN's latent space, and turn StyleGAN into fully controllable 3D neural renderer, which we refer to as StyleGAN-R. Note that StyleGAN in fact synthesizes more than just an object, it also produces the background, i.e., the entire scene. Ideally we want control over the background as well, allowing the neural renderer to render 3D objects into desired scenes. To get the background from a given image, we simply mask out the object.
We propose to learn a mapping network to map the viewpoint, shape (mesh), texture and background into the StyleGAN's latent code. Since StyleGAN may not be completely disentangled, we further fine-tune the entire StyleGAN model while keeping the inverse graphics network fixed.
Mapping Network: Our mapping network, visualized in Figure 3, maps the viewpoints to first layers and maps the shape. texture and background to the last 12 layers of W For simplicity, we denote the first 4 layers as Wi and the last 12 layers as WSTB- where Wy E 12048 and WSTB E I. 3008 Specifically, the mapping network 9. for viewpoint and 9s for shape S separate MLPs while for texture T and Gb for background B are CNN layers:
z view
(V; z zhape = =S.(S; 2 (T(;)))), z bck go (B;6))
(3)
where view 2048 shappe zxx bbck R 3008 and Our 0., 02 0, are network parameters.
We
softly combine the shape, texture and background codes into the final latent code as follows:
tD
shape + + z txt +
z bek
(4)
where @ denotes element-wise product, and s s s E R80000 are shared across all the samples. To achieve disentanglement, we want each dimension of the final code to explained by only one property (shape, texture or background). We thus normalize each dimension of using softmax.
In practice, we found that mapping V to a high dimensional code challenging since our dataset only contains limited number of views, and V is limited to azimuth, elevation and scale. We thus map to the subset of Wi. where we empirically choose 144 of the 2048 dimensions with the highest correlation with the annotated viewpoints. Thus, view E R111 in our case.
Training Scheme: We train the mapping network and fine-tune StyleGAN in two separate stages. We first freeze StyleGAN's weights and train the mapping network only. This warms up the mapping network to output reasonable latent codes for StyleGAN. We then fine-tune both StyleGAN and the mapping network to better disentangle different attributes. We provide details next.
In the warm up stage, we sample viewpoint codes w among the chosen viewpoints, and sample the remaining dimensions of w" E W*. We try to minimize the Ly difference between the mapped code v and StyleGAN's 's code w" To encourage the disentanglement in the latent space, we penalize the entropy of each dimension i of s. Our overall loss function for mapping network is:
mmppne (0,, 0g,0,, = || - w" - Σ km(
S log(s
(5)
By training the mapping network, we find that view, shape and texture can be disentangled in the original StyleGAN model but the background remains entangled. We thus fine-tune the model to get a better disentanglement. To fine-tune the StyleGAN network we incorporate a cycle consis- tency loss. In particular, by feeding sampled shape, texture and background StyleGAN
5

---

Published as a conference paper at ICLR 2021
Inpul
Prediction
Multiple
Input
Prediction
Multiple Views
Figure 4: 3D Reconstruction Results: Given input images (Ist column), predict 3D shape, texture, and render them into the viewpoint (2nd column). We also show renderings in 3 other views in remaining columns showease 3D quality. model able reconstruct cars with various shapes, textures and viewpoints. We also show the same approach on harder (articulated) objects, i.e., bird and horse.
obtain a synthesized image. We encourage consistency between the original sampled properties and the shape, texture and background predicted from the MhyleGAN-syythesiizee image via the inverse graphics network. We further feed the same background B with two different {.S, T pairs gen- erate two images I and Iz. We then encourage the re-synthesized backgrounds B₁ and Bg to be similar. This loss tries disentangle the background from the foreground object. During training, we find that imposing the consistency loss on B in image space results in blurry images, thus we constrain it in the code space. Our fine-tuning loss takes the following form:
Lyyyegan (0gm)) = ||S S112 + I|T - 12 (l)b(B) - gb (B) + (lg)(B,) - go (6)
4 EXPERIMENTS
In this section, showcase approach on inverse graphics tasks (3D image reconstruction). as well as on the task of 3D neural rendering and 3D image manipulation.
Image Datasets for training StyleGAN:
We use three category-specific StyleGAN models, one representing rigid object class, and two representing articulated (and thus more challenging) classes. We use the official car and horse model from StyleGAN2 (Karras et al., 2019b) repo which trained LSUN Car and LSUN Horse with 5.7M and 2M images. We also train bird model on NABirds (Van Horn et al., 2015) dataset, which contains 48k images.
Our "StyleGAN" Dataset: We first randomly sample 6000 cars, 1000 horse and 1000 birds with diverse shapes, textures, and backgrounds from StyleGAN. After filtering out images with bad masks as described in Sec. 3, 55429 cars, 16392 horses and 7948 birds images remain in our dataset which significant larger than the Pascal3ID car dataset (Xiang et al., 2014) (4175 car images). Note that nothing prevents us from synthesizing a significantly larger amount of data, but in practice, this amount turned out to be sufficient to train good models. We provide more examples in Appendix.
4.1 3D RECONSTRUCTION RESULTS
Training Details: Our DIB-R based inverse graphics model was trained with Adam (Kingma & Ba (2015)), with learning rate of lc-4. We set Aiou, Acol. Alap- Amm and Amov to 3, 20, 5, 5, and 2.5, respectively. We first train the model with Leot loss for 3K iterations, and then fine-tune the model by adding Lpecept to make the texture more realistic. We set Apercept to 0.5. The model converges in 200K iterations with batch size 16. Training takes around 120 hours four V V00 GPUs.
We show 3D reconstruction results in Fig. Notice the quality of the predicted shapes and textures, and the diversity of the 3D car shapes we obtain. Our method also works well on more challenging (articulated) classes, e.g. horse and bird. We provide additional examples in Appendix.
Results:

---

Published as a conference paper at ICLR 2021
Figure 5: Comparison Pascal3D test set: We compare inverse graphies networks trained on Pas- cal3D and our StyleGAN datasct. Notice consider- ably higher quality of pre- diction when training on the StyleGAN dataset.
Piscal3D
Ours
Impul
Prediction
Texture
Multiple Rendered Views Prediction
Full
w.M. Y.
Input
Prodiction
Texture
Prodiction rendered in Multiple Views
Figure 6: Ablation Study: We ablate the use multi-view consistency loss. Both texture are shape are worse without this loss, especially in the invisible parts (rows 2, 5, denoted by "w.o M. V." no multi-view consistency used during training), showcasing the importance of our EayleGAN-mulivee dataset.
Qualitative Comparison: To showcase our approach, we compare our inverse graphics network trained on our StyleGAN dataset with exactly the same model but which we train on the Pascal3D car dataset. Pascal3D dataset has annotated keypoints, which we utilize to train the baseline model, termed as as Pascal3D-model. We show qualitative comparison on Pascal3D test set in Fig. 5. Note that the images from Pascal3D dataset are different from those our StyleGAN-model was trained on. Although the Pascal3I)- predictions in other views are of noticeably lower quality than ours, which demonstrates that we recover 3D geometry and texture better than the bascline.
prediction is visually good in the input image view, rendered
Dataset
Size
Annotation
Model
Pascal3D
StyleGAN
Overall
Shape
Texture
Pascul3D
4K
200-350h Imin
Pascal3D
0.80
0.81
Ours D-edellDe
57.5%
61.6%
56.3%
25, 9%
26.4%
32.8%
StyleGAN
50K
Ours
0.76
0.95
No Preference
11.9%
10.8%
(a) Datasct Comparison
(b) 2D IOU Evaluation
(0) User Study
Table 1: (a): We compare dataset size and annotation time of Pascal3D with our StyleGAN dataset. (b):
We evaluate re-projected 2D IOU score of our StyleGAN-model vs the baseline Pascal3D-model on the two datasets. (c): We conduct user study to judge the quality of 3D estimation.
We evaluate the two networks in Table for the car class. We report the estimated annotation time in Table. (a) showcase efficiency behind StyleGAN dataset. It takes 3-5 minutes to annotate keypoints for one object, which we empirically verify. Thus, labeling Pascal3D required around 200-350 hours while ours takes only minute to annotate a 10 times larger dataset. In Table (b), we evaluate shape prediction quality by the re-projected 2D IOU score. Our model outperforms the Pascal3D-mode on the Pascal test set. This is not surprising since there is domain gap between two datasets and thus each one performs best on their own test set. Note that this metric only evaluates quality of the prediction in input view and thus not reflect the actual quality of the predicted 3D shape/texture.
Quantitative Comparison:
on the SyleGAN test set while Pascal3D-moder is better
To analyze the quality of 3D prediction, we conduct an AMT user study on the Pascal3D test set which contains 220 images. We provide users with the input image and predictions rendered in views (shown in Fig. 5, right) for both models. We ask them to choose the model with a more realistic shape and texture prediction that matches the input object. We provide details of the study in the Appendix. We report results in Table. (c). Users show significant preference of our results versus the baseline, which confirms that the quality of our 3D estimation.
Ablation study: In Fig 6 we ablate the importance of using multiple views in dataset, i.e., by encouraging multi-view consistency loss during training. We compare predictions from inverse graphics networks trained with and without this loss, with significant differences in quality.
7

---

Published as a conference paper at ICLR 2021
Input
DIBR-R
Stylegan-R
Inpur
DIBR-R
Stylegan-R
Figure 7: Dual Renderer: Given input images (Ist column), we predict mesh and texture, and render
them with the graphics renderer (2nd column), and our StyleGAN-R (3rd column).
Input
Mapping from 3D properties
latent code optimization
Figure 8: Latent code manipulation: Given an input image (col 1), we predict 3D properties and synthesize a new image with StyleGAN-R, by manipulating the viewpoint (col 2, 3, 4). Altematively, we directly opti- mize the (original) StyleGAN latent code w.r.t. image, however this leads to blurry reconstruction (col 5), Morcover, when we try to adjust the style for the optimized code, we get low quality results (col 7).
Elevation Arimath Scale
Figure Camera Controller:
We manipulate azimuth, scale, elevation parameters with StyleGAN-R to synthesize images in viewpoints while keeping content code fixed.
Sampled
Shupe Swap
Texture Swap
Background Swap
Figure 10: 3D Manipulation:
We sample 3 cars in column 1. We replace the shape of all cars with the shape of Car I (red box) in 2nd column. We transfer tex- ture
Car
2 (green box) to other cars (3rd col). In last column, we paste background of Car 3 (cyan box) to the other cars. Exam- ples indicated with boxes changed. Zoom to see details.
Car-
Car
Car
Input
Neural R
Shape Swap
Texture Swap
Bek Swap
Figure 11: Real Image Manipulation: Given input images (Ist col), we predict 3D properties and use our StyleGAN-R to render them back (2nd col). We swap out shape, texture & background in cols 3-5,
4.2 DUAL RENDERERS
Training Details: We train StyleGAN-R using Adam with learning rate of le-5 and batch size 16. Warmup stage takes 700 iterations, and we perform joint fine-tuning for another 2500 iterations.
With the provided input image, we first predict mesh and texture using the trained inverse graphics model, and then feed these 3D properties into StyleGAN-R to generate a new image. For com- parison, we feed the same 3D properties to the DIB-R graphics renderer (which is the OpenGL renderer). Results are provided in Fig. 7. Note that DIB-R can only render the predicted object, while StyleGAN-R also has the ability render the object into desired background. We find that StyleGAN-R R produces relatively consistent images compared to the input image. Shape and texture well preserved, while only the background has slight content shift.
8

---

Published as a conference paper at ICLR 2021
4.3 3D IMAGE MANIPULATION WITH STYLEGAN-R
We test our approach in manipulating SyyeGAAN-yythhssin images from our test set and real im- ages, Specifically, given an input image, we predict 3D properties using the inverse graphics net- work, and extract background by masking out the object with Mask-RCNN, We then manipulate and feed these properties StyleGAN-R synthesize new views.
Controlling Viewpoints: We first freeze shape, texture and background, and change the camera viewpoint. Example is shown in Fig. 9. We obtain meaningful results, particularly for shape and texture. For comparison, an alternative way that has been explored literature is to directly optimize the GAN's latent code (in our case the original StyleGAN's code) via an L2 image reconstruction loss. Results are shown in the last three columns in Fig. As also observed in Abdal et al. (2019), this approach fails to generate plausible images, showcasing the importance of the mapping network and fine-tuning the entire architecture with 3D inverse graphics network in the loop.
Controlling Shape, Texture and Background: We fürther aim to manipulate 3D properties, while keeping the camera viewpoint fixed. In the second column of Fig 10, we replace the shapes of all cars to one chosen shape (red box) and perform neural rendering using StyleGAN-R. We suc- cessfully swap the shape of the car while maintaining other properties. We are able to modify tiny parts of the car, such as trunk and headlights. We do the same experiment but swapping texture and background in the third and forth column of Fig 10. We notice that swapping textures also slightly modifies the background, pointing that further improvements are possible in disentangling the two.
Real Image Editing: As shown in Fig. 11, our framework also works well when provided with real images, since StyleGAN's images, which we use in training, are quite realistic.
4.4 LIMITATIONS
While recovering faithful 3D gemetry and texture, our model fails predict correct lighting. Real images and BoyleGAN-generatee images contain advanced lighting effects such as reflection, trans- parency and shadows, and our spherical harmonic lighting model is incapable in dealing with it successfully. We also only partly succeed at disentangling the background, which one can see by noticing slight changes in background in Fig. 7, Fig. 10 and Fig. 11. Predicting faithful shapes for out-of-distribution objects as discussed in Appendix is also a significant challenge. We leave improvements to future work.
CONCLUSION
In this paper, we introduced a new powerful architecture that links two renderers: a state-of-the-art image synthesis network and differentiable graphics renderer. The image synthesis network gen- erates training data for an inverse graphics network. In turn, the inverse graphics network teaches the synthesis network about the physical 3D controls. We showcased our approach to obtain signif- icantly higher quality 3D reconstruction results while requiring 10,000x less annotation effort than standard datasets. We also provided 3D neural rendering and image manipulation results demon- strating the effectiveness of our approach.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? CoRR, abs//1904.0 0989,3199,,.. 2019. httttt///aaaa wwwpxxiivvrrraoooooooooooooeeee wtw.//rg////.//...//......e////. 111./9900......... 03189.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich model reposi- tory. arXiv preprint Xivi 1512.03012, 2015.
Wenzheng Chen, Jun Gao, Huan Ling, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation based differentiable renderer. In Advances In Neural Information Processing Systems, 2019.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, llya Sutskever, and Picter Abheel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2172-2180. 2016.
Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In ECCV, 2016.

---

Published as a conference paper at ICLR 2021
Alexey Dosovitskiy, Jost Tobias Springenberg, Maxim Tatarchenko, and Thomas Brox. Learning generate chairs, tables and cars with convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 39(4)692-7705 2016.
Jun Gao, Wenzheng Chen, Tommy Xiang, Clement Fuji Tsang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Learning deformable tetrahedral meshes for 3d reconstruction. In Advances In Neural Information Processing Systems, 2020.
Shubham Goel, Angjoo Kanazawa, and Jitendra Malik. Shape and viewpoints without keypoints. In ECCV, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair. Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q Weinherger (eds.), Advances Neural Information Processing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014. URL http:/ httt::/papprss nii...ipp....................... www.:ic..ppppppppppo cc/paper/ 5423- www.rraivvveeaaveerrraaa----aaee
Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan Russell, and Mathieu Aubry. AtlasNet: A Papier- Mâché Approach to Learning 3D Surface Generation. In Proceedings IEEE Conf. Computer Vision and Puttern Recognition (CVPR), 2018
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. arXiv preprint arXiv:2004. 02546, 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. CoRR, abs// 703.06870, 2017. URL htt.htt::/// / // ..w.vvvoorro...oo......... at//ab///110 700010067....0.. 06870.
Paul Henderson and Vittorio Ferrari. Learning to generate and reconstruct 3d meshes with only 2d supervision. arXiv preprint arXiv: arXiv:/807.0000 800,00999.99 2018
Krishna Murthy J., Edward Smith, Jean- Francois Lafleche, Clement Fuji Tsang, Artem Rozantsev, Wenzheng Chen, Tommy Xiang. Rev Lebaredian, and Sanja Fidler. Kaolin: A pytorch library for accelerating 3d deep learning research. arXiv: /9/ I. 1.05063, 2019.
Angjoo Kanazawa, Shubham Tulsiani, Alexci A Efros, and Jitendra Malik. Learning category-specing mesh reconstruction from image collections. ECCV, pp. 371-386, 2018
Levent Karacan, Zeynep Akata, Aykut Erdem, and Erkut Erdem. Learning to generate images outdoor scenes from attributes and semantic layouts. arXiv preprint arXiv:1612.00215. 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of StyleGAN. CoRR, ahs/1912.04958,, 2019b.
Hiroharu Kato and Tatsuya Harada. Self-supervised learning of 3d objects from natural images, 2019.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In CVPR, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun ). 3rd International Conference on Learning Representatiinss ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: /ttt:://arxiin /awpiv.vvii..//......../aaaaa.va org/ abs/ 11112.11112 6980.
Wonkwang Lec, Donggyun Kim, Seunghoon Hong, and Honglak Lee. High-fidelity synthesis with disentangled representation. arXiv preprint arXiv:2001.04296 2020.
Daiging Li, Junlin Yang. Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In CVPR, 2021.
Tzu-Mao Li, Miika Aittala. Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. In SIGGRAPH Asia 2018 Technical Papers, pp. 222. ACM, 2018.
Xucting Li, Sifci Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Hing-Hsuan Yang, and Jan Kautz. Self-supervised single-view 3d reconstruction via semantic consistency. In ECCV, 2020.
Zinan Lin, Kiran Koshy Thekumparmmpi,, Giulia Fanti, and Sewoong Oh. Infogan-cr: Disentangling generative adversarial networks with contrastive regularizers. arXiv preprint 1906::9906.060:. 2019.
10

---

Published as a conference paper at ICLR 2021
Hsueh-Ti Derek Liu, Michael Tao, Chun-Liang Li. Derek Nowrouzezahrai, and Alec Jacobson. Beyond pixel norm-balls: Parametric adversaries using an analytically differentiable renderer. In ICLR, 2019a.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: differentiable renderer for image-based 3d reasoning. ICCV, 2019b.
Matthew M. Loper and Michael J. Black. Opendr: An approximate differentiable renderer. In David J. Flect, Tomás Pajdla, Bernt Schiele. and Tinee Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Swizzrlandd, September 6-12, 2014, Proceedings. Part VII, volume 8695 of Lecture Notes Computer Science, pp. 154-169. Springer, 2014. doi: 10.1007/978-3-319 10584-0) URL https: sttps://dee /tt:///cciic/ ctt. 1000000007770 3 10584400011-1111
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowazin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Com- puier Vision and Puttern Recognition, pp. 4460 4470. 2019.
Jeong Joon Park. Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Guim Perarnau, Jonst Van De Weijer, Bogdan Raducanu, and Jose M Álvarez. Invertible conditional gans for image editing. arXiv preprint arXiv: /t/::///11000 1.06355, 2016.
Felix Petersen, Amit H. Bermano, Oliver Deussen, and Daniel Cohen-Or. Pix2vex: Image-to-goomeet recon- struction using smooth diffcrentiable renderer. CoRR, abs/1903.11144 2019. URL http://arxix
org/abs /100.. 11149.
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan- Yen Lo, Justin Johnson, and Georgia Gkioxari. Pytorch3d. https://githbbe com/
hwtpcchrpppthtcteeeoooooo 2020.
Scott Reed, Zeynep Akata, Xinehen Yan, Lajanugen Logeswaran, Berut Schiele, and Honglak Lee. Generative adversarial text to image synthesis. arXiv preprint 160500600 2016.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans, arXiv preprint arXiv:2005.09635, 2020.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-wware neural scene representations. In Advances in Neural Information Processing Systems, 2019.
Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zällhofer, and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait images, cvpr 2020. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEF, june 2020.
Antonio Torralba, Bryan C Russell, and Jenny Yuen. Labelme: Online image annotation and applications. Proceedings of the IEEE, 98(8): 1467-1444, 2010. doi: 10.1 (009//pro.2010.2000000
Julien Valentin, Cem Keskin, Pavel Pidlypenskyi, Ameesh Makadia, Avneesh Sud, and Sofien Bouaziz. Ten- sortlow graphics: Computer graphics meets deep learning. 2019.
G. Van Horn, S. Branson, R. Farrell, S. Haber, J. Barry. P. Ipeirotis. Perona, and S. Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print fine-grained dataset collection. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 595-604, 2015.
Nanyang Wang. Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In ECCV, 2018.
Weiyue Wang. Xu Qiangeng, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view reconstruction. arXiv preprint 10711 2019.
Welinder, Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and Perona. Caltech-UCSD Birds 200.
Technical Report 2NS-TR-2010-001, California Institute of Technology, 2010.
Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Unsupervised learning of probably symmetric de- formable 3d objects from images in the wild. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
11

---

Published as a conference paper at ICLR 2021
Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond paseal: A benchmark for 3d object detection in the wild. In JEEE Winter Conference Applications of Computer Vision (WACV), 2014.
Shunyu Yao, Tzu Ming Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, Bill Freeman, and Josh Tenenbaum. 3d-aware scene manipulation inverse graphics. Advances neural information processing systems, pp. 1887-1898, 2018.
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean- Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR, 2021.
Jun- Yan Zhu, Taesung Park, Phillip Isola, Alexei Efros. Unpaired image-to-ine to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
Jun Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenenbaum, and Bill Freeman. Visual object networks: Image generation with disentangled 3d representations. In Advances in neural information pmcessing systems, pp. 118-129, 2018.
12

---

Published as a conference paper at ICLR 2021
APPENDIX
A OVERVIEW
In the Appendix, we first show feature visualization of StyleGAN layers Sec. B. We then provide detailed explanation of our StyleGAN dataset creation in Sec. C, including examples of the generated images and selected viewpoints. Next, we do a systematic analysis our camera initialization method in Sec. D. Finally, we show additional results on the 3D inverse graphics task in Sec. E, additional details the user study in Sec. F, futher examples of StyleGAN disentanglement in Sec. G, with ablation studies and discussion of limitations in Sec. H and Sec. K, respectively.
B STYLEGAN LAYERS VISUALIZATION
The official StyleGAN code repository provides models different object categories at different resolutions. Here take the 512 x 384 model as the example. This model contains 16 layers, where every two consecutive layers form a block. Each block has a different number of channels. In the last block, the model produces 32-channel feature map at a 512 x 384 resolution. Finally, learned RGB transformation function is applied to convert the feature map into an RGB image.
We visualize the feature map for each block via the learned RGB transformation function. Specif- ically, for the feature map in each block with the size of h x w X C, we first sum along the feature dimension, forming a X tensor. We then repeat the feature 32 times and generate h w 32 new feature map. This allows us to keep the information of all the channels and directly apply the RGB transformation function in the last block convert it to the RGB image.
As shown in Fig A, we find that blocks and do not exhibit interpretable structure while the car shape starts appear blocks 3-5. We observe that there is a rough car contour in block 4 which further becomes clear in block 5. From blocks 6 to 8, the car's shape becomes increasingly finer and background scene also appears. This supports some of our findings, i.e., the viewpoint controlled in block and 2 (first 4 layers) while shape, texture, and background exist in the last 12 layers. Block
Block 2
Block 3
Block 4
Block 5
Block 6
Block 7
Block 8
Generated Image
Figure A: Layer Visualization for Each Block: Notice that the car contour starts to appear in blocks and higher. This supports some of our findings that the early blocks control viewpoint (and other global properties), while shape, texture and background controlled in the higher layers.
C OORR "STYLEGAN" DATASET
We visualize all of our selected viewpoints in our dataset in Fig. B. Our car training dataset contains 39 viewpoints. For the horse and bird datasets, we choose 22 and views, respectively. We find that these views are sufficient to learn accurate 3D inverse graphics networks. We could not find views that would depict the object from a higher up camera, i.e., a viewpoint from which the roof of the the back of the horse would be more clearly visible. This is mainly due to the original dataset on which StyleGAN was trained on, which lacked such views. This leads to challenges in training inverse graphics networks to accurately predict the top of the objects.
13

---

Published as a conference paper at ICLR 2021
Car Viewpoints
Horse Viewpoints
Bird Viewpoints
Figure B: All Viewpoints: We show an example of car, bird and a horse synthesized in all of our chosen viewpoints. While shape and texture not perfectly consistent across views, they sufficiently accurate enable training accurate inverse graphics networks in downstream tasks. Horses and birds are especially challenging due to articulation. One can notice small changes articulation across viewpoints. Dealing with articulated objects is subject to future work.
Notice the high consistency of both the car shape and texture as well as the background scene across the different viewpoints. Note that for articulated objects such as the horse and bird classes, StyleGAN does not perfectly preserve object articulation in different viewpoints, which leads to challenges training high accuracy models using multi-view consistency loss. We leave further investigation of articulated objects to future work.
We further show examples from our StyleGAN--enerated dataset Fig. C. Our dataset contains objects with various shapes, textures and viewpoints. In particular, in the first six rows, one can notice a diverse variants of car types (Standard Car, SUV, Sports car, Antique Car, etc). We find that StyleGAN can also produce rare shapes like trucks, but with a lower probability.
14

---

Published as a conference paper at ICLR 2021
oonne
Cafe
Figure C: Dataset Overview: synthesize multi-view datasets for three classes: car, horse. bird. Our datasets contain objects with various shapes, textures and viewpoints. Notice the consistency of pose object each column (for each class). Challenges include the fact that for all of these objects StyleGAN has not learned to synthesize views that overlook the object from above due to the photographer bias in the original dataset that StyleGAN trained on.
15

---

Published as a conference paper at ICLR 2021
D CAMERA INITIALIZATION
Inverse graphics tasks require camera pose information during training, which is challenging to ac- quire for real imagery. Pose is generally obtained by annotating keypoints for each object and run- ning frructurefrrommmotie camera parameters. However, keypoint annotation is quite time consuming requiring roughly 3- Sminutes per object which we verify in practice using the LabelMe interface (Torralba et al., 2010). In our work, we utilize StyleGAN significantly reduce annotation effort since samples with the same wy share the same viewpoint. Therefore, we only need to assign a few selected w,t into cam- era poses. In particular, we assign poses into several bins which we show is sufficient for training inverse graphics networks where, along with the network parameters, cameras get jointly optimized during training using these bins as initialization.
(SFM) techniques (Welinder et al., 2010; Xiang et al., 2014) to compute
Specifically, we assign poses into 39, 22 and 8 bins for the car, horse and bird classes, respectively. This allows us to annotate all the views (and thus the entire dataset) in only I minute. We do acknowledge additional time in selecting good views out of several candidates.
We annotate each view with a rough absolute camera pose (which we further optimize during train- ing). To be specific, we first select 12 azimuth angles: [ 60P, 90", 120", 150", 180 210", 240 2700, 300 330%]. Given a StyleGAN viewpoint, we manually classify which azimuth angle it is close to and assign it to the corresponding label with fixed elevation (09) and camera distance.
To demonstrate the effectiveness of our camera initialization, we make comparison with another inverse graphics network trained with a more accurate camera initialization. Such initialization is done by manually annotating object keypoints in each of the selected views (wx) of a single car example, which takes about 3-4 hours (around 200 minutes, 39 views). Note that this is still a significantly lower annotation effort compared to 200-350 hours required to annotate keypoints for every single object in the Pascal3D dataset. We then compute the camera parameters using SfM We refer to the two inverse graphics networks trained with different camera initializations as view- model and keypoint -model, respectively.
We visualize our two different annotation types in Fig D. We show annotated bins in the top. We annotated keypoints for the (synthesized) car example in the first image row based on which we com- pute the accurate viewpoint using SIM. To showcase how well aligned the objects for the same viewpoint code, we visualize the annotated keypoints on all other synthesized car examples. Note that we do not assume that these keypoints are accurate for these cars (only the implied viewpoint).
We quantitatively evaluate two initialization methods in Table, D. We first compare the annotation and training times. While it takes the same amount of time train, view-model saves annotation time. The performance of view-model and keypoint -model are comparable with almost the same 2D IOU re-projection score on the StyleGAN test set. Moreover, during training the camera systems converge to the same position. We evaluate this by converting all the views into quaternions and compare the difference between the rotation axes and rotation angles. Among all views, the average difference of the rotation axes is only 1.43 and the rotation angle is 0.42", The maximum difference of the rotation axes is only 2.95 and the rotation angle is 1.11
We further qualitatively compare the two methods in Fig. E, showing that they perform very simi- larly. Both, qualitative and quantitative comparisons, demonstrated that tdew-camera initialization is sufficient for training accurate inverse graphics networks and no additional annotation is required. This demonstrates a scaleable way for creating multi-view datasets with StyleGAN, with roughy a minute of annotation time per class.
E 3D INFERENCE
We here present additional 3D prediction results and compare our model, which is trained on our StyleGAN generated dataset (StyleGAN-model), with the one trained on the Pascal 3D dataset (Xi- ang et al., 2014) (PASCAL-model), We qualitatively compare two models on the Pascal3D test set in Fig. and web imagery in Fig. G. Our StyleGAN model produces better shape and texture predictions in all the testing datasets, which is particularly noticeable when looking different ren- dered views of the prediction. We also present additional 3D prediction results on horses and birds in Fig. H.
16

---

Published as a conference paper at ICLR 2021
Azimuth=0"
Azimuth=30
Azimuth=30
Azimuth=180"
Azimuth=210
Azimuth=270*
Figure D: We show examples of cars synthesized in chosen viewpoints (columns) along with annotations. Top row shows the pose bin annotation, while the images show the annotated keypoints. We annotated keypoints for the example the first image-row hased on which To showcase how well aligned the objects are for the same viewpoint latent code, we visualize the annotated keypoints on all other synthesized car examples. Note that we do not assume that these keypoints are accurate for these cars (only the implied viewpoint). Annotating pose bins took I min for the car class, while keypoint annotation took 3-4 hours, both types of annotations thus being quite efficient. We empirically find that pose bin annotation is sufficient in training accurate inverse graphics networks (when optimizing camera parameters during training in addition to optimizing the network parameters).
compute the accurate camera parameters using SfM.
knyppiint fait. voleit-niit
Input
Pred.
Multiple Views for the Predicted Shape and Texture
The first row shows predictions from keypoint- Initialization (cameras computed by running SFM on annotated keypoints) and the second row show results obtained by training with view-Initialization (cameras are coarsely annotated into 12 view bins). Notice how close the two predictions are, indicating that coarse viewpoint annotation is sufficient for training accurate inverse graphics networks. Coarse viewpoint annotation can be done in I minute.
Figure E: Comparison of Different Camera Initializations:
Annotation Type
Annotation Time
Training Time
2D IOU
Quaternion
Mean
Max
keypoint view
3-4h Imin
60h
0.953
1.43
2.95
60h
0.952
Gu
0.42
1.11
(a) Time & Performance
(b) Camera Difference after Training
Table A: Comparison of Different Camera Initializations: First table shows annotation time required for the StyleGAN dataset, and training times the view-model and kegpoint- on the dataset with respective annotations (binned viewpoints or cameras computed with SFM from annotated kcypoints). The view-model requires significantly less annotation time, and its final performance is comparable to the keypoint- model.. Sec- ond table shows the difference of the camera parameters after training both methods (which optimize cameras during training). They converge to very similar camera positions. This shows that coarse view annotation along with camera optimization during training is sufficient in training high accuracy inverse graphics networks.
17

---

Published as a conference paper at ICLR 2021
Paxcai3
Ohea
PascalDD
Oura
Pascal5
Oan
Pasca
Chrs
ParcalDD
Oons
Phess
Omes
Input
Pred.
Multiple Views
Figure F: Comparison PASCAL3D imagery: We compare PASCAL-model with StyleGAN-mode -odell on PASCAL3D test set. While the predictions from both models are visually good in the corresponding image view, the prediction from StyleGAN-model have much better shapes and textures as observed in other views.
18

---

Published as a conference paper at ICLR 2021
PiscaliD
Ouns
PaccalD
Ours
PoccalDD
Ouss
PiscalSD
OLES
PiscalDD
Owns
Input
Prod. Multiple Views
Figure G: Comparison on Images from the Web: We compare PASCAL-model with our StyleGAN. model on images downloaded from the web. While the predictions from both models are visually good in the corresponding image view, the prediction from StyleGAN -odell have much better shapes and textures as observed in other views.
19

---

Published as a conference paper at ICLR 2021
Input
Pred.
Multiple Views for the predicted shape and texture
Figure Reconstruction Results for Car, Horse and Bird Classes: show car, horse bird examples tested on the images from the StyleGAN datasct test sets. Notice that the model struggles little in reconstructing the top of the back of the horse, since such views are lacking in training.
20

---

Published as a conference paper at ICLR 2021
Instruction
An until inaiiience agent is sociing the phode and rrvnng to gueas difieen mnphe imapes toal, Please acent
In tehhe fieit when par sheoe the colore of fevert on
onope objes
prefemence
sacond cbject
T
oecrall bestl partn
Co pedformee
4/5
Next
Submi
Figure I: User Study Interface (AMT): Predictions are rendered in 6 views and we ask users to choose the result with more realistic shape and texture that is relevant to the input object. We compare hoth the baseline (trained Pascal3D dataset) and (trained on StyleGAN dataset). We randomize their order in each HIT.
F
USER STUDY
We provide user study details in this section. We implement our user interface, visualized in in Fig. on Amazon Mechanical Turk. We show the imput image and predictions rendered in 6 views such that users can better judge the quality of 3D reconstruction. We show results for both, our inverse graphics network (trained on the StyleGAN dataset) and the one trained on the Pascal3D dataset. We show shape reconstruction and textured models separately, such that users can judge the quality of both, shape and texture, more easily. We randomize the order of ours baseline in each HIT to avoid any bias. We ask users to choose results that produce more realistic and representative shape, texture and overall quality with respect to the input image. We separate judgement of quality into these three categories to disentangle effects of 3D reconstruction from texture prediction. We also provide "no preference" options in of ties. Our instructions emphasize that more "representative" results of the input should be selected, to avoid users being biased by good looking predictions that are not consistent with the input (e.g., such as the case of overfit networks).
We evaluate the two networks on all 220 images from the Pascal3D test set (which are "in-domain" for the Pascal3D-trainee network). For each image we ask three users perform evaluation, which results in 660 votes in total. We report the average of all votes as our final metric. We further report annotator agreement analysis in Table B. For shape, texture, and overall evaluation, there are 88 88.2%, 89.2 89.2%, and 87. 97.2% cases where at least two out of three users choose the same option.
Overall
Shape
Texture
Overall
Shape
Texture
Ours
57.5% %
61.6% %
56.3%
All Agree
26.1 %
29.6%
27.1 % 62.1%
Pascal3D-moot
25.9%
26.4%
32.8%
Two Agree
61.1% 1%
58.6%
No Preference
16.6%
11.9%
10.8%
No Agreement
12.8%
11.8%
10.8%
(a) 3D Quality Study
(b) Annotator Agreement
Table B: User study results: (a): Quality of 3D estimation (shape, texture and overall). (b): Annotators agreement analysis. "No agreement" stands for the case where all three annotators choose different options.
G STYLEGAN-R DISENTANGLEMENT
Given an input image, we infer 3D properties of an object (shape, texture, background) using our inverse graphics network, but can also map these properties back to the latent code and use our StyleGAN-R to synthesize a new image. We show the results in Fig. J. Similar to Fig. in the main paper, we show DIB-R-rendered predictions and neural rendering StyleGAN-R's predictions, and manipulate their viewpoints in rows (1, 4) and (2, 5). We further show "neural rendering" results from the original StyleGAN in row (3, 6), where we only leam the mapping network but keep the StyleGAN's weights fixed. We find that fine-tuning is necessary and StyleGAN-R produces more consistent shape, texture and background.
21

---

Published as a conference paper at ICLR 2021
DIB-2
StylGANN
StyleGAN
DIB-R
SpleGAN
StyeGAN
Input
Pred.
Multiple Views
Figure J: Dual Rendering: Given the input image, we show the DeB-R-rendered predictions in rows (1, 4) and StyleGAN-R's results rows (2, 5). We further shows the neural rendering results from the original StyleGAN model, where we only learn the mapping network but keep the StyleGAN weights fixed. Clearly, after fine-tuning, StyleGAN-R produces more consistent results.
Input
w. Light
W.O. Light
Input
w. Light
w.o. Light
Figure K: Light Prediction: Given the input image, we show rendering (using the OpenGL renderer used DIB-R) results with light (columns 2, 5) and results with just textures (columns 3, 6). We find that the two results are quite similar, which indicates that we did not learn a good predictor for lighting. Moreover, we find that higher order lighting, such reflection, high-specular light are merged into texture, shown in the second row. We aim to resolve this limitation in future work.
Real Image Editing:
We show additional real-image editing examples in Fig. L. With our StyleGAN-R, easily change the car's size, azimuth and elevation and synthesize new image while preserving the shape and texture of the car with a consistent background.
H ABLATION STUDIES
We find that the multi-view consistency and perceptual losses play import role in training, as shown in Fig. P. Multi-view consistency loss helps in training a more accurate inverse graphics network in terms of shape, while the perceptual loss helps to keep texture more realistic.
22

---

Published as a conference paper at ICLR 2021
Input
StyleGAN-R
Manipulate Scales
Input
StyleGAN-R
Manipulate Azimuths
Input
StyleGAN-R
Manipulate Elevations
Figure L: Real Image Editing, Given an input image (column 1), we use our inverse graphics network to predict the 3D properties and apply StyleGAN-R to re-render these (column 2, 3). We manipulate the car size/scale (row 1-3), azimuth (row 4-6) and elevation (Row 7-9).
1 STYLEGAN MANIPULATION
We show that our method for manipulating StyleGAN is generalizable and can be generalized to other class, as illustrated in the StyleGAN-R manipulation results for the bird in Fig M and Fig N.
FAILURE CASES
We find that our inverse graphics network fails on out-of-distribution images/shapes, as shown in Fig. O. For example, the reconstruction results for Batmobile and Flinstone cars are not representa- tive of the input cars. We anticipate that this issue can be addressed by augmenting the dataset on which StyleGAN is trained with more diverse objects. Part of the issue is also caused by GANs not capturing the tails of the distribution well, which is an active area research.
23

---

Published as a conference paper at ICLR 2021
Scale
Azimath
Flevation
Figure M: Bird Camera Controller: We manipulate azimuth, scale, elevation parameters with StyleGAN-R to synthesize images in new viewpoints while keeping content code fixed.
Sampled Cars
Shape Swap
Texture Swap
Background Swap
Figure N: Bird 3D Manipulation: We sample 3 birds in column 1. We replace the shape of all birds with the shape of Bird (red box) 2nd column. We transfer texture Bird 2 (green box) to other (3rd col). last column, we paste background of Bird (cyan box) to the other birds. Examples indicated with boxes are unchanged.
K LIMITATIONS
Our simple spherical harmonics model fails to separate light from textures. We show several exam- ples Kig.K. We leave this issue for future work.
24

---

Published as a conference paper at ICLR 2021
Input
Multiple Views for the predicted shape and texture
We show examples of failure cases for car, bird and horse. Our method tends to fail to produce relevant shapes for objects with -ittr---iitribttion shapes (or textures).
Figure O: 3D Reconstruction Failure Cases:
Fril
moo Mi V
w.P.
Fulll
Vou hi V.
wio P
Input
Pred.
Texture
Pred. Multiple Views
We ablate use multi-view consistency and perceptual losses by showing results of 3D predictions. Clearly, the texture becomes worse in the invisible part if we remove the multi- view consistency loss (rows 2, 5, denoted by "w.o M. V.". which denotes that multi-view consistency was used during training). showcasing the importance become quite smooth and lose details we do not use the perceptual (rows 3, 6, noted by "w.o which denotes that no perceptual loss was used during training).
Figure P: Ablation Study:
StyleGAN-multivew dataset. Moreover, the textures
25


Reviews:
[{"review_id": "g0fsfF2Kcx0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors provided a framework for inverse graphics, i.e., infer 3D mesh, light, and texture from a 2D image. They first use StyleGAN to generate realistic multi-view images, and then trained their model with a differentiable graphics renderer.\n\nPros\n- Training on data generated by StyleGAN is novel. It is easy to control the view of rendered images, but they usually do not look real. Real images look real, but we usually do not know the viewpoint (most annotated datasets are either small or have limited annotation quality). The authors proposed a method to generate images of the same category of objects with the same viewpoint (Figure 2), which addresses this issue -- the viewpoints are known, and the images look real.\n- The experimental part is impressive. 3D reconstructions look realistic, and the authors demonstrated the effectiveness to train on StyleGAN by comparing their model to a neural network trained on PASCAL3D+.\n- The way the authors generate data by StyleGAN is also novel. They empirically realized some latent code in StyleGAN controls the camera viewpoint, so that they only need to choose some latent codes that well spanned over the space. It is a wise and elegant way to control the viewpoints of an object in StyleGAN.\n\nIn summary, compared to previous methods, generating multi-view images from StyleGAN solves the unrealistic and unknown viewpoint issue, and the results look impressive.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DjxMybcgm5b", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes to couple a GAN, an inverse graphics network, and a differentiable renderer. The authors base their work on StyleGAN, and use the observation that a specific part of the latent code corresponds to camera view-point to rapidly annotate a large amount of synthetic images with approximate camera pose. They then use these images and rough annotations to train the inverse graphics network to provide 3D and texture data. The differentiable renderer is used to synthesize 2D images from 3D, which can be compared to the input for consistency. In a second step, the authors use the inferred 3D data to disentangle the latent space of StyleGAN to allow to use it as a controllable renderer.\n\n--- Strengths ---\n\nThe presented results look great. The inverse graphics network, seems to get both shape and textures mostly right. This is especially remarkable, given the small amount of supervision that was used. The controllable StyleGAN provides very plausible results. Most importantly, results  on real images are shown, and indicate the the presented approach does make progress in brining inverse graphics networks to real images.\n\nWhile the individual parts of this work are taken from other prior works, the specific combination of different components is novel and quite creative. Using a GAN to train an inverse graphics network is to the best of my knowledge novel. So is using an inverse graphics network to turn a GAN into a controllable renderer. There are also various key-insights that make the approach work, such as the that StyleGAN is partially  disentangled with respect to view-point, or the use of multi-view consistency when training the inverse graphics network.\n\n--- Weaknesses ---\n\nThe paper is overall well written, but leaves out some technical descriptions which make it not self-contained and hard to reproduce. Specifically, I'd like to ask the authors to elaborate more on the individual loss terms in Equation (1). Especially, the 3D-related losses $L_{lap}$ and $L_{mov}$ are unclear. \n\nWhile many results are shown in the supplement, I would have loved to see a video that shows more results. For example, rotating the obtained 3D models, or showing interpolations of various factors for the controllable renderer. Even if the results are not perfect, a video would help to judge the overall quality of the results.\n\n--- Summary ---\n\nThis paper shows an interesting pipeline with good results. It presents some non-trivial observations and effectively leverages them into a complete system that looks quite impressive.\n\nTypos:\n\nPage 3, last sentence: \"conten\"\n\n--- Post rebuttal ---\nAfter reading the other reviewers comments and the rebuttal, I'm keeping my initial score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0DYLDDo8as", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use StyleGAN images to train a differential renderer, then in turn to use this differential renderer to train a more controllable GAN (dubbed StyleGAN-R) which allows to perform independent manipulation of shape, background, and texture. There are many different things in this paper, and I am struggling a bit to decide what's the main contribution, here are the ones that seem the main one to me (note I am neither a GAN nor a neural renderer expert):\n\n1. training a neural renderer with GAN images: to the best of my knowledge, that has never been demonstrated and is nice in itself. However, the way this is achieved is not completely satisfying: the entire idea is based on the empirical observation that the first part  of styleGAN latent codes correspond to viewpoint; the training requires manual selection of \"good\" latent codes and annotation of the viewpoints (once, this is fast, but remains very unsatisfying and strongly limits the number of viewpoints); further image filtering is required using a trained MaskRCNN and masks are necessary for training. Additionally, while this is nice to see, there seem to be very little/no technical contribution in this part which is mainly engineering and empirical results\n\n2. Another way to see the first contribution is to say that the paper demonstrate that using GAN generated images allows to improve over using datasets of manually annotated images. This has been an important hope as an application of GAN, and the beginning of the results section (4.1) seems to emphasize this aspect: less annotation time and higher quality than training a differential renderer on Pascal3D. Again, while this would make a very nice story, this is not related to any technical contribution. Additionally, it's hard to really trust the results on this, since the training of the networks are complicated and the results are hard to evaluate, it might be that the hyperparameters are simply better adapted to training with the StyleGAN training set than with Pascal3D (the presented AMT experiment is good for evaluation, but it is too expensive to do to use it to choose training hyperparameters)\n\n3. StyleGAN-R for decorelated image manipulations. This is again very appealing, but done in an unsatisfying way and with limited results (only on cars). There are many tricks involved in making this work, from the choice of 144 (!?) dimension among the 2048 for the latent vector to the complicated stage-wise training. The results of this step look good, but are only shown for cars, since reconstruction was also done for horses and birds, I can only assume that this doesn't work for other categories.\n\nTo summarize, I think there are several cool stories in this paper and some appealing results. While the methods remain unsatisfying, the paper shows interesting proof of concepts, I would thus tend to accept it, but I could be convinced otherwise especially if other reviewers who know the field better than I do point to other papers demonstrating similar points. \n\nI list bellow more specific issues/requests:\n\na. whether it works or not, I want to see the equivalent of figures 8-9-10 with other categories. This should also be discussed in limitations (or the paper should demonstrate it works). I am annoyed that this is not discussed more visibly in the paper and kind of hidden. No clear answer on this point would lead me to recommend rejection (but I could still recommend accept even if the method doesn't work very well on other categories)\n\nb. I would like to see a comparison between StyleGAN (using the annotated viewpoints) and styleGAN-R for camera controler (figure 8, but using the code predicted by the approach before styleGAN finetuning - not the optimized code as in fig 7), that seems the natural baseline\n\nc. I don't agree at all with the claim in 4.1 that the approach works with articulated classes: horses are indeed articulated, but looking at the horse dataset used (figure C in appendix) reveals that the selected images from styleGAN do *not* present any articulation (on the contrary to most real horse datasets). Thus the approach works simply because styleGAN do not present the same diversity as natural images, and would not work for actual articulated horses, this is more a limitation that a strength.\n\nd. fig 7 comparison with optimization: to me the natural approach/baseline would be to use a much simpler CNN to predict the latent code (and potentially do some local optimization afterward). This type of learning for optimization approach often regularize the problem well, and lead to good results. This would also be more similar in spirit to the proposed approach.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to bring together a GAN, a differentiable renderer, and an inverse graphics model. This combined model learns 3D-aware image analysis and synthesis with very limited annotation effort (order of minutes). The results look impressive, even compared to training on a labeled dataset annotation of which took several orders of magnitude more time.

The reviewers point out the novelty of the proposed system and the very high quality of the results. On the downside, R2 mentions that the model appears over-engineered and some important experimental results are missing. The authors’ response addresses these concerns quite well.

Overall, this is a really strong work with compelling results, taking an important step towards employing generative models and neural renderers “in the wild”. I think it can make for a good oral. 

Decision: Accept (Oral)