Intrinsic Autoencoders for Joint Deferred Neural Rendering
and Intrinsic Image Decomposition
Hassan Abu Alhaija∗,1 Siva Karthik Mustikovela∗,1 Justus Thies2
Varun Jampani3 Matthias Nießner2 Andreas Geiger4,5 Carsten Rother1
1

4

2
3
Heidelberg University
Technical University Munich
Google Research
5
Max Planck Institute for Intelligent Systems, Tübingen
University of Tübingen

Abstract
Neural rendering techniques promise efficient photorealistic image synthesis while providing rich control overscene parameters by learning the physical image formation
process. While several supervised methods have been proposed for this task, acquiring a dataset of images with accurately aligned 3D models is very difficult. The main contribution of this work is to lift this restriction by training
a neural rendering algorithm from unpaired data. We propose an autoencoder for joint generation of realistic images
from synthetic 3D models while simultaneously decomposing real images into their intrinsic shape and appearance
properties. In contrast to a traditional graphics pipeline,
our approach does not require to specify all scene properties, such as material parameters and lighting by hand.
Instead, we learn photo-realistic deferred rendering from asmall set of 3D models and a larger set of unaligned real images, both of which are easy to acquire in practice. Simultaneously, we obtain accurate intrinsic decompositions of real
images while not requiring paired ground truth. Our experiments confirm that a joint treatment of rendering and decomposition is indeed beneficial and that our approach outperforms state-of-the-art image-to-image translation baselines both qualitatively and quantitatively.

Figure 1: Deferred Neural Rendering and Intrinsic Image Decomposition. At training time, our model exploits
normals, albedo and reflections from a small set of 3D models as well as a large set of unpaired RGB images of thesame object category. Our model solves two tasks simultaneously: (i) generating photo-realistic images given the input geometry and basic intrinsic properties, and (ii) decomposing real images back into their intrinsic components.

required for turning 3D content into photo-realistic images.
These parameters are often tuned individually with eachrendered image, making it hard to create a large and diverse
set of rendered images. On the other hand, obtaining a large
number of real images which capture the complex interaction of light with scene geometry and surface properties iseasy. This makes the idea of learning neural image synthesis from real images very attractive.

1. Introduction
State-of-the-art sampling-based rendering engines (e.g.,
Mitsuba[20]) are able to generate photo-realistic images ofvirtual objects which are nearly indistinguishable from realworld photographs. However, this is not an easy task toaccomplish since all intrinsic physical aspects of the virtual object must be accurately modeled, such as accurate 3D
geometry, detailed textures and physically-based materials.
While some of these intrinsics are abundant on the internet, such as the geometry of 3D objects (e.g. Turbosquidand 3D Warehouse), others are hard to obtain, such as highquality materials – ideally in the form of a highly-accurate
spatially-varying BRDF. In addition, sophisticated and slow
rendering algorithms with many tunable parameters (lighting, environment map, camera model, post-processing) are

Several works on conditional image generation [36,41, 19, 9] have exploited paired datasets of real images
with semantic information, including semantic segmentation [36, 9] and body part labels [26] for training realistic
image synthesis models. However, such sparse inputs only
allow limited control over the generated image. This limits the applicability of these methods, e.g., in virtual reality or video game simulations where precise control over
the output is essential. Training a conditional image generation model from richer control inputs would require alarge dataset of paired real images with pixel aligned intrin1

sic properties such as 3D structure, textures, materials andreflections. Obtaining such a dataset is hard in practice.
Our goal is to take a step towards learning a highly controllable realistic image synthesis model without requiring
real world images with aligned 3D models. Our key insightis that learning the inverse task of intrinsic decompositionis helpful for learning image synthesis from real images
and vice-versa. We therefore train both, the forward rendering process and the reverse intrinsic decomposition process, jointly using a single objective as illustrated in Fig. 1.
Inspired by recent results in unpaired image-to-image translation [17, 29, 46], we train our model using an small set ofsynthetic 3D models of an object category as well as a large
unpaired dataset of real images of the same category.
Towards this goal, we exploit a technique from real-timerendering called Deferred Rendering which splits the rendering process into two stages and thus improves efficiency.
In the first stage, the geometry of the scene along with its
textures and material properties are projected onto a 2D
pixel grid, resulting in a set of 2D intrinsic images which
capture the geometry and appearance of the object. This
step is efficient since it does not require physically accurate
path tracing but relies on simple rendering operations. In
the second “deferred” stage, lighting, shading and texturaldetails are added to form the final rendered image. Our goal
is to replace this second deferred stage of the rendering process with a neural network which we call Deferred Neural
Rendering (DNR) network. To ensure that the input information is represented in the output image, we decompose
it back into its intrinsics using a second Intrisic Image Decomposition (IID) network. However, we found that using
this cycle alone leads to overfitting, especially in the IID
network. To improve the IID network, we introduce a second Decomposition cycle in which we train the IID network
to decompose real images.
Overall, our model follows a similar dual cycle training
setup as proposed in [46] and [44]. However, an importantconceptual difference to these works is that our task is not a
one-to-one but a one-to-many mapping. Different realistic
images can be generated from the same set of intrinsic maps
as the intrinsics do not uniquely define the image. Likewise,a single image can be explained using different intrinsic decompositions due to projection from the higher dimensional
intrinsics into the RGB image space.
We therefore introduce a shared adversarial discriminator between the input and the reconstruction at the end ofeach cycle. Our model enables both highly photo-realistic
image synthesis and accurate intrinsic image decomposition. We summarize our main contributions as follows:• We propose the Intrinsic Autoencoder, a method tojointly train photo-realistic image synthesis and intrinsic image decomposition using cycle consistency
losses without using any paired data.

• We propose a shared discriminator network that enables better generalization and proves key for learning
both tasks without paired training data.
• We analyze the importance of various model components using quantitative metrics and human experiments. We also show that our method recovers accurate intrinsic maps from challenging real images.

2. Related Work
Differentiable Rendering. A standard way of synthesizing images from a given geometry and material is to use
rendering engines. Several works try to implement the rendering process in a differentiable manner, amenable to neural networks. The work of [5] used differentiable renderingwith deformable face models for face reconstruction. The
works of [31] and [22] proposed rasterization-based differentiable renderers but only support local illumination. In
order to support more realistic image formation, some otherworks [8, 12, 13, 27] propose to back-propagate though pathtracing. Differentiable rasterizers are relatively fast, but at
the same time highly restrictive as they do not support complex global illumination. While differentiable path tracers
produce more realistic images, they are usually quite slow,
thus restricting their usage to specific applications. Anotherdrawback of differentiable renderers is that they require a
detailed representation of the rendering input in terms of geometry, illumination, materials and viewpoint. In this work,
we bypass the specification of complex image formation bytraining a CNN to directly generate realistic images fromgiven geometry and material inputs.
Neural Image Synthesis. Generative models such as Generative Adversarial Networks [14] and Variational AutoEncoders (VAE) [24] are widely use to synthesize realistic images from a latent code. In contrast, our goal is toperform conditional image synthesis which allows morefine-grained control over the image generation process.
Some popular conditional image generation approaches arelabel-to-image translation [35, 34], image-to-image translation [19, 11, 9, 46, 29, 41] and text-to-image generation [38, 45, 16, 42]. Earlier works [19, 11, 9] on conditional image-to-image generation are mostly supervisedwith paired data from both domains. Several works [46, 29]
propose a way to use unpaired data from both domains forconditional image generation. Other advances in conditional image generation include innovations in network architectures and loss functions for generating high resolution
images [41] and generating multiple diverse images [10, 17,
43, 18]. In this work, we develop a model for photo-realistic
geometry-to-image translation using only unpaired training
data as supervision. Our work is closely related to [1] which
also considers geometry-to-image translation, but requires
paired training data. Our work belongs to the family of un-paired conditional image generation models with architecture and losses (e.g., shared discriminator) specialized forthe geometry-to-image translation. Our model outperforms
state-of-the-art unpaired image-to-image translation models[46, 17] by a large margin.
Intrinsic Image Decomposition is a long standing problem in computer vision. [3] poses the task as an optimization problem with a set of hand-crafted priors for shape,shading albedo etc. On the other hand supervised methods
like [40, 21, 39] use synthetic data to train the model followed by refinement on real images. However, synthetic
data might not capture all the real world statistics and models trained with synthetic data might now generalize well
to real images. Recently, several self-supervised intrinsic
decomposition methods have been proposed [28, 32, 30].
[30] uses single images during both training and inference
stages. [28, 32] make use of multiview images or video
sequences of the scene during training and infer on a single image . Our work falls in the realm of self-supervised
intrinsic image decomposition. We do not use any paired
synthetic data or multiview sequences to train out model.
Instead, we rely on jointly training models for neural rendering and image decomposition.

3. Method
Our Intrinsic Autoencoder model (Fig. 2) consists of two
generator networks R and H for Deferred Neural Rendering and Intrinsic Image Decomposition, respectively. The
Deferred Neural Rendering Network R : M → Iˆ takes
as input a set of intrinsic maps M = {A, N, F }. The object’s surface normal vectors in the view coordinate system
N ∈ RH×W ×3 provides the Deferred Neural Rendering
Network important information about the local shape of theobject which is necessary for creating shading and reflection in the output image. The albedo A ∈ RH×W ×3 is a
pixel-wise RGB value that describes the material or texture
color at every pixel, ignoring any lighting effects. Finally,
the environment reflections F ∈ RH×W ×3 are computed byprojecting a high dynamic range environment map onto the
3D model. Note that this simple projection operation does
not involve any complicated sampling or ray-tracing operations. As shown in our experiments, the Deferred Neural
Renderer can also be trained with a subset of those inputs
since it is able to compensate for the missing information.
The DNR network R : M → Iˆ transforms all the input intrinsics M into a realistic image Iˆ ∈ RH×W ×3 that corresponds to the input intrinsics. Similarly, the Intrinsic Image
Decomposition (IID) network H : I → M̂ performs the
opposite task by taking an input image I and predicting its
intrinsics M̂ ∈ RH×W ×9 .
Supervised training of R and H on real data is typically difficult due to the lack of real training image and intrinsics
pairs (Ir , Mr ). Instead, we use a combination of cycle-

consistency losses and adversarial losses that require nopaired training examples. This allows us to leverage a large
dataset of real images {Iri }ni=0 and an unpaired set of synthetically generated intrinsic maps {Msi }mi=0 . In the following, we detail our cycle consistency losses and the novel
shared adversarial losses.

3.1. Cycle Consistency
Rendering Cycle. The goal of the rendering cycle is totrain R in order to produce realistic images Iˆs = R(Ms )from synthetic intrinsic maps Ms . To train R without paired
data, we use the inverse transformation H which decomposes the predicted image Iˆs back into its intrinsic mapsM̂s = H(R(Ms )) as illustrated in Fig. 2. We encourage
consistency of the intrinsics using the rendering cycle consistency loss which is defined as the Smooth-L1 distance
between the input and reconstructed intrinsics
Lren (R, H, Ms ) = kH(R(Ms )) − Ms k1 .

(1)

Decomposition Cycle. Similarly, we train H to generate
intrinsic maps M̂r = H(Ir ) from real images Ir . To ensure
consistency with the input Ir , the output intrinsics M̂r are
passed to the deferred neural renderer R to reconstruct theimage Iˆr = R(H(Ir )). The decomposition cycle consistency loss is defined by:
Ldec (R, H, Ir ) = kR(H(Ir )) − Ir k1 .

(2)

The combined cycle consistency loss is then defined as:
Lcyc (R, H, Ir , Ms ) = Lren (R, H, Ms ) + Ldec (R, H, Ir )(3)
To ensure that the predicted normals N̂s = HN (Ir ) and thereconstructed real normals N̂r = HN (R(Ms )) are properly normalized, we exploit an additional normalization loss
Lnorm :
Lnorm (R, H, I) =| 1−kHN (Ir )k2 | + | 1−kHN (R(Ms ))k2 |3.2. Shared Adversarial Loss
While the cycle consistency loss ensures that the network
input can be reconstructed from its output, it doesn’t place
any importance on the realism of that output. Additionally,
the cycle consistency loss assumes a one-to-one deterministic mapping between the input and output. While this isa reasonable condition for some image-to-image translation
tasks [46], it is violated when translating between images
and their intrinsic properties. Decomposing an RGB image into its high-dimensional intrinsic properties is a oneto-many transformation since multiple decompositions canbe consistent at the same time with the same image, e.g.,a gray patch may correspond to a gray diffuse surface ora black glossy surface with specular highlight. Likewise,
the process of creating an image from an incomplete set

Figure 2: Intrinsic Autoencoder. Our model comprises two cycles: The first cycle (blue) auto-encodes a set of intrinsics
rendered from 3D CAD models using appearance as latent representation. The second cycle (red) auto-encodes real images
using image intrinsics as representation. Consistency is achieved through a combination of cycle losses and shared adversarial
losses. Networks sharing the same weights are illustrated with the same color (green/yellow).
of intrinsic properties involves making additional predictions about missing attributes like lighting conditions, optical aberrations, noise or higher-order light interactions. To
better capture this multi-modal relationship, we use an adversarial loss between the input and its reconstruction.
An adversarial discriminator D is a classification modeltrained to predict if a data sample is produced by a generative model or if it stems from the true data distribution.
To train our Intrinsic Autoencoder, we use two adversarial discriminators, DI for discriminating generated images
Iˆ{s,r} from real images Ir , and DM for discriminating generated intrinsic maps M̂{r,s} from synthetic intrinsic maps
Ms . The discriminators help our model to learn the distribution of real images and synthetic intrinsics by optimizing
the following adversarial [14] loss function
Ladv (R, H, DI , DM ) = LIadv (R, H, DI )+LM
adv (R, H, DM ) (4)where

LIadv (R, H, DI ) = log(DI (Ir )) + log(1 − DI (R(Ms ))+ log(1 − DI (R(H(Ir )))(5)

is our novel shared adversarial image loss which discriminates both between the real image Ir and the generated
synthetic image Iˆs = R(Ms ), as well as between the real
image Ir and the reconstructed real image Iˆr = R(H(Ir )).
Similarly, we define the shared adversarial intrinsic loss asLM
adv (R, H, DM ) = log(DM (Ms )) + log(1 − DM (H(Ir ))+ log(1 − DM (H(R(Ms ))))
(6)

Using the reconstructed inputs Iˆr and M̂s in addition to
the generated samples Iˆs and M̂r for training DI and DM
makes the discriminators more robust and prevents overfitting. This is especially important when a relatively small
number of 3D objects are used to create the synthetic intrinsic maps which can lead to a discriminator that recognizes
the model features rather that the image realism.

3.3. Implementation and Training
We train our Intrinsic Autoencoder networks R, H in
addition to the adversarial discriminators DM , DI fromscratch by optimizing the joint objective
min max Lcyc + Lnorm + Ladv
R,H DI ,DM(7)

Our framework is implemented in PyTorch [37] and trainedusing Adam [23] with a learning rate of 0.0002. The Deferred Neural Rendering Network is a coarse-to-fine generator introduced in [41] for the deferred neural rendering
network. The input to the network is of size 256 × 512
constructed by concatenating normals, albedo and reflections. The output of the network is an RGB image of size
256 × 512 × 3. We use three networks H = {HN , HA , HF }for estimating the surface normals N , Albedo A and environment reflections F , respectively, from an image I. Each
network has a ResNet architecture with 5 ResNet blocks.
Adversarial Discriminator Networks. Since the local
structure of the generated images is mostly controlled by
the input intrinsics, we want the image discriminator DI tomainly focus on the global realism of the output. To address this, we use a multi-scale PatchGAN [41] discriminator which comprises two fully-convolutional networks thatclassify the local image patches at two scales, full and half
resolution. The discriminator outputs a realism score foreach patch instead of a single prediction per image. Thishas been shown to produce more detailed images for similar conditional image generation tasks [19, 41, 46]. The intrinsics discriminator DM has the same architecture exceptthat the input is a 9-channel stack combining all three intrinsic maps. We found that using a single discriminator forthe combination of the intrinsic maps performs better thanseparate networks for each. This is likely due to the interdependence between the different intrinsic properties thatallows the discriminator to detect inconsistencies between
the generated intrinsic maps. We provide more architecture
and training details in the supplementary material.

4. Experiments
Synthetic Data Generation. To generate the synthetic
training data, we use dataset from [2] containing 28 3D car
models covering 6 car categories (SUV, sedan, hatchback,station wagon, mini-van and van). Apart from the geometry,
we do not need any physically-based materials or textures
for the models. Instead, we assign to each car part a simple material with only two properties, the color and a scalar
glossiness factor for computing reflection maps. We assign
each 3D car part a fixed material from a set of 18 fixed materials. Additionally, we randomly pick one of 15 materials
with different colors for the car body during the rendering
process. Next, a camera position is randomly chosen withina radius of 8 meters and a maximum height of 3 meters. We
use a fast OpenGL based rendering engine which operates
at around 3 frames per second including the model loading
time. It outputs the surface normals of the car model in thecamera coordinate space and the albedo channels indicating the material color at each pixel without any lighting or
shading. Finally, we produce the environmental reflections
by using a 360 degree environment map from [2]. Thesekind of reflections are very efficient to compute since they
only require the view vector and the surface normal and do
not rely on expensive path-tracing. We render 20,000 synthetic samples of normals, albedo and reflections.
Real training data. We obtain the real images from a finegrained car classification dataset presented in [25]. For convenience, we refer to this as the real car dataset. It contains
16,000 images of cars captured in various lighting conditions, resolutions and poses and with different camera sensors and lenses.

4.1. Baselines
Since our goal is to train with only unpaired data, wechoose to benchmark our method against two state-of-the-art unpaired image generation approaches, CycleGAN[46]and MUNIT[17]. However, since both methods wereoriginally designed for image-to-image translation rather
than deferred rendering, we setup two additional strong
baselines that highlight the importance of our contributionsin improving the quality of our results.
CycleGAN and MUNIT. CycleGAN[46] is a genericmethod for translating between two domains without
available paired data. MUNIT[17] aims at producing adiverse set of translations between different domains. We
modify the two methods slightly to use our stacked 9channel synthetic intrinsic maps as inputs.
Without shared discriminator. In this setup, we do not
use the shared adversarial discriminator discussed in 3.2.
Instead, we only use the discriminator DI between generated image Iˆs , real image Ir . Similarly, the discriminator
DM is used only between synthetic intrinsics Ms andgenerated intrinsics M̂r .
Only rendering cycle. Here, we train the model using only
the deferred rendering cycle discussed in (Sec. 3.1) and do
not use the decomposition cycle.

4.2. Deferred Neural Rendering
To evaluate our approach for deferred neural rendering,
we use the network R to produce images given synthetic
intrinsic maps (albedo, normals, reflections) and compare it
to other baselines, both qualitatively and quantitatively.
4.2.1

Qualitative results

Fig. 3 shows car images generated using our deferred neural renderer from the input synthetic intrinsic maps shown
above them. The car models in the evaluation set havebeen previously seen by the generator, but the unique combination of pose and paint color has not been seen during
training. Our approach is able to generate detailed photorealistic images of cars with consistent geometry and distinct parts. We emphasize that the deferred neural rendering
network is trained without any rendered or real geometryimage pairs. Instead, it is able to learn the appearance ofdifferent car parts from a large set of real car images.
In Fig. 4 we compare the results of our full model to various baselines. The results clearly show the improvements invisual quality achieved when using our full model. Specifically, MUNIT appears to be unable to preserve the geometry and albedo of the input in the generated image, CycleGAN images has significant artefacts on the windows,
body, etc. When training our model without the shared discriminator, the resulting images suffer from irregular reflection patterns and a noisy image. This is likely due to thestrong overfitting required by the network to reproduce theinput image exactly when using only an L1 loss. The modeltrained without the decomposition cycle is not able to preserve the input intrinsics in terms of albedo and reflection.

Figure 3: Images generated using our Deferred Neural Renderer. Inputs to the network are intrinsic maps consisting ofalbedo, normals and reflections, shown above the generated images.

Figure 4: Qualitative Comparison with baselines on Neural Rendering. Inputs to the network are intrinsic maps consistingof albedo, normals and reflections, shown above the generated images. Additional higher resolution results are provided in
the supplementary materials.
In figure 5, we show the effect of input intrinsic maps on
the quality of rendered images. When the model is trained
only with normals as intrinsic input, the geometry of the result is well rendered but the color of different parts poorly
defined. The model trained on both normals and albedo
demonstrates sharper image quality but the hallucinated reflections by the network lacks lack realistic details. Finally,using the environmental reflections helps the network produce consistent and realistic images with sharp details.

Figure 5: Images generated using models trained withablated inputs.
4.2.2 Quantitative results
We evaluate the quality of generated images using Fréchet
Inception Distance(FID) [15] and Kernel Inception Dis-tance(KID) [4]. Both metrics compute the distance between
the features of two sets of images, obtained from a pretrained CNN. Table 1 presents both the FID and KID between the images generated using various methods and thereal images. Our full model achieves the lowest FID and
KID values (47.6, 4.2) indicating that the rendered images
from our model are closest to the distribution of real images compared to MUNIT [17] and CycleGAN [46]. Further, when we ablate each of the intrinsic map inputs, both
FID and KID increase substantially. Notably, in the case ofablating albedo input, the highest increase in distances canbe observed (88.7, 5.4), implying its importance for photorealistic image generation. We conclude that albedo is themost important for our task followed by normals and reflections maps. In both cases where we ablate the decomposition cycle or rendering cycle, we observe a huge increase in
the distances signifying the importance of using both cycle
consistency losses during training. Finally, training with the
setup of separate discriminators as mentioned in 4.1 leads toan increase in the distances.

w/
w/o
Cycle
Shared Decom. w/o w/o w/o
GAN MUNIT Ours Discr Cyc.
A N F
FID 103.3
KID 10.2

99.0
13.5

47.6
4.2

59.2
4.8

99.6
11.8

88.7 60.2 56.7
5.4 4.9 5.9Table 1: FID and KID between real images and generated samples. All inputs are provided to the generator(Albedo, Normals and Reflections).
Cycle
w/o Shared w/o Decom
GAN MUNIT Ours
Discr.
Cyc.
Real Im 77.7% 75.6% 67.5%
Ours
80.0% 85.8%
–

68.9%
57.6%

71.0%
63.8%

Table 2: Human Subject Study. Comparisons to identify
realistic images in an A/B test using Amazon Mechanical
Turk. The numbers indicate the ratio of trials where theimage from real or our model was chosen as more realisticcompared to the image from the method on the header.
4.2.3 Human Experiments
We design two experiments to measure the visual realismof generated car images using the Amazon Mechanical Turk
platform to crowd source human evaluations. For each comparison, we presented 40 human subjects each with 50 image pairs to choose the more realistic looking image. The
results are presented in Table 2. The first row presents experiments where one image is picked from the real images
and the other is from one of the synthesis methods and presented in a random order. Images from our full model seemto be most confused with real car image since only 67.5%
of choices were correct while in 32.5% of the trials the subjects choose our images to be the real one.
In the second experiment subjects are presented with an image generated by our full model and a matching image generated by one other synthesis methods. The results in thesecond row of Table 2 show that subjects choose our results
to be more realistic over 80% of times when compare to CycleGAN and MUNIT. This clearly indicates a high level ofvisual quality of our generated images compared to thosegenerated from existing methods. On the other hand, images from our ablated models appear to be much closer toour full model visual quality.

4.3. Intrinsic Image Decomposition
4.3.1 Qualitative results
In fig. 6, we show that the intrinsic decomposition network is able to decompose real car images into their intrinsic maps. We would like to emphasize that the model does
not have access to ground-truth intrinsic maps for real images during the training phase. Also, these car models are
not present in the synthetic training data.

w/o Shared w/o Decom. Cycle
Discr.
cycle
GAN MUNIT Ours
Normal Err.
Albedo Err.
Reflection Err.

17.75◦
54.00
55.60

18.80◦
67.21
71.00

27.82◦ 29.15◦ 14.73◦
68.18 81.44 52.74
73.18 74.75 51.74

Table 3: Errors for the Intrinsic Decomposition Task.
Our method achieves the lowest error on all tasks.
Figure 7 compares the decompositions produced by ourmodel to those from other baselines. Both CycleGAN [46]and MUNIT [17] show significant artificats and inconsistencies when trained to decompose real images. The USI3D[30] fails to generalize to real models since it was trainedusing synthetic data from ShapeNet [7]. Our model without
decomposition cycle also recovers noisy albedo and normals due to overfit only to synthetic data. On the other
hand, training without the shared discriminator leads to severe artefacts. This is because the rendering network tries
to encode intrinsics information in the generated images in
the form of high frequency artefacts such that the decomposition network can easily recover them.
4.3.2 Quantitative results
To evaluate the intrinsic maps predicted by the intrinsic image decomposition network (H) we construct a synthetic
dataset containing rendered RGB images and their corresponding intrinsic maps rendered using a standard Physically Based Renderer (Blender [6]). To obtain the error
between predicted and ground truth normals, we computethe average cosine distance between them. The errors foralbedo and reflection are the average `1 distances between
the predicted and ground truth maps. Table 3 presents theerrors of various methods for predicting intrinsic maps. Ourfull model has the least error for all the modalities followedby our model without the shared discriminator, without decomposition cycle and finally MUNIT and CycleGAN. This
indicates that our model is able to learn accurate image decomposition while keeping generalization. Note that these
PBR-rendered images have not been presented to our network during training.

4.4. Results on ShapeNet Aeroplanes
We train our model for the object class ”Aeroplanes”.
We obtain the real images from FGVC-Aircraft dataset [33]which contains 10,000 images of aeroplanes. We use the
3D models of aeroplanes from the Shapenet dataset [7] toobtain our intrinsic maps. We follow the process mentioned
in sec.4 to generate input training data. We use the normals
and albedo as inputs to the network. Figure 8 illustrates realistic images generated using our deferred rendering network, demonstrating the ability of our method to handle
low-quality mesh and texture models.

Figure 6: Results of our intrinsic decomposition network on real images. The first column shows the inputs to the
network. Our model is able to decompose the sport car in first row accurately even though our synthetic training dataset
doesn’t include any sport cars at all. The car models of other inputs images are also not present in our synthetic dataset.

Figure 7: Comparison with Baselines for intrinsic decomposition. Note that USI3D [30] only produces albedo and shading
and not reflections or normals.

Figure 8: Images generated by our network trained on aeroplanes from ShapeNet [7].

5. Conclusion
In this paper, we presented a joint approach for training adeferred rendering network for generating realistic images
from synthetic image intrinsics and an intrinsic image decomposition network for decomposing real images of an object into its intrinsic properties. We trained the model using
unpaired 3D models and real images. Our qualitative andquantitative experiments revealed that using a combinationof shared adversarial losses and cycle consistency losses isable to produce images that are both realistic and consistent
with the control input.

6. Acknowledgement
This project has received funding from European ResearchCouncil (ERC) under the European Unions Horizon 2020
programme (grant No. 647769) and Heidelberg Collaboratory for Image Processing (HCI).

References[1] H. A. Alhaija, S. K. Mustikovela, A. Geiger, and C. Rother.
Geometric image synthesis. Asian Conference on Computer
Vision (ACCV), 2018. 2[2] H. A. Alhaija, S. K. Mustikovela, L. Mescheder, A. Geiger,
and C. Rother. Augmented reality meets computer vision:
Efficient data generation for urban driving scenes. International Journal of Computer Vision (IJCV), 2018. 5[3] J. T. Barron and J. Malik. Shape, illumination, and reflectance from shading. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(8):1670–1687, 2015. 3[4] M. Bikowski, D. J. Sutherland, M. Arbel, and A. Gretton.
Demystifying MMD GANs. In International Conference on
Learning Representations, 2018. 6[5] V. Blanz, T. Vetter, et al. A morphable model for the synthesis of 3d faces. In Siggraph, volume 99, pages 187–194,
1999. 2[6] Blender Online Community. Blender - a 3D modelling andrendering package. Blender Foundation, Blender Institute,
Amsterdam, 7[7] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan,
Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su,
J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rich
3D Model Repository. Technical Report arXiv:1512.03012[cs.GR], Stanford University — Princeton University —Toyota Technological Institute at Chicago, 2015. 7, 8
[8] C. Che, F. Luan, S. Zhao, K. Bala, and I. Gkioulekas. Inverse
transport networks. arXiv preprint arXiv:1809.10820, 2018.
2[9] Q. Chen and V. Koltun. Photographic image synthesis withcascaded refinement networks. In Proceedings of the IEEE
International Conference on Computer Vision, pages 1511–
1520, 2017. 1, 2
[10] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo.
Stargan: Unified generative adversarial networks for multidomain image-to-image translation. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 8789–8797, 2018. 2[11] A. Dosovitskiy, J. T. Springenberg, M. Tatarchenko, and
T. Brox. Learning to generate chairs, tables and cars withconvolutional networks. IEEE transactions on pattern analysis and machine intelligence, 39(4):692–705, 2017. 2[12] I. Gkioulekas, A. Levin, and T. Zickler. An evaluation ofcomputational imaging techniques for heterogeneous inverse
scattering. In European Conference on Computer Vision,
pages 685–701. Springer, 2016. 2[13] I. Gkioulekas, S. Zhao, K. Bala, T. Zickler, and A. Levin.
Inverse volume rendering with material dictionaries. ACM
Transactions on Graphics (TOG), 32(6):162, 2013. 2[14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information
processing systems, pages 2672–2680, 2014. 2, 4
[15] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, andS. Hochreiter. Gans trained by a two time-scale update
rule converge to a local nash equilibrium. In I. Guyon,

U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6626–6637. Curran
Associates, Inc., 2017. 6[16] S. Hong, D. Yang, J. Choi, and H. Lee. Inferring semantic
layout for hierarchical text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 7986–7994, 2018. 2[17] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz. Multimodal
unsupervised image-to-image translation. In Proceedingsof the European Conference on Computer Vision (ECCV),
pages 172–189, 2018. 2, 3, 5, 6, 7
[18] L. Hui, X. Li, J. Chen, H. He, and J. Yang. Unsupervised multi-domain image translation with domain-specificencoders/decoders. In 2018 24th International Conference
on Pattern Recognition (ICPR), pages 2044–2049. IEEE,
2018. 2[19] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-toimage translation with conditional adversarial networks. In
Proceedings of the IEEE conference on computer vision andpattern recognition, pages 1125–1134, 2017. 1, 2, 5[20] W. Jakob. Mitsuba renderer, 2010. http://www.mitsubarenderer.org. 1[21] M. Janner, J. Wu, T. Kulkarni, I. Yildirim, and J. B. Tenenbaum. Self-Supervised Intrinsic Image Decomposition. In
Advances In Neural Information Processing Systems, 2017.
3[22] H. Kato, Y. Ushiku, and T. Harada. Neural 3d mesh renderer.
In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pages 3907–3916, 2018. 2[23] D. P. Kingma and J. Ba. Adam: A method for stochastic
optimization. CoRR, abs/1412.6980, 2014. 4[24] D. P. Kingma and M. Welling. Auto-encoding variational
bayes. arXiv preprint arXiv:1312.6114, 2013. 2[25] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013. 5[26] C. Lassner, G. Pons-Moll, and P. V. Gehler. A generativemodel for people in clothing. In ICCV, 2017. 1[27] T.-M. Li, M. Aittala, F. Durand, and J. Lehtinen. Differentiable monte carlo ray tracing through edge sampling. In
SIGGRAPH Asia 2018 Technical Papers, page 222. ACM,
2018. 2[28] Z. Li and N. Snavely. Learning intrinsic image decomposition from watching the world. In Computer Vision and Pattern Recognition (CVPR), 2018. 3[29] M.-Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-toimage translation networks. In Advances in Neural Information Processing Systems, pages 700–708, 2017. 2[30] Y. Liu, S. You, Y. Li, and F. Lu. Unsupervised learning for intrinsic image decomposition from a single image. In CVPR,
2020. 3, 7, 8[31] M. M. Loper and M. J. Black. Opendr: An approximate differentiable renderer. In European Conference on Computer
Vision, pages 154–169. Springer, 2014. 2

[32] W.-C. Ma, H. Chu, B. Zhou, R. Urtasun, and A. Torralba.
Single image intrinsic decomposition without a single intrinsic image. In V. Ferrari, M. Hebert, C. Sminchisescu, and
Y. Weiss, editors, Computer Vision – ECCV 2018, 2018. 3[33] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi.
Fine-grained visual classification of aircraft. Technical report, 2013. 7[34] T. Miyato and M. Koyama. cgans with projection discriminator. arXiv preprint arXiv:1802.05637, 2018. 2[35] A. Odena, C. Olah, and J. Shlens. Conditional image synthesis with auxiliary classifier gans. In Proceedings of the 34th
International Conference on Machine Learning-Volume 70,
pages 2642–2651. JMLR. org, 2017. 2[36] T. Park, M. Liu, T. Wang, and J. Zhu. Semantic image synthesis with spatially-adaptive normalization. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR
2019, Long Beach, CA, USA, June 16-20, 2019, pages 2337–
2346. Computer Vision Foundation / IEEE, 2019. 1[37] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017. 4[38] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, andH. Lee. Generative adversarial text to image synthesis. arXiv
preprint arXiv:1605.05396, 2016. 2[39] S. Sengupta, J. Gu, K. Kim, G. Liu, D. W. Jacobs, and
J. Kautz. Neural inverse rendering of an indoor scene froma single image. In International Conference on Computer
Vision (ICCV), 2019. 3[40] S. Sengupta, A. Kanazawa, C. D. Castillo, and D. W. Jacobs.
Sfsnet: Learning shape, refectance and illuminance of faces
in the wild. In Computer Vision and Pattern Regognition(CVPR), 2018. 3[41] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and
B. Catanzaro. High-resolution image synthesis and semantic
manipulation with conditional gans. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 8798–8807, 2018. 1, 2, 4, 5[42] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang,and X. He. Attngan: Fine-grained text to image generation
with attentional generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 1316–1324, 2018. 2[43] Z. Yang, H. Liu, and D. Cai. On the diversity of realistic
image synthesis. arXiv preprint arXiv:1712.07329, 2017. 2[44] Z. Yi, H. Zhang, P. Tan, and M. Gong. Dualgan: Unsupervised dual learning for image-to-image translation. CoRR,
abs/1704.02510, 2017. 2[45] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and
D. N. Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE International Conference on Computer
Vision, pages 5907–5915, 2017. 2[46] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired imageto-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference
on Computer Vision, pages 2223–2232, 2017. 2, 3, 5, 6, 7