Supplementary Material for
Towards Unsupervised Learning of Generative Models
for 3D Controllable Image Synthesis†

Yiyi Liao1,2,∗ Katja Schwarz1,2,∗ Lars Mescheder1,2,3,
Andreas Geiger1,2
1
Max Planck Institute for Intelligent Systems, Tübingen 2 University of Tübingen 3 Amazon, Tübingen{firstname.lastname}@tue.mpg.de

Abstract
In this supplementary document, we first provide details on the network architectures and the geometric consistency
loss in Section 1. In Section 2, we present the implementation details for all baselines. Finally we show additional results
and investigate how well our model captures the underlying 3D distribution in Section 3. The supplementary video shows
synthesized animations by controlling the camera viewpoint and object poses.

1. Implementation Details
1.1. 3D Generator
Fig. 1 illustrates the network architecture of our 3D generator gθ3D . As introduced in Section 3.1 of the main paper,
O = {obg , o1 , . . . , oN } denotes the set of primitives that represent all objects in the scene. Each foreground object is
described by its pose parameters {si , Ri , ti } and a feature vector φi determining its appearance. Considering that pose andappearance are independent variables, we split the pose generation and feature generation using two MLPs as shown in Fig. 1.
The background feature φbg is a texture map attached to the inside of the background sphere. We generate φbg ∈F ×64×128
Rusing transposed convolutional layers. {sbg , Rbg , tbg } are not shown in the figure as they are constant andtherefore not generated by the network.

1.2. 2D Generator
Our 2D generator gθ2D is an encoder-decoder structure based on ResNet [1]. Fig. 2 describes the network architecture indetail. Note that gθ2D is shared across the foreground and background primitives.

1.3. Alpha Composition
We fuse all outputs (X0i , A0i , D0i ) of our 2D generator using alpha composition in ascending order of depth at each pixel.
For a single pixel, let {x01 , . . . , x0N +1 } denote the sorted RGB values of the N foreground objects and the background and let
0{α10 , . . . , αN+1 } denote the corresponding alpha values. We calculate the composed pixel value x̂ as follows:
Algorithm 1 Alpha Compositionx̂ = x01 α10
α = α10for 2 ≤ i ≤ N dox̂ = x̂ + x0i αi0 (1 − α)α = α + αi0 (1 − α)end for1

Figure 1: Network Architecture of 3D Generator. The first two branches generate the pose and features of the foregroundobjects. The last branch generates the texture map of the background object. FC (din , dout ) denotes fully connected layers,
ConvT (din , dout , k × k, stride) refers to transposed 2D convolutions, BatchNorm (d) refers to batch normalization [3] andLeakyRelu (negative slope) is the non-linear activation function.

LeakyReLU (0.2)
Conv (128,128,3x3)
AdaIN

Conv (5,128,3x3)

Conv (128,256,4x4,s=2)BatchNorm
LeakyReLU (0.2)

LeakyReLU (0.2)
Conv (128,128,3x3)
AdaIN

LeakyReLU (0.2)
Conv (256,256,3x3)
AdaIN
LeakyReLU (0.2)
Conv (256,256,3x3)
AdaIN

256x32x32
5x64x64

128x64x64

LeakyReLU (0.2)
Conv (128,128,3x3)
AdaIN

128x64x64

LeakyReLU (0.2)
Conv (256,256,3x3)
AdaIN

ConvT (256,128,4x4,s=2)BatchNorm
LeakyReLU (0.2)

LeakyReLU (0.2)
Conv (256,256,3x3)
AdaIN

LeakyReLU (0.2)
Conv (128,128,3x3)
AdaIN

128x64x64

256x32x32

128x64x64

Conv (128,5,3x3)

5x64x6432

256x32x32
128x64x64

Figure 2: Network Architecture of 2D Generator. Conv (din , dout , k × k, stride = 1) refers to 2D convolutions, ConvT(din , dout , k × k, stride) refers to transposed 2D convolutions, BatchNorm (d) refers to batch normalization [3], AdaIN
refers to adaptive instance normalizations [2] and LeakyRelu (negative slope) is the non-linear activation function.

Applying this algorithm to all pixels yields the composite image Î which is the final output of our image synthesis pipeline.
In practice, we unroll this iterative algorithm and backpropagate gradients through it in the backward pass.

1.4. Geometric Consistency Loss
In this section, we present the warping process of the geometric consistency loss in details. Our geometric consistency
loss is implemented similar to [6]. In contrast to [6] that predicts the depth of the full image, we estimate the depth for eachobject and apply the geometric consistency loss to each object individually.
More
we add random noise to the pose parameters [Ri ; ti ] of each f oreground primitive and observe a newh specifically,ipose R̂i ; t̂i . Next, we render this primitive in both poses and generate two pairs of RGB and depth images, (X0i , D0i ) and(X̂0i , D̂0i ). Given the intrinsic matrix K and the camera extrinsic matrix [R; t], we obtain the projection functions for both
(X0i , D0i ) and (X̂0i , D̂0i ) respectively:
d1 [u1 , v1 , 1]T = K [R; t] [Ri ; ti ] [x, y, z, 1]Thid2 [u2 , v2 , 1]T = K [R; t] R̂i ; t̂i [x, y, z, 1]T(1)where [u, v] denotes the coordinate of a pixel in the image space, i.e., d1 = D0i (u1 , v1 ), d2 = D̂0i (u2 , v2 ) and x1 = X(u1 , v1 ),x2 = X(u2 , v2 ). When [x, y, z]T marks the same 3D point in the primitive coordinate, its projected RGB values in both
images (x1 and x2 ) should be the same and the depth values (d1 and d2 ) should conform to multi-view geometry constraint(in case of no occlusion). To apply this constraint, we convert each pixel [u2 , v2 ; d2 ] to a 3D point and project it into the otherpose as:
h
i−1
−1
(2)
d2→1 [u2→1 , v2→1 , 1]T = K [R; t] [Ri ; ti ] R̂i ; t̂i[R; t] K−1 d2 [u2 , v2 , 1]Tthis yields a projected RGB image proj(X̂0i ) where proj(X̂0i )(u2→1 , v2→1 ) = x2→1 and a projected depth image proj(D̂0i )where proj(D̂0i )(u2→1 , v2→1 ) = d2→1 . Note that the pixel values of proj(X̂0i ) and proj(D̂0i ) are located on pixels{u2→1 , v2→1 }. We warp the projected images into image space {u1 , v1 } using bilinear interpolation:X̃0i = warp(proj(X̂0i ))(3)D̃0i = warp(proj(D̂0i ))We thus obtain our geometric consistency loss as (equation (6) in the main paper):
#
"N
X
0
0
kAi (Xi − X̃i )k1
Lgeo (θ) = Ep(z)i=1+ Ep(z)

"N
X

#
kA0i

(4)

(Di − D̃0i )k1

i=1

where A0i is the alpha map of the foreground objects, i.e. we compute the consistency loss only in foreground regions where
the appearance and depth information is valid. We apply the consistency loss in both directions 1 ↔ 2 in our implementation.

2. Baselines
We describe the implementation details of the baselines in this section, including state-of-the-art methods and variants ofour method.

2.1. Vanilla GAN
We compare with [5] which is a typical representative of a vanilla 2D GAN method. We use the official implementation1 .
As we use spectral normalization in the discriminator which is not considered in [5], we train the vanilla GAN method bothwith and w/o spectral normalization and report the best performance. Empirically we observe that spectral normalization
boosts the performance of [5].
1 https://github.com/LMescheder/GAN stability

2.2. Layout2Im
We use the official implementation of Layout2Im2 . As this baseline requires 2D bounding boxes as input, we generate
2D bounding boxes and corresponding class labels for both the Car and Indoor dataset to train the model. We choose all
hyperparameters as suggested in [7].

2.3. 2D baseline
The 2D baseline is a variant of our model in which the 3D generator is replaced by a 2D generator which generates 2D
primitives thus not requiring a differentiable rendering layer. We now describe the details of the 2D baseline by discussing
the differences to our full model.
2D
2D
3D Generator: The “3D Generator” of the 2D baseline generates a set of 2D primitives. Let O2D = {o2D
bg , o1 , . . . , oN }2D
2D
denote the set of 2D primitives. Each foreground primitive o2D
is described by a set of attributes o2D= (s2Diii , ti , φi )F ×D×D22D
2
2D
2D
denote a 2D feature map. We do notwhere si ∈ R denote scale, ti ∈ R denote translation and φi ∈ R
consider 2D in-plane rotation for this baseline.

Differentiable Rendering: Instead of “rendering”, we use bilinear sampling to place the feature maps φ2D
on Xi accordingi2D
to its pose parameters {s2D,t}.
Therefore
thegradients
arebackpropagated
to
both
the
feature
and
the
pose
parameters.
ii
We generate both a feature map Xi and an alpha map Ai , but do not predict depth information.
2D Generator: The 2D generator then generates (X0i , A0i , D0i ) from (Xi , Ai ), i.e. a photorealistic image X0i , a refined alphamap A0i and a depth map D0i . Another difference of the 2D baseline is in alpha composition. The alpha composition we usefor our method is not differentiable wrt. the depth, thus the object depths are adopted to provide a sorting order. As this is notpossible for the 2D baseline, we apply differentiable soft composition [4] to learn D0i :A0 exp(D0i /γ)Wi = P i 0
Ai exp(D0i /γ)XiÎ =Wi X0i(5)iHere, γ = 0.1 controls the sharpness of the composition function.

2.4. Ours w/o c
For this baseline, we train our model without the background images, i.e. the model is only supervised with composite
images I. While the generator still generates both composite and background images, the discriminator is not conditioned on
the class label c. Therefore, the adversarial loss for this baseline is formulated as follow:
Ladv (θ, ψ) = Ep(z) [f (dψ (gθ (z, c = 1)))] + EpD (I) [f (−dψ (I))](6)

where gθ (z, c = 1) denotes the composite image from the generator.

3. Additional Experimental Results
We show additional qualitative results of all methods on the synthetic datasets in Fig. 3 and Fig. 4. We also analyze the
distribution of our learned 3D parameters.

3.1. Background Supervision
As shown in Fig. 3, our method is able to disentangle the foreground objects without background supervision (ours w/oc) if the background appearance is simple. For the Indoor dataset Fig. 4 where the background appearance is more complex,
our method fails to disentangle the foreground objects without any supervision. The foreground primitives vanish and thebackground object generates the entire image in this case. In contrast, our method is also able to disentangle foregroundobjects in this challenging scenario when provided with unpaired background images for supervision.
2 https://github.com/zhaobozb/layout2im3.2. Depth Maps
We visualize the predicted depth maps in Fig. 3 and Fig. 4. While recovering depth in an unsupervised fashion from
unpaired images remains challenging, our results indicate that the overall geometric structure of the objects can be recovered.
In addition, our depth prediction reflects the object translations precisely as it is refined from the depth of the projected 3D
primitives.

3.3. 3D Distribution
We investigate if our 3D generator is able to capture the underlying 3D distribution of the real images given only 2D
supervision on the Car w/o BG dataset. We randomly draw 3k samples from both the real images and our generated images
for evaluation where each sample contains 1 to 3 cars.
Translation: We first analyze the distribution of the 3D translations t1 , . . . , tN of the N foreground objects. For our
ground truth data the distribution over translations is created as follows: The simulator sequentially draws the location of theforeground objects avoiding collisions among objects. Therefore, the translation distribution of of an object depends on its
index in the set of foreground objects. More specifically, let P (t) denote the discrete probability distribution of t that we
obtain from our simulator. We assume that each ti is drawn from P (t) conditioned on its index i
P (t) =NX

P (t|I = i)P (I = i)

(7)

i=1

where I is a random variable denoting the indices of the foreground primitives. Similarly, let Q(t) denote the distribution oft estimated by our 3D generator. For both P (t) and Q(t), we uniformly discretize t into K bins on each axis, resulting in
K × K × K possible values of t. Note that an alternative is to consider each ti as a random variable and compute the joint
probability P (t1 , . . . , tN ), however, this leads to very high-dimensional sample space and thus hard to evaluate.
We quantitatively evaluate the distance between P (t) and Q(t) based on Jensen–Shannon divergence
1
1
KL(P kM ) + KL(QkM )
2
 2 
X
P (t)
KL(P kQ) =P (t) log
Q(t)t
JS(P kQ) =(8)

where M = 12 (P + Q) and KL(P kQ) is the Kullback–Leibler divergence from Q to P .
Note that P (t) and Q(t) are not necessarily aligned due to the scale ambiguity and different camera parameters (we do notassume access to the camera parameters of our simulator for training). Specifically, there are several ambiguous parameters,
including the global scale of t, the rotation of t about the z-axis and the translation offset of t at the z-axis. Note thatmodifying these parameters does not change the underlying property of the distribution. Therefore, we apply grid search onthese parameters to align Q(t) to P (t). The same transformation is applied to all t1 , . . . , tN of all samples so that the spatial
arrangement remains the same for each image.
Fig. 5 shows P (t) and Q(t), as well as JS(P kQ). Note that P (t) is not a uniform distribution due to the collision checkin our simulator for rendering multiple objects. For Q(t) we consider all 3D representation variants, including point cloud,cuboid and sphere. Our point cloud representation yields the best JS score while all the representations achieve very similarscores. We observe that our 3D generator learns to place objects on a ground plane. Moreover, it also learns a non-uniform
distribution and avoids putting objects in the center which can reduce object collisions.
Rotation: We then compare the rotation distributions P (R) and Q(R). Here, we adopt the axis-angle representation whichparameterizes the rotation matrix R by an axis of rotation e and the rotation angle ω about the axis. In contrast to P (t) thatis not uniform due to the collision check, P (R) can be represented by a uniformly distributed ω ∼ U(0, 2π) with e denoting
the z-axis, as the cars independently rotate about the z-axis in our simulator:∆
P (R) =P (ω)(9)

It can be seen that the rotations in our real images have only one degree of freedom.
For comparing Q(R) to P (R), we first investigate if our estimated rotations follow this low degree of freedom or not. In
principle the generator can create primitives with various rotations that match the coarse object shape. For instance, rotating

a cube 90 degrees around any of its symmetry axes will yield an identical shape but with another rotation. Therefore, therotation axes e over all primitives are not aligned and we cannot compare them directly. However, we find that the network
learns a consistent orientation over the ordered primitives, i.e. the first primitives in all samples have a similar orientation,
the second and so forth. Thus, we compute the mean rotation axis ēi for each index i = 1, . . . , N over all samples. We
then evaluate the angles between the rotation axes ei and the mean axis ēi . Fig. 6 (top) illustrates the distribution over theseangles. As can be seen, the distribution peaks near 0 for all 3D representations (point cloud, cuboid and sphere), indicatingthat each primitive rotates about a stable rotation axis and mimics our ground truth distribution where the degree of freedom
is 1.
Next, we evaluate the distribution of the rotation angle around these relatively fixed rotation axes∆
Q(R|I = i) =Q(ωi )

Q(R) =N
X

Q(R|I = i)Q(I = i)

(10)

i=1

We uniformly discretize ω into K bins and compare Q(R) in Fig. 6 (bottom). We observe that none of our 3D representations
captures the uniform distribution P (R). A possible explanation is that our randomly sampled camera rotations compensate
for the rotation of the objects.

References[1] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2016. 1[2] X. Huang and S. J. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017. 2[3] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. of theInternational Conf. on Machine learning (ICML), 2015. 2[4] S. Liu, T. Li, W. Chen, and H. Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2019. 4[5] L. Mescheder, A. Geiger, and S. Nowozin. Which training methods for gans do actually converge? In Proc. of the International Conf.
on Machine learning (ICML), 2018. 3[6] A. Noguchi and T. Harada. RGBD-GAN: unsupervised 3d representation learning from natural image datasets via RGBD image
synthesis. arXiv.org, 1909.12573, 2019. 3[7] B. Zhao, L. Meng, W. Yin, and L. Sigal. Image generation from layout. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2019. 4, 7, 8

[7] (t)2D (t)
Ours w/o c (t)Ours (t)Ours (R)
Ours (θ) Ours (φ)

Figure 3: Car Dataset. We translate one object for all methods. Additionally, we rotate one object and manipulate camera
poses with our method which cannot be achieved with the baselines. For Ours w/o c, we additionally visualize the composite
alpha A0i . For Ours (t) and Ours (R), we additionally visualize the composite alpha A0i and the composite depth maps D0i .
The colorbar below each depth map illustrates the transition from zero to maximum depth. Ours (φ) refers to rotating thecamera by updating the azimuthal angle φ. Ours (θ) refers to rotating the camera by updating the polar angle θ.

[7] (t)2D (t)
Ours w/o c (t)Ours (t)Ours (R)
Ours (θ) Ours (φ)

Figure 4: Indoor Dataset. We translate one object for all methods. Additionally, we rotate one object and manipulate camera
poses with our method which cannot be achieved with the baselines. For Ours w/o c, we additionally visualize the composite
alpha A0i . Note here A0i = 0 as the foreground primitive vanishes. For Ours (t) and Ours (R), we additionally visualize
the composite alpha A0i and the composite depth maps D0i . The colorbar below each depth map illustrates the transition
from zero to maximum depth. Ours (φ) refers to rotating the camera by updating the azimuthal angle φ. Ours (θ) refers torotating the camera by updating the polar angle θ.

Ground truth

Point cloudCuboidSphereFigure 5: Distribution of 3D translations. From left to right, we show the underlying distribution of the 3D object translations of the real images P (t), the distribution Q(t) of our generated samples based on point cloud, cuboid and sphere respectively. The top row illustrates the 3D distribution and the bottom row shows the distribution in bird-eye view (marginalizedout z-axis). Darker color denotes a higher probability that the translation t falls into the corresponding 3D voxel/2D grid.

Point cloudCuboidSphereFigure 6: Distribution of 3D rotations. In the first row, we show the angle between the rotation axis ei and the mean rotation
axis ēi . Results show that most of the angles are close to 0, indicating that the rotation axis of each primitive is relatively
stable across all samples. In the second row, we show the distribution of the rotation angle ω about the rotation axis e. We
omit the ground truth distribution here as it is a uniform distribution.