RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis
Authors: Atsuhiro Noguchi
Tatsuya Harada
Venue:  (2020)
Keywords: image generation
3D vision
unsupervised representation learning

Abstract: Understanding three-dimensional (3D) geometries from two-dimensional (2D) images without any labeled information is promising for understanding the real world without incurring annotation cost. We herein propose a novel generative model, RGBD-GAN, which achieves unsupervised 3D representation learning from 2D images. The proposed method enables camera parameter--conditional image generation and depth image generation without any 3D annotations, such as camera poses or depth. We use an explicit 3D consistency loss for two RGBD images generated from different camera parameters, in addition to the ordinal GAN objective. The loss is simple yet effective for any type of image generator such as DCGAN and StyleGAN to be conditioned on camera parameters. Through experiments, we demonstrated that the proposed method could learn 3D representations from 2D images with various generator architectures.

Published as a conference paper at ICLR 2020
RGBD-GAN: UNSUPERVISED 3D REPRESENTATION LEARNING FROM NATURAL IMAGE DATASETS VIA
RGBD IMAGE SYNTHESIS
Atsuhiro Noguchi & Tatsuya Harada
'The University of Tokyo, 'RIKEN {noguchi, harada)@mi. t. uutookyo.cc..
ABSTRACT
Understanding three-dimensional (3D) geometries from two-dimensional (2D) images without any labeled information promising for understanding the real world without incurring annotation cost. We herein propose novel generative model, RGBD-GAN, which achieves unsupervised 3D representation learning from 2D images. The proposed method enables camera parameter- condittion image generation and depth image generation without any 3D annotations, such as camera poses or depth. We use an explicit 3D consistency loss for RGBD im- ages generated from different camera parameters, in addition to the ordinal GAN objective. The loss is simple yet effective for any type of image generator such as DCGAN and StyleGAN to be conditioned camera parameters. Through exper- iments, we demonstrated that the proposed method could learn 3D representations from 2D images with various generator architectures.
INTRODUCTION
Generated face images from PGGAN and car images from StyleGAN. Images in even rows are generated depth images with colormaps. Though the models are trained on unlabeled RGB image datasets, they achieve RGBD image generation as well as explicit control over the camera poses.
Figure 1:
Understanding three-dimensional (3D) geometries from two-dimensional (2D) images is important in computer vision. An image of real-world objects comprises independent components: object identity and camera pose. Object identity represents the shape and texture of an object, and camera pose comprises camera rotation, translation, and intrinsics such focal length. Learning the rep- resentation of these two components independently facilitates in understanding the real 3D world. For example, camera pose invariant feature extraction can facilitate object identification problems, and camera pose variant feature representations are beneficial for the pose estimation of the objects. These tasks are easy for humans but difficult for machines.
Recently, 3D representation learning through 3D object generation has been actively researched. Many techniques are available for learning the relationship between 2D images and 3D objects. Typ- ically used 3D representations Henzler et al., 2019), point clouds (Fan et al., 2017), and meshes (Rezende et al., 2016; Kato al., 2018; Wang al., 2018; Kato & Harada, 2019). For most of the methods, 3D annotations such
voxel grids (Yan et al., 2016; Wu et al., 2016; Choy et al., 2016;

---

Published as a conference paper at ICLR 2020
as ground truth 3D models (Choy et al., 2016; Fan et al., 2017: Wang et al., 2018), multiple-view images (Yan et al., 2016), or silhouette annotations of objects (Yan et al., 2016: Kato et al.. 2018; Kato & Harada, 2019) must be used to reconstruct 3D shape from 2D images. Although these meth- ods achieve 3D object generation by controlling the object identity and camera poses independently. the construction of such datasets requires considerable time and effort. Therefore, method that can learn 3D representations without any labeled information must be developed. Though some research tackles this problem, their performance is limited when applied to natural images. Rezende et al. (2016) proposed unsupervised single-view view 3D mesh reconstruction, but it can only be applied to a primitive dataset. Henzler et al. (2019) recently proposed unsupervised single-view voxel recon- struction on natural images, but the resolution is limited due to the memory constraint.
To realize unsupervised 3D object generation, we employ different approach, i.e., RGB-Depth (RGBD) image generation. RGBD images comprise the color and depth information of each pixel. The proposed RGBD image generation can be achieved through simple extension of recently developed image generation models. We propose RGBD Generative Adversarial Networks (RGBD- GAN), which learns to generate RGBD images from natural RGB image datasets without the need of any annotations, such camera pose and depth annotations, multiple viewpoints for the single ob- jects. The proposed model uses an explicit 3D consistency loss for the generated images; the model generates two RGBD images with different camera parameters and learns them be consistent with the 3D world. This training pipeline is simple yet effective for generating depth images without su- pervision and for disentangling a camera pose from the image content. Because the proposed model does not restrict the generator architecture, we can condition any type of image generator (c.g.. PG- GAN (Karras et al.. 2018), StyleGAN (Karras et al., 2019)) on camera parameters. Figure I shows the generation results from the proposed models. As such, our model can generate RGBD images from arbitrary viewpoints without any supervision, and therefore we regard this "unsupervised 3D representation learning" though single output cannot represent full 3D scene.
Our contributions are as follows.
We propose new image generation technique, i.e., RGBD image generation, which can be achieved from RGB images without any labeled information such as annotations of camera parameters, depth, or multiple viewpoints for single objects.
The proposed method can disentangle camera parameters from the image content without any supervision.
Our method can be used to condition any type of generator camera parameters because the proposed loss function does not restrict the generator architecture.
2 RELATED WORKS
Recently, image generation models have shown significant progress, especially generative adver- sarial networks (GANs) (Goodfellow et al., 2014). GAN trains discriminator that estimates the distribution distance between generated and real images; additionally it trains a generator that min- imizes the estimated distance. As such, the distribution of training images can be estimated pre- cisely without supervision. Recent interest in generative models pertain to their training stability (Arjovsky et al., 2017; Gulrajani et al., 2017; Miyato et al., 2018) and improvement in quality and diversity (Karras et al.. 2018; Brock et al., 2019; Karras et al., 2019). Furthermore, methods to learn 3D morphable generative models from 2D images have been proposed (Tran al., 2017: Shen al., 2018; Sitzmann et al., 2019; Nguyen-Phig Phuoc et al., 2019). Tran et al. (2017) and Shen et al. (2018) learned to generate images by controlling camera poses using camera pose annotations or images captured from multiple viewpoints. Although these methods can successfully control an object pose, the scalability limited owing to the annotation costs. Nguyen-Phuoe proposed a method to disentangle object identity and camera poses without any annotations. This method uses latent 3D features and learns to generate images from the feature projected from the 3D feature with rigid-body transformations. That is, this method uses strong inductive biases regarding the 3D world to learn the relationship between camera poses and images. These image generation models cannot output explicit 3D representations, thus limiting the comprehensibility of the out- put. Sitzmann et al. (2019) achieved RGB and depth image synthesis from 2D image datasets by unsupervisingly learning occlusion aware projection from 3D latent feature to 2D. The model, how-
al. (2019) recently
2

---

Published as a conference paper at ICLR 2020
requires multiple viewpoints for single object and camera pose annotations, thus limiting the scalability.
Similarly, Rajeswar et al. (2019) proposed depth image generator trained unlabeled RGB images. The model can control the pose of generated images without supervision. However, it only works on synthetic dataset, where the surface-normal of each pixel is easily estimated by the color and location.
3 METHOD
In this study, unsupervised 3D representation learning is achieved via RGBD image synthesis. In this section, first describe the motivation RGBD representation in Section 3.1 and provide the details of our method in Section 3.2.
3.1 MOTIVATION
goal of this research is to construct model that can generate images I conditioned on camera pa- rameters However, it is impossible to perfectly model the relationship between and / without any annotations. Therefore, we alleviate the problem by considering optical flow consistency. Although optical flow is typically used for two different frames in movie, used it for images captured with different camera parameters. Optical flow consistency is expressed as the pixel movement between two images.
[æ, y, c) = I(x + Az, y Ay, Ac) for Vz, V.C.
(1)
Here, I and y are pixel coordinates in the image. Considering a small Ac, this equation can be written the following partial differential equation.
01 dx 01 dy Fa de
dc =0 for Va, y.
(2)
dy de
de and dy Be can be calculated. This term can be helpful for conditioning the generator on are known, then 0I
01 and 0f be estimated using ordinary image generation models. Therefore, if du By
and dy remain unknown, consider a geometric constraint on a homogeneous coordinate. Let 2 be the depth, p = (x, y, 1) the homogeneous coordinate of the pixel, Pworld the world coordinate of the pixel, R the rotation matrix, t the translation vector, and K the camera intrinsics. The camera parameters are represented herein as { {K, R. f). Pworld constant to c. Then, we can calculate the position on an image and the depth from the world coordinate Puorld-
the camera parameters when optimizing the GAN objective. As de
Dp = KR Pworld + Kt.
This facilitates in calculating and dy by estimating the depth D. Hence, used the RGBD representation for camera parameter conditioning. For depth image D, an optical flow consistency as an RGB image exists, considering the camera parameter change. This facilitates in estimating the depth image D.
de
D(r y. c) = D(x + Az, + Ay. + Ac) + AD for Vz, y,
(4)
Here, AD be calculated from Equation 3.
Briefly, training a GAN with the constraints in Equation 1, 3, and 4 is beneficial for learning 8r which benefits camera parameter- conditioonnniii synthesis. Additionally, learning camera parameter- conditional image generation model facilitates in learning depth distributions with the constraint from Equation and 3. The details for each module are explained below.
3.2 PROPOSED PIPELINE
The proposed model comprises three components: an RGBD image generator conditioned cam- era parameters, RGB image discriminator for adversarial training, and self-supervised RGBD con- sistency loss. The overview of the pipeline is shown in Figure 2.

---

Published as a conference paper at ICLR 2020
Generated Depth
Camera parameters
LGAN
K1. R
RGB discriminator
z-N(0, Latent vector
RGBD generator
Generated RGB
L3D
K2, R2,
Figure 2: Proposed pipeline. We train the RGBD image generator with the self-supervised 3D consistency loss and adversarial loss for RGB channels. The model generates two RGBD images with different camera parameters and learns them to be consistent with the 3D world.
3.2.1 RGBD IMAGE GENERATOR
Considering the success in image generation, the generator of a GAN can estimate complicated distributions. Therefore, we used ordinary RGB image generators such as DCGAN (Radford al., 2016) or StyleGAN for RGBD synthesis. RGBD synthesis is achieved by adding channel to the final layer of the RGB generator. Moreover, as described in the experimental section, we can use image generation models through 3D latent representations such as HoloGAN (Nguyen-Phan Phuoc al., 2019) or Voxels (Sitzmann et al., 2019), which models the 3D world more naturally.
In the proposed pipeline, the generator is conditioned camera parameters and trained with gra- dient descent to minimize the self-supervised consistency loss and the adversarial loss described bellow. Because constraint exists for the generator architecture, any type of generator architec- ture can be used for RGBD image synthesis, thus resulting in the high applicability of our method.
3.2.2 SELF-SUPERVISED RGBD CONSISTENCY LOSS
In Section 3.1, we showed that the optical flow consistency for RGB and depth can facilitate in learn- ing camera darameter-conditions image generation. We approximated the constraint in Equation and by sampling two camera parameters с₁ and C2 and minimizing the difference of both sides of the equations for two generated images conditioned
those camera parameters, where с₁ and c + Ac = C2. In this study, the camera parameters are sampled from a predefined distribution p(c) according the dataset, similarly to HoloGAN. The detailed settings explained in Section 4.2 and appendix. We limit the maximum values of Ac to 30 to avoid large occlusion.
The objective function for Equation is similar to the loss used in monocular video depth estimation (Zhou et al., 2017). Using Equation 3, we can calculate the 3D position of each pixel when an RGBD) image is viewed from different viewpoints. Therefore, images captured from e₁ be rendered by sampling the pixel values from RGBD images captured from This operation is typically called "warp" operation and implemented with bilinear sampling. We applied this loss the generated RGBD images conditioned on C₁ and e2- The main difference between depth estimation (Zhou al., 2017) and the proposed method is that our method optimizes both the RGB and depth image generator, while depth estimation only optimizes the depth estimator.
Moreover, for the constraints of the depth map in Equation 4, we define consistency loss on the generated depth maps. This loss, which is similar to the left-right disparity consistency loss in (Godard et al., 2017), attempts to equate the depth map generated from c₁ to that generated from C2 in 3D space. The overall proposed 3D loss function can be written as Equation 5.
1
LAD E. Ee-p(s))
3HW [G warp(Graz(z.c))
pprojctiinn(p((;.a))) - (5)
HW

---

Here, GRGB(Z,0 c) and GD(2.e) denote the generated RGB and depth image from latent vector & and camera parameters с respectively, W and H denote the width and height of the images re- spectively, and C1-+2 is a relative tramsformation matrix from C₁ to C2. The "projection" operation Published as a conference paper at ICLR 2020
calculates the depth value viewed from different viewpoints from the input depth map using Equa- tion 3. For simplification, we omit the loss for the inverse transformation c2-1 in the equation. The detailed explanations of "warp" and "projection" are provided in the appendix.
This loss function causes inaccurate gradients for the occluded pixels during the transformation c1-2 because it does not consider those regions. Therefore, in this study, we used the technique proposed in (Gordon et al., 2019). This technique propagates gradients only to pixels where the projected depth is smaller than the depth of the other viewpoint image. This prevents inaccurate gradients in pixels that move behind other pixels during projection.
Finally, we add a depth constraint term to stabilize the training. The loss above can be easily min- imized when the generated depth is extremely small. Therefore, we set the minimum limit for the depth value as Dmin and add regularization for depth values smaller than D Diin-
Ldepth
Σ 2.
max (0, Dmnn - D(x,
(6)
HW
3.2.3 RGB IMAGE DISCRIMINATOR
To achieve the training of RGBD generator from unlabeled RGB images, apply adversarial loss only for the RGB channels of generated images. Although the loss can only improve the reality of the images, this loss beneficial for learning depth images and camera parameter conditioning through the optimization of the loss in Equation 5.
Based on the above, the final objective for the generator LG is follows.
LG LGAN + AaDCaD + Meepthhodeghh
Here, LGAN is adversarial loss function, and λAD and Adepth are hyperparameters.
4 EXPERIMENTS
4.1 MODEL ARCHITECTURES
The proposed method does not restrict the generator architecture: any type of image generators can be conditioned camera parameters. To demonstrate the effectiveness of method, tested three types of image generation models: PGGAN, StyleGAN, and DeepVoxels. The model architectures are shown in Figure 3. Because perspective information is difficult obtain from single image, in this experiment, the camera intrinsics K are fixed during training. We controlled only the azimuth O, (left-right rotation) and elevation 0g (up-down rotation) parameters based on the training setting of HoloGAN, In the following, we provide the details of each model architecture
PGGAN: PGGAN (Karras al., 2018) is state-of-the-art DCGAN. In this experiment, we ditioned the model on two camera parameters, azimuth and elevation, as follows: First, these values are input to cos and sin functions, respectively, and the outputs are concatenated to a single four- dimensional vector Cegelic. Subsequently, Cepelic is concatenated to the latent vector which is input the generator. This operation allows the generated images to change continuously for a 360 angle change. We start with a resolution of 32 x 32 and increase it progressively to 128 x 128.
StyleGAN (Karras al., 2019) is state-of-the- -rrt--rrtt GAN model that controls the out- put "style" of each convolutional layer by performing adaptive instance normalization (AdaIN) (Huang & Belongie, 2017) and acquires hierarchical latent representations. We used Ceyelic only control the style of features resolutions of 4 X 4 and 8 8, as it is known that styles at low- resolution layers control global features such as the pose and shape of an object. More concretely, concatenated Ceyetic and the output of the mapping network w, which with a multilayer perceptron. Please refer to Figure 3. The image resolution is the same as PGGAN.
StyleGAN:
then converted to w'
Deep Voxels: HoloGAN enables the disentanglement of camera parameters by using 3D latent fea- ture representations. This is more natural modeling of the 3D world than the two models above be- cause considers explicit transformations in 3D space. However, HoloGAN cannot consider depth
5

---

Published as a conference paper at ICLR 2020
Image festure
R,
projection
AdaiN
Randen PO
DeepVosels
16:30
Ergeiic
Aroaton
30 cone Deephooees
Cayelic
K08, 2it/
208, ei10
0g. Pz
fa.%
PGGAN-based
StyleGAN-based
DeepVoxels-
Figure 3: Generator architectures tested. PGGAN-based model (left), StyleGAN- -aased model (mid- dle), and Deep Beepeoxeesbbase model (right).
information as the projection unit of HoloGAN only calculates the weighted sum of the feature on depth dimension. Therefore, we used the model inspired by Deep Voxels (Sitzmann et al., 2019) to apply the proposed method. Deep Voxels is method that can learn the 3D latent voxel representa- tion objects using images from multiple viewpoints of a single object; additionally, it can generate novel-view images. This method uses the occlusion-award projection module that learns which vox- els are visible from the camera viewpoint along the depth axis. This is achieved via unsupervised learning. Therefore, depth image can be acquired from the model, which is suitable for combining with our method. In this experiment, we combined DeepVoxel and a voxel feature generator that generates features from random latent vector z, for the random image generation task. We used 3D convolution and AdaIN for the voxel feature generator, similarly to HoloGAN. Deep uses an explicit camera model acquire the feature visible in the camera frustum, whereas HoloGAN uses bgdid-boly transformations. Therefore, DeepVoxels enables more accurate reasoning about the 3D) world. We compare the three settings for the models using 3D feature representations. The first model uses the weighted sum the depth dimension instead of occlusion- modules, similarly to HoloGAN. The second model uses occlusion-award -warre projection modules but does not use the proposed 3D loss. The final model uses Voxelsxels and the proposed 3D loss. The methods are called "HoloGAN-like: "Deep Voxels," and "Deep Voxels + 3D loss" in the figures and tables. It is noteworthy that "HoloGAN-Hi is not the same model as the original HoloGAN because it is based on Deep Voxels' network structures.
projection
4.2 DATASETS
We trained model using FFHQ (Karras et al., 2019), from ShapeNet (Chang et al., 2015), car images (Krause et al., 2013), and the LSUN bedroom dataset (Yu et al., 2015). We used 128 128 images for the PGGAN and StyleGAN, and 64 64 images for models using 3D latent feature representations owing to memory constraints. We used 35" for the elevation angle range for all experiments, 120" for the azimuth range for the FFHQ and bedroom datasets, and 360 for the azimuth range for the Car and ShapeNet car datasets. For the ShapeNet car dataset and car image dataset, we used a new occlusion reasoning algorithm for Voxels based models to stabilize the training. The details are explained in the appendix.
4.3 RESULTS
Qualitative results The generative results from each model controlling the camera parameters on the FFHQ and ShapeNet car datasets are shown in Figures 4, 5, 10, and 11. In the figures, im- ages with colormaps show the generated depth images. The depth is normalized (subtracted by the minimum value and divided by the range) and visualized with colormaps. For all models using the proposed loss (top three in the figures), images can be generated by controlling the camera param- eters while preserving their identity. Moreover, the models can generate depth images that do not exist the training samples. To confirm the depth consistency, we show normal maps and rotated images for the generative results from each model, shown in Figure 6. The white regions of the

---

Published as a conference paper at ICLR 2020
-60"
Azimuth
+60*
-17.5'
Elevation
+17.5
PGGAN 3D loss
StyleGAN + 3D loss
DeepVoxels
DeepVoxels
HoloGAN-
Figure 4: Visualization of comparison for the generated images from each model FFHQ dataset. Images in each row are generated from the same latent vector 2 but different azimuth or elevation angles. The images with colormaps are the generated depth images.
1800
Azimuth
1180
-17.5"
Elavation
17.5
3D loss
SlyleGAN 3D loss
DeepVoxels 3D loss
DeepVoxels
HoloGAN-
Figure 5: Visualization of comparison for the generated images from each model on ShapeNet car images. Images in each row are generated from the same latent vector = but different azimuth or elevation angles. The images with colormaps are the generated depth images.
ShapeNet car dataset are omitted for the visualization of point clouds. As shown in the figure, the models can generate the convex shape of a face and the rectangular shape of a car without any anno- tations regarding the 3D world. In particular, although the PGGAN and StyleGAN use 2D CNN, consistent rotation and depth estimation are achieved, which is impossible with previous methods. This implies that the proposed method has good generalization performance on the generator archi- tecture. The Vexeloxels-brar
method with the proposed loss performs well on both FFHQ and the ShapeNet car dataset. They can acquire more consistent rotation and generate more consistent depth images than 2D CNN-based models. This is thanks to the explicit 3D space modeling, though does consume much memory and has high computational cost. In the experiments, although the out- put images for Deep Voxels-base models were half the size than that of StyleGAN based models, they required 2.5 times higher memory and 2.9 times longer computational times for one iteration.
7

---

Published as a conference paper at ICLR 2020
Generated Image'
From different vowpoints
Generated image
From different viewpoints
Noraa map
Normel map
PGGAN + 3D loss
StyleGAN
+ 3D loss
DeepVoxels 3D loss
DeepVoxels
Normal map and point cloud visualization for FFHQ and ShapeNet car datasets. Point clouds in occluded region are not visualized in the figure.
Figure 6:
Table 1:
Performance comparison of unconditional generation models and proposed camera parameter-conditiona models. We report FID, Vdepth, and Veotor (lower is better) for each model.
FFHQ
ShapeNet Car
METRICS
FID
V depth
color
FID
Veepth
Vcolor
PGGAN
28.5
16.7
PGGAN 3D Joss
30.3
0.00141
0,0142
14,5
0.00043
0,0444
StyleGAN
20.9
15.5
StyleGAN + 3D loss
24.2
0.00077
0,0092
13,5
0.00027
0.0469
HoloGAN-like
23.4
33.5
Voxels
19.4
0.00487
0.0153
28.6
0.00045
0.0398
DeepVoxels- 3D loss
21.1
0.00067
0.0072
31.2
0.00030
0.0283
For the ShapeNet car dataset in Figure 5, PGGAN- and StyleGAN-based methods can generate consistently rotated images. However, for the PGGAN, only 180" azimuth change is acquired. This is because the model cannot distinguish between the front and back of the car, it is difficult to achieve only with unsupervised learning. Meanwhile, StyleGAN- based methods can learn con- sistent azimuth and elevation angle changes. This is because the StyleGAN is stable owing to its hierarchical latent representation.
Here we will compare the three 3DD-Latent-feature-bne "HoloGAN-like" method works well on the FFHQ dataset but cannot acquire consistent 360 ro- tation on the ShapeNet car dataset. DeepVoxels-based methods, on the other hand, can control 360 object rotation on the dataset, realizing the depth map generation without any supervised in- formation. This result shows that the depth reasoning helps generate images considering the 3D geometry of the objects. Moreover, VeepVoxels- based method with the proposed loss can gener- ate more consistent images for the FFHQ dataset. For example, in "DeepVoxels"" the depth of the background is smaller than that of the face, and the background pixels hide foregrounds when the generated images are rotated as shown in Figure 6. However, this is not observed with the proposed loss, our method considers warped images from different viewpoints, which facilitates learning the 3D world accurately.
methods. In our training settings, the
Moreover, additional generative results the car image and bedroom datasets are depicted in Fig- ure 7. These datasets are more difficult to train than the other two datasets due to the imbalanced distribution of the camera poses and the diversity of the object layouts. Although the models can leam consistent rotation reasonably for both datasets, the models cannot generate consistent depth maps. The results show the difficulty of the unorganized datasets.
8

---

Published as a conference paper at ICLR 2020
PGGAN 3D loss
StyleGAN 3D loss
DeepVoxels 3D loss
DeepVoxels
RX
RR
StyleGAN + 3D loss
DespVoxsis 3D loss
Figure 7: Generated and bedroom images changing the azimuth angle range.
As result, the proposed method effectively helps various generators to learn both depth information and explicit controls on camera poses. These are achieved without the need for 3D latent representa- tions as required in HoloGAN. Moreover, the proposed method further improves the results for the models using 3D latent representations.
Quantitative evaluation on RGB images We compared the Fréchet inception distance (FID) (Heusel et al., 2017) between models with and without the proposed method for each generator architecture. FID is a typical evaluation metric for the quality and diversity of the generated RGB images. The results are shown in Table 1. The results show that the proposed camera parameter- conditional image generation models can generate images with comparable or even better FIDs than unconditional or RGB image generation models for all generator architecture types. Notably, this is an unfair comparison because the models without 3D loss were trained just to minimize the distri- bution distance between the appearance of the real and generated images, although the models with 3D loss learn camera parameter conditioning. The results show the robustness and effectiveness of method against the generator architectures.
Quantitative evaluation on RGBD images Evaluating the generated RGB and depth in the 3D space difficult as obtaining the ground truth color depth for the generated images is impossi- ble. A possible approach to evaluate RGBD images without ground truth images is by calculating the inception score (IS) (Salimans et al., 2016) or FID on the generated images. However, this is inappropriate, IS and FID are estimated in the feature space of pre-trained CNN, and they not consider the 3D world geometry. Therefore, the evaluation
the generated RGBD with the 3D space is unattainable. Instead, evaluated the color and depth consistency across the views to quantitatively compare the RGBD images generated by different methods. For the point clouds generated from the same latent vector but different camera parameters c, all points should be on a single surface in the 3D space. Therefore, by calculating the variation of the generated RGBD across the views, the 3D consistency can be quantitatively evaluated. We calculated the variance of the point clouds for generated images as Vdepth and Veotor in Table 1. The details are provided in the appendix. notably, these values cannot calculated for PGGAN and StyleGAN without 3D loss, and HoloGAN-like method.
The results show that Deep with 3D loss exhibited the best scores, due to the strong inductive bias of the 3D- latent-representatiin Compared to DeepVoxel, without 3D loss, DeepVoxels with 3D loss generated consistent color and depth, exhibiting the effectiveness of the proposed loss.
CONCLUSION
We herein proposed an RGBD image synthesis technique for camera parameter-conditionall conditional image generation. Although the proposed method does require any laheled dataset, it can explicitly con-

---

Published as a conference paper at ICLR 2020
trol the camera parameters of generated images and generate consistent depth images. The method does not limit the generator architecture, and can be used to condition any type of image generator on camera parameters. As the proposed method can learn the relationship between camera param- eters and images, future works will include extending the method for unsupervised camera pose estimation and unsupervised camera pose invariant feature extraction from images.
ACKNOWLEDGEMENT
This work was partially supported by JST CREST Grant Number JPMJCRI403, and partially sup- ported by JSPS KAKENHI Grant Number JP19H01115. We would like to thank Antonio Tejero de Pablos, Dexuan Zhang, Hiroaki Yamane, James Borg, Mikihiro Tanaka, Sho Maeoki, Shunya Wakasugi, Takayuki Hara, Takuhiro Kaneko, and Toshihiko Matsuura for helpful discussions.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks In ICML, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2019.
Angel Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 1et..0302220.... 2015.
Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In ECCV. Springer, 2016.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. point set generation network for 3d object recon- struction from single image. In CVPR, 2017.
Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estima- tion with left-right consistency. In CVPR, 2017.
Ian J Goodfellow, Jean Pouget-Abadis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In NIPS, 2014.
Ariel Gordon, Hanhan Li, Rico Jonschkowski, and Anelia Angelova. Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras. In ICCV, 2019.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In NIPS, 2017.
Philipp Henzler, Niloy J. Mitra, and Tobias Ritschel. Escaping plato's cave: 3d shape from adver- sarial rendering. In ICCV, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In NIPS, 2017.
Xun Huang and Serge J Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In ICCV, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In ICLR, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
Hiroharu Kato and Tatsuya Harada. Learning view priors for single-view 3d reconstruction. In CVPR, 2019.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In CVPR, 2018.
10

---

Published as a conference paper at ICLR 2020
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fne-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) 13), Sydney, Australia, 2013.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In ICML, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
Sai Rajeswar, Fahim Mannan, Florian Golemo, David Vazquez, Derek Nowrouzezahrai, and Aaron Courville. Pix2scene: Learning implicit 3d representations from images, 2019. URL https: /oppere//ppeeeeieww net wtt.foommmm---eeeeeees0
Danilo Jimenez Rezende, SM Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3d structure from images. In NIPS, 2016.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, In NIPS, 2016.
Yujun Shen, Ping Luo, Junjie Yan, Xiaogang Wang, and Xiaoou Tang. Faceid-gan gan: Learning symmetry three-player gan for
face synthesis. In CVPR, 2018.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In CVPR, 2019.
Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning gan for pose-invariant
face recognition. In CVPR, 2017.
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating mesh models from single rgb images. ECCV, 2018
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning proba- bilistic latent space object shapes via 3d
modeling. In NIPS, 2016.
Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. NIPS, 2016.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 500 03365, 2015.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In CVPR, 2017.
A "WARP" AND "PROJECTION"
Here, we explain the "warp" and "projection" operation in Equation 5.
Warp When we warp (GB 02) to GRan(Z, first we calculate the position of pixels Pey in GRGB(2,0 e1) when they viewed from C2.
Dr,Pes -K--KKK--D.... Kt1-12
Here, R1-2 and f1-22 are relative rotation and translation matrices respectively. We warp GRGB(2, (2, 02) according to the calculated positions, Pe.- The operation is implemented with bilinear sampling between the four neighboring pixel colors of warped coordinates, such that the operation is differentiable. Same operation performed to warp generated depth images.
11

---

Published as a conference paper at ICLR 2020
Camera
Softmax weighting
Ours
Figure 8: Comparison between the softmax weighting (left) and our occlusion reasoning algorithm (right). The values denote the voxel weight, and the orange regions are the visible voxels in the camera. For softmax weighting, the occlusion network needs to change the voxel weight, according to the camera location. Our method accumulates the weight along the camera ray and ignored the voxels where the accumulative values exceed one, thus the occlusion network does not need to change the weights according to the camera location.
Projection As the depth values in (Dz c2), C142) are sampled from (D(z,c2) which is viewed from cg2 to compare GD(z, e1) with warp(G 02), c1-2), we need to project the depth values of each pixel in GD(z, e1) to the viewpoint Cz. This operation is the same as in Equation 8, and Day is used for the projected depth values. We denote as "projection".
B
deepth AND V
voltor
These metrics evaluated the depth and color consistency across the views. First, the images were generated from the same z but different c. Second, the point clouds were plotted for each image in the real-world coordinate. Third, the coordinates were converted to polar coordinates from the origin, and the angle coordinates (azimuth and elevation) were quantized. Fourth, for each image, the points were aggregated following the quantized coordinates, which denote "cell". For each cell, the point with the smallest radial coordinates were sampled. This prevented inaccurate variance for cells with multiple surfaces. Fifth, the variance of the depth and color for each cell, across different c, was calculated. Sixth, the variance was averaged across cells and different 2. Randomly sampled 100 and 100 for each were employed to calculate the metrics.
For face images, the origin was set as -0.5) (behind the head), and used -23 to 23 for angular coordination. For ShapeNet car dataset, the origin was (0, O, 0), and used 180 to 180 for the azimuth angle and -90% to 90º for the elevation angle. The white region was ignored for the evaluation of the ShapeNet car images.
C CAMERA PARAMETER DISTRIBUTIONS
To randomly sample two similar camera parameters, Ci and c2, sampled ei from uniform distribution and sampled 02 from an area near to cı, within the angle range. We limited the maximum distance between C and 02 as 30, to avoid large occlusions. Since it is difficult to obtain camera intrinsics from single image, we fix K and the camera distance during training. used as,
2s
0
K
0
0
wherein is the size of images. We fixed the distance between the camera and origin coordinates
1.
D IMPLEMENTATION DETAILS FOR DEEPVOXELS
The Deep Vexelsxsssbbar based methods were implemented using the projection layer, occlusion module, and rendering module proposed in (Sitzmann et al., 2019). We implemented them with simpler
12

---

Published as a conference paper at ICLR 2020
Ours
Softmax weighting
Figure 9: Comparison of the occlusion reasoning algorithms. The proposed method can acquire more consistent rotation and generate consistent depth maps than the softmax weighting.
structures than those in the original implementation, to reduce the computational and memory pense. We used fewer 3D convolutional layers for the occlusion module and U-Net-like network with AdaIN for the rendering module. Moreover, for simplicity, lariser or style discriminator proposed in (Nguyen-Phuoc et al., 2019).
did not the Identity regu-
We employed different algorithm in occlusion reasoning enable consistent depth for the ShapeNet car and car image datasets. The occlusion reasoning used to get image features in Deep- Voxels is softmax weighting along the depth axis, which visualized on the left side in Figure 8. This algorithm needs the occlusion module to calculate the weight of the voxels according to the camera poses, which is difficult through unsupervised learning. Therefore, reduce the training ex- penses of the occlusion network, we employ an explicit reasoning algorithm. The network estimates the probability of each voxel be on the surface of the object, i.e., the opacity of each voxel. This is implemented using sigmoid activation function. Further, the weights are accumulated along the rays from the camera by adding-up the values. When the accumulated values exceed I, the later voxels are ignored by replacing the weight values with Os. By doing this, the occlusion module does not need to change voxel weight according to the camera poses. The algorithm overview is shown in the right side of Figure 8.
The image generation results on the ShapeNet car dataset using "Deep + 3D loss", with each algorithm, are depicted in Figure 9. The proposed occlusion reasoning model acquire more consistent 360 rotation, whereas the softmax weighting cannot. Moreover, the proposed algorithm can generate consistent depth maps compared the softmax weighting method. The results show the effectiveness of the proposed method for unsupervised learning.
E TRAINING DETAILS
We trained PGGAN- and StyleGAN- models for 250,000 iterations using batch-size of 32, and 3D-latent-fcatuu based models for 65,000 iterations with batch-size of 10. All models are trained with Adam optimizer with equalized learning rates 0.001 for the generators, 0.00001 for the map- ping networks, and 0.003 for the discriminators. In the experiments, we used a ResNet-based dis- criminator and non-saturating loss (Goodfellow et al., 2014) with gradient penalty (Mescheder et al., 2018). Training with a single NVIDIA P100 GPU required 30, 40, and 50 hours for Deep Voxels-, StyleGAN-, and PGGAN- based methods, respectively.
F ADDITIONAL RESULTS
13

---

Published as a conference paper at ICLR 2020
-60
Azimuth
+60"
-17.5'
Elevation
+17.5
PGGAN + +3D loss
StyleGAN + +3D loss
DeepVoxels 3D
DeepVoxels
HoloGAN-
Figure 10: Additional results on FFHQ dataset. Images in each row are generated from the same la- tent vector but different azimuth or elevation angles. The images with colormaps are the generated depth images.
14

---

Published as a conference paper at ICLR 2020
-180*
Azimuth
+180*
-17.5'
Elevation
+17.5'
PGGAN 3D
StyleGAN + 3D loss
DeepVoxels + 3D loss
DeepVoxels
HoloGAN- like
Figure 11: Additional results on ShapeNet car dataset. Images in each row are generated from the same latent vector but different azimuth elevation angles. The images with colormaps are the generated depth images.
15

---

Published as a conference paper at ICLR 2020
2
PGGAN 3D loss
PGGAN
Figure 12: Randomly generated RGB images the FFHQ dataset from PGGAN, with and without proposed loss.
16

---

Published as a conference paper at ICLR 2020
PGGAN + 3D loss
PGGAN
Figure 13: Randomly generated RGB images the ShapeNet dataset from PGGAN, with and without proposed loss.
17

---

Published as a conference paper at ICLR 2020
U
StyleGAN + 3D loss
StyleGAN
Figure 14: Randomly generated RGB images the FFHQ dataset from StyleGAN, with and with- out proposed loss.
18

---

Published as a conference paper at ICLR 2020
StyleGAN + 3D loss
StyleGAN
Figure 15: Randomly generated RGB images the ShapeNet car dataset from StyleGAN, with and without proposed loss.
19

---

Published as a conference paper at ICLR 2020
DeepVoxels + 3D loss
DeepVoxels
Randomly generated RGB images the FFHQ dataset from DeepVoxels, with and without proposed loss.
Figure 16:
20

---

Published as a conference paper at ICLR 2020
DeepVoxels + 3D loss
DeepVoxels
Figure 17: Randomly generated RGB images the ShapeNet dataset from Deep Voxels with and without proposed loss.
21


Reviews:
[{"review_id": "r1lqcDSiYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Review ICLR20, RGBD-GAN\n\nThis review is for the originally uploaded version of this article. Comments from other reviewers and revisions have deliberately not been taken into account. After publishing this review, this reviewer will participate in the forum discussion and help the authors improve the paper.\n\n## Overall\n\nThe article proposes a method of modifying image-generating networks to also produce depth maps in an unsupervised way by enforcing rotational consistency.\n\nI enjoyed reading this work and I'm recommending it to be accepted. However, first there are some (in my opinion straight-forward) changes that need to be made to this work before I can recommend its publication: \n\n- The common \"Related Works\" section is missing and some of the literature is taking place in the introduction. I find this unorganized and I'd recommend keeping the intro shorter and just moving the literature either behind the intro or to the end of the paper.\n- Most figures and especially your headline figure (1) suffer from not having the depth normalized and not having a scale to it. The fix for this is simple and two-fold: for each depth image, subtract the minimum value and divide by the range (to normalize it and increase contrast), then write in the caption or as a legend that white is closer to the camera and black is further back.\n- 3D vs. 2.5D - If the common geometric definition of \"3D\" was applied here, the article's title was correct. However, in computer vision and especially 3D vision, the term is commonly used to refer only to models that include full scene geometry, including the occluded backs of objects and the term 2.5D is used to describe assigning depth values to pixels in an RGB image (and therefore only covering the view-dependent front of the object), which I think is the case here. However, this is not a hill that I'll die on so if you insist on that terminology, I won't block acceptance.\n- When you first discuss HoloGAN, you mention one of its main downsides being scalability and then proceed to not only explain that but also use a HoloGAN-like architecture in one of your experiments. I'd either remove the scalability argument or justify not just that but also how that's not relevant to your experiments.\n- The following phrase occurs multiple times throughout: \"camera parameter conditional image generation\". I _think_ you're missing a dash between \"parameter\" and \"conditional\".\n\n\n## Specific comments and questions\n\n### Abstract\n\nAll good.\n\n### Intro\n\n- Fig.1 normalize image \n- The literature section in intro mentions \"For all methods, 3D annotations must be used...\" - that's not true. See [Rezende, 2016][1] and [Rajeswar, 2018][2]\n- I understand how some literature is required to position your method, but I think it's better to not have the entire literature section in the center of the introduction\n\n[1]: https://arxiv.org/abs/1607.00662\n[2]: https://openreview.net/forum?id=BJeem3C9F7\n\n### Method\n\n- 2.1 clear + nicely written\n- Figure 2 good, caption a bit too short - figure+caption should be able to stand on their own\n- Illustration of Figure 3 nice, except for unclear DeepVoxel part: what's the wavy orange flag stand for?\n\n### Experiments\n\n- You mention K is fixed, but where does the initial K come from? I assume it's just neglected (since it's not important for StyleGAN/PGGAN), but then this needs to be mentioned in the methods sections closer to the formulas dealing with K.\n- Figure 4 - the depth maps need to be normalized. All we see here is a grey mush, even worse in Fig. 7\n- For ShapeNet cars, the model seems to suffer from not having a reference for the top and bottom of the image - have you tried adding floor/sky?\n- Figure 6, the tire marker is a good idea but image still unclear - I recommend slightly less rotation or an intermediate step between generated image and e.g. front view\n- For quantitative results/FID: try using Hausdorff or Chamfer distance on the rendered scenes' pixels. We don't care about the goodness of the RGB generation but the depth.\n\n### Conclusion\n\nAll good, albeit a bit short.\n\n### Appendix\n\nI don't think I saw any references to the appendix in the main paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJxNx3Optr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY: Unsupervised/Self-supervised generative model for image synthesis using 3D depth and RGB consistency across camera views\n\nCLAIMS:\n- New technique for RGBD synthesis using loss in 3D space\n- Can disentangle camera parameters from content (I disagree slightly with \"disentangle\" since you are conditioning on camera parameters in the first place)\n- Different generator architectures can be used\n\nMETHOD:\nGenerate RGBD images of 2 different views, have an adversarial loss on the RGB image, have a content loss between RGB1 and warp(RGB2), have a depth loss between D1 and warp(D2)\nEquation 5:\n- Possibly either \"c_{1->2}\" needs to be replaced by \"c_{2->1}\", or \"G_{RGB}(z, c_1) - warp(G_{RGB}(z, c_2), c_{1->2})\" needs to be replaced by \"warp(G_{RGB}(z, c_1), c_{1->2}) - G_{RGB}(z, c_2)\" (or am I missing something?)\n- Not entirely sure why there is a different \"projection\" operation, since both \"warp\" and \"projection\" are calculated from Equation 3. I understand that \"warp\" is the combined Rt matrix that is estimated using the two views and Equation 3, assuming that the \"d\"s are correct. Not sure what \"projection\" does though, possibly explain it better?\n\nDECISION: Very clearly written paper, simple idea executed well\n\nThe paper is clearly written and well organized. It uses a simple idea, and performs sufficient number of experiments to explore the idea. It is not very novel, but the paper shows its applicability with multiple architectures as a bonus.\n\nThe figures showed results almost only from their method. It would be great to pick one generator architecture, and elucidate more on the differences between not using their 3D loss and using it. Good attempt though.\n\nADDITIONAL FEEDBACK:\n- Might not be \"representation learning\", instead it is learning a generative model.\n- \"3 EXPERIMETNS\" -> \"3 EXPERIMENTS\"\n- The appendix should have more details on the equations and the specific formulations of warp  and projection operations", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gPwuOTtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes a technique to learn RGBD image synthesis from RGB images. A distinctive feature proposed by the technique is the user-controllable camera rotation parameters, learned in an unsupervised manner. The technique can be used in conjunction with various models, such as PGGAN, StyleGAN, and DeepVoxels.\n\nThis paper provides an interesting approach that can be a useful building block for future investigations.\n\nThe main issue I see with the paper is the number of results provided. Only 2 different images are shown per combination of model and dataset, limiting the reader's ability to assess the technique's performance. Would it be possible to provide a large number of results in a supplementary material or appendix?\n\nIn my opinion, this may be due to a difference in writing style, but the paper, in general, is slightly hard to read.\n\nThe depth in figures 1, 4, 5, 7 and 9 would be easier to read if it was displayed as a colormap (with the corresponding color bar) instead of grayscale. Additionally, a reference sphere would be appreciated near the normal maps shown in fig. 6 to inform the reader of the coordinates system used.\n\nSec. 3.3 states that “the depth of the background is smaller than that of the face [for DeepVoxels]; however, this does not occur when the proposed loss is used”, however fig. 6 seems to show the contrary. Is it due to the depth discontinuity?\n\nMinor details\n- Sec. 3 “Experimetns”: typo.\n- Sec. 3.3, “[...] use the 2D CNN”: I would replace “the” by “a”.\n- Sec. 3.3, Third paragraph, the first sentence is hard to read.\n- Sec. B “the later voxels are ignore[d]”\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has initially received mixed reviews, with two reviewers being weakly positive and one being negative. Following the author's revision, however, the negative reviewer was satisfied with the changes, and one of the positive reviewers increased the score as well. 

In general, the reviewers agree that the paper contains a simple and well-executed idea for recovering geometry in unsupervised way with generative modeling from a collection of 2D images, even though the results are a bit underwhelming. The authors are encouraged to expand the related work section in the revision and to follow our suggestion of the reviewers.
Decision: Accept (Poster)