Title: PIX2SCENE: LEARNING IMPLICIT 3D REPRESENTA-TIONS FROM A SINGLE IMAGE

Abstract: We aim to model 3D properties of the scenes from single 2D images. Learning 3D scenes from 2D images is a long-standing problem in computer vision with applications in related fields such as simulation and robotics. We propose pix2scene, a deep generative-based approach that represents the 3D scene in a learnt latent variable decoded into a viewpoint-dependent representation that can be rendered. Our method learns the depth of the scene and leverages a local smoothness assumption to extract the orientation of visible scene points. We achieve this using an encoderdecoder adversarial learning mechanism and a novel differentiable renderer to train the 3D model in an end-to-end fashion, using only images. We showcase the generative ability of our model qualitatively on the ShapeNet dataset (Chang et al., 2015) . We also demonstrate that our model can predict the structure of scenes from various, previously unseen view points. Finally, we evaluate the effectiveness of the learned 3D scene representation in supporting 3D spatial reasoning.

1. INTRODUCTION:

Understanding the 3-dimensional (3D) world from its 2-dimensional (2D) projections is a fundamental problem in computer vision with a broad range of application in robotics, simulation and design. Given that the majority natural scene data is available exclusively in the form of 2D images, the ability to directly infer knowledge about 3D structure from these images would be of great utility in scene understanding.

Inferring the 3D structure from multiple images of a scene has been pursued extensively, such as in stereo or structure from motion tasks (Hartley & Zisserman, 2004) . Since most available natural image data informative about the real world comes with only a single view of a given scene, it is perhaps more important to explore the development of models which can infer the 3D structural properties from a single image. On the other hand, single image 3D recovery is an extremely challenging and heavily under constrained task. The system has to rely on prior knowledge and 2D visual cues such as textures, shadows or occlusions in order to provide hints to the 3D structure of the scene. Practically, building a machine learning model that learns to infer 3D structure from images requires either a strong inductive bias or supervision. While some have used the 3D ground truth as explicit supervision (Wu et al., 2016; 2015) , in most cases of interest, such supervision will not be available. Consequently, our long term goal is to infer the 3D structure of realistic scenes from single images. In this paper we take a step towards this direction via a method of unsupervised learning of the 3D structure, directly from a single 2D image of each scene. Our method based on the adversarial learning framework (Goodfellow et al., 2014) and exploits a uniquely suitable 3D representation (i.e., surfels (Pfister et al., 2000) ) and a differentiable renderer.

Most 3D reconstruction methods rely on representing 3D objects explicitly using either voxels (Rezende et al., 2016; Yan et al., 2016) or meshes (Kanazawa et al., 2018; Wang et al., 2018) . Explicit representations store all the rendering-relevant information from a given 3D space and are easily transferable, i.e., they can be loaded with any 3D modeling software and viewed from any angle. However, approaches using explicit representations typically scale very poorly (O(n 3 ) or require a sparse/discrete representation which can be challenging for deep learning methods. As a result, these representations have only been applied to the reconstruction of single objects. As an alternative we propose to learn an implicit 3D representation which produces only the 3D geometry which is directly relevant for a particular viewpoint. Our viewpoint-specific 3D geometry is captured Figure 1 : Implicit vs explicit representations. Explicit voxel and mesh representations are viewpoint-independent and constitutes the complete scene. Our implicit surfel-based representation is viewpoint-dependent and it adapts the resolution to the viewpoint. The full scene is contained in a high-dimensional latent variable and only when the scene is to be rendered, the latent variable is serialized to surfels for a specific view.

using camera facing surfels (Pfister et al., 2000) which are surface elements defined by its position, orientation and material properties. Given an image we can infer its implicit 3D representation and then recreate novel surfel representations of the underlying scene from unobserved viewpoints. In general, we note that in a 3D scene, only a small fraction of the entities are perceivable from the camera. As the camera moves, and the occluded regions become visible, our method then generates surfels for those newly unoccluded regions. Another advantage of this approach is that minimal number of primitives (surfels) are required to obtain a high-resolution image as the camera moves closer to a part of the scene. Moreover this representation fits well with image based convolutional architectures.

Our model, Pix2Scene, is a deep generative-based approach for modelling the 3D structure of a scene directly from images. This model is unsupervised in the sense that it does not require 3D groundtruth or any other kind of image annotations. We base our model on Adversarially Learned Inference (ALI) approach (Dumoulin et al., 2016) . ALI extends the GAN (Goodfellow et al., 2014) framework by learning to infer the latent representation of a given image. In pix2scene the learned latent space embeds the 3D information of the underlying scene. The latent representation is mapped via a decoder network to a view-dependent 3D surface and then projected to image space by a differentiable renderer. The resulting image is then evaluated by an adversarial critic.

While our long-term goal is to be able to infer the 3D structure of a real-world photograph, in this paper we experiment exclusively with synthetically-constructed scenes and adopt several simplifying assumptions. In particular, we assume that the world is piece-wise smooth and that for each input image the illumination, view and object materials are known.

This work has the following main contributions, (1) we propose a novel unsupervised method for 3D understanding from a single image;

(2) we propose a new implicit 3D representation based on view-space surfels;

(3) we propose a surfel-based differentiable 3D renderer that can be used as a layer of a neural network; and (4) we propose 3D-IQTT a new 3D understanding evaluation benchmark. This task evaluates the model's ability to perform mental rotation by obtaining comprehensive understanding of underlying 3D structure. We also estimate the camera pose as part of the learnt latent variable for this particular task.

2. RELATED WORK:

The generation and reconstruction of 3D objects from images has been studied extensively in the computer vision and graphics communities (Saxena et al., 2009; Chaudhuri et al., 2011; Kalogerakis et al., 2012; Chang et al., 2015; Rezende et al., 2016; Soltani et al., 2017) . Our work bears some conceptual similarities to Kulkarni et al. (2015) which casts the 3D reconstruction problem as a more traditional inverse graphics task. By using Variational Auto-Encoder(VAEs) (Kingma & Welling, 2014) , they learn a representation of objects that disentangles factors of variations from images (i.e., object pose and configuration) and use the approach for specific transformations such as out of axis rotation. However, unlike their approach, ours is fully unsupervised and we implicitly generate 3D structure of scenes from single images. Our mechanism learns a latent representation for the underlying scene, which can later be used to render from different views and lighting conditions. Similar to ours, Rezende et al. (2016) infer the 3D configuration at their output. They adopt a probabilistic inference framework to build a generative model for 3D by combining a standard projection mechanism with gradient estimation methods. In particular, their approach requires multiple runs with mechanisms such as REINFORCE (Williams, 1992) in order to infer gradient from the projection layer. In addition, their use of mesh and voxel representations could become an obstacle to scaling their method to more complex scenes. Our approach is not susceptible to restrictions imposed by meshes or other scaling issues and has the potential to adapt to arbitrary scene configurations.

3. METHOD:



3.1. IMPLICIT 3D REPRESENTATION AND SURFELS:

Explicitly representing 3D structure presents different challenges for generative models (Kobbelt & Botsch, 2004) . Representing entire objects using voxels scales poorly given its (O(n 3 ) complexity.

The vast majority of the generated voxels aren't relevant to most viewpoints, such as the voxels that are entirely inside of objects. A common workaround is to use a sparse representation such as meshes. However, these too come with their own drawbacks, such as the need to discretise complex objects. This makes mesh representation difficult to generate using neural networks. Current mesh based methods mainly rely on deforming a pre-existing mesh.

On the other hand, our implicit approach represents the 3D scene in a high-dimensional latent variable. In our framework, this latent variable (or vector) is decoded using a generator network into a viewpoint-dependent representation of surface elements -similar to the surfels (Pfister et al., 2000) -that constitute the visible part of the scene. This representation is very compact: given a renderer's point of view, we can represent only the part of the 3D surface needed by the renderer. Also, as the camera moves closer to a part of the scene, our generator will allocate more surfels to represent that part of the scene and thereby increasing the resolution. Figure 1 illustrates these different representations. For descriptive purpose, surfels are shown as squares, but in general they do not have any shape.

Formally, surfels are represented as a tuple (P, N, ρ), where P = (p x , p y , p z ) is its 3D position, N = (n x , n y , n z ) is the surface normal vector, and ρ = (k r , k g , k b ) is the reflectance of the surface material. Since we are interested in modelling structural properties of the scenes i.e., geometry and depth, we assume that objects in the scene have a uniform material. We represent the surfels in the camera coordinate system. This significantly reduces the number of surfels by considering only the ones that will get projected onto a pixel in the rendered image. Moreover, this allows to reduce the position parameters to only p z being this the distance along a ray going through the surfel to the center of its pixel.

3.2. DIFFERENTIABLE 3D RENDERER:

As the critic operates only on image space, we need to project the generated 3D representations back to the 2D space using a renderer. In our setting, each stage of the rendering pipeline must be differentiable to allow us to take advantage of gradient-based optimization and backpropogate the critic's error signal to the surfel representation. The rendering process can be partitioned into two stages. During forward-propagation, the first stage finds the mapping between the surfels and the pixels; and the second stage computes the color of each pixel. During back-propagation, the first stage directs the gradients only to the surfels that get projected onto the image; and the second stage is differentiable as long as the shading operations are differentiable.

The first stage of the rendering involves finding the mapping between the surfels and the pixels. This requires performing the expensive operation of ray object intersection (See Figure 2a ). Our model requires a fast rendering engine as it will be use in every learning iteration. Conventional ray tracing algorithms are optimized for generating multiple views from the same scene. However in our setting during learning we render only one image from each scene. Moreover ray tracing algorithms require from representing the full scene, which is very inefficient as we only represent the part visible by the camera. To resolve these issues, our generator proposes one surfel for each pixel in the camera's The color of a surfel depends on the material reflectance, its position and orientation, and the ambient and point light source colors. See Figure 2b . Given a surface point P i , the color of its corresponding pixel I rc is given by the shading equation:

EQUATION

where ρ i is the surface reflectance, L a is the ambient light's color, L j is the j th positional light source's color, with d ij = L pos j -P i , or the direction vector from the scene point to the point light source, and k l , k q being the linear and quadratic attenuation terms respectively. Equation 1 is an approximation of rendering equation proposed in Kajiya (1986) .

3.3. PIX2SCENE MODEL:

The adversarial training paradigm allows the generator network to capture the underlying target distribution by competing with an adversarial critic network. Pix2scene employs bi-directional adversarial training to model the distribution of surfels from just 2D images.

3.3.1. BI-DIRECTIONAL ADVERSARIAL TRAINING:

ALI (Dumoulin et al., 2016) or Bi-GAN (Donahue et al., 2016) extends the GAN (Goodfellow et al., 2014) framework by including the learning of an inference mechanism. Specifically, in addition to the decoder network G x , ALI provides an encoder G z which maps data points x to latent representations z. In these bi-directional models, the critic, D, discriminates in both the data space (x versus G x (z)), and latent space (z versus G z (x)), maximizing the adversarial value function over two joint distributions. The final min-max objective can be written as:

min G max D L ALI (G, D) := E q(x) [log(D(x, G z (x)))] + E p(z) [log(1 -D(G x (z), z))], (2)

where q(x) and p(z) denote encoder and decoder marginal distributions.

3.3.2. MODELLING DEPTH AND CONSTRAINED NORMAL ESTIMATION:

Based on the ALI formulation, as depicted in Figure 3 , our model has an encoder network which captures the distribution over the latent space given an image data point x. The decoder network maps a fixed latent distribution p(z) (a standard normal distribution in our case) to the 3D surfel representation. Next, the surfel representation are rendered into a 2D image using our differentiable renderer. The resulting image is then given as input to the critic to distinguish from the real image data. Note that the input to the critic comes from the joint space of data with its corresponding latent code, as in ALI. A straightforward way to model the decoder network could be to learn a conditional distribution to produce the surfel's depth (p z ) and normal (N ). But this could lead to inconsistencies between the local shape and the surface normal. For instance, the decoder can fake an RGB image of a 3D shape simply by changing the normals while keeping the depth fixed. To avoid this issue, we exploit the fact that real-world surfaces are locally planar, and surfaces visible to the camera have normals constrained to be in the half-space of visible normal directions from the camera's view point. Considering the camera to be looking along the -z direction, the estimated normal has the constraint n z > 0. Therefore, the local surface normal is estimated by solving the following problem for every surfel,

EQUATION

where the spatial gradient ∇P is computed using 8 neighbour points, and P is the position of the surfels in the camera coordinate system obtained by backprojecting the generated depth along rays.

This approach enforces consistency between the predicted depth field and the computed normals. If the depth is incorrect, the normal-estimator outputs an incorrect set of normals, and result in an RGB image inconsistent with the data distribution, which would in-turn get penalized by the critic. The decoder and the encoder networks are thus incentivized to predict realistic depths.

3.3.3. MODEL TRAINING:

The Wasserstein-GAN (Arjovsky et al., 2017) formalism provides stable training dynamics using the 1-Wasserstien distance between the distributions. We adopt the gradient penalty setup as proposed in Gulrajani et al. (2017) for more robust training, however, we modify the formulation to take into account the bidirectional training.

Architectures of our networks, and training hyper parameters are explained in detail in appendix A. Breifly, we used Conditional Normalization (Dumoulin et al., 2016; Perez et al., 2017) for conditioning the view point (or camera pose) in the encoder, decoder and the discriminator networks. The view point is a three dimensional vector representing positional co-ordinates of the camera. In our training, the affine parameters of the Batch-Normalization layers (Ioffe & Szegedy, 2015) are replaced by learned representations based on the view point. The final objective includes a bi-directional reconstruction loss as formulated in Equation 4. This in-turn enforces the reconstructions from the model to stay close to the corresponding inputs. We also use a reconstruction error in the objective function for the encoder and decoder networks as it has been empirically shown to improve reconstructions in ALI-type models Li et al. (2017) .

EQUATION

where function rend(•) denotes rendered image on the decoder side. Evaluation. In order to evaluate the 3D reconstruction ability of the model we used Hausdorff distance (Taha & Hanbury, 2015) and MSE. Hausdorff distance measures the model's 3D reconstruction's correspondence with the input for a given camera pose. We measure the correctness of the recovered depth using standard MSE with respect to ground truth depth maps. We evaluate the 3D generation generation qualitatively. Finally the evaluation metric for the 3D-IQTT is the percentage of correctly answered questions.

Datasets. We have created multiple different scene datasets ranging from simple to complex in nature. Those scenes are composed by a room containing one or more objects placed at random positions and orientations. Each 3D scene is rendered into a single 128 × 128 × 3 image taken from a camera in a random sampled uniformly on the positive octant of a sphere containing the room.

Technically, the probability of seeing the same configuration of a scene from two different views is near zero. Box scenes is created with simple box 3D shape (as depicted in Figure 14 ). Shape scenes is created with basic 3D shapes (i.e., box, sphere, cone, torus, teapot etc). ShapeNet scenes is composed by 6 objects from the ShapeNet dataset (Chang et al., 2015) (i.e., bowls, bottles, mugs, cans, caps and bags) . For 3D-IQTT task we generated a test where each IQ question instance consists of a reference image of Tetris-like shape, as well as 3 other images, one of which is a randomly rotated version of the reference (see Figure 10 for an example). The training set is formed by 200k questions where only a few are labelled with the information about the correct answer (i.e., either 5% (10k) or 0.5% (1k) of the total training data). The validation and test sets each contain 100K labelled questions. More details on experimental setup and evaluation can be found in appendix B.

4.2. IMPLICIT 3D SCENE RECONSTRUCTION:

Figure 4 shows the input Shape scenes data and its corresponding reconstructions, along with its recovered depth and normal maps. The depth map is encoded in such a way that the darkest points are closer to the camera. The normal map colors correspond to the cardinal directions (red/green/blue for x/y/z axis respectively). Table 1 shows a quantitative evaluation of the forward and reverse Hausdorff distances on three different datasets. The table also depicts mean squared error of the generated depth map with respect to the input depth map. Figure 6 shows the reconstructions from the model on more challenging multiple-shape scenes where the number of objects as well as their shape varies. Figure 16 in the appendix showcases more qualitative evaluations.

To showcase that our model can reconstruct unobserved views, we infer the latent code z of an image x and then we decode and render different views while rotating the camera around the scene. Table 2 shows the Hausdorff distance and MSE of reconstructing a scene from different unobserved view angles. As the view angle increases from 0 • (original) to 80 • for shape scenes the reconstruction Error and MSE tend to increase. However, for the multiple-shape scenes setup the trend is not as clear because of the complexity of the scene an the inter-object occlusions. Figure 5 qualitatively shows how pix2scene correctly infers the extents of the scene not in view in a consistent manner, demonstrating true 3D understanding of the scene. 

4.3. IMPLICIT 3D SCENE GENERATION:

We trained pix2scene on scenes composed of ShapeNet objects. Figure 7 shows qualitative results on unconditional generation. This shows how our model is able to generate correct 3D interpretations of the world. We also trained our model conditionally by giving the class label of the ShapeNet object to the decoder and critic networks (Mirza & Osindero, 2014) . Figure 8 shows the results of conditioning the generator on different target classes.

In order to explore the manifold of the learned representations we select two images x 1 and x 2 from the held out data, then we linearly interpolate between their encodings z 1 and z 2 and decode the intermediary points into their corresponding images. Figure 9 shows this for two different settings. In each case, our representations capture the major geometrical aspects of the scene.

4.4. 3D-IQ TEST TASK:

We have designed a quantitative evaluation for 3D reconstruction which we refer to as the 3D-IQ test task (3D-IQTT). In their landmark work, Shepard & Metzler (1971) introduced the mental rotation task into the toolkit of cognitive assessment. The authors presented human subjects with reference images and answer images and the subjects had to quickly decide if the answer was either a 3D-rotated version or a mirrored version of the reference. The speed and accuracy with which people can solve this mental rotation task has since become a staple of IQ tests like the Woodcock-Johnson test (Woodcock et al., 2001) . We took these as inspiration when designing a quantitative evaluation: we are using the same kind of 3D objects but instead of confronting our model with pairs of images and only two possible answers, we include several distractor answers and the subject (human or computer) has to to pick the correct answers out of 3 that is a 3D-rotated version of the reference object. To verify that our approach is able to learn accurate embeddings of these shapes we first assessed the reconstruction of these shapes qualitatively as shown in Figure 11 .

4.4.1. SEMI-SUPERVISED CLASSIFICATION ON THE 3D-IQ TEST TASK:

For training pix2scene in a semi-supervised setting, in addition to the unlabeled data, we also used the labelled data. The training with the unlabeled samples differs from the approach described for previous experiments as we do not assume that we have the knowledge of camera position. Thus, part of the latent vector z encodes the actual 3D object (denoted as z scene ) and the remainder estimates the camera-pose (denoted as z view ). For the supervised training two additional loss terms were added: (a) a loss that enforces the object component (z scene ) to be the same for both the reference object and the correct answer, (b) a loss that maximizes the distance between object component of reference and the distractors. Losses (a) and (b) are contained in Equation 5where d i denotes the distractors, x ref is the reference and x ans the correct answer. Algorithm is detailed in Table 1 .

L θ (x ref , x d1 , x d2 , x ans ) = 1 2 D θ (x ref , x ans ) - 1 2 2 i=1 D θ (x ref , x di )

where

EQUATION

During the training we also minimize the mutual information between z scene and z view to explicitly disentangle and make sure that the learnt latent code has distinct source of information present in its dimensions. This is implemented via MINE (Belghazi et al., 2018) . The strategy of MINE is to parameterize a variational formulation of the mutual information in terms of a neural network:

EQUATION

)

This objective is optimized in an adversarial paradigm where T , the statistics network, plays the role of the critic and is fed with samples from the joint and marginal distribution. We added this loss to our pix2scene objective to minimize the mutual information estimate in both unsupervised and We compared our model to two different baselines. The first one is composed of 4 ResNet-50 modules (He et al., 2016) with shared weights followed by 3 fully-connected layers. We trained this CNN only on the labeled samples. Our second baseline has a similar architecture as the previous one but the fully-connected layers were removed. Instead of the supervised loss provided in the form of correct answers, it is trained on the contrastive loss (Koch et al., 2015) . This loss reduces the feature distance between the references and correct answers and maximizes the feature distance between the references and incorrect answers. A more detailed description of the networks and contrastive loss function can be found in the appendix D.

Table 3 shows 3D-IQTT results for our method and baselines. The baselines were not able to interpret the underlying 3D structure of the data and its results are only slightly better than a random guess. The poor performance of the Siamese CNN might be in part because the contrastive loss rewards similarities in pixel space and has no notion of 3D similarity. However, pix2scene achieved significantly better accuracy by leveraging the learned 3D knowledge of objects.

5. CONCLUSIONS:

In this paper we proposed a generative approach to learn 3D structural properties from single images in an unsupervised and implicit fashion. Our model receives an image of a scene with uniform material as input, estimates the depth of the scene points and then reconstructs the input scene. We also provided quantitative evidence that support our argument by introducing a novel IQ-task in a semi-supervised setup. We hope that this evaluation metric will be used as a standard benchmark to measure the 3D understanding capability of the models across different 3D representations. The main drawback of our current model is that it requires the knowledge of lighting and material properties. Future work will focus on tackling the more ambitious setting of learning complex materials and texture along with modelling the lighting properties of the scene. In this case we show the normal maps of those views. Figure 16 shows the recovered shading, depth and normal images from reconstructions of complex scenes such as bedrooms and bunny. 

:

three

two

Figures and Tables:

Figure 2: Differentiable 3D renderer. (a) A surfel is defined by its position P , normal N , and reflectance ρ. Each surfel maps to an image pixel P im . (b) The surfel's color depends on its reflectance ρ and the angles θ between each light I and the surfel's normal N .

Figure 3: Pix2scene model. Pix2scene generates realistic 3D views of scenes by training on 2D images alone. Its decoder generates the surfels depth p z from a noise vector z conditioned on the camera pose. The surfels normal is estimated from its predicted depth. The surfels are then rendered into a 2D image and together with image samples from the target distribution are fed to the critic.

Figure 4: Scene reconstruction. Left: Input images of rotated objects into a room with its depth and normal groundtruth maps. Right: pix2scene reconstructions with its depth and normal maps.

Figure 5: View point reconstruction. Given a scene (first column), we rotate the camera around to visualize the models understanding of 3D shape. As shown, the model correctly infers the unobserved geometry of the objects, demonstrating true 3D understanding of the scene. Videos of these reconstructions can be seen at https://bit.ly/2zADuqG.

Figure 6: Multiple-shape scenes reconstruction. Implicit 3D reconstruction of scenes composed by multiple ShapeNet objects.

Figure 7: Unconditional scene generation. Generated samples from pix2scene model trained on ShapeNet scenes. Left: shaded images; Right: depth maps

Figure 9: Manifold exploration. Exploration of the learned manifold of 3D representations. Generated interpolations (middle columns) between two images x 1 and x 2 (first and last columns).

Figure10: Sample questions from the 3D-IQ test task. For this "mental rotation" task, a set of reference images and 3 possible answers are presented. The goal is to find the rotated version of the reference 3D model. To solve this task, the human or the model has to infer the 3D shape of the reference from the 2D image and compare that to the inferred 3D shapes of the answers. The correct answers to these two examples are in the footnote.

Figure 13: 3D-IQTT baseline architecture. The ResNet-50 all share the same weights and were slightly modified to support our image size. "FC" stands for fully-connected layer and the hidden node sizes are 2048, 512, and 256 respectively. The output of the network is encoded as one-hot vector.

Figure 14: Scene reconstruction. (a) Input images of rotated cubes into a room. (b) pix2scene reconstructions with its (c) associated depth maps.

Figure 15: Normal views reconstruction. For each row, the first column is the input image and other columns are the extrapolated normal maps of that image from different views.

Figure 16: Reconstruction of complex scenes. Reconstruction of bedroom scenes and bunny.

Scene reconstruction results. Hausdorff metric on 3D surfels and MSE on the depth maps.

Table content: <table><tr><td>4 EXPERIMENTS</td></tr><tr><td>4.1 EXPERIMENTAL SETUP</td></tr><tr><td>Tasks. Our model is capable of both reconstructing 3D scenes and generate new ones. We evaluate</td></tr><tr><td>the 3D understanding capability of the model on 3D-IQTT: a spatial reasoning based semi-supervised</td></tr><tr><td>classification task. The goal of the 3D-IQTT is to quantify ability of our model to perform 3D spatial</td></tr><tr><td>reasoning test by using large amounts of the unlabeled training data and considerably small set of</td></tr><tr><td>labelled examples.</td></tr></table>

E(x ref ); ∀(x ref , x d1 , x d2 , x ans ) ∈ D ALI + L recon + I Θ (z scene , z view ) + L θ (x ref , x d1 , x d2 , x ans ) 3D-IQTT quantitative results.The test accuracy of the 3D-IQ test task show that the CNN baselines struggle to solve this task Pix2scene is able to understand the underlying 3D structure of the images and solve the task. The results show that although our model performs better than the baselines, we are still lagging behind the human level.supervised training iterations. Once the model is trained, we answer 3D-IQTT questions, by inferring the latent 3D representation for each of the 4 images and we select the answer closest to the reference image as measured by L2 distance.

Table content: <table><tr><td colspan="4">Algorithm 1 Semisupervised classification</td><td/></tr><tr><td colspan="2">1: while iter &lt; max iter do</td><td/><td/><td/></tr><tr><td>2:</td><td>D ← MiniBatch()</td><td/><td/><td/></tr><tr><td colspan="4">3: z ∼ 4: L ← L 5: if supervised-training-interval(iter) then</td><td/></tr><tr><td colspan="2">6: L ← L 7: end if</td><td/><td/><td/></tr><tr><td>8:</td><td colspan="2">optimize networks with L</td><td/><td/></tr><tr><td colspan="2">9: end while</td><td/><td/><td/></tr><tr><td/><td>Labeled Samples</td><td>CNN</td><td>Siamese CNN</td><td>Human Evaluation</td><td>Pix2Scene (Ours)</td></tr><tr><td/><td colspan="3">0 (Unsupervised) 0.3385 0.3698 200 0.3350 0.3610 1,000 0.3392 0.3701 10,000 0.3649 0.3752</td><td colspan="2">0.7329 + -0.1488 0.4372 + -0.0301 -0.4691 + -0.0259 -0.5567 + -0.0095 -0.5983 + -0.0021</td></tr></table>

Peer review (iclr2019):
 This paper introduces a method to create a 3D scene model given a 2D image and a camera pose. The method is: (1) an "encoder" network maps the image to some latent code vector, (2) a "decoder" network uses the code and the camera pose to create a depthmap, (3) surface normals are computed from the depthmap, and (4) these outputs are fed to a differentiable renderer which reconstructs the input image. At training time, a discriminator provides feedback to (and simultaneously trains on) the latent code and the reconstructions. The model is self-supervised by the reconstruction error and the GAN setup. Experiments show compelling results in 3D scene generation for simple monochromatic synthetic scenes composed of an empty room corner and floating ShapeNet shapes. This is a nice problem, and if the approach ever works in the real world, it will be useful. On synthetic environments, the results are impressive. The paper seems to claim more ground than it actually covers. The abstract says "Our method learns the depth and orientation of scene points visible in images", but really only the depth is learned, and the "orientation" is an automatically-computed surface normal, which is a free byproduct of any depth estimate. The "surfel" description includes a reflectance vector, but this is never estimated or further described in the paper, so my guess is that it is simply treated as a scalar (which equals 1). Taking this reflectance issue together with the orientation issue, the model is not really estimating surfels at all, but rather just a depthmap, which makes the method seem considerably less novel. Furthermore, the differentiable rendering (eq. 1) appears to assume that all light sources are known exactly -- this is not a trivial assumption, and yet it is never mentioned in the paper. The text suggests that only an image is required to run the model, but Figure 3 shows that the networks are conditioned on the camera pose -- exact knowledge of the camera pose is difficult to obtain precisely in real settings, so this again is not an assumption to ignore. To rewrite the paper more plainly, one might say that it receives a monochrome image as input, estimates a depthmap, and then shades this depthmap using perfect knowledge of lighting and camera pose, which reconstructs the input. This may sound less appealing, but it also seems more accurate. The paper is also missing some details of the method and evaluation, which I hope can be cleared up easily. - What is happening with the light source? This is critical in the shading equation (eq. 1), and yet no information is given on it -- we need the color and the position of every light in the scene. - How is the camera pose represented? Section 3.3.3 says conditional normalization is used, but what exactly is fed to the network that estimates these conditional normalization parameters? - What is the exact form of the reconstruction error? An equation would be great. - How is the class-conditioning done in 4.2? - In Eq. 4, the first usage of D_.theta should use only the object part of the vectors, and the second usage should use only the geometric part, right? Maybe this can be cleared up with a second D_subscript. - I do not understand the "interleaved" training setup in 4.4.1. Please explain that more. - It is not clear to me why the task in 4.4.2 needs any supervised training at all, if the classification is just done by computing L2 distances in the latent space. What happens with "0 sampled labels"? Overall, I like the paper, and I can imagine others in my group liking it. I hope it gets in, assuming the technical details get cleaned up and the language gets softer.