Pix2Scene: Learning Implicit 3D Representations from Images

1 INTRODUCTION . Understanding the 3-dimensional ( 3D ) world from its 2-dimensional ( 2D ) projections is a fundamental problem in computer vision with a broad range of application in robotics , simulation and design . Given that the majority natural scene data is available exclusively in the form of 2D images , the ability to directly infer knowledge about 3D structure from these images would be of great utility in scene understanding . Inferring the 3D structure from multiple images of a scene has been pursued extensively , such as in stereo or structure from motion tasks ( Hartley & Zisserman , 2004 ) . Since most available natural image data informative about the real world comes with only a single view of a given scene , it is perhaps more important to explore the development of models which can infer the 3D structural properties from a single image . On the other hand , single image 3D recovery is an extremely challenging and heavily under constrained task . The system has to rely on prior knowledge and 2D visual cues such as textures , shadows or occlusions in order to provide hints to the 3D structure of the scene . Practically , building a machine learning model that learns to infer 3D structure from images requires either a strong inductive bias or supervision . While some have used the 3D ground truth as explicit supervision ( Wu et al. , 2016 ; 2015 ) , in most cases of interest , such supervision will not be available . Consequently , our long term goal is to infer the 3D structure of realistic scenes from single images . In this paper we take a step towards this direction via a method of unsupervised learning of the 3D structure , directly from a single 2D image of each scene . Our method based on the adversarial learning framework ( Goodfellow et al. , 2014 ) and exploits a uniquely suitable 3D representation ( i.e. , surfels ( Pfister et al. , 2000 ) ) and a differentiable renderer . Most 3D reconstruction methods rely on representing 3D objects explicitly using either voxels ( Rezende et al. , 2016 ; Yan et al. , 2016 ) or meshes ( Kanazawa et al. , 2018 ; Wang et al. , 2018 ) . Explicit representations store all the rendering-relevant information from a given 3D space and are easily transferable , i.e. , they can be loaded with any 3D modeling software and viewed from any angle . However , approaches using explicit representations typically scale very poorly ( O ( n3 ) or require a sparse/discrete representation which can be challenging for deep learning methods . As a result , these representations have only been applied to the reconstruction of single objects . As an alternative we propose to learn an implicit 3D representation which produces only the 3D geometry which is directly relevant for a particular viewpoint . Our viewpoint-specific 3D geometry is captured using camera facing surfels ( Pfister et al. , 2000 ) which are surface elements defined by its position , orientation and material properties . Given an image we can infer its implicit 3D representation and then recreate novel surfel representations of the underlying scene from unobserved viewpoints . In general , we note that in a 3D scene , only a small fraction of the entities are perceivable from the camera . As the camera moves , and the occluded regions become visible , our method then generates surfels for those newly unoccluded regions . Another advantage of this approach is that minimal number of primitives ( surfels ) are required to obtain a high-resolution image as the camera moves closer to a part of the scene . Moreover this representation fits well with image based convolutional architectures . Our model , Pix2Scene , is a deep generative-based approach for modelling the 3D structure of a scene directly from images . This model is unsupervised in the sense that it does not require 3D groundtruth or any other kind of image annotations . We base our model on Adversarially Learned Inference ( ALI ) approach ( Dumoulin et al. , 2016 ) . ALI extends the GAN ( Goodfellow et al. , 2014 ) framework by learning to infer the latent representation of a given image . In pix2scene the learned latent space embeds the 3D information of the underlying scene . The latent representation is mapped via a decoder network to a view-dependent 3D surface and then projected to image space by a differentiable renderer . The resulting image is then evaluated by an adversarial critic . While our long-term goal is to be able to infer the 3D structure of a real-world photograph , in this paper we experiment exclusively with synthetically-constructed scenes and adopt several simplifying assumptions . In particular , we assume that the world is piece-wise smooth and that for each input image the illumination , view and object materials are known . This work has the following main contributions , ( 1 ) we propose a novel unsupervised method for 3D understanding from a single image ; ( 2 ) we propose a new implicit 3D representation based on view-space surfels ; ( 3 ) we propose a surfel-based differentiable 3D renderer that can be used as a layer of a neural network ; and ( 4 ) we propose 3D-IQTT a new 3D understanding evaluation benchmark . This task evaluates the model ’ s ability to perform mental rotation by obtaining comprehensive understanding of underlying 3D structure . We also estimate the camera pose as part of the learnt latent variable for this particular task . 2 RELATED WORK . The generation and reconstruction of 3D objects from images has been studied extensively in the computer vision and graphics communities ( Saxena et al. , 2009 ; Chaudhuri et al. , 2011 ; Kalogerakis et al. , 2012 ; Chang et al. , 2015 ; Rezende et al. , 2016 ; Soltani et al. , 2017 ) . Our work bears some conceptual similarities to Kulkarni et al . ( 2015 ) which casts the 3D reconstruction problem as a more traditional inverse graphics task . By using Variational Auto-Encoder ( VAEs ) ( Kingma & Welling , 2014 ) , they learn a representation of objects that disentangles factors of variations from images ( i.e. , object pose and configuration ) and use the approach for specific transformations such as out of axis rotation . However , unlike their approach , ours is fully unsupervised and we implicitly generate 3D structure of scenes from single images . Our mechanism learns a latent representation for the underlying scene , which can later be used to render from different views and lighting conditions . Similar to ours , Rezende et al . ( 2016 ) infer the 3D configuration at their output . They adopt a probabilistic inference framework to build a generative model for 3D by combining a standard projection mechanism with gradient estimation methods . In particular , their approach requires multiple runs with mechanisms such as REINFORCE ( Williams , 1992 ) in order to infer gradient from the projection layer . In addition , their use of mesh and voxel representations could become an obstacle to scaling their method to more complex scenes . Our approach is not susceptible to restrictions imposed by meshes or other scaling issues and has the potential to adapt to arbitrary scene configurations . 3 METHOD . 3.1 IMPLICIT 3D REPRESENTATION AND SURFELS . Explicitly representing 3D structure presents different challenges for generative models ( Kobbelt & Botsch , 2004 ) . Representing entire objects using voxels scales poorly given its ( O ( n3 ) complexity . The vast majority of the generated voxels aren ’ t relevant to most viewpoints , such as the voxels that are entirely inside of objects . A common workaround is to use a sparse representation such as meshes . However , these too come with their own drawbacks , such as the need to discretise complex objects . This makes mesh representation difficult to generate using neural networks . Current mesh based methods mainly rely on deforming a pre-existing mesh . On the other hand , our implicit approach represents the 3D scene in a high-dimensional latent variable . In our framework , this latent variable ( or vector ) is decoded using a generator network into a viewpoint-dependent representation of surface elements — similar to the surfels ( Pfister et al. , 2000 ) — that constitute the visible part of the scene . This representation is very compact : given a renderer ’ s point of view , we can represent only the part of the 3D surface needed by the renderer . Also , as the camera moves closer to a part of the scene , our generator will allocate more surfels to represent that part of the scene and thereby increasing the resolution . Figure 1 illustrates these different representations . For descriptive purpose , surfels are shown as squares , but in general they do not have any shape . Formally , surfels are represented as a tuple ( P , N , ρ ) , where P = ( px , py , pz ) is its 3D position , N = ( nx , ny , nz ) is the surface normal vector , and ρ = ( kr , kg , kb ) is the reflectance of the surface material . Since we are interested in modelling structural properties of the scenes i.e. , geometry and depth , we assume that objects in the scene have a uniform material . We represent the surfels in the camera coordinate system . This significantly reduces the number of surfels by considering only the ones that will get projected onto a pixel in the rendered image . Moreover , this allows to reduce the position parameters to only pz being this the distance along a ray going through the surfel to the center of its pixel . 3.2 DIFFERENTIABLE 3D RENDERER . As the critic operates only on image space , we need to project the generated 3D representations back to the 2D space using a renderer . In our setting , each stage of the rendering pipeline must be differentiable to allow us to take advantage of gradient-based optimization and backpropogate the critic ’ s error signal to the surfel representation . The rendering process can be partitioned into two stages . During forward-propagation , the first stage finds the mapping between the surfels and the pixels ; and the second stage computes the color of each pixel . During back-propagation , the first stage directs the gradients only to the surfels that get projected onto the image ; and the second stage is differentiable as long as the shading operations are differentiable . The first stage of the rendering involves finding the mapping between the surfels and the pixels . This requires performing the expensive operation of ray object intersection ( See Figure 2a ) . Our model requires a fast rendering engine as it will be use in every learning iteration . Conventional ray tracing algorithms are optimized for generating multiple views from the same scene . However in our setting during learning we render only one image from each scene . Moreover ray tracing algorithms require from representing the full scene , which is very inefficient as we only represent the part visible by the camera . To resolve these issues , our generator proposes one surfel for each pixel in the camera ’ s coordinate system . Our PyTorch implementation of the differentiable renderer can render a 128×128 surfel-based scene in under 1.4 ms on a mobile NVIDIA GTX 1060 GPU . The color of a surfel depends on the material reflectance , its position and orientation , and the ambient and point light source colors . See Figure 2b . Given a surface point Pi , the color of its corresponding pixel Irc is given by the shading equation : Irc = ρi ( La + ∑ j 1 kl‖dij‖+ kq‖dij‖2 Lj max ( 0 , NTi dij/‖dij‖ ) ) , ( 1 ) where ρi is the surface reflectance , La is the ambient light ’ s color , Lj is the jth positional light source ’ s color , with dij = L pos j − Pi , or the direction vector from the scene point to the point light source , and kl , kq being the linear and quadratic attenuation terms respectively . Equation 1 is an approximation of rendering equation proposed in Kajiya ( 1986 ) .

Summary: This paper introduces a method to create a 3D scene model given a 2D image and a camera pose. The method is: (1) an "encoder" network maps the image to some latent code vector, (2) a "decoder" network uses the code and the camera pose to create a depthmap, (3) surface normals are computed from the depthmap, and (4) these outputs are fed to a differentiable renderer which reconstructs the input image. At training time, a discriminator provides feedback to (and simultaneously trains on) the latent code and the reconstructions. The model is self-supervised by the reconstruction error and the GAN setup. Experiments show compelling results in 3D scene generation for simple monochromatic synthetic scenes composed of an empty room corner and floating ShapeNet shapes.