OBJECT-ORIENTED REPRESENTATION OF 3D SCENES

1 INTRODUCTION . The shortcomings of contemporary deep learning such as interpretability , sample efficiency , ability for reasoning and causal inference , transferability , and compositionality , are where the symbolic AI has traditionally shown its strengths ( Garnelo & Shanahan , 2019 ) . Thus , one of the grand challenges in machine learning has been to make deep learning embrace the benefits of symbolic representation so that symbolic entities can emerge from high-dimensional observations such as visual scenes . In particular , for learning from visual observations of the physical world , such representation should consider the following criteria . First , it should focus on objects ( and their relations ) which are foundational entities constructing the physical world . These can be considered as units on which we can build a modular model . The modular nature also helps compositionality ( Andreas et al. , 2016 ) and transferability ( Kansky et al. , 2017 ) . Second , being three-dimensional ( 3D ) is a decisive property of the physical world . We humans , equipped with such 3D representation in our brain ( Yamane et al. , 2008 ) , can retain consistency on the identity of an object even if it is observed from different viewpoints . Lastly , learning such representation should be unsupervised . Although there have been remarkable advances in supervised methods to object perception ( Redmon et al. , 2016 ; Ren et al. , 2015 ; Long et al. , 2015 ) , the technology should advance toward unsupervised learning as we humans do . This not only avoids expensive labeling efforts but also allows adaptability and flexibility to the evolving goals of various downstream tasks because “ objectness ” itself can vary on the situation . In this paper , we propose a probabilistic generative model that can learn , without supervision , objectoriented 3D representation of a 3D scene from its partial 2D observations . We call the proposed model ROOTS ( Representation of Object-Oriented Three-dimensional Scenes ) . We base our model on the framework of Generative Query Networks ( GQN ) ( Eslami et al. , 2018 ; Kumar et al. , 2018 ) . However , unlike GQN which provides only a scene-level representation that encodes the whole 3D scene into a single continuous vector , the scene representation of ROOTS is decomposed into objectwise representations each of which is also an independent , modular , and 3D representation . Further , ROOTS learns to model a background representation separately for the non-object part of the scene . The object-oriented representation of ROOTS is more interpretable , composible , and transferable . Besides , ROOTS provides the two-level hierarchy of the object-oriented representation : one for a global 3D scene and another for local 2D images . This makes the model more interpretable and provides more useful structure for downstream tasks . In experiments , we show the above abilities of ROOTS on the 3D-Room dataset containing images of 3D rooms with several objects of different colors and shapes . We also show that these new abilities are achieved without sacrificing generation quality compared to GQN . Our proposed problem and method are significantly different from existing works on visual 3D learning although some of those partly tackle some of our challenges . First , our model learns factorized object-oriented 3D representations which are independent and modular , from a scene containing multiple objects with occlusion and partial observability rather than a single object . Second , our method is unsupervised , not using any 3D structure annotation such as voxels , cloud points , or meshes as well as bounding boxes or segmentation annotation . Third , our model is a probabilistic generative model learning both representation and rendering with uncertainty modeling . Lastly , it is trained end-to-end . In Section 4 , we provide more discussion on the related works . The main contributions are : ( i ) We propose , in the GQN framework , a new problem of learning object-oriented 3D representations of a 3D scene containing multiple objects with occlusion and partial observability in the challenging setting described above . ( ii ) We achieve this by proposing a new probabilistic model and neural architecture . ( iii ) We demonstrate that our model enables various new abilities such as compositionality and transferability while not losing generation quality . 2 PRELIMINARY : GENERATIVE QUERY NETWORKS . The generative query networks ( GQN ) ( Eslami et al. , 2018 ) is a probabilistic generative latentvariable model providing a framework to learn a 3D representation of a 3D scene . In this framework , an agent navigating a scene i collects K images xki from 2D viewpoint v k i . We refer this collection to context observations Ci = { ( xki , vki ) } k=1 , ... , K . While GQN is trained on a set of scenes , in the following , we omit the scene index i for brevity and discuss a single scene without loss of generality . GQN learns scene representation z from context C. The learned representation z of GQN is a 3Dviewpoint invariant representation of the scene in the sense that , given an arbitrary query viewpoint vq , its corresponding 2D image xq can be generated from the representation . In the GQN framework , there are two versions . The standard GQN model ( Eslami et al. , 2018 ) uses the query viewpoint to generate representation whereas the Consistent GQN ( CGQN ) ( Kumar et al. , 2018 ) uses the query after generating the scene representation in order to obtain queryindependent scene-level representation . Although we use CGQN as our base framework to obtain query-independent scene-level representation , in the rest of the paper we use the abbreviation GQN instead of CGQN to indicate the general GQN framework embracing both GQN and CGQN . The generative process of GQN is written as follows : p ( xq|vq , C ) = ∫ p ( xq|vq , z ) p ( z|C ) dz . As shown , GQN uses a conditional prior p ( z|C ) to learn scene representation z from context . To do this , it first obtains a neural scene representation r from the representation network r = frepr-gqn ( C ) which combines the encodings of ( vk , xk ) ∈ C in an order-invariant way such as sum or mean . It then uses ConvDRAW ( Gregor et al. , 2016 ) to generate the scene latent variable z from scene representation r by p ( z|C ) = ∏ l=1 : L p ( z l|z < l , r ) = ConvDRAW ( r ) with L autoregressive rollout steps . Due to intractability of the posterior distribution p ( z|C , vq , xq ) , GQN uses variational inference for posterior approximation and the reparameterization trick ( Kingma & Welling , 2013 ) for backpropagation through stochastic variables . The objective is to maximize the following evidence lower bound ( ELBO ) via gradient-based optimization . log pθ ( x q | vq , C ) ≥ Eqφ ( z|C , vq , xq ) [ log pθ ( x q | vq , z ) ] −KL ( qφ ( z | C , vq , xq ) ‖ pθ ( z | C ) ) . Note that although in this paper we use a single target observation D = ( xq , vq ) for brevity , the model is in general trained on a set of target observations D = { ( xqj , v q j ) } j . 3 ROOTS : REPRESENTATION OF OBJECT-ORIENTED 3D SCENES . 3.1 GENERATIVE PROCESS . The main difference of our model from GQN is that we have a 3D representation per object present in the target 3D space while GQN has a single 3D representation compressing the whole 3D space into a vector without object-level decomposition . We begin this modeling by introducing the number of objects M in the target space as a random variable . Then , we can write the representation prior of ROOTS as p ( z , M |C ) = p ( M |C ) ∏M m=1 p ( z ( m ) |C ) . To implement such a model with a variable number of objects , in AIR ( Eslami et al. , 2016 ) , the authors proposed to use an RNN that rolls out M steps , processing one object per step . However , according to our preliminary investigation ( under review ) and other works ( Crawford & Pineau , 2019 ) , it turned out that this approach is computationally inefficient and shows severe performance degradation with growing M . Object-Factorized Conditional Prior . To resolve this problem , in ROOTS we propose to process objects in a spatially local and parallel way instead of sequential processing . This is done by first introducing the scene-volume feature-map . The scene-volume feature-map is obtained by encoding contextC into a 3D tensor ofN = ( H×W×L ) cells . Each cell n ∈ { 1 , . . . , N } is then associated to D-dimensional volume feature rn ∈ RD . Thus , the actual output of the encoder is a 4-dimensional tensor r = frepr-scene ( C ) . Each volume feature rn represents a local 3D space in the target 3D space in a similar way that a feature vector in a 2D feature-map of CNN models a local area of an input image . Note , however , that the introduction of the scene-volume feature-map is not the same as the feature-map of 2D images because , unlike the CNN feature-map for images , the actual 3D target space is not directly observable—it is only observed through a proxy of 2D images . For the detail implementation of the encoder frepr-scene , refer to the Appendix A.3 . Given the scene-volume feature-map , for each volume cell n = 1 , . . . , N but in parallel , we obtain three latent variables ( zpresn , z pos n , z what n ) = zn from the 3D-object prior model p ( zn|rn ) . We refer this collection of object latent variables z = { zn } Nn=1 to the scene-object latent-map as z is generated from the scene-volume feature-map r. Here , zpresn is a Bernoulli random variable indicating whether an object is associated ( present ) to the volume cell or not , zposn is a 3-dimensional coordinate indicating the position of an object in the target 3D space , and zwhatn is a representation vector for the appearance of the object . We defer a more detail description of the 3D-object prior model p ( zn|rn ) to the next section . Note that in ROOTS we obtain zwhatn as a 3D representation which is invariant to 3D viewpoints . The position and appearance latents for cell n are defined only when the cell has an associated object to represent , i.e. , zpresn = 1 . From this modeling using scenevolume feature-map , we can obtain M = ∑ n z pres n and the previous prior model can be written as follows : p ( z , M | C ) = p ( M | C ) ∏M m=1 p ( z ( m ) | C ) = N∏ n=1 p ( zn | rn ) = N∏ n=1 p ( zpresn |rn ) [ p ( zposn | rn ) p ( zwhatn | rn , zposn ) ] zpresn . ( 1 ) In addition to allowing spatially parallel and local object processing , another key idea behind introducing a presence variable per volume cell is to reflect the inductive bias of physics : two objects can not co-exist at the same position . This helps remove the sequential object processing because dealing with an object does not need to consider other objects if their features are from spatially distant areas . Note also that the scene-volume feature-map is not to strictly partition the target 3D space and that the presence variable represents the existence of the center position of an object not the full volume of an object . Thus , information about an object can exist across neighboring cells . Hierarchical Object-Oriented Representation . The object-oriented representations z = { zn } provided by the above prior model is global in the sense that it contains all objects in the whole target 3D space , independently to a query viewpoint . From this global representation and given a query viewpoint , ROOTS generates a 2D image corresponding to a query viewpoint . This is done first by learning the view-dependent representation of the target image . In a naive approach , this may be done simply by learning a single vector representation p ( zq|z , vq ) but in this case , we lose important information : the correspondence between a rendered object in the image and a global object representation zn . That is , we can not track from which object representation zn an object in the image is rendered . In ROOTS , we resolve this problem by introducing local 2D-level objectoriented representation layer . This local object-oriented and view-dependent representation allows additional useful structure and more interpretability . This 2D local representation is similar to those in AIR ( Eslami et al. , 2016 ) and SPAIR ( Crawford & Pineau , 2019 ) . Specifically , for n for which zpresn = 1 , a local object representation sn is generated by conditioning on the global representation set z and the query vq . Our local object representation model is written as : p ( s|z , vq ) = ∏N n=1 p ( sn|z , vq ) . Similar to the decomposition of zn , local object representation sn consists of ( spresn , s pos n , s scale n , s what n ) . Here , s pres n indicates whether an object n should be rendered in the target image from the perspective of the query . Thus , even if an object exists in the target 3D space , i.e. , zpresn = 1 , s pres n can be set to zero if that object should be invisible from the query viewpoint . Similarly , sposn and s scale n represent respectively the position and scale of object n in the image not in the 3D space , and swhatn represents the appearance to be rendered into the image ( thus not 3D invariant ) . For more details about how to obtain ( spresn , s pos n , s scale n , s what n ) from z and vq , we describe in the next section . Given s = { sn } , we then render to the canvas to obtain the target image p ( xq|s ) . Combining all , the generative process of ROOTS is written as follows : p ( xq|vq , C ) = ∫ p ( xq|s ) N∏ n=1 p ( sn|z , vq ) N∏ n=1 p ( zn|C ) dzds . ( 2 ) See Figure 1 for the overview of the generation process .

Summary: The paper proposes a model building off of the generative query network model that takes in as input multiple images, builds a model of the 3D scene, and renders it. This can be trained end to end. The insight of the method is that one can factor the underlying representation into different objects. The system is trained on scenes rendered in mujoco.