Year: 2019

Abstract:
In this paper, we propose a probabilistic generative model, called ROOTS, for unsupervised learning of object-oriented 3D-scene representation and rendering. ROOTS bases on the Generative Query Network (GQN) framework. However, unlike GQN, ROOTS provides independent, modular, and object-oriented decomposition of the 3D scene representation. In ROOTS, the inferred object-oriented representation is 3D in the sense that it is 3D-viewpoint invariant as the scene-level representation of GQN is so. ROOTS also provides hierarchical object-oriented representation: at 3D global-scene level and at 2D local-image level. In experiments, we demonstrate on datasets of 3D rooms with multiple objects, the above properties by focusing on its abilities for disentanglement, compositionality, transferability, and generalization. ROOTS achieves this without performance degradation on generation quality in comparison to GQN.

heading: 1 INTRODUCTION
text: The shortcomings of contemporary deep learning such as interpretability, sample efficiency, ability for reasoning and causal inference, transferability, and compositionality, are where the symbolic AI has traditionally shown its strengths (Garnelo & Shanahan, 2019). Thus, one of the grand challenges in machine learning has been to make deep learning embrace the benefits of symbolic representation so that symbolic entities can emerge from high-dimensional observations such as visual scenes.
In particular, for learning from visual observations of the physical world, such representation should consider the following criteria. First, it should focus on objects (and their relations) which are foundational entities constructing the physical world. These can be considered as units on which we can build a modular model. The modular nature also helps compositionality (Andreas et al., 2016) and transferability (Kansky et al., 2017). Second, being three-dimensional (3D) is a decisive property of the physical world. We humans, equipped with such 3D representation in our brain (Yamane et al., 2008), can retain consistency on the identity of an object even if it is observed from different viewpoints. Lastly, learning such representation should be unsupervised. Although there have been remarkable advances in supervised methods to object perception (Redmon et al., 2016; Ren et al., 2015; Long et al., 2015), the technology should advance toward unsupervised learning as we humans do. This not only avoids expensive labeling efforts but also allows adaptability and flexibility to the evolving goals of various downstream tasks because “objectness” itself can vary on the situation.
In this paper, we propose a probabilistic generative model that can learn, without supervision, objectoriented 3D representation of a 3D scene from its partial 2D observations. We call the proposed model ROOTS (Representation of Object-Oriented Three-dimensional Scenes). We base our model on the framework of Generative Query Networks (GQN) (Eslami et al., 2018; Kumar et al., 2018). However, unlike GQN which provides only a scene-level representation that encodes the whole 3D scene into a single continuous vector, the scene representation of ROOTS is decomposed into objectwise representations each of which is also an independent, modular, and 3D representation. Further, ROOTS learns to model a background representation separately for the non-object part of the scene. The object-oriented representation of ROOTS is more interpretable, composible, and transferable. Besides, ROOTS provides the two-level hierarchy of the object-oriented representation: one for a global 3D scene and another for local 2D images. This makes the model more interpretable and provides more useful structure for downstream tasks. In experiments, we show the above abilities of ROOTS on the 3D-Room dataset containing images of 3D rooms with several objects of different
colors and shapes. We also show that these new abilities are achieved without sacrificing generation quality compared to GQN.
Our proposed problem and method are significantly different from existing works on visual 3D learning although some of those partly tackle some of our challenges. First, our model learns factorized object-oriented 3D representations which are independent and modular, from a scene containing multiple objects with occlusion and partial observability rather than a single object. Second, our method is unsupervised, not using any 3D structure annotation such as voxels, cloud points, or meshes as well as bounding boxes or segmentation annotation. Third, our model is a probabilistic generative model learning both representation and rendering with uncertainty modeling. Lastly, it is trained end-to-end. In Section 4, we provide more discussion on the related works.
The main contributions are: (i) We propose, in the GQN framework, a new problem of learning object-oriented 3D representations of a 3D scene containing multiple objects with occlusion and partial observability in the challenging setting described above. (ii) We achieve this by proposing a new probabilistic model and neural architecture. (iii) We demonstrate that our model enables various new abilities such as compositionality and transferability while not losing generation quality.

heading: 2 PRELIMINARY: GENERATIVE QUERY NETWORKS
text: The generative query networks (GQN) (Eslami et al., 2018) is a probabilistic generative latentvariable model providing a framework to learn a 3D representation of a 3D scene. In this framework, an agent navigating a scene i collects K images xki from 2D viewpoint v k i . We refer this collection to context observations Ci = {(xki ,vki )}k=1,...,K . While GQN is trained on a set of scenes, in the following, we omit the scene index i for brevity and discuss a single scene without loss of generality. GQN learns scene representation z from context C. The learned representation z of GQN is a 3Dviewpoint invariant representation of the scene in the sense that, given an arbitrary query viewpoint vq , its corresponding 2D image xq can be generated from the representation.
In the GQN framework, there are two versions. The standard GQN model (Eslami et al., 2018) uses the query viewpoint to generate representation whereas the Consistent GQN (CGQN) (Kumar et al., 2018) uses the query after generating the scene representation in order to obtain queryindependent scene-level representation. Although we use CGQN as our base framework to obtain query-independent scene-level representation, in the rest of the paper we use the abbreviation GQN instead of CGQN to indicate the general GQN framework embracing both GQN and CGQN.
The generative process of GQN is written as follows: p(xq|vq, C) = ∫ p(xq|vq, z)p(z|C)dz. As shown, GQN uses a conditional prior p(z|C) to learn scene representation z from context. To do this, it first obtains a neural scene representation r from the representation network r = frepr-gqn(C) which combines the encodings of (vk,xk) ∈ C in an order-invariant way such as sum or mean. It then uses ConvDRAW (Gregor et al., 2016) to generate the scene latent variable z from scene representation r by p(z|C) = ∏ l=1:L p(z
l|z<l, r) = ConvDRAW(r) with L autoregressive rollout steps. Due to intractability of the posterior distribution p(z|C,vq,xq), GQN uses variational inference for posterior approximation and the reparameterization trick (Kingma & Welling, 2013) for backpropagation through stochastic variables. The objective is to maximize the following evidence lower bound (ELBO) via gradient-based optimization.
log pθ(x q | vq, C) ≥ Eqφ(z|C,vq,xq) [log pθ(x q | vq, z)]−KL(qφ(z | C,vq,xq) ‖ pθ(z | C)).
Note that although in this paper we use a single target observation D = (xq,vq) for brevity, the model is in general trained on a set of target observations D = {(xqj ,v q j )}j .

heading: 3 ROOTS: REPRESENTATION OF OBJECT-ORIENTED 3D SCENES

heading: 3.1 GENERATIVE PROCESS
text: The main difference of our model from GQN is that we have a 3D representation per object present in the target 3D space while GQN has a single 3D representation compressing the whole 3D space into a vector without object-level decomposition. We begin this modeling by introducing the number of objects M in the target space as a random variable. Then, we can write the representation
prior of ROOTS as p(z,M |C) = p(M |C)
∏M
m=1 p(z(m)|C). To implement such a model with a variable number of objects, in AIR (Eslami et al., 2016), the authors proposed to use an RNN that rolls out M steps, processing one object per step. However, according to our preliminary investigation (under review) and other works (Crawford & Pineau, 2019), it turned out that this approach is computationally inefficient and shows severe performance degradation with growing M .
Object-Factorized Conditional Prior. To resolve this problem, in ROOTS we propose to process objects in a spatially local and parallel way instead of sequential processing. This is done by first introducing the scene-volume feature-map. The scene-volume feature-map is obtained by encoding contextC into a 3D tensor ofN = (H×W×L) cells. Each cell n ∈ {1, . . . , N} is then associated to D-dimensional volume feature rn ∈ RD. Thus, the actual output of the encoder is a 4-dimensional tensor r = frepr-scene(C). Each volume feature rn represents a local 3D space in the target 3D space in a similar way that a feature vector in a 2D feature-map of CNN models a local area of an input image. Note, however, that the introduction of the scene-volume feature-map is not the same as the feature-map of 2D images because, unlike the CNN feature-map for images, the actual 3D target space is not directly observable—it is only observed through a proxy of 2D images. For the detail implementation of the encoder frepr-scene, refer to the Appendix A.3.
Given the scene-volume feature-map, for each volume cell n = 1, . . . , N but in parallel, we obtain three latent variables (zpresn , z pos n , z what n ) = zn from the 3D-object prior model p(zn|rn). We refer this collection of object latent variables z = {zn}Nn=1 to the scene-object latent-map as z is generated from the scene-volume feature-map r. Here, zpresn is a Bernoulli random variable indicating whether an object is associated (present) to the volume cell or not, zposn is a 3-dimensional coordinate indicating the position of an object in the target 3D space, and zwhatn is a representation vector for the appearance of the object. We defer a more detail description of the 3D-object prior model p(zn|rn) to the next section. Note that in ROOTS we obtain zwhatn as a 3D representation which is invariant to 3D viewpoints. The position and appearance latents for cell n are defined only when the cell has an associated object to represent, i.e., zpresn = 1. From this modeling using scenevolume feature-map, we can obtain M = ∑ n z pres n and the previous prior model can be written as
follows: p(z,M | C) = p(M | C) ∏M m=1 p(z(m) | C) =
N∏ n=1 p(zn | rn) = N∏ n=1 p(zpresn |rn) [ p(zposn | rn)p(zwhatn | rn, zposn ) ]zpresn . (1) In addition to allowing spatially parallel and local object processing, another key idea behind introducing a presence variable per volume cell is to reflect the inductive bias of physics: two objects cannot co-exist at the same position. This helps remove the sequential object processing because dealing with an object does not need to consider other objects if their features are from spatially
distant areas. Note also that the scene-volume feature-map is not to strictly partition the target 3D space and that the presence variable represents the existence of the center position of an object not the full volume of an object. Thus, information about an object can exist across neighboring cells.
Hierarchical Object-Oriented Representation. The object-oriented representations z = {zn} provided by the above prior model is global in the sense that it contains all objects in the whole target 3D space, independently to a query viewpoint. From this global representation and given a query viewpoint, ROOTS generates a 2D image corresponding to a query viewpoint. This is done first by learning the view-dependent representation of the target image. In a naive approach, this may be done simply by learning a single vector representation p(zq|z,vq) but in this case, we lose important information: the correspondence between a rendered object in the image and a global object representation zn. That is, we cannot track from which object representation zn an object in the image is rendered. In ROOTS, we resolve this problem by introducing local 2D-level objectoriented representation layer. This local object-oriented and view-dependent representation allows additional useful structure and more interpretability. This 2D local representation is similar to those in AIR (Eslami et al., 2016) and SPAIR (Crawford & Pineau, 2019).
Specifically, for n for which zpresn = 1, a local object representation sn is generated by conditioning on the global representation set z and the query vq . Our local object representation model is written as: p(s|z,vq) = ∏N n=1 p(sn|z,vq). Similar to the decomposition of zn, local object representation sn consists of (spresn , s pos n , s scale n , s what n ). Here, s pres n indicates whether an object n should be rendered in the target image from the perspective of the query. Thus, even if an object exists in the target 3D space, i.e., zpresn = 1, s pres n can be set to zero if that object should be invisible from the query viewpoint. Similarly, sposn and s scale n represent respectively the position and scale of object n in the image not in the 3D space, and swhatn represents the appearance to be rendered into the image (thus not 3D invariant). For more details about how to obtain (spresn , s pos n , s scale n , s what n ) from z and vq , we describe in the next section. Given s = {sn}, we then render to the canvas to obtain the target image p(xq|s). Combining all, the generative process of ROOTS is written as follows:
p(xq|vq, C) = ∫ p(xq|s) N∏ n=1 p(sn|z,vq) N∏ n=1 p(zn|C) dzds. (2)
See Figure 1 for the overview of the generation process.

heading: 3.2 IMPLEMENTATION DETAILS
text: Global 3D-object prior. The 3D-object prior p(zn|rn) generates three latents (zpresn , zposn , zwhatn ) as follows. It first obtains the presence latent from zpresn ∼ Bernoulli(fpres(rn)) and the 3-dimension position latent from zposn ∼ N (fµpos(rn), fσpos(rn)). Using these two latents, we then obtain the appearance latent zwhatn . This process is divided into object gathering and object encoding.
For object gathering, for each context image xk we attend and crop a patch that corresponds to object n. Specifically, we first notice that using the deterministic camera-coordinate projection function f3D→2D (which we do not learn), we can project from the perspective of a context viewpoint vk, a 3D position zposn in the global 3D coordinate system into a 2D position u pos k,n in context image x
k, i.e., uposk,n = f3D→2D(z pos n ,v
k). If object n should be invisible from the viewpoint vk, its projected 2D position uposk,n is out of the image x
k and we do not crop a patch. For more details about the projection function f3D→2D, refer to Wikipedia (2019). We also predict the bounding box scale uscalek,n = fscale(u pos k,n, rn,v
k). Given the center position and scale, we can crop a patch xkn ⊂ xk using the spatial transformer (Jaderberg et al., 2015). Applying this cropping to all context pairs (vk,xk) ∈ C, we gather a set of object image-patches Xn = {xkn}Kk=1 for all n with zpresn = 1.
Given object image-patches Xn, obtaining object 3D-representation zwhatn can be converted to a GQN encoding problem. That is, we can simply consider Xn and its corresponding viewpoints as a new object-level context Cn, and can run an order-invariant GQN representation network rwhatn = frepr-obj(Cn) and then run ConvDRAW to obtain zwhatn from r what n .
Local 2D-object prior. The intermediate prior p(sn|z,vq) generates (spresn , sposn , sscalen , swhatn ). This is done as follows. Similar to what we described in the above to obtain uposk,n, we use the coordinate projection function f3D→2D to find the position of a global object zn in the
target image from the perspective of query vq , i.e., sposq,n = f3D→2D(z pos n ,v q). Thus, we model the position as a deterministic variable. The scale sscaleq,n is also obtained similarly as described in the above for uscalek,n , but with random sampling. Then, we predict s pres n ∼ Bern(fpres(sposq,n , s scale q,n , zn,v
q)). If object n should be visible in the 2D target image, i.e., spresn = 1, we generate the 2D appearance representation swhatn by using an object-level GQN decoder based on ConvDRAW, i.e., swhatn = ConvDRAW(z what n ,v
q). Finally, we have p(sn|z,vq) = p(sposn |zposn ,vq)p(sscalen |sposn , zwhatn ,vq)p(spresn |sposn , sscalen , zn,vq). Rendering to 2D Canvas. A main challenge in rendering the local representation s = {sn} into the image canvas, i.e., p(xq|s), is to deal with occlusion. In ROOTS, this can easily be achieved by noticing (i) that the coordinate conversion f3D→2D(zposn ,v
q) actually converts a 3D-coordinate to another 3D-coordinate and (ii) that then the last dimension of the converted coordinate system can be interpreted as the orthogonal projection distance from the viewpoint vq to the object’s position zposn . We can use this distance as object depth from the observer’s perspective. This allows us to sort objects according to their depths and render each object accordingly without occlusion. To handle background, ROOTS has an independent module to infer the background separately at imagelevel. We also found that learning an object-level mask along with the appearance latent swhatn helps segmentation between foreground and background as well as generating less blurry images. More details on related implementation are provided in Appendix A.4.

heading: 3.3 LEARNING AND INFERENCE
text: Due to the intractability of the posterior p(z, s|C,D) with D = {(vqj ,x q j)}j the target viewpointimage pairs, we train ROOTS using variational inference with the following posterior approximation qφ(z, s|C,D) = qφ(z|C,D)qφ(s|z, C,D). To compute the gradient w.r.t. the continuous latent variables such as the position and appearance, we use reparameterization trick and for the discrete variables on the presence, we use a continuous relaxation using Gumbel-Softmax trick (Jang et al., 2016). Other methods based on the REINFORCE algorithm (Williams, 1992; Tucker et al., 2017; Grathwohl et al., 2017) can also be used. Implementation of the approximate posterior q(z|C,D) and q(s|z, C,D) is made easier by sharing the parameters of q with that of conditional prior p(z|C): we only need to provide additional data D. With D = (vq,xq) for simplicity, the objective is to maximize the following evidence lower bound (ELBO): L(θ, φ;C,D) =
Es,z∼qφ [log pθ(xq|s)−KL[qφ(s|z,vq, C,D) ‖ pθ(s|z,vq)]]−KL[qφ(z|C,D) ‖ pθ(z|C)]. (3)
Combining with Unconditioned Prior. One difficulty in using the conditional prior is the fact that, because the prior is learned, we have less control in reflecting our prior knowledge into the prior distribution. In our experiments, it turns out that biasing the posteriors of some variables towards the values of our prior preference is helpful in stabilizing the model. To implement this, in our training, we use the following objective that has additional KL terms between the posterior and unconditioned prior.
L = L(θ, φ;C,D) +KL[qφ(zpos, sscale|C,D) ‖ N (0, I)] + γKL[qφ(zpres|C,D) ‖ Geom(ρ)] + γEz∼qφ [KL[qφ(spres|zpres, C,D) ‖ Geom(ρ)]] (4)
where zpres = {zpresn }n and zpos, zscale, spres are defined similarly. The ρ and γ are hyperparameters weighting the auxiliary loss terms. We set them to 0.999 and 7 during training, respectively. This auxiliary loss can be derived if we replace the conditional prior by the product of expert prior p(z|C)p(z) divided by the posterior q(z|C).

heading: 4 RELATED WORKS
text: Although to our knowledge there has been no previous work on unsupervised and probabilistic generative object-oriented representation learning for 3D scenes containing multiple objects, there has been literature on its 2D problems. The first is the Attend, Infer, Repeat (AIR) model (Eslami et al., 2016). AIR uses spatial transformer (Jaderberg et al., 2015) to crop an object patch and uses an RNN to sequentially generate next object conditioning on the previous objects. In Crawford & Pineau (2019), the authors showed that this RNN-based rollout is inefficient and can significantly
degrade performance as the number of objects increases. SPAIR is inspired by YOLO (Redmon et al., 2016), but unlike YOLO, it does not require bounding box labels. The main limitation of SPAIR is to infer the latents sequentially. Neural Expectation Maximization (NEM) (Greff et al., 2017) considers the observed image as a pixel-level mixture of K objects and an image per object is generated and combined according to the mixture probability. In Greff et al. (2019), the authors proposed a more efficient version of NEM, called IODINE, using iterative inference (Marino et al., 2018). In MONET (Burgess et al., 2019), an RNN drawing a scene component at each time step is used. Unlike AIR and SPAIR, the representations in NEM, IODINE, and MONET do not explicitly provide natural disentanglement like presence and pose per object.
There have been plenty amount of remarkable works about visual 3D learning from the computer vision community. However, as pointed in Section 1, the problem setting and proposed model of these works are different from ours in the sense that they are either (i) not decomposing object-wise representations from a scene containing multiple objects (but working mostly on single object cases) (Wu et al., 2016; Yan et al., 2016; Choy et al., 2016; Kar et al., 2017; Nguyen-Phuoc et al., 2019), (ii) supervised approach (Huang et al., 2018; Tulsiani et al., 2018; Cheng et al., 2018; Shin et al., 2019; Du et al., 2018), (iii) learn image generation (synthesis) of a 3D scene without scene-representation (Sitzmann et al., 2019; Kato & Harada, 2019; Pinheiro et al., 2019; Tulsiani et al., 2017) or learn scene-representation without learning rendering ability (Zhou et al., 2017; Yu & Wang, 2018), or (iv) not end-to-end. Many of the more traditional works from 3D computer vision also relevant to our work but many of them are not based on neural networks and not end-to-end trainable.

heading: 5 EXPERIMENTS
text: We evaluate ROOTS quantitatively and qualitatively. We train ROOTS with the same hyperparameters on all datasets. We first briefly describe the datasets. For more details of the dataset generation and network architecture, refer to Appendix A.5 and Appendix A.4. We use MuJoCo, (Todorov et al., 2012), to simulate 3D scenes. Specifically, we generate three different datasets of scenes with 1-3 objects, 2-4 objects and 3-5 objects, respectively. We set the image size to 64×64×3 pixels. For each object, we randomly choose its position, shape, size, and color. Although we put all objects on the floor, we still predict the 3-dimension coordinate values for zposn because objects have different sizes. We generate 60K different scenes for each dataset and split them into 50k for training, 5k for validation, and 5k for testing. We use CGQN as the baseline implementation but in the rest of the section use the abbreviation ‘GQN’ instead of ‘CGQN’ to indicate the general GQN framework.

heading: 5.1 QUALITATIVE EVALUATION
text: First, we compare the generations between ROOTS with GQN by visualizing several generated samples from the same scene under the same set of query viewpoints. Note that the goal of this experiment is to show that achieving the object-wise representation in ROOTS does not deteriorate its generation quality in comparison to GQN. As seen in Figure 2, ROOTS generates slightly sharper object boundaries while GQN generates more blurry images. We believe that this is due to the object-wise generation using segmentation masks. Then, to provide a further understanding of the advantages of object-orientated representation, we visualize decomposed generations from ROOTS, as shown in Figure 3.A. We can see that ROOTS can generate clean background, complete objects, precise foreground, and detailed occlusion mask separately. Furthermore, to demonstrate the viewpoint-invariant property of the global object representation zwhatn , in Figure 3.B, we provide generations of two objects under different query viewpoints. We can see that ROOTS recovers the object with pose corresponding to query viewpoints. GQN cannot provide such decomposed generations because its representation is scene-level where objects are not decomposed.
Object-wise disentanglement. Generations from random viewpoints after changing object positions. In this section and the following sections, we use ROOTS trained on the 2-4 object dataset for demonstration unless otherwise stated. To verify the disentanglement property of our objectoriented representation learned by ROOTS, we carry out the experiment of arbitrarily modifying zposn of one object in a scene. As a good disentangled latent representation, this modification should not affect the position, existence, and appearance of other objects concurrent in the scene while handling occlusion properly. More importantly, we should be able to change either the x or y coordinate independently and this change should be consistent across viewpoints. To demonstrate this,
after changing one dimension of zposn of the red cylinder, as shown in Figure 4, we feed this modified ẑposn back to the model and follow the ROOTS generation model in the same way as we do during testing. Generations under 4 different viewpoints are shown in Figure 4. We can see that ROOTS has a strong ability to learn disentangled representation and occlusions have been handled well due to this advantage.
Compositionality. As stated in earlier sections that a 3D scene can be decomposed into several independent objects, another advantage coming with object-orientated representations is that a new scene can be easily built up with selected object components. Thus, by simple combination, we can build novel scenes. To demonstrate this, we first provide ROOTS with three sets of context images from three different scenes and save the learned object representations z for each scene, z = {zposn , zwhatn } for all n with zposn = 1. Then we swap one object between the first two scenes and add one additional object to the third scene, as shown in Figure 5. We see that the object component learned by ROOTS is fully disentangled and can be reused to create new scenes. Also, by adding one new object, we make a scene with 5 objects, which does not exist in the training dataset. Another example shown a scene 9 objects is visualized in Figure 6, where we use ROOTS trained on the 1-3 objects dataset. More details about this experiment can be found in Appendix A.2.
Partial Observability. We know that the successful generation of ROOTS, even with only partial observations provided, is coming from object gathering across viewpoints. This is because if an object is invisible for one viewpoint, ROOTS can learn it from other viewpoints. Here, we want
to push the partial observability to the extreme case, where one object is totally invisible for all context viewpoints. In this case, ROOTS should not be able to correctly predict its existence, just like humans observing the true physical world. To show this, we manually select some images from a scene that has one object missing served as context and the rest served as targets for ROOTS to generate. We show the results in Figure 7.

heading: 5.2 QUANTITATIVE EVALUATION
text: NLL and MSE. In this section, we compare the quantitative results of ROOTS and GQN on the negative log-likelihood (NLL) and the mean squared error (MSE) on the test dataset. The goal is to show quantitatively that achieving the object-wise representation in ROOTS does not deteriorate its generation quality in comparison to GQN. We approximate NLL using importance sampling with K = 50 samples and report the image NLL normalized by the number of pixels in Table ??. Both GQN and ROOTS are trained for 120 epochs on each dataset, making sure both ROOTS and GQN have converged. We see that, although learning object-factorized representations, ROOTS achieves NLL comparable to GQN. Due to object-wise representation, ROOTS recovers objects separately, together with an independent background module, ROOTS performs better on MSE than GQN. This benefit becomes clearer as the number of objects in the scene grows.
Object Detection Quality. In this section, we only provide precision and recall results as object detection evaluation of ROOTS as GQN cannot detect objects. To estimate the true positive prediction, we first filter out predictions that are too far away from any ground truth by applying a
radius threshold. The distance is calculated as the euclidean distance between two center points. Then we assign the predicted object to the ground truth object to which it has the nearest distance. Multi-association is not allowed. We normalize the coordinate value to be in the range of [−1, 1] for better interpretability of the applied radius threshold. Accordingly, the average object size is 0.3. We provide the precision and recall result under different thresholds in Table ??. Besides precision and recall, we also provide the counting accuracy of ROOTS.

heading: 6 CONCLUSION
text: We proposed ROOTS, a probabilistic generative model for unsupervised learning of 3D scene representation and rendering. ROOTS can learn object-oriented interpretable and hierarchical 3D scene representation. In experiments, we showed the generation, decomposition, and detection ability of ROOTS. For compositionality and transferability, we also showed that, due to the factorization of structured representation, new scenes can be easily built up by reusing components from 3D scenes. Interesting future directions would be to learn the knowledge of the 3D world in a sequential manner as we humans keep updating and inferring knowledge through time.

heading: A APPENDIX
text: A.1 GENERALIZATION
In this section, we quantitatively analyze ROOTS’s generalization performance on tasks of generation and detection. For the task of generation, we provide a comparison between ROOTS and GQN. Specifically, we train both ROOTS and GQN on our three datasets until full convergence. For each model trained on one dataset, e.g 1-3 objects dataset, we test it on the other two datasets, e.g 2-4 objects dataset, and 3-5 objects dataset. For the task of detection, since GQN only provide scenelevel representation (detection is not possible), we only report detection performance of ROOTS. Generation results (MSE) and detection results (precisin and recall) are shown in Table 3 and Table 4, respectively. Precision and recall are reported with the radius threshold set to 0.15 unit in Mujoco physic world. Here, we describe the trend we find from the experimental observations. When trained on dataset with small (e.g., 1-3) number of objects and tested on large (e.g., 3-5) number of objects, both ROOTS and GQN do not generalize well. This is because, during testing, latent variables are sampled from a learned prior. If during training, the model isn’t provided a chance to see scenes with more objects, it is hard to generalize well. Also, we note that, compared with GQN, ROOTS obtains larger MSE error. Together with observations in the following case, we hypothesize this sensitivity comes from the discrete variable, presenting the existence of an object. While with one vector representing the full scene, GQN has smoother performance. When trained on dataset with a large number of objects while tested on a small number of objects, ROOTS and GQN show a similar property in their performance. They both perform well. But, in this case, ROOTS produces better generations results, which we believe is a benefit from the object-wise generation and a separate background module. While in GQN all information are compressed into one vector. The above conclusion of ROOTS is consistent with its performance on detection task. One interesting finding during this experiment is that ROOTS trained on 3-5 objects dataset achieves better results (lower MSE, higher precision, and recall) than trained on 1-3 objects dataset when testing on 1-3 objects dataset. A similar phenomenon is found when testing on 2-4 objects dataset. This is an interesting finding that is worth more investigation.
A.2 COMPOSING WITH MORE OBJECTS
In this section, we provide one more example of compositionality. We first train ROOTS on 1-3 objects dataset. As in Figure 8, during testing, we collect object representations from 7 different scenes and reuse them to composite a new scene with 9 objects. Additionally, the sampled center position of each object and query viewpoints (represented as camera position, pitch, and roll) fed into ROOTS are with respect to canonical coordinates. Note that, to predict the scale of objects in 2D projection correctly (e.g., for the same object, the larger the distance between it and viewpoint is, the smaller that object should be in the 2D projection), ROOTS learns to infer local depth (position translation) for each object given specific query viewpoint. This can be observed on the yellow sphere, green cylinder and blue cube in the bottom row in Figure 8.
A.3 DETAILS OF COMPONENT MODULES
We first introduce some important building-blocks for implementing our model in this section, and then sketch the implementation steps using modules described here in the following section.
Scene Representation Network: We use Scene Representation Network to implement the order invariant encoder at scene level, frepr-scene(·). This network is modified based on Representation Network in GQN. We adjust kernel size and add a CNN3D layer to make sure the output conv-features fit our needs, shown as in Figure 9. The scene representation network takes <image, viewpoint> pair as input, output a conv-feature map. To implement the order invariant, we take the mean of these outputs from the same scene.
Object Representation Network: We use object Representation Network to implement order invariant encoder at object level, frepr-obj(·). As visualized in Figure 10, we design a branch to pass low level feature to provide richer conv-features. The dimension d in the second input equals to the summation of the dimension of {zpres, sscale, and spos}. The corresponding values are listed in Table 5. The order invanriant is implemented in the same way as in Scene Representation Network.
Convolutional LSTM Cell: We use ConvLSTM as the RNN module in the ConvDRAW module. In ConvLSTM, all fully-connected layers are substituted with convolutional layers. Its one updating step is described as follows:
(hi+1, ci+1)←− ConvLSTM(xi, hi, ci)
where hi is the output of the cell and ci is the recurrent state of the ConvLSTM, xi is the input. Both hi and ci are initialized with zeros at the i = 0 step.
ConvDRAW: We highlight one step ConvDRAW (denoted as l) used in ROOTS here for generative process and inference process separately.
• Generative Process:
(h(l+1)p , c (l+1) p )← ConvLSTMθ(x(l), z(l),h(l)p , c(l)p )
z(l+1) ∼ StatisNet(h(l+1)p )
• Inference Process:
(h(l+1)q , c (l+1) q )← ConvLSTMφ(y,x(l),h(l)p ,h(l)q , c(l)q )
z(l+1) ∼ StatisNet(h(l+1)q )
(h(l+1)p , c (l+1) p )← ConvLSTMθ(e(l), z(l),h(l)p , c(l)p )
where xl is the input at the lth step, y is target reconstruction, z(l+1) is the sampled latent at the lth step. We denote the prior module and posterior module with subscript p and q, respectively. θ and φ are neural network parameters. The StatisNet is described in the following paragraph. In the Renderer network, we replace the StatisNet with a deterministic Decoder.
Sufficient Statistic Networks: The Sufficient Statistic Networks will output sufficient statistics for pre-defined distribution, e.g. µ and σ for Gaussian distribution, given inputs. We list the configuration of all the Sufficient Statistics Networks in Table 5 used during generation. For zposn , z what n and sscalen,i , we use auto-regressive scheme (ConvDraw) to learn the sufficient statistics. For z pres n and spresq,n , we use regular convolutional layers. In the third column, we give the kernel size of the first convolutional layer, the remaining are Conv3D 1X1 layers. If the kernel size is 3, we have one zero paddings, the stride is 1 to keep the spatial size unchanged. The ”Draw Rollouts” column shows the number of DRAW steps we apply. The ”Concat” column specifies how we use sampled latent value for the subsequent process, for example, concatenating zl, for l < L or only taking zL, where L is the rollout steps.
GQN Implementation: We strictly follow the paper for implementation details. The only difference is we enhance the decoder in the renderer by adding two more convolutional layers.
A.4 IMPLEMENTATION DETAILS
Here, we give the details of the implementation of ROOTS. Details of component modules are mentioned in Appendix A.3. We first outline the generation and inference process for one object, indexed with n. Parallelizing it to multiple objects is straightforward.
Generation Process Superscript on ConvDRAW is used to indicate which variable the ConvDRAW module is responsible for. The Renderer module is implemented in the same way as ConvDRAW with one difference that we do not model any variable in Renderer, making it a deterministic decoder. All the ConvDRAW modules have a hidden state size of 128. We use ST denote spatial transformation process and ST −1 denotes the reversed process. K denote the index set of context C and θ and φ are neural network parameters.
r← ∑ k∈K frepr-scene(x k,vk, θ) (Obtain scene-volume feature-map) (5) zposn ∼ ConvDRAW pos θ (rn) (Sample 3D position) (6) zpresn ∼ CONVθ(rn) (Sample global presence) (7) uposk,n ← f3D→2D(z pos n ,v k), k ∈ K (Perspective projection) (8)
uscalek,n ∼ ConvDRAWscaleθ (rn,vk,u pos k,n), k ∈ K (Sample object scale) (9)
xkn ← ST (xk, [u pos k,n,u scale k,n ]), k ∈ K (Crop object patch) (10) rwhatn ← ∑ k∈K frepr-obj(x k n,vk), k ∈ K (Object level encoding) (11)
zwhatn ∼ ConvDRAWwhatθ (rwhatn ) (Sample global what) (12) sposq,n ← f3D→2D(zposn ,vq) (Perspective projection) (13)
sscaleq,n ∼ ConvDRAWscaleθ (rn, sposq,n ,vq) (Sample object scale) (14) spresq,n ∼ CONVθ(zpresn , rwhatn ,vq, sposq,n , sscaleq,n ) (Sample local presence) (15)
x̂qn, α q n ← Rendererθ(zwhatn ,vq) (Decode the object patch and object mask)
(16)
x̂qn ← ST −1(αqn × x̂qn, [sposq,n , sscaleq,n ]) (Generate object patch) (17) αqn ← ST −1(αqn, [s q n pos, sqn
scale]) (18) αoccq,n ← depthn × αqn == min
n (depthn × αqn) (Obtain occlusion mask) (19) x̂qfg ← ∑ n (αoccq,n × x̂qn × spresq,n ) (Generate foreground) (20)
αqfg ← ∑ n (αqn × αoccq,n × spresq,n ) (Generate foreground mask) (21)
zbg ∼ ConvDRAWbgθ (r) (Sample background) (22) x̂qbg ← BgRenderer(z bg,vq) (Decode background) (23)
x̂q ← x̂qfg + (1− α q fg)× x̂ q bg (Render generations) (24)
Inference Process The inference modules are paralleled with generative modules. We only highlight the different part below.
rq ← ∑ q frepr-scene(x q,vq, θ) (Obtain scene-volume feature-map) (25)
zposn ∼ ConvDRAW pos θ,φ(r q n, rn) (Sample 3D position for object n) (26) zpresn ∼ CONVφ(rqn, rn) (Sample global presence) (27) sposq,n ← f3D→2D(zposn ,vq) (Perspective projection) (28)
sscaleq,n ∼ ConvDRAWscaleθ,φ (rn, rqn, sposq,n ,vq) (Sample object scale) (29) xqn ← ST (xq, [sposq,n , sscaleq,n ]) (Crop object patch) (30)
rwhatq,n ← ∑ q frepr-obj(x q n,v q) (Encode object level context) (31)
zwhatn ∼ ConvDRAWwhatθ,φ (rwhatn , rwhatq,n ) (Sample global what) (32) spresq,n ∼ CONVφ(rwhatq,n , rwhatn , sposq,n , sscaleq,n ,vq) (Sample local presence) (33)
zbg ∼ ConvDRAWbgθ,φ(r q, r) (Sample background) (34)
A.5 DATASET DETAILS
The sizes of objects are randomly chosen from 0.56 to 0.66 unit in the Mujoco physic world. Each object is put on the floor of 3D space with a range of [−2, 2] along both x-axis and y-axis. We have three different types of object, cube, sphere, and cylinder with 6 different colors. For each dataset, we first randomly choose the number of objects in a scene, then randomly choose object type, color and their positions (x and y coordinates). For each scene, we have 30 cameras put at a radius of 3, pointing at a squared area located at the center, thus, the camera would not always look at the center point. The pitch is randomly chosen from [−π/6.,−π/7.] and the yaw is randomly chosen from [−π, π]. During training, we randomly divide each scene into context set and query set, where the number of <image, viewpoint> pair in context set is randomly chosen from a range of [10, 20] and the remaining are served as query set.

Review summaries:
The paper proposes a model building off of the generative query network model that takes in as input multiple images, builds a model of the 3D scene, and renders it. This can be trained end to end. The insight of the method is that one can factor the underlying representation into different objects. The system is trained on scenes rendered in mujoco.

The paper presents a framework for 3D representation learning from images of 2D scenes. The proposed architecture, which the authors call ROOTS (Representation of Object-Oriented Three-dimension Scenes), is based on the CGQN (Consistent Generative Query Networks) network. The paper provides 2 modifications. The representation is 1. factorized to differentiate objects and background and 2. hierarchical to first have a view point invariant 3D representation and then a view-point dependent 2D representation. Qualitative and qualitative experiments are performed using the MuJoCo physics simulator [1] (please add citation in the paper).