OBJECT-ORIENTED REPRESENTATION OF 3D SCENES
Authors: Chang Chen
Sungjin Ahn
Venue:  (2020)
Keywords: unsupervised learning
representation learning
3D scene decomposition
3D detection

Abstract: In this paper, we propose a generative model, called ROOTS (Representation of Object-Oriented Three-dimension Scenes), for unsupervised object-wise 3D-scene decomposition and and rendering. For 3D scene modeling, ROOTS bases on the Generative Query Networks (GQN) framework, but unlike GQN, provides object-oriented representation decomposition. The inferred object-representation of ROOTS is 3D in the sense that it is viewpoint invariant as the full scene representation of GQN is so. ROOTS also provides hierarchical object-oriented representation: at 3D global-scene level and at 2D local-image level. We achieve this  without performance degradation. In experiments on datasets of 3D rooms with multiple objects, we demonstrate the above properties by focusing on its abilities for disentanglement, compositionality, and generalization in comparison to GQN.

Under review as a conference paper at ICLR 2020
UNSUPERVISED OBJECT- 3D SCENE REPRESENTATION AND RENDERING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we propose a probabilistic generative model, called ROOTS, for unsupervised learning of object-oriented 3D-scene representation and rendering. ROOTS bases on the Generative Query Network (GQN) framework. However, unlike GQN, ROOTS provides independent, modular, and object-oriented decom- position of the 3D scene representation. In ROOTS, the inferred object-oriented representation is 3D in the sense that it 3D-viewpoint invariant as the scene-level representation of GQN is so. ROOTS also provides hierarchical object-oriented representation: at 3D global-scene level and at 2D local-image level. In experi- ments, demonstrate properties by focusing on its abilities for disentanglement, compositionality trans- ferability, and generalization. ROOTS achieves this without performance degra- dation on generation quality in comparison to GQN.
datasets of 3D rooms with multiple objects, the above
INTRODUCTION
The shortcomings of contemporary deep learning such as interpretability, sample efficiency, ability for reasoning and causal inference, transferability, and compositionality, has traditionally shown its strengths (Garnelo & Shanahan 2019). Thus, one of the grand challenges in machine learning has been make deep learning embrace the benefits of symbolic representation so that symbolic entities can emerge from -igh-dimensional observations such as visual scenes.
where the symbolic AI
In particular, for learning from visual observations of the physical world, such representation should consider the following criteria. First, it should focus on objects (and their relations) which are foun- dational entities constructing the physical world. These can be considered as units on which we can build modular model. The modular nature also helps compositionality (Andreas et al., 2016) and transferability (Kansky al.] (017). Second, being three-dimensional (3D) decisive property of the physical world. We humans, equipped with such 3D representation in our brain (Yamanel ct al. 2008), can retain consistency on the identity of an object even it is observed from different viewpoints. Lastly, learning such representation should be unsupervised. Although there have been remarkable advances in supervised methods to object perception (Redmon et al., 2016 Ren et al.) 2015 Long et al. (2055), the technology should advance toward unsupervised learning as we humans do. This not only avoids expensive labeling efforts but also allows adaptability and flexibility to the evolving goals of various downstream tasks because "objectness" itself can vary on the situation.
In this paper, we propose probabilistic generative model that can learn, without supervision, object- oriented 3D representation of a 3D scene from its partial 2D observations. We call the proposed model ROOTS (Representation of Object-Oriented Three-dimensional Scenes). We base our model on the framework of Generative Query Networks (GQN) (Eslami et al., 2018 Kumar et al. 2018). However, unlike GQN which provides only a scene-level representation that encodes the whole 3D scene into single continuous vector, the scene representation of ROOTS is decomposed into object- wise representations each of which is also an independent, modular, and 3D representation. Further, ROOTS learns to model a background representation separately for the non-object part of the scene. The object-oriented representation of ROOTS is more interpretable, composible, and transferable. Besides. ROOTS provides the two-level hierarchy of the object-oriented representation: one for a global 3D scene and another for local 2D images. This makes the model more interpretable and provides more useful structure for downstream tasks. In experiments. we show the above abilities of ROOTS the 3D--oomm dataset containing images of 3DD rooms with several objects of different

---

Under review as a conference paper at ICLR 2020
colors and shapes. quality compared to GQN.
We also show that these new abilities are achieved without sacrificing generation
Our proposed problem and method significantly different from existing works visual 3D learning although some of those partly tackle some of our challenges. First, our model learns fac- torized object-oriented 3D representations which are independent and modular, from scene taining multiple objects with occlusion and partial observability rather than a single object. Second, our method is unsupervised, not using any 3D structure annotation such as voxels, cloud points, or meshes as well as bounding boxes or segmentation annotation. Third, our model is probabilistic generative model learning both representation and rendering with uncertainty modeling. Lastly, is trained end-to-end. In Section 4 we provide more discussion on the related works.
The main contributions are: (i) We propose, in the GQN framework, a new problem of learning object-oriented 3D representations of 3D scene containing multiple objects with occlusion and partial observability in the challenging setting described above. (ii) We achieve this by proposing a new probabilistic model and neural architecture. (iii) We demonstrate that our model enables various new abilities such as compositionality and transferability while not losing generation quality.
PRELIMINARY: GENERATIVE QUERY NETWORKS
The generative query networks (GQN) (Eslami et al. 2018) is a probabilistic generative latent- variable model providing a framework to learn a 3D representation of a 3D scene. In this framework, an agent navigating a scene collects K images x from 2D viewpoint vř. We refer this collection to context observations C, = =[xx, v=)]k=1....KK While GQN trained on a set of scenes, in the following, we omit the scene index for brevity and discuss single scene without loss of generality. GQN learns scene representation z from context C. The learned representation z of GQN is a 3D- viewpoint invariant representation of the scene in the sense that, given an arbitrary query viewpoint v", its corresponding 2D image x" can be generated from the representation.
In the GQN framework, there are two versions. The standard GQN model (Eslami al. 2018) uses the query viewpoint to generate representation whereas the Consistent GQN (CGQN) (Ku- mar al. 2018) uses the query after generating the scene representation in order to obtain query- independent scene-level representation. Although we use CGQN as our base framework to obtain query-independent scene-level representation, in the rest of the paper we use the abbreviation GQN instead of CGQN to indicate the general GQN framework embracing both GQN and CGQN.
The generative process of GQN is written follows: p(x* v2, C) p(x v9, 2))(2z)))())))))))) C')z. As shown, GQN uses a conditional prior p(z|C) to learn scene representation z from context. To do this, it first obtains a neural scene representation from the representation network r frepr-egn (C) which combines the encodings of (v* x*) C' in an order-invariant way such as sum or mean. It then uses ConvDRAW (Gregor et al., 2016) to generate the scene latent variable z from scene representation by p(z =0== =I p(z Iz " r)) ConyDRAW(r) with L autoregressive rollout steps. Due to intractability of the posterior distribution píz|C, v9, x9). GQN uses variational infer- ence for posterior approximation and the reparameterization trick (Kingma & Welling, (013)) for backpropagation through stochastic variables. The objective is to maximize the following evidence lower bound (ELBO) via gradient-baat optimization.
log v2, C) 2 E (el, log Py (x" v", z)] KL (z C, x²) po(z C))-
Note that although in this paper we use single target observation (x", vo) for brevity, the model is in general trained on set of target observations D {(x}, v))
3 ROOTS: REPRESENTATION OF OBJECT- CRJECT-ORIENTED 3D SCENES
3.1 GENERATIVE PROCESS
The main difference of our model from GQN is that we have a 3D representation per object present in the target 3D space while GQN has single 3D representation compressing the whole 3D space into a vector without object-level decomposition. We begin this modeling by introducing the num- ber of objects M in the target space random variable. Then, we write the representation
2

---

Under review as a conference paper at ICLR 2020
applac
Fn
Object
fan pant buyy
x²
Jeppe-ccnn
Levet
pos
Frepaaii
Mhat
2
pfrul
Rendering
Figure 1: Overview ROOTS. (A) The context observations are fed into an encoder and obtain the scene-volume feature-map. (B) The scene-object latent-map has a cell of Zu = z zz, -z. what for each spatial volume cell of the target environment. (C) After obtaining g 2, we obtain z by finding object patches from context and then clustering them for each object. The resulting representation zn is not a 3D-representation (D) Decoding process p(x" [s)p(s/z, v*). The representation is rearranged according to query viewpoint v by using f3D-20 2 v*) and then projected to target 2D) image. (D), left-bottom is an example of 3D full view. A projection camera is shown on the left corner. On the right, we perform 2 projection steps: (1) converting the coordinate w.r.t. the projection camera (hottom) and (2) projecting onto 2D canvas (up).
prior of ROOTS as p(z, MIC) = P(M M
plz())) To implement such model with variable number of objects, in AIR (Eslami et al. 2016). the authors proposed to use an RNN that rolls out M steps, processing one object per step. However, according to our preliminary investiga- tion (under review) and other works (Crawford & Pineau 2019). it turned out that this approach is computationally inefficient and shows severe performance degradation with growing M.
Object-Factoriin Conditional Prior. To resolve this problem, in ROOTS propose process objects in a spatially local and parallel way instead of sequential processing. This is done by first introducing the voluue feature-map. The volume feature-map map is obtained by encoding context C' into a 3D tensor of N = (H x W x L) cells. Each cell n E f1...., N} is then associated to D-dimensional volume feature r,, RDD. Thus, the actual output of the encoder is 4-dimensional tensor r = Srepe-seem (C). Each volume feature r, represents a local 3D space in the target 3D space in similar way that a feature vector in 2D feature-map of CNN models a local area of an input image. Note, however, that the introduction of the scene-volume feature-map map is not the same as the feature-map of 2D images because, unlike the CNN feature- map for images, the actual 3D target space is not directly observable-it is only observed through a proxy of 2D images. For the detail implementation of the encoder frepr-scenne refer to the Appendix A.3
Given the scene-volume fea,ure-map, for each volume cell n 1...., N but in parallel, we obtain three latent variables (=free, 22000 2/ whaat) Zn from the 3D-object prior model p(z,) r.). We refer this collection of object latent variables z {2,, J-V to the scene-object latent-map as generated from the scene-volume fcature-map r. Here, 2fres is a Bernoulli random variable indicating whether object is associated (present) the volume cell or not, z is 3-dimensional coordinate indicating the position of an object in the target 3D space, and z what is a representation vector for the appearance of the object. We defer a more detail description of the 3D-object prior model p(zn (r.) to the next section. Note that in ROOTS we obtain 2 what as 3D representation which is invariant 3D viewpoints. The position and appearance latents for cell n are defined only when the cell has an associated object to represent, i.e., 2pres = 1. From this modeling using scene- 2/ 2pres and the previous prior model can be written as volume feature-map, we can obtain M Σn
follows: p(z, M C) = p(M CC) m= ((z))
[C)
N
II n=1
p(z, rT,)= = II p(> (r,) [p(z) pos x=1
what Fm, 2
(1)
In addition to allowing spatially parallel and local object processing, another key idea behind intro- ducing a presence variable per volume cell is to reflect the inductive bias of physics: two objects cannot co-exist at the same position. This helps remove the sequential object processing because dealing with object does not need consider other objects if their features are from spatially
3

---

Under review as a conference paper at ICLR 2020
distant areas. Note also that the scene- volume feature-map is not to strictly partition the target 3D space and that the presence variable represents the existence of the center position of an object not the full volume of an object. Thus, information about object can exist across neighboring cells.
Hierarchical Object-Oriented Representation. The object-oriented representations z = {zn} provided by the above prior model is global in the sense that it contains all objects in the whole target 3D space, independently to a query viewpoint. From this global representation and given a query viewpoint, ROOTS generates 2D image corresponding first by learning the view-dependent representation of the target image. In a naive approach, this may be done simply by learning a single vector representation p(z" zz v²) but in this case, we lose important information: the correspondence between a rendered object in the image and global object representation Zn. That is, we cannot track from which object representation 2, an object in the image is rendered. In ROOTS, we resolve this problem by introducing local 21>leevel object- oriented representation layer. This local object-oriented and view-dependent representation allows additional useful structure and more interpretability. This 2D local representation is similar to those in AIR (Eslami et al. 2016) and SPAIR (Crawford & Pineau 2019).
query viewpoint. This is done
Specifically, for for which z zpres = 1, a local object representation s, is generated by conditioning on the global representation set z and the query va. Our local object representation model written as: p(s|z, v9) = I 1 p(sn|z, /z, v9). Similar to the decomposition of zn, local object representa- tion Sn consists of (spree shhe spcule, ( ssaatt. Here, sere indicates whether an object a should be rendered in the target image from the perspective of the query. Thus, even if an object exists in the target 3D space, i.c., 2pree = 1. sares can be set to zero if that object should be invisible from the query viewpoint. Similarly, sh and socale represent respectively the position and scale of object n in the image not in the 3D space, and s.uhat represents the appearance to be rendered into the image (thus not 3D invariant). For more details about how to obtain (s , scale snn syhhatt from z and v4, we describe in the next section. Given = }. we then render to the canvas to obtain the target image psx* Combining all, the generative process of ROOTS is written as follows:
N
p(x*vv9,
p(x* p(..lz v2) II C() dzds.
(2)
n=1
n=1
See Figure forr the overview of the generation process.
3.2 IMPLEMENTATION DETAILS
Global 3D-object prior. The 3D-ojject prior p(z,,Ir,,) generates three latents (zpres, 2 .z, whhhtt as follows. It first obtains the presence latent from zpres Bernoulli(fpe:((.) and the 3-dimension position latent from z ~ [f (r,), ffnn (F,)). Using these two latents, we then obtain the appearance latent 2,, whaat This process is divided into object gathering and object encoding.
For object gathering, for each context image x we attend and crop a patch that corresponds to object n. Specifically, we first notice that using the deterministic camera-coordinate projection function JaD--2D (which we do not learn), we can project from the perspective of a context viewpoint v*, a 3D position z,! in the global 3D coordinate system into 2D position pos in context image x², ). If object should be invisible from the viewpoint v² its projected 2D position u is out of the image x* and do not crop patch. For more details about the projection function faD-2DD refer to WWikieddia (209)) We also predict the bounding box scale kmm = fsatee (u In, v*). Given the center position and scale, we can crop a patch x C x* using the spatial transformer (Jaderberg et al. 2015). Applying this cropping to all context pairs (v& xxxx E C, gather set of object image-patches X,, = (x;) for all n with 2 1.
ie., uk faD->2((2)) (z (
pos
Xn, obtaining object 3D-representation 2 zhahat can be converted to GQN encoding problem. That is, we can simply consider X,, and its corresponding viewpoints as a what new object-level context Ca, and can run order-invariant GQN representation network r Frepe (Cn) and then run ARW to obtain what from pakat
Given object
Local 2D-object prior. The intermediate prior p(s,,zz (z, v²) generates (sares, s socale s s.phat). This is done follows. Similar to what we described in the above to obtain u we use the coordinate projection function fap-12 to find the position of global object 2, in the
4

---

Under review as a conference paper at ICLR 2020
target image from the perspective of query vi, ie., $n fsD-2D(Zz) v9). Thus, model the position a deterministic variable. The scale stgate is also obtained similarly as Sg. described in the above for ugolle but with random sampling. Then, we predict sh knn. Bern( frres Sq, Zn v4)). If object should be visible in the 2D target image, ie., spree 1, we generate the 2D appearance representation sn swhat by using an object-level GQN decoder based on ConvDRAW, i.c., sichat = ConyDRAW(Z,, (2,chat v9). Finally, we have p(sn|z, v²) = p(s/ I v (ss ( z
whhaat, v')p((" spres i sscule, Znn v*).
Rendering to 2D Canvas. A main challenge in rendering the local representation s {s, into the image canvas, i.e., p(x" is deal with occlusion. In ROOTS, this can easily be achieved by noticing (i) that the coordinate conversion (DD=2D(22005 (z!e0 v2) actually converts 3D-coordinate to another 3D-coordinate and (ii) that then the last dimension of the converted coordinate system can be interpreted as the orthogonal projection distance from the viewpoint v to the object's position z,00 We can use this distance as object depth from the observer's perspective. This allows us to sort objects according their depths and render each object accordingly without occlusion. To handle background, ROOTS has an independent module to infer the background separately at image- level. We also found that learning an object-level mask along with the appearance latent s syhat helps segmentation between foreground and background as well as generating less blurry images. More details on related implementation are provided in Appendix A.4
3.3 LEARNING AND INFERENCE
Due to the intractability of the posterior p(z, (C, with D [(v3,x;)1; the target viewpoint- image pairs, we train ROOTS using variational inference with the following posterior approximation (C, D) (z) C, (s), C, D). To compute the gradient w.r.t. the continuous latent variables such as the position and appearance, we use reparameterization trick and for the discrete variables on the presence, we use a continuous relaxation using Gumbel- -Sofmmx trick (Jang al.
2016). Other methods based on the REINFORCE algorithm (Williams 1992, Tucker et al. 2017
Grathwohl et al. 2017) can also be used. Implementation of the approximate posterior q( C. Dj
and q(s|z, 2, C, D) is made easier by sharing the parameters of with that of conditional prior p (C):
we only need to provide additional data D. With D (v x²) for simplicity, the objective is to
maximize the following evidence lower bound (ELBO): L(,, di C, D)
E B3 [log (xx1s)) (g v9, C, || (s|z, ] ( D) po(2C) (3)
Combining with Unconditioned Prior. One difficulty in using the conditional prior is the fact that, because the prior is learned, have less control in reflecting prior knowledge into the prior distribution. In our experiments, it turns out that biasing the posteriors of some variables towards the values of our prior preference is helpful in stabilizing the model. To implement this, in our training. we use the following objective that has additional KL terms between the posterior and unconditioned prior.
L = L=((0,0 C, D) + (Il sscale IC, D) (1, +7KL[ (2 pres IC, D) Geom(p)) + BB
(SPre) ,C, D) Geom(p)] (4)
where zPres_ = {-pre*),, and pos scaaee spres are defined similarly. The and 7 are hyperparam- eters weighting the auxiliary loss terms. We set them 0.999 and during training, respectively. This auxiliary loss can be derived if we replace the conditional prior by the product of expert prior p(z(C)p(z) divided by the posterior q(z C').
RELATED WORKS
Although to our knowledge there has been no previous work on unsupervised and probabilistic generative object-oriented representation learning for 3D scenes containing multiple objects, there has been literature on its 2D problems. The first is the Attend, Infer, Repeat (AIR) model (Eslami et al. 2016). AIR uses spatial transformer (Jaderberg al. 2015) to crop object patch and uses an RNN to sequentially generate next object conditioning on the previous objects. In Crawford & Pineau 2019), the authors showed that this RNN-based rollout is inefficient and can significantly
5

---

Under review as a conference paper at ICLR 2020
degrade performance the number of objects increases. SPAIR is inspired by YOLO (Redmon jet al. 2016), but unlike YOLO, it does not require bounding box labels. The main limitation of SPAIR is to infer the latents sequentially. Neural Expectation Maximization (NEM) (Greff et al. 2017) considers the observed image as a pixel-level mixture of K objects and an image per object generated and combined according to the mixture probability. In Greff et al. (2019), the authors proposed a more efficient version of NEM, called IODINE, using iterative inference (Marino et al.) 2018). In MONET (Burgess et al. 2019), an RNN drawing a scene component at each time step is used. Unlike AIR and SPAIR, the representations in NEM, IODINE, and MONET do not explicitly provide natural disentanglement like presence and pose per object.
There have been plenty amount of remarkable works about visual 3D learning from the computer vision community. However, as pointed in Section 1, the problem setting and proposed model of these works are different from ours in the sense that they are either (i) not decomposing object-wise representations from a scene containing multiple objects (but working mostly on single object cases)
(Wu et al. 2016 Yan et al. 2016 Choy et al. 2016 Kar et al., 2017: Nguyen-nuuo et al. (0019), (ii)
supervised approach (Huang et al. 201 8 Tulsiani et al. 2018 Cheng et al. 2018: Shin et al. 2019
Du et al. 2018), (iii) learn image generation (synthesis) of a 3D scene without scene representation (Sizzmann et al. 2019: Kato & Haradal 2019 Pinheiro et al., 22019] Tulsiani et al., 2017) or learn
scene-representation without learning rendering ability (Zhou et al. 2017 Yu & Wang 2018), or (iv) not end-to-end. Many of the more traditional works from 3D computer vision also relevant to our work but many of them are not based on neural networks and not end-to-end trainable.
5 EXPERIMENTS
We evaluate ROOTS quantitatively and qualitatively. We train ROOTS with the same hyperparame- ters all datasets. We first briefly describe the datasets. For more details of the dataset generation and network architecture, refer to Appendix [A.5] and Appendix [A.4] We use MuloCo, (Todorov) al. 2012), simulate 3D scenes. Specifically, we generate three different datasets of scenes with 1-3 objects, 2-4 objects and 3-5 objects, respectively. We set the image size to 64 x 64 x pixels. For each object, we randomly choose its position, shape, size, and color. Although we put all objects on the floor, we still predict the 3-dimension coordinate values for 2,500 because objects have different sizes. We generate 60K different scenes for each dataset and split them into 50k for training. 5k for validation, and 5k for testing. We use CGQN as the baseline implementation but in the rest of the section use the abbreviation 'GQN' instead of 'CGQN' to indicate the general GQN framework.
5.1 QUALITATIVE EVALUATION
First, we compare the generations between ROOTS with GQN by visualizing several generated samples from the same scene under the same set of query viewpoints. Note that the goal of this experiment is to show that achieving the object-wise representation in ROOTS does not deterio- rate its generation quality in comparison to GQN. As seen in Figure 2 ROOTS generates slightly sharper object boundaries while GQN generates more blurry images. We believe that this is due to the object-wise generation using segmentation masks. Then, to provide further understanding of the advantages of object-orientater representation, we visualize decomposed generations from ROOTS, as shown in Figure A We can see that ROOTS can generate clean background, complete objects, precise foreground, and detailed occlusion mask separately. Furthermore, to demonstrate the newypint-invariant property of the global object representation z what in Figure B, we pro- vide generations of two objects under different query viewpoints. We can see that ROOTS recovers the object with pose corresponding to query viewpoints. GQN cannot provide such decomposed generations because its representation is scene-level where objects are not decomposed.
Object-wise disentanglement. Generations from random viewpoints after changing object posi- tions. In this section and the following sections, we use ROOTS trained on the 2-4 object dataset for demonstration unless otherwise stated. To verify the disentanglement property of our object- oriented representation learned by ROOTS, we carry out the experiment of arbitrarily modifying 2,00 of one object in a scene. As a good disentangled latent representation, this modification should not affect the position, existence, and appearance of other objects concurrent in the scene while handling occlusion properly. More importantly, we should be able to change either the x or y coor- dinate independently and this change should be consistent across viewpoints. To demonstrate this

---

Under review as a conference paper at ICLR 2020
GQN
GQN
Figure 2: Examples of generations from two different scenes together with ground truth placed in the first row for each scene. ROOTS shows hetter generation objects boundary and clearer occlusion, compared with GQN, especially when there object in the
A
B
Figure A: Generated examples for scene decomposition from three different scenes visualized as 2D images captured by a query camera (viewpoint). For example, in the first group of visualization in A, ROOTS can segment scene into the foreground and hackground first and decompose foreground into each individual object further, which show the foreground occlusion mask, which is obtained via each object's 's object and query camera. B: Recovered generations of two objects from global 3D object-wise representation under different query viewpoints. The recovered 2D projection have different pose when seen from different viewpoints.
green sphere, blue cylinder, partially observed cube and a yellow cube. We also
the distance hetween
after changing one dimension of 2 of the red cylinder, shown in Figure we feed this modified 2,00 back to the model and follow the ROOTS generation model in the same way as we do during testing. Generations under 4 different viewpoints shown in Figure 4 We can that ROOTS has a strong ability to learn disentangled representation and occlusions have been handled well due to this advantage.
Compositionality. As stated in earlier sections that a 3D scene can be decomposed into several independent objects, another advantage coming with object-orientated representations is that new scene can be easily built up with selected object components. Thus, by simple combination, we can build novel scenes. To demonstrate this, we first provide ROOTS with three of context images from three different scenes and save the learned object representations Then we swap one object between the first two scenes
for each scene,
(zn0> 2z what) for all with z,00 and add one additional object to the third scene, as shown in Figure 55 We see that the object component learned by ROOTS fully disentangled and can be reused to create new scenes. Also, by adding new object, make scene with 5 objects, which does not exist in the training dataset. Another example shown scene 9 objects is visualized in Figure 6 where we use ROOTS trained on the 1-3 objects dataset. More details about this experiment can be found in Appendix
Partial Observability. We know that the successful generation of ROOTS, even with only partial observations provided, is coming from object gathering across viewpoints. This is because an object is invisible for one viewpoint, ROOTS learn from other viewpoints. Here, want

---

Under review as a conference paper at ICLR 2020
Figure 4: Visualization of changing = and = of the red cylinder in scene through generations from 4 different cameras. We also simulated the 3D scene in the center. Cameras are shown example. We put walls for readers better understand the relative height of cameras. There wall the real dataset.
Original generations from three different scenes, icons the right side highlight ohjects in each scene. Each column shows image captured by one camera. Bottom: Manipulated generations from the same three scenes under the same set of cameras. We switch the green cylinder from scene with the purple sphere from scene 2 and we copy green sphere from scene 2 and put it into scene 3.
Figure 5: TOP:
to push the partial observability to the extreme case, where one object is totally invisible for all context viewpoints. this case, ROOTS should not be able correctly predict its existence, just like humans observing the true physical world. To show this, we manually select some images from a scene that has one object missing served as context and the rest served as targets for ROOTS to generate. We show the results in Figure]) 7
5.2 QUANTITATIVE EVALUATION
NLL and MSE. In this section, we compare the quantitative results of ROOTS and GQN on the negative log-like likeliheeoood (NLL) and the mean squared error (MSE) the test dataset. The goal is to show quantitatively that achieving the object-wise representation in ROOTS does not deteriorate its generation quality in comparison
GQN. We approximate NLL using importance sampling with K = 50 samples and report the image NLL normalized by the number of pixels in Table ?? Both GQN and ROOTS are trained for 120 epochs on each dataset, making sure both ROOTS and GQN have converged. We see that, although learning object-factorizze representations, ROOTS achieves NLL comparable to GQN. Due object-wise representation, ROOTS recovers objects separately, together with independent background module, ROOTS performs better on MSE than GQN. This benefit becomes clearer as the number of objects in the scene grows.
Table 1: Negative log-likelihood and mean squared error
Training set
1-3 objects
2-4 objects
3-5 objects
Metrics
NLL
MSE
NLL
MSE
NLL
MSE
ROOTS
0.9199
15.16
0.9205
25.37
0.9213
29.82
GQN
0.9196
15.28
0.9201
26.86
0.9204
32.66
Object Detection Quality. In this section, only provide precision and recall results as object detection evaluation of ROOTS as GQN cannot detect objects. To estimate the true positive pre- diction, first filter out predictions that far away from any ground truth by applying
8

---

Under review as a conference paper at ICLR 2020
Figure novel consisted of 9 objects composited ROOTS trained on 1-3 objects dataset.
Context
Targets
Generations
Figure 7: Left: Context images provided to ROOTS, we select images that do not have the green sphere projected. Middle: Target images, from which we can see that there is green sphere existed. Right: Generations from ROOTS given context images from the left.
radius threshold. The distance is caleulated the euclidean distance between center points. Then we assign the predicted object to the ground truth object to which it has the nearest distance. Multi-association is allowed. We normalize the coordinate value be in the range of [-1, 1] for better interpretability of the applied radius threshold. Accordingly, the average object size is 0.3. We provide the precision and recall result under different thresholds Table ?? Besides precision and recall, we also provide the counting accuracy of ROOTS.
Table 2: Precision and Recall
Training
1-3 objects
2-4 objects
3-5 objects
Threshold 0.1
0.15
0.35
0.1
0.15
0.35
0.1
0.15
0.35
Precision
76.82
91.56
98.68
66.76
86.07
98.29
66.64
86.09
96.65
Recall
75.56
90.15
97.40
65.53
84.57
95.93
66.91
86.47
97.10
Count Acc.
93.18
88.16
82.21
CONCLUSION
We proposed ROOTS, a probabilistic generative model for unsupervised learning of 3D scene rep- resentation and rendering. ROOTS can learn object-oriented interpretable and hierarchical 3D scene representation. In experiments, showed the generation, decomposition, and detection ability of ROOTS, For compositionality and transferability, we also showed that, due to the factorization of structured representation, Interesting future directions would be to learn the knowledge of the 3D world in sequential manner we humans keep updating and inferring knowledge through time.
scenes be easily built up reusing components from 3D scenes.
REFERENCES
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 39-48, 2016.
Christopher P Burgess, Loic Matthey, Nicholas Watters. Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint arXiv: /9399 2019.
Ricson Cheng, Ziyan Wang, and Katerina Fragkiadaki. Geometry- aware recurrent neural networks for active visual recognition. Advances in Neural Information Processing Systems, pp. 5081 5091, 2018.

---

Under review as a conference paper at ICLR 2020
Christopher B Choy, Danfei Xu, Jun Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view object reconstruction. In European conference on computer vision, pp. 628 644. Springer, 2016.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In Proceedings of AAAI, 2019.
Yilun Du, Zhijian Liu, Hector Basevi, Ales Leonardis, Bill Freeman, Josh Tenenbaum, and Jia- jun Wu. Learning exploit stability for 3d scene parsing. In Advances in Neural Information Processing Systems, pp. 1726-1736, 2018.
SM Ali Eslami. Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225-3233 2016.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei Rusu, Ivo Danihelka, Karol Gregor, David Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloc Hillier, Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, (200 1210, 2018.
Marta Garnelo and Murray Shanahan. Reconciling deep learning with symbolic artificial intelli- gence: representing objects and relations. Current Opinion in Behavioral Sciences, 29:17-23, 2019.
Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, and David Duvenaud. Backpropagation through the void: Optimizing control variates for black-box gradient estimation. arXiv preprint arXiv:1711.00123 2017.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2017.
Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv:1903.0000 2019.
Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual compression. Advances In Neural Information Processing Systems, pp. 3549-3557, 2016.
Siyuan Huang, Siyuan Qi. Yinxue Xiao, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Cooper- ative holistic scene understanding: Unifying 3d object, layout, and camera pose estimation. In Advances in Neural Information Processing Systems, pp. 207-218, 2018.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In Advances in neural information processing systems, pp. 2017-2025, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-sofftaa arXiv preprint arXiv:1611.00 044,,111,, 2016.
Ken Kansky, Tom Silver, David A Mély, Mohamed Eldawy, Miguel Lázaro-Gredilla, Xinghua Lou, Nimrod Dorfman, Szymon Sidor, Scott Phoenix, and Dileep George. Schema networks: Zero- shot transfer with a generative causal model of intuitive physics. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1809-1818. JMLR. org, 2017.
Abhishek Kar, Christian Häne, and Jitendra Malik. Learning multi-view stereo machine. In Advances in neural information processing systems, pp. 365-376, 2017.
Hiroharu Kato and Tatsuya Harada. Learning view priors for single-view 3d reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9778- 2019. 2019.
Diederik Kingma and Max Welling, 1312v11312.61 2013.
Auto-encoding variational bayes.
arXiv preprint
10

---

Under review as a conference paper at ICLR 2020
Ananya Kumar, SM Eslami, Danilo J Rezende, Marta Gareelo, Fabio Viola, Edward Lockhart, and Murray Shanahan. Consistent generative query networks. arXiv preprint arXiv: 180.:000070033, 2018.
Jonathan Long, Evan Shelhamer, and Trevor Darrll. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3131-3344444 2015.
Joseph Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. arXiv preprint arXiv:187.003350 2018.
Thu Nguyen-Phuoe, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Holo- gan: Unsupervised learning of 3d representations from natural images. arXiv preprint arXiv:1904.01322 2019.
Pedro O Pinheiro, Negar Rostamzadeh, and Sungjin Ahn. Domain-adaptin single-view 3d recon- struction. In Proceedings of the IEEE International Conference Computer Vision, pp. 7638- 7647, 2019.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. In Advances neural information processing systems, pp. 91-99.999...0. 2015.
Daeyun Shin, Zhile Ren, Erik Sudderth, and Charless C Fowlkes. 3d scene reconstruction with multi-layer depth and epipolar transformers. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2172-2182, 2019.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2437-2446, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
George Tucker, Andriy Mnih, Chris Maddison, John Lawson, and Jascha Sohl-Dickstein. Rebar: Low-variance, unbiased gradient estimates for discrete latent variable models. In Advances in Neural Information Processing Systems, pp. 2627-2636, 2017.
Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2626-2634, 2017.
Shubham Tulsiani, Saurabh Gupta, David F Fouhey, Alexei A Efros, and Jitendra Malik. Factoring shape, pose, and layout from the 2d image of a 3d scene. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 302-310, 2018
Wikipedia. 3D projection Wikipedia, the free encyclopedia, 2019. URL http://en.
wikipedia. http:///www.neexepppppteeeeeee-/ [Online; accessed 14-November-2019].
Ronald J Williams. Simple statistical follwin algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-22) 1992.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning prob- modeling. In Advances in
abilistic latent space of object shapes via 3d neural information processing systems, pp. 82-90, 2016.

---

Under review as a conference paper at ICLR 2020
Yukako Yamane, Eric T Carlson, Katherine C Bowman, Zhihong Wang, and Charles E Connor. A neural code for three-dimensional object shape in macaque inferotemporal cortex. Nature neuroscience, 11(11):1352, 2008.
Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In Advances in Neural Information Processing Systems. pp. 1696-1704, 2016.
Chong Yu and Young Wang. 3d-scene-gan: Three-dimensional scene reconstruction with generative adversarial networks. In ICLR (Workshop), 2018.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851-1858 2017.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 GENERALIZATION
In this section, we quantitatively analyze ROOTS's generalization performance on tasks of genera- tion and detection. For the task of generation, provide comparison between ROOTS and GQN. Specifically, we train both ROOTS and GQN on our three datasets until full convergence. For each model trained on one dataset, e-g 1-3 objects dataset, objects dataset, and 3-5 objects dataset. For the task of detection, since GQN only provide scene- level representation (detection is not possible), we only report detection performance of ROOTS. Generation results (MSE) and detection results (precisin and recall) are shown in Table 3 and Table respectively. Precision and recall are reported with the radius threshold set to 0.15 unit in Mu- joco physic world. Here, we describe the trend we find from the experimental observations. When trained on dataset with small (e.g., 1-3) number of objects and tested on large (e.g.. 3-5) num- ber of objects, both ROOTS and GQN do not generalize well. This is because, during testing, latent variables are sampled from a learned prior. during training, the model isn't provided chance to scenes with more objects, it is hard to generalize well. Also, note that, compared with GQN, ROOTS obtains larger MSE error. Together with observations in the following case, we hypothesize this sensitivity comes from the discrete variable, presenting the existence of object. While with one vector representing the full scene, GQN has smoother performance. When trained on dataset with large number of objects while tested show a similar property in their performance. They both perform well. But, in this case, ROOTS produces better generations results, which believe and a separate background module. While in GQN all information are compressed into one vector. The above conclusion of ROOTS is consistent with its performance on detection task. One inter- esting finding during this experiment is that ROOTS trained on 3-5 objects dataset achieves better results (lower MSE, higher precision, and recall) than trained on 1-3 objects dataset when testing on 1-3 objects dataset. A similar phenomenon is found when testing on 2-4 objects dataset. This is an interesting finding that worth more investigation.
test it on the other two datasets, e-g 2-4
small number of objects, ROOTS and GQN
benefit from the object-wise generation
Table Quantitative generalization results on generation
Training sel
1.3 Objects
Objects 2.4 Objects
Objects 2.4 Objects
Testing set
1.3 Objects
2-4 Objects
3-5 Objects
1-3 Objects
3-5 Objects
1-3 Objects
3-5 Objects
GQN ROOTS
15.37
27.49
43.57
16.44
26.77 25.35
3766 42.21
15.02
23.73 19.62
33.38
14.72
35.22
71.43
13.97
11.48
29.88
Table 4: Quantitative generalization results on object detection
Training set
1-3 Objects
Objects
3-5 Objects
Testing set
1-3 Objects
2-4 Objects
3-5 Objects
1-3 Objects
2.4 Objects
3-5 Objects
1-3 Objects
2.4 Objects
3.5 Objects
Precision
91.43
83,25
71.81
90.56
86.55
80.83
93.24
90.10
85.91
Recall
89.84
80.02
65.67
89.22
84.68
78.29
93.66
89.94
85.91
Count Acc.
93.07
57.91
93.17
87.42
82.17
A.2 COMPOSING WITH MORE OBJECTS
In this section, we provide one more example of compositionality. We first train ROOTS on 1-3 objects dataset. As in Figure R during testing, we collect object representations from different scenes and reuse them to composite a new scene with 9 objects. Additionally, the sampled center position of each object and query viewpoints (represented as camera position, pitch, and roll) fed into ROOTS are with respect to canonical coordinates. Note that, to predict the scale of objects in 2D projection correctly (e.g., for the same object, the larger the distance between it and viewpoint is, the smaller that object should be in the 2D projection). ROOTS learns to infer local depth (position translation) for each object given specific query viewpoint. This can be observed on the yellow sphere, green cylinder and blue cube in the bottom row in Figure g
13

---

Under review as a conference paper at ICLR 2020
Figure 8: TOP: Target images taken by several query vicwpoints from different scenes with icons on the right hand-side highlighting the objects in the seene. Bottom: New scene created with objects collected from the above 9 scenes.
A.3 DETAILS OF COMPONENT MODULES
We first introduce some important building-blocks for implementing our model in this section, and then sketch the implementation steps using modules described here in the following section.
Scene Representation Network: We use Scene Representation Network to implement the order invariant encoder scene level, frepesscene( (-). This network is modified based Representation Net- work in GQN. We adjust kernel size and add CNN3D layer to make sure the output contufeatures fit our needs, shown as Figure 9 The scene representation network takes <image, viewpoint>> pair as input, output a conv-feature map. To implement the order invariant, we take the mean of these outputs from the same scene.
64x84x3
22x22x256
22x22x120
8x8x(256-7)
4scxx128
2x2x258
1x1x258
3x1x3x256
kx33.1.3)
p=]
p-2
p=11.0.1.00
P.11.0.1.00
p=O
S-1
x
prd
1xtx7
Bxbx7
Figure Scene Representation Network
Object Representation Network: We use object Representation Network to implement order in- variant encoder at object level, (.)) As visualized in Figure I0 we design a branch to pass low level feature to provide richer conv-features. The dimension in the second input equals the summation of the dimension of {zrres soaue, and goos). The corresponding values are listed in 3 The order invanriant is implemented in the same way as Scene Representation Network.
Convolutional LSTM Cell:
We use ConvLSTM as the RNN module in the ConvDRAW module. Conv LSMM, all fully-connected layers are substituted with convolutional layers. Its one updating step is described as follows:
(h4+1, G+11) ConvLSTM(X,,
14

---

Under review as a conference paper at ICLR 2020
32x32x3
8x8x(128++1284)) 128+d)
3x3x256
1x1x256
8=5
Sm2
s=1
p=2
x
16x16x128
8x8x122 128
k=5
le=3
Sm]
p=2
p=1
1xlxd
8x8xd
Figure 10: Object Representation Network
where h; is the output of the cell and o; is the recurrent state of the ConvLSTTM, x; is the input. Both h2 and O are initialized with zeros the step.
DRAYDRAW: We highlight one step ConvDRAW (denoted as 1) used in ROOTS here for generative process and inference process separately. Generative Process:
(h (1+1), p
of (((1)) t ConvLSTM, (x, (2 22 hi" c 2 ((11) StatisNet(hh+11) (1+1))
Inference Process:
(h (++1) (++1)) ConvLSTM, (y, y, ( (1) h cf)
h,
z (1+1)
StatisNet((h(t.) (1+1))
(h ((+1) (1+1)) ConVLSTMa h (hh c!!
cp
where x' is the input the 4n step, y is target reconstruction, (1+1) is the sampled latent the Ith step. We denote the prior module and posterior module with subscript and q, respectively. and Φi are neural network parameters. The StatisNet described in the following paragraph. In the Renderer network, we replace the StatisNet with deterministic Decoder.
Sufficient Statistic Networks: The Sufficient Statistic Networks will output sufficient statistics for pre-defined distribution, e.g. μl and for Gaussian distribution, given inputs. We list the configura- tion of all the Sufficient Statisties Networks in Table used during generation. For 2 zn whoot and Fn. spres
we use auto-regressive scheme (ConvDrave to learn the sufficient statistics. For 2 and q.n we use regular convolutional layers. the third column, we give the kernel size of the first convolutional layer, the remaining are Conv3D IX1 layers. If the kernel size is 3, we have one zero paddings, the stride 1 to keep the spatial size unchanged. The "Draw Rollouts" column shows the number of DRAW steps we apply. The "Concat" column specifies how we use sampled latent value for the subsequent process, for example, concatenating 2 for I < L or only taking 24, where is the rollout steps.
GQN Implementation: strictly follow the paper for implementation details. The only difference enhance the decoder in the renderer by adding two more convolutional layers.
A.4 IMPLEMENTATION DETAILS
Here, give the details of the implementation of ROOTS. Details of component modules mentioned in Appendix A.3 We first outline the generation and inference process for one object, indexed with Parallelizing multiple objects is straightforward.
15

---

Under review as a conference paper at ICLR 2020
Table 5: Configuration of Sufficient Statistic Networks
Latent Variable
Channel Numbers
Draw Rollouts
Concate last step
2
[128, 128, 64, 32, 3]
2
[128, 128, 64, 32, 4]
4
concatenate
9.n
[128, 128, 64, 32, 2]
4
last step
apres
[256, 256, 128, 64, 1]
3
gPres
[271, 256, 128, 64, 32, 1]
9,7
z"
[128, 4]
2 last step
Generation Process Superscript on ConvDRAW used to indicate which variable the Con- vDRAW module is responsible for. The Renderer module is implemented in the same way as ConvDRAW with one difference that we do not model any variable in Renderer, making it de- terministic decoder. All the ConvDRAW modules have hidden state size of 128. We use ST denote spatial transformation process and ST denotes the reversed process. K denote the index set of context C and 0 and Φ neural network parameters.
r+
Σ Seepr-sceeee (x"; v
(Obtain scene-volume feature-map)
(5)
KEK
205
COnVDRAW (rn)
(Sample 3D position)
(6)
pres CONV (r,)
(Sample global presence)
(7)
2n
+ faD-20 n v ) E K k
pos
(Perspective projection)
(8)
u
knn
scale
C (Ta, v", u K
(Sample object scale)
(9)
ST(x", [ugea ugall)) k K
(Crop object patch)
(10)
n
what
Σ Seepr-oby (X k
(Object level encoding)
(11)
KEK
2,
what
ConNDRAWghch (rwhatt
(Sample global what) (Perspective projection)
(12)
pos
faD-2D (2p v²)
(13)
q.
8
scale
ConvDRAW (In g,n. v9)
(Sample object scale)
(14)
pres
CONV e pres .ra what v sooe g.n
scale)
(Sample local presence)
(15)
q,n
8 a
what v?) + Rendererg (2
(Decode the object patch and object mask)
(16)
x - - x x lsp s
(Generate object patch)
(17)
0 + ST (a pos .8 )
(18)
00c + depthn x ag = min (depth,, ag)
(Obtain occlusion mask)
(19)
a.m
x9
Σ (a q.n
2 s
(Generate foreground)
(20)
a1
Σ (aa x agn x sTre))
(Generate foreground mask)
(21)
2 ConYDRAW bo (r)
(Sample background)
(22)
xg +- BgRenderer vd)
(Decode background)
(23)
" + 22 + (1 ag/ x x9
(Render generations)
(24)
Inference Process The inference modules are paralleled with generative modules. We only high-
light the different part below.
16

---

Under review as a conference paper at ICLR 2020
r + Σ repe scene (x", v2, v2,0) 0)
(Obtain scene-volume feature-map) (25)
200
ConvDRAW (I In)
(Sample 3D position for object n)
(26)
"
pres
CONV o ( Fn)
(Sample global presence)
(27)
+ f3D 2 v²)
(Perspective projection)
(28)
scale
Con seale (Ins r spos v²)
(Sample object scale)
(29)
q.
x
t STxx,, (sgon (
(Crop object patch)
(30)
what
Σ frepe-oby (2%) v9)
(Encode object level context)
(31)
9,7
2 what
DRAWDha 0,0 (Ia (rithat, whhhet rgg CONV (r what what scale ,. Sqn v!)
(Sample global what)
(32)
pres
(Sample local presence)
(33)
q.
29
ConvDRAW 8.0 (r
(Sample background)
(34)
A.5 DATASET DETAILS
The sizes of objects are randomly chosen from 0.56 to 0.66 unit the Mujoco physic world. Each object is put the floor of 3D space with range of [-2, 2] along both x-axis and y-axis. We have three different types of object, cube, sphere, and cylinder with 6 different colors. For each dataset, first randomly choose the number of objects in scene, then randomly choose object type, color and their positions (æ: and coordinates). For each scene, we have 30 cameras put at a radius of 3, pointing at squared area located at the center, thus, the camera would not always look at the center point. The pitch is randomly chosen from i-n/6., - and the yaw is randomly chosen from [- J.. During training, we randomly divide each scene into context set and query set, where the number of <image, viewpoint>> pair in context set is randomly chosen from range of [10, 20) and the remaining are served as query set.
17


Reviews:
[{"review_id": "HklkZ4gVqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a framework for 3D representation learning from images of 2D scenes. The proposed architecture, which the authors call ROOTS (Representation of Object-Oriented Three-dimension Scenes), is based on the CGQN (Consistent Generative Query Networks) network. The paper provides 2 modifications. The representation is 1. factorized to differentiate objects and background and 2. hierarchical to first have a view point invariant 3D representation and then a view-point dependent 2D representation. Qualitative and qualitative experiments are performed using the MuJoCo physics simulator [1] (please add citation in the paper).  \n\n[1]Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based\ncontrol. In ICIRS, 2012.\n\n+Learning 3D representations from 2D images is an important problem. \n+The proposed methodology learns representations that are more interpretable, with higher compositionally. \n\nWhile the paper takes a step towards a potentially impactful work, I cannot recommend it for publication in its current form. \n\n1. There are claims in the paper that are not supported by the experiments. For example, “As seen in Figure 2, ROOTS has clearer generations than GQN. ” However, Figure 2 does not show this at all. It shows no difference between ROOTS and GQN. \n\n2. The paper can benefit from further clarity throughout—in general it seems a bit rushed. For example, the caption on Figure 3 reads “For example, GoodNet segments a scene into foreground and background first and decompose foreground into each individual object further…” The text has not discussed what GoodNet is. Its also unclear what is depicted in each of the columns in Figure 3. This should be clearly explained. \n\n3. I suggest clarifying Figure 1 further and referring to it in section 3. Its currently not referred in the text although it is an overview of the proposed architecture. \n\n\nOther comments\n-Table 2: Why not have a precision-recall curve (as is standard) and report average precision numbers?\n-Table 2: Why not compare to CGQN?\n\nMinor\n-There are typos throughout the text. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByxbeCd0tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a model building off of the generative query network model that takes in as input multiple images, builds a model of the 3D scene, and renders it. This can be trained end to end. The insight of the method is that one can factor the underlying representation into different objects. The system is trained on scenes rendered in mujoco.\n\nSummary of positives: \n+The factoring makes sense, and the use of voxels + the physical property to enforce that two objects can't be superimposed in z_pres is a good strategy.\n+There are a number of good qualitative results for understanding the learned object-oriented representation.\n+The approach of learning 3D representations by comparing projections to observations is a good direction.\n\nSummary of negatives:\n- The method is quite complex and explained, in my view poorly (although I'm open to the other reviewers' opinion on the matter).\n- The experiments are weak\n- The manuscript makes fairly broad claims that aren't substantiated and ignores a great deal of work in the vision community on this topic. \n\nGiven the three largely orthogonal and fairly strong negatives, I lean heavily towards rejecting this paper. Independently each of these is an issue that would be push me to at least lean towards rejection. However, I would encourage a revision and resubmission with improved method explanation, stronger experiments, and a clearer picture with respect to existing work.\n\nIn more detail:\n\nMethod: \n-I found the method section quite difficult to read, in part because the method is quite complex. This isn't intrinsically a bad thing, but complex methods with lots of steps should come with few surprises and descriptions that make the method accessible. In particular, the method section would benefit from a stronger figure that in part introduces the notation, as well as a little more thought in terms of the introduction of the method. A few instances: \n1) \"This is done by an order-invariant object encoder r_n = f_{order-invar-obj}(...)\". One turns to the appendix, and tries to find this function. It's not explicitly there -- instead you need to match r_{n,C} = \\sum_{i} .... , then look up above at the note that \"ObjectRepNet is the module we use for object level order invariant encoder\", then remember that sum is order invariant. \n2) I searched throughout the paper and couldn't find precisely what model f3d->2d was. The figure suggests a projective camera and the text says \"For more details on the coordinate projection, refer to Appendix.\", but there's none in the appendix as far as I can see. \n3) STN-INV is nowhere defined -- inverse spatial transformer? \n4) s^{what} doesn't appear to be anywhere in the appendix -- is s^{what} factored into y^att and alpha^{att}? By matching the RHS, this seems to be the only possibility, but in the main body it's called ConvDRAW aka the GQN decoder, but in the appendix it's called Renderer.   \n5) There are lots of other little things -- e.g., a figure that refers to a variable that doesn't exist (see small details section )\n\nI don't see why a paper this complex can't be accepted at ICLR, but I think at a minimum, the appendix should be more complete so that things are well-defined. I'm open to the possibility that I may just be slow so long as the other reviewers think the paper is crystal clear down to the details. However, I think even if I'm just the slow one, the authors should think about writing this more clearly and using consistent notation and function names. \n\nExperiments:\n-As far as I can see, the difference between ROOTS and GQN is that GQN is a little more blurry in its output (Figure 2) and ROOTS has a slightly better MSE for lots of objects (Table 1) but produces NLLs similar to GQN. There are a few issues with this:\n(a) It's surprising that the correlation between larger numbers of objects and better MSE isn't really investigated -- why not show that GQN breaks at some point? The differences right now are fairly small, and I think the paper ought to delve into details to demonstrate that the differences are important.\n(b) There are so many changes between ROOTS and GQN that I don't know if this has to do with the object-bits of it, or something else. This is part of a larger problem where there are no ablations. A large complex system is trained, and lots of changes are made to GQN. But when there are no ablations, it's unclear what parts of the changes are important and which parts aren't.\n(c) It's not clear whether the GQN and ROOTS are being trained fairly -- do they have the same capacity? Why are they both trained for the same number of epochs? It seems entirely likely that ROOTS may train faster than GQN (or the reverse!). If there's only one experiment like this, why not train for a long enough time to ensure convergence and then take the checkpoint with best validation performance? \n-The NLL results are very weak and probably not worth putting in, at least without some sort of explanation for why this gap is significant.\n-The object detection quality experiment is incomplete -- I just do not know how to parse the numbers that are presented without some sort of simple baseline in order to make sense of things. Why not also try something like this on GQN? \n-The qualitative experiments are nice but would be substantially improved by showing that:(a) that GQN doesn't do any of these (b) that ROOTs can train on 2-3 objects and test on 3-5 objects by simply changing the prior on K -- this is one of the primary advantages of object-centric representations of scenes (the ability to handle arbitrary numbers of objects). \n\n\nRelated work:\n\nThe paper really needs to make its claims more specific and position itself better with respect to related work. \n\nTwo gratuitous examples: \n1) The title is \"Object-oriented representation of 3D scenes\", which covers decades of work in robotics and vision. This title should be changed. \n\n2) \"First unsupervised model that can identify objects in a 3D scene\" is exceptionally broad: voxel segmentation is already a standard feature in point cloud libraries (e.g., \"Voxel Cloud Connectivity Segmentation - Supervoxels for Point Clouds\" Papon et al CVPR 2013). Is the manuscript and the Papon paper the same at all? No. But are they both unsupervised models that can identify objects in scenes. I'm not demanding that the authors write out a claim of novelty that's like a patent, but claiming \"first unsupervised model that can identify objects in a 3D scene\" is, in my opinion, clearly incorrect and needs to be qualified.\n\n\nThe paper should also better position itself compared to the wide variety of work that's been done on unsupervised 3D shape estimation/feature learning using reprojection. For instance (among many): \n(1) Geometry-Aware Recurrent Neural Networks for Active Visual Recognition. Cheng et al. NeurIPS 2018\n(2) Learning a multi-view stereo machine. Kar et al. NeurIPS 2017.\n(3) Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency. Tulsiani et al. 2017.(4) Unsupervised Learning of Depth and Ego-Motion from Video. Zhou et al. CVPR 2017 (not voxels, but 2.5D or a form of 3D)\n\nas well as the vast array of work on 3D reconstruction, including work that is object-oriented\n(1) Learning to Exploit Stability for 3D Scene Parsing. Du et al. NeurIPS 2018.\n(2) Cooperative Holistic Scene Understanding: Unifying3D Object, Layout, and Camera Pose Estimation. Huang et al. NeurIPS 2018\n(3) Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene, Tulsiani et al. CVPR 2018\n(4) Potentially not out at ICLR submission deadline, but 3D Scene Reconstruction with Multi-layer Depth and Epipolar Transformers. Shin et al. ICCV 2019.\n\nI agree that there are differences between these works and the manuscript, but it's really peculiar to work on inferring a 3D volume of scenes from a 2D image or set of images, and only cite YOLO, faster RCNN, and FCNs from the world of CVPR/ICCV/ECCV etc where this work is done very frequently. These works do indeed sometimes rely on a bit more supervision (but not always). But they're tested on data that's far more complex than a set of spheres and cubes.\n\n\n\nSmall issues that do not affect my score.\n- The claim that the method is unsupervised when it has access to precise camera poses seems a bit like a stretch to me. It's common enough that I've given up quibbling about it. Peoples' sense of distance is not exact. This deserves some further thought.\n-The authors should go through and double check their use of GQN and CGQN -- it's said at the beginning that GQN just means CGQN, but then it's occasionally dropped (e.g., right before Table 1)\n- Fig 1 shows z^{where}, which I guess got renamed?\n- \"The Scene Representation Network is modified bases on Representation Network in GQN.\" -> this sentence is presumably a typo/cut off halfway through.\n- \" This helps remove the sequential object processing as dealing with an object does not need to consider other objects if the features are already from spatially distant areas.\" -> this is unclear\n- Eqn 11 appendix \"sacle\" -> \"scale\"\n- \"Object Representation Network\" in A.2 \"objcet\" -> \"object\"\n-Equation 15 -- the parentheses put i \\in \\mathcal{I}(D) inside the Renderer, which is presumably not true.\n-Table 1 -- table captions go on top\n\n\n\n\n\n\n\n\n----------------------------------------------------\nPost-rebuttal Update: \n\nAC: I would give a rating of 5 if the full revision is considered acceptable (since the paper is more clear), and increase to 4 if it is not (since there are some more experiments, although I think they are still quite weak). \n\nI'm still inclined to reject the paper on the grounds of experimental comparisons and the open question of whether  but recognize that my concerns are ones which may not be shared by all communities (and that this is not my community).\n1) Ablations/Comparisons to GQN: This may just be a cultural thing, but I'm puzzled by the claims that certain things (direct comparisons to GQN on feature representation, and ablations) don't need to be empirically done. \n\nIn my view, results really need to be empirically demonstrated. Simply stating that ROOTS has the capacity to decompose things into 3D and GQN doesn't have this built-in isn't enough. GQN has a feature vector, and it would not be surprising if it implicitly already did some of this. There have been far too many recent results in ML where a complex method is presented and it is asserted this complex method is necessary, followed by work that shows that  simple method does as well, typically due to issues in the dataset. I don't think expecting linear readouts of systems is too much of a burden to ask, if only to put the proposed work in context. \n\nIndeed: the experiments in the revision show that ROOTS often does *worse* in generalization performance to previously unseen objects (improving in only 6/9). This is surprising -- if GQN is supposed to break, why doesn't it break here? I appreciate the author's response that there's a latent variable of # of objects that needs to be adjusted in the case of ROOTs, but this should be investigated.\n\nThe same thing goes with the claim that an ablation study is only necessary for improving results. This is just baffling -- is it possible that only certain parts of the method are necessary? Surely this is a problem that is worth studying. What if it's just that some aspect of the system has higher capacity than the equivalent in GQN and just works better?\n\nAC: I realize that I'm just the cranky computer vision person shouting about numbers and I may be out of my element here, so take this as you want. But in my view, things really need to be evaluated (since vision struggled for many years because people showed a few nice qualitative results and didn't put their ideas to the test).\n\n2) Clarity: I completely disagree with the authors that clarity issues were minor -- they really weren't and all reviewers agreed on this. Typos are thing lke tihs that are easy to read through without thinking. These were things that required thinking, flipping back-and-forth-etc. \n\nThe revision appears to be close to a full-rewrite, to the point where the diff system is useless -- it's all red/green. It appears to be clearer, but I haven't checked thoroughly. The ICLR 2020 guide is unclear how you should treat this (the AC guide says \"you can ignore this revision if it is substantially different from the original version.\"). Personally, I don't think it's fair to authors who spent their time making their paper clear in the first place rather than on new results.\n\n\n\nSmaller stuff:\n\n-The authors have misunderstood my statement on paper complexity (although I now realize the comment has been edited)-- my point is that people who present a complex system have a strong obligation to present a clear explanation (since there's little opportunity for redundancy in explanation unlike a simple approach). \n\n-\"AP interpretable metric without a relative comparison\": this is just not true although openreview is probably not the place to litigate this and I recognize that this is my outlook. Accuracy is highly interpretable: 90% top-1 accuracy on mnist would have been boring in 2005, and 90% top-1 accuracy on imagenet would be very exciting today. \n\n-f_{3D-2D} There are multiple camera models. Skimming the revision suggests it's perspective projection, but the authors should realize that there are others (orthographic, weak perspective, etc) and they're often used because they're easier to learn with.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byl0r3K0Yr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "TLDR: Interesting idea that seems promising, but lacks the maturity required to pass the ICLR bar: Lacks proper citations, comparisons with the latest works, no ablation study of their contributions.\n\nThe paper presents an extension of the Generative Query Network to incorporates 1) 3D grid for the state representations 2) hierarchical representation and 3) unsupervised model for explicit object representation (which is tied to 1.\n\nThe unsupervised representation is interesting, but this is a minor contribution on top of GCN and 3D representations have been widely studied in the vision community.\n \nAlso, except for the 3D representation, I am not sure how much the hierarchical representation helps. This leads to the question of why the authors did not perform ablation studies on each component.\n\nFinally, it seems that the authors did not add proper citations. First, the 3D representation has been studied widely in the vision community and 3D-R2N2, ECCV'16 proposed using an RNN to encode a series of 2D images to learn 3D grid representation which seems quite similar to what the authors are proposing as an encoder and representation. Secondly, there are numerous methods on 3D neural rendering such as DeepVoxels, CVPR19 and all of the baselines in their experiments. The paper seems to completely ignore the works in this field.\n\n\nQuestions:\n\nPlease point out the equation number of implementation of Eq.2 in the appendix.\n\nIs the y in Eq.3 the same y defined in the preliminary? Also, consider defining y again. There are almost two pages between the definition and Eq.3.\n\n\nMinor:\nWhy learn the camera projection? $f_{3D\\rightarrow 2D}$? Isn't this deterministic using a camera matrix?\n\nIs the lighting fixed throughout the training and testing?\n\nSec.2 first sentence: an query --> a query", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1ev9InFoB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\nThis paper is certainly very interesting, unlike other papers and makes some very solid contributions. The qualitative results are very impressive. Unfortunately, the paper is very poorly written. If the authors can address the issues below and improve the quality of the writing, I would recommend that this paper be accepted. However, in its current state, I recommend that this paper be rejected.\n\nMajor:\nThe claim that “This is the first unsupervised model that can identify objects in a 3D scene” is not true. There is MONet and Iodine that can identify objects without supervision in 2D projections of 3D scenes. This claim should be revised.\n\nWhat is r_C in p(z|c, r_C)?\n\nThe authors are using non-standard GQN notation and their notion is not consistent. The authors should make their notation consistent or use the standard GQN notation. Section two would not make sense to people that are not already familiar with GQN.\n\nThe scene volume map is interesting, the inductive bias preventing two objects being present in the same location is interesting, however one could imagine a case where uncertainty in the model could lead to two objects being represented in the same cell.\n\nThe terms in Equation 3 should be explained more explicitly. The way I understand it p(s_n|z, x) is the view-point dependent representations of the objects and this is why you condition on z, x. Placing p(s_n|z, x) in the text where you talk about s_n being view dependant with some explanation would help.\n\nWhy do you use a Gaussian distribution for the position? Would it not make more sense to use a uniform distribution? Could this be related to bias in your data? I.e. more objects in the centre of the scene?\n\nWhat happens if object n is not present in the context image, y_c? In this case what is s_{c,n}^pos. Also, this notation: r_n = f({y_c,n}c)  is a little ambiguous. I assume it means that you are applying the function f to the set of all patches in y_c? Also, what is the object invariant object encoder? A reference to details in the appendix or a footnote would suffice.\n\nIt’s great that you are able to exploit the coordinate transform and use it for rendering.\n\nThe results are very impressive: being able to swap in and out objects in a scene, showing the 2D renderings of single objects from different view-points and the scene decompositions and predicting where the “missing” object is (Figure 6).\n\nMinor:\nThe introduction could be strengthened with additional references. There are claims that object-wise factorisation will help with transfer, it would be good to have references to other work that supports this view. Also the claim that humans have 3D representations for objects requires a reference.\n\nTypos (there are too many to list here, these are just a few):\n* Abstract: and and rendering\n* “and”s should be replaced with commas in the second line of the intro.\n* Generally the paper is not written well.\n* The GQN, as a conditional → is a conditional\n* Target observations (in section 2) does not need a capital.\n* This sentence does not make sense: “instead of encoding compressing the whole scene”\n* Because of intractable posterior\n\nThere are many additional grammatical errors. \n\n\n-----------\nEdit: Following changes made to the paper, I am now more satisfied. The writing should still be improved further and suggest that the authors fully revise the paper before the camera ready version, if the paper is accepted. I have increased my score to 6.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The author proposes a object-oriented probabilistic generative model of 3D scenes.  The model is based on the GQN with the key innovation being that there is a separate 3D representation per object (vs a single one for the entire scene).  A scene-volume map is used to prevent two objects from occupying the same space. The authors show that using this model, it's possible to learn the scene representation in an unsupervised manner (without the 3D ground truth). 

The submission has received relatively low scores with one weak accept and 3 weak rejects.  All reviewers found the initial submission to be unclear and poorly written (with 1 reject and 3 weak rejects initially).  The initial submission also failed to acknowledge prior work on object based representations in the 3D vision community.  Based on the reviewer feedback, the authors greatly improved the paper by reworking the notation and the description of the model, and included a discussion of related work from 3D vision.  Overall, the exposition of the paper was substantially improved.  Some of the reviewers recognize the improvement, and lifted their scores.  

However, the work still have some issues:
1. The experimental section is still weak
The reviewers (especially those from an computer vision background) questioned the lack of baseline comparisons and ablation studies, which the authors (in their rebuttal) felt to be unnecessary. It is this AC's opinion that comparisons against alternatives and ablations is critical for scientific rigor, and high quality work aims not to just propose new models, but also to demonstrate via experimental analysis how the model compares to previous models, and what parts of the model is necessary, coming up with new metrics, baselines, and evaluation when needed.

It is the AC's opinion that the authors should attempt to compare against other methods/baselines when appropriate.  For instance, perhaps it would make sense to compare the proposed model against IODINE and MONet.  Upon closer examination of the experimental results, the AC also finds that the description of the object detection quality to be not very precise.  Is the evaluation in 2D or 3D?  The filtering of predictions that are too far away from any ground truth also seems unscientific.  

2. The objects and arrangements considered in this paper is very simplistic.  

3. The writing is still poor and need improvement.
The paper needs an editing pass as the paper was substantially rewritten.  There are still grammar/typos, and unresolved references to Table ?? (page 8,9).


After considering the author responses and the reviewer feedback, the AC believe this work shows great promise but still need improvement.  The authors have tackled a challenging and exciting problem, and have provided a very interesting model.  The work can be strengthened by improving the experiments, analysis, and the writing.  The AC recommend the authors further iterate on the paper and resubmit.  As the revised paper was significantly different from the initial submission, an additional review cycle will also help ensure that the revised paper is properly fully evaluated.  The current reviewers are to be commended for taking the time and effort to look over the revision. 
Decision: Reject