Generative Hierarchical Models for Parts, Objects, and Scenes
Authors: Fei Deng
Zhuo Zhi
Sungjin Ahn
Venue:  (2020)

Abstract: Hierarchical structure such as part-whole relationship in objects and scenes are the most inherent structure in natural scenes. Learning such representation via unsupervised learning can provide various benefits such as interpretability, compositionality, and transferability, which are important in many downstream tasks. In this paper, we propose the first hierarchical generative model for learning multiple latent part-whole relationships in a scene. During inference, taking top-down approach, our model infers the representation of more abstract concept (e.g., objects) and then infers that of more specific concepts (e.g., parts) by conditioning on the corresponding abstract concept. This makes the model avoid a difficult problem of routing between parts and whole. In experiments on images containing multiple objects with different shapes and part compositions, we demonstrate that our model can learn the latent hierarchical structure between parts and wholes and generate imaginary scenes.

Under review as a conference paper at ICLR 2020
GENERATIVE HIERARCHICAL MODELS FOR PARTS, OBJECTS, AND SCENES
Anonymous authors Paper under double-blind review
ABSTRACT
Compositional structures between parts and objects are inherent in natural scenes. Modeling such compositional hierarchies via unsupervised learning can bring var- ious benefits such as interpretability and transferability, which are important many downstream tasks. In this paper, we propose the first deep latent variable model, called RICH, for learning Representation of Interpretable Compositional Hierarchies. At the core of RICH is latent scene graph representation that orga- nizes the entities of a scene into a tree structure according to their compositional relationships. During inference, taking top-down approach, RICH is able to use higher-level representation to guide lower-level decomposition. This avoids the difficult problem of routing between parts and objects that is faced by bottom-up approaches. In experiments on images containing multiple objects with different part compositions, we demonstrate that RICH is able learn the latent composi- tional hierarchy and generate imaginary scenes.
INTRODUCTION
Compositional hierarchies prevail in natural scenes where primitive entities recursively com- posed into more abstract entities. Modeling such compositional generative process allows discovery of modular primitives that can be reused across a variety of scenes. Hence, would bring inter- pretability and transferability. in which current deep learning models are not quite successful. Due to expensive labeling. such compositional relationships should ideally be learned in an unsupervised manner. Unsupervised approaches can also provide more flexibility and generalization ability since the model is allowed to choose the most appropriate compositional hierarchy for given scenario.
Despite its importance, there has not been much work on unsupervised generative modeling of the compositional hierarchy. Earlier work on hierarchical representation learning (Lee et al., 22099) ob- tains feature hierarchy that captures concepts at different levels of abstraction, with no explicit modeling of composition. Recent researches on deep latent variable models (Maaløe et al.) 2019, Zhao et al., 2017; Sønderby et al., 2016 Bachman 2016) mainly focus on architectural designs and training methods that harness the full expressive power of hicrarchical generative models. Although they have shown impressive generation quality and disentanglement of learned representation, the compositional hierarchy is still not captured in a modular and interpretable way. To obtain inter- pretable scene representation, recent work (Tieleman) 2014 Eslami
2016 Crawford & Pineau 2019 Wu et al. 2017 Yao et al.) 2018 Romaszko et al. 2017 Deng et al.) 2019) has introduced domain-specific decoders that take object pose and appearance information as input and render the object in a similar way to graphics engines. This forces the encoder to invert the rendering process, producing interpretable object-wise pose and appearance representation.
In this paper, we extend the interpretable object-wise representation to the hierarchical setting. We propose a deep generative model, called RICH (Representation of Interpretable Compositional Hier- archies), that can use its hierarchy to represent the compositional relationships among interpretable symbolic entities like parts, objects, and scenes. To this end, taking inspiration from capsule net- works (Sabour et al. 2017 Hinton et al. 2018) and the rendering process of computer graphics, we propose a probabilistic scene graph representation that describes the compositional hierarchy latent tree. The nodes in the tree correspond to entities in the scene, while the edges indicate the compositional relationships among these entities. We associate an appearance latent with each node to summarize all lower-level composition, and pose latent with each edge specify the

---

Under review as a conference paper at ICLR 2020
transformation from the current level to the upper level. To enforce interpretability. the probabilistic scene graph is then paired with a decoder that renders the scene graph by recursively applying the specified transformations. We also introduce learnable templates for the primitive entities. Once learned, RICH is able to generate all lower-level latents and render a partial scene given the latent at specific level.
To infer the scene graph is, however, challenging, since both the tree structure and the latent vari- ables need to be simultaneously inferred. Capsule networks have provided bottom-up solution to learning the tree structure, but it faces the difficult routing problem caused by the exponentially many possible compositions. Instead, RICH takes top-down approach that avoids the routing prob- lem. The intuition is that for given scene, it is natural to first decompose it into high-level objects. If we devote our attention to one of the objects, we can then figure out its constituent parts. In cases where parts are close or have occlusion, we expect the appearance latent of the higher-level object to guide lower-level decomposition, since it summarizes the typical composition for that object.
The contributions of this paper are as follows. We propose RICH, the first interpretable represen- tation learning model for compositional hierarchies through probabilistic latent variable modeling. We then implement three-level prototype of RICH and demonstrate its effectiveness in extensive experiments. RICH is able to learn the hierarchical scene graph representation from images contain- ing multiple compositional objects. Further, it shows decent generation quality and generalization ability to unseen number of objects.
2 RELATED WORK
Interpretable object-wise representation. AIR (Eslami et al.) 2016) is the first generative model that learns interpretable object-wise scene representation. It is able to assign latent vector (z 2 zuheree, 2 2 to each object in the scene, describing the presence, size, center position, and appearance of the object. SPAIR (Crawford & Pineau 2019) improves the scalability of AIR to images containing a large number of objects. It divides the image into spatially distributed cells, and regresseseyy infers the latent vector for each cell. This crucially reduces the search space for individual cells, since they are each responsible only for explaining objects near themselves. RICH builds upon SPAIR to infer the structure of the probabilistic scene graph. To enable efficient hierar- chical inference, we use mean-field field approximation for the posterior, allowing inference of all cells to be done in parallel.
Hierarchical scene representation. Modeling the part-whole relationship in scenes has attracted growing interest, and it has been utilized for improving image classification, parsing, and segmenta- tion. Two representative models that have inspired our work are hierarchical compositional models (HCMs) and capsule networks. In HCMs (Zhu et al., 2008), the hierarchical structure is represented as graph, where leaf nodes interact with image segments, and upper-level nodes store the average position and orientation of lower-level nodes (with respect to the image coordinates). In capsule networks (Sabour al.) 2017, Hinton et al.) 2018), the part-whole relationship is used for achiev- ing viewpoint invariance. The key insight is that the relative pose of parts with respect to objects viewpoint invariant, and is thus suitable to be learned as network weights. However, neither of these two approaches uses generative modeling, and they have been applied only to scenes with one dominant object. More recently, the part-whole relationship has been explored in modeling 3D shapes (Tulsiani et al. [017] Li et al. 2017]: Zhu et al.) 2018) and motion decomposition (Xu et al., 2019). Although they have employed generative modeling, the latent space is not hierarchically structured, and typically only contains the top-level representation. In contrast, RICH represents the entire scene graph in latent space, modeling both the hicrarchical structure and the object pose and appearance each level as latent variables. This representation is arguably richer, and enables simultaneous learning of a collection of generative models for parts, objects, and scenes.
3
THE PROPOSED MODEL: RICH
RICH (Representation of Interpretable Compositional Hierarchies) is a generative model that cap- tures the recursive compositional structure inherent in natural scenes. It builds tree-structured representation similar to scene graphs in computer graphics (Foley et al. 1996). The nodes in the tree describe entities at various levels of abstraction in the scene, and the edges indicate the compo-
2

---

Under review as a conference paper at ICLR 2020
B
Sppr
X
0g2
up
111
Xay
pree
Vg
U1
Figure 1: (A) Probabilistic scene graph representation. Each node represents an entity in the scene, and associated with an appearance latent. Each edge is associated dinate transformation between the child node and the parent node. (B) Top-down inference process. Inference combines information from glimpse regions and higher-level appearance latents (not shown here). Bounding boxes indicate inferred pose latents. (C) Recursive decoding process single recursive step shown). The im- age patch Xa, and mask m,, an internal node du are decoded from the image patches and masks of all its children nodes.
relative pose latent that specifies the coor-
sitional relationships among these entities. Specifically, each leaf node represents a primitive entity that is further decomposed. Each internal node represents an abstract entity that is composed from its children nodes. The composition is specified by the relative pose of each child node with respect the parent node, and this pose information is stored on the corresponding edges.
3.1 GENERATIVE PROCESS
To make a generative model, we associate appearance vector z with each node and a pose vector po with the edge between node e and its parent pa(v), as shown in Figure DA. The intuition that 2 "Prr represents the entity at pu(v) in its canonical pose, summarizing all lower-level com- "p((w)
position in the subtree rooted at (v))). Conditioning on "pe(e)> we can then sample z appr and z powe
appr
the relative pose of with respect pa(v). Thus, the latent vectors in any subtree be recursively generated. Let V be the set of all nodes, r E V be the root node, and L C be the set leaf nodes. The generative model for the entire scene
be written follows:
p(x)= =
/ p(x Vitry Z
pl
appr
II VEVV(r)
p(2 pose
z
patu)) p(z appr
appr
2
appr pa(u)) dz,
appr that have the same where assume conditional independence among all latents 2. and Z pu()). This gives disentangled and interpretable scene representation.
We design the decoder pxx|2 (r). zz to closely follow the rendering process from a given scene graph. First, for each leaf node v L, use neural network decode its appearance vector into a small image patch &, and (close to) binary mask m. the same size as & Here we assume that &, has already been masked by m. meaning (,, (i.j) = O. We then recursively compose these primitive patches into the entire scene by applying affine transformations level by level. Specifically, let be internal node, and ch(u) ) be the set of its children. We compose the higher-level image patch x, and mask m, as follows:
for all pixel locations (i.j) where
R
Σ (Eol(u)
a² ST pose).
m.
Σ ar ST (m,, 2 pose)
BEch(N)
where a spatial transformer ST (Jaderberg et al. 2015) is used to properly place X, and m, into the coordinate frame of the parent node u, according to the scaling, rotation and translation parameters given by z! pose In addition, z pose also provides relative depth information that helps deal with occlu- sion. Entities with smaller depth will appear in front of entities with larger depth. This is enforeed by a transparency map a, that assigns pixel-wise weights to each transformed patch according to its relative depth. See Figure C for an illustration. To ensure that unoccluded part of entities will

---

Under review as a conference paper at ICLR 2020
remain visible, we normalize at after applying the transformed mask m, ST namely for all pixel locations (i,j).
STT z0))),
ax (i.j) if mi.j.)) =0. =0
Σ
a.(i,)) =1 if Σ mo(i.) > >0.
(3)
vEch(a)
vEch(u)
The final decoder output p(x) z( Nix, I)), where a is hyperparameter.
3.2 INFERENCE AND LEARNING
Since computing p(x) in Equation is intractable, we train RICH with variational inference. The approximate posterior is designed to factorize in the top-down fashion similar to the generative process, such that a higher-level appearance representation guides lower-level decomposition:
p(z x) q(z x) (z
x)
II (EV)(r)
g(2 pose
Z pa(()) Xpa(v) (Z))pP
pa(e)) x,).
Here x, is the region of input image that corresponds to the entity that node represents. This region is specified by the predicted pose vectors along the path from root r to node v. More precisely, we define x, x, and recursively extract x, = (apv)) 20""), ), as shown in Figure 1B. Notice that the relative pose z of u with respect to pa(v) needs to be inferred from Xpa(o)- After applying spatial transformer ST. we assume the captured x, is in its canonical pose. This top-down inference process avoids the challenging routing problem in capsule networks (Sabour et al. 2017: Hinton et al. 2018), leading to more efficient inference. In cases where entities are close have occlusion, the higher-level appearance vector upp could provide some guidance separating these entities.
ppa(v)
In general, all latents are assumed to be continuous, with both prior and posterior being Gaussian distributions. However, it may bring additional flexibility and interpretability to introduce some discrete latents, we explain in Section 3.3 For continuous latents, we compute posterior via seeision-weighhed combination similar to Ladder- (Sønderby et al. 2016). and use reparam- eterization trick (Kingma & Welling, 2013) to sample from the posterior. For discrete latents, use Gumbel-Softmat trick (Jang et al. 2016 Maddison et al. 2016). Thus, the entire model can be trained end-to-ent via backpropagation to maximize the following evidence lower bound (ELBO):
L = Ex(x/x) [og
(r) 2 - DKLl9(z,PP)
pose
xx)
p(Z)9))]
(5)
Σ VEVMrt
E [x(e/e) KT. | (z!e0e
pa(v)) Xpa())) || p(z!,se 12pppe
pa(()) I]]
Dvev E | (g |z pper ppa(w):Xo)
p(z.ppr appe z appe
pa((x) I]}
3.3 IMPLEMENTATION DETAILS
Structural inference. In our description above, we have assumed that the tree structure is already known. We now relax this assumption and introduce structural inference. First, we set maximum out-degree for each node so that the number of all possible structures is bounded. For simplicity, in our implementation nodes within the structure, it then suffices to specify the presence of each possible edge. Hence, for an arbitrary edge between node t and its parent,
level share the same maximum out-degree. To determine
introduce Bernoulli variable fies to indicate its presence. If z.t = 0, meaning the edge is not present, then z pose along with all latents in the subtree rooted are excluded from the representation. To encourage sparse structures, we initialize the prior p(zees pres <pa(w)) and the posterior q(z prees appr
<pp(w))> Xpa(n)) to have small Bernoulli parameters.
Node grounding. Due to the symmetric tree structure, there are numerous equivalent entity-to- node assignments for given scene, each yielding a different permutation of the pose vectors. This can cause difficulties in the learning process. In particular, the model has to learn consistent assignment strategy such that the pose vector at each edge can be well captured by a unimodal Gaussian distribution. To alleviate this problem, impose some inductive bias the assignment strategy. Inspired by SPAIR [Crawford & Pineau| 2019),, for each internal node u, we divide x,, into grid of N,, cells, where N,, is the maximum out-degree of Each child of is assigned one
4

---

Under review as a conference paper at ICLR 2020
of these cells, and is responsible for explaining only the entity whose center position is within that cell. Assuming that X captures the entity u in its canonical pose, this assignment strategy ensures that each child of node is almost always associated with the same component of entity To deal with occlusion, we let neighboring cells have slight overlap.
Primitive templates. It is often reasonable to assume that the vast number of complex entities can be composed from only a modest number of primitive entities. Identifying such primitive entities through discrete latent variables would bring additional interpretability. Hence, we introduce an external memory M with variational addressing (Bornschein et al. (017)) to learn a set of primitive templates. For each leaf node e E L. we decompose its appearance vector into ,, (z ( 2 Here, zu is one-hot vector that points one of the templates in M, and z, is a continuous vector that explains the remaining variability. We assume that 2 appr ppa(o) captures the identity but not the exact
appearance of entity since appe is intended for summarization. Therefore, we factorize the prior "pa(v)
and the posterior follows:
pl p(z- upp
appr pp((v)) = =p(z) Hidt
2upprr "pa(v)) M) p( ( what
(6)
appe pa(()))
Xn) g(2gadr
2apprr pa(si) M, x.) q(2,) (( M (2addrrr X.):
(7)
where M considered as model parameter, and M( is the deterministically retrieved memory content, We implement M to be a stack of low-dimensional embeddings of templates. To decode from 2.,,, we first retrieve the embedding indexed by ziddr, and decode it into a single-channel
image patch. This serves both the template and the mask, namely m. g (zuirr)) (M[zrrr)) ) We then
apply multiplicative modification controlled by z what and obtain X = m. O h(z," ). Here, X,
has the same number of channels as the input x, and both g(-) and h(-) are implemented as spatial broadcast decoders (Watters et al. 2019).
Canonical size. To avoid undesired effects in cascaded affine transformations, constrain the spatial transformer to always preserve aspect ratio. Thus, for a square input image, each entity is assumed to occupy square image region. Considering the various possible compositions, this region may not capture the entity's 's canonical size well. Hence, we introduce a latent variable z to represent the aspect ratio of entity u. This could give a tighter bounding box inside the square region. We consider zulli as part of the appearance vector, namely z...
(2 z and factorize the
prior and the posterior as follows:
p(z:FFr
z appr
pp(()) = p(z
what
2
appe "pa(w) p(z Zu zuhaa).
(8)
(z appr
z
spp pa(u)) x..) = g(z what
uppr (x)) x.) g(2 ratio
x what xx).
For simplicity, we introduce 2 only for intermediate nodes u E V U (r}). To properly learn z feed it as additional argument the spatial transformer. This has effects. First, the region outside the bounding box given by zatio is masked out during transformation in both inference and reconstruction. This forces the bounding box capture the entity in its entirety. Second, during training, we inject zero-mean Gaussian noise to the reconstruction inside the bounding box region. The noise level is annealed as training proceeds. This encourages tight bounding boxes without affecting generation quality.
4
EXPERIMENTS
4.1 DATASETS AND A THREE-LEVEL. PROTOTYPE
We have implemented a prototype of RICH with part-, object-, and scene-level representation. We will refer to the latents as zp, zo, and zş respectively. The maximum out-degree is set to be 4 for each internal node. For evaluation, we have made two datasets of 2D and 3D scenes. Both datasets contain 128x 128 color images, split into 64000 for training, 12800 for validation, and 12800 for testing. They present challenges of (i) multi-pose, variable number of objects and parts, (ii) multiple occurrences of the same type of objects and parts within one scene, and (iii) severe occlusion in 3D scenes. In making each dataset, we first choose set of primitive shapes to be the parts, and then construct the objects and scenes by recursively composing these parts. Specifically, we have chosen three shapes as parts, and defined ten types of objects in terms of the identity of the constituent parts
5

---

Under review as a conference paper at ICLR 2020
Figure Qualitative results on dataset. (A) (Top) Input image. (Middle) Input image superimposed with pose
predicted bounding boxes, drawn according to z 20 zo 2p up pose and addr (Bottom) Reconstruction. (B) Learned part-level templates. Template colors indicate identity. Part bounding box colors indicate the chosen template.
and their relative position, scale, and orientation. Among these ten types, three contain single part, another three contain two parts, and the remaining four contain three parts. To construct a scene, first randomly sample the number of objects (between and four) and their types, and then instantiate these objects. This means for each object, we choose random color for each of its parts, apply random scaling (within 10% of object size), and draw 2D case, the instantiation process also includes random perturbation of parts and random rotation of the object whole. We ensure that different objects have minimal overlap. In 3D case, we use MuloCo (Todorov et al., 2012) to place the objects on a plane, and then take observations from ten different viewpoints, some of which can lead severe occlusion.
random position in the scene. In
4.2 SCENE DECOMPOSITION
RICH is able to give interpretable, tree-structured decomposition of scenes into objects and parts. We visualize such decomposition in Figure Figure 2D and 3D scenes respectively. where we also show the learned memory templates for parts. Notice that the templates should be in gray scale, but for visualization purposes we have assigned a color to each template. The bounding boxes drawn top of the input images, according to the inferred pose of objects and parts with g = 1. Object bounding boxes are drawn in white, while part bounding boxes are drawn in color to indicate the template chosen for each part.
We find that the templates have learned the appearance of parts at several canonical poses (rotation in 2D and viewpoint in 3D), and RICH predicts the pose of parts with respect to these canonical poses. This makes the decomposition even more interpretable. Moreover, equipped with templates, RICH able correctly identify the parts even when they have severe occlusion. See Figure A third row where a ball occludes an equally sized cube. This example (and many others) also demonstrate that RICH can successfully deal with objects composed of multiple parts that are of the same type.
In addition to part-level templates, we believe that the learned object-level appr also helps scene de- composition, especially when there is ambiguity in part assignment and occlusion between objects. For example, in Figure AA first column, the triangle and circle near the center are close to each other and may well constitute an object. However. because this pose configuration relatively rare in the training set (compare second column), RICH has correctly rejected this composition and instead assigned these two parts separate objects, which better agrees with the training distribution. In Figure 3A fourth row, object is occluded by object 3. RICH has successfully detected object I and added in the reconstruction ball the same color as the occluded part. This is quite reasonable since the augmented object is of our predefined types and appears frequently in the dataset.
To quantify RICH's s ability scene decomposition and representation learning, we report absolute counting error, precision, and recall for detection of objects and parts in Table [ and compare the negative log- likelihood RICH with a VAE (Kingma & Welling, 2013) baseline in Table [ Here the counting error measures the absolute difference between the predicted and true number of objects

---

Under review as a conference paper at ICLR 2020
Whole
Object
Object
Object
Object
Recon-
Input bbox
Recon- struction
Input bbox
Recon- struction
Inpur bbox
Recen- struction
Input bbox
Recon- struction
Input
struction
Figure Qualitative results 3D dataset. (A) Each row shows the overall reconstruction, and the predicted bounding boxes reconstruction from each object cell for given input image. (B) Learned part-level templates. Template colors indicate identity. Part bounding box colors indicate the chosen template.
Table 1: Quantitative results on object and part detection.
Dataset
2D dataset
3D dataset
Training set
1~4 objects
1&3 objects
I~4 objects
1&3 objects
Test set
I~44 objects
2 objects
objects
I~4 objects
objects
4 objects
Object count error
0.00083
0.0014
0.00036
0.094
0.26
0.47
Object precision
0.9985
0.9987
0.9996
0.9639
0.9157
0.9581
Object recall
0.9984
0.9982
0.9995
0.9597
0.9758
0.8462
Part count
0.0086
0.011
0.014
0.80
1.1
1.3
Part precision
0.9991
0.9988
0.9991
0.8282
0.7579
0.8100
Part recall
0.9989
0.9985
0.9993
0.9116
0.9258
0.8347
and parts. To obtain precision and recall, we need to match the predictions with the groundtruth. We set the matching priority as the distance between the predicted and true center positions, namely closer pairs of prediction and groundtruth will be matched first. We only match the pair if their distance is less than 10 pixels (less than half of the size of large parts). This ensures that the matched predictions will have approximately correct center positions. The VAE baseline shares the same scene-level encoder with RICH. and uses sub-pixel convolution (Shi et al. 2016) for the decoder. We approximate the negative log-likelihood using 50 importance-weightee error, precision, and recall are also averaged over 50 samples from the posterior. As can be seen from Table RICH gives almost perfect detection of objects and parts 2D dataset, and still performs reasonably well on the challenging 3D dataset. We observe that RICH tends to split a long cylinder into parts, leading to the drop in precision for parts.
samples. The counting

---

Under review as a conference paper at ICLR 2020
B
Figure (4) Generated objects on (Top) dataset and (Bottom) dataset. White boxes indicate aspect ratio, and are drawn according to zo (B) Generated scenes on (Top) 2D dataset and (Bottom) 3D dataset.
quo
Table 2: Comparison on negative log- lgglikelihood.
Dataset
2D dataset
3D dataset
Training set
1~44 objects
1&3 objects
1 1~4 objects
1&3 objects
Test set
1~44 objects
2 objects
objects
1~4 objects
2 objects
4 objects
VAE RICH
-13761.9
-13801.4
-13590.5
-13712.0
-13788.3
-13433.2
-13890.3
-13908.3
-13796.6
-13818.0
-13867.3
-13539.7
4.3 OBJECT AND SCENE GENERATION
Apart from learning the part-level templates, RICH also has the ability to generate objects and scenes by recursively composing the learned templates. We show generation results in Figure ! To generate the scenes, first sample zg (O I), and then sample other latents following the learned conditional prior distributions, and finally use the decoder to render the image. The objects generated similarly, except that decode up to the object level and ignore zo and zo We find that RICH has captured many predefined object types in the dataset, and also managed to come up with novel compositions. The generated scenes also reasonable, with moderate distance and occlusion between objects.
pres
4.4 GENERALIZATION PERFORMANCE
RICH represents the scene as composition of objects and parts. This naturally enables generalization novel scenes. Here we evaluate RICH's capacity generalize scenes with novel number of objects. The training and validation sets this task contain scenes of one and three objects only. We trained RICH and the VAE baseline again, and report the metrics in Table I and Table on two test sets, one having two-object scenes only, and the other having four-object scenes only. We also show qualitative results in Figure 5 and Figure 6 As can be seen, RICH demonstrates quite decent generalization performance in both 2D and 3D scenes. We notice that in 3D case, there is drop recall when RICH tested on four-object scenes. One reason is that four-object scenes exhibit more severe occlusion than the training set, and we have observed that when two objects are close and have occlusion, RICH would sometimes merge them into one object. Another reason is that in four-object scenes, objects are more likely to be partially outside the scene. In this case, RICH has difficulty predicting the precise object position, leading to unmatched predictions when we compute the recall.
4.5
DATA EFFICIENCY IN DOWNSTREAM TASKS
The compositional hierarchy learned by RICH can be useful in downstream tasks that require rea- soning part-object relationships. Here we consider a classification task. The input images are generated from the same distribution described in Section 4.1 but with different random seeds. The label for each image is obtained by first counting the number of distinct parts within each ob- ject, and then summing the count over all objects. We expect the representation learned by RICH to
8

---

Under review as a conference paper at ICLR 2020
B
Figure Generalization results on dataset. RICH has been trained on scenes with and objects only, and tested on scenes with (A) 2 objects and (B) 4 objects. (Top) Input image. (Middle) Input image superimposed with predicted bounding boxes. (Bottom) Reconstruction.
Whole scene
Object
Object
Object 3
Object 4
Recon-
Input
Recon-
Input
Recon-
Input
Recon-
Input bbox
Recon-
Input
struction
bbox
struction
bbox
struction
+ bbox
struction
striction
Figure 6: Generalization results on 3D dataset. RICH has been trained on scenes with and 3 objects only. and tested scenes with (A) objects and (B) 4 objects. Each shows the overall reconstruction. and the predicted bounding boxes and reconstruction from each object for given imput image.
bring better data efficiency compared to non- non-hierarchical baselines like SPAIR (Crawford & Pineau 2019). To verify this, we use pretrained RICH and SPAIR to obtain the latent representation for each image, and then train classifier top of the representation to predict the label. Specifically, the unnormalized class probabilities RICH for RICH are computed as follows:
pres M[zzddr) ))
RICH
MMLP
Σ
W. GMPP
Σ rEch(w)
W. f MLP
(Z
(10)
where and indices of objects and parts respectively, and w. attention weights also computed by MLPs. Because RICH provides the compositional hierarchy, the classifier can explic- itly merge information from parts that belong the object. For SPAIR that detects parts with no notion of objects, we include pose vectors as additional input for fair comparison. This allows the classifier to group parts into objects based their positions. The unnormalized class probabilities y SSPAIR for SPAIR are computed as follows:
Σ (()) MMP concaa|z2 appr 2 120 (0()))
SPAIR
-PMLL MLPP
Σ WH
(1I)

---

Under review as a conference paper at ICLR 2020
2D Dataset
3D Dataset
100
100
Test Accuracy (%)
80
Test Accuracy (%)
80
60
40
40
20
RICH SPAIR
20
RICH SPAIR
0
0
64
128
256
384
512
102
103
10
Training Samples
Training Samples
Figure 7: Comparison of data efficiency in downstream tasks.
The difference here is that we have multiple sets of attention weights w("), each should learn to select the parts that belong to one specific object.
We train the two classifiers using varying number of training samples. We ensure that the classi- fiers have comparable number of parameters. We use a parallel implementation of SPAIR that, as suggested by Anonymous (2020), has similar detection performance but faster training speed. We choose the best learning rate for both classifiers using fixed validation of size 12800. We report classification accuracy on fixed test set also of size 12800. As shown in Figure 7 RICH repre- sentation approximately doubles the data efficiency on this downstream task compared SPAIR representation.
5 CONCLUSION
We have proposed RICH, the first hierarchical generative model for learning interpretable composi- tional structures. RICH takes top-down approach to infer the probabilistic scene graph represen- tation for a given scene. This utilizes the higher-level appearance information to guide lower-level decomposition, thus avoiding the difficult routing problem faced by bottom-up approaches. Through extensive experiments, we have demonstrated that RICH is able to learn the compositional hierarchy from images containing multiple objects. interesting future direction is extend RICH the sequential setting.
REFERENCES
Anonymous. Spatially parallel attention and component extraction for scene decomposition. In Submitted to International Conference on Learning Representations, 2020. URL https://
under review.
Philip Bachman, An architecture for deep, hicrarchical generative models. In Advances in Neural Information Processing Systems, pp. 4826-4834 2016.
Jörg Bornschein, Andriy Mnih, Daniel Zoran, and Danilo Jimenez Rezende. Variational memory addressing in generative models. In Advances in Neural Information Processing Systems, pp- 3920-3929, 2017.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In Proceedings of AAAI, 2019.
Boyang Deng, Simon Kornblith, and Geoffrey Hinton. Cerberus: A multi-headed derenderer. arXiv preprint arxiv::905.1 /99.////5.//////// 2019.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari. Geoffrey E Hin- ton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
10

---

Under review as a conference paper at ICLR 2020
James D Foley, Foley Dan Van, Andries Van Dam, Steven K Feiner, John F Hughes, Hughes, and Edward Angel. Computer graphics: principles and practice, volume 12110. Addison Wesley Professional, 1996.
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with EM routing. In Interna- lional Conference on Learming Representations, 2018. URL https:://ppeeeeeeieeeee net/
Forum?1G-HUWLFOWRR
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et Spatial transformer networks. In Ad- vances in neural information processing systems, pp. 2017-2025, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzttion with gumbel-softmax, arXiv preprint 1rXiv.1611..00 144, 2016.
Diederik P Kingma and Max Welling. 1312.66 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Yann LeCun, Léon Bottou, Yoshua Bengio. Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 36(11):2278-233 2324, 1998.
Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Proceedings of the 26th annual international conference on machine learning, pp. 609-616. ACM, 2009.
Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. Grass: Generative recursive autoencoders for shape structures. ACM Transactions on Graphics (TOG), 36(4):52, 2017.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling, arXiv preprint arXiv:190..02102, 2019.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:161 1.007 2016.
Lukasz Romaszko, Christopher KI Williams, Pol Moreno, and Pushmeet Kohli. Vision-as-inverse-
graphics: Obtaining rich 3d explanation of a scene from single image. In Proceedings of the IEEE International Conference on Computer Vision, pp. 851-859,
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866, 2017.
Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874-1883, 2016.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, pp. 3738-3746,
2016.
Tijmen Tieleman. Optimizing neural networks that generate images. University of Toronto (Canada), 2014.
Emanuel Todorov. Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Shubham Tulsiani, Hao Su, Leonidas J Guibas, Alexei A Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2635-2643 2017.
Nicholas Watters. Loie Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vacs. arXiv preprint arXiv:1901 190100017. 2019.
11

---

Under review as a conference paper at ICLR 2020
Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 699-707, 2017.
Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Unsupervised discovery of parts, structure, and dynamics. arXiv preprint arXiv: arXiv:1903.055 1et0099930..6.. 2019.
Shunyu Yao, Tzu Ming Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, Bill Freeman, and Josh Tenenbaum. 3d-aware scene manipulation via inverse graphies. Advances in Neural Informa- fion Processing Systems, pp. 1887-1 2018.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep gener- ative models. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp.
JMLR. org, 2017.
Chenyang Zhu, Kai Xu, Siddhartha Chaudhuri, Renjiao Yi, and Hao Zhang. Scores: Shape com- position with recursive substructure priors. In SIGGRAPH Asia 2018 Technical Papers, pp. 211. ACM, 2018.
Long Leo Zhu, Chenxi Lin, Haoda Huang, Yuanhao Chen, and Alan Yuille. Unsupervised structure learning: Hierarchical recursive composition, suspicious coincidence and competitive exclusion. In European Conference on Computer Vision, pp. 759-773. Springer, 2008
12

---

Under review as a conference paper at ICLR 2020
A DEALING WITH DEFORMABLE PRIMITIVES
In Section we have demonstrated the effectiveness of RICH on two datasets of 2D) and 3D scenes. However, these datasets only contain non-deformable primitives, meaning each primitive has only one possible shape. Here, we introduce compositional MNIST dataset where the primitives are handwritten digits with considerable shape variation. Through the same set of experiments as in Section 4 we show that with slight modification, RICH can successfully deal with such deformable primitives.
A.1 COMPOSITIONAL MNIST DATASET
We define the primitive parts to be individual digits from the MNIST database (LeCun al. 1998). We have excluded the digit 1, since it can be used as sub-parts in other digits like 4 and 7, and thus may interfere with our definition of primitives. We then define nineteen types of objects, of which nine contain a single digit, six contain two digits, and the remaining four contain three digits. Here, the object definition specifies the class of each digit that an object should contain, and the relative position and orientation of these digits. We allow overlap among digits that constitute one object.
The dataset contains 128x 128 color images, split into 64000 for training. 12800 for validation, and 12800 for testing. Each image describes a scene that contains one to four objects. To instantiate an object from its definition, we first sample the digits from the corresponding classes and resize them to 36x36. Next, we assign random color to each digit. The colors are guaranteed to be distinct. Finally, we apply random scaling (within 10% of object size) and random rotation (within +30 degrees) to the object as a whole, and draw it at random position in the scene. We ensure that different objects have minimal overlap, and that disjoint sets digit samples are used for the training. validation, and test sets.
A.2 MODIFICATIONS TO RICH
RICH decomposes the appearance vector of primitive v into z aper = (zithrr 2 and decodes the mask m. and the image patch X, as my gMM[z [2:)( and x m, hz Although in principle, such element-wise multiplication has the ability to model deformation, we empirically find this to be hard. In fact, h(2,har) tends to produce an image patch filled with single color, and is thus unable to distort the shape of template given by m..
To deal with deformable primitives, we add more inductive bias to the decoder. In particular, 2 is further split into Zy shape and zoolor during the decoding process. shape is designated to explain the shape variation, and concatenated with the template embedding M to modify the shape of
template as m, = g(concat t MM addr) ; zie >hhap)).. As mentioned in Section 3.3 m, is in gray scale, so we then use 2aalor to colorize the modified template m. and produce X, m. @ h(2gglor).
One challenge here is disentangle deformation from pose changes. Ideally, zhaap should capture only the shape variation of primitive in its canonical pose, so that z and 2,000 do not interfere with each other. This can be enforced by limiting the expressiveness of z,) zhape In experiments, we find that restricting zip
shape to a one-dimensional scalar works reasonably well.
Another challenge arises during training. We have observed that RICH tends to split the digits into uub-parts at an early stage. Consequently, the strokes instead of the digits are learned as primitives, and the objects become composition of strokes, leading to incorrect compositional hierarchy. To alleviate this problem, we take inspiration from SPACE (Anonymous) 2020) and introduce a color- filtered boundary loss for each glimpse x,. where v is primitive. The key observation that if x, captures only part of digit, then there must be some pixels on the boundary of x, that have the same color as the digit. Here, the thickness of boundary is a hyperparameter. We use h(2golor) as a color filter to find such pixels. The boundary loss then penalizes the total number of those pixels. This drives z puse to produce a larger glimpse region that reduces splitting. We anneal the weight of the boundary loss as training proceeds.
13

---

Under review as a conference paper at ICLR 2020
43
0
6
U
3-
8
2
4
8
43
4/
4
33
9
2
8
7
J3
3
O
9 9 9
Figure 8: Qualitative results on compositional MNIST dataset. (A) (Top) Input image. (Middle) Input im- age superimposed with predicted bounding hoxes. (Bottom) Reconstruction. (B) Learned part-level templates. Fach decoded from the same template emhedding with distortion specified by zhipe Here for visual- ization purposes, choose hae values that are equally spaced within one standard deviation the mean. Template colors indicate identity. Part bounding box colors indicate the chosen template.
6
6
5
7
6
3
B
80
7
Figure 9: (A) Generated objects compositional MNIST dataset. (B) Generated scenes compositional
MNIST dataset.
A.3 EXPERIMENTAL RESULTS
Similar to Section 4 here we show both qualitative and quantitative results on the compositional MNIST dataset. Figure A shows qualitative results of scene decomposition. Again, the bounding boxes are fairly tight, demonstrating that the nodes the probabilistic scene graph representation have been correctly associated with objects and parts in the images. We also visualize the learned part-level templates Figure 8B. Here each row shows the templates decoded from one slot of the memory with varying degree of distortion. As be seen, each slot captures visually similar digits, and all nine digits are captured. It is worth noting that the main purpose of memory is not to perfectly separate the different classes of digits, but obtain the canonical pose for the digits. This
14

---

Under review as a conference paper at ICLR 2020
B
3
9
3
18
78
8
#3
2
4
à
Z2
2
4
18
2
3
TEL
21
1A
2
9
5
0
78
5
6
43
2
33
9
9
8
3
Generalization results on compositional MNIST dataset. RICH has been trained on scenes with and 3 objects only, and tested on seenes with (A) 2 objects and (B) 4 objects. (Top) Input image. (Middle) Input image superimposed with predicted bounding boxes. (Bottom) Reconstruction.
Figure 10:
Table 3: Quantitative results object and part detection.
Dataset
Compositional MNIST dataset
Training set
1~4 objects
1&3 objects
Test set
1~4 objects
2 objects
4 objects
Object count error
0.0015
0.00064
0,00027
Object precision
0,9963
0.9899
0,9365
Object recall
0.9962
0.9898
0.9364
Part count error
0.016
0.0086
0.023
Part precision
0.9981
0.9989
0.9982
Part recall
0.9983
0.9990
0.9989
essential for the scene graph representation to infer consistent orientation for the digits. Figure SB shows that this purpose is indeed achieved, since the digits have almost the same pose within each memory slot.
Figure 9 shows the generated objects and scenes, which resemble the training images. Figure TO shows qualitative generalization results. The learned compositional hicrarchy generalizes well to unseen number of objects. We report quantitative results in Table 3 and Table 4 The counting error, precision, and recall are mainly to show that the representation is learned properly. We com- pare negative log-likelihood with VAE baseline demonstrate that the probabilistic scene graph representation can be obtained without impairing generation quality.
Finally, consider downstream task that requires reasoning of part-object relationships, and demonstrate the transferability of RICH representation. 'he task setup, model architecture, and training procedure are similar to those described in Section The difference here that we seek to predict the sum of max digit within each object, and we treat it as a 37-way classification task (possible output can be integers in [0, 36]). Experimental results in Figure II show that compared to SPAIR, the representation learned by RICH leads to approximately doubled data efficiency.
Table 4: Comparison on negative log-likelihood.
Dataset
Compositional MNIST dataset
Training set
1-44 objects
1&3 objects
Test set
I~4 objects
2 objects
objects
VAE RICH
-13096.4
-12915.7
-10832.9
-13377.1
-13436.9
-12801.6
15

---

Under review as a conference paper at ICLR 2020
Compositional MNIST Dataset
100
Tost Accuracy (%)
80
60
40
RICH SPAIR
20
10a
103
10
Training Samples
Figure 11: Comparison of data efficiency downstrcam task.
16


Reviews:
[{"review_id": "rkxOuHUXcS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-- Update --\nThank you for the detailed response and updates. The quality has improved but as mentioned in the original review, it would be good to see results on a more significant non-synthetic task before I would argue for acceptance.\n\n------------------\nThe authors propose a generative model with a hierarchy of latent variables corresponding to a scene, objects, and object parts. The method is evaluated on a synthetic dataset with manual inspection, MSE, and object counting performance. The method can form bounding boxes around objects and parts in the examples that are shown, and the MSE is similar to that of a VAE.\n\nThe idea of decomposing a probabilistic model hierarchically is potentially interesting, but this paper has drawbacks in terms of experimental quality, significance, and presentation.\n\n-- Experiments -- \nThe experiments are done on a manually constructed synthetic dataset, so it is not clear whether the proposed method would work in more realistic or more challenging settings. For instance, the dataset was constructed using a recursive process which probably does not resemble a realistic distribution of objects and their parts, and the parts are well-separated within the object. \n\nThe experiments on the synthetic dataset are not thorough, and could be more interesting with ablations, evaluating performance as the dataset parameters vary (e.g. number of objects, amount of occlusion), better baselines, and an evaluation metric rather than manual inspection.\n\n-- Significance --\nWith respect to significance, the formulation is fairly straightforward and builds off of previous techniques introduced in e.g. SPAIR. The lack of demonstrating usefulness on a downstream application limits this paper's significance; for instance the SPAIR paper used the generative model as a front-end for a card game and Atari game. The paper should more clearly demonstrate the benefits/tradeoffs of the hierarchical aspect.\n\n-- Presentation --\nThe paper leaves many terms undefined so the paper is not self contained, e.g. the 'problem of routing', 'memory buffer', 'memory template', 'variational memory addressing', 'spatial transformer' (missing a reference here). I suspect the related work is missing references, e.g. \"Unsupervised Discovery of Parts, Structure, and Dynamics\" Xu et al ICLR 2019. Finally, the paper should be proof-read for grammatical errors.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rkgZtVmCtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I thank the auhors for the detailed response. The updated paper does look more convincing, but my main concern remains - all considered datasets and tasks are synthetic and specifically made for the method. I think experiments on more real data would be crucial to show the potential of the method. Some examples I can imagine would be part segmentation in images or 3D models, or robotic control.\n\n---\n\nThe paper proposes an unsupervised approach to learning objects and their parts from images. The method is based on the \"Attend, Infer, Repeat\" (AIR) line of work and adds a new hierarchy level to the approach, corresponding to object parts. The method is evaluated on two custom synthetic dataset: one composed of simple 2D geometric shapes and another one with 3D geometric shapes. On these datasets the method successfully infers the object-part structure and can parse and reconstruct provided images.\n\nWhile the general topic of the paper is interesting, I do not think it is fit for publication. First and foremost, the experiments are very incomplete: the method is only evaluated on two custom synthetic datasets and not compared against any baselines or ablated versions of the method. Moreover, the proposed approach seems like a relatively minor modification of SPAIR (Crawford & Pineau).\n\nPros:\n1) The topic is interesting and the method generally makes sense.\n2) The method works on the tasks studied in the paper, including relatively challenging scenarios with 3D occlusions or ambiguity in assigning parts to objects.\n3) The paper shows generalization to a number of objects different from that seen during training.\n\nCons:\n1) Experiments are limited:\n1a) The method is evaluated on custom and relatively toy datasets. It is unclear if it would apply to more practical situations. While I agree that at the high level the question of decomposition of object into parts is interesting, it is still important to connect research in this direction to potential practical applications. Where could the proposed method be used? Perhaps in some control tasks, such as robotics? Could it be applied to more realistic data, such as for instance ShapeNet objects?\n1b) There are no comparisons to baselines. One might argue that baselines do not exist since there are no methods addressing the same task. Generally, it is the job of the authors to come up with relevant baselines to show that the proposed model actually improves upon some simpler methods. One variant of obtaining baselines is via ablating the proposed model. Another is by taking prior approaches, for instance object-based ones without the part decomposition, and showing that the part decomposition allows to improve upon those in some sense.\n\n2) The novelty is somewhat limited: the method seems like a relatively straightforward extension of SPAIR by adding another hierarchy layer. It might be sufficient if the experimental results would be strong, but given that they are not, becomes somewhat concerning. If the method does indeed include significant technical innovation, it might be helpful to better highlight it.\n\n3) The related work overview is extremly limited. It is authors' job to provide a comprehensive ovreview of prior literature and I cannot do it for them here, but below are a few papers that come to mind. This is by no means a complete list.\n\n[1] Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William T. Freeman, Joshua B. Tenenbaum, Jiajun Wu. Unsupervised Discovery of Parts, Structure, and Dynamics. ICLR 2019\n[2] Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, Jitendra Malik. Learning Shape Abstractions by Assembling Volumetric Primitives. CVPR 2017\n[3] Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, Leonidas Guibas. GRASS: Generative Recursive Autoencoders for Shape Structures. SIGGRAPH 2017.\n[4] Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos Kalogerakis, Subhransu Maji. CSGNet: Neural Shape Parser for Constructive Solid Geometry. CVPR 2018.\n[5] Adam R. Kosiorek, Hyunjik Kim, Ingmar Posner, Yee Whye Teh. Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects. NeurIPS 2018.\n\n4) Presentation is at times suboptimal:\n4a) It would be very helpful to have more visuals and intuitions about the functioning of the method, as opposed to equations. Equations are definitely good to have, but they are not the easiest to parse, especially by those not intimately familiar with this specific line of work. \n4b) It is quite unclear to me how and why is the memory used in the model. If this is described in another paper, it would be useful to point there, but still briefly summarize in this paper to make it self-contained.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxcv96sFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions: this submission proposes a generative framework which employs a hierarchical decomposition of scene, objects and parts. It modifies the SPAIR framework (Crawford & Pineau) and replaces the recurrent generation with parallel generation, by assuming the number of objects is known and fixed. Results are presented on two synthetic datasets with 2D or 3D shapes.\n\nAssessment:\n- The proposed model and learning framework are closely related to previous work (AIR, SPAIR). The authors replaced the \"sequential processing ... with single forward pass\", which requires the number of objects to be fixed.\n- In the proposed model, there is no constraint on the part-object hierarchy, and it is unclear whether it can learn to discover such hierarchies directly by reconstructing single static images. It is also not evaluated in the submission.\n- There are no comparisons with AIR or SPAIR on the two datasets the authors created.\n- The authors are recommended to compare with previous work on hierarchical generative modelings with objects and parts, such as (Xu et al, ICLR 2019), and other related work, such as SPIRAL (Ganin et al.)\n\nDue to the limited contribution, lack of comparison with related work and limited empirical evaluation, I recommend rejection of the submission.\n\n[1] Xu el al, Unsupervised Discovery of Parts, Structure, and Dynamics, ICLR 2019.\n[2] Ganin et al., Synthesizing Programs for Images using Reinforced Adversarial Learning.\n\n\n--------------------------------\nPost-rebuttal:\nThank you for your detailed answers to my questions and updated manuscript. The writing has indeed significantly improved and some questions (e.g. varying number of objects) have been addressed. After reading the rebuttal, my concerns remain that: (1) lack of empirical evaluation on more complex real / synthetic datasets; (2) it is still unclear to me how such hierarchy is inferred from single images. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors proposes a generative model with a hierarchy of latent variables corresponding to a scene, objects, and object parts. 

The submission initially received low scores with 2 rejects and 1 weak reject.  After the rebuttal, the paper was revised and improved, with significant portions of the paper completely rewritten (the description of the model was rewritten and a new experiment comparing the proposed model to SPAIR was added).  While the reviewers acknowledged the improvement in the paper and accordingly adjusted their score upward, the paper is still not sufficiently strong enough to be accepted (it currently has 3 weak rejects). 

The reviewer expressed the following concerns:
1. The experiments uses only a toy dataset that does not convincingly demonstrate the generalizability of the method to more realistic/varied scenarios. In particular, the reviewers voiced concern that the dataset is tailored to the proposed method

2. Lack of comparisons with baseline methods such as AIR/SPAIR and other work on hierarchical generative models such as SPIRAL.
In the revision, the author added an experiment comparing to SPAIR, so this is partially addressed.  As a whole, the paper is still weak in experimental rigor.  The authors argue that as their main contribution is the design and successful learning of a probabilistic scene graph representation, there is no need for ablation studies or to compare against baselines because their method "can bring better compositionality, interpretability, transferability, and generalization".  This argument is unconvincing as in a scientific endeavor, the validity of such claims needs to be shown via empirical comparisons with prior work and ablation studies.

3. Limited novelty
The method is a fairly straightforward extension of SPAIR with another hierarchy layer. This would not be a concern if the experimental aspects of the work was stronger. 

The AC agrees with the issues pointed by the reviewers.  In addition, the initial presentation of the paper was very poor.  While the paper has been improved, the changes are substantial (with the description of the method and intro almost entirely rewritten). Regardless, despite the improvements in writing, the paper is still not strong enough to be accepted.  I would recommend the authors improve the evaluation and resubmit. 
Decision: Reject