Generative Scene Graph Networks
Authors: Fei Deng
Zhuo Zhi
Donghun Lee
Sungjin Ahn
Venue: ICLR 2021 Poster (2021)
Keywords: object-centric representations
generative modeling
scene generation
variational autoencoders

Abstract: Human perception excels at building compositional hierarchies of parts and objects from unlabeled scenes that help systematic generalization. Yet most work on generative scene modeling either ignores the part-whole relationship or assumes access to predefined part labels. In this paper, we propose Generative Scene Graph Networks (GSGNs), the first deep generative model that learns to discover the primitive parts and infer the part-whole relationship jointly from multi-object scenes without supervision and in an end-to-end trainable way. We formulate GSGN as a variational autoencoder in which the latent representation is a tree-structured probabilistic scene graph. The leaf nodes in the latent tree correspond to primitive parts, and the edges represent the symbolic pose variables required for recursively composing the parts into whole objects and then the full scene. This allows novel objects and scenes to be generated both by sampling from the prior and by manual configuration of the pose variables, as we do with graphics engines. We evaluate GSGN on datasets of scenes containing multiple compositional objects, including a challenging Compositional CLEVR dataset that we have developed. We show that GSGN is able to infer the latent scene graph, generalize out of the training regime, and improve data efficiency in downstream tasks.

Published as a conference paper at ICLR 2021
GENERATIVE SCENE GRAPH NETWORKS
Fei Deng Rutgers University fei deig ret
Zhuo Zhi
University of California, San Diego zzhi @uciduccd... wacsd.edu
Donghun Lee
Sungjin Ahn
ETRI
Rutgers University
donghunßetri... ree kr
sjn. ahn gma com
ABSTRACT
Human perception excels at building compositional hierarchies of parts and ob- jects from unlabeled scenes that help systematic generalization. Yet most work on generative scene modeling either ignores the part-whole relationship sumes access to predefined part labels. In this paper, we propose Generative Scene Graph Networks (GSGNs), the first deep generative model that learns to discover the primitive parts and infer the part-whole relationship jointly from multi-object scenes without supervision and in an end-to-end trainable way. We formulate GSGN as a variational autoencoder in which the latent representation is a tree- structured probabilistic scene graph. The leaf nodes in the latent tree correspond to primitive parts, and the edges represent the symbolic pose variables required for recursively composing the parts into whole objects and then the full scene. This al- lows novel objects and scenes to be generated both by sampling from the prior and by manual configuration of the pose variables, as we do with graphics engines. We evaluate GSGN on datasets of scenes containing multiple compositional objects, including a challenging Compositional CLEVR dataset that we have developed. We show that GSGN is able to infer the latent scene graph, generalize out of the training regime, and improve data efficiency in downstream tasks.
as-
INTRODUCTION
Learning to discover and represent objects purely from observations is at the core human cog- nition (Spelke & Kinzler, 2007). Recent advances in unsupervised object-centric representation learning have enabled decomposition of scenes into objects (Greff et al. 2019] Lin et al.) 20206] Locatello al. 2020), inference and rendering 3D object models (Chen et 2020), and object tracking and future generation (Crawford & Pineau 2019a Jiang et al.) 2020 Lin et al. 2020a). These neuro-symbolic approaches, where the discreteness of discovered objects provides the sym- bolic representation, facilitate various desired abilities such as out-of-distribution generalization, relational reasoning, and causal inference.
In this paper, we seek to further discover and represent the structure within objects without super- vision. Our motivation that natural scenes frequently contain compositional popects--ojecttt that are composed of primitive parts. We humans can easily identify the primitive parts and recognize the whot- relationship. Representing objects as explicit composition parts is expected to be more efficient, since a vast number of complex objects can often be compositionally explained by small set of simple primitives. It also allows us to imagine and create meaningful new objects.
A well-established representation for whrt- relationships in computer graphics is called the scene graph (Foley et al.) 1996). It is a tree whose leaf nodes store models of primitive parts, and whose edges specify affine transformations that compose parts into objects.
While in computer graphics, the scene graph is manually constructed for rendering, in this paper, interested in inferring the scene graph from unlabeled images. To this end, propose Generative Scene Graph Networks (GSGNs), We formulate this model as a variational autoencoder (Kingma& & Welling 2013 Rezende al. 2014) whose latent representation is probabilistic scene

---

Published as a conference paper at ICLR 2021
graph. In the latent tree, each node is associated with an appearance variable that summarizes the composition up to the current level, and each edge with a pose variable that parameterizes the affine transformation from the current level to the upper level.
The design of the GSGN decoder follows the rendering process of graphics engines, but with differ-
entiable operations helping the encoder learn inverse graphics (Tieleman 2014] [u et al. 2017 Romaszko et al.) 2017: Yao et al.) 2018 Deng et al.] 2019). As a result, the pose variables inferred
by GSGN are interpretable, and the probabilistic scene graph supports symbolic manipulation by
configuring the pose variables.
One major challenge is infer the structure of the scene graph. This involves identifying the parts and grouping them into objects. Notice that unlike objects, parts are often stitched together and thus can have severe occlusion, making it hard separate them. Existing methods for learning hierarchical scene representations circumvent this challenge by working on single-object scenes (Kosiorek et al. 2019) and also providing predefined or pre-segmented parts (Li et al., [017] [ Huang al.) (020). In contrast, GSGN addresses this challenge directly, and learns to infer the scene graph structure from multi-object scenes without knowledge of individual parts.
Our key observation is that the scene graph has recursive dracttre-inferring the structure of the tree should be similar to inferring the structure of its subtrees. Hence, we develop top-down inference process that first decomposes the scene into objects and then further decomposes each object into its parts. This allows us to reuse existing scene decomposition methods such as SPACE inference module shared at each level of the scene graph. However, we find
(Lin et al. 2020b) that SPACE E has difficulty separating parts that have severe occlusion, possibly due to its complete reliance on bottom-up image features. Therefore, simply applying SPACE for decomposition each level will lead to suboptimal scene graphs. To alleviate this, GSGN learns a prior over plausible scene graphs that captures typical compositions. During inference, this prior provides top-down information which is combined with bottom-up image features to help reduce ambiguity caused by occlusion.
For evaluation, we develop two datasets of scenes containing multiple compositional 2D and 3D objects, respectively. These can be regarded as compositional versions of culti-dSprites (Greff et al., 2019) and CLEVR (Johnson et al.) 2017), two commonly used datasets for evaluating unsupervised object-level scene decomposition. For example, the compositional 3D objects our dataset are made up of shapes similar to those in the CLEVR dataset, with variable sizes, colors, and materials. Hence, we name our 3D dataset the Compositional CLEVR dataset.
The contributions of this paper are: (i) we propose the probabilistic scene graph representation that enables unsupervised and end-to-end scene graph inference and compositional scene generation, (ii) develop and release the Compositional CLEVR dataset to facilitate future research on object compositionaliiy. and (iii) we demonstrate that our model is able to infer the latent scene graph, shows decent generation quality and generalization ability, and improves data efficiency in down- stream tasks.
RELATED WORK
Object-centric representations. Our model builds upon a line of recent work on unsupervised object-centric representation learning, which aims eliminate the need for supervision in struc- tured scene understanding. These methods learn a holistic model capable of decomposing scenes into objects. learning appearance representations for each object, and generating novel scenes all without supervision and in an end-to-end trainable way. We believe such unsupervised and holistic models are more desirable, albeit more challenging to learn. These models can be categorized into scene-mixture models (Greff et 2017 [019 Burgess et al.) 2019 Engelcke et al. 2020 Locatello ct al. 2020) and spatial-attention models (Eslami et al.) 2016 Crawford & Pineau| 2019b Lin et al.)
2020b Jiang& & Ahn, 2020). Compared to these models, we go step further by also decomposing objects into parts. We use graph, because they explicitly provide object positions, unlike scene- mixture models. We combine the inference module with a learned prior to help improve robustness to occlusion. This also allows sampling novel scenes from the prior, which is not possible with spatial-attention models.
models as the inference module at each level of the scene
2

---

Published as a conference paper at ICLR 2021
"
C
glose
40
(A) Probabilistic scene graph representation. Each node represents an entity in the scene, and as- sociated with an appearance variable. Each edge is associated with a pose variable that specifies the coordinate transformation between the child node and the parent node. (B) Spatial attention during inference. We first decompose the scene into high-level objects, and then attend to each object figure out the constituent parts. (C) Recursive decoding process (a single recursive step shown). The image patch Xaj mask lib, an internal node u1 are decoded from the image patches and masks of all its children nodes.
Figure 1:
Hierarchical scene representations. Modeling the part-whole relationship in scenes has attracted growing interest, and has been utilized for improving image classification (Sabour et al. 2017 Hin- ton et al. 2018 Kosiorek et al. 2019). parsing, and segmentation (Zhu et al.] 2008). However, these models have been applied to scenes with dominant object only, and cannot perform
gen- 3D shape modeling also learns the phrt-whole relationship
cration. Recent work on assembly-base
(Tulsiani et al.. 2017 Li al. 2017] Zhu et al. 2018: Huang et al. 2020 Kania et al. 2020), but these methods require predefined or pre-segmented parts as input, and can only model single shapes with no background. By contrast,
model learns the part-whole relationship from multi-object scenes without knowledge of individual parts. There has also been work on 3D part decomposition (Chen al. 2019 Deng al., 2020), but they require voxels point clouds input, and typi- cally focus on geometry (c.g., part occupancy) without learning to represent appearance (e.g.. color, material). Part hierarchies have also been used for shape generation (Mo al. 2019), where the hi- erarchy is provided as input rather than inferred from the input. Our approach infers compositional structures from static scenes, and is orthogonal to methods that use motion cues for decomposing dynamic scenes (Xu et al.] (2019) and methods that infer physical interactions from dynamic scenes (Li et al. 2020; Stanić et al. (020). Hinton (202I) recently proposed an iterative procedure that expected to form the part hierarchy through multiple rounds of message passing among adjacent levels. While our model works without iterative message passing, we believe this is important for parsing more complex scenes.
Hierarchical latent variable models. Our model can be regarded as a hicrarchical latent variable model, and is inspired by several recent advances (Bachman) 2016 Sønderby et al. (2016] Zhao al.) 2017 Maaløe et al.; 2019) that have achieved impressive generation quality. While these methods Tocus on designing the hierarchical structure and training method that harness the full expressive power of generative models, our goal is to learn the hierarchical structure from unlabeled images that captures the compositional relationship among symbolic entities like objects and parts.
GENERATIVE SCENE GRAPH NETWORKS
3.1 GENERATIVE PROCESS
We assume that the image x generated by a set foreground variables collectively denoted Zig and background variable Zog as follows:
/ʃ
p(x)
p(x Zig, Zng) p Zzg (2gz) p(Ztg) dzig dZigg
To represent the compositional structures within foreground objects, GSGN models Øig as a tree- structured probabilistic scene graph, as shown in Figure DA. Each leaf node represents a primitive entity that is not further decomposed. Each internal node represents an abstract entity that is com- posed from its children nodes. Similar to graphics engines, the composition is modeled affine
3

---

Published as a conference paper at ICLR 2021
transformations, and is specified by the relative pose (including rotation, scaling, and translation) of each child node u with respect to its parent pa(v). We use a pose variable z to represent the relative pose, and associate it with the corresponding edge. We also associate an appearance variable z.ppr with each node v. It is expected to represent the appearance of entity in its canonical pose summarizing all lower-level composition in the subtree rooted v. In particular, the appearance variable z.ppr at the rool node r summarizes the full scene. Due to this summarization assumption, given zypee, we can generate the pose and appearance variables for all children nodes of in a con- ditionally independent way. Hence, for a given tree structure with V being the set of all nodes, the prior over foreground variables can be factorized according to the tree structure:
p(zig) = p(zzppr) ILvEV p(zz)
pose
Z
appe pa(u)) p(z,
(PppPrr 2
pa(())
appr
(2)
Here we further assume conditional independence between zpose and zappr, since in graphics en- gines, one should be able to specify the pose and appearance separately.
Representing tree structures. The above factorization only works for given tree structure. To deal with variable tree structures, we need to include them in the latent representation as well. We start by setting a maximum out-degree for each node so that the total number of possible structures is bounded. To determine the structure, it then suffices to specify the presence of each possible edge. Hence, for an arbitrary edge between node v and its parent, we introduce a Bernoulli variable 2, to indicate its presence. If zpres O, meaning the edge is not present, then the pose variable associated with the edge along with all variables in the subtree rooted at u are excluded from the probabilistic scene graph. More precisely, let us define zeres to be the product of all the presence variables along the path from root r to node U:
zpres 11
zpres
Zpres for V V V (r). 2.rres
The foreground variables now become 2íg factorizes as follows:
(zpppr) U {zpres zpase 2:prr MEV, and the prior
p(zig) - p(z,Pr) HOEE [p(2]
Appe pe(m))
spree putv)
[p(zp0se
z pp(())
p 2
appr
2 pa(()) ) | p
(4)
We implement the pose and appearance variables as Gaussian variables, p(z,pPr) = NO 1), and
the parameters of each conditional distribution are output by an MLP.
Differentiable decoder. We design the decoder to follow the recursive compositing process in graphics engines, helping the encoder learn inverse graphics. First, for each leaf node v, we use neural network y(.) to decode its appearance variable into small image patch x, and a (close to)
binary mask m, (X.; m.,) = (ZZ.Prr) Here, g()) implemented as a spatial broadeast decoder (Watters et al. 2019) optionally followed by sub-pixel convolutions (Shi et al. 2016).
We then recursively compose these primitive patches into whole objects and the full scene by ap- plying affine transformations parameterized by the pose variables. Specifically, let u be an internal node, and ch(u) be the set of its children. We compose the higher-level image patch X and mask m. follows:
x. = =vvch(u) pres a, 2 OTSTSTIL '(te, 2po0e),
(5)
m. Enechhu) pres a. 3 ST ST
(me, 2 pase).
(6)
Here, @ denotes pixel-wise multiplication, and ST 1 is an inverse spatial transformer (Jaderberg eet al. (015) that differentiably places x, and m. into the coordinate frame of the parent node u. To deal with occlusion, we include relative depth in z,cce, and compute a transparency map og by the softmax over negative depth values. This ensures that entities with smaller depth will appear in front of entities with larger depth. See Figure [ for an illustration.
When we reach the root node, we obtain an image &, and mask m of all foreground objects. We then use a spatial broadcast decoder (Watters et al., 2019) to decode Zbg into a background image &bg. The full scene can be modeled pixel-wise mixture of foreground and background, where mn serves as the mixing weight:
p(x Zig bbg)= = m A( &, of 1) + (1 m,) N &bg og1)
Here, oig and 0bg are hyperparameters.
We use to refer both node in the prohahilistic scene graph and the entity that node represents.
4

---

Published as a conference paper at ICLR 2021
3.2 INFERENCE AND LEARNING
Since computing p(x) in Equation I involves an intractable integral, we train GSGN with variational inference. The approximate posterior factorizes similarly as the generative process:
p(zig: Zug x) g(Zig, Zog
x) g(Zig x) ((bbg Zig. x)
(8)
To infer the foreground variables, we make the key observation that the probabilistic scene graph has recursive structure. This suggests recursive inference process, in which the children nodes
of the root node are first inferred, and then inference recursively performed within each subtree rooted at the children nodes. Hence, we design top-down factorization:
(Zigg (x) qZ,,pDr x) IoEeeer (g(2
pa(()) Xpo(v)) putv)
gprea
[g(2 p00e
appr pp((s) Xxo(w))) M pr
x HovEVrr) (z(z,
paree pa(v)).X.)) x. 22
where the inference of child node is conditioned on the inferred appearance variable z 2 of its appr parent. This crucially provides top-down information for separating lower-level entities, since pa(v) summarizes lower-level composition. We combine this top-down information with bottom-up image features. Specifically, we use spatial attention to crop local image region X, that is expected to capture the entity v. This provides more relevant information for inferring the appearance and struc- ture of entity root r to node v. More precisely, we define x, x, and recursively extract x,, ST =TT(xpal (Xpa(v): 2pose) using a spatial transformer ST (Jaderherg et al. 2015). as shown in Figure DB.
ppa(t)
The region x, is specified by all the predicted pose variables along the path from
Parameter sharing between prior and posterior. Inspired by Ladder VAES (Sonderby et al,, 2016), we find it beneficial to share parameters between the prior and the posterior. Specifically, we implement each factorized posterior
product of the corresponding prior and posterior obtained solely from image features. This also allows us to reuse existing scene decomposition methods such SPACE (Lin et al. 2020b) as inference module of GSGN. For example.
g( (PP
pa(u)) xx) x p(z.ppr p)(()) ( x.) p(z (2P
2 appr
ISPACE (ZACEE(ZPPL)) (Z.PPr (x.). (10)
Training. We use reparameterization trick (Kingma & Welling
2013
Rezende et al. 2014) to
sample the continuous pose and appearance variables, and
trick (Jang al. 2016
Maddison et al. 2016) to sample the discrete presence variables. The entire model can be trained end-to-end via backpropagation maximize the evidence lower bound (ELBO))
L [p([ Zig; Zng) DKI.(g(Z;, xx || [(()) - DKt.(y/ (g(2g) Zigs x) || 25)
(11)
Auxiliary KL loss. As pointed out by recent work (Chen et al. 2020), learnable priors have diffi- culty reflecting our prior knowledge. Hence, similar Chen et al. (2020). we introduce auxiliary KL terms between the posterior and some fixed prior that reflects our preference. In particular, we find important to introduce KL term between qzzpre 1 zapee with low success probability to encourage sparse tree structures.
"pa(v)> Xpa(n) and a Bernoulli prior distribution
4 EXPERIMENTS
Datasets. In order to evaluate GSGN's ability to discover scene graph structures, we develop com- positional versions of Multi-dSprites (Greff et al. 2019 and CLEVR (Johnson et al.) 2017). two commonly used datasets for evaluating unsupervised object-centric representation learning models. We refer to our new datasets as 2D Shapes and Compositional CLEVR datasets, respectively. For each dataset, we define three types of primitive parts, and ten types of objects composed from these parts. Three of the object types contain a single part, another three contain two parts, and the re- maining four contain three parts. To construct a scene, we randomly sample the number of objects (between one and four) and their types, sizes, positions, and orientations. The color and material of each part are also randomly sampled, thus covering broad range of compositional structures. Each dataset consists of 128x x 128 color images, split into 64000 for training, 12800 for validation, and 12800 for testing.
5

---

Published as a conference paper at ICLR 2021
Object
Inpit
Input
Impur
boox
strction
Figure Visualization inferred scene graphs on Shapes dataset.
Fall scene
Objoct
Cijjet
Olject
Object
Recon-
Ingux boox
Recon- struction
Inprit -boxx
Recon-
Irput 3boxs
Recos- struction
Iapur tbox
Recon
Expu
sinacton
Parts
sirctionn
Puns
Puns
strucios
Pani
Figure 3: Visualization of inferred scene graphs Compositional CLEVR dataset.
GSGN implementation. While our formulation of GSGN allows representing and inferring scene graphs of arbitrary depth, in our experiments, we have only investigated the effectiveness of three- level GSGN as a proof-of-concept, containing part-, object-, and scene-level representations. We set the maximum out-degree
for each node. We also implemented another version, denoted GSGN-9, where the maximum out-degree is 9. This is to evaluate the effectiveness of our model when structural hyperparameters are mismatched to data, and show some potential for scalability.
Baselines. Previous work hierarchical scene representations assumes single-object scenes (Ko- siorek et al. 2019) and requires predefined or pre-segmented parts (Li et al., 2017 Huang et al, 2020), and thus cannot work a state-of tfeehhrrrt non- hierarchical scene decomposition model. Although SPACE cannot infer the scene graph, its hyperparameters can be tuned decompose the scene into either objects (SPACE-O) or parts (S This allows us to compare the inference quality of individual nodes.
datasets. Hence, we compare with SPACE din al. (020b),
We consider ablated versions of GSGN. CSGN-No-Share does not share parameters between prior and posterior, namely does not factorize the posterior as in Equation I0 Instead, the posterior mean and variance computed by another network that takes in both spp and x GSGN-No-Aux directly optimizes the ELBO without auxiliary KI. terms.
Ablations.
pa(())
Scene graph inference. We visualize the inferred scene graphs in Figure and Figure Here, the bounding boxes visualize the inferred pose variables (including rotation, scaling, and transla- tion), and the reconstructions are obtained by feeding the inferred appearance variables through the decoder. Empty slots indicate that the corresponding presence variables are zero. As can be seen, GSGN is able to correctly separate the parts and group them into objects, even when objects close (Figure 2 Row 1-4, Figure 3 Row 1-2), parts have similar color (Figure [ Row 4-5, Figure Row 3), and there is occlusion between objects and parts (Figure Row 4-5).

---

Published as a conference paper at ICLR 2021
Table 1: Quantitative results on 2D Shapes dataset.
Object Count Accuracy
Object Fl Score
Part Count Accuracy
Part
Metric
ELBO
Fl Score
SPACE-O O
3,96
99.79%
99.95%
SPACE-P P
3,95
98.75%
96.73%
GSGN
3.96
99.76%
99.85%
99.54%
99.91% %
GSGN-No- -Ssaare
3.96
99.72%
99.90%
99.49% 95.79%
99.89%
GSGN-No -No-Aux
3.96
25.00%
72.75%
97.34%
Table 2: Quantitative results Compositional CLEVR dataset.
Object Count Accuracy
Object F1 Score
Part Count Accuracy
Part FI Score
Metric
ELBO
SPACE-O
3,50
98.02%
99.17%
SPACE-P
3,48
92.71%
99.03%
GSGN
3.49
98.66%
99.37%
98.63%
99.77 .7%7
GSGN-9
3.49
96.27%
97.21
97.73%
97.59%
GSGN-No-Share
3.50
96.21%
98.26%
97.09%
99.38%
GSGN-No-Aux
3.50
25.13%
72.87%
12.38%
79.09%
Table 3: Robustness occlusion Compositional CLEVR dataset.
Min Visible
<100
100~200
>00
Pixels Per Part
Part Count Accuracy
Part Recall
Part Count Accuracy
Part Recall
Count
Part
Metric
Accuracy
Recall
SPACE-P
12.24%
86.03%
85.66%
97.95%
96.11%
99.48%
GSGN
95.92%
98.93%
98.33%
99.77%
98.76%
99.86%
GSGN-9
89.80%
97.35%
96.92%
97.85%
98. 12%
97 97.62%
GSGN-No-Share
85.71%
96.34%
96.13%
99.15%
97.56%
99.46%
We report quantitative results in Table and Table Here, counting accuracy measures the correct- ness of the number of nodes in the inferred scene graph. F1 score is the harmonic mean precision is sufficiently close to groundtruth entity.
and recall. We define node to be true positive Hence, Fl score reflects the inferred nodes indeed capture entities. We measure closeness by the distance between center positions. The distance threshold for objects and parts are 10 and 5 pixels respectively. As can be seen, all models obtain similar reconstruction quality. GSGN-No-Aur fails to infer the scene graph structure. We find that it tends to set all presence variables to be one, sulting in redundant nodes. With auxiliary KL terms, GSGN and GSGN-No-Share are able to infer sparse structures, achieving comparable object-level decomposition performance with SPACE-O.
On the Compositional CLEVR dataset, GSGN achieves better part-level decomposition than SPACE-P. We observe that this is because SPACE-P has difficulty separating parts that have vere occlusion. We further investigate this by splitting the test set into three occlusion levels. Here measure the amount of occlusion in an image by the minimum number of visible pixels per part. As shown in Table 3 when an image contains part that has less than 100 visible pixels, SPACE-P performs significantly worse than GSGN. Its low recall suggests that SPACE-P tends to miss oc- cluded parts. We also find GSGN-No-Shann that parameter sharing helps better combine top-down information from the prior to reduce ambi- guity caused by occlusion. For GSGN-9, although its number of leaf nodes is far more than the maximum number of parts per scene in our dataset, it only shows a slight drop in performance when compared to GSGN, and it is still much better than SPACE-P at identifying parts that have severe occlusion.
performs worse than GSGN in this scenario, indicating
The counting accuracy and Fl score evaluate the object- and pert-level separately, without taking into account the overall tree structure. To directly measure the quality of the inferred tree structure, perform manual inspection on 100 test images from each dataset. GSGN achieves 100% and

---

Published as a conference paper at ICLR 2021
Figure 4: Image manipulation results. (A) Object-level manipulation. (B) Part-level manipulation.
99% structural accuracy on the 2D Shapes and Compositional CLEVR datasets, respectively. We also consider heuristic baseline, which starts from the parts predicted by SPACE-P, and groups the parts whose bounding boxes overlap. This baseline achieves 95% and 80% structural accuracy on the two datasets, respectively.
Scene graph manipulation. The scene graphs inferred by GSGN have interpretable tree structures and pose variables. Therefore, new objects and scenes can be composited by directly manipulating the latent variables in an inferred scene graph. In Figure 4A, we show object-level manipulations. We manipulate the inferred scene graph of the fourth scene of Figure 3 by changing the scale (first row) and z-coordinate (second row) of an object in the scene. As can be seen, object occlusion handled correctly. We can also make these modifications after replacing the object with some other object from a different scene (third and fourth rows). This is achieved by replacing the subtree corresponding to that object. Because the subtree stores the relative pose (rather than the absolute pose) of the parts, the part structure of the newly added object can be automatically preserved. In Figure 4B, we show part-level manipulations on the first scene Figure 3 In the first row, we modify the size of the bronze ball. This requires changing both the relative position and scale of the ball, so that remains in contact with the green cube below it. In the second row, we replace the bronze ball with other parts that can be found in the five scenes of Figure 3 This is achieved by replacing the appearance and scale variables with those of the new part, and recalculating the position so that the new part will appear right above the green cube. We note that this allows novel objects to be composited that have never been seen during training (see the last two columns).
Object and scene generation. GSGN able to generate objects and scenes by sampling the la- tent variables from the learned prior and feeding them through the decoder. We show generation results in Figure We find that GSGN has captured many predefined object types in the two datasets considered, and also managed to come up with novel compositions. The generated scenes are also reasonable, with moderate distance and occlusion between objects. The objects generated by SPACE-O are much less interpretable, since SPACE does not explicitly model the part-whole relationship. Also, SPACE cannot generate meaningful scenes, because the presence variables sam- pled from its prior will almost always be zero, leading to object being put onto the scene.
Generalization performance. We evaluate GSGN's capacity to generalize to scenes with novel number of objects. We report the metrics in Table and 1 A). As can be seen, GSGN demonstrates quite decent generalization performance.
Data efficiency in downstream tasks. The compositional structure learned by GSGN can be useful in downstream tasks that require reasoning of part-object relationships. Here we consider a classi- fication task. The input images generated from the same distribution but with different random
8

---

Published as a conference paper at ICLR 2021
Figure 5: (4) Objects sampled from GSGN learned prior. (B) Objects sampled from SPACE-O prior. (C) Scenes sampled from GSGN learned prior.
seeds. The label for each image is obtained by first counting the number of distinct parts within each object, and then summing the count over all objects. Here, two parts are considered the same if they have the same shape. regardless of their pose, color, and material. We expect the represen- tation learned by GSGN to bring better data efficiency compared to SPACE. Notice that SPACE-P cannot group the parts by itself and will fail in this task. Hence, compare with SPACE-O, whose object-level representation implicitly encodes the constituent parts. We use pretrained GSGN and SPACE-O obtain the latent representation for each image, and then train classifier top of the representation to predict the label. The classifier for SPACE-O first feeds the appearance variable of each object through an MLP. and then performs a learned weighted sum aggregate them into a scene-level representation vector, which is fed to another MLP to output class probabilities. The classifier for GSGN first aggregates the appearance variables of parts into object-level representa- tion vectors, and then performs the same operations as the classifier for SPACE-O. The tree structure learned by GSGN crucially allows us to use a graph net (Battaglia et al., 2018) within the subtree of each object for aggregation, providing more inductive bias for extracting interactions. We train the two classifiers using 64, 128, 256, 512, 1024, 2048, 4096, 8192, and 16000 training samples. We choose the best learning rate for both classifiers using fixed validation set of size 12800. We report classification accuracy on a fixed test set also of size 12800. As shown in Figure 6 (Section (). GSGN approximately doubles the data efficiency on this downstream task compared to SPACE-O.
5 CONCLUSION
We have proposed GSGN, the first deep generative model for unsupervised scene graph discovery from multi-object scenes without knowledge of individual parts. GSGN infers the probabilistic scene graph by combining top-down prior and bottom-up image features. This utilizes the higher-level ap- pearance information to guide lower-level decomposition when there is compositional ambiguity caused by severe occlusion. GSGN is also able to generate novel -ut-of-distributtio objects and scenes through scene graph manipulation. While we have demonstrated the effectiveness of three- level GSGN
prrof-off-oncepp it remains open question whether GSGN scale more realistic data, with deeper hierarchies and more complex appearance. Interesting future directions include developing recurrent module dynamically control the depth of the scene graph, using it- erative inference as suggested by Hinton (202I) for better coordination across levels, and improving the single-level inference module.
ACKNOWLEDGMENTS
This work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government [21ZR1100, A Study of Hyper-Connected Thinking Internet Technology by autonomous connecting, controlling and evolving ways]. The authors would like to thank Skand Vishwanath Peri, Chang Chen, and Jindong Jiang for helpful discussion, and anony- mous reviewers for constructive comments.

---

Published as a conference paper at ICLR 2021
REFERENCES
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrrnkk. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE transac- tions pattern analysis and machine intelligence, 34(1 34(11):2274-2282 2012.
Philip Bachman. An architecture for deep, hierarchical generative models. In Advances in Neural Information Processing Systems, pp. 4826 4834, 2016.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Banchez-Gonzake Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 2018.
Jörg Bornschein, Andriy Mnih, Daniel Zoran, and Danilo Jimenez Rezende. Variational memory addressing in generative models. In Advances in Neural Information Processing Systems, pp- 3920-3929, 2017.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representa- tion. arXiv preprint arXiv: 2019.
Chang Chen, Fei Deng, and Sungjin Ahn. Object-centric represttttonn and rendering of 3D scenes. arXiv preprint arXiv:2006.06/. 2020.
Zhiqin Chen, Kangxue Yin, Matthew Fisher, Siddhartha Chaudhuri, and Hao Zhang. BAE-NET: Branched autoencoder for shape co-segmentation. In Proceedings of the IEEE International Con- ference on Computer Vision, pp. 8490-8499, 2019.
Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. arXiv preprint arXiv:1911.00033 2019a.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 33, pp. 3412-3420, 2019b.
Boyang Deng, Simon Kornblith, and Geoffrey Hinton. Cerberus: A multi-headed derenderer. arXiv preprint arXiv:1905.11940 2019.
Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi. CvxNet: Learnable convex decomposition. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 31-44, 2020.
Martin Engelcke, Adam Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative
scene inference and sampling with object-centric latent representations. In International Confer- ence on Learning Representations, 2020. URL htttps://openreviiet net/ forum?id
BkxfaTVFwE w
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hin- ton, al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
Pedro Felzenszwalb and Daniel P Huttenlocher. Efficient graph- based image segmentation. Inter- national journal of computer vision, 1981 2004.
James D Foley, Foley Dan Van, Andries Van Dam, Steven K Feiner. John F Hughes, J Hughes, and Edward Angel. Computer graphics: principles and practice, volume 12110. Addison Addison-Wesley Professional, 1996.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2017.
10

---

Published as a conference paper at ICLR 2021
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pp. 2424-2433, 2019.
Irina Higgins, Loic Matthey, Arka Pal. Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning basic visual concepts with a con- strained variational framework. International Conference on Learning Representations. 2017.
Geoffrey Hinton. How to represent part-whole hierarchies in a neural network. arXiv preprint arXiv:2102. 12627, 2021.
Geoffrey E. Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with EM routing. In Interna- tional Conference on Learning Representations, 2018. URL htt.ttt:///ppeeeeeeeeeeeeee
net/
Jialei Huang, Guanqi Zhan, Qingnan Fan, Kaichun Mo. Lin Shao, Baoquan Chen, Leonidas Guibas, and Hao Dong. Generative 3D part assembly via dynamic graph learning. arXiv preprint arXiv:2006.0779) 07793, 2020.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Ad- vances neural information processing systems, pp. 2017-2025, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-Softmax. arXiv preprint arXiv:1611.011 111111111.014,,, 2016.
Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. In Advances in Neural Infor-
mation Processing Systems, pp. 12572-12582, 2020.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. SCALOR: Generative world models with scalable object representations. In International Conference on Learning Representations, 2020. URL https://openreeee/ee http:/wwwwww...eeeeeeeeeeeeeeeee nett/oorummo==SSxxkoo
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei. C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic datasct for compositional language and elementary visual
reasoning. In CVPR, 2017.
Kacper Kania, Maciej Zieba, and Tomasz Kajdanowicz. UCSG-NET - -Unsupervised discovering of constructive solid geometry tree. In Advances in Neural Information Processing Systems, pp- 8776-87866 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1xXiv:1412.60 met.:.12266990 6980, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 14, 2013.
arXiv preprint
Adam Kosiorek, Sara Sabour. Yee Whye Teh, and Geoffrey E Hinton. Stacked capsule autoencoders. In Advances Neural Information Processing Systems, pp. 15486-15496. 2019.
Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. GRASS: Generative recursive autoencoders for shape structures. ACM Transactions on Graphics (TOG), 36(4):52, 2017.
Yunzhu Li, Antonio Torralba, Animashree Anandkumar, Dieter Fox, and Animesh Garg. Causal discovery in physical systems from videos. arXiv preprint arXiv:2007.0063/ 2020.
Zhixuan Lin, Yi-Fu Wu, Skand Peri, Bofeng Fu, Jindong Jiang, and Sungjin Ahn. Improving gen- erative imagination in object-centric world models. In International Conference Machine Learning, 2020a.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang. and Sungjin Ahn. SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020b.
U http::/://ppaaeeeeeeeeeee//eeeer
wwt. wwtpwe/wwwwweeetteeeeeeeeeee 03y5YDH

---

Published as a conference paper at ICLR 2021
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centrie learning with slot atten- tion. arXiv preprint arXiv:2006. 5505, 2020.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: A very deep hierarchy of latent variables for generative modeling. In Advances in neural information processing systems, pp. -358, 2019.
Chris J Maddison. Andriy Mnih, and Yee Whye Teh. The Concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.0077 2016.
Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas. Struc-
tureNet: Hierarchical graph networks for 3D shape generation. arXiv preprini S0kiv:9908.00555 2019.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML, 2010.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. International conference on machine learning, pp. 1278-1286. PMLR. 2014.
Lukasz Romaszko, Christopher KI Williams, Pol Moreno, and Pushmeet Kohli. Inversse-- Graphics: Obtaining a rich 3D explanation of a scene from a single image. In Proceedings of the IEEE International Conference on Computer Vision. pp. 851-859, 2017.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856 3866, 2017.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874-1883, 2016.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pp. 3738-3746, 2016.
Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89-96, 2007.
Aleksandar Stanić, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Hicrarchical relational infer- ence. Workshop Object-Oriented Learning at ICML, 2020.
Tijmen Tieleman. Optimizing neural networks that generate images. University of Toronto (Canada), 2014.
Shubham Tulsiani, Hao Su, Leonidas J Guibas, Alexei A Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2635-2643, 2017.
Nicholas Watters, Loic Matthey, Christopher Burgess, and Alexander Lerchner. Spatial Broadcast decoder: A simple architecture for learning disentangled representations in VAEs. arXiv preprint arXiv:1901.07017, 2019.
Xing Wei, Qingxiong Yang, Yihong Gong, Narendra Ahuja, and Ming-Hsuan Yang. Superpixel hierarchy. IEEE Transactions on Image Processing, 27(10)48888443 2018.
Jiajun Wu. Joshua B Tenenbaum, and Pushmeet Kohli. Neural scene de-rendering, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 699-707, 2017.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018
12

---

Published as a conference paper at ICLR 2021
Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Unsupervised discovery of parts, structure, and dynamics. In International Confer- Learning Representations, 2019. URL http:::://pperevie.... net/forum?id
rJel 01C577
Shunyu Yao, Tzu Ming Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, Bill Freeman, and Josh Tenenbaum. 3D-aware scene manipulation via inverse graphics. Advances in Neural Informa- tion Processing Systems, pp. 1887-1898 2018.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep gener- ative models. In Proceedings of the 34th International Conference on Machine Learning-Vor Volume 70, pp.
JMLR. org. 2017.
Chenyang Zhu, Kai Xu, Siddhartha Chaudhuri, Renjiao Yi, and Hao Zhang. SCORES: Shape com- position with recursive substructure priors. In SIGGRAPHI Asia 2018 Technical Papers, pp. 211. ACM, 2018.
Long Leo Zhu, Chenxi Lin, Haoda Huang, Yuanhao Chen, and Alan Yuille. Unsupervised structure learning: Hierarchical recursive composition, suspicious coincidence and competitive exclusion. In European Conference on Computer Vision, pp. 759-773. Springer, 2008
13

---

Published as a conference paper at ICLR 2021
A
GENERALIZATION PERFORMANCE
Table 4: Generalization performance on 2D Shapes dataset.
Training Set
1 & 3 Objects
Test Set
2 Objects
4 Objects
Model
SPACE-O
SPACE-P
GSGN
SPACE-O
SPACE-P
GSGN
ELBO
4.00
3.99
4.00
3.85
3.83
3.84
Object Count Accuracy
99.47%
99.51%
100.0%
100.0%
Object F1 Score
99.89%
99.73%
100.0%
99.47%
Part Count Accuracy
98. 12%
99.21%
96.82%
99.39%
Part F1 Score
95.66%
99.86%
96.50%
99.95%
Table 5: Generalization performance Compositional CLEVR dataset.
Training Set
3 Objects
Test Set
Objects
Objects
Model
SPACE-O
SPACE-P
GSGN
SPACE-O
SPACE-P
GSGN
ELBO
3.62
3.60
3,60
3.06
3.10
3.02
Object Count Accuracy
98.27%
98.93%
88.86%
87.78%
Object F1 Score
99.64%
99.72%
96.72%
96.41% %
Part Count Accuracy
93.02%
98.69%
84.96%
87.20%
Part F1 Score
95.66%
99.71%
98.92%
97.99%
B DATA EFFICIENCY IN DOWNSTREAM TASKS
2D Shapes
Compositional CLEVR
100
100
Test Accuracy (%)
Test Accuracy (%)
60
40
GSGN SPACE-O
GSGN SPACE-O
20
20
10"
10
10
102
10s
104
Training Samples
Training Samples
Figure Comparison of data efficiency downstream classification task that requires reasoning of the part- whole relationship.
C LEARNING PRIMITIVE TEMPLATES
is often reasonable to assume that the vast number of complex entities can composed from only modest number of primitive entities. Identifying such primitive entities through discrete latent vari- ables would bring additional interpretability. Hence, we consider an extension to GSGN (denoted GSGN-Mem) where the leaf nodes become pointers jointly learned memory M of primitive templates. Each slot in M stores the low-dimensional shape embedding of a learned primitive part. These embeddings To link the probabilistic scene graph with the memory primitives, we decompose the appearance variable zappr for each leaf node into zappr (2æddr 2 What). Here, 2nddr is one-hot addressing
considered model parameters, and trained jointly via backpropagation.
14

---

Published as a conference paper at ICLR 2021
scene
Ofject
Olyeet
Object
Object
Recoe- struction
Eaput Bbox
Recat- sinuction
Ispu: Mbox
Recoer unyctios
Iuput bhos
Recour stracion
Iaput blox
Recor- sirution
Irpui
Parts
Pars
Part
Part
Figure 7: Visualization of graphs inferred by GSGN-Mem Compositional CLEVR dataset.
Figure 8: Learned memory of primitive parts. Each column corresponds to one slot. The first row shows the decoded mask that captures the shape of the parts. The remaining rows samples from the learned prior that fill plausible color, material, and lighting effects.
variable that points to one of the slots in M, and z,hhat is continuous variable that explains the remaining variability in the appearance of primitives, such as color, material, and lighting effects. Similar to Bornschein et al. (2017), we factorize the prior as:
pp(())
p(zaddr
appr M) p(2, ([zzddrr)
(12)
where M[2eddr] is the retrieved memory content.
To decode from the templates, first retrieve the embedding indexed by zaddr and decode into a (close to) binary mask m. g[M[z.ddrr) that captures the shape of v. We then add color, material, and lighting effects by applying multiplicative modification controlled by z what and obtain x, m. @ hz)))s)) Here, @) denotes pixel-wise multiplication, g(.) is implemented as spatial broadcast decoder (Watters al. 2019), and h(.) spatial broadcast decoder followed by sub-pixel convolutions Shi et al.) 20)
trained GSGN-Mem on Compositional CLEVR dataset. We find able infer the scene graph, as we show qualitatively in Figure 7 The primitives have learned the appearance of all the parts in the dataset, while reserving some memory capacity for potential future use (see Figure 8)
D IMPLEMENTATION DETAILS
D.1 FOREGROUND INFERENCE
We provide the implementation outline for computing q(Zgg x) in Algorithm I and Algorithm 2 The inference carried out in a top-down fashion, starting at level that corresponds to the root node, The core inference module, GSGN(/+1) takes as input the appearance variable zaprr and the local image region x, for node in level 1, and outputs the latent variables Zek(v) and local image regions Xch(e) for all children nodes of Here, ch(u) denotes the of all children nodes of v, and
15

---

Published as a conference paper at ICLR 2021
(ch(v) and Xch(u) defined as:
Be)(v) {(z zu pose 2 (PPP)
(wEoh(u): (
Xeh(o) {xu (Xuuuech(())
(13)
GSGN(++1) is shared for all nodes in level 1, and serves generic module for decomposing an entity at level into its parts at level (1 + 1).
There are three submodules in GSGN(++1), namely PresPoseApprPrii for computing the condi- tional prior, and PresPoseEnc( and ApprEnc((+1) that are adapted from the SPACE encoder. All three submodules can be implemented as CNNs. For PresPoscApppPri each cell in the output feature map corresponds to one child node, and the number of cells deter- mines the maximum out-degree at level I. Hence, the latent variables at the children nodes can be inferred parallel through a feedforward pass each submodule.
and PresPoseEnc((t))
Algorithm Foreground Inference
Input: image number of scene graph levels L I: Infer the appearance variable for the root node:
((2,pPr x) ApprEnc((=)) (x)
z, appr q(z,, appr x); X, = X
2: for each scene graph level
...... do
3: for each node v in level parallel do
4: Infer the latent variables and crop local image regions for children nodes of
Zch(w): xch(o) = GSGN(++1) (z.per x,)
5: end for
6: end for
7: return all foreground variables Zig
Algorithm GSGN(I+1); Foreground Inference at Level (l + 1)
Input: appearance variable z.Ppr and local image region x, for node v in level Compute conditional prior:
p(ZcM(u) (:per) = PresPoseApprPri(tt)) (2.pe))
2: Use SPACE to predict presence and pose variables from X,:
QSPACE
ch(s)
pose,pose
x,) PresPoseEnc ((+1) (x+x)
Sample the presence and pose variables:
prss,pose
ggzz (k()
pres,pose ppppr x.) p ((()(()))
pres, zPPPr) @SPACE pres, x..) (ch(r)
ch((w)
4: for each child node u E ch(v) parallel do
5: Use spatial transformer crop local image region for node
x,, ST Pase)
6: Use SPACE to predict appearance variable from X,;
QSPACE(2.) (z,Ppr pppr xx,) ApprEne (I+1) (x.)
7: Sample the appearance variable:
uppr q(z")Pr zupprr x.)) xx p(z (ZPPr) (SPACE (z (SPACE(2.:PP))) xx.)
8: end for
9: return latent variables Zeh(v) and local image regions Xah(o) for all children nodes of
16

---

Published as a conference paper at ICLR 2021
A
Input Il 64)
B
Input (64 do4 xs
Iipul (32x 32 s
ConsBlock
CorvBlock (ow64, 5m2, p=l)
ComBloock 10+32, 5m2, pe]]
10n256, sml, pw0)
ComeBlock
CorvBlack (om64, Lod, 1n2,
GomBkkk Jem32, Jes,, 1n2, prl)
Destdeuta,, kil,
ShouSSullla/3
CorvBlock (en64, kid, x22,p:1) p=1)
ComBlock ke:3, ked, v:2, prll)
ConBBook (o-64, kel, sel,, pHO)
(o-64, ked, sn2, p=l)
1c-64, sel, poC)
CondBlock (ox32, Kol,
CondBoock Iceld, lol,
Condilloc (cebd, sel,
1ce64, bel,
Ic-6d, sel, p=li
Corv [c-12, kel, pelp
Condlinn's
Landllack (o64 le3, se), pel!
Output (1 x1x 64)
(o66, Joel, jnl, pa0))
Outpal (2x2x 12)
Oulput x2x2x
(6.126, ken, ael, pmil)
ConvBlock (0+120, p=O)
Corvilock (0-64, kel, smil p=0)
CorvBlock 1o-32, kw], s-1, p-0)
pato
Ourput x
Figure 9; CNN architectures of (A) PresPoseApprPri (B) PresPoseEnce and (C) ApprEne (9) used in three-level GSGN. To implement GSGN-9, which the maximum out-degree is we just need to change the layers in green so that the output feature maps have spatial size 3
For concreteness, in Figure 9 we provide the CNN architectures of the submodules of GS))(33 used in our experiments. They are responsible for decomposing objects into parts. The submodules at the other levels are similar. In Figure
9 each ConvBlock involves convolution layer followed by
group normalization (Wu & He, 2018) and ReLU nonlinearity (Nair & Hinton, 2010). ConvReLU denotes convolution layer directly followed by ReLU nonlinearity. The PixelShufile(r) layer (Shi et al. 2016) converts an input feature map of size H,, W,C C x. r] to an output feature map of size [ r, W C].
PresPoseApprPrii" (Figure 9 takes in the 64-dimensional appearance variable of an object, and outputs the conditional prior for each of its four possible children. Here, output 1 includes two (unnormalized) logits for the presence variable, and the mean and standard deviation of the 5- dimensional pose variable, including 2-dimensional position and I-dimensional rotation, scaling, and depth values. Output 2 includes the mean and standard deviation of the 32-dimensional appear- ance variable of part. Notice that due to the PixelShuffle layer, not all parameters the four children. This allows the network to predict a different distribution for the latent variables of each child. (3)) (Figure 3) takes in the cropped image region of an object (resized to 64 x 64), and infers the presence and pose of its children. ApprEnc (Figure 9C) takes in the image region of a part (cropped from the input to PresPoseEncc3) and resized to 32 x 32), and infers the appearance of that part. Notice that the cropping and appearance inference can be done parallel for all parts. We apply Softplus nonlinearity to ensure that the standard deviations are positive.
shared across
17

---

Published as a conference paper at ICLR 2021
D.2 PARAMETERIZATION OF POSE VARIABLES
As mentioned in Section D.I the pose variable includes position, scaling, rotation, and depth:
thhta z = (z. where scale Z. Zu zepph)..
(14)
These are all Gaussian variables. Before feeding them to the spatial transformer, we first apply nonlinear squashing functions to convert their values into the desired range. In the following, we describe each variable in detail.
2 where E R represents the center position of in the image patch of its parent Xpa(e)- Similar to SPAIR (Crawford & Pineau 2019b) and SPACE (Lin et al. 2020b). we divide xpa(s) into > grid cells, with r" being the maximum out-degree. Each v of the same parent is associated with a cell index (i, and is expected to identify an entity in Xpa(v) that is close to cell (1, Specifically, zwhere is used to compute the deviation of entity v from the center of cell (i, Let C,j E IR? denote the center of cell ji. and E R denote the side length of each cell. The center position of entity v can be computed as:
Position.
Zv zyyhere Ca.j + a d. tanh(zzhorr))
(15)
Here, a is a hyperparameter. If a 0.5, then the center of entity e must be within cell (i.j). Typically, is set to be greater than 0.5 so that the network can learn to distribute the entities that are close together across multiple cells. In our experiments, we set a = 0.55 for GSGN (r = 2), and 0.75 for GSGN-9 (r 3).
Scaling, zi represents the scale in Xpa(v)- If v is an object, then zicale E R2, and the object scale is computed as:
z
sscale = 0.3 + 0.29 tanh(z,
(16)
If v is part, then for simplicity we let z. E R, and compute the part scale as:
zecale 0.5 + 0.49 tanh(z,cale).
(17)
Here, Z 1 would mean that entity v has the same size as its parent. We find that limiting the range of object scale is necessary for GSGN-No-Aux to learn object-level decomposition (other- wise, GSGN-No-Aux tends to explain the entire scene as one object). With the auxiliary KL. terms introducing some preferred prior (described in Section D.3), in GSGN the object scale can probably be computed in the same way as the part scale, but we have not investigated this.
Rotation. z!heta IR represents the rotation of o in Xpa(e)- Since there is groundtruth, the network is free to decide what each entity should look like when there is no rotation, and predict the rotation angles accordingly. On the 2D Shapes dataset, the rotation is computed as:
Zg ztheta = π tanh(z (hatta
(18)
On the Compositional CLEVR dataset, since all objects are placed upright on plane, we compute the rotation as:
Zy
ttheta
tanh(z (2ehta)
(19)
10
Depth. depth E IR represents the relative depth of with respect to its siblings (other children of pa(v)). It transformed to a positive value:
zdepch = Softplus(z4epth)
(20)
2 zdepth is then used to construct a transparency map Os that is the same size as Xpa(v)- Ideally, for pixel (i, (i. if entity contains this pixel and at the same time has smaller zdepth than all its siblings that also contain the pixel, and (i, = 0 otherwise. To determine which pixels are contained in entity v, we can simply use the decoded mask m. and the presence variable zpres, Specifically, we first place the mask into the parent coordinate frame by an inverse spatial transformer:
m = pres ST awhere zealle, 7 5theta
(21)
where m, is the same size as Xpa(e). and ideally (i,j 1 if entity is present and contains pixel (i,j), and (i. = 0 otherwise (in practice. Qn and m, are not strictly binary). We can now compute a (i.j) by a masked Softmax over negative depth values:
m (1, exp (-zdeptty
a (1, =
(22)
Eucoh(pa(o)) (i. j)- exp ( (-zeepth)
18

---

Published as a conference paper at ICLR 2021
D.3 AUXILIARY KL TERMS
In this section, we list the auxiliary KL terms used in our experiments. They have the same weight the original KL terms that appear in the ELBO.
Table 6 lists the auxiliary KI. terms that are the same across all levels. We use a Bernoulli prior with low success probability for z encourage sparse tree structures. The prior for zeeph has its mean value set to 4. This helps avoid the negative half of Softplus nonlinearity that tends to make the transformed zdepth values (Equation 20 less distinguishable from each other.
Table lists the auxiliary KL terms that are only used the object level for the Compositional CLEVR dataset. The prior for zacale has its mean value set to -0.2. According to Equation 16 this corresponds to an approximate mean value of 0.24 for object size zicale, We note that this value does not match the average object size in the dataset. It is simply set to be small to encourage decomposition and avoid explaining the entire scene as one object. The prior for p is used only in GSGN-9. We find that it helps prevent posterior collapse, which we observed in GSGN-9 but not in GSGN. On the 2D Shapes dataset, we use a Gaussian mixture prior for zicale, as shown in Table This is reflect the prior knowledge that these 2D objects can have significantly different scales. Neither the mean values nor the mixing weights are chosen to match the actual distribution of object scales in the dataset. We estimate this auxiliary KL term using 100 samples from the posterior. This is not expensive since gcaae has only two dimensions.
Finally, shown in Table 9 we use Gaussian mixture prior for the part scales. The component mean values, after being transformed according to Equation 17 are approximately 0.91, 0.5, and 0.24. Again, note that neither the mean values nor the mixing weights match the actual distribu- tion of part scales in our datasets. These values are chosen to encourage decomposition of objects into parts and at the same time reflect the prior knowledge that parts can have significantly different scales. We estimate this auxiliary KL term using 100 samples from the posterior.
Table 6: Auxiliary KI. terms used at all levels.
Posterior
Unconditioned Prior
(zprre
appr "pa(u): Xpa(o))
Bernoulli(1 x 10 10)
q
where
xappe
Xpa(u))
10,
q( (2thota
"pa(u)* Xpa(v))
10,
q( (2
aappr
pa(()) xpa(r))
144
Table 7: Auxiliary KL terms used only at the object level on Compositional CLEVR dataset.
Posterior
Unconditioned Prior
( scale
appr pa(e)) Xpa(v))
((-0.22.0 0.1)
12 (2
uppr pa(n)) x.)
Table 8: Auxiliary KL term used only at the object level 2D Shapes dataset.
Posterior
Unconditioned Prior
g (
2 appr
pp) Xpa(s))
0.2N 0.1) 0.& 0.1)
19

---

Published as a conference paper at ICLR 2021
Table 9: Auxiliary KL term used only the part level.
Posterior
Unconditioned Prior
z appr
pa(()) Xpa(v)) )
+ 0.05) 0.88(-00.6,0.
D.4 BACKGROUND INFERENCE
The prior and posterior for the background variable are implemented
p(Zbg Zig) = p(Zbg 2PPP))) (Z Ztg. x) = ((Zbg 2 x)
(23)
We provide the implementation outline for computing q(Zbg x,pprr x) Algorithm The two submodules, BgPri and BgEne, are implemented as CNNs, and their architectures Figure I0 Both submodules output the mean and standard deviation of the background variable. We apply Softplus nonlinearity to ensure that the standard deviations are positive. 2.ppr and Bbg have dimensions 128 and 8, respectively.
be found in
Algorithm Background Inference
Input: image x, appearance variable of the root node z,pprr I: Compute conditional prior:
p(Zog (z.PP")) BgPri(z,,PP)
2: Predict the background variable from x:
QSPACE (Zbg Zbg x) BgEnc(x)
3: Sample the background variable:
Zhg q Bbg 12perr x) p(Zbg x,
aPPE @SPACE ZZg lx)
4: return background variable Zbg
Inpui (1 Lxx 128)
Irpul (128% 128x 3)
ConcBlock Jem8, vil, pr1)
1om128, kmll prff)
Corviee Jea32, kat, pat!
Convillock sul,
Carw Jow18 kel, sel, pat)
ConeBiokk Icwll, ked, 5-2, p=1)
Output
Conotibok Jen8, xal, p=1)
CaneBlock p=1)
ConvReLU [tm3) ken, pm0)
Cony Iest kel, path
Duppi
Figure 10: CNN architectures of (A) BgPri and (B) BgEne used in three-level GSGN,
20

---

Published as a conference paper at ICLR 2021
D.5 TRAINING DETAILS
We train GSGN on single GPU, using Adam optimizer (Kingma & Ba 2014) with batch size of 64 and learning rate of 3 10 for up 500 epochs. We use gradient clipping ensure that the infinity norm of the gradient does not exceed 1.0. The temperature for Gumbel-Softmar (Jang et al.) 2016 Maddison et al., 2016) is exponentially annealed from 2.5 to 0.5 during the first 20 epochs. Similar to Slot Attention (Locatello et al. 2020), the learning rate is linearly increased from 0 to x 10 during the first 10 epochs, and exponentially decayed to half of its value every 100 epochs. We set dig = 0.3 and 0bg = 0.1. On the Compositional CLEVR dataset, ofg has an initial value of 0.152, and linearly increased from 0.15² 0.3 during epochs 20-40. Similar to SPACE (Lin) et al. 2020b), the mixing weight m, is fixed at the start of training. On the 2D Shapes dataset, we fix m, x 10 for epoch, while on the Compositional CLEVR dataset, we fix m,, = 0.1 for 2 epochs. GSGN-9 is trained on two GPUs, each taking batch size of 32, using the same schedule.
E COMPARISON WITH SUPERPIXEL HIERARCHY
In this section, we qualitatively compare GSGN with a simple baseline called Superpixel Hierarchy (Wei al. 2018). This algorithm hierarchically merges pixels into superpixels based on connectiv- ity and color histograms, until the full image is merged into one superpixel. Using this hierarchy,
algorithm able to produce segmentation results when given the desired number of superpixels
(can be any number between one and the total number of pixels). It has been shown that Super-
pixel Hierarchy outperforms the widely used FH (Felzenszwalb & Huttenlocher 2004) and SLIC (Achanta et al. 2012) methods and also some more recent methods.
Object Grouping
Part Grouping
20 Super- Pixels
30 Super-
Object Grouping
Part Grouping
20 Super- Pixels
30 Super- Pixels
Input
Pixels
Input
Figure 11: Segmentation results produced by Superpixel Hierarchy.
In principle, Superpixel Hierarchy (SH) should be able to perform object and part grouping, by assigning superpixel
object or a part. We show qualitative segmentation results on our datasets in Figure 11 using the publicly available code?] Here, we compute the mean color for each superpixel, and each pixel is drawn the mean color of the superpixel to which belongs. For object and part grouping, we provide SH with the groundtruth number of components (the number of objects/parts plus one for background). As can be seen, when doing object grouping, SH tends to merge some parts into the background. This seems to be caused by some characteristics of our datasets. Specifically, in the 2D Shapes dataset, the parts within an object do not touch each other, making it hard to merge them into one superpixel because superpixels 4-connected. In the Compositional CLEVR dataset, the light and dark regions of the background tend to be split into different superpixels due dissimilarity in color histograms. For part grouping.
supposed to be
https: thub. com/ semi
http:/ssmiiuurrrrm quark hww.l//bbbvkkee -uperrippereeeeee
21

---

Published as a conference paper at ICLR 2021
SH sometimes merges tiny parts into the background the 2D Shapes dataset, and splits metal parts that have specular highlights on the Compositional CLEVR dataset. When more superpixels allowed, SH is generally able to distinguish foreground from background, but this kind of over- segmentation cannot serve our purpose of building the scene graph.
F RESULTS ON SEVEN-PART COMPOSITIONAL CLEVR DATASET
In this section, we show results on a slightly more complex version of the Compositional CLEVR dataset, in which introduce four new parts and replace some of the objects. The dataset contains ten types of objects composed from a total of seven parts. Three of the object types contain single part, another three contain two parts, and the remaining four contain three parts. We the same network architecture and auxiliary KL terms as those we used for the original Compositional CLEVR dataset. We find that adjusting the weight of the KL terms like 8- VAES (Higgins al, 2017) helps improve training stability. Specifically, we multiply the original and auxiliary KI. terms of part-level presence and pose variables by a hyperparameter B. We keep
for the first 50 epochs, then linearly increase f from to 5 during the next 30 epochs, and finally keep 3 = 1 for
the remaining epochs. We report qualitative results in Figure 12 and quantitative results in Table I0 and Table MII
Pull scene
Object
Object
Object
Object 4
Recen- sirsction
Expur bbox
Reant- struction
Traut boox
Razin- siruction
Input bbox
Rezm-- stoction
Irpuf bbox
Reann-
Input
Purt
Parts
Purts
simiction
Pars
Figure Visualization inferred scene graphs on seven-part Compositional CLEVR dataset.
Table Quantitative results on seven-part Compositional CLEVR dataset.
Object Count Accuracy
Object F1 Score
Part Count Accuracy
Part F1 Score
Metric
ELBO
SPACE-O SPACE-P
3.56
96.60%
98.49%
3.55
89.69% 97.48%
92.10% 94.1 1 %
GSGN
3.55
96.38%
98.29
Table 11: Robustness to occlusion seven-part Compositional CLEVR dataset.
Min Visible
<100
100~200
>000
Pixels Per Part
Part Count Accuracy
Part Recall
Part Count Accuracy
Part
Part Count Accuracy
Part Recall
Metric
Recall
SPACE-P
0.00% 100.0%
81.05% 95.63%
82.78% 96.72%
94.50%
93.39%
90.25%
GSGN
96.90%
97.85%
92.76%
22

---

Published as a conference paper at ICLR 2021
G ANALYSIS OF POSSIBLE SCENE GRAPHS
In this section, we provide brief analysis how variational the scene graphs can be on the Com- positional CLEVR dataset.
Let us first investigate the possible scene graph structures (i.e., ignoring the difference caused by pose and appearance). There 1-4 objects in scene, each consisting of 1-3 parts. This leads to total of Σ ("+3-1)= 34 possibilities.
Now account for the pose variations of parts in the scene graph. Note that the relative position and scale of parts in each composite object are different. So the number of variations within an object
is (singlle-part + 3(two-part) + 4(three-part) 8. This leads to a total of Σ (*+8-1) = 494 possibilities.
Finally, we account for the appearance variations. There are colors and materials, leading to 14
choices of appearance for each part, Combining with pose variations, we obtain 3 x 14 +-++3 x 142 + 4 x 143 = 11606 possible subtrees for each object. Hence, the total number of possible scene graphs Σ Ex=1 (=+11666--) 7.57 10!4,
We note that the above calculation still does account for the pose variations of objects. In fact, object poses are sampled from continuous distributions as opposed to discrete ones, leading to infinite variations.
23


Reviews:
[{"review_id": "zaYaEoYRhV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=====Post-Rebuttal Comment=====\n\nI thank the authors for the detailed response and the updated results. While my overall opinion of the work is slightly more positive post-rebuttal, I still maintain that this is a clear reject, primarily for the following reason:\n\n- The technical contribution (adding a hierarchical layer to SPAIR and demonstrate the hierarchy can also be learned without supervision) is not significant enough to accept purely based on a \"proof-of-concept\" of a \"new\" direction.\n- For incremental contributions, I expect the experimental results to be more convincing to be acceptable, a few points that I still really expect to see:\n    - comparisons on harder datasets when one doesn't have to go into a specific metric to show an edge over a prior art that is targeting a different application\n    - results on decomposition with more significant overlaps (especially in 2D) and on objects where part boundaries are harder to infer (actual 3D objects is still preferable)\n    - object-level manipulation (row 3&4 in fig 4 did not match description) and latent space interpolation\n\nI would also strongly encourage the authors to highlight the similarity and different between SPAIR and SPACE when introducing the latent code formulation.\n\n=====Summary=====\n\nThis work proposes a method that can learn a three-level (scene, object, part) hierarchical representation of scenes in an unsupervised manner. A variational autoencoder is used here, where the encoder recursively infers the shape and pose of individual objects and parts, and the decoder recomposites the inferred parts are then recomposited to the inferred poses. The proposed method is evaluated on two datasets created for this paper. The results suggest that the proposed method can reliably breaks down scenes into meaningful objects and parts, and performs slightly better than another method designed for a slightly different task in terms of reconstruction quality, learned representation, and data efficiency for downstream tasks.\n\n=====Strengths=====\n\n- The motivation of the need for hierarchy is solid, and the solution proposed seems to me to be a reasonable way to impose some sort of hierarchy.\n- The model seems to be well-tuned, utilizing appropriate training tricks and architectures.\n- Good performance for the evaluations chosen in the paper.\n\n=====Weaknesses=====\n\n- *Very* inadequate attribution of ideas. I am not too familiar with the AIR line of work, but I think quite a few ideas can be traced back to prior works. It would be much better if the authors can, in addition to a brief one sentence mention in related works, add clear discussions for the inspirations of the main design choices.\n- Missing discussions of relevant works that do: 1. unsupervised part decomposition e.g. “UCSG-NET - Unsupervised Discovering of Constructive Solid Geometry Tree”, \"Bae-net: Branched autoencoder for shape co-segmentation\", “CvxNet: Learnable Convex Decomposition”; 2. Learning hierarchical representations e.g. “StructureNet: Hierarchical Graph Networks for 3D Shape Generation”.\n- I think the comparisons in this work are neither adequate nor fair. I am not convinced that the two toy dataset used here can prove the superiority of the method. The authors claim that “other works can’t work on our dataset”, but I think the burden of the proof is on the authors to show that their method is superior, even under a more specific setting. In other words, if the method is indeed “general” and can learn good decompositions, then I would expect it to perform better even under an slightly unfair setting i.e. comparing against metrics/datasets adopted in other works. Furthermore, the datasets used in this paper appears to be way too simple as compared to real world data. The authors argue that dataset with a single shape is easier, but I disagree: datasets like partnet contains much more complicated part structures, as well as joints between parts, than what is used here, even with only a single shape. (And it is pretty evident from the qualitative examples that the challenging part is decomposing objects into parts, not decomposing scenes into objects). Last but not least, I want to see more evidence that the proposed method is actually useful in real applications.\n- The learned representation does not seem to be of very good quality, as seen in Figure 4.\n- A lot of overclaims, to name a few: 1. “GSGN is a general framework for representing and inferring scene graphs of arbitrary depth”: I don’t think a model being able to work a toy setting with three levels will mean that the same framework can be used for more complex settings of arbitrary depth (as an analogy: MLP works for MNIST but not on ImageNet). If the framework can handle more general cases, then show it. 2. “Closely follow the rendering process in graphics engines”: I don’t think applying affine transformations and compositing alone is enough to warrant this claim, it is pretty clear that the learned representation lacks a good sense of “objectness”, as textures, lighting and etc. are all entangled together (evident in Fig 3). There does not seem to be a straightforward way to extend the method to truly parallel the 3D rendering process, neither. 3. “First deep generative model for unsupervised scene-graph discovery”: there are a lot of works that infer structures in an unsupervised way, I don’t think it’s fair to give a very narrow definition of “scene graph” and claim “the first”. 4. “GSGN has capture many predefined object types in the dataset”: I don’t think one can make this claim when there are only three primitives and ten types of objects…\n\n=====Reasons for Score=====\n\nOverall, I have the impression that this work is cherry picking a very specific setting where the proposed architecture works reasonably well. For a work making a quite big claim of being “the first deep generative model that learns …”, I would expect much more comprehensive evaluations than what is currently shown here. Furthermore, many ideas in this work are not attributed properly, making the novelty of the method quite unclear. From my limited knowledge of the direct predecessors of this work, I don’t think there is too much novelty in this paper. I would be more than willing to change my score if the authors can 1. Provide more comprehensive evaluations 2. State the novelty / discuss prior works more clearly. But for now, I tend to give a pretty clear reject.\n\n=====Additional Comments & Questions=====\n\n- I am not sure if translation & rotation alone is a good way to handle 2D renders of 3D objects, since any translation & rotation will result in change of perspectives and illumination e.g. rotating the bronze sphere in Figure 3, row 1 will make the specular highlight inaccurate and translating the blue cube in row 3 will make the top surface less visible. Could the authors justify why predicting translation/rotation make sense, when the perspective/illumination of the object already provides a really strong cue?\n- Following previous point: would like to see examples of the same learned object being used in multiple scenes.\n- The quality of the learned primitives, as seen in Figure 4, seems to be pretty underwhelming. If the aim of the work is discovering those primitives, would it make more sense to impose a stronger prior on the properties of the primitives?\n- Table 1 & 2: why are all the ELBO terms the same? I would imagine them to be different, especially for SPACE-O/P, which, if I understand correctly, is a *completely different* model with different architecture and loss formulation? \n- Table 2 & 3: why are the metrics so close between SPACE-P & GSCN in table 2 but so different in Table 3? Does that suggest unbalanced dataset?\n- Still Table 2 & 3: why no comparison between SPACE-O & GSGN for object level occlusion?\n- The paper claims that being able to handle background is a unique advantage as compared to other works, but the background used in the toy dataset is quite simple. Would like to see more complex examples.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "lygkHt1HkxM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Generative Scene Graph Networks (GSGN) is a variational auto-encoder with the intermediate representation being tree-like scene graphs. The leaf nodes stand for primitive parts and edges stand for poses to compose parts into objects recursively. The experiments are done in two image datasets of single color, simple shape 2D/3D objects: Multi-dSprites and CLEVR, and the model is able to discover objects without supervision.\n\n**Strength**: I find the direction important, and the method well established in a variational inference framework with graphics-inspired designs. Experiment numbers look good in general.\n\n**Weakness**: Perhaps my biggest concern is that the current datasets are a bit weak. First, objects are too simple (single color, simple shape), so we are not sure if GSGN can work with more realistic visual domains where objects have more complex 3D structure or texture. Second, the object/primitive decomposition is slightly weird to me. I would expect hierarchal structure like an object being human body, and parts being legs, arms, head and so on. But in this work a \"part\" is a single-color object, and an “object” is a bunch of single-color adjacent objects. Based on these two points, I believe the paper will be much stronger with experiments on more complex objects like humans or tables or \n\nAnother concern is the application of learned scene graph. The paper only shows unconditional sampling, but not really how to use the learned scene graph. For example, I'd expect scene graphs to be used for image manipulation, as one can change part of the object (shape, color, pose) without changing the rest. Showing the learned scene graph is useful for any downstream tasks can be a great plus for the current work.\n\nFinally, I wonder how variational the learned scene graphs can be, as the objects in the datasets are fairly simple and the learning might be easy. I'd be happy to see some analysis but this is not my main concern.\n\n---------\n\nAfter rebuttal: I'm glad they added some experiments and analysis I wanted to see, so I raise the score to 6. As the authors said, the paper is a proof-of-concept of unsupervised hierarchal scene graph learning, and the rebuttal to some degree reassured me. For example, modeling a cube on top of another top as two parts of an object (which was weird to me: why not each cube as an object?) helps edit tasks where the top cube is enlarged but the \"on top\" relation is maintained. The downstream representation transfer also makes sense. Of course experiments are still toy from computer vision perspective, but I'm now okay with acceptance.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WS49JqZPDhf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**PAPER SUMMARY**\n\nThe paper presents a generative model for scenes that uses tree-structured latent variables to recursively decompose images into objects and parts, without any object or part supervision during training. The model is trained using variational inference. Experiments are performed on two new datasets (2D Shapes and Compositional CLEVR), demonstrating that the model is able to successfully uncover recursive scene/object/part decompositions in an unsupervised setting. The model is compared against prior work (SPACE) that performs non-hierarchical scene modeling.\n\n**STRENGTHS**\n\n- The paper presents a novel generative model that can infer tree-structured latent variables\n- The method is technically impressive, and a clear improvement over the non-hierarchical modeling used in prior work\n- The paper presents two new datasets (2D Shapes and Compositional CLEVR) for studying hierarchical scene decomposition. I hope these can be publicly released!\n\n**WEAKNESSES**\n- The method is quite complex, and though it is technically impressive I wish that it had been compared against very simple baselines\n- No experiments on real-world data\n- Unclear how the model will scale to wider and deeper trees\n- Many implementation details are unclear\n\n**SCALABILITY**\n\nA big selling point of the proposed model is that it can model the hierarchy of scenes into objects and parts, and the tree-based formulation of the latent space used to achieve this is technically impressive. However, how well would the method scale to larger scenes? All experiments used a relatively small three-level hierarchy where all nodes have a fixed out-degree of 4; thus in all experiments the full tree has just 21 nodes total. How well can the method scale to much larger trees, with deeper hierarchies or wider out-degrees?\n\n**PRIOR KNOWLEDGE**\n\nThe method requires choosing a tree depth and node out-degree as hyperparameters. To some extent, these hyperparameters encode very strong prior knowledge about the data being modeled (levels of hierarchy, and number of subparts within each part). In the experiments, these hyperparameters are perfectly matched to the synthetic datasets: you use a two-level tree with out-degree of four, and each image has between 1 and 4 objects, each of which is composed of between 1 and 3 parts. However in more complex real-world scenarios, you may not have such detailed knowledge of the world’s compositional structure. For this reason, I’m curious as to how the method would behave when the structural hyperparameters are mismatched to the underlying statistics of the dataset.\n\n**SIMPLE BASELINES**\n\nThe model is evaluated on two synthetic datasets -- 2D Shapes, and Compositional CLEVR. How difficult is the scene graph inference problem on these datasets? I wish that the authors had compared to very simple baselines in order to give a sense for how difficult the problem really is.\n\nFor example, I suspect that a simple “handcrafted” baseline that oversegmented the image into superpixels, then recursively merged superpixels based on proximity and simple appearance features (e.g. color histograms) could produce very plausible scene graphs for these two datasets.\n\nAt a high level, the promise of an end-to-end learning-based approach compared to a “handcrafted” approach like the above is that a learning-based approach should require less tuning, should be more adaptable to new datasets and tasks, and should scale better to complex real-world datasets where the assumptions of the model don’t perfectly match the statistics of messy data from the real world. However in this case, I’m not convinced that this complex variational method would be any simpler to implement or tune, or even give better results than, a very simple “handcrafted” baseline like the above on these synthetic datasets. There are also no experiments to demonstrate its scalability to real-world datasets.\n\nThis leads to a pointed question: If someone wanted to infer scene graphs from images, why should they prefer your approach over a very simple “handcrafted” approach?\n\n**MANY IMPLEMENTATION DETAILS UNCLEAR**\n\nThere are many implementation details that are unclear from the paper and supplementary material, without which reproducing the results are practically impossible. Even if some of these details do make sense as part of the main text, they should be specified more explicitly in the supplementary material. For example:\n\nWhat is the generative process for the presence variables $z_v^{pres}$? This is not clear from Equations 3 or 4, nor the surrounding discussion. From Equation 9, the encoder predicts presence variables conditioned on image patches; but how can you predict the presence variables when sampling an image from scratch?\n\nWhat is the dimension of the latent variables $z^{appr}_r?$\n\nYou say that the pose variables are Gaussian, but how is the pose parameterized in terms of relative location and scale?\n\nFrom the discussion after Equation 6, the pose variables also include a depth map -- how are these depth maps represented and parameterized? Are they per-pixel Gaussians like the other pose variables?\n\nHow exactly are the transparency maps $\\alpha_v$ computed from the depth maps? The text states that they are computed “by the softmax over negative depth values”, but this is hard to understand -- what are the sets of values being fed to softmax? If the depth map contains a per-pixel depth then a softmax over space wouldn’t make sense, since this would cause the transparency of an object to depend on its spatial size (since it would have more pixels competing in the softmax).\n\nIn Equation 9, the distributions of the latents $z_v^{pres}$, $z_v^{pose}$ are conditioned only on data from the parent node $z_{pa(v)}^{appr}$ and $z_{pa(v)}$. Thus all children of a node will have the same distribution for their pose and appearance. Is this correct? If so, how do you encourage the child nodes to cover all parts of the parent, and not collapse to a single part of the parent?\n\nThere are no details about any of the neural network architectures used to implement the model, nor any details about any training hyperparameters (e.g learning rates, training schedule, regularization strengths etc).\n\n**SUMMARY**\n\nThe model is technically impressive, and a clear improvement over prior work. However on the whole I’m not sure whether the complexity of the method is actually necessary to solve the problem at hand; I wish that the authors had done a better job demonstrating the benefits of the proposed method over very simple baselines. There are also many implementation details that are very unclear, for which reason I fear that the paper as written is utterly unreproducible.\n\nOn the whole I lean slightly toward acceptance, but I hope the authors can address my concerns in their rebuttal.\n\n\n**AFTER REBUTTAL**\n\nThe rebuttal largely addresses my concerns about implementation details.\n\nI am pleased to see the additional experimental results provided by the authors; I think that these do improve the paper. I still feel that some well-tuned handcrafted approach could likely perform on-par with the results of the proposed method, but the comparison with [Wei et al] show that achieving such results is at least not trivial, which does help to better ground the complexity of the task. The additional experiments with a three-level hierarchy show a bit more evidence for scalability than provided in the original paper. While these extra experiments do strengthen the paper, I feel that they don't really address the core issue with the paper, which is whether there is any hope for the proposed method to scale to more complex and realistic datasets.\n\nOverall I think that this is a reasonable paper and I still lean slightly toward acceptance, so I maintain my original rating of 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2R26oAjJTID", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "== Update ==\n\nThank you for your response and clarifications. I have left my score as is.\n\n== Original Review ==\n\nThe paper presents an unsupervised method for inferring scene graphs from images. Building upon\nscene-attention methods such as AIR and SPACE, it hierarchically decomposes a scene into objects and\nthose objects into parts, giving rise to a tree structure. It is shown that this model successfully\nrecovers the hierarchies underlying the data on two newly proposed hierarchical variants of the\nSprites and CLEVR datasets.\n\nStrengths:\n 1. The paper is well written, and despite the considerable complexity of the method, its\n    presentation is relatively easy to follow.\n 2. The task of interest is well-defined, and has clearly been effectivly solved on the datasets\n    considered. Both quantitative and qualitative evaluations make it very clear that the model has\n    learned to infer the correct scene graphs as desired. Its ability to infer the appearance of\n    occluded parts is especially impressive.\n 3. While there are no direct competitors on this newly defined task, the paper does a decent job of\n    comparing to the closest available baseline, showing how the additional structure can be\n    beneficial.\n\nWeaknesses:\n 1. One may argue that the datasets have been deliberately constructed to showcase the model. While\n    that is probably true, I think this is a valid approach given the novel nature of the task and\n    the lack of supervision. Despite the clearly helpful structure (limited number of objects and\n    parts), the datasets still appear sufficiently challenging.\n 2. In the experiments, scene graphs are limited to trees of height 2 and degree 4. This is a\n    significant constraint, however, each additional level of hierarchy introduces ambiguities and\n    makes it harder to learn the graph in an unsupervised manner. More complicated structures would\n    likely require supervision.\n 3. As far as I can tell, the object types were chosen once to generate the datasets, and then kept\n    fixed across the experiments. Reporting results for multiple different datasets with randomly\n    chosen object types would be somewhat more convincing.\n 4. As is common for unsupervised scene models, the proposed method likely only works on synthetic\n    images in its current state. However, due to the additional structural assumptions on the data,\n    it seems especially challenging to find suitable real-world use-cases.\n\nOverall, the paper presents an effective new method for the task it sets out to solve. While it is\nquestionable how it would work on real-world data, I believe the paper is of sufficient interest as\na proof of concept, and am therefore leaning towards acceptance.\n\nQuestions:\n 1. It is stated that auxiliary KL terms are added, with the sparseness constraint on $z^{pres}$\n    being one of them. But is not clear if there are others. This would be important to know in\n    order to evaluate how strong the model's inductive biases are.\n 2. The downstream task used for Fig. 5 is not clear to me. If the number of parts is computed for\n    each object, and these numbers are then summed, isn't the result equal to the total number of\n    parts in the scene, which SPACE-P can also infer? If only distinct parts are counted,\n    how is equality of parts defined on the dataset?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission generated significant discussion between the reviewers; three of them ended up on the "accept" side, but one remained firmly in the "reject" camp.

The main strength of the paper is that it tackles a very hard problem: learning an unsupervised generative model (and accompanying inference model) of scene graph structures given only image data. As one reviewer mentioned, it is remarkable that the authors were able to get their system to work at all, given the seeming intractability of this problem. The work builds upon a clear line of prior work in this area, and the type of data on which it is evaluated ("toy" synthetic datasets a la CLEVR) is consistent with prior art.

Multiple reviewers brought up the "toy" nature of the dataset as a drawback to the paper, but most agreed that this is not reason to reject the paper. Rather, the paper demonstrates a convincing proof of concept that this kind of model can be built, and improvements in the elements out of which the model is composed (generative and inference networks) should improve its applicability to real-world data.

Another question mark raised by multiple reviewers: could a simpler, handcrafted inference procedure work just as well or better? The authors included a new experiment against a hand-coded heuristic in their rebuttal, and their method outperforms it. One reviewer noted that more careful tuning might make a heuristic perform as well as the proposed method, but it is still clear that it is not trivial to get a hand-coded solution to perform well (even for this "toy" data). Another reviewer pointed out that this is one of the main attractions of variational inference methods: the ability to specific knowledge as simple generative priors rather than complex bottom-up inference procedures.

One reviewer, R1, remains negative about the paper. His (it is a he; I know this reviewer) main concern is that the scene graphs used are shallow and have a simple structure, and thus (a) it's not clear what value they add, (b) a simple postprocess could reconstruct them, assuming the individual object parts could be detected, and (c) it's not clear whether the method would generalize to deeper/more complex hierarchies. He believes this calls into question the validity of the entire method.

I am sympathetic to this argument, but I think setting the bar this high may prevent progress in this field. For point (a), the authors included an image-manipulation application in their rebuttal--again, a proof of concept, not a directly useful tool. For point (b), the authors did compare against a hand-coded inference baseline and achieved better results, so while this may be possible, it is probably not as easy as the reviewer suggests. (c) remains an open question, to me. But even if this method as presented cannot generalize to more complex scene graphs, it likely paves the way for future work that can.
Decision: Accept (Poster)