Generating Scenes with Latent Object Models
Authors: Patrick Emami
Pan He
Sanjay Ranka
Anand Rangarajan
Venue: ICLR 2022 Submitted (2022)
Keywords: deep generative models
slots
scene generation
object-centric
VAEs

Abstract: We introduce a structured latent variable model that learns the underlying data-generating process for a dataset of scenes. Our goals are to obtain a compositional scene representation and to perform scene generation by modeling statistical relationships between scenes as well as between objects within a scene. To make inference tractable, we take inspiration from visual topic models and introduce an interpretable hierarchy of scene-level and object-level latent variables (i.e., slots). Since generating scenes requires modeling dependencies between objects, we cannot make a bag-of-words assumption to simplify inference. Moreover, assuming that slots are generated with an autoregressive prior requires decomposing scenes sequentially during inference which has known limitations. Our approach is to assume that the assignment of objects to slots during generation is a deterministic function of the scene latent variable. This removes the need for sequential scene decomposition and enables us to propose an inference algorithm that uses orderless scene decomposition to indirectly estimate an ordered slot posterior. Qualitative and quantitative analysis establishes that our approach successfully learns a smoothly traversable scene-level latent space. The hierarchy of scene and slot variables improves the ability of slot-based models to generate samples displaying complex object relations. We also demonstrate that the learned hierarchy of representations can be used for a scene-retrieval application with object-centric re-ranking.

Under review as a conference paper at ICLR 2022
GENERATING SCENES WITH LATENT OBJECT MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce a structured latent variable model that learns the underlying data- generating process for dataset of scenes. Our goals scene representation and to perform scene generation by modeling statistical re- lationships between scenes as well as between objects within a scene. We take inspiration from visual topic models and introduce an interpretable hierarchy of scene-level and object-level latent variables (i.e., slots). To generate scenes, de- pendencies between objects must be modeled. While we could achieve this by assuming slots are generated autoregressively. algorithms for estimating autore- gressive slot posteriors impose an unnatural order on objects in scene and are known to be limited in visually complex environments. We climinate the need for autoregressive posterior inference by assuming that the assignment of objects to slots during generation is a deterministic function of the scene latent variable. We propose inference algorithm that indirectly estimates ordered slot posterior by leveraging an orderless slot posterior and the deterministically--reees ppeddict-edeeeddd object generation order. Qualitative and quantitative analysis establishes that approach successfully learns a smoothly traversable scene-level latent space and that the hierarchy scene and slot variables improves the ability of slot-based models generate scenes with complex multi-object relations.
obtain compositional
INTRODUCTION
This paper considers the problem of learning the data-generating process for multi-object scenes. We want to train likelihood- based deep generative models that can encode scenes into representation hierarchy of scene and object latent variables and that support novel scene generation. A scene-level representation can capture statistical relationships between scenes and can model complex interac- tions between objects (Jiang & Ahn, 2020). Object-level representations increase compositionality, interpretability, and robustness of the model to out-of- distrrbbittion data (Dittadi et al., 2021). Much progress has been made on models that encode scenes into sets of object-centric distributed repre- sentations, slots (Greff et al., 2017: Burgess et al., 2019; Greff et al., 2019; Engelcke al., 2019; Locatello et al., 2020; Nanbo et al., 2020; Emami et al., 2021; Engelcke et al., 2021). These models have demonstrated promising generalization and object-level disentanglement and have been applied successfully to object-centric visual model-based reinforcement learning (Veerapaneni et al., 2019).
However, training blot-based models to jointly decompose and generate novel scenes remains chal- lenging. The limited prior work on this problem has approached it by either ignoring dependencies between slots (Greff al., 2019; Emami al., 2021) or by factorizing the joint distribution over slots using the chain rule into a sequence of autoregressive distributions (Finellcke et al., 2019; 2021). None of these models possess a scene-level representation and are therefore limited in their ability to capture scene-level statistics and higher-order object relations (Jiang & Ahn, 2020). Also. although autoregressive generation is capable of modeling dependencies between objects, autoregressive inference is extremely difficult (Engelcke al., 2021) and remains open problem.
In this work, we introduce Latent Object Models (LOMs), a deep generative modeling framework for fitting the mata-generating process for scenes with latent variable hierarchy consisting of scene-level variable and set of slot variables. The analogy used to design the hierarchical graphical model is inspired by visual topic models (Hofmann, 1999; Blei et al., 2003; Sivic et al., 2005; Fritz &

---

Under review as a conference paper at ICLR 2022
Figure The hierarchical graphical model for LOMs. We assume that scenes r are generated from K object-centric slot latent variables 21:K- The slots are independent conditional on causal sequence of deterministic variables (shown in boxes). This causal sequence is pre- dicted from the top-level scene latent variable s. By conditioning each slot one of the deterministic variables, we provide the order-dependent relational information necessary for scene generation.
h1
2
K
Table 1: Latent Object Models introduce the idea of using scene-slot latent hierarchy to fit the mata-generating process. Unlike spatial latents, slots are assigned to scene objects during inference and generation, improving interpretability and disentanglement of the learned representations.
Model
scene latent
spatial latents
object slots
VAE (Kingma & Welling, 2013)
NVAE (Vahdat Kautz, 2020)
GNM (Jiang & Ahn, 2020)
EMORL (Emami et al., 2021)
GENESIS (Engelcke et al., 2019)
GENESIS-v2 (Engelcke et al., 2021)
LOMs (Ours)
Schicle, 2008; Tuytelaars et al., 2010):
documents scenes
topics slots
words percepts
where a percept is any discrete token that makes up a scene observation, such as a pixel or 3D point. LOMs have three-level hierarchy. Each percept modeled K K-component mixture, where each component dependent on one of intermediate-leve slots. Then, the slot variables are dependent on single top-level scene variable. Although topic models make "bag-of-words" assumption that topics are orderless within a document, multi-object scene generation requires modeling dependencies between slots. LOMs therefore cannot assume slots are orderless in a scene.
Instead of modeling slot generation with an autoregressive (i.c., stochastic) prior, which requires inferring an autoregressive posterior, we propose to assign objects slots for ordered scene generation deterninistically. We train function to take a sample from the scene prior and predict causal sequence of deterministic variables (Figure 1). The slot priors are designed be independent of each other when conditioned on this sequence of deterministic variables, which eliminates the need to estimate an autoregressive posterior. We instead propose an inference algorithm for LOMs that assumes auxiliary generative model which decomposes scenes into orderless set of slots. This used to infer a slot-order- invariant scene-level posterior from which we obtain an ordered slot prior. To obtain ordered slot posterior for maximum likelihood training, compute matching between the means of the orderless posterior and the ordered prior. This matching is used to permute the orderless posterior and convert into ordered distribution.
We introduce a latent variable model with a latent scene-slot hierarchy and variational inference algorithm for modeling the data-generating process of multi-object scenes. We demonstrate that this framework is capable of learning a low-dimensional scene-level manifold that can be smoothly traversed. We also show that generated scenes accurately capture higher-order object relations present in the dataset.
Contributions:
2 RELATED WORK
We compare LOMs with related deep generative models Table 1. Likelihood based deep latent variable models based the variational autoencoder (VAE) framework (Kingma & Welling, 2013)
2

---

Under review as a conference paper at ICLR 2022
have recently demonstrated impressive scene generation capabilities for high-resolution images with spatial latent variables split across a hierarchy of stochastic layers (Sønderby et al., 2016; Kingma et al., 2016; Child, 2020; Vahdat & Kautz, 2020). Differently. LOMs model perceptual data using scene and slot latent variables. During inference and generation, each slot is bound to a single entity in the scene which improves interpretability and compositionality (Greff al., 2020).
Generative Neurosymbolic Machines (GNMs) (Jiang & Ahn, 2020) present the idea of combining scene-level prior and symbolic latent variables within hierarchical VAE for multi-object scene generation. GSGN (Deng et al., 2021) explores learning part-whole hierarchies for VAEs with similar symbolic latent variables. However, these VAEs make a strong assumption that the symbolic object-centric latent variables are spatial variables, i.c. bound to each cell of a fixed grid (Eslami al., 2016; Crawford & Pineau, 2019; Lin et al., 2020). While this works well for synthetic image datasets with similarly sized objects. LOMs tackle the more challenging problem of learning scene-slot hierarchy. LOMs fall into the category of VAEs that represent scenes as mixture models whose components are generated by set of slots (Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2019; 2021; Locatello et al., 2020; Nanbo et al., 2020; Emami et al., 2021), l). EfficientMORL (EMORL) is orderless slot-based VAE with hierarchical independent slot prior that combines efficient bottom-up stochastic slot attention with top-down iterative refinement. GENESIS (GEN) (Engelcke et al., 2019) and GENESIS-v2 (GENv2) (Engelcke et al., 2021) use autoregressive slot prior for multi-object scene generation. Unlike LOMs, neither possess a scene-level representation. Furthermore, while theoretically an autoregressive slot prior can learn dependencies between objects necessary for generating structured scenes, learning this distribution is difficult. GEN uses sequential attention to estimate an autoregressive posterior that imposes an unnatural order on objects, biases slot gradients, and does not work well on visually complex scenes (Engelcke et al., 2021). GENv2 replaces GEN's posterior with an orderless slot posterior which does not suffer from these limitations. However, orderless posteriors assume slots are independent which means GENv2's autoregressive prior will struggle to learn object dependencies.
Other VAFs with structured latent variables have been proposed for hierarchical scene generation (An- ciukevicius et al., 2020; von Kügelgen et al., 2020) but make simplifying independence assumptions in their graphical models that prevents them from scaling to more complex scenes. The Set (Kim et al., 2021) learns to generate set-structured data such as point clouds with a tree-structured hierarchy of sets of latent variables. However, unlike slot-based models, its latent variables not explicitly trained to segregate the scene representation.
Generative adversarial methods for compositional scene synthesis (van Steenkiste al., 2020; Nguyen-Phuud et al., 2020; Liao et al., 2020; Ehrhardt et al., 2020; Niemeyer & Geiger, 2021; Hudson & Zitnick, 2021) avoid the computational challenges of Bayesian inference that LOMSs address but lack the ability to decompose scenes into an interpretable hierarchy of representations.
3 LATENT OBJECT MODELS
3.1 GENERATIVE MODEL
We restrict the focus of this paper to unlabeled datasets D of i.i.d. samples of images x E NxC. N = HW, for clarity. Our goal is to fit the underlying data-generating process, which we propose to approach by using a three-level latent hierarchy. To that end, we define scene-level latent variable E IR D=, K object-level latent variables Z1, 22...., 2K, Zς E R. (or slots, which we abbreviate with k1 and pixel observations X, E RC, i L...., N. The joint distribution over observed and latent variables can be factorized as follows:
(x(x, Zi:K, s) = (( s)) ZI:K).
(1)
We adopt the VAE framework for approximate Bayesian inference (Kingma & Welling, 2013; Rezende al., 2014) since posterior inference is intractable and since we would like to parameterize these distributions with deep neural networks.
The scene-level prior p(s) is left standard Gaussian in this work for simplicity. However, the slot prior is more complicated, since generating coherent scenes requires modeling relational information between slots. Unlike topic models, cannot assume slots are i.i.d. when conditioned on We
3

---

Under review as a conference paper at ICLR 2022
instead assume fixed and deterministic order the slots informed by
pe(zi.K 8) II po(zz hes,, where h2 = fo (11:
(2)
where So is a deterministic function that transforms a sample from the scene prior and h1 - {8()). Each hig contains the order-aware information about which object the kthh slot should generate. By designing the slot prior such that the kth slot indirectly depends on slots Zi:k- through hg. the joint distribution over slots factorizes product of Gaussian conditionally independent (but not identically distributed) distributions. Notice that we replaced each h2 with & the LOM generative model becomes a bag-of- bag-of-words model.
The choice of
strongly influences the complexity of the learned scene representation. We use slot order-invarian Gaussian likelihood with the mean given by sum of the product of the image-shaped decoded masks ma and components Tk:
N
K Σ
( z1:K)
III N 1=1
M.A.Ti,,
The Gaussian likelihood is well-suited for generating complex scenes since it does not penalize solutions that use multiple slots fit parts of the background (Locatello al., 2020; Emami et al., 2021). By contrast, the Mixture-of-Gaussine et al., 2019; Greff et al., 2019; Engelcke et al., 2019; 2021) heavily penalizes the model for splitting parts of the scene across multiple slots. A scene can be sampled from LOM as follows:
likelihood used by some slot-based models (Burgess
1. Sample s from the scene prior p(s).
2. Compute h, = fo(s), h2 = So (11,,))) s)..,,,,.... = fo(h1xx-1,,5
3. Sample one slot from each the ps(z1 h1), Pe(22 h2)..... (.p)......pp(x hx)).
4. Render the image by decoding the K slots in parallel into masks my and pixel components and then aggregating them with Equation 3.
3.2 INFERENCE
We have introduced how LOMs model the scene generation process. Now, we turn our attention to the inference algorithm for the slot and scene latent variables. We want avoid directly estimating an ordered slot posterior using sequential scene decomposition since an optimal object ordering is unknown and difficult to learn without supervision. Our inference algorithm, visualized in Figure uses an auxiliary generative model that assumes slots are orderless to indirectly obtain an ordered slot posterior. Orderless scene decomposition uses randomized iterative assignment to bind percepts the slot variables (Greff et al., 2019; Locatello et al., 2020; Emami et al., 2021; Engelcke et al., 2021). The main idea behind our inference algorithm to impose an ordering on auxiliary orderless posterior with object ordering information provided by the deterministic variables k. The algorithm shares similarities with recent work that proposes to learn good canonical ordering for orderless sets to ease certain modeling tasks (Vinyals et al., 2016; Zhang al., 2019). An important consequence of having an orderless slot posterior is that we can define a coot-ordrr-nnaaiiin scene-level posterior:
(s g (o)
(4)
where 9o is permutation-invarian function. Orderless slots are denoted by o1:k. Each step of LOM inference is summarized here and visualized in Figure 2:
1. Estimate orderless slot posterior: Compute and sample from the orderless posterior qu
estimated by the auxiliary model (with inference network parameters w): O1:K (w1:K
x) T Uu (og x).
2. Estimate scene posterior: Compute and sample from the scene-level posterior: qs(s
(o(0).)))
3. Estimate scene-conditioned ordered slot prior: Compute the ordered prior po (0 using single sample and the causal function fo-
4

---

Under review as a conference paper at ICLR 2022
Step I.
(b) Step
Step 3.
(d) Step 4.
Step
Figure 2: Inference for Latent Object Models. Circles are slot latents. Dotted edges signify variables from the auxiliary orderless model and solid edges are variables from the LOM model. Boxes are deterministic variables. In steps 1-2 we infer an orderless slot posterior and the scene posterior. In step we use the scene posterior to generate slots via the deterministic variables. Step we align the orderless slot posterior to the prior. Finally in step 5 we estimate new order-aware scale parameters for the aligned slot posterior through another set of deterministic variables.
4. Align the orderless slot posterior: Compute matching between the K means from pe [Z1:K s) and the K means from 9u (O1:k x), e.g., by minimizing the pairwise L2 distance, then permute the order of the slot posterior means by to get (x(1):=(K)-
5. Estimate an ordered slot posterior: Compute new order-aware scale parameters ((k) for the permuted slot posteriors to obtain ordered slot posteriors qs (N) h (K)= x).
Here "= = wi (ha (s)))) with h==(== = (0()=)=<((-1) and Wy is a linear transformation. We introduce another causal function f' transform a sample from the scene posterior and the permuted orderless slots (L)) into the sequence h (1)) A simple and robust choice for computing an (K) greedy alignment (Algorithm in the appendix) that sequentially finds the best match for each slot. Each *a)) provides the ordered slot posterior with its aware- scale parameter:
K
K x) = II (O
II (o
(5)
k=1
Here, (*())) is just the kthh mean of the permuted orderless slot posterior.
Architectures for ferf' and got this work we use a Transformer (Vaswani et al., 2017) with multiple blocks of single-head causal self-attention (SA) to implement fe. The first token in the input is sample s ~ y(s) and the next K tokens are shared trainable embedding initialized with zeros. We implement l using nearly the same causal Transformer architecture as fa. The differences that the first token is now sample from the scene posterior and instead of using trainable embedding as input for the other tokens we pass the K permuted orderless slots (I)))=( For Jo aeruuaaion-invariiat Transformer with multiple blocks of residual SA and no positional encoding or causal masking. We aggregate the set output with a global sum pooling layer, which we find essential for capturing complex object relations in scenes. Details provided in the appendix.
3.3 TRAINING
LOMs can be seen as hybrid model that jointly trains an orderless VAE and an ordered VAE. Gradients are prevented from flowing back the orderless VAE from the ordered VAE during training (via stop_o Sarpp_grrdd gradd(ooda )) to ensure that orderless scene decomposition remains unbiased. To estimate qu (01:K x), we can use any slot-based VAE with orderless decomposition. In this work, we train LOMs with both EMORL and GENv2. For simplicity, we re-use their respective decoder architectures to implement the LOM image likelihood.
The ELBO for training LOMs is maximized jointly with the ELBO for the auxiliary orderless generative model. The LOM ELBO resembles the ELBO used by previous deep generative models with similar topic-model-inspires hierarchical factorization (Edwards & Storkey, 2016). We show it here presented as a sum of three terms and defer its derivation to the appendix.
Objective:
First is log- likelihodooo (i.e., reconstruction) term:
Lu = (eloeeoo() (egstennee (e [log 2xx)]]]
(6)
5

---

Under review as a conference paper at ICLR 2022
Then, have reverse Kullback-Leible (KL) term for the slots, which can be factorized sum over since the slot posterior and prior are conditionally independent and aligned via w:
K
DKL DKI(9o(Zz()) (((()) x) pa hu)))
LL = (slo
(7)
Finally, we have a scene-level reverse KL term:
LsceneLL
-D (g0(s go((1:))) || p(s)).
(8)
Altogether, we maximize L, the sum of these three terms and the auxiliary orderless model's ELBO:
L = Londerless + LLL + LotoIKL + L.ceneKL.
Impact the ELBO on fo: Intuitively, maximizing the ELBO encourages Jo to generate objects in consistent order that results in a low average value for LaskI. In practice, fo often learns to generate parts (or all) of the background with the same slot(s) and generate foreground objects using the same subset of slots (Figure 3).
To dynamically balance the reconstruction and KL terms for all models we use GECO (Rezende & Viola, 2018). GECO reformulates maximization of the ELBO into constrained minimization of the KL terms, where the constraint dictates a desired reconstruction quality C. Please refer the appendix for discussion
Optimization:
choosing C' as it is crucial for obtaining good performance. All other implementation details including architecture description, hyperparameters, and compute requirements also deferred the appendix.
EXPERIMENTS
Datasets: We use three synthetic multi-object datasets: Objects Room and CLEVR6 from the standard Oblti-OOject Dataset (Kabra et al., 2019) and ShapeStacks (Groth et al., 2018). All three datasets consist of rendered images of 3D scenes containing 3+ objects and have variable illumination, shadow, and camera perspectives. Successful object-centric scene generation requires reasoning about local object relations (e.g., occlusion in CLEVR6, placement of objects relative the floor in Objects Room) as well as global relations (e.g., stacking multiple blocks in ShapeStacks). We focus on synthetic scenes in this work since we wish to first establish correctness of the proposed LOM framework before tackling real-world scenes.
We compare LOM-EMORL and LOM-GENv2-G against the state-of-the-art slot-based VAE for scene generation GENv2, as well as EMORL, GNM, GENESIS (GEN), and the NVAE. EMORL has an independent slot prior that does not capture dependencies between objects; therefore, it serves as simple scene generation baseline. We train GENv2 with the default Mixture-of- Gaussians image likelihood (GENv2-MoG) on CLEVR6 as these results were not reported their paper. Results for GEN and GENv2-MoG (2021). We train GENv2 with the Gaussian likelihood on all three datasets (GENv2-G) for a more direct comparison the LOMs. See Section for detailed discussion on these models and the appendix for training details and hyperparameters.
Baselines:
the other two datasets provided by Engelcke et al.
Metrics: We multiple metrics to analyze the performance of all models in terms of object- centric decomposition and disentanglement, image generation quality, and the structural accuracy of generated scenes. The Frechet Inception Distance (FID) (Heusel al., 2017 7) is standard metric for quantifying the ability of a model to generate images similar to the training set. We follow Engelcke al. (2021) and use 10K real and generated samples to calculate this score. However, we found that the FID score heavily penalizes blurriness in generated images while ignoring whether the semantics of the generated scenes are accurate. Therefore, inspired by the evaluation of GNM (Jiang Ahn, 2020), on ShapeStacks we generate 100 images with each model and count the fraction of images containing a stack where all blocks in the stack are touching each other (structural accuracy, or S-Acc). To compare the ability to jointly generate high-quality scenes and learn object-centric representations, we compute the adjusted rand index for segmented foreground objects (ARI-FG) (Rand, 1971; Hubert & Arabie, 1985) all datasets and disentanglement scores (DCI) (Eastwood & Williams, 2018) on CLEVR6. The NVAE is only trained on CLEVR6 since it is the only considered dataset that provides ground-truth latent factors.

---

Under review as a conference paper at ICLR 2022
(a) ShapeStacks
(b) CLEVR6
(c) Objects Room
(d) Objects Room masks
(e) EMORL
(f) GENESIS
GENv2-G
(h) LOM-GENV2-G
ii GENv2-G
LOM-GENv2-G
Figure 3: (a-c) Unconditional samples from LOM (GenV2-G). d) LOMs learn relatively stable and deterministic assignment of objects to specific slots for generating scenes. The color of each segment indicates the slot number. e-h) A few generated ShapeStacks scenes. GEN scenes generated with a trained model released by the authors. LOM (GENv2-G) generates scenes that most accurately reflect multi-object structure in the data and with the fewest visual artifacts.
We ablate attributes of LOMs on a 48x48 resolution variant of Objects Room but defer these results to the appendix (Section A.5). Specifically, measure the importance of: the expressivity of fe.fø- and gø. computing order-aware scale parameters during LOM inference, and not sharing parameters between fo and f-- We also consider variants with autoregressive slot priors where try to (unsuccessfully) train them by directly minimizing the KL with respect to an orderless slot posterior.
4.1 QUALITATIVE EVALUATION
We first verify that LOMs are able to fit the data- generating process for each dataset by inspecting unconditional samples from the prior (Figures 3a-3d). Note that slightly scale down the standard deviations in the prior using a temperature for hierarchical VAEs (Vahdat & Kautz, 2020) to help keep samples in high probability regions. See
= 0.8 when sampling. which is common practice
7

---

Under review as a conference paper at ICLR 2022
Figure LOMs can generate interpolated scenes between two random samples from the scene prior by smoothly changing individual object and scene-level attributes. We use spherical linear interpolation to generate the intermediate latents between two random samples from p(s). This suggests the LOM has learned a good scene-level manifold.
Figure 7 in the appendix for temperature sensitivity analysis. Upon close inspection of the Objects Room scene masks (Figures 3d and 3j), we see that LOMs can consistently generate the wall, floor, and shapes with the same groups of slots (as indicated by mask color). By learning consistent and deterministic order with which to generate objects, LOMs are able to learn important dependencies between objects needed for generating scenes.
We compare Objects Room and ShapeStacks samples from slot-based VAEs in Figures 3e-3j. which the two datasets with the most complex multi-object structure. As expected, EMORL generates incoherent scenes with floating blocks due to its independent prior. GEN generates seemingly incoherent scenes from the visually challenging ShapeStacks dataset. In theory, its autoregressive posterior should be capable of learning ordered dependencies between slots; however, GEN inference tends heavily rely on color cues, which fails in datasets with textures and shadows. Samples from GENv2-G's autoregressive prior contain visual artifacts and structural inaccuracies (e.g., extra walls, missing floors Objects Room, floating blocks in ShapeStacks). Its orderless posterior cannot capture object dependencies, which makes it difficult for the autoregressive prior to learn to sequentially generative scenes from the challenging Objects Room and ShapeStacks datasets. By contrast, we see that LOMs use their ability to learn deterministic generation order and their scene-level representation to generate scenes that more accurately reflect the ground truth data.
It is also helpful inspect the I.OM scene-level latent space by interpolating between two random samples from the scene prior using spherical linear interpolation (Figure 4). The interpolated images resemble the training data which suggests that the LOM has learned good scene-level manifold. Scene-level attributes such as camera perspective and floor color change smoothly and the number of objects roughly remains at two. More interpolation examples (Figures 10, 11, 12) as well as qualitative comparisons (Figures 13-15). Figures 16-20 visualize reconstructions and segmentations for all models. We also explore how LOMs can hierarchically cluster scene dataset using scene-level and object-level relationships in the appendix (Section A.6).
provided in the appendix
4.2 QUANTITATIVE EVALUATION
We present the results in Table GNM has difficulty discovering and segmenting individual objects in Objects Room and StapeStacks which leads to much lower ARI-FG scores compared to the slot-based models (see, e.g., Figures 14 and 19). Despite this, GNM achieves low FID scores on all three environments. Its inability to decompose scenes in two out of three environments suggests that the slot-based approaches are more promising for likelihood- based multi-object scene generation.
Out of the slot-based models, LOM-GENV2-G achieves the best FID score and S-Ace on ShapeStacks. LOM-EMORL achieves the best FID score on CLEVR6. To quantify the improvement in generation quality gained by the LOM framework, LOM-EMORE and GENv2-G with LOM-GE GENv2-G. G. Removing sum pooling in the scene encoder Os decreases LOM-EMORI'S S-Ace by 36%, highlighting the role it plays in encoding relations
compare FID and S-Ace scores for EMORL with
8

---

Under review as a conference paper at ICLR 2022
Table Mean + std. dev. over three training runs. n.s.p. no sum pooling.
CLEVR6
Objects Room
ShapeStacks
Model
FID1
ARI-FGt
FID:
ARI-FG+
FID:
ARI-FG-
S-Ace (%)
GNM
27.5
0.97
51.645
4.20
49.3
0.37
100
GEN
62.8+a
0.63+o.08
186.8-1s
0.70+0.06
GENv2-MoG
61.0=3
0.98
52.6#a
0.84
112.7
#0.00
59
EMORL
244.0111
0.961...00 02
178.3127
0.47
10.22
258.41.87
0.6010.04
LOM-EMORL
52.0
0.9610.01
67.413
0.51 Lo.25
132.2115
0.604
76
LOM-EMORL-nsp.
139.1
0.62
40
GENv2-G
61.0aa
0.98 0.96+
87.6+4
0.00
115.3
0.68#0.02
42
LOM-GENY2-G
6
63.1
0.79+0.00
91.6+3
0.68
82
f The ARLEG EMORI: 0:64, 0.16 and (EMORL.) 0.69, 0.70, 0.16.
Table 3: Comparing object-centric representation quality of models with spatial latents (NVAE) and scene-slot hicrarchical latents on CLEVR6. Metrics are mean std. dev. over three training runs.
Model
FID,
ARI-FG:
Disentanglementr
Completenesst
Informativenesst
LOM-EMORL
52.0+6
0.9610.01
0.48-0.04
0.45
0.25
NVAE
28.314
0.1900.0
0.1610.0
0.0+0.0
involving many objects. In almost all settings, observe clear improvements. LOM-GENv2-G on CLEVR6 did not improve the FID compared to GENv2-G despite there being a noticeable qualitative improvement (Figure 15). We believe this is because GENv2's encoder introduces visual artifacts that artificially inflates the FID score, but this requires further investigation.
GENv2-MoG achieves lower FID scores than GENv2-G in Objects Room. We observed that GENv2- MoG segments the walls, floor, and sky into a single slot (Figure 13), whereas GENv2-G tends to segment the wall and floor separately (Figure 3i). GENv2's orderless posterior's inability learn slot dependencies means that GENv2-G G has the more difficult task of orienting the objects, walls, and floor during generation whereas GENv2-MoG simplifies this task, leading to GENv2-G's worse FID scores. By contrast, LOM-GENv2-6 successfully learns dependencies between objects and clearly improves GENv2-G's Objects Room FID score (Figure 3j). We believe that GEN achieves lower FID score than GENv2-G in Objects Room because GEN's autoregressive posterior, which is heavily reliant on color cues, able to successfully model slot dependencies this dataset. GEN clearly has difficulty with the more challenging StapeStacks dataset and achieves much higher FID scores. In general, LOMs maintain the decomposition performance (as measured by ARI-FG) as the auxiliary orderless VAE. Finally, Table shows that although the NVAE achieves low FID on CLEVR6, its poor disentanglement scores suggests that its spatial latents do not encode interpretable object-centric information about scenes.
5 CONCLUSION
We presented probabilistic framework for learning an interpretable hierarchy of scene and slot latent variables. The learned representation shares advantages of both single scene-level representation as well as object-centric slots. This work improves our understanding on how to develop object-centric VAEs, particularly with respect to accurately modeling scene-level statistics.
We believe the interpretation of the deterministic variables in the LOM graphical model could benefit from further study. They are essential to the success of LOMs but trade off the ability to model stochasticity in the order in which objects are generated. We observed that LOMs occasionally generate objects that intersect each other due to lack of 3D understanding in the latent representations. Although unsupervised 3D scene inference is particularly difficult due to ambiguities introduced by perspective geometry, obtaining 3D object representations should improve generation quality (Chen et al., 2020). Our LOM implementation could also be improved by replacing the standard Gaussian scene prior with more flexible distribution such mixture-of mixture-of-Gauusians

---

Under review as a conference paper at ICLR 2022
REFERENCES
Titas Anciukevicius, Christoph H Lampert, and Paul Henderson. Object-centric image generation with factored depths, locations, and appearances. arXiv preprint arXiv:2004.00644 2020.
David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3:993-1022, 2003.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 1390,, 2019.
Chang Chen, Fei Deng, and Sungjin Ahn. Object-centric representation and rendering of scenes. arXiv preprint arXiv:2006.06110. 2020.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprint arXiv:2011. 2020.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1409.4009//00 /259, 2014.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3412-3420, 2019.
Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. International Conference on Learning Representations, 2021.
Andrea Dittadi, Samuele Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning, arXiv preprint arXiv:2107.00637, 2021.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
Harrison Edwards and Amos Storkey. Towards a neural statistician. arXiv preprint arXiv:1606.02188 2016.
Sebastien Ehrhardt, Oliver Groth, Aron Monszpart, Martin Engelcke, Ingmar Posner, Niloy Mitra, and Andrea Vedaldi. Relate: Physically plausible multi-object scene synthesis using structured latent spaces. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 11202-11213. Curran As- sociates, Inc., 2020. URL https: http:///pocccceeseeeeeeeeeec neurips http//aper/22000 file/ 806bea 54 032 Bet.bb811884444440eeeee
Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan. Efficient iterative amortized inference for learning symmetrie and disentangled multi-object representations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2970-2981. PMLR, 18-24 Jul 2021. URL https: / pptppeeiccenggssee mlr press//e/////e..eee / 39/ emami 2 21a. html.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gener- ative scene inference and sampling with object-centric latent representations. arXiv preprint arXiv:1907.13052 2019.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-V2: Inferring unordered object representations without iterative refinement. arXiv preprint arXiv:2104.099: 09400998 2021.
SM Ali Eslami, Nicolas Heess. Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
10

---

Under review as a conference paper at ICLR 2022
Mario Fritz and Bernt Schiele. Decomposition, discovery and detection of visual categories using topic models. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2008.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings. 2010.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel
Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In Kamalika Chaudhuri and Ruslan Salakhutdinoy (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2424-2433, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http: httpp/ppoooens.sep/eepp ml press /v37/v/////// /99/ggeff1 9a. html.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05200. 2020.
Oliver Groth, Fabian B Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision- based physical intuition for generalised object stacking. In Proceedings of the European Conference Computer Vision (ECCV), pp. 702-717, 2018.
Dan Hendrycks and Kevin Gimpel. 1606,08660555 2016.
Gaussian error linear units (gelus). arXiv preprint
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Thomas Hofmann. Probabilistic latent semantic indexing. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 50-57, 1999.
Lawrence Hubert and Phipps Arabic. Comparing partitions. Journal of classification, 2(1):193-218 1985.
Drew Hudson and Larry Zitnick. Generative adversarial transformers. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4487-4499. PMLR, 18-24 Jul 2021. URL https: http:::://proceediii egss mlr press// / vv13vv399/// htt.ddsoo21111 html.
Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines.
arXiv preprint
1221:22010.12152. 2020.
Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. object-datasets/ 2019,
Jinwoo Kim, Jaehoon Yoo, Juho Lee, and Seunghoon Hong. Setvae: Learning hierarchical composi- tion for generative modeling of set-structured data. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 15059-15068, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980. 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 2013.
arXiv preprint
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29:4743-475) 2016.
11

---

Under review as a conference paper at ICLR 2022
Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3d controllable image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5871-5880. 2020.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint arXiv:200..02007, 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv preprint arXiv:2006. / 15055, 2020.
Lars Maalée, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: http:///ccccc.cc.cc.////////..se neur neurips. cc/ paper/2018 9/ file/ 9bdb8bl faffa 4b344444331111 97b44499994499 9 Paper. pdf.
James Munkres. Algorithms for the assignment and transportation problems. Journal of the society for industrial and applied mathematics, 5(1):32-38, 1957.
Li Nanbo, Cian Eastwood, and Robert Fisher. Learning object-centric representations of multi-object scenes from multiple views. Advances in Neural Information Processing Systems, 33, 2020.
Thu Nguyen-Phuoc,, Christian Richardt, Long Mai, Yong-L.ang Yang, and Niloy Mitra. Block- gan: Learning 3d object-aware scene representations from unlabelled images. arXiv preprint arXiv:2002.08988 2020.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11453-11444 11464, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performant Deep Learning Library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846-850 1971.
Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint
2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. 31st International Conference on Machine Learning, ICML 2014, 4:3057-3070, 2014.
Josef Sivic, Bryan C Russell, Alexei A Efros, Andrew Zisserman, and William T Freeman. Discover- ing objects and their location in images. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, volume 1, pp. 370-377. IEEE, 2005.
Casper Kaae Sønderby, Tapani Raiko, Lars Maalde, Spren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neurul information processing systems, pp. 3738-3746,
2016.
Tinne Tuytelaars, Christoph H Lampert, Matthew B Blaschko, and Wray Buntine. Unsupervised object discovery: A comparison. International journal of computer vision, 88(2):284-302. 2010.
12

---

Under review as a conference paper at ICLR 2022
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 19667-19679. Curran Asso- neurips httpssccppapee/222//5/ee
ciates, Inc., 2020. URL 33b2125511@ae7722244666bb 6.5 pdf.
Sjoerd van Steenkiste. Karol Kurach, Jürgen Schmidhuber, and Sylvain Gelly. Investigating object compositionality in generative adversarial networks. Neural Networks, 130:309-325, 2020. ISSN 0893-6080. doi: https///d..ogg///.0000d....emmm 20200.0.0...00 URL https: www.sccience... rect. wttp:/ccoocccciiiee /ttpc//cc////caaa///////////////
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua B Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. arXiv preprint arXiv: 12827, 2019.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. International Conference Learning Representutions, 2016.
Julius Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. arXiv preprint arXiv:2004.12906 2020.
Yan Zhang, Jonathon Hare, and Adam Prüngel-Benntt. Learning representations of sets through optimized permutations. International Conference on Learning Representations, 2019.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of AAAI, 2021.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
Ps. d
surn pooling MLP
residual self-attention block
3x
residual self-atlention block
O,
(a) 9
(b) gi w/out sum pooling
d' Mo; d' (4,, de Hs, d5
1
1
N
h1
hg
h1
hg
h5
residual self-attention block
2x
residual self-attention block
2x
position encoding
position encoding
Oz[1; Ox(2) Or(S) 0x(4) On(5)
(c) fo
(d) So
Figure 5: The Transformer architectures used to implement the LOM. visualize each with K The output(s) of each function is mapped to Gaussian parameters μ,, a, which we annotate onto each diagram. a) The scene encoder 9g. We do not use positional encoding or causal attention masking in the slot-order-invarian scene encoder. The last input o, is trainable embedding. b) For the ablation study on the impact of the global sum pooling layer on Jé. we directly map the (K + 1)st element of the output set to the Gaussian scene posterior parameters. The function estimates the ordered slot prior parameters from a sample from the scene posterior s and a shared trainable embedding (grey). d) We estimate the ordered slot posterior scale parameters with the outputs h' of f/g
A. NEURAL ARCHITECTURES
We describe the details of the neural architectures used in this paper implement LOMs. We use GELU non-linear activations (Hendrycks & Gimpel, 2016) and xavier weight initialization (Glorot & Bengio, 2010).
We construct the commutation-invarian scene encoder and the causal slot prior/posterior by stacking residual self-attention (SA) blocks (Figure 5). Each SA block based on the Transformer (Vaswani et al., 2017) design. For each such block, we apply single-head scaled dot-product attention an input æ E ND
Residual self-attention block:
M())g(=)
a = softmax
)u(x))
(10)
VD
M (Laeerayeo + +a).
(11)
The k; u linear maps D D projections and the MLP is D. D, projection. In experiments we set D2 to 64.
Scene encoder 9s: The scene encoder 96 (Figure 5a) has 3 residual SA blocks. It takes input an orderless set of slots oı:k where each slot is a D--dim vector. We use the means of the orderless slot posterior for the input in practice. We append Dim trainable scene embedding o, position
14

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Greedy matching algorithm for π. Uses O(K) time and O(K2) space.
pk 2: C(i. [ = 11 - 19112, V(i,j)
1: Input: Ordered means uf...., ......... orderless means pi 1.. 1k with pe E IR.
3: π -=
4: for index i= 1....... K do
5:
j= arg min Cii.
6:
+l:j"+==nnt
7:
π T||= j
8: end for
9: return Permutation π
K + 1 so that the input to Is is set of size K + 1. After the 3 residual SA blocks we apply sum pooling layer that aggregates the output set by summing the K + vectors and projecting the result to the scene latent dimension D. with an MLP. Then, two linear layers are used to obtain the mean and softplus-actives activated standard deviation for the Gaussian scene posterior. Our ablation study analyzes the number of residual SA blocks and the impact of the sum pooling layer on the ability to capture higher-order object relations (Figure 5b).
Causal Transformer fa: A straightforward architectural choice for implementing fo is recurrent neural network (RNN) such as a GRU (Cho et al., 2014), which have a linear time dependency the
on the input sequence length. By contrast, Transformers have quadratic dependency input sequence length; however, this computation can be efficiently parallelized, resulting in a constant time dependency in practice. Moreover, they have superior ability to capture long-range interactions between elements of the sequence due to the use of self-attention (Radford et al., 2019; Brown al., 2020; Zhou et al., 2021). Therefore we use a Transformer with single-head causal self-attention (SA) to implement fo. In detail, the first token in the input sequence is the sample s p(s) and the subsequent K tokens in the sequence are set to a shared trainable embedding of dimension Dz initialized with zeros. The scene sample is projected to the same dimension as the D.-dimensional embeddings with a single layer MLP. To provide extra positional information add positional encoding the input sequence. This is implemented by mapping the values [0/(K- + 1), 1/(K + 1)..... (K + 1)/(K + 1)] to Dj -dddimensional vectors with a linear layer and adding this to the input. We use the standard approach for implementing causal self-attention by masking out attention values in each of the residual SA blocks. The length K output sequence, starting from the second output token, are the h1.k- These are each mapped to Gaussian parameters for each pa [z (z generation networks maximize the ELBO, which we discussed Section 3.3.
hx) with two linear layers. We train fo end-to-end alongside the inference and
Causal Transformer ln= For the causal Transformer that predicts the order-dependent scale pa- rameters for the ordered slot posterior, we use the same architecture as fe except for few minor differences. Primarily, the inputs are now sample from the scene posterior and the means of the permuted orderless slot posterior. The outputs of the stack of SA blocks are mapped with a single linear layer to the D.-dim softplus-activated standard deviations of the ordered slot posterior. The means of the ordered slot posterior are fixed be the means of the permuted orderless posterior.
A.2 GREEDY MATCHING ALGORITHM
Pseudocode for the greedy matching algorithm we use to align the orderless slot posterior to the ordered slot prior is provided in Algorithm 1. Alternative matching algorithms such as the Hungarian algorithm (Munkres, 1957) can be used instead at a higher computational cost. We did not empirically explore other matching algorithms since the greedy solution worked well initially.
15

---

Under review as a conference paper at ICLR 2022
A.3 DERIVATION OF THE LOM ELBO
We assume that we are given a sample O1:K from an orderless slot posterior estimated by an auxiliary orderless generative model. We can derive lower-bound the log marginal likelihood as follows:
log p(x) = log /
po s, (m():==(K)) dds, dz1:K
(12)
(o (x(()==((K)
x)
log
Po (x, Z1.K) dz
(13)
(s, ZT(1)==(())
x)
(x, 21:Kk)
log Eg
(14)
(s, (x())=
x)
Applying Jensen's inequality, we get:
po 8, 21:k)
log log)(x) (y log (s. (T))s
(15)
x)
po 21:k (oo (21: |s)((s)
B Faste
log ( go( (lg.)))g(ee (2x()())
h (x)= x)
(16)
We can treat the inner log product of terms sum of logs to get the following three loss terms.
First is a log- likelihood loss term:
LL = (5lo((())))))))) E (x)) (x)))))
[logpe(x (x(1)=[(<1]]]
(17)
Second is slot reverse KL. divergence loss:
log pa (Z [s)
LKI_ = E (o
E
(18)
2 (o(2x(1)s<())
h ((
(s DKL (go(Za())
Ih's [K)) x) po (21:K {))
(19)
Since both the ordered slot posterior and the ordered slot prior are products of K conditionally independent distributions, this reverse KL further factorizes as a sum over K:
DEL (15((()))) (a(a): x) po(2x mas)
LKI. = (eas
(20)
Finally, there is a scene-level reverse KL divergence loss:
p(s)
CKL.s = E (xlo((oo)))
Bo log go(s (oLk))
(21)
log p(s) go(s go(or.k))
= (B
(22)
log (s (o(()))))
(23)
p(s)
=-D =Doki(gos go(ol:k)) p(s))
(24)
Therefore, we have the following ELBO we wish to maximize with respect to parameters 0,
logp(x) 2 Lu. + LKI_ + LKI.S-
(25)
In practice, we approximate this ELBO by averaging over a minibatch dataset samples of size B and use stochastic gradient ascent.
A.4 EXPERIMENT DETAILS
An open source implementation of the code for replicating all experiments will released with link placed here. Videos demonstrating latent space random walks will also be released. We provide the generated samples and our labels used to compute the structure accuracy metric in the supplementary material.
16

---

Under review as a conference paper at ICLR 2022
A.4.1 HYPERPARAMETERS
We trained two LOM variants, one that uses EfficientMORL (Emami et al. 2021) (EMORL) to extract the auxiliary orderless posterior and one which uses GENESIS-v2 (GENv2) (Engeleke et al., 2021). The authors of both models have released open source implementa- tions. 12 For EMORL, we keep their architecture and hyperparameters fixed except that the number of iterative refinement steps is set at 2 for all experiments. We use the Gaussian image likelihood with a global standard deviation of 0.7 for the Objects Room and ShapeStacks datasets. For CLEVR6, we needed to lower the global standard deviation to 0.1 to achieve sharp object segmentations. We use the refined approximate EMORL posterior for LOM inference. When using GENv2 to imple- ment LOM, we disable the autoregressive prior and replace it with independent Gaussian slot prior. We also used a Gaussian image likelihood instead of the default mixture-of- ofasssssannss so that both LOM-EMORL and global standard deviation of 0.7 for the Gaussian likelihood on all three env environments. We use the approximate posterior estimated from the output of GENv2's non-parametrie clustering algorithm for LOM inference. We set the slot dimension to 64 for all models. Please see the respective papers and code for EMORL and GENv2 for complete architecture and hyperparameter details.
Auxiliary orderless models:
G used the same likelihood. LOM-GENv2- uses
We maintain the same slot dimension (D= 64) and image likelihoods as the auxiliary orderless models on all environments. The scene latent dim D, is 128 which is chosen to be twice the slot dimension. We use the same decoder architectures from EMORL and GENv2, respectively, for mapping slots to image-shaped masks and components. The optimization hyperparameters are also kept the same as EMORL and GFNv2 for each LOM. For LOM-EMORI, both the auxiliary model and the LOM are trained jointly with the Adam optimizer (Kingma Ba, 2014) with default PyTorch (Paszke et al., 2019) hyperparameters and a learning rate of 4c-4. EMORL's learning rate schedule is a linear warmup for 10K steps then multiplicative decay with by rate of 0.5 every 100K steps. RDM-GENv2-G- G is trained in similar manner, again using Adam and learning rate of 1e-4 but without any learning rate schedule.
LOM:
Datasets:
Following Engelcke et al. (2021) we use K 7 and K 9 slots respectively for Objects Room and ShapeStacks. For CLEVR6, we use K = 7 slots following standard practice on this popular multi-object benchmark. LOM-EMORL pre-processes all images by converting RGB values to the range [-1, 1] before passing them as input to the model. Objects Room can be accessed freely online. 3 This dataset contains 64x64 RGB training images. The StapeStacks dataset is also freely available for download online and also has 64x64 RGB images.". 4, We use the same preprocessing protocol for CLEVR6 as Emami et al. (2021), which is to center crop the images to 192x195 and the resize them to size 96x96. This dataset is available as part of the Multi-Object Dataset.
A.4.2 BALANCING RECONSTRUCTION AND KI.
GECO: Both EMORL and GENv2 use GECO to balance reconstruction and KI. losses (Rezende & Viola, 2018). We use the respective implementations of GECO from each code base for training LOM-EMORL and LOM-GENv2-G.
For LOM-EMORL we customize the GECO target for each dataset, which is dependent on the image resolution and the choice of image likelihood. For Objects Room, we used per-pixel and per-channel target of -0.352 (for unnormalized Gaussian log- likelliiioood..... Due to the increased visual complexity of ShapeStacks, we lowered this target to -0.351 to achieve a better trade-off of KI. and reconstruction. On CLEVR6 we used a target of -2.265 (for unnormalized Gaussian log-likelihood with a = 0.1). The rule of thumb used to tune the GECO target is that the target should be reached after about 15-20% of the training steps. After the target is reached, the GECO Lagrange parameter is automatically decreased to I so that a valid ELBO is maximized. We use a GECO learning rate of le-6 for updating the Lagrange parameter and multiply this by an acceleration factor once the target is reached. Acceleration factors of 5, 10, and 50 were used on Objects Room, CLEVR6, and StapeStacks respectively.
17

---

Under review as a conference paper at ICLR 2022
As mentioned, LOM-GENV2-G G is trained to optimize unnormalized Gaussian image likelihood like LOM-EMORL. For Objects Room and ShapeStacks we used a per-pixel and per-channel target of -0.353 and increased this to -0.356 for CLEVR6. We found that needed to decrease the GECO learning rate from le-5 to 1e-6 for these datasets compared to the default GECO learning rate used by GENv2. We note that the GENv2 codebase had a better GECO implementation than EMORL that was less sensitive to particular hyperparameters and with more robust annealing schedules. Improving the GECO implementation used by EMORL should help make tuning these hyperparameters easier for replication.
Freebits: We found it beneficial to use a freebits parameter À on some of the KL terms to help avoid posterior collapse (Maalée et al., 2019). In detail, for minibatch of size B the greebits-uugmntt scene KI. is:
B
DKI(uols go(o1.)) || p(s)) ~
max (A, DxL (go(s) go (o(()))) ll p(s)))
(26)
B
This encourages the model to keep the KL at least as large A. For slot-based distributions that factorize as products over K, we take the max of X and the sum over K slot-wise KL. terms. We use this for each of EMORL's hierarchical KL terms as well as the LOM scene KL (but not for the LOM slot KL since the posterior means are fixed via the stopped gradients, meaning that we would like to push this KL as close zero as possible). We use freebits for the L scene KL well. We use À = 0.75 x D= for each KL term in EMORL and À = 0.25 x D2 for the LOM scene KL. Inspired by Maaløe et al. (2019) we tried annealing À down slightly to 0.167 D, for the scene KL term during the last 25K steps of training and found that this slightly increased performance.
A.4.3 BASELINES AND COMPUTE
The authors of GENv2 have released pre-trained weights for GEN trained on StapeStacks and GENv2 trained on Objects Room and ShapeStacks. We use these weights for model visualizations and to compute the structure accuracy metric. To train GENv2 on CLEVR6 with the default mixture-of- -fa--sssiaaaanss image likelihood, we had to lower the standard deviation to 0.1 to get it to work decently well. We adjusted the GENv2 GECO target accordingly, tuning it to -2.265 (per-pixel and per-channel value). We arrived at this value by starting with -61000, which is for the unnormalized miittue-of--Gausiian 0.1 used by Emami et al. (2021)), and then increasing the target slightly to achieve a high-quality reconstruction. We also trained GENv2 on all three environments using the same Gaussian image likelihood as LOM-GEN-2-G for a more fair comparison.
GENESIS and GENESIS-V2:
summed over pixels and channels with standard deviation of
GNM: We the official GNM Py Torch implementation provided by the authors. 5 We train GNM on the 128x 128 resolution version of CLEVR6 so that we could use the default GNM hyperparameters that the authors used for the 128x 128 resolution CLEVR-based environment from their paper. This worked well for CLEVR6. By default, GNM organizes its symbolic variables into a 4x4 spatial grid, which we maintained. We treat each spatial grid cell as one slot for a total of 16 slots. To train GNM on the 64x64 Objects Room and ShapeStacks scenes, we removed one downsampling and one upsampling layer from the encoder and decoder to account for the resolution being one-half that of CLEVR6. For all environments, we used the default global latent dim of 32 and Swha dim of 64.
The NVAE is trained with the open source implementation provided by the authors.5 6 We changed the image likelihood from the discrete mixture of logistic distribution to the Gaussian likelihood with standard deviation of 0.7 for a fair comparison with other considered models. We also modified the default NVAE architecture for 64x64 images for use with CLEVR6 as follows. To decrease memory consumption, we remove the normalizing flows and use 16 initial channels in the encoder and decoder towers. The NVAE has three scales of spatial latents, and we extract the top-level at the lowest resolution of 12x12. Since each spatial latent group has 20 channels, this becomes 2880-dim vector when flattened. We follow the same protocol as (Greff et al., 2019; Dittadi et al., 2021) to compute disentanglement scores for NVAE by imposing a fixed order on objects in the scene based on their attributes.
NVAE:
ang ang/GNM
'https: / http:://ggtttubbuc thub. com wwtp///1lbbb...... abs /NAAR
18

---

Under review as a conference paper at ICLR 2022
340
180
blocks)
160
black
divergence
320
140
3. causal Trans. block
causal Trans. blacks ardertess postertor
120
6. shared ceusel Trans.
7. autoreg. prior, no
8. autoreg. prior, w/ s
240
1
2
3
4
5
7
8
Model
(b)
Figure 6: Ablation and variation results. Box plots shown for the five random seeds. a) Test KL for ablated models. Lower is better. b) FID scores for ablated models. Lower is better. c) Unconditional samples from the Model variant, scene-conditioned autoregressive slot prior trained to directly match an orderless posterior (using EMORL as the orderless generative model). The autoregressive prior cannot generate coherent scenes as orderless posteriors do not model any dependencies between objects. We observe similar results with the V2nV2-G baseline.
Compute: We use batch size of 32 for LOM-EMORL and LOM-GENY2-G, For LOM-EMORL, we split this across 8 NVIDIA A 100 GPUs and train for 400K steps on Objects Room and ShapeStacks, which takes approximately 27 wall-clock hours each run. We train LOM-EMORL for 300K steps on CLEVR6 which takes hours. On Objects Room and ShapeStacks, training the GENv2 baseline with GPUs and batch size of 32 takes about 20 hours to reach 500K steps (following the authors's training protocol). We train GENv2 for 300K steps on CLEVR6, which takes 24 hours. LOM- GENv2-G takes approximately similar amount of time. GNM takes about 9 hours to reach 500K steps with a batch size of 32 on one A100 GPU for Objects Room and ShapeStacks. It takes about 12 hours to reach 500K steps CLEVR6
A100 GPU. The NVAE converges quickly CLEVR6 because we are able to use large batch size of 256 split across 8 GPUs. We found it takes about 10- 14 hours to converge in this setup depending random seed.
A.5 ABLATION AND VARIATION STUDIES
Setup: We train each ablation model variant of LOM-EMORL, defined below, across five random seeds on a 48x48 resolution version of Objects Room.
Model 0:
The baseline LOM-EMORL model, which uses 2 residual SA blocks both the scene encoder 96 and the causal Transformers fe. fg-
Model 1:
The number of residual SA blocks in 96 is decreased to 1.
Model 2:
The number of residual SA blocks in de is increased to 3.
Model 3:
The number of residual SA blocks in fe. fa is decreased to 1.
Model 4: The number of residual SA blocks in fe. fa is increased to 3.
Model 5:
During LOM inference, skip the step of computing scale parameters for the aligned orderless slot posterior. Instead, we re-ur re-use the scale parameters estimated by the auxiliary orderless model.
Model 6: We share parameters between So and f/-
19

---

Under review as a conference paper at ICLR 2022
80
fid
70
60
0.6
0.7
0.8 temp T
0.9
1.0
Figure 7: FID sampling temperature (7) Objects Room. Error bars show std. dev. across three seeds. The blue solid curve LOM-EMORL and the orange dashed curve is LOM-GenV2-G. We use 0.8 for all LOM results.
Model 7: We consider an alternative approach to scene generation by means of an autoregressive prior. We minimize the reverse KL divergence between EMORL's orderless slot posterior and an RNN-basee autoregressive slot prior, stopping gradients from passing back through the posterior. This KL is added as auxiliary loss term to EMORL's ELBO. Note that the scene-level latent is removed in this variant and the multi-step LOM inference algorithm is not used since we are simply directly minimizing the KL between an orderless posterior and the autoregressive prior.
Model 8: We extend Model by adding the scene-level latent. We condition the first slot of the autoregressive slot prior on the scene-level latent The scene-level reverse KL term is also added to EMORL's ELBO as an auxiliary loss.
Results: See Figure We found that increasing the expressivity of the scene encoder Is by using 3 SA blocks (Model 2) instead of 2 (Model 0) slightly increased FID, whereas decreasing the number of SA blocks
in Je (Model 1) and in the causal Transformers fa. f% (Model 3) lead to pronounced decline in performance. Increasing the number of SA blocks to 3 for fe. fg saw negligible improvement (Model 4). We observed that not computing new order-aware scale parameters for the LOM ordered approximate slot-level posterior (Model 5) resulted in unfavorably high test KI. and FID scores. This verifies that computing the order-aware scale parameters for the LOM approximate ordered posterior is important for learning the LOM prior. Next, we found that sharing parameters between fo and f/ severely degraded the FID (Model 6).
The autoregressive prior variants (Model 7 and Model 8) achieve the worst FID scores and KL across all variants. However, adding the scene latent to the autoregressive prior (Model 8) improves the FID score relative Model 7. We visualize samples from Model 8 in Figure 6c. The orderless EMORL slot posterior (i.e., slots are independent of each other) does not provide sufficient slot dependency information train the autoregressive prior sequentially generate objects in a coherent way. The generated scenes appear to be mostly incoherent and have multiple walls, missing floors, and floating objects. We observed similar phenomena with GENv2, which attempts to fit an autoregressive slot prior with an orderless approximate slot posterior.
Sampling temperature: We plot how the Objects Room FID varies with respect the temperature E {0.6, 0.7, 0.8, 0.9, 1.0} in Figure 7. Decreasing the temperature to 0.8 helps samples stay within regions of high probability when sampling from the hierarchical prior. We observe small corresponding increase in FID.
A.6 HIERARCHICAL CLUSTERING EXAMPLES
We provide illustrative examples of how the hierarchical representation inferred by LOMs hierarchi- cally clusters the dataset via to use the scene-level latent variable to first retrieve nearest- neighbt scenes based on scene-level similarities. Then, the retrieval results can be re-ranked by selecting single object's slot from the
retrieval application. For given query scene, the basic idea is
20

---

Under review as a conference paper at ICLR 2022
(a) Top-K K nearest seone retricval using s.
(b) Re-ranking results based similarity to the green cone.
Figure 8: Hierarchical clustering example. The top 5 retrieved scenes based on Euclidean distance between their scene-level latent and the query scene latent (the query is shown the left, and scenes are shown left-right in decreasing similarity rank). a) The most similar scene has similar camera perspective, wall color, object color, and object distance from the camera the query. b) We the slot for the green cone (boxed) to re-rank the results against the slots for each retrieved scene. The top retrieved scene has the exact same camera perspective and relative object placement (shown with arrows) as the green cone.
query and matching against slots of the retrieved scenes. We would expect that after re-ranking, the most similar scene the query would also contain object that matches the selected query object in one or more object-level attributes.
We created database of 50K scenes stored as (s, z) pairs of vectors from the Objects Room training set. For the Euclidean distance between latent vectors to return a meaningful value, we found necessary first PCA to project the 128-dim scene latents down to 6-dim vectors. We ran PCA the database of 50K scene latents and inspected the explained variances of the principle components select the projection dimension of We select all queries from held-out samples not included in the initial 50K used to create the database. We visualize examples in Figure and Figure 9.
A.7 ADDITIONAL QUALITATIVE RESULTS
Additional examples spherical linear interpolation (SLERP) between two random samples from the LOM scene prior are shown in Figures 10, 11, 12. To create these images, normalize each sample and then multiply by VD, before computing the intermediate latents.
21

---

Under review as a conference paper at ICLR 2022
Figure 9: Additional hierarchical clustering examples. Similar to Figure 8, the query is shown on the left, the top shows the initially retrieved scenes in decreasing order of similarity, and the bottom row shows the re-ranking based on the selected slot (shown with black boxes). For StapeStacks, found that the initial scene-level retrieval was highly successful at finding scenes with similar camera perspective and scene background as the query.
Figure 10: Objects Room SLERP between two random samples from the LOM scene prior,
22

---

Under review as a conference paper at ICLR 2022
(a)
(b)
Figure 11: CLEVR6 SLERP between random samples from the LOM scene prior.
(a)
Figure 12: ShapeStacks SLERP between two random samples from the LOM scene prior.
23

---

Under review as a conference paper at ICLR 2022
GNM
EMORL
GENv2- -MW
GENv2-G
LOM GENv2-G
LOM EMORL
Figure 13: Objects Room unconditional samples from each generative model with both images and masks displayed.
24

---

Under review as a conference paper at ICLR 2022
GNM
GENESIS
GENv2-MOG
GENv2-G
LOM GENv2-G
LOM EMORL
Figure 14: ShapeStacks unconditional samples from each generative model with both images and masks displayed.
25

---

Under review as a conference paper at ICLR 2022
GNM
EMORL
GENv2-MOG
GENv2-G
LOM GENv2-G
LOM EMORL
Figure 15: CLEVR6 unconditional samples from each generative model with both images and masks displayed. For EMORL, the samples shown in columns one through three are drawn from the prior of model that scores average ARI-FG of 96 and FID of 244. The remaining three samples are drawn from a different model trained keep the KI. low throughout training and which achieves lower average ARI-FG of 89 but FID score of 75.
26

---

Under review as a conference paper at ICLR 2022
input
recons
segmentation
input
recons
segmentation
Input
recons
segmentation
Figure 16: LOM-EMORI. reconstruction and segmentation.
27

---

Under review as a conference paper at ICLR 2022
Input
recons
seg mententiion
input
recons
segmentation
Input
recons
segmentation
Figure 17: LOM-GenV2-G reconstruction and segmentation.
28

---

Under review as a conference paper at ICLR 2022
Input
recons
segmentation
input
recons
segmentation
input
recons
segmentation
Figure 18: GENv2-G reconstruction and segmentation.
29

---

Under review as a conference paper at ICLR 2022
input
recons
segmentation
Input
recons
segmentation
Input
segmentation recons
Figure 19: GNM reconstruction and segmentation. Its symbolic variables, which are bound spatial grid of fixed resolution, are unable to properly bind to individual objects in the Objects Room and ShapeStacks scenes. Objects Room, the 3D shapes are segmented into strips and grouped alongside parts of the floor. The wall is erroneously not considered one of the foreground objects. In ShapeStacks, distinct blocks are often segmented together when they fall within the same grid cell.
30

---

Under review as a conference paper at ICLR 2022
input
recons
(a) Reconstruction
(b) Random samples
Figure 20: NVAE qualitative results.
31


Reviews:
[{"review_id": "wPfmDMamSM", "reviewer": "Reviewer_4aYc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n1. The paper is well-written and the proposed method is clearly described.\n\n2. The proposed method aims to tackle the difficulty of modeling and inferring relationships of objects, which is an important problem in object-centric representation learning of scenes.\n\n**Weaknesses**\n1. The novelty is limited as the generative model is similar to GENESIS (Engelcke et al., 2019) in that the relationships of objects are modeled by an RNN, while the inference method is similar to Slot Attention (Locatello et al., 2020).\n\n2. The main claim/contribution that using a sequence of deterministic variables (referred as `causal sequence’ in the paper) to block dependency between scene-level random variable $s$ and slot-level random variables $z_{1:K}$ seems problematic. As shown in the generative process (Eq.1), $z_{1:K}$ still have dependences conditioned on unobserved $s$.\n\n3. Using a causal sequence to model orders of slots is questionable. Although there are dependences between slots, the joint distribution of slot-level latent variables should be invariant under permutation because there are no causal effects among static objects. \n\n4. Relationships between objects are not considered when estimating orderless slot posteriors. Besides the regularization of the KL divergence term, how to improve the inference of slot-level latent variables by utilizing the learned priors of object relationships is unexploited.\n\n5. Only the scene generation performance is compared in the paper, the scene decomposition performance (e.g. ARI and AMI), which is of vital importance to evaluating object-centric learning performance, is not included. In addition, the reconstruction performance (e.g. Log-Likelihood and MSE) is also missing.\n\n6. The proposed method is not compared with GNM (Jiang & Ahn, 2020), which also has scene latent and considers relationships among objects, and many existing relevant object-centric scene representation methods. The overall scene generation performance of the proposed method is inferior to GENESISv2 (Engelcke et al., 2021).\n\n**Questions**\n1. How to select the hyperparameter $K$?\n2. How to avoid representing the same object with multiple slots?\n3. Is the proposed method able to generate images containing a specific number of objects?\n4. Is the proposed method able to generate images containing fewer or more objects than those used for training?\n", "summary_of_the_paper": "This paper proposes a structured latent variable model, called Latent Object Models (LOM), for compositional scene generation. To generate a scene image, LOM first samples a scene-level latent variable, and then samples multiple slot-level latent variables based on a sequence of deterministic variables modeled by an autoregressive model. To infer the ordered slot-level latent variables, LOM first estimates orderless slot posteriors, and then determines the permutation of orderless slot posteriors based on a greedy matching algorithm. After learning, the model is able to generate scene images with reasonable relationships of objects. Experiments are conducted on three datasets, and the proposed method is compared with several existing methods.", "summary_of_the_review": "This paper considers an important problem in object-centric scene representation, that is, modeling and inferring relationships among objects in a scene. However, both modeling and inference methods have heavy overlaps to existing works and, the main contribution of the paper, inserting a deterministic layer between scene-level random variable $s$ and slot-level random variables $z_{1:K}$ seems unable to block dependences among slots. The proposed method has not been compared to closely relevant methods, and the evaluation metrics for decomposition and reconstruction performance are also not fully investigated. The overall scene generation performance of the proposed method is inferior to GENESISv2.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qKd4-d0YRhs", "reviewer": "Reviewer_T7nj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1- Paper strengths:\n- The proposed method does not depend on the ordering of objects in their slots\n- The proposed object slots distinctly correspond to masks for each object in a scene\n- Thanks to deterministic ordering of the slots, specific object slots correspond to specific semantic concepts (e.g walls, different blocks etc)\n\n\n2- Paper weaknesses\n- The paper needs some polishing in general both for writing and structure. Parts of the paper are written in a casual way and need to be rewritten with a more scientific perspective. Also, some of the sentences are a bit vague and hard to understand \n- The Introduction section is not written well and has vague sentences; parts of it are written more similar to Related Works\n- I agree that inference over all orderings of objects in a slot is hard. However, this is not well-motivated in the paper. In other words, it is not clear to me why the authors have tried to tackle the problem of integrating object orderings in a slot. This is because in practice some prior works seem to propose/fill object slots in an autoregressive fashion and produce good results at the end of the day. I think this could probably be because the models proposed in prior works do not really rely on the order of objects. As one of the experiments, the authors need to show that the results obtained from priors models are sensitive to such orderings\n- Qualitatively, the produced images are not sharp enough compared to some some prior works (MONet, IODINE, some  of the missing references)\n- The authors do not show any experiments for demonstrating minimal generalization of their proposed method for unseen scenes\n- The main comparison of the proposed method with prior work is done quantitatively. This is not sufficient. The authors need to show results for qualitative comparison as well instead of different variations of the proposed method.\n\n3- Additional Comments:\n\nAbstract:\n- The abstract is too wordy and not terse to the point.\n- The abstract is a little bit technical to understand for general readers. It’d be better if the authors do not assume that most future readers are in their field necessarily. Examples of such technical terms: “bag-of-words assumption”, “slots are generated with an autoregressive prior”\n- “known limitations”: vague, relies on authors’ prior knowledge\n\n\nIntroduction:\n- The introduction starts by quickly talking about this specific work. It lacks a good, high-level motivation on why this problem is important. \n- Pragraph2 (P2): “These models have demonstrated strong generalization and object-level disentanglement” → strong and subjective assessment. I would argue no computational model in the machine learning literature has shown strong generalization capabilities as measured against human generalization ability.\n- P2 → Veerapaneni et al. does not seem to be using any of the models cited two sentences before\n- “However, we understand little about how to jointly decompose scenes and sample novel\nscenes from noise with slot-based models” → vague\n\nExperiments:\n- Section 4.1: I do not see an experiment in this section that clearly relate to “EXPLORING THE SCENE-LEVEL LATENT SPACE”. I was expecting some results for latent traversal here, given the section name.\n\nReferences:\n- Many of the references do not have the correct publication venu/information. For instance “Entity abstraction in visual model-based reinforcement learning” is an ICLR 2019 paper but is cited as an arXiv paper. Other examples: “Adam: A method for stochastic optimization”, “Auto-encoding variational bayes”, “Generative neurosymbolic machines.”, “Multi-object representation learning with iterative variational inference” etc\n\n4- Missing References:\n- White, Tom (2016): Sampling Generative Networks. Open Access Victoria University of Wellington | Te Herenga Waka. Journal contribution. https://doi.org/10.26686/wgtn.12585362.v1 \n- Eslami, SM Ali, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman et al. \"Neural scene representation and rendering.\" Science 360, no. 6394 (2018): 1204-1210..\n- Sitzmann, Vincent, Michael Zollhoefer, and Gordon Wetzstein. \"Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations.\" Advances in Neural Information Processing Systems 32 (2019): 1121-1132.\n- Nanbo, Li, Cian Eastwood, and Robert Fisher. \"Learning object-centric representations of multi-object scenes from multiple views.\" Advances in Neural Information Processing Systems 33 (2020).\n- Crawford, Eric, and Joelle Pineau. \"Learning 3D Object-Oriented World Models from Unlabeled Videos.\" Workshop on Object-Oriented Learning at ICML. 2020.\n- Deng, Fei, et al. \"Generative scene graph networks.\" International Conference on Learning Representations. 2020.\n- Deng, Fei, Zhuo Zhi, Sungjin Ahn. Hierarchical Decomposition and Generation of Scenes with Compositional Objects,  Workshop on Object-Oriented Learning at ICML. 2020\n- Chen, Chang, Fei Deng, and Sungjin Ahn. \"Object-Centric Representation and Rendering of 3D Scenes.\" JMLR (2021).", "summary_of_the_paper": "This work proposes a VAE-based hierarchical generative model (named Latent Object Models ) for scenes that contain multiple objects. This is done by modeling the hierarchical relationship between scenes and objects via a set of latent variables for scenes and objects. The object latent variables are a sample from their corresponding object slots. The object slots are inferred deterministically from the scene latent variable.  Having deterministic object slots makes the inference more tractable by removing the need for sequential decomposition of a scene into object slotsIn the experiments, the authors show results for random sampling of the latent variables and some qualitative results for their method and different versions of the proposed method. The authors also show quantitative comparison with some prior works", "summary_of_the_review": "Overall, I believe the paper needs a lot more work in order to be accepted. The authors reasoning for proposing a generative model that does not depend on ordering of is not well motivated and the contributions of this work are not strong. Also, the results of the experiments the authors have performed are far from convincing, specially because they have not properly compared their method to more relevant prior works. Finally, the paper is not written well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bb5svzFu3YF", "reviewer": "Reviewer_tg1E", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strenghts\nThis paper builds on recent methods in unsupervised object-centric segmentation and scene generation which have seen recent success in a number of synthetic image datasets. The authors give a good explanation of the problems in many of these pieces of work and motivate the need for a method that can better handle the orderless structure of the object in these scenes.  In general this paper is well written and the ideas proposed are clear and technically sound.  To my knowledge, while some of the contributions are inspired by similar ideas, the combined approach is novel and can be a useful contribution to the literature. With some exceptions noted below, the most closely related literature is compared either in the discussion and/or empirically. The experimental methodology focuses on relevant metrics, namely sampling quality, and the structure of the learned representations (e.g. structural accuracy and disentanglement), and results show for the most part that the proposed method leads to clear benefits over related work, and a large set of ablations are carried out to establish the usefulness of the different parts of the method.\n\n*Weaknesses\n1) While the key ideas are clear I find the resulting architecture, which relies on having a separately generative model, an odd choice. It makes it a bit more difficult to understand how the two models interact with each other. The new model attempts to address shortcomings of the original model, but depends on the original model to work well as it relies on its approximate posterior -- this could in some cases lead to learning limitations since gradients are not passed through the object latents of the original model. While choosing to stop the gradients is convienent (as otherwise the ordering algorithm's non-differentiability would need to be addressed), the lack of an end-to-end solution might limit the new model's benefits to propagate to learning the original model, or conversely, the limitations of the original model prevents the new model to learn e.g. robust object-centric representations. \n\n2) The idea to deterministically generate an order to ease the task of modelling a set is not new.  For instance, Vinyals et al. show the performance benefits of finding a canonical ordering, and Zhang et al. learn to optimize these permutations. While these works don't apply it to latent set representations, I think the paper should at least discuss how some of these works relate to the proposed idea.\n\n3) I agree with the authors that introducing a hierarchy (e.g. in the form of a global scene latent) in many of these slot-based models is a great idea. However, I would like to see more evidence of the benefits of having the prior *deterministically* produce the order of slots, as opposed to having an autoregressive slot prior. Conceptually, the burden of reasoning about the correlations between objects (in all its permutations) has now been shifted the the global prior since p(z_{1:k}|s) are now independent. This might still work for some types of scenes but I'm not sure how that would scale to scenes with larger and more complex compositions of objects. Perhaps the authors can provide some examples of what they expect the global latent to represent.\n\n4) It's unclear what architecture is used for the likelihoood model. Is it using/sharing the same parameters as EMORL? This should be clearer in Sec. 3. \n\n5) Wrt. the main methods compared, GENESIS and GENv2, I did not follow the explanation as to why these methods can handle richer background and textures (leading e.g. to the lower FID in Objects Room) compared to LOM. I would also like to see the resulting metrics using EMORL alone, as that would clearly show the gains of incorporating the ordered prior.\n\n6) I don't think the error bars in Table 2 are described. It would be also interesting to see the ShapeStacks S-Acc performance across the different runs.\n\nSome other minor comments:\n\n- The LOM-u and LOM-s model performance was hard to find. The labelling in the text and the table in the appendix should be consistent.\n\n- In the appendix, it is stated the Transformers have constant time dependency on the sequence length unlike RNN. While this is true if one can fully parallelize the Transformer's SA, they in general have quadratic complexity on the sequence length vs RNN's linear complexity.\n\n**Post-rebuttal update**\nWhile I believe there are interesting ideas in this paper, quantitative results in the paper are still mixed, with LOM-based methods mostly clearly showing its usefulness (vs e.g. GENv2-MoG) on one dataset and one metric (S-Acc). For this reason, I think the paper would benefit from having experiments using carefully designed datasets that more directly tackles the problem of modelling orderless elements using the proposed method. I would like to see the benefits on this problem compared to other ways of modelling sets. In its current form, reliance on an pretrained models complicates understanding the extent of the benefits and limitations of the method. On the point of limitations, I don't think the authors resolved my concern regarding relying on a scene-level Gaussian to capture all the structure between objects.\nI appreciate the efforts of the authors to incorporate additional results and overall improving the paper, but for the reasons above  I will keep the current score. \n\nReferences.\n\nVinyals et al: Order Matters: Sequence to sequence for sets. ICLR 2016\n\nZhang et al. Learning Representations of Sets through Optimized Permutations. ICLR  2019\n\n\n", "summary_of_the_paper": "This work tackles the problem of learning object-centric generative models of scenes, and in particular it addresses the challenge of handling the orderless nature of objects (represented by latent variables) when learning the model's prior. Most recent models in the literature either assume an arbitrary sequential scene generation (which can hinder the model's performance), or attempt to model the orderless set of object latent variables with an autoregressive prior which can cause discontinuities during training when the allocations of object change.\nThe proposed method relies on incorporating (a) a scene-level prior that deterministically generates a per-object distribution ordering and (b) an ordering algorithm to match object posterior and prior distributions. By learning to generate an object order from a scene latent variable, it effectively sidesteps the problem of modelling the large equivalent class of sets. Quantitative and qualitative results show that incorporating these two ideas allows learning better generative models that can sample scenes with plausible composition of objects.", "summary_of_the_review": "The overall ideas proposed are novel and a potentially useful contribution to the space of image VAEs. The model proposed is shown to work well on well-known multi-object datasets, and a large set of ablation studie disentangle the contributions of the different parts of their method. I'd like to see more discussions about the conceptual limitations of their ideas, as well as references to related work in finding canonical orders for set-based tasks. The authors should also address the clarity issues in the text and figures mentioned above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "F6C6XFT7Dzu", "reviewer": "Reviewer_XQMo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - The paper addresses an important problem, namely how object-centric models can formulate a prior distribution over set-structured latents, while avoiding issues with autoregressive solutions.\n - The proposed model is novel to my knowledge. It follows the existing framework of probabilistic, object-centric modelling, but differs from existing systems in important ways.\n - The paper is generally well written.\n - The construction of the model is sensible given the design goal of eliminating the autoregressive prior from GENESIS-type models.\n\nWeaknesses:\n - The paper lacks a clear theoretical argument as to why the autoregressive prior used in previous systems is a problem. Training autoregressive models while implicitly integrating out the factorization order has been demonstrated to work well at large scale (see e.g. the work deriving from orderless NADE, such as XLNet). It is not a priori clear to me that the solution explored here should necessarily be more effective. It might be better to introduce this issue as an open question, rather than to claim that one solution is clearly better than the other.\n - The quantitative results are mixed. LOM does not outperform previous models in terms of FID score on any of the three datasets. The main positive result is the improved structural accuracy on ShapeStacks, which is a somewhat soft metric due to its manual evaluation.\n - I am unconvinced by the qualitative results as currently presented. The interpolations in Fig. 4 do not strike me as particularly smooth, and contain inconsistent configurations with overlapping objects. Even the top ranked examples in Fig 5 b,c do not share very many commonalities with the given scene. In any case, qualitatively inspecting a single sample scene with no baseline provides too weak of a signal to support the arguments made in the text.", "summary_of_the_paper": "The paper presents LOM, a new structured latent variable model for object-centric probabilistic modelling of scenes. It differs from previous models of this type, such as GENESIS, by avoiding the use of an autoregressive prior over slots/objects. In prior work, the use of such priors has meant that either inference needed to introduce an order among slots (e.g., by also occuring autoregressively), or the order of slots had to be implicitly integrated out during training. It is argued that either option is suboptimal. The proposed model instead models slot assignments in the prior deterministically given a scene-level variable. Inference commences largely in an orderless manner; the inferred slots are matched with their counterparts in the prior only after the fact for training.\nThe model is evaluated experimentally on a suite of synthetic multi-object image datasets, and compared to prior work such as GENESIS.\n", "summary_of_the_review": "The paper proposes an interesting solution to the issue of formulating priors over set-structured latent variables. However, in its current state, it does not offer clear enough evidence that this solution is actually beneficial compared to previous approaches. In particular, its (potentially) beneficial properties should be more clearly demonstrated using quantitative metrics.\n\nPost rebuttal: I have increased my score to a 6, as the authors have added additional results supporting the argument that LOMs improve the unconditional sample quality of the underlying VAE.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a VAE-based hierarchical generative model (Latent Object Model) to model scenes with multiple objects. 
The paper would benefit from a substantial revision to improve text quality and clarity.
The experiments lack proper quantitative baselines and imputations; and the overall results are quite underwhelming relative to existing models.
Decision: Reject