Year: 2021

Abstract:
We introduce a structured latent variable model that learns the underlying datagenerating process for a dataset of scenes. Our goals are to obtain a compositional scene representation and to perform scene generation by modeling statistical relationships between scenes as well as between objects within a scene. We take inspiration from visual topic models and introduce an interpretable hierarchy of scene-level and object-level latent variables (i.e., slots). To generate scenes, dependencies between objects must be modeled. While we could achieve this by assuming slots are generated autoregressively, algorithms for estimating autoregressive slot posteriors impose an unnatural order on objects in a scene and are known to be limited in visually complex environments. We eliminate the need for autoregressive posterior inference by assuming that the assignment of objects to slots during generation is a deterministic function of the scene latent variable. We propose an inference algorithm that indirectly estimates an ordered slot posterior by leveraging an orderless slot posterior and the deterministically-predicted object generation order. Qualitative and quantitative analysis establishes that our approach successfully learns a smoothly traversable scene-level latent space and that the hierarchy of scene and slot variables improves the ability of slot-based models to generate scenes with complex multi-object relations.

text: We introduce a structured latent variable model that learns the underlying datagenerating process for a dataset of scenes. Our goals are to obtain a compositional scene representation and to perform scene generation by modeling statistical relationships between scenes as well as between objects within a scene. We take inspiration from visual topic models and introduce an interpretable hierarchy of scene-level and object-level latent variables (i.e., slots). To generate scenes, dependencies between objects must be modeled. While we could achieve this by assuming slots are generated autoregressively, algorithms for estimating autoregressive slot posteriors impose an unnatural order on objects in a scene and are known to be limited in visually complex environments. We eliminate the need for autoregressive posterior inference by assuming that the assignment of objects to slots during generation is a deterministic function of the scene latent variable. We propose an inference algorithm that indirectly estimates an ordered slot posterior by leveraging an orderless slot posterior and the deterministically-predicted object generation order. Qualitative and quantitative analysis establishes that our approach successfully learns a smoothly traversable scene-level latent space and that the hierarchy of scene and slot variables improves the ability of slot-based models to generate scenes with complex multi-object relations.

heading: 1 INTRODUCTION
text: This paper considers the problem of learning the data-generating process for multi-object scenes. We want to train likelihood-based deep generative models that can encode scenes into a representation hierarchy of scene and object latent variables and that support novel scene generation. A scene-level representation can capture statistical relationships between scenes and can model complex interactions between objects (Jiang & Ahn, 2020). Object-level representations increase compositionality, interpretability, and robustness of the model to out-of-distribution data (Dittadi et al., 2021). Much progress has been made on models that encode scenes into sets of object-centric distributed representations, or slots (Greff et al., 2017; Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2019; Locatello et al., 2020; Nanbo et al., 2020; Emami et al., 2021; Engelcke et al., 2021). These models have demonstrated promising generalization and object-level disentanglement and have been applied successfully to object-centric visual model-based reinforcement learning (Veerapaneni et al., 2019).
However, training slot-based models to jointly decompose and generate novel scenes remains challenging. The limited prior work on this problem has approached it by either ignoring dependencies between slots (Greff et al., 2019; Emami et al., 2021) or by factorizing the joint distribution over slots using the chain rule into a sequence of autoregressive distributions (Engelcke et al., 2019; 2021). None of these models possess a scene-level representation and are therefore limited in their ability to capture scene-level statistics and higher-order object relations (Jiang & Ahn, 2020). Also, although autoregressive generation is capable of modeling dependencies between objects, autoregressive inference is extremely difficult (Engelcke et al., 2021) and remains an open problem.
In this work, we introduce Latent Object Models (LOMs), a deep generative modeling framework for fitting the data-generating process for scenes with a latent variable hierarchy consisting of a scene-level variable and a set of slot variables. The analogy used to design the hierarchical graphical model is inspired by visual topic models (Hofmann, 1999; Blei et al., 2003; Sivic et al., 2005; Fritz &
Figure 1: The hierarchical graphical model for LOMs. We assume that scenes x are generated from K object-centric slot latent variables z1:K . The slots are independent conditional on a causal sequence of deterministic variables (shown in boxes). This causal sequence is predicted from the top-level scene latent variable s. By conditioning each slot on one of the deterministic variables, we provide the order-dependent relational information necessary for scene generation.
Schiele, 2008; Tuytelaars et al., 2010):
documents− scenes topics− slots words− percepts
where a percept is any discrete token that makes up a scene observation, such as a pixel or a 3D point. LOMs have a three-level hierarchy. Each percept is modeled as a K-component mixture, where each component is dependent on one of K intermediate-level slots. Then, the slot variables are dependent on a single top-level scene variable. Although topic models make a “bag-of-words” assumption that topics are orderless within a document, multi-object scene generation requires modeling dependencies between slots. LOMs therefore cannot assume slots are orderless in a scene.
Instead of modeling slot generation with an autoregressive (i.e., stochastic) prior, which requires inferring an autoregressive posterior, we propose to assign objects to slots for ordered scene generation deterministically. We train a function to take a sample from the scene prior and predict a causal sequence of deterministic variables (Figure 1). The slot priors are designed to be independent of each other when conditioned on this sequence of deterministic variables, which eliminates the need to estimate an autoregressive posterior. We instead propose an inference algorithm for LOMs that assumes an auxiliary generative model which decomposes scenes into an orderless set of slots. This is used to infer a slot-order-invariant scene-level posterior from which we obtain an ordered slot prior. To obtain an ordered slot posterior for maximum likelihood training, we compute a matching between the means of the orderless posterior and the ordered prior. This matching is used to permute the orderless posterior and convert it into an ordered distribution.
Contributions: We introduce a latent variable model with a latent scene-slot hierarchy and variational inference algorithm for modeling the data-generating process of multi-object scenes. We demonstrate that this framework is capable of learning a low-dimensional scene-level manifold that can be smoothly traversed. We also show that generated scenes accurately capture higher-order object relations present in the dataset.

heading: 2 RELATED WORK
text: We compare LOMs with related deep generative models in Table 1. Likelihood-based deep latent variable models based on the variational autoencoder (VAE) framework (Kingma & Welling, 2013)
have recently demonstrated impressive scene generation capabilities for high-resolution images with spatial latent variables split across a hierarchy of stochastic layers (Sønderby et al., 2016; Kingma et al., 2016; Child, 2020; Vahdat & Kautz, 2020). Differently, LOMs model perceptual data using scene and slot latent variables. During inference and generation, each slot is bound to a single entity in the scene which improves interpretability and compositionality (Greff et al., 2020).
Generative Neurosymbolic Machines (GNMs) (Jiang & Ahn, 2020) present the idea of combining a scene-level prior and symbolic latent variables within a hierarchical VAE for multi-object scene generation. GSGN (Deng et al., 2021) explores learning part-whole hierarchies for VAEs with similar symbolic latent variables. However, these VAEs make a strong assumption that the symbolic object-centric latent variables are spatial variables, i.e., bound to each cell of a fixed grid (Eslami et al., 2016; Crawford & Pineau, 2019; Lin et al., 2020). While this works well for synthetic image datasets with similarly sized objects, LOMs tackle the more challenging problem of learning a scene-slot hierarchy. LOMs fall into the category of VAEs that represent scenes as mixture models whose components are generated by a set of slots (Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2019; 2021; Locatello et al., 2020; Nanbo et al., 2020; Emami et al., 2021). EfficientMORL (EMORL) is an orderless slot-based VAE with a hierarchical independent slot prior that combines efficient bottom-up stochastic slot attention with top-down iterative refinement. GENESIS (GEN) (Engelcke et al., 2019) and GENESIS-v2 (GENv2) (Engelcke et al., 2021) use an autoregressive slot prior for multi-object scene generation. Unlike LOMs, neither possess a scene-level representation. Furthermore, while theoretically an autoregressive slot prior can learn dependencies between objects necessary for generating structured scenes, learning this distribution is difficult. GEN uses sequential attention to estimate an autoregressive posterior that imposes an unnatural order on objects, biases slot gradients, and does not work well on visually complex scenes (Engelcke et al., 2021). GENv2 replaces GEN’s posterior with an orderless slot posterior which does not suffer from these limitations. However, orderless posteriors assume slots are independent which means GENv2’s autoregressive prior will struggle to learn object dependencies.
Other VAEs with structured latent variables have been proposed for hierarchical scene generation (Anciukevicius et al., 2020; von Kügelgen et al., 2020) but make simplifying independence assumptions in their graphical models that prevents them from scaling to more complex scenes. The SetVAE (Kim et al., 2021) learns to generate set-structured data such as point clouds with a tree-structured hierarchy of sets of latent variables. However, unlike slot-based models, its latent variables are not explicitly trained to segregate the scene representation.
Generative adversarial methods for compositional scene synthesis (van Steenkiste et al., 2020; Nguyen-Phuoc et al., 2020; Liao et al., 2020; Ehrhardt et al., 2020; Niemeyer & Geiger, 2021; Hudson & Zitnick, 2021) avoid the computational challenges of Bayesian inference that LOMs address but lack the ability to decompose scenes into an interpretable hierarchy of representations.

heading: 3 LATENT OBJECT MODELS

heading: 3.1 GENERATIVE MODEL
text: We restrict the focus of this paper to unlabeled datasets D of i.i.d. samples of images x ∈ RN×C , N = HW , for clarity. Our goal is to fit the underlying data-generating process, which we propose to approach by using a three-level latent hierarchy. To that end, we define a scene-level latent variable s ∈ RDs , K object-level latent variables z1, z2, . . . , zK , zk ∈ RDz (or slots, which we abbreviate with z1:K), and pixel observations xi ∈ RC , i = 1, . . . , N . The joint distribution over observed and latent variables can be factorized as follows:
pθ(x, z1:K , s) = p(s)pθ(z1:K | s)pθ(x | z1:K). (1)
We adopt the VAE framework for approximate Bayesian inference (Kingma & Welling, 2013; Rezende et al., 2014) since posterior inference is intractable and since we would like to parameterize these distributions with deep neural networks.
The scene-level prior p(s) is left as a standard Gaussian in this work for simplicity. However, the slot prior is more complicated, since generating coherent scenes requires modeling relational information between slots. Unlike topic models, we cannot assume slots are i.i.d. when conditioned on s. We
instead assume a fixed and deterministic order on the slots informed by s:
pθ(z1:K | s) = K∏ k=1 pθ(zk | hk), where hk = fθ(h1:k−1, s), (2)
where fθ is a deterministic function that transforms a sample s from the scene prior and h1 = fθ(s). Each hk contains the order-aware information about which object the kth slot should generate. By designing the slot prior such that the kth slot indirectly depends on slots z1:k−1 through hk, the joint distribution over slots factorizes as a product of Gaussian conditionally independent (but not identically distributed) distributions. Notice that if we replaced each hk with s the LOM generative model becomes a bag-of-words model.
The choice of image-likelihood strongly influences the complexity of the learned scene representation. We use a slot order-invariant Gaussian likelihood with the mean given by a sum of the product of the image-shaped decoded masks mk and components xk:
pθ(x | z1:K) = N∏ i=1 N ( K∑ k=1 mi,kxi,k, σ 2 ) . (3)
The Gaussian likelihood is well-suited for generating complex scenes since it does not penalize solutions that use multiple slots to fit parts of the background (Locatello et al., 2020; Emami et al., 2021). By contrast, the Mixture-of-Gaussians likelihood used by some slot-based models (Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2019; 2021) heavily penalizes the model for splitting parts of the scene across multiple slots. A scene can be sampled from a LOM as follows:
1. Sample s from the scene prior p(s).
2. Compute h1 = fθ(s),h2 = fθ(h1, s), . . . ,hK = fθ(h1:K−1, s).
3. Sample one slot from each of the pθ(z1 | h1), pθ(z2 | h2), . . . , pθ(zK | hk). 4. Render the image by decoding the K slots in parallel into masks mk and pixel components xk and then aggregating them with Equation 3.

heading: 3.2 INFERENCE
text: We have introduced how LOMs model the scene generation process. Now, we turn our attention to the inference algorithm for the slot and scene latent variables. We want to avoid directly estimating an ordered slot posterior using sequential scene decomposition since an optimal object ordering is unknown and difficult to learn without supervision. Our inference algorithm, visualized in Figure 2, uses an auxiliary generative model that assumes slots are orderless to indirectly obtain an ordered slot posterior. Orderless scene decomposition uses randomized iterative assignment to bind percepts to the slot variables (Greff et al., 2019; Locatello et al., 2020; Emami et al., 2021; Engelcke et al., 2021). The main idea behind our inference algorithm is to impose an ordering on the auxiliary orderless posterior with object ordering information provided by the deterministic variables h1:K . The algorithm shares similarities with recent work that proposes to learn a good canonical ordering for orderless sets to ease certain modeling tasks (Vinyals et al., 2016; Zhang et al., 2019). An important consequence of having an orderless slot posterior is that we can define a slot-order-invariant scene-level posterior:
qφ(s | gφ(o1:K)), (4)
where gφ is a permutation-invariant function. Orderless slots are denoted by o1:K . Each step of LOM inference is summarized here and visualized in Figure 2:
1. Estimate orderless slot posterior: Compute and sample from the orderless posterior qω estimated by the auxiliary model (with inference network parameters ω): o1:K ∼ qω(o1:K | x) = ∏K k=1 qω(ok | x). 2. Estimate scene posterior: Compute and sample from the scene-level posterior: s ∼ qφ(s | gφ(o1:K)).
3. Estimate scene-conditioned ordered slot prior: Compute the ordered prior pθ(z1:K | s) using a single sample s and the causal function fθ.
4. Align the orderless slot posterior: Compute a matching π between the K means from pθ(z1:K | s) and the K means from qω(o1:K | x), e.g., by minimizing the pairwise L2 distance, then permute the order of the slot posterior means by π to get oπ(1):π(K).
5. Estimate an ordered slot posterior: Compute new order-aware scale parameters σzπ(k) for the permuted slot posteriors to obtain ordered slot posteriors qφ(zπ(k) | h′π(k),x).
Here σzπ(k) = wφ(h ′ π(k)), with h ′ π(k) = f ′ φ(oπ(1):π(k−1), s) and wφ is a linear transformation. We introduce another causal function f ′φ to transform a sample from the scene posterior and the permuted orderless slots oπ(1):π(K) into the sequence h′π(1):π(K). A simple and robust choice for computing π is an O(K) greedy alignment (Algorithm 1 in the appendix) that sequentially finds the best match for each slot. Each h′π(k) provides the ordered slot posterior with its order-aware scale parameter:
K∏ k=1 qφ(zπ(k) | h′π(k),x) = K∏ k=1 N (µoπ(k), (σ z π(k)) 2I). (5)
Here, µoπ(k) is just the kth mean of the permuted orderless slot posterior.
Architectures for fθ,f ′φ and gφ: In this work we use a Transformer (Vaswani et al., 2017) with multiple blocks of single-head causal self-attention (SA) to implement fθ. The first token in the input is a sample s ∼ p(s) and the next K tokens are a shared trainable embedding initialized with zeros. We implement f ′φ using nearly the same causal Transformer architecture as fθ. The differences are that the first token is now a sample s from the scene posterior and instead of using a trainable embedding as input for the other tokens we pass the K permuted orderless slots oπ(1):π(K). For gφ we use a permutation-invariant Transformer with multiple blocks of residual SA and no positional encoding or causal masking. We aggregate the set output with a global sum pooling layer, which we find essential for capturing complex object relations in scenes. Details are provided in the appendix.

heading: 3.3 TRAINING
text: LOMs can be seen as a hybrid model that jointly trains an orderless VAE and an ordered VAE. Gradients are prevented from flowing back to the orderless VAE from the ordered VAE during training (via stop_grad(o1:K)) to ensure that orderless scene decomposition remains unbiased. To estimate qω(o1:K | x), we can use any slot-based VAE with orderless decomposition. In this work, we train LOMs with both EMORL and GENv2. For simplicity, we re-use their respective decoder architectures to implement the LOM image likelihood.
Objective: The ELBO for training LOMs is maximized jointly with the ELBO for the auxiliary orderless generative model. The LOM ELBO resembles the ELBO used by previous deep generative models with a similar topic-model-inspired hierarchical factorization (Edwards & Storkey, 2016). We show it here presented as a sum of three terms and defer its derivation to the appendix.
First is a log-likelihood (i.e., reconstruction) term: LLL = Eqφ(s|gφ(o1:K)) [ Eqφ(zπ(1):π(K)|h′π(1):π(K),x) [log pθ(x | z1:K)] ] . (6)
Then, we have a reverse Kullback-Leibler (KL) term for the slots, which can be factorized as a sum over K since the slot posterior and prior are conditionally independent and aligned via π:
LslotKL = −Eqφ(s|gφ(o1:K)) [ K∑ k=1 DKL ( qφ(zπ(k) | h′π(k),x) ‖ pθ(zk | hk) )] . (7)
Finally, we have a scene-level reverse KL term: LsceneKL = −DKL ( qφ(s | gφ(o1:K)) ‖ p(s) ) . (8)
Altogether, we maximize L, the sum of these three terms and the auxiliary orderless model’s ELBO:
L = Lorderless + LLL + LslotKL + LsceneKL. (9)
Impact of the ELBO on fθ: Intuitively, maximizing the ELBO encourages fθ to generate objects in a consistent order that results in a low average value for LslotKL. In practice, fθ often learns to generate parts (or all) of the background with the same slot(s) and generate foreground objects using the same subset of slots (Figure 3).
Optimization: To dynamically balance the reconstruction and KL terms for all models we use GECO (Rezende & Viola, 2018). GECO reformulates maximization of the ELBO into a constrained minimization of the KL terms, where the constraint dictates a desired reconstruction quality C. Please refer to the appendix for a discussion on choosing C as it is crucial for obtaining good performance. All other implementation details including architecture description, hyperparameters, and compute requirements are also deferred to the appendix.

heading: 4 EXPERIMENTS
text: Datasets: We use three synthetic multi-object datasets: Objects Room and CLEVR6 from the standard Multi-Object Dataset (Kabra et al., 2019) and ShapeStacks (Groth et al., 2018). All three datasets consist of rendered images of 3D scenes containing 3+ objects and have variable illumination, shadow, and camera perspectives. Successful object-centric scene generation requires reasoning about local object relations (e.g., occlusion in CLEVR6, placement of objects relative to the floor in Objects Room) as well as global relations (e.g., stacking multiple blocks in ShapeStacks). We focus on synthetic scenes in this work since we wish to first establish correctness of the proposed LOM framework before tackling real-world scenes.
Baselines: We compare LOM-EMORL and LOM-GENv2-G against the state-of-the-art slot-based VAE for scene generation GENv2, as well as EMORL, GNM, GENESIS (GEN), and the NVAE. EMORL has an independent slot prior that does not capture dependencies between objects; therefore, it serves as a simple scene generation baseline. We train GENv2 with the default Mixture-ofGaussians image likelihood (GENv2-MoG) on CLEVR6 as these results were not reported in their paper. Results for GEN and GENv2-MoG on the other two datasets are provided by Engelcke et al. (2021). We train GENv2 with the Gaussian likelihood on all three datasets (GENv2-G) for a more direct comparison to the LOMs. See Section 2 for a detailed discussion on these models and the appendix for training details and hyperparameters.
Metrics: We use multiple metrics to analyze the performance of all models in terms of objectcentric decomposition and disentanglement, image generation quality, and the structural accuracy of generated scenes. The Frechet Inception Distance (FID) (Heusel et al., 2017) is a standard metric for quantifying the ability of a model to generate images similar to the training set. We follow Engelcke et al. (2021) and use 10K real and generated samples to calculate this score. However, we found that the FID score heavily penalizes blurriness in generated images while ignoring whether the semantics of the generated scenes are accurate. Therefore, inspired by the evaluation of GNM (Jiang & Ahn, 2020), on ShapeStacks we generate 100 images with each model and count the fraction of images containing a stack where all blocks in the stack are touching each other (structural accuracy, or S-Acc). To compare the ability to jointly generate high-quality scenes and learn object-centric representations, we compute the adjusted rand index for segmented foreground objects (ARI-FG) (Rand, 1971; Hubert & Arabie, 1985) on all datasets and disentanglement scores (DCI) (Eastwood & Williams, 2018) on CLEVR6. The NVAE is only trained on CLEVR6 since it is the only considered dataset that provides ground-truth latent factors.
We ablate attributes of LOMs on a 48x48 resolution variant of Objects Room but defer these results to the appendix (Section A.5). Specifically, we measure the importance of: the expressivity of fθ,f ′θ, and gφ, computing order-aware scale parameters during LOM inference, and not sharing parameters between fθ and f ′θ. We also consider two variants with autoregressive slot priors where we try to (unsuccessfully) train them by directly minimizing the KL with respect to an orderless slot posterior.

heading: 4.1 QUALITATIVE EVALUATION
text: We first verify that LOMs are able to fit the data-generating process for each dataset by inspecting unconditional samples from the prior (Figures 3a-3d). Note that we slightly scale down the standard deviations in the prior using a temperature of τ = 0.8 when sampling, which is common practice for hierarchical VAEs (Vahdat & Kautz, 2020) to help keep samples in high probability regions. See
Figure 7 in the appendix for a temperature sensitivity analysis. Upon close inspection of the Objects Room scene masks (Figures 3d and 3j), we see that LOMs can consistently generate the wall, floor, and shapes with the same groups of slots (as indicated by mask color). By learning a consistent and deterministic order with which to generate objects, LOMs are able to learn important dependencies between objects needed for generating scenes.
We compare Objects Room and ShapeStacks samples from slot-based VAEs in Figures 3e-3j, which are the two datasets with the most complex multi-object structure. As expected, EMORL generates incoherent scenes with floating blocks due to its independent prior. GEN generates seemingly incoherent scenes from the visually challenging ShapeStacks dataset. In theory, its autoregressive posterior should be capable of learning ordered dependencies between slots; however, GEN inference tends to heavily rely on color cues, which fails in datasets with textures and shadows. Samples from GENv2-G’s autoregressive prior contain visual artifacts and structural inaccuracies (e.g., extra walls, missing floors in Objects Room, floating blocks in ShapeStacks). Its orderless posterior cannot capture object dependencies, which makes it difficult for the autoregressive prior to learn to sequentially generative scenes from the challenging Objects Room and ShapeStacks datasets. By contrast, we see that LOMs use their ability to learn a deterministic generation order and their scene-level representation to generate scenes that more accurately reflect the ground truth data.
It is also helpful to inspect the LOM scene-level latent space by interpolating between two random samples from the scene prior using spherical linear interpolation (Figure 4). The interpolated images resemble the training data which suggests that the LOM has learned a good scene-level manifold. Scene-level attributes such as camera perspective and floor color change smoothly and the number of objects roughly remains at two. More interpolation examples are provided in the appendix (Figures 10, 11, 12) as well as qualitative comparisons (Figures 13-15). Figures 16-20 visualize reconstructions and segmentations for all models. We also explore how LOMs can hierarchically cluster a scene dataset using scene-level and object-level relationships in the appendix (Section A.6).

heading: 4.2 QUANTITATIVE EVALUATION
text: We present the results in Table 2. GNM has difficulty discovering and segmenting individual objects in Objects Room and ShapeStacks which leads to much lower ARI-FG scores compared to the slot-based models (see, e.g., Figures 14 and 19). Despite this, GNM achieves low FID scores on all three environments. Its inability to decompose scenes in two out of three environments suggests that the slot-based approaches are more promising for likelihood-based multi-object scene generation.
Out of the slot-based models, LOM-GENv2-G achieves the best FID score and S-Acc on ShapeStacks. LOM-EMORL achieves the best FID score on CLEVR6. To quantify the improvement in generation quality gained by the LOM framework, we can compare FID and S-Acc scores for EMORL with LOM-EMORL and GENv2-G with LOM-GENv2-G. Removing sum pooling in the scene encoder gφ decreases LOM-EMORL’s S-Acc by a 36%, highlighting the role it plays in encoding relations
involving many objects. In almost all settings, we observe clear improvements. LOM-GENv2-G on CLEVR6 did not improve the FID compared to GENv2-G despite there being a noticeable qualitative improvement (Figure 15). We believe this is because GENv2’s encoder introduces visual artifacts that artificially inflates the FID score, but this requires further investigation.
GENv2-MoG achieves lower FID scores than GENv2-G in Objects Room. We observed that GENv2MoG segments the walls, floor, and sky into a single slot (Figure 13), whereas GENv2-G tends to segment the wall and floor separately (Figure 3i). GENv2’s orderless posterior’s inability to learn slot dependencies means that GENv2-G has the more difficult task of orienting the objects, walls, and floor during generation whereas GENv2-MoG simplifies this task, leading to GENv2-G’s worse FID scores. By contrast, LOM-GENv2-G successfully learns dependencies between objects and clearly improves GENv2-G’s Objects Room FID score (Figure 3j). We believe that GEN achieves a lower FID score than GENv2-G in Objects Room because GEN’s autoregressive posterior, which is heavily reliant on color cues, is able to successfully model slot dependencies this dataset. GEN clearly has difficulty with the more challenging ShapeStacks dataset and achieves much higher FID scores. In general, LOMs maintain the decomposition performance (as measured by ARI-FG) as the auxiliary orderless VAE. Finally, Table 3 shows that although the NVAE achieves low FID on CLEVR6, its poor disentanglement scores suggests that its spatial latents do not encode interpretable object-centric information about scenes.

heading: 5 CONCLUSION
text: We presented a probabilistic framework for learning an interpretable hierarchy of scene and slot latent variables. The learned representation shares advantages of both a single scene-level representation as well as object-centric slots. This work improves our understanding on how to develop object-centric VAEs, particularly with respect to accurately modeling scene-level statistics.
We believe the interpretation of the deterministic variables in the LOM graphical model could benefit from further study. They are essential to the success of LOMs but trade off the ability to model stochasticity in the order in which objects are generated. We observed that LOMs occasionally generate objects that intersect each other due to a lack of 3D understanding in the latent representations. Although unsupervised 3D scene inference is particularly difficult due to ambiguities introduced by perspective geometry, obtaining 3D object representations should improve generation quality (Chen et al., 2020). Our LOM implementation could also be improved by replacing the standard Gaussian scene prior with a more flexible distribution such as a mixture-of-Gaussians.

heading: A APPENDIX

heading: A.1 NEURAL ARCHITECTURES
text: We describe the details of the neural architectures used in this paper to implement LOMs. We use GELU non-linear activations (Hendrycks & Gimpel, 2016) and xavier weight initialization (Glorot & Bengio, 2010).
Residual self-attention block: We construct the permutation-invariant scene encoder and the causal slot prior/posterior by stacking residual self-attention (SA) blocks (Figure 5). Each SA block is based on the Transformer (Vaswani et al., 2017) design. For each such block, we apply single-head scaled dot-product attention to an input x ∈ RN×D:
α = softmax( k(x)q(x)ᵀ√
D )v(x) (10)
x = x+ MLP(LayerNorm(x+ α)). (11)
The k, q, v linear maps are D × Dz projections and the MLP is a Dz × Dz projection. In our experiments we set Dz to 64.
Scene encoder gφ: The scene encoder gφ (Figure 5a) has 3 residual SA blocks. It takes as input an orderless set of slots o1:K where each slot is a Dz-dim vector. We use the means of the orderless slot posterior for the input in practice. We append a Dz-dim trainable scene embedding os at position
Algorithm 1 Greedy matching algorithm for π. Uses O(K) time and O(K2) space.
1: Input: Ordered means µz1, . . . , µzK , orderless means µo1 , . . . , µoK , with µk ∈ RDz . 2: C[i, j] = ‖µzi − µoj ‖2, ∀(i, j) 3: π = [ ] 4: for index i = 1 . . .K do 5: j∗ = argminC[i, :] 6: C[:, j∗] = + inf 7: π[i] = j∗ 8: end for 9: return Permutation π
K + 1 so that the input to gφ is a set of size K + 1. After the 3 residual SA blocks we apply a sum pooling layer that aggregates the output set by summing the K+1 vectors and projecting the result to the scene latent dimension Ds with an MLP. Then, two linear layers are used to obtain the mean and softplus-activated standard deviation for the Gaussian scene posterior. Our ablation study analyzes the number of residual SA blocks and the impact of the sum pooling layer on the ability to capture higher-order object relations (Figure 5b).
Causal Transformer fθ: A straightforward architectural choice for implementing fθ is a recurrent neural network (RNN) such as a GRU (Cho et al., 2014), which have a linear time dependency on the input sequence length. By contrast, Transformers have a quadratic dependency on the input sequence length; however, this computation can be efficiently parallelized, resulting in a constant time dependency in practice. Moreover, they have a superior ability to capture long-range interactions between elements of the sequence due to the use of self-attention (Radford et al., 2019; Brown et al., 2020; Zhou et al., 2021). Therefore we use a Transformer with single-head causal self-attention (SA) to implement fθ. In detail, the first token in the input sequence is the sample s ∼ p(s) and the subsequent K tokens in the sequence are set to a shared trainable embedding of dimension Dz initialized with zeros. The scene sample is projected to the same dimension as the Dz-dimensional embeddings with a single layer MLP. To provide extra positional information we add a positional encoding to the input sequence. This is implemented by mapping the values [0/(K + 1), 1/(K + 1), . . . , (K + 1)/(K + 1)] to Dz-dimensional vectors with a linear layer and adding this to the input. We use the standard approach for implementing causal self-attention by masking out attention values in each of the residual SA blocks. The length K output sequence, starting from the second output token, are the h1:K . These are each mapped to Gaussian parameters for each pθ(zk | hk) with two linear layers. We train fθ end-to-end alongside the inference and generation networks to maximize the ELBO, which we discussed in Section 3.3.
Causal Transformer f ′θ: For the causal Transformer that predicts the order-dependent scale parameters for the ordered slot posterior, we use the same architecture as fθ except for a few minor differences. Primarily, the inputs are now a sample from the scene posterior and the means of the permuted orderless slot posterior. The outputs of the stack of SA blocks are mapped with a single linear layer to the Dz-dim softplus-activated standard deviations of the ordered slot posterior. The means of the ordered slot posterior are fixed to be the means of the permuted orderless posterior.

heading: A.2 GREEDY MATCHING ALGORITHM
text: Pseudocode for the greedy matching algorithm we use to align the orderless slot posterior to the ordered slot prior is provided in Algorithm 1. Alternative matching algorithms such as the Hungarian algorithm (Munkres, 1957) can be used instead at a higher computational cost. We did not empirically explore other matching algorithms since the greedy solution worked well initially.

heading: A.3 DERIVATION OF THE LOM ELBO
text: We assume that we are given a sample o1:K from an orderless slot posterior estimated by an auxiliary orderless generative model. We can derive a lower-bound on the log marginal likelihood as follows:
log p(x) = log ∫ pθ(x, s, zπ(1):π(K))ds, dz1:K (12)
= log ∫ qφ(s, zπ(1):π(K) | x) qφ(s, zπ(1):π(K) | x) pθ(x, s, z1:K)ds, dz1:K (13)
= logEq [ pθ(x, s, z1:K)
qφ(s, zπ(1):π(K) | x)
] . (14)
Applying Jensen’s inequality, we get: log p(x) ≥ Eqφ(s,zπ(1):π(K)|x) [ log
pθ(x, s, z1:K)
qφ(s, zπ(1):π(K) | x)
] (15)
= Eqφ(s|gφ(o1:K)) [ Eqφ(zπ(1):π(K)|h′π(k),x) [ log
pθ(x | z1:K)pθ(z1:K | s)p(s) qφ(s | gφ(o1:K))qφ(zπ(1):π(K) | h′π(k),x)
]] .
(16) We can treat the inner log product of terms as a sum of logs to get the following three loss terms. First is a log-likelihood loss term:
LLL = Eqφ(s|gφ(o1:K)) [ Eqφ(zπ(1):π(K)|h′π(k),x) [ log pθ(x | zπ(1):π(K)) ]] . (17)
Second is a slot reverse KL divergence loss:
LKL_z = Eqφ(s|gφ(o1:K)) [ Eqφ(zπ(1):π(K)|h′π(k)x) [ log
pθ(z1:K | s) qφ(zπ(1):π(K) | h′π(k),x)
]] (18)
= −Eqφ(s|gφ(o1:K)) [ DKL ( qφ(zπ(1):π(K) | h′π(k),x) ‖ pθ(z1:K | s) )] . (19)
Since both the ordered slot posterior and the ordered slot prior are products of K conditionally independent distributions, this reverse KL further factorizes as a sum over K:
LKL_z = −Eqφ(s|gφ(o1:K)) [ K∑ k=1 DKL ( qφ(zπ(k) | h′π(k),x) ‖ pθ(zk | hk) )] . (20)
Finally, there is a scene-level reverse KL divergence loss: LKL_s = Eqφ(s|gφ(o1:K)) [ Eqφ(zπ(1):π(K)|h′π(k),x) [ log
p(s)
qφ(s | gφ(o1:K))
]] (21)
= Eqφ(s|gφ(o1:K)) [ log
p(s)
qφ(s | gφ(o1:K))
] (22)
= −Eqφ(s|gφ(o1:K)) [ log qφ(s | gφ(o1:K)) p(s) ] (23)
= −DKL ( qφ(s | gφ(o1:K)) ‖ p(s) ) . (24)
Therefore, we have the following ELBO we wish to maximize with respect to parameters θ, φ: log p(x) ≥ LLL + LKL_z + LKL_s. (25)
In practice, we approximate this ELBO by averaging over a minibatch of dataset samples of size B and use stochastic gradient ascent.

heading: A.4 EXPERIMENT DETAILS
text: An open source implementation of the code for replicating all experiments will be released with link placed here. Videos demonstrating latent space random walks will also be released. We provide the generated samples and our labels used to compute the structure accuracy metric in the supplementary material.

heading: A.4.1 HYPERPARAMETERS
text: Auxiliary orderless models: We trained two LOM variants, one that uses EfficientMORL (Emami et al., 2021) (EMORL) to extract the auxiliary orderless posterior and one which uses GENESIS-v2 (GENv2) (Engelcke et al., 2021). The authors of both models have released open source implementations.12 For EMORL, we keep their architecture and hyperparameters fixed except that the number of iterative refinement steps is set at 2 for all experiments. We use the Gaussian image likelihood with a global standard deviation of 0.7 for the Objects Room and ShapeStacks datasets. For CLEVR6, we needed to lower the global standard deviation to 0.1 to achieve sharp object segmentations. We use the refined approximate EMORL posterior for LOM inference. When using GENv2 to implement a LOM, we disable the autoregressive prior and replace it with an independent Gaussian slot prior. We also used a Gaussian image likelihood instead of the default mixture-of-Gaussians so that both LOM-EMORL and LOM-GENv2-G used the same likelihood. LOM-GENv2-G uses a global standard deviation of 0.7 for the Gaussian likelihood on all three environments. We use the approximate posterior estimated from the output of GENv2’s non-parametric clustering algorithm for LOM inference. We set the slot dimension to 64 for all models. Please see the respective papers and code for EMORL and GENv2 for complete architecture and hyperparameter details.
LOM: We maintain the same slot dimension (Dz = 64) and image likelihoods as the auxiliary orderless models on all environments. The scene latent dim Ds is 128 which is chosen to be twice the slot dimension. We use the same decoder architectures from EMORL and GENv2, respectively, for mapping slots to image-shaped masks and components. The optimization hyperparameters are also kept the same as EMORL and GENv2 for each LOM. For LOM-EMORL, both the auxiliary model and the LOM are trained jointly with the Adam optimizer (Kingma & Ba, 2014) with default PyTorch (Paszke et al., 2019) hyperparameters and a learning rate of 4e-4. EMORL’s learning rate schedule is a linear warmup for 10K steps then multiplicative decay with by a rate of 0.5 every 100K steps. LOM-GENv2-G is trained in a similar manner, again using Adam and a learning rate of 1e-4 but without any learning rate schedule.
Datasets: Following Engelcke et al. (2021) we use K = 7 and K = 9 slots respectively for Objects Room and ShapeStacks. For CLEVR6, we use K = 7 slots following standard practice on this popular multi-object benchmark. LOM-EMORL pre-processes all images by converting RGB values to the range [−1, 1] before passing them as input to the model. Objects Room can be accessed freely online.3 This dataset contains 64x64 RGB training images. The ShapeStacks dataset is also freely available for download online and also has 64x64 RGB images.4. We use the same preprocessing protocol for CLEVR6 as Emami et al. (2021), which is to center crop the images to 192x192 and the resize them to size 96x96. This dataset is available as part of the Multi-Object Dataset.

heading: A.4.2 BALANCING RECONSTRUCTION AND KL
text: GECO: Both EMORL and GENv2 use GECO to balance reconstruction and KL losses (Rezende & Viola, 2018). We use the respective implementations of GECO from each code base for training LOM-EMORL and LOM-GENv2-G.
For LOM-EMORL we customize the GECO target for each dataset, which is dependent on the image resolution and the choice of image likelihood. For Objects Room, we used a per-pixel and per-channel target of -0.352 (for an unnormalized Gaussian log-likelihood). Due to the increased visual complexity of ShapeStacks, we lowered this target to -0.351 to achieve a better trade-off of KL and reconstruction. On CLEVR6 we used a target of -2.265 (for an unnormalized Gaussian log-likelihood with σ = 0.1). The rule of thumb used to tune the GECO target is that the target should be reached after about 15-20% of the training steps. After the target is reached, the GECO Lagrange parameter is automatically decreased to 1 so that a valid ELBO is maximized. We use a GECO learning rate of 1e-6 for updating the Lagrange parameter and multiply this by an acceleration factor once the target is reached. Acceleration factors of 5, 10, and 50 were used on Objects Room, CLEVR6, and ShapeStacks respectively.
1https://github.com/pemami4911/EfficientMORL 2https://github.com/applied-ai-lab/genesis 3https://github.com/deepmind/multi_object_datasets 4https://ogroth.github.io/shapestacks/
As mentioned, LOM-GENv2-G is trained to optimize an unnormalized Gaussian image likelihood like LOM-EMORL. For Objects Room and ShapeStacks we used a per-pixel and per-channel target of -0.353 and increased this to -0.356 for CLEVR6. We found that we needed to decrease the GECO learning rate from 1e-5 to 1e-6 for these datasets compared to the default GECO learning rate used by GENv2. We note that the GENv2 codebase had a better GECO implementation than EMORL that was less sensitive to particular hyperparameters and with more robust annealing schedules. Improving the GECO implementation used by EMORL should help make tuning these hyperparameters easier for replication.
Freebits: We found it beneficial to use a freebits parameter λ on some of the KL terms to help avoid posterior collapse (Maaløe et al., 2019). In detail, for a minibatch of size B the freebits-augmented scene KL is:
DKL ( qφ(s | gφ(o1:K)) ‖ p(s) ) ≈ 1 B B∑ i=1 max ( λ,DKL ( qφ(s | gφ(oi1:K)) ‖ p(s) )) . (26)
This encourages the model to keep the KL at least as large as λ. For slot-based distributions that factorize as products over K, we take the max of λ and the sum over K slot-wise KL terms. We use this for each of EMORL’s hierarchical KL terms as well as the LOM scene KL (but not for the LOM slot KL since the posterior means are fixed via the stopped gradients, meaning that we would like to push this KL as close to zero as possible). We use freebits for the LOM-GENv2-G scene KL as well. We use λ = 0.75 ×Dz for each KL term in EMORL and λ = 0.25 ×Ds for the LOM scene KL. Inspired by Maaløe et al. (2019) we tried annealing λ down slightly to 0.167×Ds for the scene KL term during the last 25K steps of training and found that this slightly increased performance.

heading: A.4.3 BASELINES AND COMPUTE
text: GENESIS and GENESIS-v2: The authors of GENv2 have released pre-trained weights for GEN trained on ShapeStacks and GENv2 trained on Objects Room and ShapeStacks. We use these weights for model visualizations and to compute the structure accuracy metric. To train GENv2 on CLEVR6 with the default mixture-of-Gaussians image likelihood, we had to lower the standard deviation to 0.1 to get it to work decently well. We adjusted the GENv2 GECO target accordingly, tuning it to -2.265 (per-pixel and per-channel value). We arrived at this value by starting with -61000, which is for the unnormalized mixture-of-Gaussians summed over pixels and channels with standard deviation of 0.1 used by Emami et al. (2021)), and then increasing the target slightly to achieve a high-quality reconstruction. We also trained GENv2 on all three environments using the same Gaussian image likelihood as LOM-GENv2-G for a more fair comparison.
GNM: We use the official GNM PyTorch implementation provided by the authors.5 We train GNM on the 128x128 resolution version of CLEVR6 so that we could use the default GNM hyperparameters that the authors used for the 128x128 resolution CLEVR-based environment from their paper. This worked well for CLEVR6. By default, GNM organizes its symbolic variables into a 4x4 spatial grid, which we maintained. We treat each spatial grid cell as one slot for a total of 16 slots. To train GNM on the 64x64 Objects Room and ShapeStacks scenes, we removed one downsampling and one upsampling layer from the encoder and decoder to account for the resolution being one-half that of CLEVR6. For all environments, we used the default global latent dim of 32 and zwhat dim of 64.
NVAE: The NVAE is trained with the open source implementation provided by the authors.6 We changed the image likelihood from the discrete mixture of logistic distribution to the Gaussian likelihood with standard deviation of 0.7 for a fair comparison with other considered models. We also modified the default NVAE architecture for 64x64 images for use with CLEVR6 as follows. To decrease memory consumption, we remove the normalizing flows and use 16 initial channels in the encoder and decoder towers. The NVAE has three scales of spatial latents, and we extract the top-level at the lowest resolution of 12x12. Since each spatial latent group has 20 channels, this becomes a 2880-dim vector when flattened. We follow the same protocol as (Greff et al., 2019; Dittadi et al., 2021) to compute disentanglement scores for NVAE by imposing a fixed order on objects in the scene based on their attributes.
5https://github.com/JindongJiang/GNM 6https://github.com/NVlabs/NVAE
Compute: We use a batch size of 32 for LOM-EMORL and LOM-GENv2-G. For LOM-EMORL, we split this across 8 NVIDIA A100 GPUs and train for 400K steps on Objects Room and ShapeStacks, which takes approximately 27 wall-clock hours each run. We train LOM-EMORL for 300K steps on CLEVR6 which takes 36 hours. On Objects Room and ShapeStacks, training the GENv2 baseline with 4 GPUs and a batch size of 32 takes about 20 hours to reach 500K steps (following the authors’s training protocol). We train GENv2 for 300K steps on CLEVR6, which takes 24 hours. LOMGENv2-G takes approximately a similar amount of time. GNM takes about 9 hours to reach 500K steps with a batch size of 32 on one A100 GPU for Objects Room and ShapeStacks. It takes about 12 hours to reach 500K steps on CLEVR6 on one A100 GPU. The NVAE converges quickly on CLEVR6 because we are able to use a large batch size of 256 split across 8 GPUs. We found it takes about 10-14 hours to converge in this setup depending on random seed.

heading: A.5 ABLATION AND VARIATION STUDIES
text: Setup: We train each ablation or model variant of LOM-EMORL, defined below, across five random seeds on a 48x48 resolution version of Objects Room.
Model 0: The baseline LOM-EMORL model, which uses 2 residual SA blocks in both the scene encoder gφ and the causal Transformers fθ, f ′θ.
Model 1: The number of residual SA blocks in gφ is decreased to 1. Model 2: The number of residual SA blocks in gφ is increased to 3. Model 3: The number of residual SA blocks in fθ, f ′θ is decreased to 1. Model 4: The number of residual SA blocks in fθ, f ′θ is increased to 3. Model 5: During LOM inference, we skip the step of computing new scale parameters for the
aligned orderless slot posterior. Instead, we re-use the scale parameters estimated by the auxiliary orderless model.
Model 6: We share parameters between fθ and f ′θ.
Model 7: We consider an alternative approach to scene generation by means of an autoregressive prior. We minimize the reverse KL divergence between EMORL’s orderless slot posterior and an RNN-based autoregressive slot prior, stopping gradients from passing back through the posterior. This KL is added as an auxiliary loss term to EMORL’s ELBO. Note that the scene-level latent is removed in this variant and the multi-step LOM inference algorithm is not used since we are simply directly minimizing the KL between an orderless posterior and the autoregressive prior.
Model 8: We extend Model 7 by adding the scene-level latent. We condition the first slot of the autoregressive slot prior on the scene-level latent s. The scene-level reverse KL term is also added to EMORL’s ELBO as an auxiliary loss.
Results: See Figure 6. We found that increasing the expressivity of the scene encoder gφ by using 3 SA blocks (Model 2) instead of 2 (Model 0) slightly increased FID, whereas decreasing the number of SA blocks to 1 in gφ (Model 1) and in the causal Transformers fθ, f ′θ (Model 3) lead to a pronounced decline in performance. Increasing the number of SA blocks to 3 for fθ, f ′θ saw negligible improvement (Model 4). We observed that not computing new order-aware scale parameters for the LOM ordered approximate slot-level posterior (Model 5) resulted in unfavorably high test KL and FID scores. This verifies that computing the order-aware scale parameters for the LOM approximate ordered posterior is important for learning the LOM prior. Next, we found that sharing parameters between fθ and f ′θ severely degraded the FID (Model 6).
The autoregressive prior variants (Model 7 and Model 8) achieve the worst FID scores and KL across all variants. However, adding the scene latent to the autoregressive prior (Model 8) improves the FID score relative to Model 7. We visualize samples from Model 8 in Figure 6c. The orderless EMORL slot posterior (i.e., slots are independent of each other) does not provide sufficient slot dependency information to train the autoregressive prior to sequentially generate objects in a coherent way. The generated scenes appear to be mostly incoherent and have multiple walls, missing floors, and floating objects. We observed similar phenomena with GENv2, which attempts to fit an autoregressive slot prior with an orderless approximate slot posterior.
Sampling temperature: We plot how the Objects Room FID varies with respect to the temperature τ ∈ {0.6, 0.7, 0.8, 0.9, 1.0} in Figure 7. Decreasing the temperature to 0.8 helps samples stay within regions of high probability when sampling from the hierarchical prior. We observe a small corresponding increase in FID.

heading: A.6 HIERARCHICAL CLUSTERING EXAMPLES
text: We provide illustrative examples of how the hierarchical representation inferred by LOMs hierarchically clusters the dataset via a scene retrieval application. For a given query scene, the basic idea is to use the scene-level latent variable to first retrieve nearest-neighbor scenes based on scene-level similarities. Then, the retrieval results can be re-ranked by selecting a single object’s slot from the
query and matching it against all slots of the retrieved scenes. We would expect that after re-ranking, the most similar scene to the query would also contain an object that matches the selected query object in one or more object-level attributes.
We created a database of 50K scenes stored as (s, z) pairs of vectors from the Objects Room training set. For the Euclidean distance between latent vectors to return a meaningful value, we found it necessary to first use PCA to project the 128-dim scene latents down to 6-dim vectors. We ran PCA on the database of 50K scene latents and inspected the explained variances of the principle components to select the projection dimension of 6. We select all queries from held-out samples not included in the initial 50K used to create the database. We visualize examples in Figure 8 and Figure 9.

heading: A.7 ADDITIONAL QUALITATIVE RESULTS
text: Additional examples of spherical linear interpolation (SLERP) between two random samples from the LOM scene prior are shown in Figures 10, 11, 12. To create these images, we normalize each sample s and then multiply by √ Ds before computing the intermediate latents.
in pu t
re co ns
se gm en ta tio n
in pu t
re co ns
se gm en ta tio n
in pu t
re co ns
se gm en ta tio n
in pu t
re co ns
se gm en ta tio n
in pu t
re co ns
se gm en ta tio n
in pu t
re co ns
se gm en ta tio n
in pu t
re co ns

Review summaries:
This work proposes a VAE-based hierarchical generative model (named Latent Object Models ) for scenes that contain multiple objects. This is done by modeling the hierarchical relationship between scenes and objects via a set of latent variables for scenes and objects. The object latent variables are a sample from their corresponding object slots. The object slots are inferred deterministically from the scene latent variable.  Having deterministic object slots makes the inference more tractable by removing the need for sequential decomposition of a scene into object slotsIn the experiments, the authors show results for random sampling of the latent variables and some qualitative results for their method and different versions of the proposed method. The authors also show quantitative comparison with some prior works

The paper presents LOM, a new structured latent variable model for object-centric probabilistic modelling of scenes. It differs from previous models of this type, such as GENESIS, by avoiding the use of an autoregressive prior over slots/objects. In prior work, the use of such priors has meant that either inference needed to introduce an order among slots (e.g., by also occuring autoregressively), or the order of slots had to be implicitly integrated out during training. It is argued that either option is suboptimal. The proposed model instead models slot assignments in the prior deterministically given a scene-level variable. Inference commences largely in an orderless manner; the inferred slots are matched with their counterparts in the prior only after the fact for training. The model is evaluated experimentally on a suite of synthetic multi-object image datasets, and compared to prior work such as GENESIS. 

This work tackles the problem of learning object-centric generative models of scenes, and in particular it addresses the challenge of handling the orderless nature of objects (represented by latent variables) when learning the model's prior. Most recent models in the literature either assume an arbitrary sequential scene generation (which can hinder the model's performance), or attempt to model the orderless set of object latent variables with an autoregressive prior which can cause discontinuities during training when the allocations of object change. The proposed method relies on incorporating (a) a scene-level prior that deterministically generates a per-object distribution ordering and (b) an ordering algorithm to match object posterior and prior distributions. By learning to generate an object order from a scene latent variable, it effectively sidesteps the problem of modelling the large equivalent class of sets. Quantitative and qualitative results show that incorporating these two ideas allows learning better generative models that can sample scenes with plausible composition of objects.

This paper proposes a structured latent variable model, called Latent Object Models (LOM), for compositional scene generation. To generate a scene image, LOM first samples a scene-level latent variable, and then samples multiple slot-level latent variables based on a sequence of deterministic variables modeled by an autoregressive model. To infer the ordered slot-level latent variables, LOM first estimates orderless slot posteriors, and then determines the permutation of orderless slot posteriors based on a greedy matching algorithm. After learning, the model is able to generate scene images with reasonable relationships of objects. Experiments are conducted on three datasets, and the proposed method is compared with several existing methods.