Generating Scenes with Latent Object Models

We introduce a structured latent variable model that learns the underlying datagenerating process for a dataset of scenes . Our goals are to obtain a compositional scene representation and to perform scene generation by modeling statistical relationships between scenes as well as between objects within a scene . We take inspiration from visual topic models and introduce an interpretable hierarchy of scene-level and object-level latent variables ( i.e. , slots ) . To generate scenes , dependencies between objects must be modeled . While we could achieve this by assuming slots are generated autoregressively , algorithms for estimating autoregressive slot posteriors impose an unnatural order on objects in a scene and are known to be limited in visually complex environments . We eliminate the need for autoregressive posterior inference by assuming that the assignment of objects to slots during generation is a deterministic function of the scene latent variable . We propose an inference algorithm that indirectly estimates an ordered slot posterior by leveraging an orderless slot posterior and the deterministically-predicted object generation order . Qualitative and quantitative analysis establishes that our approach successfully learns a smoothly traversable scene-level latent space and that the hierarchy of scene and slot variables improves the ability of slot-based models to generate scenes with complex multi-object relations . 1 INTRODUCTION . This paper considers the problem of learning the data-generating process for multi-object scenes . We want to train likelihood-based deep generative models that can encode scenes into a representation hierarchy of scene and object latent variables and that support novel scene generation . A scene-level representation can capture statistical relationships between scenes and can model complex interactions between objects ( Jiang & Ahn , 2020 ) . Object-level representations increase compositionality , interpretability , and robustness of the model to out-of-distribution data ( Dittadi et al. , 2021 ) . Much progress has been made on models that encode scenes into sets of object-centric distributed representations , or slots ( Greff et al. , 2017 ; Burgess et al. , 2019 ; Greff et al. , 2019 ; Engelcke et al. , 2019 ; Locatello et al. , 2020 ; Nanbo et al. , 2020 ; Emami et al. , 2021 ; Engelcke et al. , 2021 ) . These models have demonstrated promising generalization and object-level disentanglement and have been applied successfully to object-centric visual model-based reinforcement learning ( Veerapaneni et al. , 2019 ) . However , training slot-based models to jointly decompose and generate novel scenes remains challenging . The limited prior work on this problem has approached it by either ignoring dependencies between slots ( Greff et al. , 2019 ; Emami et al. , 2021 ) or by factorizing the joint distribution over slots using the chain rule into a sequence of autoregressive distributions ( Engelcke et al. , 2019 ; 2021 ) . None of these models possess a scene-level representation and are therefore limited in their ability to capture scene-level statistics and higher-order object relations ( Jiang & Ahn , 2020 ) . Also , although autoregressive generation is capable of modeling dependencies between objects , autoregressive inference is extremely difficult ( Engelcke et al. , 2021 ) and remains an open problem . In this work , we introduce Latent Object Models ( LOMs ) , a deep generative modeling framework for fitting the data-generating process for scenes with a latent variable hierarchy consisting of a scene-level variable and a set of slot variables . The analogy used to design the hierarchical graphical model is inspired by visual topic models ( Hofmann , 1999 ; Blei et al. , 2003 ; Sivic et al. , 2005 ; Fritz & Figure 1 : The hierarchical graphical model for LOMs . We assume that scenes x are generated from K object-centric slot latent variables z1 : K . The slots are independent conditional on a causal sequence of deterministic variables ( shown in boxes ) . This causal sequence is predicted from the top-level scene latent variable s. By conditioning each slot on one of the deterministic variables , we provide the order-dependent relational information necessary for scene generation . Schiele , 2008 ; Tuytelaars et al. , 2010 ) : documents− scenes topics− slots words− percepts where a percept is any discrete token that makes up a scene observation , such as a pixel or a 3D point . LOMs have a three-level hierarchy . Each percept is modeled as a K-component mixture , where each component is dependent on one of K intermediate-level slots . Then , the slot variables are dependent on a single top-level scene variable . Although topic models make a “ bag-of-words ” assumption that topics are orderless within a document , multi-object scene generation requires modeling dependencies between slots . LOMs therefore can not assume slots are orderless in a scene . Instead of modeling slot generation with an autoregressive ( i.e. , stochastic ) prior , which requires inferring an autoregressive posterior , we propose to assign objects to slots for ordered scene generation deterministically . We train a function to take a sample from the scene prior and predict a causal sequence of deterministic variables ( Figure 1 ) . The slot priors are designed to be independent of each other when conditioned on this sequence of deterministic variables , which eliminates the need to estimate an autoregressive posterior . We instead propose an inference algorithm for LOMs that assumes an auxiliary generative model which decomposes scenes into an orderless set of slots . This is used to infer a slot-order-invariant scene-level posterior from which we obtain an ordered slot prior . To obtain an ordered slot posterior for maximum likelihood training , we compute a matching between the means of the orderless posterior and the ordered prior . This matching is used to permute the orderless posterior and convert it into an ordered distribution . Contributions : We introduce a latent variable model with a latent scene-slot hierarchy and variational inference algorithm for modeling the data-generating process of multi-object scenes . We demonstrate that this framework is capable of learning a low-dimensional scene-level manifold that can be smoothly traversed . We also show that generated scenes accurately capture higher-order object relations present in the dataset . 2 RELATED WORK . We compare LOMs with related deep generative models in Table 1 . Likelihood-based deep latent variable models based on the variational autoencoder ( VAE ) framework ( Kingma & Welling , 2013 ) have recently demonstrated impressive scene generation capabilities for high-resolution images with spatial latent variables split across a hierarchy of stochastic layers ( Sønderby et al. , 2016 ; Kingma et al. , 2016 ; Child , 2020 ; Vahdat & Kautz , 2020 ) . Differently , LOMs model perceptual data using scene and slot latent variables . During inference and generation , each slot is bound to a single entity in the scene which improves interpretability and compositionality ( Greff et al. , 2020 ) . Generative Neurosymbolic Machines ( GNMs ) ( Jiang & Ahn , 2020 ) present the idea of combining a scene-level prior and symbolic latent variables within a hierarchical VAE for multi-object scene generation . GSGN ( Deng et al. , 2021 ) explores learning part-whole hierarchies for VAEs with similar symbolic latent variables . However , these VAEs make a strong assumption that the symbolic object-centric latent variables are spatial variables , i.e. , bound to each cell of a fixed grid ( Eslami et al. , 2016 ; Crawford & Pineau , 2019 ; Lin et al. , 2020 ) . While this works well for synthetic image datasets with similarly sized objects , LOMs tackle the more challenging problem of learning a scene-slot hierarchy . LOMs fall into the category of VAEs that represent scenes as mixture models whose components are generated by a set of slots ( Burgess et al. , 2019 ; Greff et al. , 2019 ; Engelcke et al. , 2019 ; 2021 ; Locatello et al. , 2020 ; Nanbo et al. , 2020 ; Emami et al. , 2021 ) . EfficientMORL ( EMORL ) is an orderless slot-based VAE with a hierarchical independent slot prior that combines efficient bottom-up stochastic slot attention with top-down iterative refinement . GENESIS ( GEN ) ( Engelcke et al. , 2019 ) and GENESIS-v2 ( GENv2 ) ( Engelcke et al. , 2021 ) use an autoregressive slot prior for multi-object scene generation . Unlike LOMs , neither possess a scene-level representation . Furthermore , while theoretically an autoregressive slot prior can learn dependencies between objects necessary for generating structured scenes , learning this distribution is difficult . GEN uses sequential attention to estimate an autoregressive posterior that imposes an unnatural order on objects , biases slot gradients , and does not work well on visually complex scenes ( Engelcke et al. , 2021 ) . GENv2 replaces GEN ’ s posterior with an orderless slot posterior which does not suffer from these limitations . However , orderless posteriors assume slots are independent which means GENv2 ’ s autoregressive prior will struggle to learn object dependencies . Other VAEs with structured latent variables have been proposed for hierarchical scene generation ( Anciukevicius et al. , 2020 ; von Kügelgen et al. , 2020 ) but make simplifying independence assumptions in their graphical models that prevents them from scaling to more complex scenes . The SetVAE ( Kim et al. , 2021 ) learns to generate set-structured data such as point clouds with a tree-structured hierarchy of sets of latent variables . However , unlike slot-based models , its latent variables are not explicitly trained to segregate the scene representation . Generative adversarial methods for compositional scene synthesis ( van Steenkiste et al. , 2020 ; Nguyen-Phuoc et al. , 2020 ; Liao et al. , 2020 ; Ehrhardt et al. , 2020 ; Niemeyer & Geiger , 2021 ; Hudson & Zitnick , 2021 ) avoid the computational challenges of Bayesian inference that LOMs address but lack the ability to decompose scenes into an interpretable hierarchy of representations . 3 LATENT OBJECT MODELS . 3.1 GENERATIVE MODEL . We restrict the focus of this paper to unlabeled datasets D of i.i.d . samples of images x ∈ RN×C , N = HW , for clarity . Our goal is to fit the underlying data-generating process , which we propose to approach by using a three-level latent hierarchy . To that end , we define a scene-level latent variable s ∈ RDs , K object-level latent variables z1 , z2 , . . . , zK , zk ∈ RDz ( or slots , which we abbreviate with z1 : K ) , and pixel observations xi ∈ RC , i = 1 , . . . , N . The joint distribution over observed and latent variables can be factorized as follows : pθ ( x , z1 : K , s ) = p ( s ) pθ ( z1 : K | s ) pθ ( x | z1 : K ) . ( 1 ) We adopt the VAE framework for approximate Bayesian inference ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) since posterior inference is intractable and since we would like to parameterize these distributions with deep neural networks . The scene-level prior p ( s ) is left as a standard Gaussian in this work for simplicity . However , the slot prior is more complicated , since generating coherent scenes requires modeling relational information between slots . Unlike topic models , we can not assume slots are i.i.d . when conditioned on s. We instead assume a fixed and deterministic order on the slots informed by s : pθ ( z1 : K | s ) = K∏ k=1 pθ ( zk | hk ) , where hk = fθ ( h1 : k−1 , s ) , ( 2 ) where fθ is a deterministic function that transforms a sample s from the scene prior and h1 = fθ ( s ) . Each hk contains the order-aware information about which object the kth slot should generate . By designing the slot prior such that the kth slot indirectly depends on slots z1 : k−1 through hk , the joint distribution over slots factorizes as a product of Gaussian conditionally independent ( but not identically distributed ) distributions . Notice that if we replaced each hk with s the LOM generative model becomes a bag-of-words model . The choice of image-likelihood strongly influences the complexity of the learned scene representation . We use a slot order-invariant Gaussian likelihood with the mean given by a sum of the product of the image-shaped decoded masks mk and components xk : pθ ( x | z1 : K ) = N∏ i=1 N ( K∑ k=1 mi , kxi , k , σ 2 ) . ( 3 ) The Gaussian likelihood is well-suited for generating complex scenes since it does not penalize solutions that use multiple slots to fit parts of the background ( Locatello et al. , 2020 ; Emami et al. , 2021 ) . By contrast , the Mixture-of-Gaussians likelihood used by some slot-based models ( Burgess et al. , 2019 ; Greff et al. , 2019 ; Engelcke et al. , 2019 ; 2021 ) heavily penalizes the model for splitting parts of the scene across multiple slots . A scene can be sampled from a LOM as follows : 1 . Sample s from the scene prior p ( s ) . 2 . Compute h1 = fθ ( s ) , h2 = fθ ( h1 , s ) , . . . , hK = fθ ( h1 : K−1 , s ) . 3 . Sample one slot from each of the pθ ( z1 | h1 ) , pθ ( z2 | h2 ) , . . . , pθ ( zK | hk ) . 4 . Render the image by decoding the K slots in parallel into masks mk and pixel components xk and then aggregating them with Equation 3 .

Summary: This paper proposes a structured latent variable model, called Latent Object Models (LOM), for compositional scene generation. To generate a scene image, LOM first samples a scene-level latent variable, and then samples multiple slot-level latent variables based on a sequence of deterministic variables modeled by an autoregressive model. To infer the ordered slot-level latent variables, LOM first estimates orderless slot posteriors, and then determines the permutation of orderless slot posteriors based on a greedy matching algorithm. After learning, the model is able to generate scene images with reasonable relationships of objects. Experiments are conducted on three datasets, and the proposed method is compared with several existing methods.