SCALOR: Generative World Models with Scalable Object Representations

1 INTRODUCTION . Unsupervised structured representation learning for visual scenes is a key challenge in machine learning . When a scene is properly decomposed into meaningful entities such as foreground objects and background , we can benefit from numerous advantages of abstract symbolic representation . These include interpretability , sample efficiency , the ability of reasoning and causal inference , as well as compositionality and transferability for better generalization . In addition to symbols , another essential dimension is time . Objects , agents , and spaces all operate under the governance of time . Without accounting for temporal developments , it is often much harder if not impossible to discover certain relationships in a scene . Among a few methods that have been proposed for unsupervised learning of object-oriented representation in temporal scenes , SQAIR ( Kosiorek et al. , 2018 ) is by far the most complete model . As a probabilistic temporal generative model , it can learn object-wise structured representation while modeling underlying stochastic temporal transitions in the observed data . Introducing the propagation-discovery model , SQAIR can also handle dynamic scenes where objects may disappear or be introduced in the middle of a sequence . Although SQAIR provides promising ideas and shows the potential of this important direction , a few key challenges remain , limiting its applicability merely to synthetic toy tasks that are far simpler than typical natural scenes . The first and foremost limitation is scalability . Sequentially processing every object in an image , SQAIR has a fundamental limitation in scaling up to scenes with a large number of objects . As such , the state-of-the-art remains at the level of modeling videos containing only a few objects , such as MNIST digits , per image . Considering the complexity of typical natural scenes as well as the importance of scalable unsupervised object perception for applications such as self-driving systems , it is thus a challenge of the highest priority to scale robustly to scenes with a large number of objects . Scaling up the object-attention capacity is an important problem because it allows us to maximize the modern parallel computation that can maximize search capacity . This is contrary to humans , who can attend only to a few objects at a time in a time-consuming sequential manner . The †Authors with equal contribution . Email addresses : { jindong.jiang , sepehr.janghorbani , gerard.demelo , sungjin.ahn } @ rutgers.edu AlphaGo system ( Silver et al. , 2017 ) is an example demonstrating the power of such parallel search ( attention ) beyond human capacity . The second limitation is that previous models including SQAIR lack any form of background modeling and thus only cope with scenes without background , whereas natural scenes usually have a dynamic background . Thus , a temporal generative model that can deal with dynamic backgrounds along with many foreground objects is an important step toward natural video scene understanding . In this paper , we propose a model called SCALable Sequential Object-Oriented Representation ( SCALOR ) . SCALOR resolves the aforementioned key limitations and hence can model complex videos with several tens of moving objects along with dynamic backgrounds , eventually making the model applicable to natural videos . In SCALOR , we achieve scalability with respect to the object density by parallelizing both the propagation and discovery processes , reducing the time complexity of processing each image from O ( N ) to O ( 1 ) , with N being the number of objects in an image . We also observe that the sequential object processing in SQAIR , which is based on an RNN , not only increases the computation time but also deteriorates discovery performance . To this end , we propose a parallel discovery model with superior discovery capacity and performance . SCALOR can also be regarded as a generative tracking model since it not only detects object trajectories but is also able to predict trajectories into the future . In our experiments , we demonstrate that SCALOR can model videos with nearly one hundred moving objects along with a dynamic background on synthetic datasets . Furthermore , we showcase the ability of SCALOR to operate on natural-scene videos containing tens of objects with a dynamic background . The contributions of this work are : ( i ) We propose the SCALOR model , which significantly improves ( two orders of magnitude ) the scalability in terms of object density . It is applicable to nearly a hundred objects while providing more efficient computation time than SQAIR . ( ii ) We propose parallelizing the propagation–discovery process by introducing the propose–reject model , reducing the time complexity to O ( 1 ) . ( iii ) SCALOR can model scenes with a dynamic background . ( iv ) SCALOR is the first probabilistic model demonstrating its working on a significantly more complex task , i.e. , natural scenes containing tens of objects as well as background . 2 PRELIMINARIES : SEQUENTIAL ATTEND INFER REPEAT ( SQAIR ) . SQAIR models a sequence of images x = x1 : T by assuming that observation xt at time t is generated from a set of object latent variables zOt = { zt , n } n∈Ot withOt the set of objects present at time t. Latent variable zt , n corresponding to object n consists of three factors ( z pres t , n , z where t , n , z what t , n ) , which represent the existence , pose , and appearance of the object , respectively . SQAIR also assumes that an object can disappear or be introduced in the middle of a sequence . To model this , it introduces the propagation–discovery model . In propagation , a subset of currently existing objects is propagated to the next time-step and those not propagated ( e.g. , moved out of the scene ) are deleted . In discovery , after deciding how many objects Dt will be discovered , Dt objects are newly introduced into the scene . Combining the propagated set Pt and discovered set Dt , we obtain the set of currently existing objects Ot . The overall process can be formalized as : p ( x1 : T , z fg 1 : T , D1 : T ) = p ( D1 , z D 1 ) T∏ t=2 p ( xt|zfgt ) p ( Dt , zDt |zPt ) p ( zPt |zfgt−1 ) . ( 1 ) For SQAIR , we use zfgt ( “ fg ” standing for foreground ) to denote z O t because SQAIR does not have any latent variables for background . Due to the intractable posterior , SQAIR is trained through variational inference with the following posterior approximation : q ( D1 : T , z fg 1 : T |x1 : T ) = T∏ t=1 q ( Dt , z D t |xt , zPt ) ∏ n∈Ot−1 q ( zt , n|zt−1 , n , x≤t ) . ( 2 ) SQAIR is trained using the importance-weighted autoencoder ( IWAE ) objective ( Burda et al. , 2015 ) . The VIMCO estimator ( Mnih & Rezende , 2016 ) is used to backpropagate through the discrete random variables while using the reparameterization trick ( Kingma & Welling , 2013 ; Williams , 1992 ) for continuous variables . SQAIR has two main limitations in terms of scalability . First , for propagation , SQAIR relies on an RNN , which sequentially processes each object by conditioning on previously processed objects . Second , the discovery is also sequential because it uses RNN-based discovery based on AIR ( Eslami et al. , 2016 ) . Consequently , SQAIR has a time complexity of O ( |Ot| ) per step t. In Crawford & Pineau ( 2019b ) , the authors demonstrated that this sequential discovery can easily fail beyond the scale of a few objects . Moreover , SQAIR lacks any model for the background and its temporal transitions , which is important in modeling natural scenes . 3 SCALOR . In this section , we describe the proposed model , SCALOR . We first describe the generative process along with the proposal-rejection mechanism , which is designed to prevent propagation collapse , and then the inference process and learning . 3.1 GENERATIVE PROCESS . SCALOR assumes that an image xt is generated by background latent z bg t and foreground latent zfgt . The foreground is further factorized into a set of object representations z fg t = { zt , n } n∈Ot . In SCALOR , we represent an object by zt , n = ( z pres t , n , z where t , n , z what t , n ) similarly to SQAIR . The appearance representation zwhatt , n is a continuous vector representation , and z where t , n is further decomposed into the center position zpost , n , scale z scale t , n , and depth z depth t , n . The depth representation , which is missing in SQAIR , represents the relative depth between objects from the camera viewpoint . This depth modeling helps deal with object occlusion . The foreground mask mt , n obtained from zwhatt , n is used to distinguish background and foreground . We adopt the propagation–discovery model from SQAIR , but improve it in such a way to resolve the scalability problem . The generative process of SCALOR can be written as : p ( x1 : T , z1 : T ) = p ( z D 1 ) ( z bg 1 ) T∏ t=2 p ( xt|zt ) ︸ ︷︷ ︸ rendering p ( zbgt |zbg < t , zfgt ) ︸ ︷︷ ︸ background transition p ( zDt |zPt ) ︸ ︷︷ ︸ discovery p ( zPt |z < t ) ︸ ︷︷ ︸ propagation , ( 3 ) where zt = ( z bg t , z fg t ) . As shown , the generation process is decomposed into four modules : ( i ) propagation , ( ii ) discovery , ( iii ) background transition , and ( iv ) rendering . Propagation . The propagation in SCALOR is modeled as follows : p ( zPt |z < t ) = ∏ n∈Ot p ( zprest , n |z < t , n ) { p ( zwheret , n |z < t , n ) p ( zwhatt , n |z < t , n ) } zprest , n , ( 4 ) where p ( zprest , n |z < t , n ) is a Bernoulli distribution with parameter βt , n . The distributions of “ what ” and “ where ” are defined only when the object is propagated . To implement this , for each object n we assign an object-tracker RNN denoted by its hidden state ht , n . The RNN is updated by input zt , n for all t where the object n is present in the scene . The parameter βt , n is obtained as βt , n = fmlp ( ht , n ) . If zprest , n = 0 , the object n is not propagated and the tracker RNN is deleted . Importantly , unlike the RNN-based sequential propagation in SQAIR , the propagation in SCALOR is fully parallel . Discovery by Proposal-Rejection . The main contribution in making our model scalable with respect to the the number of objects is our new discovery model that consists of two phases : proposal and rejection . In the proposal phase , we assume that the target image can be covered by H ×W latent grid cells , and we propose an object latent variable z̃t , h , w per grid cell . This proposal phase can be written as : p ( z̃Dt |zPt ) = HW∏ h , w=1 p ( z̃Dt , h , w|zPt ) = HW∏ h , w=1 p ( z̃prest , h , w|zPt ) { p ( z̃wheret , h , w|zPt ) p ( z̃whatt , h , w|zPt ) } z̃prest , h , w . ( 5 ) In the rejection phase , our goal is to reject some of the proposed objects if a proposed object largely overlaps with a propagated object . We realize this by using the mask variable mt , n . Specifically , if the overlapping area between the mask of a proposed object and that of a propagated object is over a threshold τ , we reject the proposed object . This procedure can be described as ( i ) proposal : z̃Dt ∼ p ( z̃Dt |zPt ) and ( ii ) accept-reject : zDt = faccept-reject ( z̃Dt , zPt , τ ) . In this way , the final discovery set zDt is always a subset of the proposal set z̃ D t , i.e. , z D t ⊆ z̃Dt . Although we use a deterministic function for the rejection , it can be a design choice to implement this as a stochastic decision . While one rationale behind this design is to reflect an inductive bias of a Gestalt principle saying that two objects can not coexist in the same position , we shall also see later further reasons as to why this design is effective . The final discovery model can then be written as : p ( zDt |zPt ) = p ( z̃Dt |zPt ) ∏HW h , w=1 p ( z D t , h , w|zPt , z̃Dt , τ ) , where the acceptance model is : p ( zDt , h , w|zPt , z̃Dt , τ ) = f ( zacceptt , h , w|zPt , z̃Dt , τ ) p ( z̃wheret , h , w ) z accept t , h , wp ( z̃whatt , h , w ) zacceptt , h , w . ( 6 ) Background Transition . Unlike SQAIR , SCALOR is endowed with a background model . The background image is encoded into a D-dimension continuous vector zbgt from the background transition p ( zbgt |zbg < t , zfgt ) . The background RNN encodes the temporal transition of background images . Rendering . The implementation of the rendering process is the same as in SPAIR ( Crawford & Pineau , 2019b ) , except that we process the objects in parallel . Implementation details are shown in Appendix A .

Summary: This paper proposes a generative model for scalable sequential object-oriented representation. The paper proposes several improvements based on the method SQAIR (Kosiorek et al. 2018b), (1) modeling the background and foreground dynamics separately; (2) parallelizing the propagation-discovery process by introducing the propose-reject model which reducing the time complexity. Finally, the proposed model can deal with orders of magnitude more objects than previous methods, and can model more complex scenes with complex background.