SCALOR: GENERATIVE WORLD MODELS
Year: 2020

Abstract:
Scalability in terms of object density in a scene is a primary challenge in unsupervised sequential object-oriented representation learning. Most of the previous models have been shown to work only on scenes with a few objects. In this paper, we propose SCALOR, a probabilistic generative world model for learning SCALable Object-oriented Representation of a video. With the proposed spatiallyparallel attention and proposal-rejection mechanisms, SCALOR can deal with orders of magnitude larger numbers of objects compared to the previous stateof-the-art models. Additionally, we introduce a background module that allows SCALOR to model complex dynamic backgrounds as well as many foreground objects in the scene. We demonstrate that SCALOR can deal with crowded scenes containing up to a hundred objects while jointly modeling complex dynamic backgrounds. Importantly, SCALOR is the first unsupervised object representation model shown to work for natural scenes containing several tens of moving objects. https://sites.google.com/view/scalor/home

heading: 1 INTRODUCTION
text: Unsupervised structured representation learning for visual scenes is a key challenge in machine learning. When a scene is properly decomposed into meaningful entities such as foreground objects and background, we can benefit from numerous advantages of abstract symbolic representation. These include interpretability, sample efficiency, the ability of reasoning and causal inference, as well as compositionality and transferability for better generalization. In addition to symbols, another essential dimension is time. Objects, agents, and spaces all operate under the governance of time. Without accounting for temporal developments, it is often much harder if not impossible to discover certain relationships in a scene.
Among a few methods that have been proposed for unsupervised learning of object-oriented representation in temporal scenes, SQAIR (Kosiorek et al., 2018) is by far the most complete model. As a probabilistic temporal generative model, it can learn object-wise structured representation while modeling underlying stochastic temporal transitions in the observed data. Introducing the propagation-discovery model, SQAIR can also handle dynamic scenes where objects may disappear or be introduced in the middle of a sequence. Although SQAIR provides promising ideas and shows the potential of this important direction, a few key challenges remain, limiting its applicability merely to synthetic toy tasks that are far simpler than typical natural scenes.
The first and foremost limitation is scalability. Sequentially processing every object in an image, SQAIR has a fundamental limitation in scaling up to scenes with a large number of objects. As such, the state-of-the-art remains at the level of modeling videos containing only a few objects, such as MNIST digits, per image. Considering the complexity of typical natural scenes as well as the importance of scalable unsupervised object perception for applications such as self-driving systems, it is thus a challenge of the highest priority to scale robustly to scenes with a large number of objects. Scaling up the object-attention capacity is an important problem because it allows us to maximize the modern parallel computation that can maximize search capacity. This is contrary to humans, who can attend only to a few objects at a time in a time-consuming sequential manner. The
†Authors with equal contribution. Email addresses: {jindong.jiang, sepehr.janghorbani, gerard.demelo, sungjin.ahn}@rutgers.edu
AlphaGo system (Silver et al., 2017) is an example demonstrating the power of such parallel search (attention) beyond human capacity.
The second limitation is that previous models including SQAIR lack any form of background modeling and thus only cope with scenes without background, whereas natural scenes usually have a dynamic background. Thus, a temporal generative model that can deal with dynamic backgrounds along with many foreground objects is an important step toward natural video scene understanding.
In this paper, we propose a model called SCALable Sequential Object-Oriented Representation (SCALOR). SCALOR resolves the aforementioned key limitations and hence can model complex videos with several tens of moving objects along with dynamic backgrounds, eventually making the model applicable to natural videos. In SCALOR, we achieve scalability with respect to the object density by parallelizing both the propagation and discovery processes, reducing the time complexity of processing each image from O(N) to O(1), with N being the number of objects in an image. We also observe that the sequential object processing in SQAIR, which is based on an RNN, not only increases the computation time but also deteriorates discovery performance. To this end, we propose a parallel discovery model with superior discovery capacity and performance. SCALOR can also be regarded as a generative tracking model since it not only detects object trajectories but is also able to predict trajectories into the future. In our experiments, we demonstrate that SCALOR can model videos with nearly one hundred moving objects along with a dynamic background on synthetic datasets. Furthermore, we showcase the ability of SCALOR to operate on natural-scene videos containing tens of objects with a dynamic background.
The contributions of this work are: (i) We propose the SCALOR model, which significantly improves (two orders of magnitude) the scalability in terms of object density. It is applicable to nearly a hundred objects while providing more efficient computation time than SQAIR. (ii) We propose parallelizing the propagation–discovery process by introducing the propose–reject model, reducing the time complexity to O(1). (iii) SCALOR can model scenes with a dynamic background. (iv) SCALOR is the first probabilistic model demonstrating its working on a significantly more complex task, i.e., natural scenes containing tens of objects as well as background.

heading: 2 PRELIMINARIES: SEQUENTIAL ATTEND INFER REPEAT (SQAIR)
text: SQAIR models a sequence of images x = x1:T by assuming that observation xt at time t is generated from a set of object latent variables zOt = {zt,n}n∈Ot withOt the set of objects present at time t. Latent variable zt,n corresponding to object n consists of three factors (z pres t,n , z where t,n , z what t,n ), which represent the existence, pose, and appearance of the object, respectively. SQAIR also assumes that an object can disappear or be introduced in the middle of a sequence. To model this, it introduces the propagation–discovery model. In propagation, a subset of currently existing objects is propagated to the next time-step and those not propagated (e.g., moved out of the scene) are deleted. In discovery, after deciding how many objects Dt will be discovered, Dt objects are newly introduced into the scene. Combining the propagated set Pt and discovered set Dt, we obtain the set of currently existing objects Ot. The overall process can be formalized as:
p(x1:T , z fg 1:T , D1:T ) = p(D1, z D 1 ) T∏ t=2 p(xt|zfgt ) p(Dt, zDt |zPt ) p(zPt |zfgt−1) . (1)
For SQAIR, we use zfgt (“fg” standing for foreground) to denote z O t because SQAIR does not have any latent variables for background. Due to the intractable posterior, SQAIR is trained through variational inference with the following posterior approximation:
q(D1:T , z fg 1:T |x1:T ) = T∏ t=1 q(Dt, z D t |xt, zPt ) ∏ n∈Ot−1 q(zt,n|zt−1,n,x≤t) . (2)
SQAIR is trained using the importance-weighted autoencoder (IWAE) objective (Burda et al., 2015). The VIMCO estimator (Mnih & Rezende, 2016) is used to backpropagate through the discrete random variables while using the reparameterization trick (Kingma & Welling, 2013; Williams, 1992) for continuous variables. SQAIR has two main limitations in terms of scalability. First, for propagation, SQAIR relies on an RNN, which sequentially processes each object by conditioning on previously processed objects. Second, the discovery is also sequential because it uses RNN-based
discovery based on AIR (Eslami et al., 2016). Consequently, SQAIR has a time complexity of O(|Ot|) per step t. In Crawford & Pineau (2019b), the authors demonstrated that this sequential discovery can easily fail beyond the scale of a few objects. Moreover, SQAIR lacks any model for the background and its temporal transitions, which is important in modeling natural scenes.

heading: 3 SCALOR
text: In this section, we describe the proposed model, SCALOR. We first describe the generative process along with the proposal-rejection mechanism, which is designed to prevent propagation collapse, and then the inference process and learning.

heading: 3.1 GENERATIVE PROCESS
text: SCALOR assumes that an image xt is generated by background latent z bg t and foreground latent zfgt . The foreground is further factorized into a set of object representations z fg t = {zt,n}n∈Ot . In SCALOR, we represent an object by zt,n = (z pres t,n , z where t,n , z what t,n ) similarly to SQAIR. The appearance representation zwhatt,n is a continuous vector representation, and z where t,n is further decomposed into the center position zpost,n, scale z scale t,n , and depth z depth t,n . The depth representation, which is missing in SQAIR, represents the relative depth between objects from the camera viewpoint. This depth modeling helps deal with object occlusion. The foreground mask mt,n obtained from zwhatt,n is used to distinguish background and foreground. We adopt the propagation–discovery model from SQAIR, but improve it in such a way to resolve the scalability problem. The generative process of SCALOR can be written as:
p(x1:T , z1:T ) = p(z D 1 )(z bg 1 ) T∏ t=2 p(xt|zt)︸ ︷︷ ︸ rendering p(zbgt |zbg<t, zfgt )︸ ︷︷ ︸ background transition p(zDt |zPt )︸ ︷︷ ︸ discovery p(zPt |z<t)︸ ︷︷ ︸ propagation , (3)
where zt = (z bg t , z fg t ). As shown, the generation process is decomposed into four modules: (i) propagation, (ii) discovery, (iii) background transition, and (iv) rendering.
Propagation. The propagation in SCALOR is modeled as follows: p(zPt |z<t) = ∏ n∈Ot p(zprest,n |z<t,n) { p(zwheret,n |z<t,n) p(zwhatt,n |z<t,n) }zprest,n , (4) where p(zprest,n |z<t,n) is a Bernoulli distribution with parameter βt,n. The distributions of “what” and “where” are defined only when the object is propagated. To implement this, for each object n we assign an object-tracker RNN denoted by its hidden state ht,n. The RNN is updated by input zt,n for all t where the object n is present in the scene. The parameter βt,n is obtained as βt,n = fmlp(ht,n). If zprest,n = 0, the object n is not propagated and the tracker RNN is deleted. Importantly, unlike the RNN-based sequential propagation in SQAIR, the propagation in SCALOR is fully parallel.
Discovery by Proposal-Rejection. The main contribution in making our model scalable with respect to the the number of objects is our new discovery model that consists of two phases: proposal and rejection. In the proposal phase, we assume that the target image can be covered by H ×W latent grid cells, and we propose an object latent variable z̃t,h,w per grid cell. This proposal phase can be written as:
p(z̃Dt |zPt ) = HW∏ h,w=1 p(z̃Dt,h,w|zPt ) = HW∏ h,w=1 p(z̃prest,h,w|zPt ) { p(z̃wheret,h,w|zPt ) p(z̃whatt,h,w|zPt ) }z̃prest,h,w . (5) In the rejection phase, our goal is to reject some of the proposed objects if a proposed object largely overlaps with a propagated object. We realize this by using the mask variable mt,n. Specifically, if the overlapping area between the mask of a proposed object and that of a propagated object is over a threshold τ , we reject the proposed object. This procedure can be described as (i) proposal: z̃Dt ∼ p(z̃Dt |zPt ) and (ii) accept-reject: zDt = faccept-reject(z̃Dt , zPt , τ). In this way, the final discovery set zDt is always a subset of the proposal set z̃ D t , i.e., z D t ⊆ z̃Dt . Although we use a deterministic function for the rejection, it can be a design choice to implement this as a stochastic decision. While one rationale behind this design is to reflect an inductive bias of a Gestalt
principle saying that two objects cannot coexist in the same position, we shall also see later further reasons as to why this design is effective. The final discovery model can then be written as: p(zDt |zPt ) = p(z̃Dt |zPt ) ∏HW h,w=1 p(z D t,h,w|zPt , z̃Dt , τ), where the acceptance model is:
p(zDt,h,w|zPt , z̃Dt , τ) = f(zacceptt,h,w|zPt , z̃Dt , τ)p(z̃wheret,h,w)z accept t,h,wp(z̃whatt,h,w) zacceptt,h,w . (6)
Background Transition. Unlike SQAIR, SCALOR is endowed with a background model. The background image is encoded into a D-dimension continuous vector zbgt from the background transition p(zbgt |zbg<t, zfgt ). The background RNN encodes the temporal transition of background images. Rendering. The implementation of the rendering process is the same as in SPAIR (Crawford & Pineau, 2019b), except that we process the objects in parallel. Implementation details are shown in Appendix A.

heading: 3.2 LEARNING AND INFERENCE
text: Due to the intractability of the true posterior distribution p(z1:T |x1:T ), we train our model using variational inference with the following posterior approximation:
q(z1:T |x1:T ) = T∏ t=1 q(zt|z<t,x≤t) = T∏ t=1 q(zbgt |zfgt ,xt) q(zDt |zPt ,x≤t) q(zPt |z<t,x≤t) . (7)
Posterior Propagation. q(zPt |z<t,x≤t) is similar to the propagation in generation, except that we now provide observation x≤t through an RNN encoding. Here, the propagation for each object n is done by q(zt,n|z<t,n,x≤t) using attention at,n = fatt(x≤t) on the feature map for object n. To compute the attention, we use the previous position zpost−1,n as the center position and extract half the width and height of the convolutional feature map using bilinear interpolation. This attention mechanism is motivated by the observation that only part of the image contains information for tracking an object and an inductive bias that objects cannot move a large distance within a short time span (i.e., objects do not teleport).
Posterior Discovery. The posterior discovery also consists of proposal and rejection phases. The main difference is that we now compute the proposal in spatially-parallel manner by conditioning on the observations x≤t, i.e., q(z̃Dt |zPt ,x≤t) = ∏HW h,w=1 q(z̃ D t,h,w|zPt ,x≤t). Here, the observation x≤t is encoded into the feature map of dimensionalityH×W×D using a Convolutional LSTM (Xingjian et al., 2015). Then, from each feature we obtain z̃Dt,h,w. Importantly, this is done over all the feature cells h,w in parallel. A similar approach is used in SPAIR (Crawford & Pineau, 2019b), but it infers the object latent representations sequentially and thus is difficult to scale to a large number
of objects (Lin et al., 2020). Even if this spatially-parallel proposal plays a key role in making our model scalable, we also observe another challenge due to this high capacity of the discovery module. The problem is that the discovery module tends to dominate the propagation module and thus most of the objects in an image are explained by the discovery module, i.e., objects are rediscovered at every time-step while nothing is propagated. We call this problem propagation collapse.
Why would the model tend to explain an image through discovery while suppressing propagation? First, the model does not care where—either from discovery or propagation—an object is sourced from as long as it can make an accurate reconstruction. Second, the propagation step performs a much harder task than the discovery. For the propagation to properly predict, tracker n needs to learn to find the matching object from an image containing many objects. Although the propagation attention plays an important role in balancing the discovery and propagation, we found that it does not eliminate the problem of re-discovery, and without rejection, its effectiveness varies across different experiment settings. On the contrary, the discovery module does not need to solve such a difficult association problem because it only performs local image-to-latents encoding without associating latents of the previous time-step. Therefore, it is much easier for the discovery encoder to produce latents that are more accurate than those inferred from propagation. If we limit the capacity of the discovery module and sequentially process objects like in SQAIR, we may mitigate this problem because the propagation module is naturally encouraged to explain what this weakened discovery module cannot. This approach, however, cannot scale.
We employ two techniques to resolve the aforementioned problem. First, we simply bias the initial network parameter so that it has a high propagation probability at the beginning of the training. This helps the model prefer to explain the observation first through propagation. The second technique is our proposal-rejection mechanism, which is implemented the same way as in the generation process. This prevents the discovery model from redundantly explaining what is already explained by the propagation module. The posterior for the discovery model can be written as:
q(zDt |zPt ,x≤t) = q(z̃Dt |zPt ,x≤t) HW∏ h,w=1 paccept(z D t,h,w|zPt , z̃Dt ) , (8)
where the acceptance model is paccept(zDt,h,w|zPt , z̃Dt ) = p(z pres t,h,w|zPt , z̃Dt )(p(z̃wheret,h,w)p(z̃whatt,h,w))z pres t,h,w .
Posterior Background. The posterior of the background q(zbgt |zfgt ,xt) is conditioned on the input image and currently existing objects. Here, we provide the foreground latents so that the remaining parts in the image can be explained by the background module.
Training. We train our model by maximizing the following evidence lower bound L(θ, φ) = T∑ t=1 Eqφ(z<t|x<t) [ Eqφ(zt|z<t,x≤t) [log pθ(xt|zt)]−KL [qφ(zt|z<t,x≤t) ‖ pθ(zt|z<t)] ] . (9)
We use the reparameterization trick (Williams, 1992; Kingma & Welling, 2013) for continuous random variables such as zwhat, and the Gumbel-Softmax trick (Jang et al., 2016) for discrete variables such as zpres. We found that our proposed model works well and robustly with these simpler training methods than what is used in SQAIR, i.e., VIMCO and IWAE.

heading: 4 RELATED WORK
text: Different approaches have been taken to tackle the problem of unsupervised scene representation learning. Object-oriented models such as AIR (Eslami et al., 2016) and SPAIR (Crawford & Pineau, 2019b) decompose scenes into latent variables representing the appearance, position, and size of the underlying objects. While AIR makes use of a recurrent neural network, SPAIR applies spatially invariant attention to extract local feature maps. Although the latter provides better scalability than AIR, it is still limited as it performs sequential inference on objects. SQAIR (Kosiorek et al., 2018), discussed in Section 2, extends the ideas proposed in AIR to temporal sequences. On the other hand, scene-mixture models (Greff et al., 2017; Van Steenkiste et al., 2018; Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2019) decompose scenes into a collection of components, each being a full-image level representation. Although such models allow decomposition of the input image into components, they are not object-wise disentangled as multiple objects can be in the same
component. Furthermore the obtained representation does not contain explicit interpretable features like position and scale, etc. SPACE (Lin et al., 2020) combines both of the above approaches by using object detection for foreground and mixture decomposition for background. It improves upon SPAIR by parallelizing the latent inference process.
DDPAE (Hsieh et al., 2018) is another object-oriented sequential generative model that models each object with an appearance and a position vector. The model assumes the appearance of an object to be fixed, and thus shares the content vector across different time-steps. NEM (Greff et al., 2017) and RNEM (Van Steenkiste et al., 2018) introduce a spatial mixture model to disentangle the scene into multiple components representing each entity. Since each component generates a full scene image, the latent representations are not interpretable. Tracking-By-Animation (He et al., 2019) introduces a deterministic model to tackle the task of object tracking in an unsupervised fashion. Furthermore, there is a substantial amount of work on object tracking from the computer vision community using the same “bounding box”-representation approach proposed in SCALOR (Kosiorek et al., 2017; Ning et al., 2017; Nam & Han, 2016; Tao et al., 2016), but such methods use provided labels to tackle the problem of object tracking and thus are usually fully or semi-supervised and not probabilistic object-oriented models.
We also note that Crawford & Pineau (2019a) has independently and concurrently developed a similar architecture to ours. This model also emphasizes the scalability problem with a similar idea motivated by parallelizing SPAIR and extending it to sequential modeling. The main differences are the usage of the proposal-rejection mechanism and the background modeling that make our model work on complex natural scenes.

heading: 5 EXPERIMENTS
text: In this section, we describe the experiments conducted to empirically evaluate the performance of SCALOR. We propose two tasks, (i) synthetic MNIST/dSprites shapes and (ii) natural-scene CCTV footage of walking pedestrians. We will show SCALOR’s abilities to detect and track objects, to generate future trajectories, and to generalize to unseen settings. Furthermore, we provide a quantitative comparison to state-of-the-art baselines.

heading: 5.1 TASK 1: LARGE-SCALE MNIST AND DSPRITES SHAPES
text: We first evaluate our model on datasets of moving dSprites shapes as well as moving MNIST digits. In all experiments, the image sequence covers a 64 × 64 partial view of the center of the whole environment. Therefore, while there is a fixed number of objects in the environment at each timestep, only a subset of them are visible in the observed image. The environment size is customized for each setting in a way that objects can conveniently move out of the viewpoint completely and re-enter within a few time-steps. We test on five different scale settings. In each setting, the number of objects in each trajectory is sampled uniformly from an interval [min, max]. Each scale setting is specified with a triplet (min, avg,max), where min and max are as mentioned and avg represents the average number of visible objects in the trajectories in that setting. The five settings are referred to as Very Low Density (VLD) [(2, 2.9, 4)], Low Density (LD) [(8, 8, 11)], Medium Density (MD) [(18, 20, 24)], High Density (HD) [(50, 55, 64)] and Very High Density (VHD) [(90, 90, 110)]. For
example, in the MD setting, there are always between 18 to 25 objects in the overall environment, while only about 20 of them are visible on average in each frame.
Experiment 1 - Tracking Performance. This experiment evaluates the tracking performance in an environment without background. We use the following tracking metrics: Multi-Object Tracking Accuracy (MOTA), precision-recall of the inferred bounding boxes (Bernardin & Stiefelhagen, 2008), reconstruction mean squared error, and normalized counting mean absolute error (CountMAE). CountMAE measures the difference between the number of predicted objects and the actual number of objects, normalized by the latter. MOTA measures the tracking accuracy and consistency of the bounding boxes. Precision-recall measures the number of false-positive and negatives, regardless of the associated IDs. For computing the MOT metrics, we choose the Euclidean distance threshold to be twice the actual object size in each setting. Figure 2 shows the quantitative results of SCALOR compared to baselines. More detailed quantitative results are given in Appendix B.
We compare the performance of the proposed model with SQAIR in the VLD setting for both MNIST and dSprites datasets, and LD setting for dSprites. Note that we were not able to make SQAIR work on other settings due to the high object density. As shown in Figure 2, SCALOR outperforms SQAIR in all these settings, obtaining significantly higher accuracy and recall. SQAIR either misses some objects or misidentifies distinct objects as one. In addition, SCALOR has lower values of CountMAE in comparison to SQAIR, showing that SCALOR can infer the number of objects in the scene more accurately. Furthermore, we observe that increasing the number of objects in the scene does not significantly impede SCALOR’s tracking ability, which demonstrates the strength of SCALOR when applied to images with a high number of objects. SCALOR can achieve relatively high precision-recall even for scenes containing about 100 objects. Note that in the VHD case, the number of objects (about 90) in the first time-step exceeds the number of detection grid cells (8×8) the discovery model has. Thus, the model can only detect up to 64 objects at the first time-step, and detects the rest in the following time-steps. This results in lower performance on the tracking and detection accuracy in the VHD case. This is demonstrated in Figure 10 in Appendix C.
Figure 3 demonstrates the qualitative performance of SCALOR on dSprites and MNIST (HD). To clarify tracking consistency, bounding boxes with distinct ids are represented by distinct colors. Discovered objects are emphasized by white circles. As shown in Figure 3(a), the discovery module of SCALOR can identify newly introduced objects and put them in the propagation list while the propagation module keeps tracking existing objects. Figure 3(c) shows object-wise rendering of inferred zwhat latent variables. For clear visualization, object-wise rendering is shown only for a
subset of objects present in all the time-steps. SCALOR infers consistent object-wise representation even when objects are largely occluded.
Experiment 2 - Dynamic Background. Another interesting yet challenging setting is the presence of a dynamic background. As we can see in Figure 4, SCALOR can decompose the video sequence to a set of foreground objects as well as a dynamic background. While the background module can model the background dynamics, it can also model the dynamics of the whole environment including the moving objects. This usually brings competition between background module and foreground module on explaining the foreground part. However, the background module and foreground module cooperate properly in SCALOR. We believe this is because of our modeling that the background module obtains information from the foreground part by conditioning on the foreground latent variables. Table 1 also provides the tracking performance of the model for this setting to show how complex images affect the tracking quality. We observe that it achieves comparable performance to the default no-background setting. These results are provided in Table 1 under “SCALOR – BG”.
Experiment 3 - Future Time Prediction/Generation. This section showcases the generation ability of SCALOR via two experiments. The first experiment is aimed at showing conditional generation. Here, the model is provided with the first 5 frames, and then tested to generate the next 5 frames. Latent variables at each time-step are sampled independently from the prior distribution of the propagation step conditioned on latent variables from previous time-step. Because the focus is on conditional generation of background and objects, discovery prior of zpres is manually set to zero. Figure 5 shows a generated sample. The first 5 frames (before the red line) correspond to reconstruction while the next 5 frames represent conditional generation1. As we can see in Figure 5, SCALOR can generate dynamic background and object trajectories that are reasonably consistent.
The second experiment showcases video generation from scratch. In this setting, the model generates all latent variables from the discovery prior at the first time-step. For the following time-steps, latent variables are sampled by conditioning on the latent variables of the previous time-steps. The object generation probability for each grid cell at the first time-step, i.e. zpres1,h,w, is set to 0.2. Figure 6 shows one generated sample. At the first time-step where all variables are sampled from a fixed prior, the background is generated reasonably, but the objects are generated only partially. This behavior is expected because newly introduced objects are mostly partially observed at image boundaries. This makes the inference network learn a zwhat representation corresponding to the partial views. Furthermore, since the zwhat prior is a standard Gaussian, it is hard to model the multi-modal object appearance. Yet, it is interesting to see that the conditional generation of each object’s zwhat gradually converges to a consistent complete view of an object. This demonstrates that in our model,
1Examples with longer duration are in the project website
the distribution of zwhatt≥2 in propagation can actually achieve multi-modality when conditioned on zwhat1 from the first time-step. Furthermore, it shows that the propagation prior network is capable of maintaining a consistent representation of the object along the sequence as is reflected in the dataset. Therefore, SCALOR is capable of generating objects with independent appearance and behavior.

heading: 5.2 TASK 2: REAL-WORLD SCENARIO
text: This section considers the performance of SCALOR on real-world natural video, which previous models have not been able to handle due to the scalability issue and lack of a background model. Compared to synthetic data, the challenges in this setting are significantly more difficult. Specifically, we consider the Crowded Grand Central Station dataset (Zhou et al., 2012), which was collected from CCTV cameras at New York City’s Grand Central Station. Due to the complex pedestrian behavior, the density of the dataset can be considered a mix of LD and HD.
Figure 7 shows the tracking result on a sample sequence. We can see that SCALOR performs reasonably well on this pedestrian tracking dataset by maintaining consistent temporal trajectories. As shown in Figure 7(c), the background module infers the background component and reconstructs the extracted background correctly. As for object detection, SCALOR succeeds in accurate pedestrian detection and tracking. Furthermore, the foreground mask produced by zwhat provides the segmentation of each individual pedestrian, as shown in Figure 7(d). We draw tracking trajectories in Figure 7(e) for each pedestrian in the natural scene. Trajectories in different colors correspond to different pedestrian ids inferred by the identity of the latent variable of each object. Additional figures and the dataset details are provided in Appendix D.
Figure 14 in Appendix D shows the conditional generation. The first 5 frames are the inference reconstruction while the last 5 frames are model generation. Starting from the 6th frame, the latent transition of the propagation trackers is modeled by the sequential prior network. In the generation process, a different prior of zpres is introduced in the discovery module to introduce new objects emerging in the scene (see Appendix E for more details). As shown in the figure, the model tends to generate movement aligned with the previous frames. This also applies to newly generated objects from the discovery phase as shown in Figure 14(f). Although the trajectories are consistent, the appearance of generated objects fails to maintain its consistency across different time frames. Noticeably, in the generated sequence the segmentation mask for each object tends to deform into a different shape. This may stem from imperfections during the inference that makes the learning of appearance transition difficult. Additional figures for generation are provided in Appendix D.
The ground truth trajectories of the Grand Central Station Dataset were not available when this experiment was conducted. We instead use negative log-likelihood (NLL) to compare our model with two baselines, a sequential VAE and a Recurrent Latent Variable Model (VRNN). The sequential VAE has one latent variable z of dimensionality 64, and a sequential prior p(zt|z<t). It is similar to our background module with the same number of latent variables for encoder and decoder. As for the VRNN, we implement a VRNN with 128 dimensions of the LSTM hidden state and latent variable z
and choose a convolutional network as the image encoder and decoder. The NLL value for our model is 28.30, and for the VAE and VRNN baseline it is 27.59 and 27.79 respectively. While learning a highly structured representation, SCALOR can still obtain a comparable generation quality.

heading: 5.3 ABLATION STUDY AND COMPUTATIONAL EFFICIENCY COMPARISON
text: Ablation Study. We perform an ablation study on the rejection mechanism and the propagation attention mechanism. Here, we train our model on the moving dSprites dataset where the number of objects varies from 6 to 36. We use two metrics to evaluate different architectures. The first metric is the propagation rate, which measures how long an object is propagated. Since, by design, objects always stay in a scene for more than one frame, the propagation rate can be regarded as a proxy to measure the success rate of tracking through propagation. The second metric counts the number of discoveries that occurred in the whole sequence. If the discovery module dominates, a high value will be observed. As we can see in Figure 8 (a), with no rejection (Rej) or attention (Att) in propagation, the propagation rate goes down to 30% in early training iterations and keeps decreasing as the training progresses. In this setting, we found that the model re-discovers objects in every frame without tracking them properly. This is also shown in Figure 8 (b), where it has a significantly larger number of discoveries. With the propagation attention mechanism, the propagation rate increases to 95%. This is because the attention mechanism reduces the cost of finding the matching object between time-steps in propagation. The model thus favors tracking in propagation over re-discovery. It also prevents the discovery module from propagation collapse. However, as we can see in Figure 8 (b), the re-discovery problem still exists in this setting. The rejection mechanism, however, makes the propagation rate converge to 1 even without propagation attention. The propagation attention together with the rejection produced more accurate boxes.
Computational Efficiency. In Figure 8 (c) and (d), we measure the inference time (c), and training convergence time (d). For measuring the inference time, we consider a hypothetical situation with N objects in the first frame. The model is desired to discover and propagate them accordingly. Furthermore, we set the discovery capacity of SQAIR to be the same as the number of discovery grid cells in SCALOR. Figure 8(c) demonstrates how much time one forward step takes, as the number of objects/discovery capacity increases from 4 to 64. SQAIR’s speed decreases linearly as both its discovery and propagation mechanisms process objects sequentially. In contrast, SCALOR does not suffer from such phenomena as discovery and propagation are done in parallel. Figure 8 (d) shows MSE convergence vs. training time when both models are trained on the MNIST VLD setting. SCALOR converges to a lower MSE than SQAIR and does so orders of magnitude faster.

heading: 6 CONCLUSION
text: We introduce SCALOR, a probabilistic generative world model aimed at visually modeling an environment of crowded dynamic objects. With the proposed parallel discovery-propagation and proposal-rejection mechanism, we improve the capacity of object density from a few objects to up to a hundred objects. Unlike previous models, the proposed model can also deal with dynamic backgrounds. These contributions consequently makes our model applicable, for the first time in this line of research, to natural scenes. An interesting future direction is to introduce more structure to the background and to model interactions among objects and the background.

heading: ACKNOWLEDGMENTS
text: SA thanks Kakao Brain and Center for Super Intelligence (CSI) for their support.

heading: A ALGORITHMS
text: Algorithm 1: Discovery Proposal-Rejection Inference Input: x1:t - image sequence up to current time-step
MPt - propagated mask from current time-step τ - rejection hyper-parameter
/* Feature encoding */ eimgt = ConvLSTM(x1:t) emaskt = MaskEncoder(M P t ) eaggt = Concat[e img t , e mask t ] /* Proposal-Rejection (done in parallel) */ AcceptedListt = [] for h← 1 to H do
for w ← 1 to W do /* Objects proposal */ z̃prest,h,w ∼ Bern(·|f pres nn (e agg t,h,w))
if z̃prest,h,w == 0 then continue end z̃wheret,h,w ∼ N (·|fwherenn (e agg t,h,w)) gattt,h,w = STN(xt, z̃ where t,h,w) // Attended Glimpse z̃whatt,h,w ∼ N (·|GlimpseEnc(gattt,h,w)) ot,h,w,mt,h,w = STN−1(GlimpseDec(z̃whatt,h,w)), z̃ where t,h,w) /* Accept-Reject Test */ δ = A(mDt,h,w∩MPt ) A(mDt,h,w)
// A - pixel area if δ < τ then
AcceptedListt.Add(z̃ what t,h,w, z̃ where t,h,w, z̃ pres t,h,w)
end end
end Output: AcceptedListt
Algorithm 2: Propagation Inference Input: x1:t - image sequence up to current time-step PropListt−1 = {zt−1,n,ht−1,n}n∈Ot−1 - latent variable from previous time-step /* Feature encoding */ eimgt = ConvLSTM(x1:t) /* Object tracking (done in parallel) */ PropListt = PropListt−1 for n← 1 to Ot−1 do
eattt,n = f att nn(STN(e img t,n , (z pos t−1,n, z scale t−1,n))) // Feature map attention ht,n = GRU(fnn([eattt,n, z what t−1,n, z pos t−1,n, z scale t−1,n, z pres t−1,n]),ht−1,n) eaggt,n = f agg nn ([eattt,n, z what t−1,n, z pos t−1,n, z scale t−1,n,ht,n]) zpost,n, z scale t,n ∼ N (.|fwherenn (eaggt,n)) gattt,n = STN(xt, (z pos t,n, z scale t,n )) // Attented Glimpse zwhatt,n ∼ N (·|GlimpseEnc(gattt,n)) zdeptht,n ∼ N (·|fnn(eattt,n, zwhatt,n ,ht,n)) zprest,n ∼ Bern(·|f presnn (zpost,n, zscalet,n , zwhatt,n ,ht,n)) zwheret,n = (z pos t,n, z scale t,n , z depth t,n ) if zprest,n == 1 then PropListt,n.Update(z what t,n , z where t,n , z pres t,n ) else PropListt,n.Delete()
end end Output: PropListt
Algorithm 3: Background Module and Rendering Input: xt - image at current time-step,
ot = {ot,n}n∈Ot - object RGB glimpses mt = {mt,n}n∈Ot - object masks glimpses, {(zpost,n, zscalet,n ), zprest,n }n∈Ot - object latents for position and presence
/* Foreground object rendering (done in parallel) */ for n← 1 to Ot do
xfgt,n = STN −1(ot,n, (z pos t,n, z scale t,n )) γt,n = STN −1(mt,n · zprest,n σ(−zdeptht,n ), (zpost,n, zscalet,n ))
γt,n = normalize(γt,n,∀n) end xfgt = ∑ n∈Ot x fg t,nγt,n /* Foreground mask rendering */ for n← 1 to Ot do
Mt,n = STN−1(mt,n, (z pos t,n, z scale t,n ))
end Mt = min( ∑ n∈Ot Mt,n, 1) /* Background rendering */ ebg = BackgroundEncoder(Concat[xt, (1−Mt)]) zbg ∼ N (.|f bgnn(ebg)) xbgt = BackgroundDecoder(z
bg) /* Foreground background combination */ xt = x fg t + (1−Mt) xbgt Output: xt

heading: B ADDITIONAL QUANTITATIVE RESULT
text: Table 1 includes Multi Object Tracking Accuracy (MOTA) as well as precision-recall of the inferred bounding boxes (Bernardin & Stiefelhagen, 2008).
Table 2 provides a comparison of SCALOR to SQAIR and VRNN in terms of the reconstruction error (MSE) and negative log-likelihood (NLL). NLL is computed per pixel across the whole sequence.

heading: C ADDITIONAL EXPERIMENTS ON DSPRITES AND MNIST
text: C.1 FREQUENT DENSE DISCOVERY
This experiment evaluates the ability to discover many newly introduced objects across time-steps. This is important because in many applications only key-frames of a video are available, i.e., frames at which significant changes happen. An example of a key-frame is a sudden change in the observer’s viewpoint. This change of viewpoint introduces many new objects in the frame. Figure 9 shows one such instance. In this setting, 10–15 objects are introduced at the first, fifth, and ninth time-steps, respectively. As is shown, SCALOR is able to discover many new objects in each frame.
C.2 VERY HIGH DENSITY
Figure 10 shows samples from the Very High Density experiment. This experiment places 90–110 objects in the overall environment, around 90 of which are visible at every time-step on average. The discovery module contains 8 × 8 detection grid cells and thus can only detect up to 64 objects. Interestingly, as is shown, the model will identify as many objects as possible in the first time-step and discover the remaining in the second time-step. Furthermore, if too many objects are densely packed in one local region of the space, SCALOR will perform similarly detecting them through multiple frames.
C.3 ABILITY TO HANDLE OVERLAP AND OCCLUSION
Figure 11 shows a sample of the MNIST settings where objects move towards each other more aggressively. In this setting, there is a higher chance of overlapping and occlusion. As shown, the identity of the objects is preserved when objects overlap with each other.
C.4 GENERALIZATION TEST
We conduct three sets of experiments on generalization. In the first experiment, we investigate generalization to longer sequences. In this setting, the model is trained on trajectories of length 10 while being tested on trajectories of length 20. In the second experiment, we evaluate generalization in more crowded scenes. In this setting, the model is trained on 15–25 objects and tested on 50–60 objects. The third experiment tests the generalization of the model to unseen objects. The model is trained only on moving MNIST images containing digits 0 to 5 while being tested on images containing digits 6 to 9. Figures 12 and 13 show samples from these experiments. “SCALOR – LG” in Table 1 also provides tracking results for the “length generalization” setting.

heading: D EXPERIMENT DETAIL AND ADDITIONAL QUALITATIVE RESULTS ON GRAND CENTRAL STATION DATASET
text: For natural-scene experiments, we spatially split the video into 8 parts and create a dataset of 400k frames in total. We choose the first 360k frames for training and 40k frames for testing. Since the movement of pedestrians is too slow under 25 fps in the original video, we treat every other 7 frames as two consecutive frames in the dataset sequence. The length of the input sequence is 10, and each image is resized to 128× 128.
The generation result mentioned in Section 5.2 is provided in Figure 14. We provide additional qualitative results in Figures 15 to 20. Rows from top to bottom represent input sequence, overall reconstruction, extracted background, foreground segmentation mask with IDs, center position from each zpost,n latent, and extracted trajectories based on the transition of center position. Additionally, Figures 18 to 20 show examples of conditional generation, where images starting from time-step 6 (after the red line) are generated sequences. Rows from top to bottom are conditional generations of the overall images, conditional generations of background, extracted conditional generation of each segmentation mask, and conditional generation of trajectories.

heading: E MODEL ARCHITECTURE DETAILS
text: In this section, we provide additional details of the architecture and hyperparameters used for pedestrian detection. When a new frame is provided, the network uses a fully convolutional encoder to obtain aH×W feature map. The feature map is fed into a convolutional LSTM to model the sequential information along the sequence. The convolutional LSTM is shared across discovery module and propagation module. The discovery module and propagation module also share the zwhat encoder and decoder. The encoder has a convolutional network followed by one fully connected layer, while the glimpse decoder uses a fully convolutional network with sub-pixel layer (Shi et al., 2016) for upsampling. The background module shares a similar structure with the zwhat encoder and decoder. It takes a 4-dimensional input, i.e., RGB and foreground mask, and outputs a 3-dimensional image. We use GRUs in propagation trackers and in prior transition networks.
We choose a batch size of 20 for the natural scene experiments and a batch size of 16 for MNIST/dSprites experiments. The learning rate is fixed at 4e-5 for natural image experiments and 5e-4 for dSprites/MNIST experiments. We use RMSprop for optimization during training. The standard deviation of the image distribution is chosen to be 0.1 for natural experiments and 0.2 for toy experiments. The prior for all Gaussian posteriors is set to standard normal. For the pedestrian tracking dataset, we constrain the range of zscale so that the inferred width can vary from 5.2 pixels to 11.7 pixels, and the height can vary from 12.0 to 28.8, and both with a prior of the middle value in discovery. Similarly, we constrain zscale on synthetic datasets so that it can vary from half to 1.5 times the actual object size. The zpos variable in the propagation phase is modeled as the deviation of the position from the previous time-step instead of the global coordinate. The prior for zpres in discovery is set to be 0.1 at the beginning of training and to quickly anneal to 1e-3 for natural image experiments and 1e-4 for dSprites/MNIST experiments. The temperature used for modelling zpres is set to be 1.0 at the beginning and anneal to 0.3 after 20k iterations.
Full details of the architecture will be released along with our code.

Review summaries:
This paper proposes a generative model for scalable sequential object-oriented representation. The paper proposes several improvements based on the method SQAIR (Kosiorek et al. 2018b), (1) modeling the background and foreground dynamics separately; (2) parallelizing the propagation-discovery process by introducing the propose-reject model which reducing the time complexity. Finally, the proposed model can deal with orders of magnitude more objects than previous methods, and can model more complex scenes with complex background.