SCALOR: Generative World Models with Scalable Object Representations
Authors: Jindong Jiang*
Sepehr Janghorbani*
Gerard De Melo
Sungjin Ahn
Venue:  (2020)

Abstract: Scalability in terms of object density in a scene is a primary challenge in unsupervised sequential object-oriented representation learning. Most of the previous models have been shown to work only on scenes with a few objects. In this paper, we propose SCALOR, a probabilistic generative world model for learning SCALable Object-oriented Representation of a video. With the proposed spatially parallel attention and proposal-rejection mechanisms, SCALOR can deal with orders of magnitude larger numbers of objects compared to the previous state-of-the-art models. Additionally, we introduce a background module that allows SCALOR to model complex dynamic backgrounds as well as many foreground objects in the scene. We demonstrate that SCALOR can deal with crowded scenes containing up to a hundred objects while jointly modeling complex dynamic backgrounds. Importantly, SCALOR is the ﬁrst unsupervised object representation model shown to work for natural scenes containing several tens of moving objects.

Published as a conference paper at ICLR 2020
SCALOR: GENERATIVE WORLD MODELS WITH SCALABLE OBJECT REPRESENTATIONS
Jindong Jiang', Sepehr Janghorbani' Gerard de Melo & Sungjin Ahn
Rutgers University
ABSTRACT
Scalability in terms of object density in a scene is a primary challenge in unsu- pervised sequential object-oriented representation learning. Most of the previous models have been shown to work only on scenes with few objects. In this pa- per, we propose SCALOR, a probabilistic generative world model for learning SCALable Object-oriented Representation of a video. With the proposed spatially- parallel attention and proposal-rejection mechanisms, SCALOR can deal with orders of magnitude larger numbers of objects compared to the previous state- of-the-art models. Additionally, we introduce background module that allows SCALOR to model complex dynamic backgrounds as well as many foreground objects in the scene. We demonstrate that SCALOR can deal with crowded scenes containing up to a hundred objects while jointly modeling complex dynamic back- grounds. Importantly, SCALOR is the first unsupervised object representation model shown to work for natural scenes containing several tens of moving ob- jects. httts: http:///sites...ogeeee.e www/view http://wwwsall//hhhm...
INTRODUCTION
Unsupervised structured representation learning for visual scenes is key challenge in machine learning. When a scene is properly decomposed into meaningful entities such as foreground objects and background, can benefit from numerous advantages of abstract symbolic representation. These include interpretability, sample efficiency, the ability of reasoning and causal inference, as well as compositionality and transferability for better generalization. In addition to symbols, another essential dimension is time. Objects, agents, and spaces all operate under the governance of time. Without accounting for temporal developments, it is often much harder if not impossible to discover certain relationships in a scene.
Among few methods that have been proposed for unsupervised learning of object-oriented repre- sentation in temporal scenes, SQAIR (Kosiorek et al. 2018) is by far the most complete model. As a probabilistic temporal generative model, it learn object-wise structured representation while modeling underlying stochastic temporal transitions in the observed data. Introducing the propagation propagation-discovery model. SQAIR can also handle dynamic scenes where objects may disap- pear or be introduced in the middle of sequence. Although SQAIR provides promising ideas and shows the potential of this important direction, a few key challenges remain, limiting its applicability merely to synthetic toy tasks that are far simpler than typical natural scenes.
The first and foremost limitation scalability. Sequentially processing every object in an image, SQAIR has fundamental limitation in scaling up to scenes with large number of objects. As such, the state-of-the-art remains at the level of modeling videos containing only few objects, such as MNIST digits, per image. Considering the complexity of typical natural scenes as well as the importance of scalable unsupervised object perception for applications such as self-driving systems, it is thus challenge of the highest priority scale robustly to scenes with large number of objects. Scaling up the object-attention capacity is an important problem because it allows us to maximize the modern parallel computation that maximize search capacity. This is contrary to humans, who can attend only to few objects at a time in a time-consuming sequential manner. The
Authors with equal contribution. Email addresses: fjindong. j jiang, sepehr, janghorbani, ge gerard d e, sungji n.

---

Published as a conference paper at ICLR 2020
AlphaGo system (Silver al. 2017) is an example demonstrating the power of such parallel search (attention) beyond human capacity.
The second limitation is that previous models including SQAIR lack any form of background mod- cling and thus only cope with scenes without background, whereas natural scenes usually have a dynamic background. Thus, a temporal generative model that can deal with dynamic backgrounds along with many foreground objects is an important step toward natural video scene understanding.
In this paper, we propose model called SCALable Sequential Object-Oriented Representation (SCALOR). SCALOR resolves the aforementioned key limitations and hence can model complex videos with several tens of moving objects along with dynamic backgrounds, eventually making the model applicable to natural videos. In SCALOR, we achieve scalability with respect to the object density by parallelizing both the propagation and discovery processes, reducing the time complexity of processing each image from ((N) to (1)), with N being the number of objects in an image. We also observe that the sequential object processing in SQAIR, which is based on an RNN, not only increases the computation time but also deteriorates discovery performance. To this end, we propose a parallel discovery model with superior discovery capacity and performance. SCALOR can also be regarded as generative tracking model since it not only detects object trajectories but is also able to predict trajectories into the future. In our experiments, we demonstrate that SCALOR can model videos with nearly one hundred moving objects along with dynamic background on synthetic datasets. Furthermore, we showcase the ability of SCALOR to operate on natural-scene videos containing tens of objects with dynamic background.
The contributions of this work are: (i) We propose the SCALOR model, which significantly im- proves (two orders of magnitude) the scalability in terms of object density. It is applicable nearly a hundred objects while providing more efficient computation time than SQAIR. (ii) We propose parallelizing the propagation- -ippgattionncscooveyyy process by introducing the propose-reyy rejeect model, reducing the time complexity to (1). (iii) SCALOR can model scenes with a dynamic background. (iv) SCALOR is the first probabilistic model demonstrating its working on a significantly more complex task, i.e., natural scenes containing tens of objects as well as background.
PRELIMINARIES SEQUENTIAL ATTEND INFER REPEAT (SQAIR)
SQAIR models a sequence of images x = X1:T by assuming that observation x, at time is gener- hnEo, with O the set of objects present at time t. Latent variable nr corresponding to object consists of three factors (z ziyhere Li,n2 2 which represent the existence, pose, and appearance the object, respectively. SQAIR also assumes that an object can disappear or be introduced in the middle of a sequence. To model this, it introduces the propagation-discoverr model. In propagation, a subset of currently existing objects is propagated to the next time-step and those not propagated (e.g., moved out of the scene) are deleted. In discov- ery, after deciding how many objects D, will be discovered, Dr objects are newly introduced into the scene. Combining the propagated set P, and discovered set Dr, we obtain the set of currently existing objects O., The overall process can be formalized as:
ated from set of object latent variables 2
T
p(xI.T, DI.)) = = DDD, xP) II P( 21 1 12(1))
(1)
For SQAIR, we use 2,5 ("fgg standing for foreground) to denote 2, because SQAIR does not have any latent variables for background. Due to the intractable posterior, SQAIR is trained through variational inference with the following posterior approximation:
2 II a(r, 2
X1, 2
II neO...
g(2t, 1,, Xgx)
SQAIR is trained using the emprrtance-weighted autoencoder (IWAE) objective (Burda et al. 2015). The VIMCO estimator (Mnih & Rezende, 2016) is used to backpropagate through the discrete ran- dom variables while using the reparameterization trick (Kingma & Welling 2013 Williams 1992) for continuous variables. SQAIR has two main limitations in terms of scalability. First, for prop- agation, SQAIR relies on an RNN, which sequentially processes each object by conditioning on previously processed objects. Second, the discovery is also sequential because it uses RNN-based
2

---

Published as a conference paper at ICLR 2020
Consequently, SQAIR has time complexity of O(1O,l) per step t. In Crawford & Pineau (2019b), the authors demonstrated that this sequential discovery can easily fall beyond the scale of few objects. Moreover, SQAIR lacks any model for the background and its temporal transitions, which important in modeling natural scenes.
discovery based AIR (Eslami al., 2016).
SCALOR
In this section, we describe the proposed model, SCALOR. We first describe the generative process along with the proposal-rejection mechanism, which is designed to prevent propagation collapse, and then the inference process and learning.
3.1 GENERATIVE PROCESS
SCALOR assumes that an image x, generated by background latent 2 and foreground latent The foreground is further factorized into a set of object representations z = {zen) In SCALOR, we represent an object by zt,n = (=f,n zi,n whhatt similarly to SQAIR. The appearance representation z,,re is continuous vector representation, and zt,n
where is further decomposed into
the center position zi, scale zt,n and depth =t.m The depth representation, which is missing in SQAIR, represents the relative depth between objects from the camera viewpoint. This depth modeling helps deal with object occlusion. The foreground mask m,,, obtained from zt.n is used to distinguish background and foreground. We adopt the propagation-discoveee but improve it in such a way to resolve the scalability problem. The generative process of SCALOR can be written as:
what
model from SQAIR,
T p (z) p(z,
P(XIT, p(z) (z)
Il rendering background transition
p(z
p(zz (2Z))
(3)
discovery
propagation
where Z,
(zt bg 2 As shown, the generation process is decomposed into four modules: (i)
propagation, (ii) discovery, (iii) background transition, and (iv) rendering.
Propagation. The propagation in SCALOR is modeled as follows:
piz [z<()== = II
mes p( z<en)) {p(z),
where (zee,)) p(zi, what Z z2))) St.
(4)
nEO.
where p(= ()) ) a Bernoulli distribution with parameter Se,.. The distributions of "what" and "where" are defined only when the object is propagated. To implement this, for each object we assign an object-tracker RNN denoted by its hidden state h The RNN is updated by input 2,,n for all f where the object n is present in the scene. The parameter Be,x is obtained as 8t,n fulp (h,,)) If 2 0, the object is not propagated and the tracker RNN is deleted. Importantly, unlike the RNN-based sequential propagation SQAIR, the propagation in SCALOR is fully parallel.
Discovery by Proposal-Rejection spect to the the number of objects is our new discovery model that consists of two phases: proposal and rejection. In the proposal phase, assume that the target image can be covered by H x W latent grid cells, and we propose an object latent variable hr,h,w per grid cell. This proposal phase can be written as:
The main contribution in making model scalable with re-
HW II h,w=1
HW II R,w=1
p(
22
Epres
where |z,) ( (p(zt,h, (z(") p wwaat 2.,, 12?
(5)
In the rejection phase, our goal is to reject some of the proposed objects if a proposed object largely overlaps with a propagated object. We realize this by using the mask variable mt,n- Specifically, if the overlapping area between the mask of proposed object and that of propagated object is over a threshold т. we reject the proposed object. This procedure can be described as (i) pro- posal: zP p(Z xP) and (ii) accept-reject: z faweep ( z -). In this way, the final discovery set z is always subset of the proposal set z,, i.c. z, C zP. Although
deterministic function for the rejection, it can be design choice to implement this as a stochas- tic decision. While rationale behind this design is reflect inductive bias of Gestalt
3

---

Published as a conference paper at ICLR 2020
Ireker INh0
Encecay
Propagatin Mask
Decoddrr
Accepe lost
Input
Mask
Ouee
Reconstrucion
Proposal
Rejection Mask
Accepted Proposal
Figure SCALOR inference procedure: (A) Proposal, Accept-Reject, (C) Propagation, (D) Background Module and Rendering Process. (A) The proposal module takes the input image and propagation mask and combines them to make the proposal representation. is generated, and then compared to the propagation mask to make accept-reject decision. Only the accepted proposals considered discovered objects. (C) The tracker RNNs decide what and where propagate after looking at the imput image. The gray boxes represent what not propagated. (D) Given inferred foreground objects and the input image, the background module infers the background representation. The rendering process combines the foreground and background representations according to the foreground mask assignment
(B) From the proposal representation, the proposal mask
principle saying that two objects cannot coexist in the same position, we shall also see later fur- ther reasons to why this design is effective. The final discovery model can then be written as: p(zple [2]") = p(z? [z I1 p(z t,,,0 1 ,P,,), r), where the acceptance model is:
pl 22,22, .)
(P))) [ ( 1 2P ()p(p((()))) (p(2 where (,,,,, F1,h. (on.mpp(pit.. p( ".
Secept what Jezepe
(6)
Background Transition. Unlike SQAIR, SCALOR is endowed with background model. The background image is encoded into D D-dimension continuous vector z, from the background tran- sition p(2) be 22t: (.. The background RNN encodes the temporal transition background images.
Rendering. The implementation of the rendering process is the same in SPAIR (Crawford & Pincau 2019b), except that we process the objects in parallel. Implementation details are shown in Appendix A
3.2 LEARNING AND INFERENCE
Due to the intractability of the true posterior distribution p(Z)r(XiTT), we train our model using variational inference with the following posterior approximation:
T
T
((21:T (1:T) = II
q() Rect x5t) = II t=1
(( 2 xx) g(2? 12 x5t) ((2 [ee,,
(7)
1=1
Posterior Propagation. (z, |acee Xxee) ) is similar to the propagation in generation, except that we now provide observation Xst through an RNN encoding. Here, the propagation for each object n is done by (Z.tt,,, using attention ar,, fau(Xet) on the feature map for object To compute the attention, we use the previous position 2 as the center position and extract half width and height of the convolutional feature map using bilinear interpolation. This attention mechanism is motivated by the observation that only part of the image contains information for tracking an object and an inductive bias that objects cannot move a large distance within a short time span (i.c., objects do not teleport).
Posterior Discovery. The posterior discovery also consists of proposal and rejection phases. The main difference is that the observations X i.c. q(, |z? xxx) "u q( |z?" x<t). Here, the observation Xxt is encoded into the feature map of dimensionality H x W x using Convolutional LSTM (Xingjian) et al. 2015). Then, from each feature we obtain 2 Importantly, this is done over all the feature cells h, w in parallel. A similar approach is used in SPAIR (Crawford & Pineau (019b), but infers the object latent representations sequentially and thus is difficult scale large number
compute the proposal in spatially parallll manner by conditioning
4

---

Published as a conference paper at ICLR 2020
of objects (Lin et al., 2020). Even if this ceatially-parallee proposal plays key role in making model scalable, we also observe another challenge due to this high capacity of the discovery module. The problem is that the discovery module tends to dominate the propagation module and thus most of the objects in an image are explained by the discovery module, i.e., objects are rediscovered at every time-step while nothing is propagated. We call this problem propagation collapse.
Why would the model tend to explain an image through discovery while suppressing propagation? First, the model does not care where-either from discovery or propagation-an object is sourced from as long as it can make an accurate reconstruction. Second, the propagation step performs a much harder task than the discovery. For the propagation to properly predict, tracker needs to learn to find the matching object from an image containing many objects. Although the propaga- tion attention plays an important role in balancing the discovery and propagation, we found that does not eliminate the problem of re-discovery, and without rejection, its effectiveness varies across different experiment settings. On the contrary, the discovery module does not need to solve such a difficult association problem because it only performs local image-to-latents encoding without as- sociating latents of the previous time-step. Therefore, it is much easier for the discovery encoder to produce latents that more accurate than those inferred from propagation. If we limit the pacity of the discovery module and sequentially process objects like in SQAIR, we may mitigate this problem because the propagation module is naturally encouraged to explain what this weakened discovery module cannot. This approach, however, cannot scale.
We employ techniques resolve the aforementioned problem. First, we simply bias the initial network parameter so that it has a high propagation probability at the beginning of the training. This helps the model prefer to explain the observation first through propagation. The second technique is our proposal-rejection mechanism, which is implemented the same way as in the generation process. This prevents the discovery model from redundantly explaining what is already explained by the propagation module. The posterior for the discovery model can be written as:
HW II
(Z x. x(t) 2(2p [2? xsi)
Paccept 2 12 2..).
(8)
h,w=1
where the acceptance model is Paccept hz / zP))
p( lz, ZP)) (PP)p (p(z),,,n )pppppppp(ppp (h,,,))))) Fi.h... h,
Posterior Background. The posterior of the background q(z |2 x,) is conditioned the input image and currently existing objects. Here, we provide the foreground latents so that the remaining parts in the image can be explained by the background module.
Training. We train our model by maximizing the following evidence lower bound L(0, o)
T
Σ Eg (anttacelxcc (ego( [log Pe Pe (xx| [x)] KL [go Zer,, Xge) || pa (Ze [c)i]]
(9)
We use the reparameterization trick (Williams) 1992 Kingma & Welling 2013) for continuous ran- dom variables such as wha and the Gumbel-Softmax trick (Jang al. 2016) for discrete variables such as p We found that our proposed model works well and robustly with these simpler training methods than what is used in SQAIR, i.c.. VIMCO and IWAE.
4 RELATED WORK
Different approaches have been taken to tackle the problem of unsupervised scene representation leamiigg Object-oriented models such as AIR (Eslami et al., (0166) and SPAIR (Crawford & Pineau 2019b) decompose scenes into latent variables representing the appearance, position, and size of the underfying objects. While AIR makes use of recurrent neural network, SPAIR applies spatially invariant attention to extract local feature maps. Although the latter provides better scalability than AIR, it is still limited as it performs sequential inference on objects. SQAIR (Kosiorek et al.) 2018), discussed in Section 2, extends the ideas proposed in AIR to temporal sequences. On the other hand, scene- mixture models (Greff et al., 2017 Van Steenkiste et al. 2018 Burgess et al. 2019 scene-mixture
Greff al. 2019 Engelcke et al. 2019) decompose scenes into collection of components, each being a full-image level representation. Although such models allow decomposition of the input image into components, they are not object-wise disentangled multiple objects can be in the same
5

---

Published as a conference paper at ICLR 2020
Tracking and Detection Accuracy
Counting Reconstruction Error
0.3
MSE
CountMAe
CountMAE MSE
Accuracy
0.7
SGMOTA 5Q-MOTA
0.5
PraPiiiio SQ--Peccion
Recol SQ-aecall
0.2
VED(--)
Li(0-111
MD(18-24)
H1ap0-00 VHD290013))
ID(5333
(a) Dataset Density
(b) Dataset Density
Figure 2: Quantitative result showing superior performance of SCALOR (SC) compared SQAIR (SQ). (a) Tracking Accuracy (b) Object Count and Reconstruction Error
component. Furthermore the obtained representation does not contain explicit interpretable features like position and scale, etc. SPACE (Lin et al.) (020) combines both of the above approaches by using object detection for foreground and mixture decomposition for background. improves upon SPAIR by parallelizing the latent inference process.
DDPAE (Hsieh et al. 2018) is another object-oriented sequential generative model that models each object with an appearance and a position vector. The model assumes the appearance of an object to be fixed, and thus shares the content vector across different time-steps. NEM (Greff et al.) 2017) and RNEM (Van Steenkiste et al. 2018) introduce spatial mixture model to disentangle the scene into multiple components representing each entity. Since each component generates a full scene image, the latent representations not interpretable. Tracking-B;; By-Animation (He et al. 2019) introduces a deterministic model to tackle the task of object tracking in an unsupervised fashion. Furthermore, there is substantial amount of work on object tracking from the computer vision community using the same "bounding box" box""-eerre represeentation approach proposed in SCALOR (Kosiorek et al., 2017; Ning et 2017 Nam & Han [016]] Tao al.] (016)). but such methods use provided labels to tackle the problem of object tracking and thus are usually fully or semi-supervised and not probabilistic object-oriented models.
We also note that Crawford & Pineau (2019a) has independently and concurrently developed similar architecture to ours. This model also emphasizes the scalability problem with a similar idea motivated by parallelizing SPAIR and extending it to sequential modeling. The main differences are the usage of the proposal-rejection mechanism and the background modeling that make our model work complex natural scenes.
EXPERIMENTS
In this section, we describe the experiments conducted to empirically evaluate the performance of SCALOR. We propose two tasks, (i) synthetic MNIST/dSprites shapes and (ii) natural-scene CCTV footage of walking pedestrians. We will show SCALOR's abilities to detect and track objects, to generate future trajectories, and to generalize to unseen settings. Furthermore, we provide quantitative comparison to state-of-the-art baselines.
5.1 TASK 1: LARGE-SCALE MNIST AND DSPRITES SHAPES
We first evaluate our model on datasets of moving dSprites shapes as well as moving MNIST digits. In all experiments, the image sequence covers 64 64 partial view of the center of the whole environment. Therefore, while there is fixed number of objects in the environment at each time- step, only a subset of them are visible in the observed image. The environment size is customized for each setting in a way that objects can conveniently move out of the viewpoint completely and re-enter within a few time-steps. We test on five different scale settings. In each setting, the number of objects in each trajectory is sampled uniformly from an interval [min, max). Each scale setting is specified with a triplet (min, avg. max), where min and max are as mentioned and avg represents the average number of visible objects in the trajectories in that setting. The five settings to as Very Low Density (VLD) [(2, 2.9, 4)], Low Density (LD) [(8, 8, 11)], Medium Density (MD) [(18, 20, 24)], High Density (HD) [(50, 55, 64)] and Very High Density (VHD) ((90, 90, 110).. For
referred

---

Published as a conference paper at ICLR 2020
2 100
m
(a)
(b)
5 8 0
5
0
T
5
7
0
6
5
8 8
0
C 5 3
0 8 7 4
T
8
33 4
/ O
4
3
/
4
4
(c)
9
0
0
4
0
4
4
0
4
O
0
4
400 3
O
3
3
4
0
3
Qualitative results of SCALOR for Moving dSprites and Moving MNIST (HD) tasks: a) Inferred bounding boxes superimposed on the original image sequence. White circles indicate discovery at that timestep, b) Reconstructed sequence, c) Per-object reconstruction
Figure 3:
(a)
17
(b)
Figure 4: Qualitative samples of tracking on Moving dSprites task with dynamic background: a) Original image sequence with inferred bounding boxes, b) Reconstructed sequence
example, in the MD setting, there are always between 18 to 25 objects in the overall environment, while only about 20 of them are visible average in each frame.
Experiment - Tracking Performance. This experiment evaluates the tracking performance in an environment without background. We the following tracking metrics: Multi-Object Track- ing Accuracy (MOTA), precision-recall of the inferred bounding boxes (Bernardin & Stiefelhagen 2008 reconstruction mean squared error, and normalized counting mean absolute error (Count- MAE). CountMAE measures the difference between the number of predicted objects and the actual number of objects, normalized by the latter. MOTA measures the tracking accuracy and consistency of the bounding boxes. Precision-recall measures the number of calse-positive and negatives, re- gardless of the associated IDs. For computing the MOT metrics, we choose the Fuclidean distance threshold to be twice the actual object size in each setting. Figure[]shows the quantitative results of SCALOR compared to baselines. More detailed quantitative results are given in Appendix
We compare the performance of the proposed model with SQAIR in the VLD setting for both MNIST and dSprites datasets, and LD setting for dSprites. Note that we were not able to make SQAIR work other settings due to the high object density. As shown in Figure 2 SCALOR outperforms SQAIR in all these settings, obtaining significantly higher accuracy and recall. SQAIR either misses some objects or misidentifies distinct objects as one. In addition, SCALOR has lower values of CountMAE in comparison to SQAIR, showing that SCALOR can infer the number of objects in the scene more accurately. Furthermore, we observe that increasing the number of ob- jects in the scene does not significantly impede SCALOR's 'ss tracking ability, which demonstrates the strength of SCALOR when applied to images with a high number of objects. SCALOR can achieve relatively high precision-recall even for scenes containing about 100 objects. Note that in the VHD case, the number of objects (about 90) in the first time-step exceeds the number of detection grid cells (8x8) 8) the discovery model has. Thus, the model can only detect up to 64 objects at the first time-step, and detects the rest in the following time-steps. This results in lower performance on the tracking and detection accuracy in the VHD case. This is demonstrated in Figure 10 in Appendix C
Figure 3 demonstrates the qualitative performance of SCALOR on dSprites and MNIST (HD). To clarify tracking consistency, bounding boxes with distinct ids are represented by distinct colors. Discovered objects are emphasized by white circles. As shown in Figure 3 ()), the discovery module of SCALOR can identify newly introduced objects and put them in the propagation list while the propagation module keeps tracking existing objects. Figure 3))) shows object- rendering of inferred z that latent variables. For clear visualization, object-wise rendering is shown only for
7

---

Published as a conference paper at ICLR 2020
(b)
Figure 5: Conditional generation. The first 5 frames provided the model while the next 5 gener- ated. The red line indicates where generation starts from. a) Image reconstructiodggenraaiin.. byDynamic background inference/generation
(a)
(b)
Figure 6: Generation from scratch. Objects are sampled from discovery prior and propagated in the next time frames. a) Generated sequence. b) Generated dynamic background
subset of objects present in all the time-steps. SCALOR infers consistent object-wise representation even when objects are largely occluded.
Experiment 2 Dynamic Background. Another interesting yet challenging setting is the presence of dynamic background. As we can see Figure 4 SCALOR can decompose the video sequence to set of foreground objects well as dynamic background. While the background module can model the background dynamics, can also model the dynamics of the whole environment including the moving objects. This usually brings competition between background module and foreground background module and foreground mod-
module on explaining the foreground part. However, ule cooperate properly in SCALOR. We believe this is because of our modeling that the background module obtains information from the foreground part by conditioning on the foreground latent vari- ables. Table T also provides the tracking performance of the model for this setting to show how complex images affect the tracking quality. We observe that achieves comparable performance to the default no-background setting. These results provided in Table Table[under Junder "SCALOR BG".
Experiment Future Time Prediction//Generation.. This section showcases the generation abil- ity of SCALOR via two experiments. The first experiment aimed at showing conditional gen- eration. Here, the model provided with the first 5 frames, and then tested to generate the next frames. Latent variables at each time-step are sampled independently from the prior distribution of the propagation step conditioned on latent variables from previous time-step. Because the focus on conditional generation of background and objects, discovery prior of zrre is manually set to zero. Figure 5 shows generated sample. The first 5 frames (before the red line) correspond to reconstruction while the next 5 frames represent conditional generation]] As we can see in Figure 5 SCALOR can generate dynamic background and object trajectories that are reasonably consistent.
The second experiment showcases video generation from scratch. In this setting, the model gener- ates all latent variables from the discovery prior at the first time-step. For the following time-steps, latent variables are sampled by conditioning on the latent variables of the previous time-steps. The the object generation probability for each grid cell at the first time-step, i.c. z is set to 0.2. Figure 6 shows generated sample. At the first time-step where all variables the background generated reasonably, but the objects are generated only partially, This behavior expected because newly introduced objects mostly partially observed at image boundaries. This makes the inference network learn a z what representation corresponding to the partial views. Furthermore, since the z what prior is standard Gaussian, it is hard to model the multi-modal object appearance. Yet, it is interesting to see that the conditional generation of each object's z what grad- ually converges to a consistent complete view of an object. This demonstrates that in our model,
sampled from a fixed prior,
Exammless with longer duration the project website
8

---

Published as a conference paper at ICLR 2020
Figure 7: SCALOR's performance on Grand Central Station Dataset. (a) input sequence, (b) overall recon- struction of objects and background, (c) reconstruction of the extracted background, (d) segmentation for each object, colors indicate tracking ID, (e) extracted object trajectories
the distribution of what in propagation can actually achieve multi-modality when conditioned on 2:2
what from the first time-step. Furthermore, it shows that the propagation prior network is capable of maintaining consistent representation of the object along the sequence as is reflected in the dataset. Therefore, SCALOR is capable of generating objects with independent appearance and behavior.
5.2 TASK 2: REAL- WORLD SCENARIO
This section considers the performance of SCALOR on real-world natural video, which previous models have not been able handle due to the scalability issue and lack of background model. Compared to synthetic data, the challenges in this setting are significantly more difficult. Specifi- cally, we consider the Crowded Grand Central Station dataset (Zhou et al. 2012), which was col- lected from CCTV cameras at New York City's Grand Central Station. Due to the complex pedes- trian behavior, the density of the dataset can be considered a mix of LD and HD.
Figure 7 shows the tracking result on sample sequence. We can see that SCALOR performs rea- sonably well on this pedestrian tracking dataset by maintaining consistent temporal trajectories. As shown in Figure (]c), the background module infers the background component and reconstructs the extracted background correctly. As for object detection, SCALOR succeeds in accurate pedes- trian detection and tracking. Furthermore, the foreground mask produced by whaa provides the segmentation of each individual pedestrian, as shown Figure d). We draw tracking trajectories in Figure e for each pedestrian in the natural scene. Trajectories in different colors correspond to different pedestrian ids inferred by the identity of the latent variable of each object. Additional figures and the dataset details are provided in Appendix D
Figure 14 in Appendix D shows the conditional generation. The first frames are the inference reconstruction while the last 5 frames are model generation. Starting from the 6th frame, the latent transition of the propagation trackers is modeled by the sequential prior network. In the generation process, a different prior of zpres is introduced in the discovery module to introduce new objects emerging in the scene (see Appendix D for more details). As shown in the figure, the model tends generate movement aligned with the previous frames. This also applies newly generated ob- jects from the discovery phase as shown in Figure [4) Although the trajectories are consistent, the appearance of generated objects fails to maintain its consistency across different time frames. Noticeably, in the generated sequence the segmentation mask for each object tends to deform into different shape. This may stem from imperfections during the inference that makes the learning of appearance transition difficult. Additional figures for generation are provided in Appendix D
The ground truth trajectories of the Grand Central Station Dataset were not available when this ex- periment was conducted. We instead use negative log-likelihood (NLL) to compare our model with two baselines, a sequential VAE and a Recurrent Latent Variable Model (VRNN). The sequential VAE has one latent variable of dimensionality 64, and sequential prior p(z,) (Z<)). It is similar to our background module with the same number of latent variables for encoder and decoder. As for the VRNN, we implement VRNN with 128 dimensions of the L.STM hidden state and latent variable

---

Published as a conference paper at ICLR 2020
-10-
2.5
Number discovery
Inference Lasency (In
SCALOR SQAIR
Mcan Square Error
32
0.8
OR Astention Rejection
77
0.7
1.5
2.4
Propagatoo
0.6
55
1.6
0.5
0.4
0.5
0.8
0.3
0.2
20
40
20
30
Training iteration (k)
Training iteration (k)
Number of Objects
Training Time (in Hours)
(a)
(b)
(c)
(d)
Figure 8: Attention-Rejeet Rejection Ablation study and computational efficiency comparison. (a) Propagation rate, (b) Number of discovered objects, (c) Inference time and, (d) Training convergence time
and choose a convolutional network as the image encoder and decoder. The NL. value for our model 28.30, and for the VAE and VRNN baseline it 27.59 and 27.79 respectively. While learning a highly structured representation, SCALOR can still obtain a comparable generation quality.
5.3 ABLATION STUDY AND COMPUTATIONAL EFFICIENCY COMPARISON
Ablation Study. We perform ablation study on the rejection mechanism and the propagation attention mechanism. Here, we train our model on the moving dSprites dataset where the number of objects varies from 6 to 36. We use two metries evaluate different architectures. The first metric is the propagation rate, which measures how long an object is propagated. Since, by design, objects always stay in a scene for more than one frame, the propagation rate can be regarded as a proxy to measure the success rate of tracking through propagation. The second metric counts the number of discoveries that occurred in the whole sequence. If the discovery module dominates, a high value will be observed. As we can see in Figure 8 (a), with no rejection (Rej) or attention (Att) in propagation, the propagation rate goes down to 30% in early training iterations and keeps decreasing as the training progresses. In this setting, we found that the model re-discovers objects in every frame without tracking them properly. This is also shown in Figure (b), where it has a significantly larger number of discoveries. With the propagation attention mechanism, the propagation rate increases to 95%. This is because the attention mechanism reduces the cost of finding the matching object between time-steps in propagation. The model thus favors tracking in propagation over re-discovery. It also prevents the discovery module from propagation collapse. However, as we can see in Figure [ (b), the re-discovery problem still exists in this setting. The rejection mechanism, however, makes the propagation rate converge to I even without propagation attention. The propagation attention together with the rejection produced more accurate boxes.
Computational Efficiency. In Figure 8 (c) and (d), we measure the inference time (c), and training convergence time (d). For measuring the inference time, we consider a hypothetical situation with N' objects in the first frame. The model is desired to discover and propagate them accordingly. Furthermore, we set the discovery capacity SQAIR to the same as the number of discovery grid cells in SCALOR. Figure ((c) demonstrates how much time one forward step takes, as the number of objects/discovery capacity increases from 4 to 64. SQAIR's s speed decreases linearly as both its discovery and propagation mechanisms process objects sequentially. In contrast, SCALOR does not suffer from such phenomena as discovery and propagation are done in parallel. Figure (d) shows MSE convergence vs. training time when both models trained on the MNIST VLD setting. SCALOR converges to a lower MSE than SQAIR and does so orders of magnitude faster.
6 CONCLUSION
We introduce SCALOR, a probabilistic generative world model aimed at visually modeling an environment of crowded dynamic objects. With the proposed parallel proposal-rejection mechanism, we improve the capacity of object density from few objects to up to hundred objects. Unlike previous models, the proposed model can also deal with dynamic back- grounds. These contributions consequently makes our model applicable, for the first time in this line of research, to natural scenes. An interesting future direction to introduce more structure to the background and to model interactions among objects and the background.
and
ACKNOWLEDGMENTS
SA thanks Kakao Brain and Center for Super Intelligence (CSI) for their support.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: the clear mot metrics. Journal on Image and Video Processing, 2008:1, 2008.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arviv: XXV9:/509.000 00559, 2015.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint /901. 2019.
Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. arXiv preprint arXivv1911.09033 2019a.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In Proceedings of AAAI, 2019b.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations, 2019.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2017.
Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran,
Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-object representation learning with iterative variational inference. arXiv preprint arXiv:1903. 1ete10093300000 2019.
Zhen He, Jian Li, Daxue Liu, Hangen He, and David Barber. Tracking by animation: Unsupervised learning of multi-object attentive trackers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1318-1327, 2019.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning de- compose and disentangle representations for video prediction. In Advances Neural Information Processing Systems, pp. 517-526. 2018.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzatio with gumbel-softmax. arXiv preprint arXiv: /6/ /. 1.01 144, 2016.
Diederik Kingma and Max Welling. arXiv:1312.6114.00 2013.
Auto-encoding variational bayes.
arXiv preprint
Adam Kosiorek, Alex Bewley, and Ingmar Posner. Hierarchical attentive recurrent tracking. In Advances in Neural Information Processing Systems, pp. 3053-3061, 2017.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems, pp. 1616,8616, 2018.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang. and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020. 2020..
Andriy Mnih and Danilo Rezende. Variational inference for monte carlo objectives. arXiv preprint arXiv: 1602.06725. 2016.
Hyeonseob Nam and Bohyung Han. Learning multi-domaii convolutional neural networks for vi- sual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4293 4302, 2016.
11

---

Published as a conference paper at ICLR 2020
Guanghan Ning, Zhi Zhang, Chen Huang, Xiaobo Ren, Haohong Wang. Canhui Cai, and Zhihai He. Spatially supervised recurrent convolutional neural networks for visual object tracking. In 2017 IEEE International Symposium Circuits and Systems (ISCAS), pp. 1-4. IEEE, 2017.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874-1883, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354 2017.
Ran Tao, Efstratios Gavves, and Aroold WM Smeulders. Siamese instance search for tracking. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1420-1429, 2016.
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural ex- pectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint 1rXiv 10353, 2018.
Ronald J Williams.
Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learming, 8(3-4)2229-25 256, 1992.
SHI Xingjian, Zhourong Chen, Hao Wang, Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting, In Ad- vances neural information processing systems, pp. 802-810, 2015.
Bolei Zhou, Xiaogang Wang, and Xiaoou Tang. Understanding collective crowd behaviors: Learn- ing mixture model of dynamic pedestrian-agent In 2012 IEEE Conference Computer Vision and Pattern Recognition, pp. 2871-2878. IEEE, 2012.
12

---

Published as a conference paper at ICLR 2020
A ALGORITHMS
Algorithm 1: Discovery
Rejection Inference
Input: X14 image sequence up to current time-step
M. propagated mask from current time-step rejection hyper-parameter
Feature encoding
img
(x)
7 gmask Faskenno (M,)
Concat(e, img e
Proposal. Troposal-Reeess ReessllRRejeeiion (done in parallel)
AcceptedList, = for h 1 to do for +- to W do Objects proposal
=pres "t,h.w Bern( f HES (e),).)))
if "t,, = 0 then
gPres
continue
end
7. whbere
(t.h.w (X zwheere h.
Attended Gl Glimpse
St.hww
2
Swhat
6,0,00 A(((|GlimppeEne he
M4A,1 STN SIS-STT--AAA (GlimpseDcc22yhh. Zuuu wwhhere
Ot,,,w
(,h,w) k,
Test
8 A(m nM?
11 A pixel area
A(m
if 8 < then
Add(Z Whaat Acceptedl.ist Add t, L.h, 2t,,,w)
end end end
Output: AcceptedList,
13

---

Published as a conference paper at ICLR 2020
Algorithm 2: Propagation Inference
Input: X1#: image sequence up current time-step
PropList,-1 = ( 1 1xeo..... latent variable from previous time-step
/* Feature encoding
img (
ez
/* Object tracking (done in parallel)
/
PropList, PropList,
for to O--1 do
eft |nn (STN(e img ( sccalo h GRU(Snn (let, what 4 Il, 1-1, 1-1,n h1-1,)
((-1,n)) // Feature map attention pos
scale pres
Ctn fun ([c?.. 24 I,nt 2 L 24 hz,zl)
pos
scale
2 t, scale (.I. rwhere
(.lhwherre ((()))
att
(x (2 zt.n)) ))
11 Attented Climpse
what
N(c/GlimpseEnc geth
t.
depth
(-ISnnnn (e what ht,n))
tn
iin
2nn ~ Bern( f-Immm (z) 2,.0 2 ht,)))
pres
2 where
( in 2t,
scaal depthy
pres In
= then PropList,, Update (z,, 2 2. poss
else ,n
end end
Output: PropList,
Algorithm 3: Background Module and Rendering
Input: x, image at current time-step,
Ineo, object RGB glimpses
m, = [m,,xnee object masks glimpses.
[ 2,,, pres Ineo, object latents for position and presence
Foreground object rendering (done in parallel)
for n -- 1 to O do
x STN (or,ns (z,os zzcale)
Ye. = STN (m,, =n, a-
pres
depth ) (2,00 2oalee)
Ytn = normalize (Ye, Vn)
end
x EnEo, x Vt.
Foreground mask rendering
for t 1 to O do
M., STN (miny (2 sqale)
end
M. = min(E Mtn, 1)
/* Background rendering
bg = BackgroundFnooder(CCnaa... (1-M)))
2 b [h (e²²))
bg BackgroundDeecader(22))
X,
/* Foreground background combination
X, X (1 -0,M o x
Output: X²
14

---

Published as a conference paper at ICLR 2020
B ADDITIONAL QUANTITATIVE RESULT
Table Jincludes Multi Object Tracking Accuracy (MOTA) as well as precision-recall of the inferred bounding boxes (Bernardin & Stiefelhagen, 2008).
Table provides a comparison of SCALOR to SQAIR and VRNN in terms of the reconstruction
error (MSE) and negative liliiiiihoodd (NLL). NLL is computed per pixel across the whole quence.
Experimental Setting
MOTA
Precision
Recall
MAE
SCALOR VHD
84.6%
96.8%
88.1% 1%
0.091
dSprites
SCALOR HD
92.8%
97.6%
95,4%
0.033
SCALOR MD
93.5%
97.8%
95.9%
0.041
SCALOR LLD
92.5%
97.9%
94.9%
0.059
SCALOR - VLD
93.8%
96.3%
98.9%
0.067
SQAIR LD
20.2%
97.6%
64.1%
0.335
SQAIR VLD
72.5%
98.4%
75.4%
0,239
SCALOR MD
86.9%
99.0%
87.9%
0.113
MNIST
SCALOR LD
85.9%
99.0%
87.1%
0.124
SCALOR VLD
94.8%
97.7%
98.8%
0.040
SQAIR VLD
78.5%
99.9%
78.8%
0.214
SCALOR - BG
91.9%
97.2%
95.3%
0.034
SCALOR LG
92.3%
97.0%
95.7%
0.032
Table 1: Quantitative results of SCALOR for different experimental settings. "SCALOR BG" refers to the dynamic background task for Experiment "SCALOR refer to the length generalization experiment in Section
dSprites
MNIST
Method
VLD
LD
MD
HD
VLD
LD
MD
SCALOR
22.16
22.41
22.56
22.88
7.45
7.54
7.82
MSE NLL
VRNN
22.33
22.80
23.28
23.30
7.62
7.70
7.71
SQAIR
22.63
40.50
7.75
SCALOR
0.0010
0.0030
0.0024
0.0015
0.0029
0.0026
0.0010
VRNN
0.0018
0.0071
0.0105
0.0190
0.0022
0.0054
0.0035
SQAIR
0.0135
0.0918
0.0084
Table Negative Log Likelihood and Mean Squared Error SCALOR ys. baselines
15

---

Published as a conference paper at ICLR 2020
C ADDITIONAL EXPERIMENTS ON DSPRITES AND MNIST
C.I FREQUENT DENSE DISCOVERY
Figure 9: Frequent Dense Discovery experiment: a) First row: inferred bounding boxes superimposed on the original sequence. b) Second row: discovery bounding boxes. c) Third row: discovery reconstruction. d) Last row: propagation reconstruction
This experiment evaluates the ability to discover many newly introduced objects across time-steps. This is important because in many applications only key-frames of a video are available, i.e., frames at which significant changes happen. An example of a key-frame is a sudden change in the observer's viewpoint. This change of viewpoint introduces many new objects in the frame. Figure Saaure/ssows one such instance. In this setting, 10-15 objects are introduced at the first, fifth, and ninth time-steps, respectively. As is shown, SCALOR is able to discover many new objects in each frame.
C.2 VERY HIGH DENSITY
Figure 10: Sample from Very High Density setting: a) First row: inferred bounding boxes, b) Second row: overall reconstruction, e) Third row: discovery bounding boxes, d) Fourth row: discovery reconstruction, d) Fifth row; propagation bounding boxes, Sixth row: propagation reconstruction
16

---

Published as a conference paper at ICLR 2020
shows samples from the Very High Density experiment. This experiment places 90-110 objects in the overall environment, around 90 of which are visible at every time-step on average. The discovery module contains 8 x 8 detection grid cells and thus can only detect up 64 objects. Interestingly, as is shown, the model will identify as many objects as possible in the first time-step and discover the remaining in the second time-step. Furthermore, if too many objects are densely packed in one local region of the space, SCALOR will perform similarly detecting them through multiple frames.
Figure 10
C.3 ABILITY TO HANDLE OVERLAP AND OCCLUSION
10
HH d 8
3
1/
87
g Fin
m1
m
1
4
E
Am
20
D
F
En
2
Eg
8 8818
PP
M 2
R 2 7
2 Eg
mi 2
3/4 2
Pix Ti
2
Pig.
Fr
8M
PM
wA
P Rw
F vah
E by
B
8
x
#
3
5
ng Ey
3 100
Th F
30 e L
S4
SP
Jully ton
Dis let
1 m
A m
1:
F
Pp.
Figure 11: Highly occluded setting: Sample sequences in which objects move towards each other move more aggressively so overlap and occlusion happens more frequently
Figure 11 shows sample of the MNIST settings where objects move towards each other more aggressively. In this setting, there is a higher chance of overlapping and occlusion. As shown, the identity of the objects is preserved when objects overlap with each other.
C.4 GENERALIZATION TEST
14F
M
T:
Ix
5u:
MB
MANT
90
Nlow
mmg
2m
2'
2
Ma
IM
Figure 12: Generalization with respect longer sequences. a) First row: bounding boxes for the first 10
time-steps. b) Second row: bounding boxes for the last 10 time-steps
82
4
10
2
11.
m
99
FF
R
55
2
X
mm
m
8
Figure 13: Generalization Experiment: a) First row: generalization to unseen shapes. b) Second row: general- ization larger number of objects
We conduct three sets of experiments on generalization. In the first experiment, we investigate generalization to longer sequences. In this setting, the model is trained on trajectories of length 10 while being tested on trajectories of length 20. In the second experiment, we evaluate generalization in more crowded scenes. In this setting, the model is trained on 15-25 objects and tested on 50-60 objects. The third experiment tests the generalization of the model to unseen objects. The model is trained only
moving MNIST images containing digits 0 5 while being tested images containing digits 6 to 9. Figures [nn samples from these experiments. "SCALOR- LG" in Table [ also provides tracking results for the "length generalization" setting.
17

---

Published as a conference paper at ICLR 2020
D EXPERIMENT DETAIL AND ADDITIONAL QUALITATIVE RESULTS ON GRAND CENTRAL STATION DATASET
For natural scene experiments, we spatially split the video into parts and create a dataset of 400k frames in total. We choose the first 360k frames for training and 40k frames for testing. Since the movement of pedestrians too slow under 25 fps the original video, we treat every other 7 frames as two consecutive frames in the dataset sequence. The length of the input sequence is 10, and each image is resized to 128 x 128.
(a)
(b)
(c)
(d)
Figure 14: Conditional generation on Grand Central Station Dataset. The first 5 frames are observed, the next S frames generated (after the red line). (a) Overall reconstruction and generation, (b) Conditional genera- tion of background, (c) Conditional generation of segmented objects, (d) Conditional generation of movement trajectorics
The generation result mentioned in Section 5.2 is provided in Figure 14 We provide additional qualitative results in Figures 15 to 20 Rows from top to bottom represent input sequence, overall reconstruction, extracted background, foreground segmentation mask with IDs, center position from each z latent, and extracted trajectories based on the transition of center position. Additionally, Figures to 20 show examples of conditional generation, where images starting from time-step (after the red line) are generated sequences. Rows from top to bottom are conditional generations of the overall images. conditional generations of background, extracted conditional generation of each segmentation mask, and conditional generation of trajectories.
18

---

Published as a conference paper at ICLR 2020
Figure 15: Tracking sample
Figure 16: Tracking sample 2
19

---

Published as a conference paper at ICLR 2020
Figure 17: Tracking sample 3
141
Figure Conditional generation sample
Figure 19: Conditional generation sample
20

---

Published as a conference paper at ICLR 2020
Figure 20: Conditional generation sample 3
21

---

Published as a conference paper at ICLR 2020
E MODEL ARCHITECTURE DETAILS
In this section, we provide additional details of the architecture and hyperparameters used for pedes- trian detection. When a new frame provided, the network uses fully convolutional encoder to obtain a H x W feature map. The feature map is fed into convolutional LSTM to model the sequen- tial information along the sequence. The convolutional LSTM shared across discovery module and propagation module. The discovery module and propagation module also share the z zhaa en- coder and decoder. The encoder has a convolutional network followed by one fully connected layer, while the glimpse decoder uses fully convolutional network with sub-pixel layer (Shi et al. (016) for upsampling. The background module shares a similar structure with the z what encoder and de- coder. It takes 4-dimensional input, i.e., RGB and foreground mask, and outputs 3-dimensional image. We use GRUs in propagation trackers and in prior transition networks.
We choose batch size of 20 for the natural scene experiments and batch size of 16 for MNIST/aSprites experiments. The learning rate is fixed at 4e-5 for natural image experiments and Se-4 for d'prites/MNIST experiments. We use RMSprop for optimization during training. The standard deviation of the image distribution is chosen to be 0.1 for natural experiments and 0.2 for toy experiments. The prior for all Gaussian posteriors is set to standard normal. For the pedestrian tracking dataset, we constrain the range of 2 z so that the inferred width can vary from 5.2 pixels 11.7 pixels, and the height can vary from 12.0 28.8, and both with prior of the middle value in discovery. Similarly, times the actual object size. The n variable the propagation phase is modeled as the deviation of the position from the previous time-step instead of the global coordinate. The prior for zeres in discovery is set to be 0.1 at the beginning training and to quickly anneal to le-3 for natural image experiments and le-4 for dSprites/MNIST experiments. The temperature used for modelling gres is set to be 1.0 at the beginning and anneal to 0.3 after 20k iterations.
constrain n synthetic datasets that it can vary from half to 1.5
Full details of the architecture will be released along with code.
22


Reviews:
[{"review_id": "ryeAO_hsYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE: My original main concern was the lack of baseline, but during the rebuttal period the authors have conducted the request comparison and addressed my questions satisfactorily. Therefore, I would recommend the paper be accepted.\n\n---\nSummary: This paper proposes a generative model and inference algorithm for discovering and propagating object latents in a way that scales to hundreds of objects. The key components of their approach is the parallel discovery and propagation of object latents as well as the explicit modeling of the background. The authors show that the model is able to model and generalize to scenes with hundreds of objects, including a real-world scene of a train station.\n\nResearch Problem: This paper tackles the problem of scaling object-oriented generative modeling of scenes to scenes with a large number of objects.\n\nThe main weakness of the submission is the lack of a baseline, and mainly for this result I would recommend rejecting the submission at its current state. However, if the authors are able to revise the submission to include such comparisons (with Kosiorek et al. (2018), van Steenkiste et al. (2018), and Alahi et al. (2016), detailed below), then I would highly consider accepting the paper, as the paper makes a novel contribution to modeling scenes with many more objects than previous work, as far as I am aware.\n\nStrengths:\n- The authors show that the method can model various synthetic and real-world datasets that show the efficacy of the method\n- The method can also generalize to more objects and longer timesteps than trained on.\n\nWeaknesses:\n- The main weakness of the submission is the lack of a baseline. It would be important to understand the differences between Kosiorek et al. (2018), which the authors claim is the closest work to theirs, and van Steenkiste et al (2018), which also models objects in a parallel fashion. Alah et al. (2016) also takes an approach of dividing the scene into grid cells and also demonstrate results on modeling human trajectories.\n- Motivation: Whereas the authors motivate the benefits for modeling objects, the motivation for specifically scaling to model hundreds of objects is less clear. It would be helpful for the authors to provide examples or arguments for the benefits of modeling so many objects at once. One argument against such a need is that humans only pay attention to a few number of objects at a time and do not explicitly model every possible object in parallel. One argument in favor of such a need is the ability to gain superhuman performance on tasks that could benefit from modeling multiple entities, such as playing Starcraft, detecting anamolies in medical scans, or modeling large scale weather patterns.\n- A possible limitation of the method may be in modeling interactions between entities. What mechanism in the propagation step allows for modeling such interactions, and if not, how could such a mechanism be incorporated?\n- How would SCALOR behave if the grid cells were smaller than the objects? In this case an object may occupy multiple grid cells. Would the authors provide an experiment analyzing this case? Would SCALOR model a object as multiple entities in this case (because the object spans multiple grid cells), or would SCALOR model the object with a single latent variable?\n\nQuestion:\n- What is the fundamental reason for why the structure of such a generative model would cause the latents to model objects, rather than something else, such as the image patches that show the empty space between objects?\n\nAlahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., & Savarese, S. (2016). Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 961-971).\n\nVan Steenkiste, S., Chang, M., Greff, K., & Schmidhuber, J. (2018). Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802.10353.\n\nKosiorek, A., Bewley, A., & Posner, I. (2017). Hierarchical attentive recurrent tracking. In Advances in Neural Information Processing Systems (pp. 3053-3061).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJx4wc3TKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I thank the authors for the detailed rebuttal, as well as for the updates to the text and several new experiments in the revised version of the paper. Most of my comments are addressed well. I am happy to improve my rating and recommend to accept the paper.\n\n---\n\nThe paper proposes an approach for unsupervised detection and tracking of objects in videos. The method continues the \"Attend, Infer, Repear\" (AIR) and Sequential AIR (SQAIR) line of work, but improves on these previous approaches in terms of scalability and can thus be applied to scenes with tens of objects. The scalability is obtained by replacing, wherever possible, sequential processing of objects by parallel processing. Experiments are performed on three datasets: moving DSprites, moving MNIST, and the real-world \"Crowded Grand Central Station\" dataset. The method seems to work in all cases, as confirmed by quantitative evaluation on the first two datasets and a qualitative evaluation on all three.\n\nI recommend rejecting the paper in its current state. On one hand, the results look quite good, and the method seems to indeed scale well to many objects. On the other hand, novelty is limited and the experiments are limited in that there are no comparisons with relevant baselines and no clear experiments showing the specific impact of the architectural modifications proposed in this paper. Moreover, the paper is over 9 pages, which, as far as I remember, requires the reviewers apply \"higher standards\". Overall, I would like the authors to comment on my concerns (below) and may reconsider my rating after that.\n\nPros:\n1) Relatively clear presentation.\n2) Judging from the extensive qualitative results and the (limited) quantitative evaluation, the method seems to work.\n3) I appreciate the additoinal results on generalization and parallel disovery in the appendix.\n\nCons:\n1) Novelty of the work seems quite limited. The main contribution is in improving the efficiency of object detection/tracking by parallelizing the computation, without much conceptual innovation. This might be a sufficient contribution (in the end, efficiency is very important for actually applying methods in practice), but then a thorough evaluation of the method would be expected (see further comments about it further). Moreover, the previously published SPAIR method by Crawford and Pineau seems very relevant and related, but is not compared against and is only briefly commented upon, despite the code for that method seems to be available online. I would like the authors to clarify the relation to SPAIR and preferably provide an experimental comparison.\n\n2) The experiments are restricted. While there are quite many qualitative results, several issues remain:\n2a) No baseline results are reported. It is thus impossible to judge if the method indeed improves upon related prior works. In particular, comparisons to SQAIR and SPAIR would be very useful. If possible, it would be useful to provide even more baselines, for instance some traditional tracking methods. Comparisons can be both in terms of tracking/reconstruction performanc, as well as in terms of computational efficiency. Both can be measured as functions of the number of objects in the scene.\n2b) There are few quantitative resutls. In experiments 2-4 of section 5.1 it seems it would be fairly easy to introduce some, in particular, one could compare how do these more difficult settings compare to the \"default\" one. In section 5.2 given that the ground truth tracks are not available, evaluating tracking is challenging, but one could still compare NLL/reconstruction with appropriate baselines. The provided comparison to a vanilla VAE in therms of NLL is actually somewhat confusing - not sure what it tells the reader; moreover, I would actually expect the NLL of the proposed structured model to be better. Why is it not?\n2c) Since the paper is largely about tracking objects through videos, it would be very usefuly to include a supplementary video with qualitative results. \n\n3) (minor) Some issues with the presentation:\n3a) I found the method description at times confusing and incomplete. For instance, it is quite unclear which exactly architectures are used for different components. The details can perhaps bee looked up in the SQAIR paper, but it would still be useful to summarize most crucial points in the supplementary material to make the paper more self-contained.\n3b) The use of \\citet vs \\citep is often incorrect. \\citet should be used when the author names are a part of the text, while \\ciptp if the paper is cited in passing, for instance: \"Smith et al. (2010) have shown that snow is white.\" vs \"It is known that snow is white (Smith et al. 2010).\" \n3c) Calling AIR a \"seminal work in the field of object detection\" is not quite correct - object detection is a well-established task in computer vision, and AIR is not really considered a seminal work in that field. It is a great paper, but not really in object detection.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byl5Um4m5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a generative model for scalable sequential object-oriented representation. The paper proposes several improvements based on the method SQAIR (Kosiorek et al. 2018b), (1) modeling the background and foreground dynamics separately; (2) parallelizing the propagation-discovery process by introducing the propose-reject model which reducing the time complexity. Finally, the proposed model can deal with orders of magnitude more objects than previous methods, and can model more complex scenes with complex background.\n\nAccept.\nThe paper is clearly written and the experimental results are well organized. The results in the paper may be useful for unsupervised multi-objects tracking .I have one concern here, \n（1）As argued in the paper, previous methods are difficult to deal with the nearly a hundred objects situation and there is no direct comparison for these methods. So has the author compared the method SCALOR with previous methods in few objects setting? Does the technical improvements of the method benefit in the few objects setting?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the author response and paper revision, the reviewers all came to appreciate this paper and unanimously recommended it be accepted.  The paper makes a nice contribution to generative modelling of object-oriented representations with large numbers of objects.  The authors adequately addressed the main reviewer concerns with their detailed rebuttal and revision.
Decision: Accept (Poster)