SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition
Authors: Zhixuan Lin
Yi-Fu Wu
Skand Vishwanath Peri
Weihao Sun
Gautam Singh
Fei Deng
Jindong Jiang
Sungjin Ahn
Venue:  (2020)
Keywords: Generative models
Unsupervised scene representation
Object-oriented representation
spatial attention

Abstract: The ability to decompose complex multi-object scenes into meaningful abstractions like objects is fundamental to achieve higher-level cognition. Previous approaches for unsupervised object-oriented scene representation learning are either based on spatial-attention or scene-mixture approaches and limited in scalability which is a main obstacle towards modeling real-world scenes. In this paper, we propose a generative latent variable model, called SPACE, that provides a uniﬁed probabilistic modeling framework that combines the best of spatial-attention and scene-mixture approaches. SPACE can explicitly provide factorized object representations for foreground objects while also decomposing background segments of complex morphology. Previous models are good at either of these, but not both. SPACE also resolves the scalability problems of previous methods by incorporating parallel spatial-attention and thus is applicable to scenes with a large number of objects without performance degradations. We show through experiments on Atari and 3D-Rooms that SPACE achieves the above properties consistently in comparison to SPAIR, IODINE, and GENESIS. Results of our experiments can be found on our project website: https://sites.google.com/view/space-project-page

Published as a conference paper at ICLR 2020
SPACE: UNSUPERVISED OBJECT-ORIENTER SCENE REPRESENTATION VIA SPATIAL ATTENTION AND DECOMPOSITION
Zhixuan Lin Yi-Fu Wu', Skand Vishwanath Peri',) Weihao Sun', Gautam Singh', Fei " J Sungjin Ahn
Rutgers University & 2Zhejiang University
ABSTRACT
The ability to decompose complex multi-object scenes into meaningful abstrac- tions like objects is fundamental to achieve higher-level cognition. Previous ap- proaches for unsupervised object-oriented scene representation learning are ei- ther based on spatial-attention or scene-mixture approaches and limited in scal- ability which is a main obstacle towards modeling real-world scenes, In this paper, we propose generative latent variable model, called SPACE, that pro- vides a unified probabilistic modeling framework that combines the best of spatial-attention and scene-mixture approaches. SPACE can explicitly provide factorized object representations for foreground objects while also decompos- ing background segments complex morphology. Previous models are good at either of these, but not both. SPACE also resolves the scalability problems previous methods by incorporating parallel spatial-attention and thus is ap- plicable to scenes with large number of objects without performance degra- dations. We show through experiments on Atari and 3D-Rooms that SPACE achieves the above properties consistently in comparison to SPAIR, IODINE, and GENESIS. Results of our experiments can be found on our project website:
INTRODUCTION
One of the unsolved key challenges in machine learning is unsupervised learning of structured repre- sentation for a visual scene containing many objects with occlusion, partial observability, and com- plex background. When properly decomposed into meaningful abstract entities such as objects and spaces, this structured representation brings many advantages of abstract (symbolic) representation areas where contemporary deep learning approaches with global continuous vector represen- tation of a scene have not been successful. For example, structured representation may improve sample efficiency for downstream tasks such as deep reinforcement learning agent (Mnih et al,) 2013). It may also enable visual variable binding (Sun) 1992) for reasoning and causal inference over the relationships between the objects and agents in a scene. Structured representations also provide composability and transferability for better generalization.
Recent approaches to this problem of unsupervised object-oriented scene representation can be categorized into two types of models: scene-mixture models and spatial-attention models. In scene- mixture models (Greff et al. 2017 2019, Buuggess et al.] 2019 Engelcke et al.) 2019). a visual scene is explained by mixture of finite number of component images. This type of representation provides flexible segmentation maps that can handle objects and background segments of complex morphology. However, since each component corresponds features of objects like position and scale are only implicitly encoded in the scale of full image and further disentanglement is required to extract these useful features. Also, since it does not explicitly reflect useful inductive biases like the locality of an object in the Gestalt principles (Kofika 2013),
full-scale image, important physical
VViiitig Student Rutgers University. Authors named inside () equally contributed. Correspondance to zx1 cs imma 1 com and sjn. ahn@gma i com.

---

Published as a conference paper at ICLR 2020
the resulting component representation is not necessarily representation of local area. Moreover, to obtain a complete scene, a component needs to refer to other components, and thus inference is inherently performed sequentially, resulting in limitations in scaling to scenes with many objects.
In contrast, spatial models (Eslami et al., 2016 Crawford & Pineau 2019) can explicitly obtain the fully disentangled geometric representation of objects such position and scale. Such features are grounded on the semantics of physics and should be useful in many ways (e.g., sample efficiency, interpretability, geometric reasoning and inference, transferability). However, these mod- els cannot represent complex objects and background segments that have too flexible morphology to be captured by spatial attention (i.e. based on rectangular bounding boxes). Similar to acene-mixture models, previous models in this class show scalability issues as objects are processed sequentially.
In this paper, we propose a method, called Spatially Parallel Attention and Component Extraction (SPACE), that combines the best of both approaches. SPACE learns to process foreground objects, which can be captured efficiently by bounding boxes, by using parallel spatial-attention while de- composing the remaining area that includes both morphologically complex objects and background segments by using component mixtures. Thus, SPACE provides an object-wise disentangled rep- resentation of foreground objects along with explicit properties like position and scale per object while also providing decomposed representations of complex background components. Further- more, by fully parallelizing the foreground object processing, existing spatial attention methods. In experiments on 3D-room scenes and Atari game scenes, we quantitatively and qualitatively compare the representation of SPACE other models and show that SPACE combines the benefits of both approaches in addition to significant speed-ups due to the parallel foreground processing.
resolve the scalability issue of
The contributions of the paper are as follows. First, we introduce a model that unifies the benefits of apatial-attenting and scene-mixture approaches in a principled framework of probabílistic latent variable modeling. Second, we introduce a spatially parallel multi-object processing module and demonstrate that can significantly mitigate the scalability problems of previous methods. Lastly, provide an extensive comparison with previous models where we illustrate the capabilities and limitations of each method.
THE PROPOSED MODEL: SPACE
In this section, describe our proposed model, Spatially Parallel Attention and Component Extrac- tion (SPACE). The main idea of SPACE, presented in Figure D is to propose a unified probabilistic generative model that combines the benefits of the spatial-attention and scene-mixture models.
2.1 GENERATIVE PROCESS
SPACE assumes that scene x is decomposed into two independent latents: foreground z and background ne The foreground is further decomposed into set of independent foreground objects z {z(²) { and the background is also decomposed further into a sequence of background segments by 2e = Z1.K While our choice of modeling the foreground and background independently worked well empirically, for better generation, it may also be possible to condition one on the other. The image distributions of the foreground objects and the background components are combined together with a pixel-wise mixture model to produce the complete image distribution:
p(x|z'' 2 a p((l +(1 a) Σ Th (x|Z
(1)
Foreground
Background
Here, the foreground mixing probability a is computed as a (2 This way, the foreground
is given precedence in assigning its mixing weight and the remaining is apportioned to the
background. The mixing weight assigned to the background is further sub-divided among the K background components. These weights are computed as Ty fnz (2 and Σx Tu = 1. With these notations, the complete generative model can be described as follows.
= p(x) = // p(x|z!' ppz2 (p(z () d2
(2)
2

---

Published as a conference paper at ICLR 2020
Background
Origiall image
Racigyound Components
Recomstructed Bacagrund
Foreground
Reconstructed Image
Paraliel Spatial Aborrion
Recormbuuttdd
Figure Illustration of the SPACE model. SPACE consists foreground module and background module. foreground module, the input image divided into grid encoder is used to compute the where _dptt and 2 for each cell in parallel. = where is used to identify proposal bounding boxes and spatial transformer is used to attend to each bounding box in parallel, computing = what encoding for each cell. The model selects patches using the bounding boxes and reconstructs them using VAE from all the foreground latents 2 background module segments the scene into components in the figure) using a pixel-wise mixture model. Each component consists of a set of latents n (z"", =² where =" models the mixing probability of the component and =" models the RGB distribution of the component. The components are combined to reconstruct the hackground using VAF. The reconstructed background and foreground then combined using pixel-wise mixture model generate the full reconstructed image.
W cells (4 x in figure). An image
We now describe the foreground and background models in more detail.
Foreground. SPACE implements & as a structured latent. In this structure, an image is treated as it were divided into И x W cells and each cell is tasked with modeling at most (nearby) object in the scene. This type of structuring has been used in (Redmon et al., 2016 Santoro et al. 2017 Crawford & Pineau 2019). Similar SPAIR, in order model an object, each cell is associated dapth with a set of latents (z.".e z! ziehere, z; 2zhhr). In this notation, z is a binary random variable denoting if the cell models any object not, where denotes the size of the object and its location relative to the cell, deppth denotes the depth of the object to resolve occlusions and z what models the object appearance and its mask. These latents may then be used compute the foreground image component p(x z which is modeled as Gaussian distribution A N(1s of.). In practice, we treat of hyperparameter and decode only the mean image f. In this process, SPACE reconstructs the objects associated to each cell having pees = 1. For each such cell, the model uses the z zuthur to decode the object glimpse and its mask and the glimpse is then positioned using zz where via the Spatial Transformer (Jaderberg et al. 2015). Using the object masks and z, zdopth all the foreground objects are combined into single foreground mean-image j and the foreground mask a (See Appendix D for more details).
full-resolution canvas
SPACE imposes a prior distribution on these latents as follows:
HxW p(z = II
p(2 (p(z) Where )p(z) deptt )p(z.
depth what
==1
Here, only 2!"0s is modeled using Bernoulli distribution while the remaining modeled Gaus- sian.
Background. To model the background, SPACE implements z similar GENESIS. as (z 2() where z models the mixing probabilities т of the components and z models the RGB distribu- bg of the 1tth background component Gaussian N N( og)) The following prior is tion p(x x
3

---

Published as a conference paper at ICLR 2020
imposed upon these latents.
K II A==1
p(2
p(z (p(zk" z
(4)
2.2 INFERENCE AND TRAINING
Since cannot analytically evaluate the integrals in equation due to the continuous latents z and bg i.K we train the model using variational approximation. The true posterior on these variables is
approximated as follows.
p 2 1
2
x)
q(2
x)
II
a(
(5)
k=1
This is used to derive the following ELBO to train the model using the reparameterization trick and SGD (Kingma & Welling, 2013).
K DRu(a(ZE & x) p(
[(x) Re 2/11))
1ogp(x/=!,2293
(2kk)
k=1
(6)
HxW
Σ
DKI. (a(2 p(2,)))
See Appendix B for the detailed decomposition of the ELBO and the related details.
Parallel Inference of Cell Latents. SPACE uses mean-field approximation when inferring the cell
latents, so 2 (z. pres
2i where depth
Zi .2. :2(hha) for each cell does not depend other cells.
HxW W
q( xx
II i=1
(z."T x) (g(z)/bere) (((2))bre)))(( depih ( what 2.yberr x)
(z
As shown in Figure [ this allows each cell to act as independent object detector, spatially attend- ing to its own local region in parallel. This is in contrast to inference in SPAIR, where each cell's latents auto-regressively depend on some or all of the previously traversed cells in row-major or- der i.e., (zz5||x = #o 12 x). However, this method becomes prohibitively expensive in practice the number of objects increases. While/Crawin connections are crucial for performance since they model dependencies between objects and thus prevent duplicate detections, challenge this assertion by observing that 1) due to the bottom-up encoding conditioning on the input image, each cell should have information about its nearby area without explicitly communicating with other cells, and 2) in (physical) spatial space, two objects cannot exist at the same position. Thus, the relation and interference between objects should not be severe and the mean-field approximation is good choice in our model. In our experiments, verify empirically that this is indeed the case and observe that SPACE shows comparable detection performance to SPAIR while having significant gains in training speeds and efficiently scaling to scenes with many objects.
& Pineau (219)) claim that these lateral
Preventing Box-Splitting. the prior for the bounding box size is set to be too small, then the model could split large object by multiple bounding boxes and when the size prior is too large, the model may not capture small objects in the scene, resulting in a trade-off between the prior values of the bounding box size. To alleviate this problem, auxiliary loss which we call the boundary loss. In boundary loss, we construct a boundary of thickness b pixels along the borders of each glimpse. Then, restrict object to be inside this boundary and penalize the model if an object's mask overlaps with the boundary area. Thus, the model penalized if tries to split large object by multiple smaller bounding boxes. A detailed implementation of the boundary loss mentioned in Appendix
found it sometimes helpful to introduce an
RELATED WORKS
Our proposed model is inspired by several recent works in unsupervised object-oriented scene de- composition. The Attend- (rrendInffe-Bee Infer-Repeat (AIR) (Eslami et al. 2016) framework uses a recurrent neu- ral network to attend to different objects in scene and each object is sequentially processed one
4

---

Published as a conference paper at ICLR 2020
at time. An object-oriented latent representation is prescribed that consists of 'what', 'where', and "presence" variables. The 'what' variable stores the appearance information of the object, the 'where' variable represents the location of the object in the image, and the 'presence' variable con- trols how many steps the recurrent network runs and acts as an interruption variable when the model decides that all objects have been processed.
Since the number of steps AIR runs scales with the number of objects it attends to, it does not scale well to images with many objects. Spatially Invariant Attend, Infer, Repeat (SPAIR) (Crawford & Pineau 2019) attempts to address this issue by replacing the recurrent network with convolutional network. Similar to YOLO (Redmon et al., 2016). the locations of objects are specified relative to local grid cells rather than the entire image, which allow for spatially invariant computations. In the encoder network, a convolutional neural network is first used to map the image to a feature volume with dimensions equal to a pre-specified grid size. Then, each cell of the grid is processed sequentially to produce objects. This is done sequentially because the processing of each cell takes as input feature vectors and sampled objects of nearby cells that have already been processed. SPAIR therefore scales with the pre-defined grid size which also represents the maximum number of objects that can be detected. Our model uses approach similar to SPAIR to detect foreground objects, but importantly we make the foreground object processing fully parallel to scale to large number of objects without performance degradation. Works based on Neural Expectation Maximization (Van Steenkiste et al. (018:) Greff et al. 2017) do achieve unsupervised object detection but do not explicitly model the presence, appearance, and location of objects. These methods also suffer from the problem of scaling to images with a large number of objects. In a related line of research, AIR has also recently been extended to track objects in sequence of images (Kosiorek al. 2018
Crawford & Pineau 2020 Jiang et al. 2020).
For unsupervised scene- meneemiitture models, several recent models have shown promising results. MONet (Burgess et al.] (219)) leverages a deterministic recurrent attention network that outputs pixel-wise masks for the scene components. variational autoencoder (VAE) (Kingma Welling, 2013) is then used to model each component. IODINE (Greff et al. 2019) approaches the problem from a spatial mixture model perspective and uses amortized iterative refinement of latent object rep- resentations within the variational framework. GENESIS (Engelcke et al. 2019) also uses spatial mixture model which is encoded by component-wiss latent variables. Relationships between these components are captured with autoregressive prior, allowing complete images to be modeled by a collection of components.
4 EVALUATION
We evaluate our model on two datasets: 1) an Atari (Bellemare et al. 2013) dataset that consists of random images from a pretrained agent playing the games, and 2) generated 3D-room dataset that consists of images of walled enclosure with a random number of objects on the floor. In order to test the scalability of our model, we use both a small 3D-room dataset that has 4-8 objects and a large 3D-room dataset that has 18-24 objects. Each image is taken from a random camera angle and the colors of the objects, walls, floor, and sky are also chosen at random. Additional details of the datasets can be found in the Appendix E
We compare our model against two scene-mixture models (IODINE and GENESIS) and one spatial-attention model (SPAIR). Since SPAIR does not have an explicit background compo- nent, we add an additional VAE for processing the background. Additionally, we test against two implementations of SPAIR: one where we train on the entire image using 16 16 grid and another where we train on random 32 x 32 pixel patches using a 4 x 4 grid. We denote the former model as SPAIR and the latter as SPAIR-P. SPAIR-P is consistent with the SPAIR's alternative training regime on Space Invaders demonstrated in Crawford & Pineau (2019) to address the slow training of SPAIR on the full grid size because of its sequential inference. Lastly, for performance reasons,
Baselines.
unlike the original SPAIR implementation. we use parallel processing for rendering the objects from their respective latents onto the canva for both SPAIR and SPAIR-P. Thus, because of these im- provements, our SPAIR implementation can be seen stronger baseline than the original SPAIR. More details of the basclines are given in Appendix D
It is important to note that the worst case complexity of rendering is Ohw x HW), (where (h, u)) is the image size) which is extremely time consuming when we have large image size and/or large number of objects.
5

---

Published as a conference paper at ICLR 2020
SPACE
SPAIT
N/A
N/A
N/A
N/A
N/A
SpAm Pinch
N/A
N/A
GUNCSE
N/A
M/A
N/A
SPACE
N/A
N/A
N/A
N/A
GENG26
N/A
M/A
N/A
Figure 2: Qualitative comparison between SPACE, SPAIR, SPAIR-P, IODINE and GENESIS for the 3D-
Room dataset.
4.1 QUALITATIVE COMPARISON OF INFERRED REPRESENTATIONS
In this section, we provide qualitative analysis of the generated representations of the different models. For each model, we performed a hyperparameter search and present the results for the best settings of hyperparameters for each environment. Figure 2 shows sample scene decompositions
of our baselines from the 3D-Room dataset and Figure 3 shows the results on Atari. Note that SPAIR does not component masks and IODINE and GENESIS do not separate foreground from background, hence the corresponding cells are left empty. Additionally, we only show a few representative components for TODINE and GENESIS since
those experiments with larger K than can be displayed. More qualitative results of SPACE can be found in Appendix A
IODINE GENESIS. In the 3D-Room environment, IODINE able to segment the objects and the background into separate components. However, it occasionally does not properly decompose objects (in the Large 3D-room results, the orange sphere on the right is reconstructed) and may generate blurry objects. GENESIS is able to segment the background walls, floor, and sky into multiple components. It able capture blurry foreground objects in the Small 3D-Room, but is not able to cleanly capture foreground objects with the larger number of objects in the Large 3D- Room. Atari, both IODINE and GENESIS fail to capture the foreground properly or try to encode all objects in single component. We believe this is because the objects in Atari games smaller, less regular and lack the obvious latent factors like color and shape as the 3D dataset, and thus detection-based approaches more appropriate in this case.
SPAIR & SPAIR-P. SPAIR is able to detect tight bounding boxes in both 3D-Rooms and the two Atari games. SPAIR-P, however, often fails to detect the foreground objects in proper bounding boxes, frequently uses multiple bounding boxes for one object and redundantly detects parts of the background foreground objects. This is limitation of the patch training the receptive field of

---

Published as a conference paper at ICLR 2020
Mode
Feregrennt
Famppacd Backgraad
Campooae
14th
N/A
N/A
N/A
SPAJR-P
N/A
N/A
N/A
100
N/A
N/A
N/A
N/A
N/A
N/A
Receddnscan
Fraegarsi
Foregronnt Mak Bedgroend
Mad
ENY
1601
N/A
N/A
N/A
100NE
N/A
N/A
N/A
GAMEAR
N/A
N/A
N/A
Figure 3: Qualitative comparison between SPACE, SPAIR, IODINE and GENESIS for Space Invaders, Air Raid, River Raid.
Restremaaiinn
Eameprand
Rodgard
Altens
Atoets
1211
Obei
Figure 4: Qualitative demonstration of SPACE trained jointly on a selection of 10 Atari games. We show games with complex background here.

---

Published as a conference paper at ICLR 2020
each patch is limited to 32 x 32 glimpse. prohibiting it from detecting larger objects and making it difficult to distinguish the background from foreground. These two properties are illustrated well in Space Invaders, where it is able to detect the small aliens, but it detects the long piece of background ground on the bottom of the image as foreground objects.
SPACE. In both 3D-Room, SPACE is able accurately detect almost all objects despite the large variations in object positions, colors, and shapes, while producing a clean segmentation of the back- ground walls, ground, and sky. This is in contrast the SPAIR model, while being able provide similar foreground detection quality, encodes the whole background into a single component, which makes the representation less disentangled. Notably, in River Raid where the background is con- stantly changing, SPACE is able to perfectly segment the blue river while accurately detecting all foreground objects, while SPAIR often cannot properly separate foreground and background.
Joint Training. F shows the results of training SPACE jointl across 10 Atari games. We see that even in this setting, SPACE is able to correctly detect foreground objects and cleanly segment the background.
Foreground vs Background. Typically, foreground the dynamic local part of the scene that we are interested in, and background is the relatively static and global part. This definition, though intuitive, ambiguous. Some objects, such as the red shields Space Invaders and the key in Mon- tezuma's Revenge (Figure 6) are static but important to detect as foreground objects. We found that SPACE tends to detect these as foreground objects while SPAIR considers background. Similar behavior is observed in Atlantis (Figure 4). where SPACE tends to detect some foreground objects from the middle base that is above the water. One reason for this behavior is because we limit the ca- pacity of the background module by using spatial broadcast decoder (Watters et al. 2019) which is much weaker when compared to other decoders like sub-pixel convolutional nets (Shi et al. 2016 This would favor modeling static objects as foreground rather than background.
4.2 QUANTITATIVE COMPARISON
In this section we compare SPACE with the baselines in several quantitative metric We first note that each of the baseline models has a different decompoxition capacity (C), which we define as the capability of the model to decompose the scene into its semantic constituents such as the foreground objects and the background segmented components, For SPACE, the decomposition capacity is equal to the number of grid cells / x W (which is the maximum number of foreground objects that can be detected) plus the number of background components K. For SPAIR, the decomposition capacity equal to the number of grid cells H W plus for background. For IODINE and GENESIS, it is equal to the number of components K.
For each experiment, we compare the metrics for each model with similar decomposition capacities. This way, each model can decompose the image into the same number of components. For setting in SPACE with a grid size of H x W with KSPACE components, the equivalent settings in IODINE and GENESIS would be with C (H x W/ + KSPACE- The equivalent setting in SPAIR would be a grid size of H x W.
Table 1: Comparison of SPACE the SPAIR baseline with respect to the quality of the bounding boxes in the 3D-R DD-Room setting. Results are averaged over 5 best random seeds and standard deviations are given.
Avg. Precision TaU Threshold 0.5
Avg. Procision
Object Count Erar Rate
Model
Dataset
ToU Threshold @ [0.5: 0.05 0.95]
SPACE (16x) x 16)
3D-Room Large
0.8927 = 0.0027
0.1115 + 0.0075 0.4364 + 0.0179
0.01 = 0.0026 0.0300 0.0072
SPAIR
3D-Room Large
0.9072 0.0003
SPACE (8 x 8)
3D> Room Small
0.9027 +: 0.0019
0.5069 1: 0.0030
0.0397 =: 0.0022
SPAIR (8
Ro.mRom Small
0.9081 0.0004
0.5068 0.0081
0.0200 0.0039
Gradient Step Latency. The lefumost chart of Figure shows the time taken to complete one gradient step (forward and backward propagation) for different decomposition capacities for each of the models. We see that SPAIR's latency grows with the number of cells because of the sequential
2 As previously shown. SPAIR-P does not work well in many of our environments, so we do not include it in these quantitative experiments. As in the qualitative section, we use the SPAIR implementation with sequential inference and parallel rendering in order to speed up the experiments.
8

---

Published as a conference paper at ICLR 2020
Training Latency
Convergence 3D Room (4 x 4)
Convergence 3D Room
Convergence 3D 16)
1,200
1,200
Gradent Step Laterty (In
Recorstruction Pixal MSE
1,000
2,600
1,500
2,200
800
1,800
1,400
400
200
500
200
4x4 8x6 Dx 35
26
Number of Calls (H 19)
Wall Clock Time (in 5) act
Wall Cooc Time (in
Wall Clock Time (in -a0
5)
(C= W+1)
10DINE (C H 5++5)
GENESIS
Figure 5: Quantitative performance comparison hetween SPACE, SPAIR, IODINE and GENESIS in terms of prochs-pocessing time during training, training convergence and converged pixel MSE. Convergence plots showing pixel-MSE computed held-out during training.
nature its latent inference step. Similarly GENESIS and IODINE's latency grows with the number of components K because each component is processed sequentially in both the models. IODINE is the slowest overall with its computationally expensive iterative inference procedure. Furthermore, both IODINE and GENESIS require storing data for each of the K components, so we were unable to run our experiments on 256 components or greater before running out of memory on our 22GB GPU. On the other hand, SPACE employs parallel processing for the foreground which makes scalable to large grid sizes, allowing it to detect large number of foreground objects without any significant performance degradation. Although this data was collected for gradient step latency, this comparison implies similar relationship exists with inference time which is a main component in the gradient step.
Time for Convergence. The remaining three charts in Figure 5 show the amount of time each model takes to converge in different experimental settings. We use the pixel-wise mean squared error (MSE) measurement of how close model is to convergence. In all settings, SPAIR and SPACE converge much faster than IODINE and GENESIS. In the 4 x and 8 x setting, SPAIR and SPACE converge equally fast. But as we scale up 16 16, SPAIR becomes much slower than SPACE
Average Precision and Error Rate. In order to assess the quality of our bounding box predictions, we measure the Average Precision and Object Count Error Rate of our predictions. Our results are shown in Table [ We only report these metrics for 3D-Room since we have access to the ground truth bounding boxes for each of the objects in the scene. Both models have very similar average precision and error rate. Despite being parallel in its inference, SPACE has a comparable count error rate to that of SPAIR.
From our experiments, can assert that SPACE can produce similar quality bounding boxes as SPAIR while 1) having orders of magnitude faster inference and gradient step time, 2) scaling to a large number of objects without significant performance degradation, and 3) providing complex background segmentation.
5 CONCLUSION
We propose SPACE, unified probabilistic model that combines the benefits of the object represen- tation models based on spatial attention and the scene decomposition models based on component mixture. SPACE can explicitly provide factorized object representation per foreground object while also decomposing complex background segments. SPACE also achieves significant speed-up and thus makes the model applicable to scenes with a much larger number of objects without perfor- mance degradation. Besides, the detected objects in SPACE are also more intuitive than other meth- ods. We show the above properties of SPACE on Atari and 3D-Rooms. Interesting future directions replace the sequential processing of background by parallel one and to improve the model for natural images. Our next plan is to apply SPACE for object-orienter model-base based reinforcement learning.

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
SA thanks to Kakao Brain and Center for Super Intelligence (CSI) for their support. ZL thanks to the ZJU-3DV group for its support. The authors would also like to thank Chang Chen and Zhuo Zhi for their insightful discussions and help in generating the 3D-Room dataset.
REFERENCES
Jonathan T. Barron. Continuously differentiable exponential linear units. ArXiv, abs/ 704.007488 2017.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
Christopher P Burgess, Loic Matthey Nicholas Watters, Rishabh Kabra, Trina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint arXiv:1901.11338 2019.
Djork- émé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). ICLR, 2016.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolu- tional neural networks. In Proceedings of AAAI, 2019.
Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. Proceedings of AAAI, 2020.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations, 2019.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, and Geoffrey Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2017.
Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint 1rXiv:1903.00455 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning. 2015.
Max Jaderberg. Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In Advances in neural information processing systems, 2017-2025, 2015.
Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameterrization with gumbel-softmax. arXiv preprint arXiv:1611.0114. 2016.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Scalable object-
oriented sequential generative models. In International Conference on Learning Representations, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.61 2013.
Kurt Koffka. Principles of Gestalt psychology. Routledge, 2013.
10

---

Published as a conference paper at ICLR 2020
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling moving objects. In Advances in Neural Information Processing Systems, pp. 8806-86166 2018.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pp. 4967-4976, 2017.
Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang- Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Ron Sun. On variable binding in connectionist networks. Connection Science, 4(2):93-124, 1992.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop, neural networks for machine learn- ing. Tech Report, 2012. URL https:/
cs. wtt.::stttoottoooooooooooooo http://:tttjmmmm/ccccc /
s diides/lectune
sl slides 1ec6 pdff
Emanuel Todorov. Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural ex- pectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint 1rXiv:1802. 10353, 2018.
Nicholas Watters, Loic Matthey, Christopher P. Burgess, and Alexander Lerchner. Spatial broad- cast decoder: A simple architecture for learning disentangled representations in vaes. ArXiv, abs/1901.070117, 2019.
Yuxin Wu and Kaiming He. Group normalization. The European Conference on Computer Vision (ECCV), September 2018.
Yuxin Wu al. Tensorpack. huttpss//gghhib.ccm///eeerrrg// 2016.

---

Published as a conference paper at ICLR 2020
A
ADDITIONAL RESULTS OF SPACE
Made
Type
Recesiiniion
Faregran
Forercel Mask Backeriedd
Conpannt Med RGB
SPACE
SPARE
N/A
N/A
N/A
Figure 6: Case illustration of Montezuma's Revenge comparing object-detection behaviour in SPACE and SPAIR.
Fongreant
Poregaund lite Banggaad
Componer
Sackgraal Corpoiess
43
124
Picl
Cannoc
Doatle
Nanpero
Rogan
Spez
Figure 7: Qualitative demonstration SPACE trained the jointly on selection ATARI games.
12

---

Published as a conference paper at ICLR 2020
Heczmdt aciaa
unegraani
Fssegrtanal Mack Backgraand
Companet
Figure Object detection and background segmentation using SPACE 3D-Room data set with small objects. Each row corresponds one input image.
13

---

Published as a conference paper at ICLR 2020
Heczmdi acina
uregraann
Fiseggtanaa Backgrand
Cumpasett
Figure Object detection and background segmentation using SPACE 3D-Room data with large ber of objects.
14

---

Published as a conference paper at ICLR 2020
B ELBO DERIVATIONS
In this section, we derive the ELBO for the log-likelihood p(x).
pxx) 2 E (e xx) [p(x| ==
DKL(q x) p(z())) (x ll ((2
E (oe (xx222 2b8)
DKL (q(z xx) p((2)) DKL( p(2
= xxxx)
pxx 2)
Dal(a(zi by [x, 2xk) 1(2
HxW
DKL(a(2,1x0) ((2t)))
KL Divergence for the Foreground Latents Under the SPACE 's approximate inference, the (g2 (z, p(z,")) inside the expectation can be evaluated as follows.
Eg(20 [KK((((2)1x) ll p(2)))))
-Eg pppppe) D (x) ll p(2()1)) + Bece," xx) 2 Atheree (x) (2
(xgttt bx) DKL (2(22
xx, 2, zhhere) || p(z)))aa) + DKL (g(2 (22geppth (x) p(2 (opth)
= =Rylee px))))
DKL.(a(22200 (x) p(2((0))) + 2. 2pee (K12) (hheee xx) p(2 (bbre))))
DKL (q(z, x,, 2, (xhere) || p(zz(tar) + DKL (g(2, (200))00 xx) ll p(z; (opppt
KL Divergence for the Background Latents Under our GENESIS-like modeling of inference for the background latents, the KL term inside the expectation for the background is evaluated as follows.
(KI,, x 12 x) p(z x)
= =Bay(x xx//)))
DKL xx x) || p(z x>+)) + E g ba bg xx (aw(a(LLeeee 1z , x) p(z 12
EEyaeh [DKL (z( x x) p|
DKL (g(z)
Relaxed treatment of pres In implementation, model the Bernoulli random variable z using the Gumbel-Softtax distribution (Jang et al.] (016)). We use the relaxed value of zres in the entire training and use hard samples only for the visualizations.
C BOUNDARY LOSS
In this section we elaborate on the implementation details of the boundary loss. We construct kernel of the size of the glimpse, gs gs (we use gs = 32) with a boundary gap of b = 6 having negative uniform weights inside the boundary and a zero weight the region between the boundary and the glimpse. This ensures that the model is penalized when the object is outside the boundary. This kernel is first mapped onto the global space via STN Jaderberg et al. 2015) to obtain the global kernel. This is then multiplied element-wist with global object mask obtain the boundary loss map. The objective of the loss is to minimize the mean of this boundary loss map. In addition to the ELBO, this loss is also back- propagated via RMSProp (Tieleman & Hinton. (2012)). This loss, due to the boundary constraint, enforces the bounding boxes to be less tight and results in lower average precision, disable the loss and optimize only the BLBO after the model has converged well.
15

---

Published as a conference paper at ICLR 2020
D IMPLEMENTATION DETAILS
D.1 ALGORITIMMS
Algorithm and Algorithm present SPACE's inference for foreground and background. Algo- rithm 2 describe the details rescale, function in Algorithm that transforms local shift zthili to global shift 2-hin Algorithm show the details of the generation process of the background mod- ule. For foreground generation, we simply sample the latent variables from the priors instead of conditioning on the input. Note that, for convenience the algorithms for the foreground module and background module are presented with for loops, but inference for all variables of the foreground module are implemented as parallel convolution operations and most operations of the background module (barring the LSTM module) are parallel as well.
Algorithm 1: Foreground Inference
Input: image æ
Output: foreground mask a, appearance grid grid height H and width W
é imageFncoderFig())
ResidualConnnecionne (é!"))
eing
ResiduallEncoder// ((élm,, yilm))
for i +- to HW do
* The following is performed in parallel P. = "PPresNet(e," img
[1a depth
,0, deplhy Nettt,
[agcalee ouculll] ZScaleNet(e, img
[ecghi o ZShiftNet(e,""
img
pres Bern(p;)
depih
[(, depth
o 2)
2;
Nu oi 2) 2 shit
2,
cale
Nu shifl
o 2
/* Rescale local shift to global shift as in SPAIR
2.cale a (2,calo)
shift (2,th)e
2, where = [2لcاle 2;
shint
Extract glimpses with Spatial Transformer
2; ST(x, 2 whheee)
[Rc phat a what =Giimpsenncoder(t:)
2 what ~ NN what what,2
/* Foreground mask and appearance of glimpse size
[a am GlimpseeDecoddrr(2.aall
â ait a 2eres
An
y, ā
Transform both to canvas size
á ST
ST-1(aatt (á 2 where)
y. ST (y. 2. where)
end
Compute weights for each component
w softmax(-100- (2 @ a
/* Compute global weighted mask and foreground appearance sum(w w 3 a
sum w @ "
16

---

Published as a conference paper at ICLR 2020
Algorithm 2: Rescale Iain
Input: Shift latent z, cell index i, grid height H and width Output: Rescaled shift latent 2,th
/* Get width and heigh index of cell
[k = 1% H]
** Center of this cell
C, +.5,j
Get global shift
2 shift c, + tanh(z
* No Normalize to range (-1, 1)
shift 2 21th / -1
Algorithm 3: Background Inference
Input: image æ, initial LSTM states ho: Co. initial dummy mask zi bg Output: background masks Th, appearance Mk for k = 1.....
ing ImaaeEEncoderB((z)
for k 1 to K do
hg, C2 = LSTM((Z el Ck- 1k-1,
[1 0x = PredictMask(hh)
z . (N x :
/* Actually decoded in parallel
4g MaskDecoderr22")
* Stick breaking process as described in GENESIS
TR = SBP(#1:*)
[M] of CompEncoder((ma.0)) mn)
NHHH 3 1
(CopDDecoee((z))
end
Algorithm 4: Background Generation
Input: initial LSTM states ho, Co, initial dummy mask 21t by Output: background masks mıs appearance 2e for l 1.....A K
for k 4 to do
Ck-1, h4. C2 ,.
[M o PredictMaskPrior((t)))
n ~ N( m.2
/- Actual ly decoded in parallel
44 = eeod
Stick breaking process as described in GENESIS
Th $
[14 of PredietComp(z?))
2 N( ..3 2.2
14
(CompDecode((2))
end
17

---

Published as a conference paper at ICLR 2020
D.2 TRAINING REGIME AND HYPERPARAMETERS
For all experiments we use an image size of 128 128 and batch size of 12 to 16 depending on memory usage. For the foreground module, we use the RMSProp (Tieleman & Hinton. (2012)
optimizer with a learning rate of 1 x 10-5 except for Figure for which we use a learning rate of 10- as SPAIR. For the background module, we use the Adam (Kingma & Ba 2014)) optimizer with a learning rate of 10-3, We use gradient clipping with a maximum norm 1.0. For quantitative results, SPACE is trained up to 160000 steps. For Atari games, we find it beneficial to set a to be fixed for the first several thousand steps, and vary the actual value and number of steps for different games. This allows both the foreground as well as the background module to learn in the early stage of training.
We list out hyperparameterss for 3D large dataset and joint training for 10 static Atari games below. Hyperparameters for other experiments are similar, but are finetuned for each dataset indi- vidually. In the tables below, (m - n) (p q) denotes annealing the hyperparameter value from m to n, starting from step p until step
3D Room Large
Name
Symbol
Value
ne
prior prob
(0.1 + 0.01) (4000 + 10000)
sccale
prior mean
ceale
(-1.0 + -2.) (10000 - 20000)
zscale prior stdev
scate
0.1
zshin prior
shinn shin
N(0, I)
z depph prior
depphh deppt
NO..
z what prior
1 what o what
I)..
foreground stdev
ofg
0.15
background stdev
gbg
0.15
component number
K
gumbel-softue softmaxx temperature #steps to fix
(2.5 - 0.5) (0 20000)
N/A
fixed a value
N/A
boundary loss
Yes
turn off boundary loss at step
100000
Joint Training on 10 Atari Games
Name
Symbol
Value
pres
prior prob
p
x 10 10-10 10 -2.5
scale
prior mean
p ceale
seale
prior stdev
o scale
0.1
z shin
prior
shin
o shif
N(O. I)
z depth prior
depah depth
N I)
prior
1 what
what
NOO, I)
foreground stdev
fR
0.20
background stdev
bg
0.10
component number
K
3
gunbel-softmax temperature #steps to fix a fixed value boundary loss turn off boundary loss at step
(2.5
1.0) (0 -> 10000)
4000
0.1
No
N/A
D.3 MODEL. ARCHITECTURE
Here we describe the architecture of our 16 x 16 SPACE model. The model for 8 x grid cells is the same but with stride-2 convolution for the last layer of the image encoder.
18

---

Published as a conference paper at ICLR 2020
All modules that output distribution parameters implemented with either single fully nected layer or convolution layer, with the appropriate output size. Image encoders are fully convo- lutional networks that output feature map of shape H x W, and the glimpse encoder comprises of convolutional layers followed by a final linear layer that computes the parameters of a Gaussian distribution. For the glimpse decoder of the foreground module and the mask decoder of the back- ground module we use the sub-pixel convolution layer (Shi et al. (2016)). On the lines of GENESIS (Engelcke (2019) and IODINE (Greff al. (2019), we adopt Spatial Broadcast Network Watters et al. (2019)) as the component decoder to decode z into background components.
For inference and generation of the background module, the dependence of z on zi.k-1 im- plemented with LSTMs, with hidden sizes of 64. Dependence of z on z is implemented with MI.P with two hidden layers with 64 units per layer. We apply softplus when computing standard deviations for all Gaussian distributions, and apply sigmoid when computing reconstruction and masks. We use either Group Normalization (GN) (Wu He ((08)) and CELU (Barron (2017) or Batch Normalization (BN) [loffe & Szegedy (2015)) and ELU (Clevert al. (2016)) depending the module type.
The rest of the architecture details are described below. In the following tables, ConvSub(n) denotes a sub-pixel convolution layer implemented as a stride- convolution and PyTorch PixelShuffle(n) layer, and GN(n) denotes Group Normalization with groups.
Name
Value
Comment
depth dim
2 gres dim
scale dim
for and axis for and axis
shift dim
what dim
32
z" dim
32
z dim
32
glimpse shape
(32, 32)
for attt a
image shape
(128, 128)
Foreground Image Encoder
Layer
Size/Ch.
Stride
Norm/Act.
Input
Conv 4 4
16
GN(4WCELU
Conv 4 4
32
/N(L)/
Conv 4 4
64
GN(8)CELU
Conv 3 3
128
GN(16/CELU
Conv 3
256
GN(32/CELU
Conv 1
128
GN(16/CELU
Residual Connection
Layer
Size/Ch.
Stride
Norm/Act.
Input
128
Conv 3 3 128
GN(16/CELU
Conv 3 x 3
128
GN(16)CELU
Residual Encoder
Layer
Size/Ch.
Stride
Norm//Att.
Input
128 + 128
Conv 3 3
128
GN(16/CELU
19

---

Published as a conference paper at ICLR 2020
Glimpse Encoder
Layer
Size/Ch.
Stride
Norm/Act.
Input
Conv 3 3
GN(4)/CELU
Conv 4 4
32
GN(8)CELU
Conv 3 x 3
32
GN(4)CCELU
Conv
GN(S)CELU
Conv 4 4
128
GN(8)/CELU
Conv
256
GN(16/CELU
Linear
32 + 32
Glimpse Decoder
Layer
Size/Ch.
Stride
Norm./Act.
Input
32
Conv x 1
256
GN(16)CEEU
ConvSub(2)
128
I GN(L6)/EELU
Conv x
128
GN(16)CELU
ConvSub(2)
128
I
GN(16)CELU
Conv x
128
I
GN(16)/CELU
ConvSub(2)
64
I
GN(8)CELU
Conv x
64
GN(8)CELU
ConySub(2)
32
I
GN(8YCELU
Conv x
32
I
GN(8YCELU
ConvSub(2)
16
I
GN(4)CELU
Conv 3 x 3
16
GN(4)CELU
Background Image Encoder
Layer
Size/Ch.
Stride
Norm./Act.
Input 3
Conv 3 x
64
2
BN/ELU
Conv 3 x
64
BN/ELU
Conv 3 x
64
2
BN/ELU
Conv 3 x
64
2
BN/ELU
Flatten
Linear
64
ELU
Mask Decoder
Layer
Size/Ch.
Stride
Norm./Act.
Input
32
Conv 1 x 1
256
1
GN(16)CELU
ConvSub(4)
256
I
GN(16)CELU
Conv 3 x 3
256
1
GN(16)CELU
ConvSub(2)
128
I
GN(16)CELU
Conv 3 x
128
I
GN(16)CELU
ConvSub(4)
64
I
GN(8YCELU
Conv 3 x
64
GN(8/CELU
ConvSub(4)
16
I
GN(4NCELU
Conv 3 x
16
GN(4/CELU
Conv 3 x 3
16
I
GN(4)CELU
Conv x 1
20

---

Published as a conference paper at ICLR 2020
Component Encoder
Layer
Size/Ch.
Stride
Norm//Act.
Input
3+1 (RGB+mask)
Conv 3 x
32
2
BN/ELU
Conv 3
32
2
BN/ELU
Conv 3 x 3
64
2
BN/ELU
Conv 3
64
2
BN/ELU
Flatten
Linear
32+32
Component Decoder
Layer
Size/Ch.
Stride
Norm//Act.
Input
32 (1d)
Spatial Broadcast
32+2 (3d)
Conv 3
32
BN/ELU
Conv 3 x 3
32
BN/ELU
Conv x
32
BN/ELU
Conv 3 x 3
3
D.4 BASELINES
Here we give out the details of the background decoder in training of SPAIR (both full image as well as patch-wise training). The foreground and background image encoder is same as that of SPACE with the only difference that the inferred latents are conditioned on previous cells' latents as described in Section 2.2 For the background image encoder, we add an additional linear layer so that the encoded background latent is one dimensional.
SPAIR Background Decoder
Layer
Size/Ch.
Stride
Norm./Act.
Input
32
Conv x 1
256
GN(16)CELU
ConvSub(4)
256
I
GN(16)CELU
Conv x
256
I
GN(16)CELU
ConvSub(4)
128
I
GN(16)CELU
Conv x
128
I
GN(16)CCELU
ConvSub(2)
64
I
GN(8)CELU
Conv x
64
I
GN(8)CELU
ConvSub(4)
16
I
GN(4)CELU
Conv 3 x 3
16
I
GN(4YCELU
Conv 3 x 3
16
I
GN(4/CELU
Conv 3 x 3 3
1
SPAIR Background Encoder For Patch Training
Layer
Size/Ch.
Stride
Norm//Act.
Input
Conv 2 x 2
16
2
GN(4/CELU
Conv 2
32
GN(8/CELU
Conv 2 x 2
64
2
GN(8/CELU
Conv 2
128
2
GN(16)CELU
Conv 2 x 2
32
2
GN(4/CELU
21

---

Published as a conference paper at ICLR 2020
SPAIR Background Decoder For Patch Training
Layer
Size/Ch.
Stride
Norm/Act.
Input
16
Cony x 1
256
GN(16)CELU
Conv
2048
ConvSub(4)
128
/N(16/CELU GN(16)CELU
Conv 3
128
Conv x 1
256
ConvSub(2)
64
GN(8)CELU
Conv 3 x 3
64
GN(8)CCLUU
Conv
256
ConvSub(4)
16
GN(4)CELU GN(4)CCLU GN(4)CELU
Conv 3
Conv 3 3
16
Conv x 3
For IODINE, we use our own implementation following the details as described (Greff et al. 2019 For GENESIS, we also use our own implementation following the same architecture in (Engelcke al. 2019), but the details of individual networks are similar to SPACE's background module.
E DATASET DETAILS
Atari. For each game, we sample 60,000 random images from pretrained agent (Wu et al. (016) We split the images into 50,000 for the training set, 5,000 for the validation set, and 5,000 testing set. Each image is preprocessed into size of 128 x 128 pixels with BGR color channels. We present the results for the following games: Space Invaders, Air Raid, River Raid, Montezuma's Revenge.
We also train our model on a dataset of 10 games jointly, where we have 8,000 training images, 1,000 validation images, and 1,000 testing images for each game. We the following games: Asterix, Atlantis, Carnival, Double Dunk, Kangaroo, Montezuma Revenge, Pacman, Pooyan, Qbert, Space Invaders.
Room 3D. We use MuloCo (Todorov et al. 2012) to generate this dataset. Each image consists of a walled enclosure with random number of objects the floor. The possible objects randomly sized spheres, cubes, and cylinders. The small 3D-Room dataset has 4-8 objects and the large 3D- Room dataset has 18-24 objects. The color of the objects are randomly chosen from different colors and the colors of the background (wall, ground, sky) are chosen randomly from different colors. The angle of the camera also selected randomly. We use training set 63,000 images, a validation set of 7,000 images, and a test set of 7,000 images. We use 2-D projection from the camera to determine the ground truth bounding boxes of the objects so that we can report the average precision of the different models.
22


Reviews:
[{"review_id": "r1g_bxtAYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Positives:\n+The system makes sense and is explained well\n+The factoring of scenes into objects and multiple background components is good\n+I think overall the experiments are reasonable, although I have a number of questions about whether aspects of them are apples-to-apples\n\nNegatives:\n-Some of the experiments do not appear apples-to-apples\n-There are a large number of changes, and there aren't any ablations. It's a little hard to follow and verify that the gains are credited properly. \n\nOverall, I'm favorably inclined towards accepting this paper so long as the experiments are more clearly made apples to apples. Right now, since I'm forced to give a binary decision and I'm not positive about comparisons, I have to lean towards rejection -- I'd peg my actual rating as 4.5.\n\nMethod: \n+The method is well-explained and straight-forward (in a good way). \n+The factoring of scenes into objects and multiple background components is good\n+The parallelization is good, and the fact that it works far faster than SPAIR with similar results is quite nice\n\nExperiments:\n+Overall the experiments are pretty good and compare against the baselines I would expect, and have both qualitative and quantitative results.\n+The method appears to do a good job of segmenting the objects, and if Figure 1 is representative, this is quite impressive. \n-Why does Figure 1 show results from different systems on different images? This makes comparison impossible. Paired samples are always more informative.\n\n-It's not clear to me that fair comparisons were done, especially to GENESIS. \n(a) It's never listed how K for genesis was picked -- this should presumably be tuned somewhere to optimize performance. The paper mentions in the 4.2 that it was impossible to run the experiments for GENESIS for more than 256 components -- but the GENESIS paper has numbers more like K=9. If there are an overabundance of components, this might explain some of the object splitting observed in the paper.\n(b) Unless I'm missing something, in Figure 5, for 4x4 and 8x8, it doesn't appear that IODOINE or GENESIS have converged at all. Does the validation MSE just then flatten (or go up) there? This is also wall-clock, so I'm not sure why things would stop there. This seems to conflate training speed with performance (although also note that the wall clock times being discussed are pretty small -- the rightmost side of the graph is ~14 hours -- hardly a burdensome experiment).\n(c) Similarly, for 16x16 cells, SPAIR seems to be improving consistently. Is it being cut off?-Figure 5 -- The caption for the figure things appears to not make sense: GENESIS is listed as having K = HxW+5 components and SPACE has K=5 listed. Neither make sense to me. Are they out of order?\n(d) For SPAIR in Table 1, it's not clear whether it's the slow SPAIR that was mentioned previously or the fast one (e.g., the predicted boxes are described as the same quality as SPAIR -- but is it the slow SPAIR or fast one?). I think the paper would benefit from being a bit clearer about this. I get that the parallel decomposition, in some sense, may be necessary to get any results. But I wish the paper were a bit more explicit.\n\n\n-There are some reasonable results. I realize that there isn't existing ground truth on atari and other games, but why not label a few hundred frames manually? \n-It would have been nice to have an ablation of some of the components, including the boundary loss. Unfortunately, there's a complex multi-part system and it's not clear how to break off components apart for reuse elsewhere.\n\nSmall stuff that doesn't affect my review:\n1) Figure 5 -- the figure text size is tiny and should be fixed. \n2) Eqn 3, subscript of the product \"i\" -> \"i=1\"\n3) Table 1 -- captions on tables go on top\n4) Now that the systems work like this, I'd encourage the authors to go and try stuff on more realistic data.\n5) I would be a little wary of making a big deal out of the discovery of the Montezuma's Revenge key. I realize this is indeed important, but I don't see why something like the slic superpixel objective, or felzenswalb-huttenlocher wouldn't find it either. I think it's great that there's a setting in terms of network capacity (for fg/bg networks) that yields this result, but this seems to depend heavily on the particular networks used for each of the parts of the method, and not on the general method. Also, it seems largely a function of the fact that they're a small region with a different color.\n\n-----------------------------------------\n\nPost-rebuttal update: I have read the authors' response and I am happy to increase my rating to 6.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkglMXumjr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of unsupervised scene decomposition with a foreground-background probabilistic modeling framework. Building upon the idea from the previous work on probabilistic scene decomposition [Crawford & Pineau 2019], this paper further decomposes the scene background into a sequence of background segments. In addition, with the proposed framework, scene foreground-background interactions are decoupled into foreground objects and background segments using chain rules. Experimental evaluations have been conducted on several synthetic datasets including the Atari environments and 3D-Rooms. Results demonstrate that the proposed method is superior to the existing baseline methods in both decomposing objects and background segments.\n\nOverall, this paper studies an interesting problem in deep representation learning applied to scene decomposition. Experimental results demonstrated incremental improvements over the baseline method [Crawford & Pineau 2019] in terms of object detection. However, reviewer has a few questions regarding the intuition behind the foreground-background formulation and the generalization ability to unseen combinations or noisy inputs.\n\n== Qualitative results & generalization ==\nThe qualitative improvements over the baseline method [Crawford & Pineau 2019] seem not very impressive (Figure 1: only works a bit better with cluttered scenes). First, how does the proposed method perform in real world datasets (Outdoor: KITTI, CItyscape; Indoor: ADE20K, MS-COCO)? Second, the generalization to unseen scenarios are mentioned in the introduction but not really carefully studied or evaluated in the experiments. For example, one experiment would be to train the framework on the current 3D-Rooms dataset but then test on new environments (e.g., other room layout) or new objects (e.g. other shapes such as shapenet objects). \n\n\n== Application beyond object detection ==\nEquation (4) does not seem to be natural in practice: basically, the background latents depends on the foreground object latents. Alternatively, you can assume them to be independent with each other. It’s better to clarify this point in the rebuttal. As this is a generative model, reviewer would like to know the applicability to other tasks such as pure generation, denoising and inpainting. For example, how does the pre-trained model perform with noisy input (e.g., white noise added to the image)? Also, what’s the pure generation results following the chain rules given by Equation (1), (3) & (4).\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xhornvcH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nIn this paper, the authors propose a generative latent variable model, which is named as SPACE, for unsupervised scene decomposition. The proposed model is built on a hierarchical mixture model: one component for generating foreground and the other one for generating the background, while the model for generating background is also a mixture model. The model is trained by standard ELBO with Gumbel-Softmax relaxation of the binary latent variable. To avoid the bounding box separation, the authors propose the boundary loss, which will be combined with the ELBO for training. The authors evaluated the proposed on 3D-room dataset and Atari. \n\n\nThere are several issues need to be addressed:\n\n\n1, The organization of the paper should be improved. For example, the introduction to the generative model is too succinct: the spatial attention model did not be introduced in main text. Why this model is call 'spatial attention' is not clear to me. The boundary loss seems an important component, however, it is never explicitly presented. \n\n2, The parallel inference is due the mean-field approximation, in which the posterior is approximated with factorized model, therefore, the flexibility is restricted. This is a trade-off between flexibility and parallel inference. The drawback of such parametrization should be explicitly discussed. I was wondering is there any negative effect of such the approximated posterior with fully factorized model comparing to the SPAIR?\n\n3, The empirical evaluation is not convincing. The quality illustration in Fig.1, 2 and 3 uses different examples for different methods. This cannot demonstrate the advantages of the proposed model. The quantitative evaluation only shows one baseline, SPAIR, in Table 1, and other baselines (IODINE and GENESIS) are missing. With such empirical results, the performances of the proposed method are not convincing. \n\nIn sum, I think this paper is not ready to be published. \n\n====================================================================\n\nI have read the authors' reply and the updated version. I will raise my score to 6. \n\nAlthough the mean-field inference is standard, the model in the paper looks still interesting and the performances are promising. \n\nI expect the boundary loss should be specified formally in the final version.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJep1tq-qr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes SPACE: a generative latent variable models for scene decomposition (foreground / background separation and object bounding box prediction). The authors state the following contributions relative to prior work in this space: 1) ability to simultaneously perform foreground/background segmentation and decompose the foreground into distinct object bounding box predictions, 2) a parallel spatial attention mechanism that improves the speed of the architecture relative to the closest prior work (SPAIR), 3) a demonstration through qualitative results that the approach can segment into foreground objects elements that remain static across observations (e.g. the key in Montezuma's Revenge).\n\nThe proposed model is evaluated on two sets of datasets: recorded episodes from subsets of the Atari games, and \"objects in a 3D room\" datasets generated by random placement of colored primitive shapes in a room using MuJoCo.  Qualitative results demonstrate the ability of the proposed model to separate foreground from background in both datasets, as well as predict bounding boxes for foreground objects.  The qualitative results show comparisons against SPAIR, as well as two mixture-based generative models (IODINE and GENESIS), though mostly not for direct comparisons on the same input.  Quantitative results compare the proposed model against the baselines in terms of: gradient step timings, and convergence plots of RMSE of reconstruction against wall clock time, and finally on object bounding box precision and object count error in the 3D room dataset.\n\nThe key novelty of the proposed model is that it decomposes the foreground latent variable into a set of latents (one for each detected object), and attends to these in parallel. This leads to improved speed compared to SPAIR, as demonstrated by the gradient step timings. I am convinced that the proposed model is asymptotically faster than SPAIR. However, the fact that the timings are only reported for the gradient step and not more comprehensively for entire training and inference step, is unsatisfying.  I found the qualitative comparisons to be confusing as they were mostly for different input frames, making it hard to have a direct comparison of the quality between the proposed method and baselines.  Moreover, the quantitative results reporting bounding box precision are confusing. Why report precision at exactly IoU = 0.5 and IoU in [0.5, 0.95] instead of the more standard precision at IoU >= 0.5 (and higher threshold values such as 0.95)?  The differences in the reported results seem relatively small and in my opinion, not conclusive given the above unclear points.\n\nDue to the above weaknesses in the evaluation, I am not fully convinced that the claimed contributions are substantiated empirically.  Thus I lean towards rejection.  However, since I am not intimately familiar with the research area, I am open to being convinced by other reviewers and the authors about the conceptual contributions of the model.  As it stands, I don't think this contribution is strong enough to merit acceptance.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper makes a reasonable contribution to generative modeling for unsupervised scene decomposition.  The revision and rebuttal addressed the primary criticisms concerning the qualitative comparison and clarity, which caused some of the reviewers to increase their rating.  I think the authors have adequately addressed the reviewer concerns.  The final version of the paper should still strive to improve clarity, and strengthen the evaluation and ablation studies.
Decision: Accept (Poster)