GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
Authors: Martin Engelcke
Adam R. Kosiorek
Oiwi Parker Jones
Ingmar Posner
Venue:  (2020)
Keywords: Generative modelling
object-centric representations
scene generation
variational inference

Abstract: Generative latent-variable models are emerging as promising tools in robotics and reinforcement learning. Yet, even though tasks in these domains typically involve distinct objects, most state-of-the-art generative models do not explicitly capture the compositional nature of visual scenes. Two recent exceptions, MONet and IODINE, decompose scenes into objects in an unsupervised fashion. Their underlying generative processes, however, do not account for component interactions. Hence, neither of them allows for principled sampling of novel scenes. Here we present GENESIS, the first  object-centric generative model of 3D visual scenes capable of both decomposing and generating scenes by capturing relationships between scene components. GENESIS parameterises a spatial GMM over images which is decoded from a set of object-centric latent variables that are either inferred sequentially in an amortised fashion or sampled from an autoregressive prior. We train GENESIS on several publicly available datasets and evaluate its performance on scene generation, decomposition, and semi-supervised learning.

Published as a conference paper at ICLR 2020
GENESIS: GENERATIVE SCENE INFERENCE AND SAMPLING WITH OBJECT-CENTRIE LATENT REPRESENTATIONS
Martin Engelcke Adam R. Kosiorek V Oiwi Parker Jones Ingmar Posner Applied Al Lab, University of Oxford; Dept. of Statistics, University of Oxford
ABSTRACT
Generative latent-variable models are emerging as promising tools in robotics and reinforcement learning. Yet, even though tasks in these domains typically involve distinct objects, most state-of-the-art generative models do not explicitly capture the compositional nature of visual scenes. Two recent exceptions, MONet and IODINE, decompose scenes into objects in an unsupervised fashion. Their under- lying generative processes, however, do not account for component interactions. Hence, neither of them allows for principled sampling of novel scenes. Here we present GENESIS, the first object-centric generative model of rendered 3D scenes capable of both decomposing and generating scenes by capturing relationships between scene components. GENESIS parameterises a spatial GMM over images which is decoded from a set of object-centric latent variables that are either in- ferred sequentially in an amortised fashion or sampled from an autoregressive prior. We train GENESTS on several publicly available datasets and evaluate its performance on scene generation, decomposition, and semi-supervised learning.
INTRODUCTION
Task execution in robotics and reinforcement learning (RL) requires accurate perception of and rea- soning about discrete elements in an environment. While supervised methods can be used to identify pertinent objects, is intractable collect labels for every scenario and task. Discovering structure in data-such as objects-an and learning represent data in a compact fashion without supervision are long-standing problems in machine learning (Comon 1992 Tishby al. 22000), often formu- lated as generative vatent-variable modelling (c.g. Kingma & Welling |014| Rezende et al., 2014). Such methods have been leveraged to increase sample efficiency in RL (Gregor et al.) 2019) and other supervised tasks (van Steenkiste al. 2019). They also offer the ability imagine environments for training (Ha & Schmidhuber] 2018). Given the compositional nature of visual scenes, sepa- rating latent representations into object-centric ones can facilitate fast and robust learning (Watters) et al. 2019a), while also being amenable to relational reasoning (Santoro et al. (017). Interestingly, however, state-of-the-art methods for generating realistic images do account for this discrete
structure (Brock et al.) 2018 Parmar et al. 2018).
As in the approach proposed in this work, human visual perception is not passive. Rather it involves a creative interplay between external stimulation and an active, internal generative model of the world (Rao & Ballard 1999 Friston| 2003). That this necessary can seen from the physiology of the eye, where the small portion of the visual field that can produce sharp images (fovea centralis) motivates the need for rapid eye movements (saccades) to build up a crisp and holistic percept of scene (Wandell 1995). In other words, what we perceive is largely mental simulation of the external world. Meanwhile, work in computational neuroscience tells us that visual features (see, e.g., Hubel & Wiesel 1968) can be inferred from the statistics of static images using unsupervised learning (Olshausen & Field 1996). Experimental investigations further show that specific brain areas (e.g. LO) appear specialised for objects, for example responding objects than to scenes or textures, while responding only weakly to movement (cf. MT) (e.g., Grill-
strongly to common
Spector Malach 2004).
CCorresoondigg author: marti n roboottt....... ox. ac.uk

---

Published as a conference paper at ICLR 2020
In this work, we are interested in probabilistic generative models that can explain visual scenes compositionally via several latent variables. This corresponds to fitting a probability distribution po(x) with parameters 0 to the data. The compositional structure is captured by K latent variables so that po (x)) [ 21:K (2o(Z::)) dzı.k. Models from this family can be optimised using the variational auto-encoder (VAF) framework (Kingma & Welling. 2014 Rezende al.) 2014), by maximising a variational lower bound on the model evidence (Jordan et 1999). Burgess et al. (2019) and Greff et al. (2019) recently proposed two such models, MONet and IODINE, decom- pose visual scenes into meaningful objects. Both works leverage an analysis-by-syrahest by-synthessiss approach through the machinery of VAES (Kingma & Welling 2014,] Rezende et al., 2014) train these mod- els without labelled supervision, e.g. in the form of ground truth segmentation masks. However, the models have a factorised prior that treats scene components as independent. Thus, neither provides an object-centric generation mechanism that accounts for relationships between constituent parts of a scene, c.g. two physical objects cannot occupy the same location, prohibiting the component- wise generation of novel scenes and restricting the utility of these approaches. Moreover, MONet embeds a convolutional neural network (CNN) inside of an recurrent neural network (RNN) that is unrolled for each scene component, which does not scale well to more complex scenes. Similarly, IODINE utilises a CNN within an expensive, gradient-based iterative refinement mechanism.
Therefore, we introduce GENErative Scene Inference and sampling (GENESIS) which is, the best of our knowledge, the first object-centric generative model of rendered 3D scenes capable of both de- composing and generating scenes" Compared to previous work, this renders GENESIS significantly more suitable for a wide range of applications in robotics and reinforcement learning. GENESIS achieves this by modelling relationships between scene components with an expressive, autoregres- sive prior that is learned alongside sequential, amortised inference network. Importantly, sequen- tial inference is performed in cow-dimesioonal latent space, allowing all convolutional encoders encoderss and decoders to be run in parallel to fully exploit modern graphics processing hardware.
We conduct experiments on three canonical and publicly available datasets: coloured Multi-dSprites (Burgess et al. 2019), the GQN dataset (Eslami et al., 2018), and ShapeStacks (Groth et al., 2018)) The latter two are simulated 3D environments which serve as testing grounds for navigation and object manipulation tasks, respectively. We show both qualitatively and quantitatively that in con- trast to prior art, GENESIS is able to generate coherent scenes while also performing well on scene decomposition. Furthermore, benefit of utilising general purpose, object-centric latent representations from GENESIS for tasks such predicting whether block tower is stable not.
use the scene annotations available for ShapeStacks to show the
Code and models are available at htttt::t///ggtt httt:s////gttuubb.cm/////ppppppp ai-laaddd---laaageeeeeeeee genesis
2 RELATED WORK
Structured Models Several methods leverage structured latent variables to discover objects in im-
ages without direct supervision. CST-VAE (Huang & Murphy 2015). AIR (Eslami al.) 2016),
SQAIR (Kosiorek et al. 2018), and SPAIR (Crawford & Pineau 2019) use spatial attention to par- tition scenes into objects. TAGGER (Greff et al. 2016), NEM (Greff al. 2017), and R-NEM (van)
Steenkiste et al. 2018a) perform unsupervised segmentation by modelling images as spatial mixture models. SCAE (Kosiorek et al. 2019) discovers geometric relationships between objects and their parts by using an affine-aware decoder. Yet, these approaches have not been shown to work on more complex images, for example visual scenes with 3D spatial structure, occlusion, perspective distor- tion, and multiple foreground and background components as considered in this work. Moreover, none of them demonstrate the ability to generate novel scenes with relational structure.
While Xu et al. (2018) present extension of Eslami et al. (2016) to generate images, their method only works on binary images with a uniform black background and assumes that object bounding boxes do not overlap. In contrast, train GENESIS rendered 3D scenes from Eslami et al. (2018) and Groth et al. (018)) which feature complex backgrounds and considerable occlusion to perform both decomposition and generation. Lastly, Xu al. (2019) use ground truth pixel-wise flow fields as a cue for segmenting objects or object parts. Similarly, GENESIS could be adapted to also leverage temporal information which is a promising avenue for future research.
We use the terms "object" and "scene component" synonymously in this work.
2

---

Published as a conference paper at ICLR 2020
MONet & IODINE While this work is most directly related to MONet (Burgess et al. 2019) and IODINE (Greff et al., 2019). it sets itself apart by introducing a generative model that captures rela- tions between scene components with an autoregressive prior, enabling the unconditional generation of coherent, novel scenes. Moreover, MONet relies on a deterministic attention mechanism rather than utilissigg proper probabilistic inference procedure. This implies that the training objective is not valid lower bound on the marginal likelihood and that the model cannot perform density estimation without modification. Furthermore, this attention mechanism embeds a CNN in a RNN, posing an issue in terms of scalability. These two considerations do not apply to IODINE, but IODINE employs a gradient- based, iterative refinement mechanism which expensive both in terms of com- putation and memory, limiting its practicality and utility. Architecturally, GENESTS is more similar to MONet and does not require expensive iterative refinement as IODINE. Unlike MONet, though, the convolutional encoders and decoders in GENESIS can be run in parallel, rendering the model computationally more scalable to inputs with a larger number of scene components.
Adversarial Methods A few recent works have proposed to use an adversary for scene segmenta- tion and generation. Chen et al. (2019) and Bielski & Favaro (2019) segment a single foreground object per image and Arandjelović & Zisserman (2019) segment several synthetic objects superim- posed on natural images. [Azadi et al. (2019) combine two objects or an object and a background scene in sensible fashion and van Steenkiste et al. (2018b) can generate scenes with a potentially arbitrary number of components. In comparison, GENESIS performs both inference and generation, does not exhibit the instabilities of adversarial training, and offers a probabilistic formulation which captures uncertainty, e.g. during scene decomposition. Furthermore, the complexity of GENESIS increases with (K), where K' is the number of components, as opposed to the OK²)) complexity of the relational stage in van Steenkiste et al. (0188)).
Inverse Graphies range of works formulate scene understanding as an inverse graphics problem. These well-engineered methods, however, rely on scene annotations for training and lack probabilis- tic formulations. For example, Wu al. (2017b) leverage graphics renderer to decode a structured scene description which is inferred by a neural network. RRomasko et al. (2017) pursue a similar approach but instead make use of a differentiable graphics render. Wu et al. (2017a) further employ different physics engines to predict the movement of billiard balls and block towers.
GENESIS: GENERATIVE SCENE INFERENCE AND SAMPLING
In this section, we first describe the generative model of GENESIS and simplified variant called GENESIS-S. This is followed by the associated inference procedures and two possible learning objectives. GENESIS is illustrated in Figure aan Figure the graphical model in comparison to alternative methods. An illustration of GENESIS-S is included Appendix B.I Figure
Generative model Let x E #C be an image. We formulate the problem of image genera- tion spatial Gaussian mixture model (GMM). That is, every Gaussian component & 1.....A represents an image-sized scene component X E CC K E N4 is the maximum number of scene components. The corresponding mixing probabilities T [0, 1] 1\ indicate whether the component is present at a location in the image. The mixing probabilities are normalised across scene components, i.e. V Σn Tiyjk 1, and can be regarded as spatial attention masks. Since there are strong spatial dependencies between components, we formulate an autoregressive prior distribution over mask variables 2 R.... which encode the mixing probabilities TT.
K
K
(e(z)Tk) = II ( z
II A=1
(Z( (Z
(s)) ms Ro
(1)
The dependence on previous latents zii is implemented via an RNN Rs with hidden state up-
Next, we assume that the scene components x₃ are conditionally independent given their spatial allo- cation in the scene. The corresponding conditional distribution over component variables z E R De which encode the scene components x factorises as follows,
K
po(zi.x
(z
zxz
(2)
R=
3

---

Published as a conference paper at ICLR 2020
x
1
1i X1
KK
XX
Figure I: GENESIS illustration. Given an image x. an encoder and an RNN compute the mask latents z These are decoded to obtain the mixing probabilities Ty. The image and individual masks are concatenated infer the component latents z from which the scene components X₃ are decoded.
Now, the image likelihood is given by mixture model,
p(x ziK: zikk)
Σ Th Po x
(3)
where the mixing probabilities मą = To (::e)) are created via stick-breaking process (SBP) adapted from Burgess et al. (219)) follows, slightly overloading the notation,
K
M1 mo(2?")
TT
Σ JJ
To (z
mKK
1
Note that this step is not necessary for model and instead one could use softmax normalise masks as in Greff et al. (2019).
Finally, omitting subscripts, the full generative model can be written
// po z z²,z"" ppo (z" z 1z") (Po(z")) dz dz
(x =
(5)
where we assume that all conditional distributions are Gaussian. The Gaussian components of the image likelihood have fixed scalar standard deviation o². We refer to this model as GENESIS. To investigate whether separate latents for masks and component appearances are necessary for decom- position, we consider simplified model, GENESIS-S, with single latent variable per component,
Pe (Z.
II pa (zk Z1
(6)
In this case, 2ą takes the role of o in Equation and of z in Equation while Equation no longer necessary.
is
Approximate posterior We amortise inference by using an approximate posterior distribution with parameters and structure similar the generative model. The full approximate posterior reads as follows,
go (2 (.i. z..K
x) qo (2 x) (o (x,2(K)
where
go (Z
x)
II A=1
go (2% x, -
and qo (zf. x, 2)
II
qs (2 x, 2(kk
with the dependence on z realised by an RNN Rs. The RNN could, principle, be shared with the prior, but we have not investigated this option. All conditional distributions are Gaussian. For GENESIS-S, the approximate posterior takes the form go (Z
x) I (z(2) x, 2111))-)).
4

---

Published as a conference paper at ICLR 2020
N
2
zi
21
x
ZK
VAE (b) MONET
JODINE
(d) GENESIS
(e) GENESIS-S
Figure 2: Graphical model of GENESIS compared to related methods. N denotes the number of refinment iterations in IODINE. Unlike the other methods, both GENESIS variants explicitly model dependencies between scene components.
Learning GENESIS can be trained by maximising the evidence lower bound (ELBO) on the log- marginal likelihood log (x), given by
(x) (x (>>/)>)))) log pe(x zz, zz (2o
(8)
(z n xx
(2s (x) Po z =² KL (2), n x) (2"))"""))
However, this often leads strong emphasis on the likelihood term, while allowing the marginal approximate posterior (z) (x) ( x)] to drift away from the prior distribution, hence increasing the KL-divergence. This also decreases the quality of samples drawn from the model. To prevent this behaviour, we the Generalised ELBO with Constrained Optimisation (GECO) objective from Rezende & Viola (2018) instead, which changes the learning problem to minimising the KL-divergence subject to reconstruction constraint. Let C I be the minimum allowed reconstruction log-likelihood, GECO then uses Lagrange multipliers to solve the following problem,
0* x arg min (gø(z", 2 x) po )²
0,0
(10)
such that xx)) log 2 "=,z" zn C.
EXPERIMENTS
In this section, present qualitative and quantitative results coloured Multi-dSprittt (Burgess) ct al. 2019), the "rooms-ring- cooms-ring-camera dataset from GQN (Eslami et al. 2018) and the ShapeStacks dataset (Groth al. 2018). We use image resolution of 64-by-64 for all experiments. The number of components is set to = 5, K = 7, and K = 9 for Multi-dSprites, GQN, and ShapeStacks, respectively. More details about the datasets are provided in Appendix A Implementation and training details of all models are described in Appendix
4.1 COMPONENT- WISE SCENE GENERATION
Unlike previous works, GENESIS has an autoregressive prior capture intricate dependencies be- tween scene components. Modelling these relationships is necessary to generate coherent scenes. For example, different parts of the background need fit together; do not want to create com- ponents such as the sky several times; and several physical objects cannot be in the same location. GENESIS is able to generate novel scenes by sequentially sampling scene components from the prior and conditioning each new component those that have been generated during previous steps.
After training GENESIS and MONet on the GQN dataset, Figure 3 shows the component-by- component generation process of novel scenes, corresponding to drawing samples from the respec- tive prior distributions. More examples of generated scenes are shown in Figure [6] Appendix [ With GENESIS, either object in the foreground or part of the background is generated at every step and these components fit together to make up a semantically consistent scene that looks sim- ilar to the training data. MONet, though, generates random artefacts at every step that do not form a sensible scene. These results are striking but not surprising: MONet was not designed for scene generation. The need for such model is why we developed GENESIS.
5

---

Published as a conference paper at ICLR 2020
Sample
k
k ==
k ==
k 6
k =
B
B8
Genesis: FID 80.5
RRBE
4
MONet: FID 176.4
Figure 3: Component-by-componee scene generation with GENESTS and MONet after training on the GQN dataset. The first pane shows the final scene and the subsequent panes show the components generated at each step. GENESIS first generates the sky and the floor, followed by individual objects, and finally distinct parts of the wall in the background to compose coherent scene. MONet, in contrast, only generates incomplete components that do not fit together.
Notably, GENESIS pursues consistent strategy for generation: Step one generates the floor and the sky, defining the layout of the scene. Steps two to four generate individual foreground objects. Some of these slots remain empty less than three objects present in the scene. The final three steps generate the walls in the background. We conjecture that this strategy evolves during training the floor and sky constitute large and easy model surfaces that have strong impact on the reconstruction loss. Finally, we observe that some slots contain artefacts of the sky at the top of the wall boundaries. We conjecture this due the fact that the mask decoder does not have skip connections as typically used in segmentation networks, making it difficult for the model to predict sharp segmentation boundaries. Scenes generated by GENESIS-S are shown in Figure & and Figure Appendix D While GENESIS-S does separate the foreground objects from the background, generates them one step and the individual background components are not very interpretable.

---

Published as a conference paper at ICLR 2020
4.2 INFERENCE OF SCENE COMPONENTS
Like MONet and IODINE, which were designed for unsupervised scene decomposition, GENESIS is also able segment scenes into meaningful components. Figure 4 compares the decomposition of two images from the GQN dataset with GENESIS and MONet. Both models follow similar decomposition strategy, but MONet fails to disambiguate one foreground object in the first example and does not reconstruct the background in as much detail in the second example. In Appendix E Figure TO illustrates the ability of both methods disambiguate objects of the same colour and
Figure I shows scene decomposition with GENESIS-S.
Following Greff et al. (2019), we quantify segmentation performance with the Adjusted Rand Index (ARI) of pixels overlapping with ground truth foreground objects. We computed the ARI on 300 random images from the ShapeStacks test set for five models trained with different random seeds. GENESIS achieves ARI of 0.73 + 0.03 which is better than 0.63 = 0.07 for MONet. This metric, however, does not penalise objects being over-segmented, which can give mislcading impression Appendix
with regards to segmentation quality. This is illustrated in Figure 3
Inspired by Arbelaez et al. 2010), we thus propose to use the segmentation covering (SC) of the ground truth foreground objects by the predicted masks. This involves taking weighted mean mask pairs, putting potentially undesirable emphasis on larger objects. We therefore also consider taking unweighted mean (mSC). For the same 300 images from the ShapeStacks test and five different random seeds, GENESTS (SC: 0.64 + 0.08, mSC: 0.60 + 0.09) again outperforms MONet (SC: 0.52 + 0.09, mSC: 0.49 = 0.09). More details are provided Appendix
Reconsiruction
k
k=
k=4
==
=
==
Genesis
MONet
Genesis
MONet
Figure 4: Step-by-step decomposition of the same scene from GQN with GENESIS and MONet. Unlike MONet, GENESIS clearly differentiates individual objects in the first example. In the second example, GENESIS captures the fine- mn-ggrineed pattern of the wall in the background better than MONet.
4.3 EVALUATION OF UNSUPERVISED REPRESENTATION UTILITY
Using a subset of the available labelled training images from ShapeStacks, we train set of clas- sifiers on the representations learned by GENESIS and several baselines to evaluate how well these representations capture the ground truth scene state. In particular, we consider three tasks: (1) Is a tower stable or not? (2) What is the tower's 's height in terms of the number of blocks? (3) What the camera viewpoint (out of 16 possibilities)? Tower stability is particularly interesting prop- erty as it depends on in fine-graines object information and the relative positioning of objects. We selected the third task as learning scene representations from different views has previously been prominently explored in Eslami et al. (2018). We compare GENESIS and GENESIS-S against three baselines: MONet, a VAE with a spatial broadcast decoder (BD-VAE) and a VAE with a deconvo- lutional decoder (DC-VAE). The results
summarised in Table [ The architectural details of the baselines are described in Appendix B.2 and Appendix B.3 B.3 The implementation details of the
classifiers are provided in Appendix B.5

---

Published as a conference paper at ICLR 2020
Both GENESIS and GENESIS-S perform better than than the baselines predicting tower stability and their accuracies on predicting the height of the towers is only outperformed by MONet. We conjecture that MONet benefits here by its deterministic segmentation network. Overall, this cor- roborates the intuition that object-centric representations are indeed beneficial for these tasks which focus on the foreground objects. We observe that the BD-VAE does better than the DC-VAF all three tasks, reflecting the motivation behind its design which is aimed at better disentangling the un- derlying factors of variation in the data (Watters et al. 2019b). All models achieve high accuracy at predicting the camera view. Finally, we note that none of models reach the stability prediction accu- racies reported Groth et al. (2018) which were obtained with an Inception- classifier (Szegedy) et al. 2017). This is not surprising considering that only subset the training images is used for training the classifiers without data augmentation and at a reduced resolution.
Table I: Classification accuracy in on the test sets of the ShapeStacks tasks.
Task
GENESIS
GENESIS-S
MONet
BD-VAE
DC-VAE
Random
Stability
64.0
63.2
59.6
60.1
59.0
50.0
Height
80.3
80.8
88.4
78.6
67.5
22.8
View
99.3
99.7
99.5
99.7
99.1
6.25
4.4 QUANTIFYING SAMPLE QUALITY
In order to quantify the quality of generated scenes, summarises the Fréchet Inception Dis- tances (FIDs) (Heusel et al., 2017) between 10,000 images generated by GENESIS well several baselines and 10,000 images from the Multi-dSprites and the GQN test sets, respectively. The two GENESIS variants achieve the best FID both datasets. While GENESIS-S performs better than GENESIS on GQN, Figure g and Figure 9 in Appendix D show that individual scene components are less interpretable and that intricate background patterns generated at the expense of sensible foreground objects. It is not surprising that the FIDs for MONet are relatively large given that was designed for generating scenes. Interestingly, the DC-VAE achieves smaller FID on GQN than the BD-VAE. This is surprising given that the BD-VAE representations are more useful for the ShapeStacks classification tasks. Given that the GQN dataset and ShapeStacks are somewhat simi- lar in structure and appearance, this indicates that while FID correlates with perceptual similarity, it does not necessarily correlate with the general utility of the learned representations for downstream tasks. We include scenes sampled from the BD-VAF and the DC-VAE in Figure 7 Appendix [D] where we observe that the DC-VAE models the background fairly well while foreground objects are blurry.
Table 2: Fréchet Inception Distances for GENESIS and baselines on GQN.
Dataset
GENESIS
GENESIS-S
MONet
BD-Va VAE
DC-VAE
Multi-dSprites
24.9
28.2
92.7
89.8
100.5
GQN
80.5
70.2
176.4
145.5
82.5
CONCLUSIONS
In this work, propose novel object-centric latent variable model of scenes called GENESIS. We show that GENESIS is, to the best of our knowledge, the first unsupervised model to both decompose rendered 3D scenes into semantically meaningful constituent parts, while the same time being able to generate coherent scenes in a component-wise fashion. This is achieved by capturing relationships between scene components with autoregressive prior that is learned alongside a computationally efficient sequential inference network, setting GENESIS apart from prior art. Regarding future work, an interesting challenge is scale GENESIS more complex datasets and to employ the model in robotics or reinforcement learning applications. To this end, it will be necessary to improve reconstruction and sample quality, reduce computational cost, and scale the model to higher resolution images. Another potentially promising research direction is to adapt the formulation to only model parts of the scene that are relevant for a certain task.
8

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
This research was supported by an EPSRC Programme Grant (EP/M019918/1), an EPSRC DTA studentship, and a Google studentship. The authors would like to acknowledge the use of the University of Oxford Advanced Research Computing (ARC) facility in carrying out this work, http://áx. doi. 20g//00 5281/zen.odo zen.odo... 22558, and the use of Hartree Centre resources. The authors would like to thank Yizhe Wu for his help with re-implementing MONet, Oliver Groth for his support with the GQN and ShapeStacks datasets, and Rob Weston for proof reading the paper.
REFERENCES
Relja Arandjelović and Andrew Zisserman. Object Discovery with a Copy-Pasting GAN. arXiv preprint arXiv: 1369, 2019.
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour Detection and Hier- archical Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010.
Samaneh Azadi, Deepak Pathak, Sayna Ebrahimi, and Trevor Darrell. Compositional GAN: Learn- ing Condiiooniii Binary Composition. arXiv preprint vrXiv:180..075600 2019.
Rianne van den Berg, Leonard Hasenclever, Jakub M Tomezak, and Max Welling. Sylvester Nor- malizing Flows for Variational Inference. Conference on Uncertainty Artificial Intelligence, 2018.
Adam Bielski and Paolo Favaro. Emergence of Object Segmentation in Perturbed Generative Mod- els. arXiv preprint arXiv: /905. 12663, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. arXiv preprint vrXiv::809) 10009,/106, 2018.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Trina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised Scene Decomposition and Repre- sentation, arXiv preprint 1rXiv:1901.11390 2019.
Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised Object Segmentation by Re- drawing. arXiv preprint arXiv:1905.13339) 2019.
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and Accurate Deep Network
Learning by Exponential Linear Units (ELUs). International Conference Learning Represen- tations, 2016.
Pierre Comon. Independent Component Analysis. In J-L.Lacoume (ed.), Higher-Order Statistics, pp. 38. Elsevier, 1992.
Eric Crawford and Joelle Pineau. Spatially Invariant Unsupervised Object Detection with Convolu- tional Neural Networks. AAAI Conference on Artificial Intelligence, 2019.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language Modeling with Gated Convolutional Networks. International Conference on Machine Learning, 2017.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hin- ton, et al. Attend, Infer, Repeat: Fast Scene Understanding with Generative Models. Neural Information Processing Systems, 2016.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural Scene Representation and Rendering. Science, 2018.
Karl Friston. A Theory of Cortical Responses. Philosophical Transactions of the Royal Society B: Biological Sciences, (60(1456):815-55 2005.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep Sparse Rectifier Neural Networks. Inter- national Conference Artificial Intelligence and Statistics, 2011.

---

Published as a conference paper at ICLR 2020
Klaus Greff, Antti Rasmus. Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep Unsupervised Perceptual Grouping. Neural Information Processing Systems, 2016.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural Expectation Maximization.
Neural Information Processing Systems, 2017.
Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zo- ran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-Object Representation Learning with Iterative Variational Inference. International Conference on Machine Learning, 2019.
Karol Gregor, Danilo Jimenez. Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aaron van den Oord. Shaping Belief States with Generative Environment Models for RL. arXiv preprint arXiv:1906.09237, 2019.
Kalanit Grill-Spector and Rafael Malach. The Human Visual Cortex. Annual Review of Neuro- science, 27(1):649-677, 2004.
Oliver Groth, Fabian B Fuchs, Ingmar Posner, and Andrea Vedaldi. ShapeStacks: Learning Vision- Based Physical Intuition for Generalised Object Stacking. European Conference on Computer Vision, 2018.
David Ha and Jürgen Schmidhuber. World Models. Neural Information Processing Systems, 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Neural Information Processing Systems, 2017.
Sepp Hochreiter and Jüirgnn Schmidhuber. Long Short-Term Memory, Neural Computation, 1997.
Jonathan Huang and Kevin Murphy. Efficient Inference in Occlusion-Aware Generative models of Images. arXiv preprint arXh::1511.06300 2015.
D. H. Hubel and T. N. Wiesel. Receptive Fields and Functional Architecture of Monkey Striate Cortex. The Journal of Physiology, 195(1):215-243 1968.
Sergey loffe and Christian Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. International Conference on Machine Learning, 2015.
Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An Introduction to Variational Methods for Graphical Models. Machine Learning, 37(2):183-233, 1999.
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. International
Conference on Learning Representations, 2015.
Diederik P Kingma and Max Welling. Auto-Encodin Variational Bayes. International Conference on Learning Representations, 2014.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects. Neural Information Processing Systems, 2018.
Adam R Kosiorek, Sara Sabour, Yee whye Teh, and Geoffrey Hinton. Stacked Capsule Autoen-
coders. arXiv preprint arXiv:1906.068/ 0681888 2019.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dSprites: Disentanglement 2017
Testing Sprites Dataset.
Bruno A. Olshausen and David J. Field. Emergence of Simple-Cell Receptive Field Properties by Learning Sparse Code for Natural Images. Nature, 381:607-609, 1996.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image Transformer. International Conference Machine Learning, 2018.
10

---

Published as a conference paper at ICLR 2020
Rajesh P. N. Rao and Dana H. Ballard. Predictive Coding in the Visual Cortex: A Functional Interpretation of Some Extra-Classical Receptive- Effects. Nature Neuroscience, 2(1):79- 87, 1999.
Danilo Jimenez Rezende and Fabio Viola. Taming VAEs, arXiv preprini arXiv:1810.0059)) 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. International Conference on Machine Learn- ing, 2014.
Lukasz Romaszko, Christopher KI Williams, Pol Moreno, and Pushmeet Kohli. Vision-as-Invent Graphics:
Obtaining Rich 3D Explanation of Scene from Single Image. In IEEE Interna- tional Conference on Computer Vision, 2017.
Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski. Razvan Pascanu, Peter W. Battaglia, and Timothy P. Lillicrap. A Simple Neural Network Module for Relational Reasoning. Neural Information Processing Systems, 2017.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception- Inception-Resnet and the Impact of Residual Connections on Learning. AAAI Conference on Artificial Intelligence, 2017.
Naftali Tishby, Fermando C Pereira, and William Bialek. The Information Bottleneck Method. arXiv preprint uXXiv:phyyics00004057, 2000.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions. arXiv preprint 1802 10353, 2018a.
Sjoerd van Steenkiste, Karol Kurach, and Sylvain Gelly, A Case for Object Compositionality in Deep Generative Models of Images. NeurIPS Workshop on Modeling the Physical World: Learn- ing. Perception, and Control, 2018b.
Sjoerd Steenkiste, Francesco Locatello, Jurgen Schmidhuber, and Olivier Bachem. Are Disentangled Representations Helpful for Abstract Visual Reasoning? arXiv preprint 1rXiv:1905.122:0 2019.
Brian A. Wandell. Foundations of Vision. Sinauer Associates, 1995.
Nicholas Watters, Loie Matthey, Matko Bosnjak, Christopher Burgess, and Alexander Lerchner. COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity- 2019a.
Driven Exploration. arXiv preprint
Nicholas Watters, Loic Matthey, Christopher Burgess, and Alexander Lerchner. Spatial Broadcast Decoder: Simple Architecture for Learning Disentangled Representations in VABs. arXiv
preprint arXiv:1901.07017 2019b.
Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to See Physics via Visual De-Aniiation. Neural Information Processing Systems, 2017a
Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural Scene De-rendering. IEEE Confer- on Computer Vision and Pattern Recognition, pp. 699-707, 2017b.
Kun Xu, Chongxuan Li, Jun Zhu, and Bo Zhang. Multi-Objects Generation with Amortized Struc- tural Regularization. Neural Information Processing Systems, 2018
Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Unsupervised Discovery of Parts, Structure, and Dynamies. arXiv preprint 1903v00903.55//// 2019.
11

---

Published as a conference paper at ICLR 2020
A DATASETS
Images contain between one and four randomly se- lected "sprites" from Mathhyy et al. (2017), available at /tttss://gita: com/ deepmind/ dsprites-daal dataset For each object and the background, we randomly select one of five dif- ferent, equally spread values for each of the three colour channels and generate 70,000 images. We set aside 10,000 for validation and testing each. The script for generating this data will be released with the rest of our code.
SuriiesSprites (Burgess et al., 2019)
GQN (Eslami et al. 2018) The "rooms-ring-camera" dataset includes simulated 3D scenes of a square room with different floor and wall textures, containing one to three objects of various shapes and sizes. It can be downloaded from https: com deepmind/ haapa//g--aaaaaaaa agn-datasete...
ShapeStacks (Groth et al., 2018) Images show simulated block towers of different heights (two to six blocks). Individual blocks can have different shapes, sizes, and colours. Scenes have annotations for: stability of the tower (binary), number of blocks (two to six), properties of individual blocks, locations in the tower of centre-of-mass violations and planar surface violations, wall and floor textures (five each), light presets (five), and camera view points (sixteen). More details about the dataset and download links can be found at http:::://shapstaakks robots...ooo....oo ox ac..ukkuk .uk/
B IMPLEMENTATION DETAILS
B.1 GENESIS ARCHITECTURE
We use the architecture from Berg et al. (0188 to encode and decode z with the only modification of applying batch normalisation (loffe & Szegedy 2015) before the GLU non-linearities (Dauphin et al. 2017). The convolutional layers in the encoder and decoder have five layers with size-5 kernels, strides of [1, 2, 2, 1J. and filter sizes of [32. 32, 64, 64, 64] and [64, 32, 32, 32, 32], respectively. Fully-connected layers are used at the lowest resolution.
The encoded image is passed long short-term memory (LSTM) cell (Hochreiter Schmidhuber 1997) followed by linear layer to compute the mask latents z of size 64. The LSTM state size twice the latent size. Importantly, unlike the analogous counterpart in MONet, the decoding of z is performed in parallel. The autoregressive prior Po x - is implemented as an LSTM with 256 units. The conditional distribution pe (zf, z is parameterised by multilayer perceptron (MLP) with two hidden layers, 256 units per layer, and ELUS (Clevert et al., (016)). We use the same component VAE featuring spatial broadcast decoder as MONet to encode and decode 2g but we replace RELUS GGloro al. (011) with ELUS.
For GENESIS-S. S. as illustrated in Figure 5 the encoder of Z is the same as for z above and the decoder from Berg et al. (2018) again used to compute the mixing probabilities. However, GEN- ESIS-S also has second decoder with spatial broadcasting to obtain the scene components X2 from Zk. We found the use two different decoders to be important for GENESIS-S in order for the model to decompose the input.
Z
Ti, X]
Zk
Figure 5: GENESIS-S overview. Given an image x, an encoder and an RNN compute latent variables Zj. These are decoded to directly obtain the mixing probabilities Tę and the scene components X
12

---

Published as a conference paper at ICLR 2020
B.2 MONET BASELINES
We followed the provided architectural details described in Burgess et al. (2019). Regarding un- specified details, employ an attention network with [32, 32, 64, 64, 64] filters in the encoder and the reverse in the decoder. Furthermore, we normalise the mask prior with a softmax function to compute the KL-divergence between mask posterior and prior distributions.
B.3 VAE BASELINES
Both the BD-VAE and the DC-VAE have latent dimensionality of 64 and the same encoder as in Berg al. (2118. The DC-VAE also uses the decoder from |erg et al. (2018). The BD- VAE has the same spatial broadcast decoder with ELUS as GENESIS, but with twice the number of filters to enable better comparison.
B.4 OPTIMISATION
The scalar standard deviation of the Gaussian image likelihood components is set to d, = 0.7. We use GECO (Rezende & Viola 2018) balance the reconstruction and KL divergence terms in the loss function. The goal for the reconstruction enror is set to 0.5655, multiplied by the image dimen- sions and number of colour channels. We deliberately choose comparatively weak reconstruction constraint for the GECO objective to emphasise KL minimisation and sample quality. For the remain- ining GECO hyperparameters, the default value of a 0.99 is used and the step size for updating is set to 10 We increase the step size to 10-4 when the reconstruction constraint is satisfied to accelerate optimisation as tended to undershoot at the beginning of training.
All models trained for 5 10th iterations with batch size of 32 using the ADAM optimiser Kingma & Ba. 2015) and learning rate 10-4. With these settings, training GENESIS takes about two days single GPU. However, expect performance improve with further training. This particularly extends to training GENESIS on ShapeStacks where 5 10 training iterations are enough to achieve good sample quality.
B.5 SHAPESTACKS CLASSIFIERS
Multilayer perceptrons (MLPS) with one hidden layer, 512 units, and ELU activations are used for classification. The classifiers are trained for 100 epochs 50,000 labelled examples with batch size of 128 using cross-entropy loss, the ADAM optimiser, and a learning rate of 10-4, As inputs the classifiers, concatenate z and z for GENESIS, Zţ for GENESIS-S, and the component VAE latents for the two MONet variants.
C SEGMENTATION COVERING
Following Arbelaez et al. (0010), the segmentation covering (SC) based on the intersection over union (IOU) between pairs of segmentation masks from two sets S and S'. In this work, we consider to be the segmentation masks of the ground truth foreground objects and S' to be the predicted segmentation masks. The covering of S by S' is defined as:
C(S' S) ERES
1
Σ RES
R| max (OU R'), RES"
(1I)
[R|
where |R]] denotes the number of pixels belonging to mask R. Note that this formulation is slightly more general than the one in Arbelaez al.) (2010) which assumes that masks in are non- overlapping and cover the entire image. The above takes weighted mean over IOU values, pro- portional to the number of pixels of the masks being covered, To give equal importance to masks of different sizes, also consider taking unweighted mean (mSC):
(S' + S)
Σ max TOU R, R'),
(12)
R'ES'
RES
where |S| denotes the number of non-empty masks in S. Importantly and unlike the ARI. both segmentation covering variations penalise the see-eeggmetttionn of ground truth objects as this de- creases the YOU for pair of masks. This is illustrated in Figure 13 Appendix B
13

---

Published as a conference paper at ICLR 2020
D COMPONENT- WISE SCENE GENERATION GQN
Genesis: FID 80.5
MONet: FID 176.4
Figure 6: Randomly selected scenes generated by GENESIS and MONet after training on the GQN dataset. Images sampled from GENESIS contain clearly distinguishable foreground objects and back- grounds. Samples from MONet, however, are mostly incoherent.
BD-VAE: FID 145.5
DC-VAE: FID 82.5
Figure 7: Randomly selected scenes generated by the BD-VAE and the DC-VAE after training on the GQN dataset; shown for comparison. The DC-VAE generates decent scene backgrounds but foreground objects are blurry.
14

---

Published as a conference paper at ICLR 2020
Sample
==
==3
==4
==
k
Genesis S: FID 70.2
Figure 8: component-by-componn scene generation with GENESIS-S S after training on the GQN dataset. While GENESIS-S nominally achieves the best FID in Table 2 this appears be due the generation of high fidelity background patterns rather than appropriate foreground objects. Fur- thermore, unlike the components generated by GENESIS every step in Figure the components generated by GENESIS-S are not very interpretable.
Genesis-Si S: FID 70.2
Figure 9: Randomly selected scenes generated by GENESIS-S S after training the GQN dataset.
15

---

Published as a conference paper at ICLR 2020
E
INFERENCE OF SCENE COMPONENTS
Inpul
Reconstruction
==2
== ==4 k=3 k=
Genesis
MONet
Figure 10: Step-by-step decomposition of scene from GQN with GENESIS and MONet. Two objects with the same shape and colour are successfully identified by both models. While colour and texture useful cues for decomposition, this example shows that both models perform something more useful than merely identifying regions of similar colour.
Reconstruction
=
k=3
==4
R=5
k=
k=
Genesis-S
Figure 11: Step-by-step decomposition of the same scenes as in Figure 4 and Figure 10 with GENESIS-S. While the foreground objects are distinguished from the background, they are explained together in the first step. Subsequent steps reconstruct the background haphazard fashion.
16

---

Published as a conference paper at ICLR 2020
k= 2
==
==4
==
k
Genesis
Input
Reconstruction
k=
k=
k
&
k
&
& 6
MONet
Input
Reconstruction
k
k
k
Figure 12:
A ShapeStacks tower is decomposed by GENESIS and MONet. Compared to the GQN dataset, both methods struggle segment the foreground objects properly. GENESIS captures the purple shape and parts of the background wall in step k 4. MONet explains the green shape, the cyan shape, and parts of floor step & = 9. This is reflected the foreground ARI and segmentation covering for GENESIS (ARI: 0.82, SC: 0.68, mSC: 0.58) and MONet (ARI: 0.39, SC: 0.26, mSC: 0.35); the latter being lower as the green and cyan shapes are not separated.
k
k=5
k =
Genesis
Input
Reconstruction
==7
k=8
k=
& ==
A 4
k 5
k 6
MONet
Input
Reconstruction
==
k=
Figure 13: In this example, GENESIS (ARI: 0.83, SC: 0.83, mSC: 0.83) segments the four foreground objects properly. MONet (ARI: 0.89, SC: 0.47, mSC: 0.50), however, merges foreground objects and background again in steps higher than for GENESIS. This is possible the ARI does not penalise the ser-segmentttion of the foreground objects, highlighting its limitations for evaluating unsupervised instance segmenta- tion. The segmentation covering, however, reflects the quality of the segmentatioin masks properly.
and k 9. Despite the inferior decomposition, the ARI for MONet
17


Reviews:
[{"review_id": "HJxO0sg6YH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE: I appreciated the authors' discussion. The authors addressed my questions satisfactorily, and I maintain my original rating of accept.\n\n----\nSummary: This papers tackles the question of building an object-centric latent variable generative model of scenes that can sample novel scenes with coherent objects and relationships. To do this, the authors define a generative model, GENESIS, that uses an autoregressive prior over mask variables. The component variables are generated conditioned on these mask variables. The visual appearance of the objects are generated conditioned on the component variables. Inference is done sequentially by inferring some later object varaibles conditioned on others. Results show that the model adopts a consistent strategy in generating and inferring the scene components, first considering the background then the foreground objects. The authors apply GENESIS to three datasets with monochromatic objects and show that GENESIS qualitatively generates coherent scenes and infers coherent scene components.\n\nDecision: Accept. This work clearly addresses a problem beyond current object-centric modeling approaches such as IODINE and MONet, which is the problem of generating novel scenes.\n\nStrengths:\n- The paper is well written and executed.\n- The evaluation is thorough\n- The problem and solution are well motivated\n\nWeaknesses: \n- While the authors demonstrates that GENESIS is able to model static scenes, it is not clear how straightforward it is to extend GENESIS to modeling dynamics for the purpose of robotics and reinforcement learning (as stated in the authors' motivation). Whereas approaches such as IODINE or RNEM (van Steenkiste et al. 2018) treat the object latent that can be propagated through time, maintatining that the same latent models the same object may not be a guarantee for autoregressive approaches such as MONet or GENESIS that re-parse the scene at every frame. Object temporal consistency is especially important when considering tasks that have occlusion, which are important problems in robotics.\n- The results in Appendix D seem to suggest that GENESIS decomposes a scene mostly via color segmentation, as IODINE and MONet do. One concern is that such models that rely mainly on color segmentation are not applicable for real world robotics with various lighting conditions and textures because segmenting based on color may not provide coherent object representations. Would the authors be able to provide an empirical analysis of how GENESIS models a real-world scene, analogous to Figure 11 in the IODINE paper?\n\nVan Steenkiste, S., Chang, M., Greff, K., & Schmidhuber, J. (2018). Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802.10353.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylK-ZHRYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a probabilistic generative latent variable model representing a 2D image as a mixture of latent components. It formulates the scene generation problem as a spatial Gaussian mixture model where each Gaussian component comes from the decoding of an object-centric latent variable. The contribution of the proposed method from previous works is the introduction of an autoregressive prior on the component latents. This allows the model to capture autoregressive dependencies among different components and thus help generate coherent scenes, which has not been shown in the previous works. In the experiments, the authors compare GENESIS with MONet and VAEs qualitatively and quantitatively and show that the model outperforms the baseline in terms of both scene decomposition and generation.\n\nThe proposed model seems like the right direction to improve upon MONet and it is nice to see the generation results. It is also nice to see the fully probabilistic modeling of the problem. Although it would improve over the MONet, I'm nevertheless not sure if the framework of sequential component generation (applying both MONet and GENESIS) can be a robust approach to more complex scenes, e.g., with a larger number of objects. Also, the mixture-based approach seems not guarantee the object-level decomposition. For example, if in the scene there are many objects of the same shape, size, and color, I think the proposed model may not properly distinguish them, as shown in some of the wall patterns in the experiments. But, I'm not sure what would happen if K is set to a large number to deal with this. Then, it would have the problem of long-term dependency in sequences.\n\nSome comments and questions:\n1. It would be good to show the qualitative result of the simplified Genesis-s and its qualitative result in the FID experiment.\n2. Is the VAE baselines (BD-VAE and DC-VAE) trained by maximizing the ELBO? If it is the case, will the result of the FID experiment section be different if we train the VAE baseline with GECO?\n3. Some detail of the implementation is missing, e.g. how do we model the posterior q_{\\phi}(z_{k}^{c} | x, z_{1: k}^{m}).\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bke06RuRYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a generative model for images. There's a probability mask per-pixel per-component (which yields mixing probabilities), and then a set of latents per-component that yield an image. The system is tested on a set of scenes like the GQN dataset, stacks of blocks, and the multi-dsprites dataset. The system is better than MONet, although there are a few lingering questions.\n\nSummary of positives:\n+ The factoring of the image into various components is eminently sensible and more work should build in the notion of objects\n+ The method is well explained, and I found it easy to understand the entire process.\n+ The method does appear to perform better than MONet, and qualitatively produces good results.\n\nSummary of negatives:\n- There are a few overclaims that should be fixed: namely that the system is a \"generative model of 3D visual scenes\" when in reality it is a generative model of images that has a latent space that is perhaps well-positioned to match images generated by 3D scenes composed of objects.\n- The method section sets up a few questions that are never answered in the paper: GENESIS--S gets introduced as a baseline to test a hypothesis, but this never gets quantitatively evaluated (indeed GENESIS-S beats GENESIS in 2/3 of the categories where they are compared head-to-head); similarly, the fact that GENESIS produces probabilities is repeatedly sold as an advantage over other methods, but it's never used.\n- The experiments are weak: the aforementioned questions aren't tackled, as well as a few other issues.\n\n\nOverall, I lean ever so slightly towards rejection. The results are good looking, and the method seems well-explained. However, in my view, the manuscript seems to fall short of the mark. I am not, however, strongly opposed to the paper's acceptance. If it is accepted, I would urge authors to address the issues to make their paper have maximum impact.\n\nBig picture things:\n-I'm baffled by the repeated claim that this is a generative model of 3D scenes. In the introduction \"first object-centric generative model of 3D visual scenes capable of both decomposing and generating scenes\". I'm worried I'm just missing something profoundly obvious and critical -- since it seems obvious to me that it's not 3D. \n\nAs far as I can tell, x here is a HxWxC image (where C is the number of channels and probably 3); the mixing probabilities are over HxW images, so it's a 2D segmentation system. It may be applied to photos rendered with a perspective camera -- but if the requirement for the system is being applied to images that come via projection, then is faster RCNN a 3D object detector? Are normalized cuts, SLIC superpixels, or any of the other myriad unsupervised segmentation methods from vision then 3D scene segmentation approaches?\n\n-Similarly, it's not really a model for objects -- as can be seen in the multi-colored wall example in Fig 3, and Fig 8, 9. its notion of object is a region with uniform color (presumably due to the Gaussian p(x|z) ). So while it's great that it can model regions of uniform color, the system in large part seems to work (in its current form, reading the current manuscript) because the objects are all one color, and doesn't necessarily decompose the scene into objects.\n\nUnless I'm totally missing something, this sort of claim that should be corrected very quickly. I realize that my personal bias is towards more engineering and harder data and I appreciate that there should be different operating points along the spectrum of how much information is provided. However, I think that it's worth making claims and assumptions clear, rather than claiming to solve the general problem on a specific case because of peculiarities about the specific case.\n\n\nMethod:\n-The writing of the method is quite straightforward and written well. The system is clear to me. I largely have no complaints about the method. I do, however, have concerns about the experiments that are done to validate claims in the method section.\n-GENESIS-S appears as this baseline \"To investigate whether separate latents for masks and component appearances are necessary for decomposition\", but then basically disappears. There are, as far as I can see, no qualitative results from it, and it basically appears once quantitatively, in S 4.3 / Table 1, where it appears to be as good if not better than GENESIS. In the appendix it's claimed that GENESIS trains faster than GENESIS-S, but if this is the selling point of GENESIS over GENESIS-S, it would be nice to have at least a little quantification of this.\n-There is considerable fuss made over the fact that the system produces probabilities and this is repeatedly mentioned as an advantage over existing systems, but it is never demonstrated that these probabilities are good or useful. This leaves the reader hanging a bit. \n\nExperiments:\n\nOverall the experiments are a little on the weak side.\n\n+The qualitative results are good, but the primary selling point it seems is that the system is better than MONET. This does seem clear, and GENESIS does appear quite a bit better.\n\n- While the results on factoring scenes do appear to be good, there's no quantitative evaluation of this (although the abstract promises it). Section 4.2 essentially says that ARI is bad, without demonstrating in a compelling way that it is. The paper says that this behavior can be seen in Appendix D, but this only shows the oversegmentation and not that this dramatically throws off performance -- I'd expect a figure showing a reversal of expectations -- a clearly worse result getting a clearly better ARI metric. \n\nAdditionally, computer vision has been evaluating unsupervised segmentation for close to two decades (BSDS came out in 2001). The authors should look at the metrics in e.g., \"Contour Detection andHierarchical Image Segmentation\" Arbelaez et al. PAMI 2010. I realize that coming up with metrics is hard, but I think the burden is on the authors to find the metrics to show their conclusions quantitatively.\n\n- The tower stability/height/view experiments seem incomplete. I find them quite interesting, but I'm not entirely sure of what to draw from it. The paper obliquely comments that the Groth paper gets better results with a more complex backbone network and blames it on lack of augmentation, use of a subset of the data, and a reduced resolution. But why not just do straightforward like train a MLP on an even further spatially downsampled image? Random here isn't a particularly compelling baseline or reference point. \n\nSmall stuff that doesn't affect my review:\n1) It might be worth explicitly pointing out that p_\\theta(x_k,z_k^c) is just a Gaussian soon after Eqn 3. It's written after equation 5, but without making the p_\\theta explicit. This may help some readers.\n2) The tower stability is done with a one-hidden-layer (512) MLP. I'm of the personal opinion that readouts on latent variables are more informative if it's a linear transformation (since this can't do much heavy lifting).\n3) The abstract promises semi-supervised learning -- is this the tower experiment? This strikes me more as transfer learning.\n\n-----------------------------\n\nPost review update: I have read the authors' responses, and found them thoughtful and to have answered my questions. I'm happy to accept the paper and would encourage the AC to do so.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper offers a new method for scene generation.  While there is some debate on the semantics of ‘generative’ and ‘3d’, on balance the reviewers were positive and more so after rebuttal.  I concur with their view that this paper deserves to be accepted.
Decision: Accept (Poster)