GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
Authors: Martin Engelcke
Oiwi Parker Jones
Ingmar Posner
Venue: NeurIPS 2021 Poster (2021)
Keywords: Generative models
object-centric representations
variational inference

Abstract: Advances in unsupervised learning of object-representations have culminated in the development of a broad range of methods for unsupervised object segmentation and interpretable object-centric scene generation. These methods, however, are limited to simulated and real-world datasets with limited visual complexity. Moreover, object representations are often inferred using RNNs which do not scale well to large images or iterative refinement which avoids imposing an unnatural ordering on objects in an image but requires the a priori initialisation of a fixed number of object representations. In contrast to established paradigms, this work proposes an embedding-based approach in which embeddings of pixels are clustered in a differentiable fashion using a stochastic stick-breaking process. Similar to iterative refinement, this clustering procedure also leads to randomly ordered object representations, but without the need of initialising a fixed number of clusters a priori. This is used to develop a new model, GENESIS-v2, which can infer a variable number of object representations without using RNNs or iterative refinement. We show that GENESIS-v2 performs strongly in comparison to recent baselines in terms of unsupervised image segmentation and object-centric scene generation on established synthetic datasets as well as more complex real-world datasets.

GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner Applied AI Lab, University of Oxford, UK (martin, oiwi, ingmar)@robota ox ac.
Abstract
Advances in unsupervised learning of rereettteprsssntaaiinnn have culminated in the development of a broad range of methods for unsupervised object segmentation and interpretable object-centric to simulated and real-world datasets with limited visual complexity. Moreover, object representations
generation. These methods, however, are limited
often inferred using RNNs which do not scale well large images or iterative refinement which avoids imposing an unnatural ordering objects in an image but requires the priori initialisation of fixed number of object representations. In contrast to established paradigms, this work proposes an embedding- based approach in which embeddings pixels are clustered in a differentiable fashion using stochastic ttickbbreaing process. Similar to iterative refinement, this clustering procedure also leads to randomly ordered object repre- sentations, but without the need of initialising a fixed number of clusters priori. This is used to develop new model, GENESIS-VE v2, which can infer variable number of object representations without using RNNs or iterative refinement. We show that GENESIS- v2 performs strongly in comparison to recent baselines terms of unsupervised image segmentation and object-centric scene generation on established synthetic datasets as well as more complex wallwwrldd datasets.
Introduction
Reasoning about discrete objects in an environment is foundational how agents perceive their surroundings and act in it. For example, autonomous vehicles need to identify and respond to other
road users (e.g. II 2 and robotic manipulation tasks involve grasping and pushing individual objects (e.g. (3). While supervised methods can identify selected objects (e.g. [4] it is intractable to manually collect labels for every possible object category. Furthermore, we often desire the ability to predict, or imagine, how collection of objects might behave (e.g. (6). A range of works have thus explored unsupervised segmentation and object-centric generation recent years (e.g. (73))) These models are often formulated as variational autoencoders (VAES) 37 38 which allow the joint learning of inference and generation networks to identify objects in images and to generate scenes in an object-centric fashion (e.g. [15 17 28
Moreover, such models require differentiable mechanism for separating objects in an image. While some works use spatial transformer networks (STNs) [39] to process crops that contain objects (e.g. [7- The latter avoids the use of fixed-size sampling grids which are ill-suited for objects of varying size. Instead, object representations are inferred either by iteratively refining set of randomly initialised representations
15 others directly predict pixel-wise instance segmentation masks (e.g. [16 27
(e.g. 24) or by using recurrent neural networks (RNN) (e.g. [16-18). ). One particularly
interesting model of the latter category is GENESIS [17] can perform both scene segmentation and
generation by capturing relationships between objects with an autoregressive prior.
NNo affiliated with Google DeepMind.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

As noted in Novotny et al. [40] however, using RNNs for instance segmentation requires processing high-dimensional imputs in sequential fashion which is computationally expensive and does not scale well to large images with potentially many objects. We also posit that recurrent inference is not only problematic from a computational point of view, but that it can also inhibit the learning of object representations by imposing an unnatural ordering on objects. In particular, we argue that this leads to different object slots receiving gradients of varying magnitude which provides a possible explanation for models collapsing to single object slot during training, unless the flexibility of the model is restricted (see (8). While iterative refinement instead infers unordered object representations, requires the a priori initialisation of fixed number of object slots even though the number of objects in an image is unknown.
In contrast, our work takes inspiration from the literature on supervised instance segmentation and adopts an instance colouring approach (e-g- [10 in which pixel-wise embeddings or colours-aid are clustered into attention masks. Typically, either supervised learning signal is used to obtain cluster seeds (e.g. [40] 41 or clustering is performed as a non-differentiable post-processing
operation (e.g. 42 43) Neither of these approaches is suitable for unsupervised, end-to-end learning of segmentation masks. We hence develop an instance colouring stick-breaking process (IC-SBP) to cluster embeddings in a differentiable fashion. This is achieved by stochastically sampling cluster seeds from the pixel embeddings to perform a soft grouping of the embeddings into a set of randomly ordered attention masks. It is therefore possible to infer object representations both without imposing a fixed ordering or performing iterative refinement.
Inspired by GENESIS 17]. we leverage the IC-SBP to develop GENESIS-V2. V2, novel model that learns to segment objects in images without supervision and that uses an autoregressive prior to generate scenes in an interpretable, object-centric fashion. GENESIS- V2 is comprehensively benchmarked against recent prior art [16] [7] on established synthetic datasets Objjectsoon [44] and ShapeStacks -4hree it performs strongly in comparison to several recent base- lines. We also evaluate GENESIS-V2 on more challenging real-world images from the Sketchy [46] and the MIT-Princeton Amazon Picking Challenge (APC) 2016 Object Segmentation datasets 471. where also achieves promising results. Code and pre-trained models are available at https://gtt http://ggtttu..../ thub. com/ haapmmappleed-eeeee ai- aal-labbgooe sab/genesis
2 Related Work
Unsupervised models for learning object representations are typically formulated either as autoen- coders (e.g. 7100
28) generative adversarial networks (GANs) (e.g. (6). Typical GANs are able to generate images, but lack an associated inference mechanism and often suffer from training
12
instabilities (see e.g. 48 49
A comprehensive review and discussion of the subject is provided in
Greff et al. (50)-
In order to infer object representations, STNs [39 can explicitly disentangle object location by cropping out a rectangular region from an input, allowing object appearance to be modelled in canonical pose (e.g. [7H1O 15)) This operation, however, relies a fixed-size sampling grid which is not well-suited if objects vary broadly in terms of scale. In addition, gradients are usually obtained via bi-linear interpolation and are therefore limited to the extent of the sampling grid which can impede training: for example, if the sampling grid does not overlap with any object, then its location cannot be updated in a meaningful way. In contrast, purely segmentation based approaches (1627 often use RNNs (e.g. (6618) or iterative refinement (e.g. (19)24) to infer object representations from an image. Other works either use fixed number of slots 25 26 or group pixels in a non-differentiable fashion (7). RNN based models need to learn a fixed strategy that sequentially attends to different regions in an image, but this imposes an unnatural ordering on objects in an image. Avoiding such a fixed ordering leads to a routing problem. One way to address this is by randomly initialising set of object representations and iteratively refining them. More broadly, this is also related to Deep Set Prediction Networks [5I], where set is iteratively refined in a gradient-basee based fashion. The main disadvantage of iterative refinement is that it is necessary to initialise a fixed number of clusters priori, even though ideally we would like the number of clusters to be input-dependent. This directly facilitated by the proposed IC-SBP. The IC-SBP and GENESIS-V2 v2 are in this respect also related to the Brick-Breaking VAE [52] which uses stochastic number of latent variables, but does not attempt to explicitly capture the object-based structure of visual scenes.
2

---

Unlike some other works which learn unsupervised object representations from video sequences (e.g. [4). our work considers the more difficult task of learning such representations from individual images alone. GENESIS-V2 is most directly related to GENESTS [7 and ST.OT- LLOT-ATTENTION 24]_ Like GENESIS, the model is formulated as a VAF to perform both object segmentation and object-centric scene generation, whereby the latter is facilitated by an autoregressive prior. Similar to SLOT-ATTENTION, the model uses a shared convolutional encoder to extract a feature map from which features are pooled via an attention mechanism to infer object representations with random ordering. In contrast to SLOT-ATTENTION however, the attention masks are obtained with a parameter- free clustering algorithm that does not require iterative refinement or a predefined number of clusters. Both GENESIS and ALOT-ATTENTION are only evaluated on synthetic datasets. In this work, we synthetic datasets for quantitative benchmarking, but we also perform experiments on two more challenging real-world datasets.
3 GENESIS-V2
An image x of height H. width W, with C' channels, and pixel values in the interval [0, 1] is considered to be three-dimensional tensor [0, 1] This work is only concerned with RGB images where C = 3. but other input modalities with a different number of channels could also be considered. Assigning individual pixels to object-like scene components can be formulated as obtaining a set of object masks π E [0, => x with >n Ti.i. = 1 for all pixel coordinate tuples (i, in an image, where K is the number of scene components. Inspired by prior works (e.g. [16 (22)) and identical to Engelcke et al. [1], this is achieved by modelling the image likelihood (x(x as an SGMM of the form
H W log [o ZI:K) = ΣΣΣ log Σ
K
( o.2)
(I)
The parameters 9 of the model are learned 0r is a fixed standard deviation that shared across object slots. The summation in Equation TT) implies that the likelihood is innvariiiinnnnniann to the order of the object representations Z1:K (see e.g. [53 (44) provided that Te.j (Z1.k) & inttuuttttionnvvariant This allows the generative model to accommodate for a variable number of object representations.
is also
To segment objects in images and to generate synthetic images in an object-centric fashion re- quires the formulation of appropriate inference and generative models, i.c. (o(21. x) and (x(x Z1:K) pe (z(2zx) ), respectively, where ó are also learnable parameters. In the generative model, is necessary to model relationships between object representations to facilitate the generation of coherent scenes. Inspired by GENESIS [17], this is facilitated by an autoregressive prior
Po (21.k) = II po p2 21:k-1)-
GENESIS uses two sets of latent variables to encode object masks and appearances separately. In contrast, GENESIS-VE v2 uses one set of latent variables Z1:K to encode both, which increases parameter sharing. The graphical model of GENESIS v2 is shown next to related models in Appendix A
While GENESIS relies on a recurrent mechanism in the inference model to predict segmentation masks, GENESIS-V2 instead infers latent variables without imposing a fixed ordering and assumes object latents Z to he conditionally independent given an input image x, i.e., go (2 (x) = II Io (zg x). Specifically, GENESIS-V2 first extracts an encoding with a deterministic UNet backbone. This encoding is used to predict map of semi conolltovlltio pixel embeddings E 12 xW xD; (see (40).. Scmi-convolutional embeddings are introduced in Novotny al. 40 facilitate the prediction of unique embeddings for multiple objects of identical appearance. The embeddings are computed by performing an element- wise addition of pixel coordinates to two dimensions of the embeddings. In this work, we let the pixel coordinates be in the interval [-1, 1] relative to the image centre. Subsequently, the IC-SBP converts the embeddings into set of normalised atlention masks m E [0, 1] X x with Σx M4.j. 1 via distance kernel vh. The spatial structure of the embeddings should induce the attention masks to be spatially localised, but this is not hard constraint. addition, we derive principled initialisations for the scaling- factor of different IC-SBP distance kernels e in Section 3.2
3

---

IC-SBP
TT, X]
:
mg
2K
WK XX
Figure I: GENESIS-V2 overview. The image x is passed into a deterministic backbone. The resulting encoding is used to compute the pixel embeddings Ç which are clustered into attention masks mi:K by the IC-SBP. Features are pooled according to these attention masks to infer the object latents Zı:K These are decoded into the object masks TT:K and reconstructed components X1:K-
GENESIS- v2 uses the attention masks mi:k to pool feature vector for each scene component from the deterministic image encoding. A set of object latents Z1:K is then computed from these feature vectors. This set of latents is decoded in parallel to compute the statistics of the SGMM in Equation (T. The object masks π are normalised with a softmax operator. The inference and generation models can jointly trained VAE as illustrated in Figure Further architecture details are described in Appendix
Inspired by Locatello al. 241
3.1 Instance Colouring Stick-Breaking Process
The IC-SBP is stochastic, differentiable algorithm that clusters pixel embeddings BHxWxDD
into a variable number of soft attention masks m E [0, 1|"xW* Intuitively, this is achieved by (1) sampling the location (i.j) of a pixel that has not been assigned hard cluster according to the distance of the embedding Çı at the selected pixel location to all other pixel embeddings according to kernel v. and (3) repeating the previous explained or some form of stopping condition is reached. Crucially, the stochastic selection of pixel embeddings as cluster seeds leads to set of randomly ordered soft clusters. Due its conceptual similarity, the method derives its name from more formal stick beke-breking process formulations as can, e.g., be found in Yuan et al. [1] or Nalisnick and Smyth (52).
cluster yet. (2) creating a soft or
steps until all pixels are
The IC-SBP is described more formally in Algorithm T Specifically and inspired by Burgess et al. [16]. scope s e [0, x initialised to a matrix of ones H track the degree to which pixels have been assigned to clusters. In addition, a matrix of seed scores is created once by sampling from uniform distribution
(O, 1) Hx to perform the stochastic selection of pixel embeddings. At each iteration, a single embedding vector Çijj is selected at the spatial location (i. j) which corresponds to the argmax of the element- wise multiplication of the seed scores and the current scope. This ensures that cluster seeds are sampled from pixel embeddings that have not yet been assigned clusters. An alpha mask a4 E [0. RW computed as the distance between the cluster seed embedding Çi,j and all individual pixel embeddings according to distance kernel v/. The output of the kernel y is one two embeddings are identical and decreases to zero as the distance between a pair of embeddings increases. The associated attention mask mæ is obtained by the element-wise multiplication of the alpha masks by the current scope to ensure that the final set of attention masks normalised. The scope is then updated by an element-win wise multiplication with the complement of the alpha masks. This process is repeated until a stopping condition satisfied, at which point the final scope is added as additional mask to explain any remaining pixels.
In this work, we restrict ourselves to soft cluster assignment, leading to continuous attention masks with values in m E [0, 1] H>WW" Unless the attention masks take binary values, several executions of the algorithm will lead to slightly different masks for individual objects. If the mask values are discrete and exactly equal to zero one, however, then the set of cluster seeds and the set of attention masks are uniquely defined apart from their ordering. This can be inferred from the fact that if at each step of the IC-SBP produces discrete mask, then embeddings associated with this mask cannot

---

Algorithm 1: Instance Colouring Stick-Breaking Process embeddings Ç E R Dx
Input:
Output: masks m with my E [0,1]
Initialise: masks m = Ø, scope = seed scores c ~ U0) E #dxw
while not StopCondition(m) do i.j argmax(s o c); a = DistanceKemel(G, Gig); m.append(s @ o);;
S = 2 (1 a);
end
m.append(s)
be sampled as cluster seeds later on due to the masking of the seed scores by the scope. A different cluster with an associated discrete mask is therefore created at every step until all embeddings are uniquely clustered. Another interesting modification would he to use continuous masks while making the output of the IC-SBP cemmmuattion-qqivvriiiii This could be achieved either by directly using the cluster seeds for downstream computations or by separating the mask normalisation from the stochastic seed selection. While the SBP formulation facilitates the selection of diverse set of cluster seeds, the masks could be normalised separately after the cluster seeds are selected by using a softmax operation, for example. An investigation of these ideas is left for future work.
with respect the ordering of the cluster seeds.
In contrast to GENESIS, the stochastic ordering of the masks implies that it is not possible for GENESIS-V2 V2 to learn a fixed sequential decomposition strategy. While this does not strictly apply to the last mask which set equal to the remaining scope, we find empirically that models learn strategy where the final scope is either largely unused or where it corresponds to a generic background cluster with foreground objects remaining unordered as desired. Unlike as in iterative refinement where a fixed number of clusters needs to be initialised priori, the IC-SBP can infer variable number of object representations by using a heuristic that considers the current set attention masks at every iteration in Algorithm I While we use fixed number of K masks during training for efficient parallelism on GPU accelerators, we demonstrate that a flexible number of masks can be extracted at test time with minimal impact on segmentation performance.
3.2 Kernel Initialisation with Comii-Covolltional Embeddings
For semi-convolutional embeddings be similar according a distance kemel th. the model needs to learn to compensate for the addition of the relative pixel coordinates. It can achieve this by predicting delta vector for each embedding to a specific pixel location, for example to the centre of the object that the embedding belongs to. A corollary of this is that if embeddings are equal to the relative pixel coordinates with the other dimensions being zero, then clustering embeddings based on their relative distances results in "blob-like", spatially localised masks. In this work, we make use of this property to derive meaningful initialisation for free parameters in the distance kernel &i of the IC-SBP. Established candidates for & from the literature are the Gaussian dे [40], Laplacian tiz and Epanechnikov bh kernels [55] with
tha = -xxp (- [[u-v]1
VE = exp (- [|u-vvl]
th max (1-1lu l|u-v||² .0) 0).
(3)
whereby and are embeddings of equal dimension. Each kernel contains a scaling factor (G.LLE) E R By initialising the model at the beginning of training so that the embeddings are equal to the relative pixel coordinates with the other dimensions being zero, then (GG can be initialised so that the initial attention masks are similarly circular patches. In particular, we initialise these scaling factors
-O'= K T-'
/Kln2 oF -=
which derived and illustrated in Appendix C After initialisation, ((G.L.)) is jointly optimised along with the other learnable parameters of the model as in Novotny al. [40].
5

---

3.3 Training
Following Engelcke et al. 17. GENESIS- v2 is trained by minimising the GECO objective [56]. which can be written as a loss function of the form
b = E (=/x)| In (x z)] + Bg KL (z x)
po(z]]
(5)
The relative weighting factor 3g E Rt is updated at every training iteration separately from the model parameters according to
8g - Bg - CC(C)
with a= E (1 ag)) (x)() po 2)].
(6)
E R is an exponential moving average of the negative image log- liglliielhhood, ag E [0, 1] is momentum factor, E IR is step size hyperparameter, and C' R is a target reconstruction error. Intuitively, the optimisation decreases the weighting of the KL (Kullback-eeible term as long as the reconstruction error is larger than the target C. The weighting of the KL term is increased again once the target is satisfied.
regularisation
In some applications, a practitioner might only require segmentation masks in which case, so having to reconstruct the entire input would be rather inefficient. While we observed that the attention masks m are correlated to the object masks т, they do not align as closely with object boundaries. We conjecture that this is consequence of the large receptive field of the UNet backbone which spatially dilates information about objects. Consequently, we also conduct experiments with an additional auxiliary mask consistency loss that encourages attention masks and object masks to be similar. This leads to a modified loss function of the form
Sgg = E + Bg (KL(g0 x) || Po())] + KL|m || nograd(
(7)
in which m and π are interpreted as pixel-wise categorical distributions. Preliminary experiments indi- cated that stopping the gradient propagation through the object masks helps achieve segmentation quality comparable to using the original loss function in Equation (5)
Experiments
This section presents results on two simulated datasets ObjectsRoom 44 and ShapeStacks 45 as well as two real-world
[46] and APC [47 which are described in Appendix D GENESIS-V2 2 is compared against three recent baselines: GENESIS 17 MONET [66,, and SLOT- ATTENTION 24 Even though SLOT-ATTENTION is trained with a pure reconstruction objective and not generative model, it is an informative and strong baseline for unsupervised scene segmentation. The other models are trained with the GECO objective [56] following the protocol from Engelcke al. 17 for comparability. We refer to MONET trained with GECO as MONET-G to avoid conflating the results with the original settings. Further training details are described in Appendix E
segmentation quality is quantified using the Adjusted Rand Index (ARI) (57) and the Mcan Segmentation Covering (MSC). The MSC is derived from Arbelaez et al. [58] and described in detail in Engelcke et al. [17]. These are by default computed using pixels belonging to ground truth foreground objects (ARI-FG and MSC-FG). Similar to Greff al. (22) these are averaged over 320 images from respective test sets. We also report the Evidence Lower Bound (ELBO) averaged over 320 test images as a measure how well the generative models able to fit the data. Generation quality is measured using the Fréchet Inception Distance (FID) [59 which is computed from 10,000 samples and 10,000 test set images using the implementation from Seitzer [60] When models trained with multiple random seeds,
Following prior works (e.g. [17
18 22 24
always show qualitative results for the seed that achieves the highest ARI-FG. In terms of the two real-world datasets, there are only ground truth segmentation masks available for the APC data, with the caveat that there is only a single foreground object per image. In this case, when computing the segmentation metrics from foreground pixels alone, the ARI-FG could be trivially maximised by assigning all image pixels to the same component and the MSC-FG would be equal to the largest IOU between the predicted masks and the foreground objects When considering all pixels, the optimal solution for both metrics is to have exactly one set of pixels assigned to the foreground object and another set of pixels being assigned to the background. While acknowledging that segmenting the background as a single component is arguably not the only valid way of segmenting the background, we report the ARI and MSC using all pixels instead to develop a sense of how well foreground objects are separated from the background.

---

Imputs
Segmentations Raconstructions
(a) MONET-G
(h) GENESIS
(c) STOT-ATTENTION
(d) GENESIS-V2
Figure 2: GENESIS-V2 learns better reconstructions and segmentations on ShapeStacks.
Table I: Means and standard deviations of the segmentation metrics from three seeds. Bold values in the first half of the table indicate the best values for the generative models; bold values in the second half indicate any better values achieved by the additional non-generative baseline.
ObjectsRoom
ShapeStacks
Model
Generative
ARI-FG
MSC-FG
ARI-FG
MSC-FG
MONET-G
Yes
0.54100.0
0.3310001
0.70+0.04
0.57:10.12
GENESIS
Yes 0.63:10.03
0.53+00.7
0.70:10.05
0.67:10.22
GENESIS-V2
Yes
0.85
0.59
0.01
0.67
SLOT-ATTENTION
0.79:10.02
0.64
0.76+10.01
0.70:L0.05
4.1 Benchmarking in Simulation
Figure shows qualitative results on ShapeStacks and additional qualitative results are included in Appendix F GENESIS-V2 cleanly separates the foreground objects from the background and the reconstructions. Interestingly, GENESIS-V2 is the only model that segments the entire background as single component. We conjecture that this behaviour structures are of finite variety. As result, even when viewing only a fraction of the background, is possible for the model largely predict the appearance of the rest of the background. The KL regularisation during training encourages efficient compression of the inputs, which penalises redundant information between slots and might thus explain this behaviour. trained without KL. regularisation and the decoders of MONET-G well GENESIS are possibly not flexible enough to reconstruct the entire background as single component (see Appendix for architecture details).
consequence of the fact that the background
AOT-ATTENNION is
In terms of quantitative performance, Table I summarises the segmentation results ObjectsRoom and ShapeStacks. GENESIS-V2 outperforms the two generative baselines GENESIS and MONET-G across datasets on all metries, showing that the IC-SBP is indeed suitable for learning object-centric representations. GENESIS-V2 outperforms the non- non-generative SLOT-ATTENTION baseline in terms of the ARI-FG both datasets. SLOT-ATTENTION manages achieve better mean MSC-FG. The standard deviation of the MSC-FG values is much larger, though, which indicates training is not as stable. While the ARI-FG indicates the models ability to separate objects, it does not penalise the undersegmentattion of objects (see [7]. The MSC-FG, in contrast, is an IOU based metric and sensitive
exact segmentation masks. conjecture that SLOT- ATTENTIIO manages predict slightly more accurate segmentation masks given that is trained on pure reconstruction objective and without KI. regularisation, thus leading to slightly better mean MSC-FG.
ablations for
GENESIS- v2 is also included in Appendix
7

---

Table 2: Means and standard deviations of the segmentation metries from three seeds for GENESIS- with a fixed or flexible number of object slots. Highlighting follows an analogous scheme as in Table T
Dataset
Training
Slots
Avg.
MAE 4
ARI-FG f
MSC-FG
ObjectsRoom
No mask loss
Fixed
7.0:0.00 5.0
3.3+0.0
0.8510001
0.59
Flexible
1.740.9
0.84#0.01
0.51
ShapeStacks
No mask loss
Fixed
9.0:t0.0
4,4+0.0
0.81
0.67
Flexible
6.3+0.3
1.9+0.3
0.77=0.02
0.63#00.01
With mask loss
Fixed Flexible
9.0+10.0
4.4
0.01
0.68:10.00 0.68
5.7:0..2
1.1:0.002
0.81
Table Means and standard deviations of ELBO values and FID scores from three seeds.
Model
ObjectsRoom
ShapeStacks
ELBB f
FID
ELBO 1
FID 4
MONET-G
-7217=19
205.717.6
17268119
197.8=5.2
GENESIS
-7023-12
62.8+22.5
-7082+15
186.8=18.0
SLOT-ATT.
GENESIS-V2
-7040+2
52.6+2.7
-7019+22
112.7#3.2
We also examine whether the IC-SBP can indeed be used to extract variable number of object representations. This is done by terminating the IC-SBP according to a manual heuristic and setting the final mask to the remaining scope. Specifically, we terminate the IC-SBP when the sum of the current attention mask values smaller than 70 pixels for ObjectsRoom and 20 pixels for ShapeStacks. A larger threshold is used for the former the attention masks tend to be more dilated (see Appendix F). The average number used slots, the Mean Absolute Error (MAE) to the ideal number of slots, and segmentation metrics when using fixed number and variable number slots after training are summarised in Table 2 On both datasets, allowing for a flexible number requires fewer steps and achieves smaller MAF. This is incurred, though, at drop in segmentation performance. When training GENESIS-V v2 with the auxiliary mask loss as in Equation (7) on ShapeStacks, the average number of steps and the MAE further decrease at impact the segmentation metrics. On ObjectsRoom, the auxiliary mask loss appeared to deteriorate the learning of good object segmentations and the associated results therefore not included.
terms of density estimation and scene generation, T summarises the ELBO values and FID scores for GENESIS-V2 v2 and the baselines ObjectsRoom and ShapeStacks GENESIS achieves slightly better ELBO than GENESIS-U v2 on ObjectsRoom, but GENESIS-VE v2 performs significantly better StapeStacks. We hypothesise that GENESIS benefits here from having more flexible, autoregressive posterior than GENESIS-V2. Regarding scene generation, GENESIS-V2 consistently performs best with particularly significant improvement on ShapeStacks. Results ShapeStacks, however, are not as good as for ObjectsRoom, which is likely caused by the increased visual complexity of the images in the ShapeStacks dataset. Qualitative results for scene generation shown in Figures 3 and Both GENESIS and GENESIS-V2 produce reasonable samples after training on ObjectsRoom. For ShapeStacks, samples from GENESIS contain significant distortions. In comparison, samples from GENESIS-V2 are more realistic, but also still show room for improvement.
NNote that in contrast to the MONET-G training objective, the decoded mask distribution rather than the deterministic attention masks are used to compute the reconstruction likelihood in the ELBO calculation for MONET-G. The KL divergence between the two mask distributions as used in the training objective is not part of this ELBO calculation.

---

@ MONET-G
MONET-G
GENESIS
(b) GENESIS
(c) GENESIS-V2
(c) GENESIS-V2
Figure 3: ObjectsRoom samples.
Figure 4: ShapeStacks samples.
Inputs
Sigmentations Recanstruction
(a) MONET-G
(b) GENESIS
(c) SLOT-ATTER ATTENTION
(d) GENESIS-V2
Figure 5: In contrast to MONET-G and GENESIS, GENESIS-V2 V2 as well as SLOT-ATTENTIO are able learn reasonable object segmentations on the challenging Sketchy dataset.
4.2 Real-World Applications
After validating GENESIS-V2 on two simulated datasets, this section present experiments on the Sketchy [46] and APC datasets [47; two significantly more challenging real-world datasets collected in the context of robot manipulation. Due to the long training time on these datasets, each model is only trained with single random seed. While this makes it infeasible draw statistically strong conclusions, the aim of this section is to provide an indication and early exploration of how these models fare on more complex real-world datasets. Reconstructions and segmentations after training GENESIS-V2 on Sketchy and APC are shown in Figures 5 and 6 For Sketchy, it can be seen that GENESIS-V2 and SLOT-ATTENTION disambiguate the individual foreground objects and the robot gripper fairly well. SLOT-ATTENTION produces slightly more accurate reconstructions. which is likely facilitated by the pure reconstruction objective that the model is trained with. However, GENESIS-VE v2 is the only model that separates foreground objects from the background in APC images. Environment conditions Sketchy are highly controlled and SLOTT ATTENTION appears to unable to handle more complex conditions in APC images. Nevertheless, GENESIS v2 also struggles to capture the fine-grained details and oversegments one the foreground objects into several parts, leaving room for improvement in future work. Additional qualitative results are included in Appendix

---

Imputs
Segmentttiinn on Raconstructions
OME
(a) MONET-G
(b) GENESIS
(c) SLOT-ATTENTION
(d) GENESIS-V2 V2
Figure 6: GENESIS-V2 the only model that separates the foreground objects in images from APC.
Table 4: Segmentation metrics on APC.
Table 5: FID scores on Sketchy and APC.
ARI
MSC
Sketchy
APC
MONET-G
0.11
0.48
MONET-G
294.3
269.3
GENESIS
0.04
0.29
GENESIS
241.9
183.2
SLOT-ATT.
0.03
0.25
SLOT-ATT.
GENESTS-V2
0.55
0.67
GENESIS-V2
208.1
245.6
Table reports the ARI and MSC scores computed from all pixels GENESIS- V2 and the baselines.
GENESIS-V v2 stands out in terms of both metrics, corroborating that GENESIS-V v2 takes valuable step towards learning unsupervised object-representations from real-world datasets. FID scores for generated images are summarised in Table 5 and qualitative results are included in Appendix F GENESIS-V2 v2 achieves the best FID on Sketchy, but outperformed by GENESIS on APC. Both models consistently outperform MONET-G. All of the FID scores, however, are fairly large which is not surprising given much higher visual complexity of these images. It is therefore difficult to draw strong conclusions from these beyond rough sense of sample quality. Further work is required to generate high-fidelity images after training on real-world datasets.
Conclusions
This work develops GENESIS-V2. v2, novel object-centric latent variable model of scenes which is able to both decompose visual scenes into semantically meaningful constituent parts while at the same time being able to generate coherent scenes in object-centric fashion. GENESIS-V2 leverages differentiable clustering algorithm for grouping pixel embeddings into a variable number of attention masks which used infer unordered set of object representations. This approach is validated empirically on two established simulated datasets as well as two additional real-world datasets. The results show that GENESIS-V v2 takes a step towards learning better object-centric representations without labelled supervision from real-world datasets.
In terms of future work, there is still room for improvement in terms of reconstruction, segmentation, and sample quality. would also be interesting to investigate the ability GENESIS- v and the IC-SBP to handle Moreover, several interesting variations of the IC-SBP were also already described in Section [3.1 Additional promising avenues include the extension of GENESIS-V2 to learn object representations from video (see e.g. [33 14) or to leverage recent advances in hierarchical latent variable models such Nouveau VAEs (NVAEs) 661.
images that contain more objects than seen during training.
10


Reviews:
[{"review_id": "ja6yvdptdsG", "reviewer": "Reviewer_jTRB", "summary": "Object-centric generative models aim to decompose a scene image $x$ into multiple object-wise (and background) latent $z_{1:K}$. To extract the *set* of latent vectors, prior work considered a fixed-order recurrent approach (e.g., GENESIS) or an iterative refinement approach (e.g., IODINE, DSPN, SetTransformer, and Slot Attention). However, the former may impose an unnatural ordering and suffer from the forgetting issue for long sequences, while the latter requires a predefined fixed set cardinality. Instead, the paper suggests iteratively cluster the local pixels to form an object, which does not suffer from the ordering issue (except the last background) and can naturally decide the set cardinality (by a stopping heuristic). The proposed method produces more visually plausible scene decomposition results in standard synthetic and newly demonstrated real (e.g., robot arm) datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a new idea for object-centric generative models and outperforms prior work, including more challenging robot arm and Amazon picking datasets. This would be a nice contribution stepping towards more realistic scenarios, and thus I recommend weak accept. However, I have some concerns and questions, which I expect to be clarified in the revised paper.\n\n\n**Basically a local clustering method**\n\nAt first glance, it was hard to catch the core idea. After reading the paper carefully, I found that the proposed method is pretty simple (well described in Algorithm 1). Specifically, it repeats:\n1. Choose a random location (i,j) from the (soft) remaining area.\n2. Cluster the pixels which are similar to the one of (i,j).\n3. (Softly) remove the chosen cluster and goto 1.\n\nDespite its simplicity, I think the current description in Section 3 is a little confusing. For example, when I was reading lines 142-144, I could not catch what is the scope $s$ and seed score $c$ because the technical details for soft clustering assignment (i.e., scope) and random permutation (i.e., seed score) suddenly appeared before I understand the entire picture. Also, I was curious why one should use the semi-convolutional pixel embeddings in line 125, but found that it is just a positional encoding after reading the original paper. The paper could be kinder for newcomers by (a) presenting the high-level idea first, then explain the technical details, and (b) making contents self-contained. I think the description could be more concise, and there would be enough space for being self-contained.\n\n\n**How does the attention masks $m$ shaped?**\n\nIdeally, the attention masks $m$ should be identical to the final mask $\\pi$, and there is no reason to generate $\\pi$ again from the latent $z$ (may with some thresholding trick if needed). What is the reality? The auxiliary mask loss results on ObjectsRoom give negative signals. It would be interesting to analyze how the attention masks $m$ changes during training; check if it converges to the one-hot mask as training goes.\n\n\n**Discussion on steak-breaking processes**\n\nThe paper named its method as an instance colouring stick-breaking process. Aside from being a *stick-breaking process* formally (e.g., the remaining background is not symmetric), the paper could discuss the related literature that inspired this work, e.g., how finite realization of the Dirichlet process can be implemented. Also, there have been some trials to define a stochastic process prior to VAEs [1,2], although they do not consider the spatial structure of latent.\n\n[1] Nalisnick and Smyth. Stick-Breaking Variational Autoencoders. ICLR 2017.\\\n[2] Joo et al. Dirichlet Variational Autoencoder. arXiv 2019.\n\n\n**Is it fundamentally possible to discover objects from a static image?**\n\nThe proposed method heavily relies on the heuristic of the *locality* (by distance kernel) and *large size* (by stopping criteria) of the objects. As a result, it shows more visually plausible results on the demonstrated benchmark datasets. However, it may not work well for *non-Gaussian-like* objects. Also, is it possible to distinguish the pattern of background and objects (e.g., square patterns in Figure 2)? I suspect that the current success of GENESIS-v2 comes from the dataset assumptions: (a) objects are Gaussian-like, and (b) backgrounds are relatively simple. It would be interesting to check the results on more complex objects.\n\n\n**Possible future directions**\n\nWhile it is out-of-the-scope of the paper, there would be several directions to improve the proposed model. First, to address the fundamental limitations of object discovery, the model would utilize the additional information, e.g., sequential movements of objects like SQAIR. Second, recent work on VAEs found that hierarchical structure is the key to high-resolution image generation (e.g., [3]). It would be interesting to combine current object-centric structures with hierarchical VAEs.\n\n[3] Vahdat and Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. NeurIPS 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper discussed its limitation, e.g., in lines 250-255. The paper also discussed the potential negative societal impacts but in Appendix. Following the suggestion of NeurIPS style guide, the revised paper could contain the explicit limitations and societal impacts section in the main text.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "aev0jUYXzwT", "reviewer": "Reviewer_nSnK", "summary": "The paper proposes an object-centric generative model that combines the best of previous approaches that use RNNs or iterative refinement. The model can infer a variable number of object slots at test time as in RNNs, and the inferred object slots are randomly ordered as in iterative refinement. This is achieved by a novel differentiable iterative clustering algorithm. At each iteration, a random pixel (not yet assigned to any cluster) is chosen as the cluster seed, and a new cluster is formed by computing similarity between the per-pixel embeddings and the seed embedding. At test time, a stopping heuristic can be applied so that the number of clusters is flexible. On two simulated datasets, the model achieves better segmentation and generation quality than MONET and GENESIS, and comparable or slightly worse segmentation quality than Slot Attention (which cannot do generation). The model also shows promising results on two real-world datasets. In particular, on the APC dataset, the model shows significantly better segmentation quality than all the baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**STRENGTHS**\n\n- The model unifies two desirable properties of object-centric representations: flexible number and unordered slots.\n- The method is novel. It is inspired by supervised instance segmentation literature, with improvements that made it suitable for unsupervised, end-to-end segmentation.\n- The results on real-world datasets look promising, especially on the APC dataset where previous methods completely failed.\n\n**WEAKNESSES**\n\n- While it is conceptually desirable to be able to infer a flexible number of object slots, it is not clear from the paper what computational benefits this can bring. Specifically,\n    - In iterative refinement, the number of iterations is usually small and independent of the number of objects. However, in the proposed model, the number of iterations increases with the number of objects, which can be inefficient when there are many objects.\n    - Using a flexible number of slots only showed success on one of the datasets, and it required additional loss to match the performance when using a fixed number of slots.\n    - Perhaps one advantage of using a flexible number of slots is that it can reduce oversegmentation, which seems to be the case in the qualitative results. However, this is not supported by quantitative evaluation. Maybe the authors could include background ARI, and see whether it is improved when using a flexible number of slots.\n    - It is shown in Table 2 that using the stopping heuristic can reduce the number of slots. However, it is not clear how close this is to the true number of objects.\n\n**MINOR COMMENTS**\n\n- It is mentioned (L175-176) that the last slot tends to be the background. I am curious how this emerged. In the datasets considered, most pixels seem to belong to the background, so when a cluster seed is randomly sampled, it should have high probability to be a background cluster. This seems to suggest the opposite, that the first few slots are more likely to be the background.\n- How many slots are used in Figure 2? Table 2 suggests that you used 9 slots for this dataset, but the number of clusters seems to be far less than 9 in Figure 2.\n- The paper could be strengthened by showing generalization to more objects at test time, like what was shown in Slot Attention. This may also demonstrate the benfit of having a flexible number of slots.\n- Does the KL term introduce some implicit ordering to the posterior?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The ability to predict a flexible number of object slots at test time seems limited, and the model may have scalability issues when the scene contains many objects.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "QLXhEHFltIp", "reviewer": "Reviewer_mGsC", "summary": "This paper proposes a new object-centric generative model and demonstrates its effectiveness on a number of datasets (notably including 2 real-world datasets, unlike most previous work) compared to 3 SOTA baselines, both in terms of unsupervised segmentation and image generation. The model is reminiscent of GENESIS in terms of generative model and Slot Attention in its approach to approximate inference. However, thanks to a novel differentiable clustering method based on a stick-breaking process, this model does away with recurrent attention mechanisms and iterative refinement methods common in previous work.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is novel, interesting, significant for the community, and proven to be effective by a sufficiently extensive experimental section. Weaknesses of the model are clearly shown. Overall this is a solid contribution to the field.\n\nThe paper is generally very well-written. However, the paragraph at lines 157-171 is not as easy to follow, and the points therein could be made clearer. Similarly, the approach outlined in Sec. 3.2, based on kernel distance and adding spatial information, is a nice way of including an inductive bias, but the argument with the \"delta vector\" is not very clear.\n\nOther comments/questions:\n1. Since GENESIS does not seem perform particularly well on standard datasets like multi-dSprites and CLEVR, it would have been interesting to see if v2 closes this gap with other methods like MONet and slot attention. On the other hand, the additional compute required to run these experiments was arguably put to better use by the authors by testing the models on 2 real-world datasets.\n1. I assume the reconstruction target for GECO is chosen empirically, starting from an educated guess of possible good values (e.g. taken from GENESIS which is also trained with GECO). Is this \"annealing schedule\" necessary, or could one empirically choose a reasonable sigma for the likelihood components? Have you tried learning sigma per-component or even per-pixel (probably wouldn't work at first, but maybe after an initial training phase with fixed sigma)?\n1. Have you considered or tried different likelihoods? Since these datasets are probably discrete (uint8), the right thing to do would be to use a discrete distribution (although it might not matter in practice unless we want to measure ELBO or likelihood) with support [0,1]. Using Gaussians makes sense considering prior work, but I'm curious if you have insights on this.\n1. lines 25-26: many models for object-centric representation learning are in fact not generative, but they are cited as generative here\n1. line 105: not all cited works use a SGMM as likelihood, e.g. slot attention uses MSE as loss, which is equivalent to using a Gaussian log likelihood where the mean is a linear (convex) combination of pixel values from all slot reconstructions.\n1. the colors of segmentation mask are sometimes hard to tell apart (e.g. Fig 12 in appendix)\n1. lines 208-209: which applications are you referring to?\n1. why measure segmentation metrics on 320 images only? (also, is there any reason for this particular number?)\n1. line 237: as far as I remember, ARI does not penalize _over_-segmentation, i.e. when an object is segmented in multiple clusters, as much as MSC. Could you double-check this? I think this is also mentioned in the GENESIS paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The societal impact is addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "O-O_OSF4S2q", "reviewer": "Reviewer_9w6y", "summary": "Paper proposed a novel unsupervised pixel-level clustering algorithm that detects meaningful object representations on multiple datasets.\nThe proposed model also supports novel data sampling.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nWhile GENESIS-V2 shares similar components with GENESIS, the pixel-level clustering mechanism is of great novelty.\n\nClarity:\nThe Paper is well written. I found it easy to follow.\n\nQuality:\n\nThe generative model proposed is sound.\nEvaluation on multiple datasets (including real-world datasets) is provided and the proposed model shows superior performance on all of them.\nImages sampled from the proposed model are of high quality.\n\nI do have one major concern about the proposed model.\nGENESIS-V2 has an inductive bias that ignores background structures.\nDo the kernel functions impose a shape prior or implicit limits on the size of the objects?\nThe author, in the APC case, set \"object\" to be the one individual small item in images, excluding any background structures.\nActually, outstanding background structures in the APC dataset are ignored by GENESIS-V2 but picked out by all other baselines.\nIn Fig 12, floors, walls, and sky are segmented as one single object by GENESIS-V2.\nAccording to my memory, almost all previous works on the Object-Room dataset treat those structures separately.\n\nPlease correct me if I am wrong but I don't think this inductive bias is mentioned in the main paper.\nI am not against such inductive bias at all and actually find it rather interesting.\nHowever, IMHO, a more detailed discussion on such strong inductive bias should be added.\n\nSignificance:\nAfter the physical scene graph, GENESIS-V2 is another strong evidence showing that the clustering-based algorithm has its unique potential and deserves more attention from the community.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact is not addressed and there is not much to address I think.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: - The proposed method is tackling an important problem. The reviewers found the approach is reasonable and has some novelty. Demonstration of the method on the real world dataset is a step forward in the line of research. 
- The major concerns from the reviewers are addressed well enough by the rebuttal. 
- The clarity is fair enough but some clarification as pointed by the reviewers would make the paper better. 
Decision: Accept (Poster)