GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
Authors: Martin Engelcke
Oiwi Parker Jones
Ingmar Posner
Venue: NeurIPS 2021 Poster (2021)
Keywords: Generative models
object-centric representations
variational inference

Abstract: Advances in unsupervised learning of object-representations have culminated in the development of a broad range of methods for unsupervised object segmentation and interpretable object-centric scene generation. These methods, however, are limited to simulated and real-world datasets with limited visual complexity. Moreover, object representations are often inferred using RNNs which do not scale well to large images or iterative refinement which avoids imposing an unnatural ordering on objects in an image but requires the a priori initialisation of a fixed number of object representations. In contrast to established paradigms, this work proposes an embedding-based approach in which embeddings of pixels are clustered in a differentiable fashion using a stochastic stick-breaking process. Similar to iterative refinement, this clustering procedure also leads to randomly ordered object representations, but without the need of initialising a fixed number of clusters a priori. This is used to develop a new model, GENESIS-v2, which can infer a variable number of object representations without using RNNs or iterative refinement. We show that GENESIS-v2 performs strongly in comparison to recent baselines in terms of unsupervised image segmentation and object-centric scene generation on established synthetic datasets as well as more complex real-world datasets.

GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner Applied AI Lab, University of Oxford, UK (martin, oiwi, ingmar) ox ac.
Abstract
Advances in unsupervised learning of rereettteprsssntaaiinnn have culminated in the development of a broad range of methods for unsupervised object segmentation and interpretable object-centric to simulated and real-world datasets with limited visual complexity. Moreover, object representations
generation. These methods, however, are limited
often inferred using RNNs which do not scale well large images or iterative refinement which avoids imposing an unnatural ordering objects in an image but requires the priori initialisation of fixed number of object representations. In contrast to established paradigms, this work proposes an embedding- based approach in which embeddings pixels are clustered in a differentiable fashion using stochastic ttickbbreaing process. Similar to iterative refinement, this clustering procedure also leads to randomly ordered object repre- sentations, but without the need of initialising a fixed number of clusters priori. This is used to develop new model, GENESIS-VE v2, which can infer variable number of object representations without using RNNs or iterative refinement. We show that GENESIS- v2 performs strongly in comparison to recent baselines terms of unsupervised image segmentation and object-centric scene generation on established synthetic datasets as well as more complex wallwwrldd datasets.
Introduction
Reasoning about discrete objects in an environment is foundational how agents perceive their surroundings and act in it. For example, autonomous vehicles need to identify and respond to other
road users (e.g. II 2 and robotic manipulation tasks involve grasping and pushing individual objects (e.g. (3). While supervised methods can identify selected objects (e.g. [4] it is intractable to manually collect labels for every possible object category. Furthermore, we often desire the ability to predict, or imagine, how collection of objects might behave (e.g. (6). A range of works have thus explored unsupervised segmentation and object-centric generation recent years (e.g. (73))) These models are often formulated as variational autoencoders (VAES) 37 38 which allow the joint learning of inference and generation networks to identify objects in images and to generate scenes in an object-centric fashion (e.g. [15 17 28
Moreover, such models require differentiable mechanism for separating objects in an image. While some works use spatial transformer networks (STNs) [39] to process crops that contain objects (e.g. [7- The latter avoids the use of fixed-size sampling grids which are ill-suited for objects of varying size. Instead, object representations are inferred either by iteratively refining set of randomly initialised representations
15 others directly predict pixel-wise instance segmentation masks (e.g. [16 27
(e.g. 24) or by using recurrent neural networks (RNN) (e.g. [16-18). ). One particularly
interesting model of the latter category is GENESIS [17] can perform both scene segmentation and
generation by capturing relationships between objects with an autoregressive prior.
NNo affiliated with Google DeepMind.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

As noted in Novotny et al. [40] however, using RNNs for instance segmentation requires processing high-dimensional imputs in sequential fashion which is computationally expensive and does not scale well to large images with potentially many objects. We also posit that recurrent inference is not only problematic from a computational point of view, but that it can also inhibit the learning of object representations by imposing an unnatural ordering on objects. In particular, we argue that this leads to different object slots receiving gradients of varying magnitude which provides a possible explanation for models collapsing to single object slot during training, unless the flexibility of the model is restricted (see (8). While iterative refinement instead infers unordered object representations, requires the a priori initialisation of fixed number of object slots even though the number of objects in an image is unknown.
In contrast, our work takes inspiration from the literature on supervised instance segmentation and adopts an instance colouring approach (e-g- [10 in which pixel-wise embeddings or colours-aid are clustered into attention masks. Typically, either supervised learning signal is used to obtain cluster seeds (e.g. [40] 41 or clustering is performed as a non-differentiable post-processing
operation (e.g. 42 43) Neither of these approaches is suitable for unsupervised, end-to-end learning of segmentation masks. We hence develop an instance colouring stick-breaking process (IC-SBP) to cluster embeddings in a differentiable fashion. This is achieved by stochastically sampling cluster seeds from the pixel embeddings to perform a soft grouping of the embeddings into a set of randomly ordered attention masks. It is therefore possible to infer object representations both without imposing a fixed ordering or performing iterative refinement.
Inspired by GENESIS 17]. we leverage the IC-SBP to develop GENESIS-V2. V2, novel model that learns to segment objects in images without supervision and that uses an autoregressive prior to generate scenes in an interpretable, object-centric fashion. GENESIS- V2 is comprehensively benchmarked against recent prior art [16] [7] 24] on established synthetic datasets- Objjectsoono [44] and ShapeStacks -4hree it performs strongly in comparison to several recent base- lines. We also evaluate GENESIS-V2 on more challenging real-world images from the Sketchy [46] and the MIT-Princeton Amazon Picking Challenge (APC) 2016 Object Segmentation datasets 471. where also achieves promising results. Code and pre-trained models are available at https://gtt http://ggtttu..../ thub. com/ haapmmappleed-eeeee ai- aal-labbgooe sab/genesis
2 Related Work
Unsupervised models for learning object representations are typically formulated either as autoen- coders (e.g. 7100
28) generative adversarial networks (GANs) (e.g. (6). Typical GANs are able to generate images, but lack an associated inference mechanism and often suffer from training
12
instabilities (see e.g. 48 49
A comprehensive review and discussion of the subject is provided in
Greff et al. (50)-
In order to infer object representations, STNs [39 can explicitly disentangle object location by cropping out a rectangular region from an input, allowing object appearance to be modelled in canonical pose (e.g. [7H1O 15)) This operation, however, relies a fixed-size sampling grid which is not well-suited if objects vary broadly in terms of scale. In addition, gradients are usually obtained via bi-linear interpolation and are therefore limited to the extent of the sampling grid which can impede training: for example, if the sampling grid does not overlap with any object, then its location cannot be updated in a meaningful way. In contrast, purely segmentation based approaches (1627 often use RNNs (e.g. (6618) or iterative refinement (e.g. (19)24) to infer object representations from an image. Other works either use fixed number of slots 25 26 or group pixels in a non-differentiable fashion (7). RNN based models need to learn a fixed strategy that sequentially attends to different regions in an image, but this imposes an unnatural ordering on objects in an image. Avoiding such a fixed ordering leads to a routing problem. One way to address this is by randomly initialising set of object representations and iteratively refining them. More broadly, this is also related to Deep Set Prediction Networks [5I], where set is iteratively refined in a gradient-basee based fashion. The main disadvantage of iterative refinement is that it is necessary to initialise a fixed number of clusters priori, even though ideally we would like the number of clusters to be input-dependent. This directly facilitated by the proposed IC-SBP. The IC-SBP and GENESIS-V2 v2 are in this respect also related to the Brick-Breaking VAE [52] which uses stochastic number of latent variables, but does not attempt to explicitly capture the object-based structure of visual scenes.
2

---

Unlike some other works which learn unsupervised object representations from video sequences (e.g. [4). our work considers the more difficult task of learning such representations from individual images alone. GENESIS-V2 is most directly related to GENESTS [7 and ST.OT- LLOT-ATTENTION 24]_ Like GENESIS, the model is formulated as a VAF to perform both object segmentation and object-centric scene generation, whereby the latter is facilitated by an autoregressive prior. Similar to SLOT-ATTENTION, the model uses a shared convolutional encoder to extract a feature map from which features are pooled via an attention mechanism to infer object representations with random ordering. In contrast to SLOT-ATTENTION however, the attention masks are obtained with a parameter- free clustering algorithm that does not require iterative refinement or a predefined number of clusters. Both GENESIS and ALOT-ATTENTION are only evaluated on synthetic datasets. In this work, we synthetic datasets for quantitative benchmarking, but we also perform experiments on two more challenging real-world datasets.
3 GENESIS-V2
An image x of height H. width W, with C' channels, and pixel values in the interval [0, 1] is considered to be three-dimensional tensor [0, 1] This work is only concerned with RGB images where C = 3. but other input modalities with a different number of channels could also be considered. Assigning individual pixels to object-like scene components can be formulated as obtaining a set of object masks π E [0, => x with >n Ti.i. = 1 for all pixel coordinate tuples (i, in an image, where K is the number of scene components. Inspired by prior works (e.g. [16 (22)) and identical to Engelcke et al. [1], this is achieved by modelling the image likelihood (x(x as an SGMM of the form
H W log [o ZI:K) = ΣΣΣ log Σ
K
( o.2)
(I)
The parameters 9 of the model are learned 0r is a fixed standard deviation that shared across object slots. The summation in Equation TT) implies that the likelihood is innvariiiinnnnnaann to the order of the object representations Z1:K (see e.g. [53 (44) provided that Te.j (Z1.k) & inttuuttttionnvvariant This allows the generative model to accommodate for a variable number of object representations.
is also
To segment objects in images and to generate synthetic images in an object-centric fashion re- quires the formulation of appropriate inference and generative models, i.c. (o(21. x) and (x(x ZI:K) pe (z(2zx) ), respectively, where ó are also learnable parameters. In the generative model, is necessary to model relationships between object representations to facilitate the generation of coherent scenes. Inspired by GENESIS [17], this is facilitated by an autoregressive prior
Po (21.k) = II po p2 21:k-1)-
GENESIS uses two sets of latent variables to encode object masks and appearances separately. In contrast, GENESIS-VE v2 uses one set of latent variables Z1:K to encode both, which increases parameter sharing. The graphical model of GENESIS v2 is shown next to related models in Appendix A
While GENESIS relies on a recurrent mechanism in the inference model to predict segmentation masks, GENESIS-V2 instead infers latent variables without imposing a fixed ordering and assumes object latents Z to he conditionally independent given an input image x, i.e., go (2 (x) = II Io (zg x). Specifically, GENESIS-V2 first extracts an encoding with a deterministic UNet backbone. This encoding is used to predict map of semi conolltovlltio pixel embeddings E 12 xW xD; (see (40).. Scmi-convolutional embeddings are introduced in Novotny al. 40 facilitate the prediction of unique embeddings for multiple objects of identical appearance. The embeddings are computed by performing an element- wise addition of pixel coordinates to two dimensions of the embeddings. In this work, we let the pixel coordinates be in the interval [-1, 1] relative to the image centre. Subsequently, the IC-SBP converts the embeddings into set of normalised atlention masks m E [0, 1] X x with Σx M4.j. 1 via distance kernel vh. The spatial structure of the embeddings should induce the attention masks to be spatially localised, but this is not hard constraint. addition, we derive principled initialisations for the scaling- factor of different IC-SBP distance kernels e in Section 3.2
3

---

IC-SBP
TT, X]
:
mg
2K
WK XX
Figure I: GENESIS-V2 overview. The image x is passed into a deterministic backbone. The resulting encoding is used to compute the pixel embeddings Ç which are clustered into attention masks mi:K by the IC-SBP. Features are pooled according to these attention masks to infer the object latents Zı:K These are decoded into the object masks TT:K and reconstructed components X1:K-
GENESIS- v2 uses the attention masks mi:k to pool feature vector for each scene component from the deterministic image encoding. A set of object latents Z1:K is then computed from these feature vectors. This set of latents is decoded in parallel to compute the statistics of the SGMM in Equation (T. The object masks π are normalised with a softmax operator. The inference and generation models can jointly trained VAE as illustrated in Figure Further architecture details are described in Appendix B
Inspired by Locatello al. 241
3.1 Instance Colouring Stick-Breaking Process
The IC-SBP is stochastic, differentiable algorithm that clusters pixel embeddings BHxWxDD
into a variable number of soft attention masks m E [0, 1|"xW* Intuitively, this is achieved by (1) sampling the location (i.j) of a pixel that has not been assigned hard cluster according to the distance of the embedding Çı at the selected pixel location to all other pixel embeddings according to kernel v. and (3) repeating the previous explained or some form of stopping condition is reached. Crucially, the stochastic selection of pixel embeddings as cluster seeds leads to set of randomly ordered soft clusters. Due its conceptual similarity, the method derives its name from more formal stick beke-breking process formulations as can, e.g., be found in Yuan et al. [1] or Nalisnick and Smyth (52).
cluster yet. (2) creating a soft or
steps until all pixels are
The IC-SBP is described more formally in Algorithm T Specifically and inspired by Burgess et al. [16]. scope s e [0, x initialised to a matrix of ones H track the degree to which pixels have been assigned to clusters. In addition, a matrix of seed scores is created once by sampling from uniform distribution
(O, 1) Hx to perform the stochastic selection of pixel embeddings. At each iteration, a single embedding vector Çijj is selected at the spatial location (i. j) which corresponds to the argmax of the element- wise multiplication of the seed scores and the current scope. This ensures that cluster seeds are sampled from pixel embeddings that have not yet been assigned clusters. An alpha mask a4 E [0. RW computed as the distance between the cluster seed embedding Çi,j and all individual pixel embeddings according to distance kernel v/. The output of the kernel y is one two embeddings are identical and decreases to zero as the distance between a pair of embeddings increases. The associated attention mask mæ is obtained by the element-wise multiplication of the alpha masks by the current scope to ensure that the final set of attention masks normalised. The scope is then updated by an element-win wise multiplication with the complement of the alpha masks. This process is repeated until a stopping condition satisfied, at which point the final scope is added as additional mask to explain any remaining pixels.
In this work, we restrict ourselves to soft cluster assignment, leading to continuous attention masks with values in m E [0, 1] H'WW" Unless the attention masks take binary values, several executions of the algorithm will lead to slightly different masks for individual objects. If the mask values are discrete and exactly equal to zero one, however, then the set of cluster seeds and the set of attention masks are uniquely defined apart from their ordering. This can be inferred from the fact that if at each step of the IC-SBP produces discrete mask, then embeddings associated with this mask cannot

---

Algorithm 1: Instance Colouring Stick-Breaking Process embeddings Ç E R Dx
Input:
Output: masks m with my E [0,1]
Initialise: masks m = Ø, scope = = seed scores c ~ U0) E #dxw
while not StopCondition(m) do i.j argmax(s o c); a = DistanceKemel(G, Gig); m.append(s @ o);;
S = 2 (1 a);
end
m.append(s)
be sampled as cluster seeds later on due to the masking of the seed scores by the scope. A different cluster with an associated discrete mask is therefore created at every step until all embeddings are uniquely clustered. Another interesting modification would he to use continuous masks while making the output of the IC-SBP cemmmuattion-qqivariiiii This could be achieved either by directly using the cluster seeds for downstream computations or by separating the mask normalisation from the stochastic seed selection. While the SBP formulation facilitates the selection of diverse set of cluster seeds, the masks could be normalised separately after the cluster seeds are selected by using a softmax operation, for example. An investigation of these ideas is left for future work.
with respect the ordering of the cluster seeds.
In contrast to GENESIS, the stochastic ordering of the masks implies that it is not possible for GENESIS-V2 V2 to learn a fixed sequential decomposition strategy. While this does not strictly apply to the last mask which set equal to the remaining scope, we find empirically that models learn strategy where the final scope is either largely unused or where it corresponds to a generic background cluster with foreground objects remaining unordered as desired. Unlike as in iterative refinement where a fixed number of clusters needs to be initialised priori, the IC-SBP can infer variable number of object representations by using a heuristic that considers the current set attention masks at every iteration in Algorithm I While we use fixed number of K masks during training for efficient parallelism on GPU accelerators, we demonstrate that a flexible number of masks can be extracted at test time with minimal impact on segmentation performance.
3.2 Kernel Initialisation with Comii-Covolltional Embeddings
For semi-convolutional embeddings be similar according a distance kemel th. the model needs to learn to compensate for the addition of the relative pixel coordinates. It can achieve this by predicting delta vector for each embedding to a specific pixel location, for example to the centre of the object that the embedding belongs to. A corollary of this is that if embeddings are equal to the relative pixel coordinates with the other dimensions being zero, then clustering embeddings based on their relative distances results in "blob-like", spatially localised masks. In this work, we make use of this property to derive meaningful initialisation for free parameters in the distance kernel &i of the IC-SBP. Established candidates for & from the literature are the Gaussian dे [40], Laplacian tiz and Epanechnikov bh kernels [55] with
tha = -xxp (- [[u-v]12)
VE = exp (- [|u-vvl]
th max (1-1lu-vi l|u-v||² .0) 0).
(3)
whereby and are embeddings of equal dimension. Each kernel contains a scaling factor (G.LLE) E R By initialising the model at the beginning of training so that the embeddings are equal to the relative pixel coordinates with the other dimensions being zero, then (GG can be initialised so that the initial attention masks are similarly circular patches. In particular, we initialise these scaling factors
-O'= K T-'
/Kln2 oF -=
which derived and illustrated in Appendix C After initialisation, ((G.L.)) is jointly optimised along with the other learnable parameters of the model as in Novotny al. [40].
5

---

3.3 Training
Following Engelcke et al. 17. GENESIS- v2 is trained by minimising the GECO objective [56]. which can be written as a loss function of the form
b = E (=/x)| In (x z)] + Bg KL (z x)
po(z]]
(5)
The relative weighting factor 3g E Rt is updated at every training iteration separately from the model parameters according to
8g - Bg - CC(C)
with a= E (1 ag)) (x)( po 2)].
(6)
E R is an exponential moving average of the negative image log- liglliielhhood, ag E [0, 1] is momentum factor, E IR is step size hyperparameter, and C' R is a target reconstruction error. Intuitively, the optimisation decreases the weighting of the KL (Kullback-eeible term as long as the reconstruction error is larger than the target C. The weighting of the KL term is increased again once the target is satisfied.
regularisation
In some applications, a practitioner might only require segmentation masks in which case, so having to reconstruct the entire input would be rather inefficient. While we observed that the attention masks m are correlated to the object masks т, they do not align as closely with object boundaries. We conjecture that this is consequence of the large receptive field of the UNet backbone which spatially dilates information about objects. Consequently, we also conduct experiments with an additional auxiliary mask consistency loss that encourages attention masks and object masks to be similar. This leads to a modified loss function of the form
Sgg = E + Bg (KL(g0 x) || Po())] + KL|m || nograd(
(7)
in which m and π are interpreted as pixel-wise categorical distributions. Preliminary experiments indi- cated that stopping the gradient propagation through the object masks helps achieve segmentation quality comparable to using the original loss function in Equation (5)
Experiments
This section presents results on two simulated datasets ObjectsRoom 44 and ShapeStacks 45 as well as two real-world
[46] and APC [47 which are described in Appendix D GENESIS-V2 2 is compared against three recent baselines: GENESIS 17 MONET [66,, and SLOT- ATTENTION 24 Even though SLOT-ATTENTION is trained with a pure reconstruction objective and not generative model, it is an informative and strong baseline for unsupervised scene segmentation. The other models are trained with the GECO objective [56] following the protocol from Engelcke al. 17 for comparability. We refer to MONET trained with GECO as MONET-G to avoid conflating the results with the original settings. Further training details are described in Appendix E
segmentation quality is quantified using the Adjusted Rand Index (ARI) (57) and the Mcan Segmentation Covering (MSC). The MSC is derived from Arbelaez et al. [58] and described in detail in Engelcke et al. [17]. These are by default computed using pixels belonging to ground truth foreground objects (ARI-FG and MSC-FG). Similar to Greff al. (22) these are averaged over 320 images from respective test sets. We also report the Evidence Lower Bound (ELBO) averaged over 320 test images as a measure how well the generative models able to fit the data. Generation quality is measured using the Fréchet Inception Distance (FID) [59 which is computed from 10,000 samples and 10,000 test set images using the implementation from Seitzer [60] When models trained with multiple random seeds,
Following prior works (e.g. [17
18 22 24
always show qualitative results for the seed that achieves the highest ARI-FG. In terms of the two real-world datasets, there are only ground truth segmentation masks available for the APC data, with the caveat that there is only a single foreground object per image. In this case, when computing the segmentation metrics from foreground pixels alone, the ARI-FG could be trivially maximised by assigning all image pixels to the same component and the MSC-FG would be equal to the largest IOU between the predicted masks and the foreground objects When considering all pixels, the optimal solution for both metrics is to have exactly one set of pixels assigned to the foreground object and another set of pixels being assigned to the background. While acknowledging that segmenting the background as a single component is arguably not the only valid way of segmenting the background, we report the ARI and MSC using all pixels instead to develop a sense of how well foreground objects are separated from the background.

---

Imputs
Segmentations Raconstructions
(a) MONET-G
(h) GENESIS
(c) STOT-ATTENTION
(d) GENESIS-V2
Figure 2: GENESIS-V2 learns better reconstructions and segmentations on ShapeStacks.
Table I: Means and standard deviations of the segmentation metrics from three seeds. Bold values in the first half of the table indicate the best values for the generative models; bold values in the second half indicate any better values achieved by the additional non-generative baseline.
ObjectsRoom
ShapeStacks
Model
Generative
ARI-FG
MSC-FG
ARI-FG
MSC-FG
MONET-G
Yes
0.54100.0
0.3310001
0.70+0.04
0.57:10.12
GENESIS
Yes 0.63:10.03
0.53+00.7
0.70:10.05
0.67:10.22
GENESIS-V2
Yes
0.85
0.59
0.01
0.67
SLOT-ATTENTION
0.79:10.02
0.64
0.76+10.01
0.70:L0.05
4.1 Benchmarking in Simulation
Figure shows qualitative results on ShapeStacks and additional qualitative results are included in Appendix F GENESIS-V2 cleanly separates the foreground objects from the background and the reconstructions. Interestingly, GENESIS-V2 is the only model that segments the entire background as single component. We conjecture that this behaviour structures are of finite variety. As result, even when viewing only a fraction of the background, is possible for the model largely predict the appearance of the rest of the background. The KL regularisation during training encourages efficient compression of the inputs, which penalises redundant information between slots and might thus explain this behaviour. trained without KL. regularisation and the decoders of MONET-G well GENESIS are possibly not flexible enough to reconstruct the entire background as single component (see Appendix for architecture details).
consequence of the fact that the background
AOT-ATTENNION is
In terms of quantitative performance, Table I summarises the segmentation results ObjectsRoom and ShapeStacks. GENESIS-V2 outperforms the two generative baselines GENESIS and MONET-G across datasets on all metries, showing that the IC-SBP is indeed suitable for learning object-centric representations. GENESIS-V2 outperforms the non- non-generative SLOT-ATTENTION baseline in terms of the ARI-FG both datasets. SLOT-ATTENTION manages achieve better mean MSC-FG. The standard deviation of the MSC-FG values is much larger, though, which indicates training is not as stable. While the ARI-FG indicates the models ability to separate objects, it does not penalise the undersegmentattion of objects (see [7]. The MSC-FG, in contrast, is an IOU based metric and sensitive
exact segmentation masks. conjecture that SLOT- ATTENTITN manages predict slightly more accurate segmentation masks given that is trained on pure reconstruction objective and without KI. regularisation, thus leading to slightly better mean MSC-FG.
ablations for
GENESIS- v2 is also included in Appendix
7

---

Table 2: Means and standard deviations of the segmentation metries from three seeds for GENESIS- with a fixed or flexible number of object slots. Highlighting follows an analogous scheme as in Table T
Dataset
Training
Slots
Avg.
MAE 4
ARI-FG f
MSC-FG
ObjectsRoom
No mask loss
Fixed
7.0:0.00 5.0
3.3+0.0
0.8510001
0.59
Flexible
1.740.9
0.84#0.01
0.51
ShapeStacks
No mask loss
Fixed
9.0:t0.0
4,4+0.0
0.81
0.67
Flexible
6.3+0.3
1.9+0.3
0.77=0.02
0.63#00.01
With mask loss
Fixed Flexible
9.0+10.0
4.4
0.01
0.68:10.00 0.68
5.7:0..2
1.1:0.002
0.81
Table Means and standard deviations of ELBO values and FID scores from three seeds.
Model
ObjectsRoom
ShapeStacks
ELBB f
FID
ELBO 1
FID 4
MONET-G
-7217=19
205.717.6
17268119
197.8=5.2
GENESIS
-7023-12
62.8+22.5
-7082+15
186.8=18.0
SLOT-ATT.
GENESIS-V2
-7040+2
52.6+2.7
-7019+22
112.7#3.2
We also examine whether the IC-SBP can indeed be used to extract variable number of object representations. This is done by terminating the IC-SBP according to a manual heuristic and setting the final mask to the remaining scope. Specifically, we terminate the IC-SBP when the sum of the current attention mask values smaller than 70 pixels for ObjectsRoom and 20 pixels for ShapeStacks. A larger threshold is used for the former the attention masks tend to be more dilated (see Appendix F). The average number used slots, the Mean Absolute Error (MAE) to the ideal number of slots, and segmentation metrics when using fixed number and variable number slots after training are summarised in Table 2 On both datasets, allowing for a flexible number requires fewer steps and achieves smaller MAF. This is incurred, though, at drop in segmentation performance. When training GENESIS-V v2 with the auxiliary mask loss as in Equation (7) on ShapeStacks, the average number of steps and the MAE further decrease at impact the segmentation metrics. On ObjectsRoom, the auxiliary mask loss appeared to deteriorate the learning of good object segmentations and the associated results therefore not included.
terms of density estimation and scene generation, T summarises the ELBO values and FID scores for GENESIS-V2 v2 and the baselines ObjectsRoom and ShapeStacks GENESIS achieves slightly better ELBO than GENESIS-U v2 on ObjectsRoom, but GENESIS-VE v2 performs significantly better StapeStacks. We hypothesise that GENESIS benefits here from having more flexible, autoregressive posterior than GENESIS-V2. Regarding scene generation, GENESIS-V2 consistently performs best with particularly significant improvement on ShapeStacks. Results ShapeStacks, however, are not as good as for ObjectsRoom, which is likely caused by the increased visual complexity of the images in the ShapeStacks dataset. Qualitative results for scene generation shown in Figures 3 and Both GENESIS and GENESIS-V2 produce reasonable samples after training on ObjectsRoom. For ShapeStacks, samples from GENESIS contain significant distortions. In comparison, samples from GENESIS-V2 are more realistic, but also still show room for improvement.
NNote that in contrast to the MONET-G training objective, the decoded mask distribution rather than the deterministic attention masks are used to compute the reconstruction likelihood in the ELBO calculation for MONET-G. The KL divergence between the two mask distributions as used in the training objective is not part of this ELBO calculation.

---

@ MONET-G
MONET-G
GENESIS
(b) GENESIS
(c) GENESIS-V2
(c) GENESIS-V2
Figure 3: ObjectsRoom samples.
Figure 4: ShapeStacks samples.
Inputs
Sigmentations Recanstruction
(a) MONET-G
(b) GENESIS
(c) SLOT-ATTER ATTENTION
(d) GENESIS-V2
Figure 5: In contrast to MONET-G and GENESIS, GENESIS-V2 V2 as well as SLOT-ATTENTIO are able learn reasonable object segmentations on the challenging Sketchy dataset.
4.2 Real-World Applications
After validating GENESIS-V2 on two simulated datasets, this section present experiments on the Sketchy [46] and APC datasets [47; two significantly more challenging real-world datasets collected in the context of robot manipulation. Due to the long training time on these datasets, each model is only trained with single random seed. While this makes it infeasible draw statistically strong conclusions, the aim of this section is to provide an indication and early exploration of how these models fare on more complex real-world datasets. Reconstructions and segmentations after training GENESIS-V2 on Sketchy and APC are shown in Figures 5 and 6 For Sketchy, it can be seen that GENESIS-V2 and SLOT-ATTENTION disambiguate the individual foreground objects and the robot gripper fairly well. SLOT-ATTENTION produces slightly more accurate reconstructions. which is likely facilitated by the pure reconstruction objective that the model is trained with. However, GENESIS-VE v2 is the only model that separates foreground objects from the background in APC images. Environment conditions Sketchy are highly controlled and SLOTT ATTENTION appears to unable to handle more complex conditions in APC images. Nevertheless, GENESIS v2 also struggles to capture the fine-grained details and oversegments one the foreground objects into several parts, leaving room for improvement in future work. Additional qualitative results are included in Appendix

---

Imputs
Segmentttiinn on Raconstructions
OME
(a) MONET-G
(b) GENESIS
(c) SLOT-ATTENTION
(d) GENESIS-V2 V2
Figure 6: GENESIS-V2 the only model that separates the foreground objects in images from APC.
Table 4: Segmentation metrics on APC.
Table 5: FID scores on Sketchy and APC.
ARI
MSC
Sketchy
APC
MONET-G
0.11
0.48
MONET-G
294.3
269.3
GENESIS
0.04
0.29
GENESIS
241.9
183.2
SLOT-ATT.
0.03
0.25
SLOT-ATT.
GENESTS-V2
0.55
0.67
GENESIS-V2
208.1
245.6
Table reports the ARI and MSC scores computed from all pixels GENESIS- V2 and the baselines.
GENESIS-V v2 stands out in terms of both metrics, corroborating that GENESIS-V v2 takes valuable step towards learning unsupervised object-representations from real-world datasets. FID scores for generated images are summarised in Table 5 and qualitative results are included in Appendix F GENESIS-V2 v2 achieves the best FID on Sketchy, but outperformed by GENESIS on APC. Both models consistently outperform MONET-G. All of the FID scores, however, are fairly large which is not surprising given much higher visual complexity of these images. It is therefore difficult to draw strong conclusions from these beyond rough sense of sample quality. Further work is required to generate high-fidelity images after training on real-world datasets.
Conclusions
This work develops GENESIS-V2. v2, novel object-centric latent variable model of scenes which is able to both decompose visual scenes into semantically meaningful constituent parts while at the same time being able to generate coherent scenes in object-centric fashion. GENESIS-V2 leverages differentiable clustering algorithm for grouping pixel embeddings into a variable number of attention masks which used infer unordered set of object representations. This approach is validated empirically on two established simulated datasets as well as two additional real-world datasets. The results show that GENESIS-V v2 takes a step towards learning better object-centric representations without labelled supervision from real-world datasets.
In terms of future work, there is still room for improvement in terms of reconstruction, segmentation, and sample quality. would also be interesting to investigate the ability GENESIS- v and the IC-SBP to handle Moreover, several interesting variations of the IC-SBP were also already described in Section [3.1 Additional promising avenues include the extension of GENESIS-V2 to learn object representations from video (see e.g. [33 14) or to leverage recent advances in hierarchical latent variable models such Nouveau VAEs (NVAEs) 661.
images that contain more objects than seen during training.
10


Reviews:
[{"review_id": "VEmXaySqEcm", "reviewer": "Reviewer_HXzc", "summary": "Authors propose an object-centric generative model of images. This is derived from GENESIS, but replaces iterative inference with a clustering approach inspired by recent instance segmentation techniques. Results on two synthetic and two real datasets show significantly better performance than recent methods GENESIS and Slot Attention.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper is generally clear and well-written. Components of the model are motivated reasonably and described comprehensibly.\n\nThe model itself is reasonably simple and elegant -- particularly the idea of using an embedding-clustering approach to allow pixels to become grouped in a more 'bottom-up' fashion than existing methods. This approach is novel within the field of structured generative modelling, but inspired by existing techniques in instance segmentation.\n\nThe evaluation is reasonably comprehensive, and includes some real-world (albeit rather tightly controlled) data, which is welcome. Results show the method out-performs GENESIS and Slot Attention, by varying degrees, on most datasets, for both generation and segmentation/decomposition tasks.\n\nConcerns / suggestions / omissions:\n\nThe technical contribution fairly small. In particular, the model combines ideas from GENESIS, MONet, and clustering-based segmentation, but it doesn't do anything radically new.\n\n155: what is the stopping condition? It seems this could be important, as it affects how likely the model is to over-/under-segment\n\nFig.2: related to the previous, please discuss what inductive bias of the proposed method means it prefers to treat floor+walls as one object? There is not really anything in the data that means this is the 'correct' segmentation (even though it might look correct to a human)\n\nTab.1: the bolding here seems inconsistent with the standard deviations -- it looks from the numbers like GENESIS is statistically closer to GENESIS-V2 than the bolding would imply?\n\n130: the described model does not seem to fit the usual structure of a stick-breaking process -- or at least, no more than MONet does --; maybe justify this description, or think of something more appropriate? (this may relate to the previous point; if the stopping condition were that some bernoulli rv is 1, the name would be more apt)\n\n\"Generative modeling of infinite occluded objects for compositional scene representation\" [Yuan, ICML 19] should be cited.\n\n35: why is GENESIS particularly interesting vs. the many other models cited?\n\n113: formulating an inference model seems to imply use of SGVB, which has not been stated yet\n\n---\n\n## Post-rebuttal\n\nThe authors have addressed several of my concerns in their comment, and I remain somewhat in favor of acceptance. The technical contribution (the IC-SBP itself) is relatively small, but still of interest. The issues raised by KxNW and BKZJ are of some concern, but seem to me well-enough addressed by the respective author responses.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, both fine.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "MlaHT30KcJS", "reviewer": "Reviewer_KxNW", "summary": "This paper proposes a new object-centric generative model called GENESIS-v2. Unlike prior RNN-based approaches using sequential attention, it yields an unordered collection of object representations; and unlike \"clustering approaches\" based on iterative refinement it does in principle not require fixing the number of slots a priori. This is achieved by extending GENESIS (Engelcke et al., 2020) with a differentiable clustering method based on an Instance Colouring Stick-Breaking Process (IC-SBP). \n\nHere, coordinate-aware (i.e. semi-convolutional) pixel-level embeddings are clustered iteratively by first randomly selecting a cluster seed and then comparing its associated pixel-embedding to all other pixel-embeddings using a learnable distance kernel to extract a mask. When the masks are binary these masks are uniquely defined up to their ordering. A heuristic can be used to decide when to terminate the clustering process and assign the remaining image content to a \"background\" mask. Once attention masks are obtained, they can be used to derive the associated feature vectors to obtain slot representations, which can be decoded separately as in prior work (eg. IODINE/Slot-Attention). GENESIS-v2 is evaluated on a number of synthetic and more real-world datasets, where it is shown to outperform existing approaches in terms of clustering and generative capabilities.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an interesting alternative to the two common paradigms explored in object-centric models for representation learning. A U-net style auto-encoder computes pixel-level embeddings, which are made coordinate-aware by adding their relative pixel coordinates to two dimensions of the embedding as proposed in Novotny et al. (2018). A sequential Instance Colouring Stick-Breaking Process (IC-SBP) then uses these embeddings to derive attention masks from which object representations can be computed similar to in prior work. In particular, the attention-masks are combined with the pixel-embeddings and pooled (and transformed) to obtain parameters of a factored posterior distribution from which object representations can be obtained. Decoding is as in Slot-Attention (Locatello et al., 2020) using separate decoders for each so, which output RGB and a segmentation logit, which can be normalized to obtain a masking. Training proceeds via the Generalized ELBO with Constrained Optimization (GECO) algorithm from Rezende et al. (2018). Additionally an \"auxiliary mask loss\" can be added to force the attention masks and the segmentation masks to be similar.\n\nThe most interesting and novel aspect of GENESIS-v2 is therefore the IC-SBP for clustering. \nThe IC-SBP makes use of a kernel distance function for comparing pixel embeddings, and it is shown how a meaningful initialisation of their parameters (which are learned) can be derived to yield an initial clustering that incorporates a spatial inductive bias (Figure 9 in Appendix C). Further, although the ordering of the cluster seeds in the IC-SBP is stochastic, it is argued that when the masks are binary, the masks are indeed uniquely defined. This is important, since otherwise the masks may depend on the order in which the cluster seeds were selected, which is undesirable as in the case of RNN-based approaches. However, actually using such discrete mask values is not explored and all experiments are conducted using continuous mask values as far as I can tell. Another potential advantage of using IC-SBP is that one can in principle dynamically decide when to terminate the clustering process using a heuristic (and thus how many slots are obtained) as opposed to committing to a fixed upper bound beforehand. However, here again the paper falls short, as no attempt is made at deriving a general/useful heuristic. Rather, only a naive heuristic to terminate IC-SBP when the sum of the attention mask values < 20 is explored, but which is (understandably) observed to perform poorly on other data sets. Finally, this heuristic is only explored during inference, while training always proceeds with a fixed K as in prior methods.\n\nThe experiments are extensive and GENESIS-v2 is compared against GENESIS, MONet, and Slot-Attention on both synthetic and more real-world data. Although the former two baselines are trained using the GECO objective and benefit from the hyper-parameter tuning done for GENESIS-v2, it appears that no further tuning was done for Slot-Attention, which seems somewhat unfair for the more real-world datasets. Regardless, on the synthetic datasets it is convincingly shown how GENESIS-V2 is often the best performing model, when using a fixed K, both in terms of generation (FID) and clustering (ARI & MSC). A metric to measure reconstruction quality such as MSE or nll is strangely absent, which would allow for a better comparison in terms of how well GENESIS-v2 is able to model the data. Currently it seems that the reconstructions by GENESIS-v2 are quite poor compared to say Slot-Attention. For example, in Figure 12 on ObjectsRoom it can be seen how it misses entire objects (4th panel from left) and are quite blurry such as in Figure 5. I agree with the authors that the generative results show room for improvement, since although GENESIS-v2 improves in terms of FID, the absolute FID is still quite high and the image quality is generally quite poor.\n\nOn the more real world datasets similar observations hold true and GENESIS-v2 is generally the best performing model. However, here I was quite surprised that K=10 is used for Sketchy even though the images only contain 1 gripper + 3 objects. Similarly, for APC the images contain only a single object, yet K=10 is used. This is significantly different for the hyper parameter choices on the synthetic datasets, where far more objects can be observed and yet K < 10. The reason that I am bringing this up is because baseline methods may be disadvantaged by this choice. For example, Slot-Attention requires all slots to participate in the iterative grouping process, and thus a large K will necessarily yield in more competition among the components. Clearly, it can be argued that this is a disadvantage of such methods and an advantage of GENESIS-v2 that it does not suffer from this. However, in that case, I do expect a set of experiments with a much more reasonable K to verify that baseline methods suffer indeed only because of the misspecification (and perform well if K is set more accurately). I would recommend K = 5-7 for Sketchy and K = 2-4 for APC.  \n\nIn summary, although I find that the contribution is interesting and a promising direction for future work on object-centric generative models, I can not yet recommend it for acceptance. Indeed, the novel clustering approach proposed here is mainly motivated as (1) leading to randomly ordered object representations without dependencies between them, and (2) not needing to set a fixed number of clusters a priori. However, (1) requires evaluating IC-SBP with discrete masks, which is not done, and (2) is insufficiently explored: no attempt is made at developing a general heuristic, no experiments are provided when training using the current heuristic, etc. I should note that I agree that it has been shown that GENESIS-v2 behaves well when K is misspecified, but this is a lot less significant and not how the current approach is motivated. Further, the experimental evaluation needs to be improved: a measure of reconstruction quality should be provided, equal tuning needs to be applied to all baselines, and on the more real-world datasets experiments need to be included with a more reasonable choice of K.\n\n**Minor Comments**\n\n* It would be good to clarify the \"relative\" in \"relative pixel coordinates\". From Figure 8 in the appendix I assumed that these coordinates are relative to the image center?\n\n* It took me a while to figure out how the MSC metric is computed and a ref to the paper where this is described (I believe it is derived from Arbelaez et al. 2010?) \n\n* I think the results in Figure 9 are quite nice and would suggest moving that to the main text.\n\n* Similarly, I would suggest moving some of the technical detail regarding decoding, etc. to the main text and integrate it into the discussion. Currently, the description in the main text is quite high-level for those not familiar with prior work in this area.\n\n* I was a little confused at the naming initially since the decoder is more like Slot-attention / IODINE rather than GENESIS. Perhaps this model deserves its own name?\n\n* Please take a moment to go through the references to correctly cite prior work that has been published (as opposed to citing the arxiv version)\n\n**Post-rebuttal Update**\n\nI think that this paper has a lot of potential, but is currently borderline mainly because the authors fail to provide sufficient empirical support regarding dynamically inferring the correct number of components. In particular:\n\n* No more general heuristic is developed as a stopping condition, which leaves it unclear what stopping condition should be to obtain good performance w.r.t. dynamic component selection\n* No experiments are provided where training proceeds using a dynamic number of objects, which leaves it unclear how this affects performance.\n\nThere are also other issues regarding empirical evaluation, such as how K is chosen for the baselines and the lack of reporting reconstruction error (or nll). I have corresponded with the authors regarding these issues, and there is a clear disagreement about the importance of these. One concern I initially had was unfounded due to a misunderstanding on my part and was clearly addressed. Unfortunately, there appears to be no interest in addressing the other issues, even though some appear easy to fix. Because of this, I still remain slightly in favor of a borderline reject (5).\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is some discussion about the limitations of this model throughout the text, which is sufficient in my view. The societal impact statement is rather brief and generic, but which is also somewhat understandable for this line of work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "L_JZDdll86s", "reviewer": "Reviewer_BKZJ", "summary": "This work approaches object-centric decomposition and generative modeling using a combination of prior works, including Genesis [1], Slot Attention [2], semi-convolutional operators for instance segmentations [3], and MONet [4].", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- IC-SBP is an interesting combination of existing approaches. It is described reasonably along with the implications of using binary or non-binary masks and possible extensions.\n- Multiple metrics (ARI, MSC, and FID) help paint a detailed picture.\n- Ablations cover the main tricks used including the mask loss term, semi-conv operators, and auto-regressive prior.\n- Decent dataset coverage (including real world data), but nevertheless simplistic and far from game changing.\n\nQuestions:\n- To what extent is instance coloring related to mean shift clustering? It would be helpful to explore any parallels with the statistics/stochastic processes literature.\n- Could you shed some light on the quality of the representations (latents) achieved by Genesis V2? My concern is the spatial pooling used to produce latent statistics (a la Slot Attention) may not be amenable to interpretable features despite the latent bottleneck. \n- What is the qualitative effect of sampling different seed scores c on the segmentation outputs?\n- What is the rationale of optimizing the scale parameter \\sigma of the distance kernel via gradient descent? To what extent did it vary over the course of training in your experiments? Does the value correlate strongly with the number of non-empty masks?\n\nWeaknesses:\n- Not enough novelty/exploration of topics of interest.\n- One of the claimed advantages (variable number of objects per scene) is not on solid ground, as it doesn’t seem achievable in ObjectsRoom. It requires a heuristic with an additional hyperparameter (number of pixels last explained) even where it works.\n- To compare fairly to the baseline models, it is not sufficient to train them using the same protocol, even if it is GECO using a common target reconstruction likelihood. Two things are missing and seem essential:\n   (1) A scatterplot showing reconstruction (log LL) versus compression (KL) tradeoffs across the models. \n   (2) Results with different reconstruction error targets. Whether a model can achieve a target log LL depends (among other things) on the capacity of its decoder. A particular concern is that MONet-G looks poorly trained on ObjectsRoom. Compared to its performance in the original paper on the same dataset, the way it is trained here suggests a training deficiency.\n- Regarding the use of metrics in the field, there’s a claim in the paper which is important but unjustified: ARI “does not penalize the undersegmentation of objects.” Such claims are important in shaping future work in the field and must be properly justified. The only citation here is to [1], where Figure 13 confusingly asserts that “ARI does not penalise the over-segmentation of the foreground objects”. To my knowledge, ARI is a clustering metric which will penalize both under- and over-segmentation. It’s only ARI-FG (which excludes background pixels from the computation) which can lead to a situation like Figure 13 in [1]. It’s unclear whether that figure is computing the ARI for all pixels or just ARI-FG. In the context of this work, it is unclear where ARI-FG will differ from MSC-FG given they’re both excluding background pixels.\n\nSuggestions:\n- Figure 1 could be more helpful. Consider labeling the UNet, Conv-GN8-ReLU blocks, etc.\n- Consider comparing to a non-parametric segmentation algorithm like QuickShift++ [5] (with a similar number of clusters).\n- Move the distance kernel descriptions to the appendix. They’re not used anywhere in the main text.\n\nReferences:\n[1] Engelcke, Martin, et al. \"Genesis: Generative scene inference and sampling with object-centric latent representations.\" arXiv preprint arXiv:1907.13052 (2019).\n[2] Locatello, Francesco, et al. \"Object-centric learning with slot attention.\" arXiv preprint arXiv:2006.15055 (2020).\n[3] Novotny, David, et al. \"Semi-convolutional operators for instance segmentation.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n[4] Burgess, Christopher P., et al. \"Monet: Unsupervised scene decomposition and representation.\" arXiv preprint arXiv:1901.11390 (2019).\n[5] Jiang, Heinrich, Jennifer Jang, and Samory Kpotufe. \"Quickshift++: Provably good initializations for sample-based mean shift.\" International Conference on Machine Learning. PMLR, 2018.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors show room for improvement in the generation (sampling) of ShapeStacks and APC scenes. They also discuss the limitations of producing a variable number of slots/object masks. It would be great if they could further describe the failure modes of the algorithm or the implications of initial conditions like the seed scores c.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "6qkUWZemITA", "reviewer": "Reviewer_dzPr", "summary": "This work proposes a new object centric generative model called genesis-v2. The main contribution is using an Instance Colouring Stick-Breaking Process for assigning pixels to slots. The idea is to randomly select a pixel as seed, assign close pixels based on a kernel, select another random pixel from the remaining unassigned pixels for K steps. Authors claim two advantages, 1: no ordering can be imposed, 2: number of clusters don't need to be set initially. This work shows results on two synthetic datasets and two real world ones and compared to their baseline and slot attention they achieve impressively better results both qualitatively and quantitatively. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using Instance colouring clustering technique with random seed although is closely related to slot attention but has some novel aspects. Since the slots in slot attention are also randomly initialized the effect is similar to the icsbp proposed here (no specific ordering). \n\nAuthors claim that slot attention needs a preset number of slots, whereas they don't. This is not completely true. First, for practical reasons of gpu utilization genesis-v2 also uses a fixed number during training. Their ablation in appendix shows their best results is with a fixed number during inference. So their flexibility claim is not beneficial or advantageous at least in this dataset. In the objectRoom the flexible inference is even disadvantageous. As explained in slot attention, they also can adapt to a flexible number of objects at inference. I would argue that slot attention also fixes the number of slots during training for training efficiency and better parallelization. So there is no restriction in that regard. \n\nFor the distance kernel, authors add the pixels position to two dimensions of the embedding and then usually use a gaussian distance kernel. Interestingly they derive an initial value for the sigma based on the set K (number of slots). Again it goes against the flexibility claim, assuming a set K. Deriving the initial sigma based on equal initial mass assignment is interesting and useful.\n\nThe most impressive aspect was the experimental results. Qualitatively, they have almost perfect background segmentation. Typically, unsupervised segmentation masks tend to over segment background. This aspect of their results is intriguing to me. There is not much discussion on why genesis-v2 is better at recognizing background. An ablation on K during training can potentially provide insight. My question is would it still segment background/foreground with a larger K, such as 32 instead of 9. \n\n----POST Rebuttal\nThe idea presented here is interesting and experiments show it most probably helps with symmetry breaking and avoiding bad local minimas. Other reviewers concerns like training all with Geco and Monet-g not being trained properly has probably merit but the results in Genesis-v2 are better in my experience. Finetuning the baselines better most probably will decrease the gap but Genesis-V2 has legitimately  good results.\nTherefore, this paper is contributing to the community in my opinion which leans me toward acceptance.\nIt would still be helpful to include results for a larger K such as 32 to showcase the applicability of this model with larger variety of number of objects in each scene.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors indicate some of the limitations in the manuscripts with future work to address them. One of the main limitations is the training speed which makes it infeasible to try on a real segmentation dataset like Pascal os MS CoCo. On the fairly simple real world APC dataset, it takes about 8 days to train.\n\nAnother limitation is probably the number of objects it can handle. I would be quite surprised if it can be scaled to handle ~ 100 objects in one scene, like images with pedestrians in busy street scenes. Mainly due to full image pixel level multiplications and attention maps per object, auto regressive prior and the for loop of the colour clustering.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach to object learning that performs well on two synthetic and two real-world benchmarks. Even after the discussion phase, the reviewers could not agree on the evaluation of the paper: two of them are in favor of acceptance, while two lean towards rejection. After reading the reviews, the author responses, the discussion, and the paper itself, below is a summary of some key points.

Pros:
1) Reasonable model, which, interestingly, potentially is less rigid in terms of pre-fixing the number of object slots than previous models
2) Good empirical performance on two real-world and two synthetic datasets. Evaluation on real-world datasets is a step in the right direction.

Cons:
1) There are doubts about the evaluation being fair to the baselines in terms of hyperparameter tuning. In particular the number of slots on the real-world datasets seems potentially mis-specified. (it is great that the proposed model can deal with this, but trying baselines with the appropriate number of slots is still a crucial experiment)
2) The paper combines ideas from several prior lines of work - GENESIS, Slot Attention, Novotny et al., stick-breaking processes. It is unclear which components actually make it work. There are some ablation studies, but they are performed on synthetic datasets and almost none of them show any significant change.  On all datasets except for one the model works very similar to Slot Attention. The only big difference is on the APC dataset, but it is very unclear where it comes from. The KL penalty? The IC-SBP? The misspecified K in slot attention? Some of the other training parameters? It is really unclear.
3) The claims about adaptive number of objects should be clarified - while I believe there is substance to them, it is apparent from the reviewers' comments that the current presentation can be confusing.

To summarize, the paper presents a well-working unsupervised object segmentation system, but there are some concerns about comparison to the baselines and there is lack of analysis of the reasons for the good performance of the model. The latter point in my opinion is crucial, since without that it is impossible for other researchers to make good use of the paper. Papers should be not (just) about showing off SOTA results, but about communicating clear and usable insights. Thus, at this point I recommend rejection. 

(The work does look promising and I encourage the authors to take the feedback into account and re-submit to a different venue. The resistance of the authors to the comments/requests from some of the reviewers is a bit surprising and does not serve them well)
Decision: Accept (Poster)