Attention Normalization Impacts Cardinality Generalization in Slot Attention
Authors: Markus Krimmel
Jan Achterhold
Joerg Stueckler
Venue: Accepted by TMLR ()

Abstract: Object-centric scene decompositions are important representations for downstream tasks in fields such as computer vision and robotics. The recently proposed Slot Attention module, already leveraged by several derivative works for image segmentation and object tracking in videos, is a deep learning component which performs unsupervised object-centric scene decomposition on input images. It is based on an attention architecture, in which latent slot vectors, which hold compressed information on objects, attend to localized perceptual features from the input image. In this paper, we demonstrate that design decisions on normalizing the aggregated values in the attention architecture have considerable impact on the capabilities of Slot Attention to generalize to a higher number of slots and objects as seen during training. We propose and investigate alternatives to the original normalization scheme which increase the generalization capabilities of Slot Attention to varying slot and object counts, resulting in performance gains on the task of unsupervised image segmentation. The newly proposed normalizations represent minimal and easy to implement modifications of the usual Slot Attention module, changing the value aggregation mechanism from a weighted mean operation to a scaled weighted sum operation.

Published in Transactions on Machine Learning Research (09/2024)
Attention Normalization Impacts Cardinality Generalization in Slot Attention
Markus Krimmel
Embodied Vision Group, Man Planck Institute for Intelligent Systems
Jan Achterhold
jon. wor..uchcertodd@itucciinn......
Embodied Vision Group, Mar Planck Institute for Intelligent Systems"
Joerg Stueckler
Jberg.stuecklerr@unt
Embodied Vision Group, Maz Planck Institute for Intelligent Systems Intelligent Perception in Technicol Systems Group, Universily of Augsburg
Reviewed on OpenReview: https: http:///pprreii.eeee net/ per/formm? id= LLQXLfbGOg
Abstract
Object-centric scene decompositions important representations for downstream tasks in fields such computer vision and robotics. The recently proposed Slot Attention module, already leveraged by several derivative works for image segmentation and object tracking in videos, is a deep learming component which performs unsupervised object-centric scene decomposition on input images. It is based on an attention architecture, in which latent slot vectors, which hold compressed information on objects, attend to localized perceptual features from the input image. In this paper, we demonstrate that design decisions on normalizing the aggregated values in the attention architecture have considerable impact the capabilities of Slot Attention to generalize to higher number of slots and objects during training. We propose and investigale alternatives to the original normalization scheme which increase the generalization capabilities of Slot Attention to varying slot and object counts, resulting performance gains on the task of unsupervised imago segmentation. The newly proposed normalizations represent minimal and easy to implement modifications of the usual Slot Attention module, changing the value aggregation mechanism from weighted mean operation to scaled weighted sum operation.
1 Introduction
Object-wise scene decompositions are ubiquitous in computer vision, robotics, and related disciplines such reinforcement. learning, since the state and actions in the environment naturally represented in relation to objects. Over recent years, unsupervised learning of object-centric representations from unlabelled images
and video has attracted significant interest in the machine learning community (Greff et al. 2017 Engelcke
et al. 2020 Locatello et al., 2020 The Slot Attention architecture (Locatello et al. 2020 decomposes a two-dimensional RGB input image object-wisc using an attention mechanism which updates slots, holding information about objects, in a recurrent manner. (Locatello al. 2020),, it was used for set prediction and image segmentation tasks on relatively simple renderings of 2D or 3D scenes, such CLEVR (Johnson et al. 2017) Later in (Seitzer et al.) 2023 Slot Attention has also been successfully applied to the task of unsupervised segmentation of more realistic images (MOVi, (Grefl et al. 2022 Delecting and tracking objects in videos using Slot Attention is described in (Kipf et al., 2022 Elsayed et al. 2022). Despite its empirical success, theoretical explanation of its inner workings and the inductive biases which lead to
Work done during PhD studies MPI-IS. Jan Achterhold is now with Robert Bosch GmbH Corporate Research, and the Rosch Center for Artificial Intelligence (BCAT), Renningen, Germany.

---

Published in Transactions on Machine Learning Research (09/2024)
the emergence of object-wisst decompositions in Slot Attention is still under active research Chang al. 2022b a...
In this paper we investigate design choices on the normalization of aggregated attention values in Slot Attention. We find that the normalization proposed in (Locatello et al. 2020 leads to suboptimal foreground segmentation performance during inference with higher number of objects or slots than used for training the model. We also investigate two alternative normalization approaches, give theoretical insights on their behavior, and assess their performance in relation to the original Slot Attention bascline. We demonstrate that these different approaches for normalizing the aggregated values can have significant impact the generalization of Slot Attention to a varying number of slots and objects during inference.
2
Background
2.1 Slot Attention
The Slot Attention module (Locatello et al. 2020) is given a set of input tokens H, E Pisptt , E NI and iteratively refines a set of K slots 0g E Date (l. In an object-wise scene decomposition scenario, slots correspond latent variables holding information on objects, while the input tokens are localized image features, e-g-, computed by convolutional neural network. Slots bind to input tokens via dot-product attention mechanism (Luong et al. 2015 Learned linear maps & and q extract D-dimensional keys and queries from the larer-nornallized (Ba el al. 2016) input lokens Hn == LayerNorm(@.)) and layer- normalized slots 0₂ == LayerNorm( (0x), respectively. In our case, we always have Dmpptt D and the layer normalization modules that produce o, and 0₂ do not share parameters.
For the attention mechanism, an unnormalized N x K matrix M of dot prod- uets is formed from the keys kn = k(xx) and queries qk = q(Ox). On each row of M, Softmax operator is then applied, yielding
r (7m,k)
[]
1 (((x)) ((())) = k qk MA = T
exp Ma,k
(1)
Yn,k Σ exp Mr,k'
(2)
T
With this, each row Yn may be interpreted as the probability of an imput token n be assigned a particular slot k. The constant r corresponds to a lemperature parameter which chosen be vD.
A linear map v Prppo R extracts values from the input tokens and the matrix I is used to accumulate values into unnormalized slot-wise update codes:
N
U = Σ x(,,)
(3)
==1
With the motivation to improve the stability of the attention mechanism, Slot Attention performs a normal- ization on the update codes. Namely, the sum in 3 is scaled in such a way that it becomes a weighted mean of the values v(æ,), Le.:
Uk
(4)
En=
Yook
This normalization scheme termed weighted mean. Locatello et al. (2020) discuss two ablations of this normalization. The weighted sum scheme normalizes the update code by multiplication with constant, i.e. u, := - ug. In the ablation study in (Locatello et al. (0200), the value chosen for C' is not discussed, and it must be assumed that C' = 1 was chosen. The second ablation of (Locatello et al.) 2020 is termed layer normalization and uses a layer normalization module that is shared across slots for normalization. Concretely, the normalized update code is computed as us = LayerNorm We refer to Appendix for an exact. definition layer normalization.
For each slot k. the aggregated value uy is used to update the latent representation 0% via a gated recurrent unit (GRU) (Cho et al. 2014) and residual multilayer perceptron with hnew update ux)-
2

---

Published in Transactions on Machine Learning Research (09/2024)
2.2 Von Mises-Fisher Distributions
Von Dises-Fiisher (vMF) distributions Fisher 1953) are probability distributions on the unit (d - 1)-sphere
in Rd. Typically, they are parametrized by mean direction 0 E Bd with ||0||2 = 1 and concentration parameter T > 0. They are defined by the following density w.r.t. the usual surface measure on the (d - 1)- sphere f(x 0, = (d,r))
20 where Z(d, r) is a normalization constant that is independent of 0. If (0,,..., 0x) and (71:-.., Tk) are parameters of vMF distributions and (T1, Fx) is contained the probability simplex, a vMF mixture model can be defined as usual via the density g(x))= == Σ Jx Tk).
1 exp [
3 Slot Attention and von Mises-Fisher Mixture Model Parameter Estimation
2023 compare Slot Attention to expectation maximization (Dempster et al. 1977 Bishop 2006) (EM) in Gaussian mixture models, i.e. to soft k-means clustering. We, however, connect it with expectation maximization in a mixture model of von FliserFiishrr (vMF) distributions (Banerjee et al. 2003), since Slot Attention uses a bilinear form on slots and inputs as a scoring function instcad of the negative Euclidean distance. In this scction, we make the parallel between Slot Attention and EM explicit. by performing EM parameter estimation in vMF mixture model and relating each step to the corresponding step in Slot Attention. We will then view the weighted mean, layer norm, and weighted sum normalization variants in the context of this analogy and compare them.
Many works (Locatello et al. 2020; Chang et al.
2022b
Kirilenko et al.
3.1 Relating Slot Attention to EM
We consider case in which N points x, are given on the unit (d- 1)-sphere in Rd We estimate the mean directions o,,... Ox of K vMF components, along with the mixture coefficients Tiy-,., TK. We assume that the vMF distributions have fixed concentration, i..,, = 1. We interpret the parameters 0₂ to relate to slots in Slot Attention and the points Fn to relate to the perceptual input features of the module. The concentration parameter T can be understood as an analogue to the temperature vD in Slot Attention.
E-Step In the expectation step, soft assignments of datapoints to clusters (slots) are computed via the likelihood functions of the vMF components:
Th exp J. 0z)
Tn E Th' exp(xr 0x.)
(5)
The resulting matrix I E [0, x| corresponds to the attention matrix in Slot Attention. Equation ( closely resembles the computation of the attention matrix in Slot Attention with some differences: While the inputs Hn in Slot Attention do not necessarily lie on the unit sphere, we do remind the reader that they are layer-normalized and therefore lie on ellipsoids. Similarly, the slots are layer-normalized before the attention step. In contrast to equation 5 Slot Attention uses key and query maps instead of directly forming a dot. product between F,, and Og. L.e., the dot products are formed between kn and qk.
While the Slot Attention architecture does not explicitly model the mixture parameters Tk, it may encode some weighting in the layer=nornallized slots. Indeed, we show in Appendix B that the keys kn are contained in some (D di-dimensional affine subspace A C Rd which may be written uniquely as A = a + V where V is a (D- - 1)-dimensional lincar space and a E V- is perpendicular to V. If pv : . + V is the orthogonal projection onto V and Pa : RDD > (a) is the orthogonal projection outo the span of a, we may decompose any x E R orthogonally as æ = pv + pa(xr) (x). For any key vector kn = k((,,)) E A we therefore have kn =a + +V The attention value Tn. in Slot Attention may now be writen as:
Ink = exp(k, qk)
exp(a T Pa (qx)) explpv (k,) T py (qx))
(6)
Ex exp(k )x))
En exp(a Pa Pa( exp(px((x)) pp pv (qx:))
Hence, the term exp(a" T Pa (qx)) may be interpreted as an analogue of Ths which assigns a weight to the th slot but is independent of the input at index n.
3

---

Published in Transactions on Machine Learning Research (09/2024)
M-Step In the maximization step, cluster (slot) parameters are updated using the soft signments r. In EM, the new mean directions and mixing coefficients are computed via:
Σ n= &nn Σn= In
Σ 7a.k
Oken
T
(8)
N
In our analogy to Slot Attention, this M-step would relate to the slot-update involving the aggregated val- ues ug. Hence, it may be of interest in this comparison to investigate whether the values us hold information about the right-hand sides equations and
3.2 Comparing Normalizations in EM Analogy
In the following paragraphs, we discuss how the update normalizations discussed previously compare in the context of our EM analogy and, in particular, whether the normalized update codes uy hold sufficient information to recover the quantities from equations and
Weighted Mean In the weighted mean case, the aggregated values uR can hold sufficient information to extract the quantities in 7 and 8 if D Dinput holds. Assuming that the value map is the identily, the uz/l/uull2- However, it is not clear how тт could be computed
right-hand side of ( may be computed from Indeed, we show in Proposition D that there can be no general formula as for the weighted sum case that generalizes without exception across slot-counts. We provide a proof in Appendix C by constructing some explicit slot settings which demonstrate that a hypothetical function f can not map every update code to corresponding unique scalar.
Proposition 1. Consider Slot Attention with weighted mean normalization and any fized model porameters and faxed impul data #1,..., &N with N 1. Then, there exists no function f IR + R such that il holds
Em In.k N
f(ux))
for arbitrary K 1, arbitrary slots 0,...., 0g and resulting normalized update codes u1 Ug.
Layer Normalization While, at least in some cases, it is possible to recover the quantity in 7 from update codes in the layo-nnommaliztioon variant, these update codes still do not contain sufficient information to infer the quantity in (8). Indeed, the reader may verify that the same argument presented in the proof of Proposition ! also holds for the layer norm variant.
Weighted Sum In the weighted sum case, we may, as for the weighted mean normalization, obtain the right-hand side of equation (7) via u the value map is the identity, In contrast to the previously discussed normalizations, we may also recover information on the column sums Σ Yn.k, which appear in equation (5) We make this rigorous in Proposition 2 and provide a proof in Appendix D where we exploit the fact that the values v, lie in a lower-dimensional subspace.
Proposition 2. Consider Slot Attention with weighted sum normalization and fixed model parameters. Let the namber of input tokens N be fized. Assume that Dinput = holds. For almost all (w.r.t. Lebesgue measure) paramelers of the inpul'' 's layernorm module and the value map v, there exists map f TR. - (which may depend on these parameters) such that
f(ux) Em Tn.k
vISSS
(10)
N
holds for any K 2 1, any slots 01,...,0x, any imput data ã ..., UN; and the resulting attention matria and update codes ui ... UK.
Since the assumptions of Proposition only exclude parameter subset of zero volume, its conclusion likely holds in practice during training.
4

---

Published in Transactions on Machine Learning Research (09/2024)
Hence, the weighted sum variant may also be seen a generalization of the weighted mean normalization: the update codes from weighted sum normalization hold sufficient. information such that weighted-mean- normalized update codes can be recovered from them (e.g. by the update network if it has sufficient capacity). The reverse is not true, as demonstrated in Proposition
4 Methods of Normalization
4.1 Weighted Sum Normalization with Fixed Scaling
As detailed in the above discussion, in contrast to the weighted mean normalization, the weighted sum normalization may preserve information on the fraction oę of input tokens assigned to the slot in the slot update code uy. While (Locatelo et al. 2020) report worse performance of the weighted sum normalization compared to the weighted mean normalization, the value chosen for C' is not further discussed. As detailed in our experiments, we observe that the weighted sum normalization can outperform the weighted mean normalization for C = N, where N the number of input tokens. For image inputs, the number of tokens is relatively large, e-g- N 1282 = 16, 384 for feature maps of CLEVR renderings. We aim to avoid unreasonably large values in the update code ug, which may lead to numerical instabilities, such vanishing gradients. With C = N, it holds that ug is bounded with |us.al I max, |a Vd {l,..., which is independent of N. Indeed, we have the following chain of inequalities:
1 N
uugal
Mn
m /Unaal
S
Σ
Mnn max lnnal S max Jun,al
(11)
n=1
1=
Here, we first the triangle inequality, followed by the crude estimate |vn,al S max, |nn,al for all Finally, we used the fact that Σn=1 Yn.k $ NN holds, since I row-stochastic with N rows. While this provides an argument for our choice C which is a suitable heuristics across tasks, we hypothesize that task-specific tuning of this hyperparameter may be beneficial.
4.2 Weighted Sum Normalization with Batch Scaling
Instead of heuristically choosing scaling parameter C' in the weighted sum normalization as above, we also investigate an approach in which the scaling factor is learned via a form of batch normalization (Toffe & Szegedy 2015) during training. Concretely, we measure the magnitude of unnormalized update vectors in the first Slot Attention iteration of each forwand pass by computing their batch statisties. These batch statistics are used during the subsequent iterations of the forward pass to scale the update codes. While other works using batch normalization in recurrent networks do not share statistics across time (Cooijmans et al. 2017 Laurent et al.) 2016), we find that our approach greatly simplifies varying the number of iterations during inference. In contrast to typical implementations of batch normalization, we propose to reduce all axes during the computation of the statistics (i.e., the batch axis, the slot axis, and the layer axis). Reducing the slot axis necessary to presorve slot-permutation equivariance, which is a desirable property in object-centric learning (Locatello el al.] 2020). Reducing the layer axis leads to scalar batch statisties, yielding a method that more closely aligns with the normalization approaches we have discussed so far.
Assuming that the tensor Ú E < holds the unnormalized update codes of the first SA iteration computed for miti- of size L, we define the batch statistics as:
Dato
Dator
LKD ΣΣΣ
1
((0)
LKD ΣΣ>
1
(0:00)
m)²
(12)
i,k.
Note that both statistics are scller-valued. As proposed by loffe & Szegedy 20155) we also learn two parameters a, 3 E R and normalize the tensor of update codes in itcration j via:
U = ū6 m
V8 VC +
(13)
5

---

Published in Transactions on Machine Learning Research (09/2024)
where E >> 0 is small constant. We stress that the values and are computed for each mini-batch in the first Slot Attention iteration and are therefore independent of j. Moreover, gradients flow through m and v. We cache an exponential moving average of the batch statistics during training and use it during inference. Hence, during inference, the normalization in equation (3) is simply an affine transformation with fixed weights; thereby closely resembling the weighted sum normalization presented in the previous subsection. While the weighted sum normalization is linear and not affine, we note that this distinction does not impact the capacity of the model. Indeed, the vectors ug are also affinely transformed within the update network, therefore making any previous affine or linear transformation redundant from capacity perspective. Notwithstanding, the normalizations presented here will significantly alter the trainig trajectory of the models. Batch normalization, in particular, has previously been shown to remedy the problem of saturating activalions and vanishing gradients (Tofle & Szegedy 2015 Pascanu et al., 2013), which may arise from improper normalization Glorot & Bengio 2010). We provide pseudocode for the two proposed normalization variants in Appendix
5 Experiments
We investigate the proposed normalizations on unsupervised object discovery lasks. To this end, we train autoencoders on the CLEVR (Johnson et al. 2017) and MOVi-C (Greff et al. 2022) datasets, utilizing autoencoder architectures that have been described in (Locatello et al. 2020) and (Seitzer et al.) 2023), ,
respectively. Additional results for a property prediction task can be found in Appendix !1 We provide visualizations of scene segmentations in Appendix H We will give brief overview on our experimental up
in the following and refer to the supplementary material for more detaile]]
Model Variants We refer to the standard normalization (weighted mean) the baseline and to the LayerNorm- based ablation from (Locatello et al.) (0200] as the layer normalization. We term the method detailed in Sec. 4.1 the weighted sum normalization, and the method from Sec. 4.2 the batch normalization. In some experiments, we will train models on filtered training sets (CLEVR6 and MOVi-C6) containing only a limited number of objects. For clarity, we annotate each model variant with a tuple (O, K), where O denotes the maximum number of objects seen in the training set and K denoles the number of slot latents used during training.
CLEVR Dataset We use an extended version of the CLEVR dataset that is provided in the Multi-object Datasets repository (Kabra et al. 2019). It consists of 100,000 2D) renderings of 3D scenes depicting up to 10 objects whose shapes are geometric primitives. Each scene is annotated with ground truth segmentation, which we use for evaluation. Following Locatello et al.] (020)) we use 70,000 images for training and further adopt the approach of (Locatello et al. 2020 Greff et al. 2019 Burgess et al. 2019 by cropping the images to highlight objects in the center. In contrast to (Locatello ct al. 2020), we also augment the data during training via random horizontal flips. As in (Locatello et al.] 2020), we also consider a subset of the CLEVR dataset, only consisting of images containing at most 6 objects. We refer to this dataset as CLEVR6 and will denote the original dataset by CLEVR10.
MOVi-C Dataset Compared to CLEVR, MOVI-C represents a significant step-up in perceptual complexity. It contains 10,986 vidco sequences, cach consisting of 24 frames. We use 250 of these video sequences for validation and hold out 999 sequences for testing. Each clip shows 3 to 10 highly textured 3D-scannedl objects from the Google Scanned Objects repository (Downs et al. 2022) flying into view and colliding. In our experiments, we only consider single RGB frames from the dataset and discard any temporal relation between them. Once again, we introduce a filtered dataset, which we denote by MOVi-C6 and which consists of frames of clips that contain at most 6 objects. For sake of clarity, we refer to the original dataset as MOVi-C10.
MOVi-D Dataset The MOVi-D dataset consists of scenes that are visually similar to those from the MOVi- C dataset. However, the scenes contain up to 23 (10 to 20 static, 1 to 3 moving) objects, thereby presenting a greater challenge to object-centric method. Structurally, the datasct resembles MOVi-C, consisting of 11,000
Codde is available at

---

Published in Transactions on Machine Learning Research (09/2024)
video sequences, each made up of 24 frames. As before, we discard any temporal relationship between frames. In our experiments we will not train on MOVi-D, but instead investigate zero-shot transfer performance of models that were trained on MOVi-C.
CNN Autoencoder We adopt the convolutional neural network (CNN) based architocture proposed in Locatello el al., 2020) to train object-centric autoencoders on the CLEVR dataset. convolutional network transforms input images into feature maps, which are enriched by positional embeddings and spatially flattened. The resulting sets of tokens are processed by the Slot Attention module to obtain object-centric latent representations. A spatial broadcast decoder (Walters et al. 2019) decodes each slot latent separately into an image and an unnormalized alpha mask. The alpha masks are normalized across the slot axis via a softmax operation and subsequently used to linearly combine the reconstructed images, thereby producing a reconstruction of the input. As in (Locatell et al.) 2020), we perform 3 Slot Attention iterations during training, and iterations during evaluation. We further follow (Locatello et al. 2020 in that we obtain segmentations from trained autoencoders by assigning each pixel to the slot for which the corresponding entry in the alpha mask attains a maximum value.
Dinosaur Autoencoder To obtain object-centric behavior on the subslantially more complex MOVi-C dataset, we adopt the approach of Dinosaur (Seitzer et al. 2023). Instead of directly operating on RGB frames, the autoencoders are trained on image features that are extracted via a pre-trained and fixed vision transformer (ViT) (Caron et al. 2021). In spirit, the autoencoder resembles the previously discussed architecture: A small encoder, in the form of a two-layer perceptron, processes the ViT features, which are then transferred into a latent representation by the Slot Attention module. Each latent is decoded individually into reconstruction of the image features and unnormalized alpha mask. An overall reconstruction of the ViT features is formed by linearly combining the individual reconstructions via the normalized alpha masks. We use the MLP--basdd decoder that is described in (Seitzer et al.) 2023). Crucially, the autoencoder exclusively operates on ViT features, neither receiving RGB frames as input, nor producing them as out put. Hence, the autoencoder's reconstruction loss is also measured on ViT features, providing a training signal that is more akin to perceptual similarity than similarity in RGB space. As in the experiments on the CLEVR dataset, we extract segmentations from the alpha masks. Since the alpha masks (and, correspondingly, the ViT feature maps) are of a lower resolution than the RGB frames of the MOVi-C dataset, we adopt the approach of [Seitzer et al. 2023 and bi-linearly upscale the alpha masks before computing segmentations.
Evaluation Following related work (Locatello et al. 2020 Scitzer et al. 2023 Kipf ct al. 2022 Greff et al,, 2019), we primarily judge model performance by the quality of foreground Degmentations as measured by the foreground adjusted Rand index (Rand 1971 Hubert & Arabie, 1985) (F-ARI). Additionally, we provide figures regarding the overall sogmentation performance when including the background (ARI). All models are evaluated on 1,280 scenes from the respective test sets. As reconstruction losses are rarely discussed in related work, we defer the investigation of this metric to Appendix A
5.1 Object Discovery on CLEVR
In this subsection, we investigate our proposed normalization approaches on an object discovery task on the CLEVR dataset. In a first set of experiments, we follow the exact training procedure detailed in (Locatello ct al. 2020) and illustrate how the different normalization methods behave as the number of slot latents K is changed during inference. The effect of choosing large numbers of slot latents during training is studied in a second set of experiments. Throughout these experiments, we additionally scrutinize the impact of the object count on model performance.
Training With 7 Slots In this first set of experiments, we follow (ocatello et al. 2020 as closely as possible and train the previously described CaN-based architecture on the CLEVR6 dataset with 7 slots. We compare the bascline and layer normalizations proposed in (Locatello et al. 2020) to the methods discussod in Section 4 For each variant, we perform 5 training runs with different seeds. The trained models are evaluated on the CLEVR6 and CLEVR10 test sets.
7

---

Published in Transactions on Machine Learning Research (09/2024)
1.00
1.00
1.00
0.35
0.56
0.95
F-AR1
0.96
0.50
Seaee 15.7)
0.90
Dustra B-71 Law POTIAL
-EW NCTTLL
1.2. Scre
0.85
0.85
0.50
13
15
17
18
21
11
13
15
17
19
21
9 10
Slots
Slots
Objects
(a) F-ART (f) CLEVR6, number of slots varies during inference.
(b) F-ART (1) CLEVR1O, num- ber of slots varies during inference.
(c) F-ARI (f) on CLEVR, number of objects varies during inference. Evaluated with 11 slots.
Lijer Norn 6o.77 Scanhhorm
Layer Narn
0.5
Leyer hormis
Maig (6.7)
16.77
Wilgheed Sanit
ARI
0.2
0.2
0.0
# Slota
0
Slots
Objects
(d) ARI (1) on CLEVR6
(e) ART (1) on CLEVR10
(f) ARI (1) on CLEVR
Figure 1: Dependence of performance on slot and object count. Models are trained on CLEVR6 with 7 slots.
Note the non-zero y-intercept.
Input
Baseline
Layer Norm
Weighted Sum
Batch Norm
Figure 2: Qualitative results on a CLEVRIO images, showing reconstruction ions and (soft.) segmentations. The models are trained on CLEVR6 with slots and evaluated with slots.
The baseline and layer norm variants lead to object-centric behavior for all seeds. For the weighted sum normalization and the batch norm variant, however, we encounter two runs each which the autoencoders decompose the input spatially instead of object-wisc. Following (Locatello al. 2020), we omit these runs our analysis.
In Subfigures la and lb illustrate how the foreground segmentation performance changes vary
the number of slots during evaluation. We note that in the bascline and layer norm variants, segmentation performance deteriorates when they are presented with more than nine slots, while our proposed normalizations appear to generalize well to these changes. In particular, we observe that both of our proposed normalizations outperform the bascline when the autocncoders are evaluated with 11 slots, is done (Locatello al. 2020). We show qualitative results of the different model variants a high slot count in Figure and refer to Appendix H for more visualizations.
8

---

Published in Transactions on Machine Learning Research (09/2024)
1.01
F-ARI
F-ARI
F.ARI
01.8
Base 34 10
Norry
Ear
0.7
0.7
Decc Nerm 10)
0.7
17
19
11
13
15
17
19
21
Slots
Slots
Objects
(a) F-ARI (f) CLEVR6, number of slots varies during inference.
(b) F-ARI (t) on CLEVRIO, mum- ber of slots varies during inference.
(c) F-ARI (t) CLEVR, object count varies during inference. Evaluated with 11 slots.
1.00
1.00
mage 11h-1
0188 Rwithteee.....
Sar KL10
Welphiee Sumi
0.6
ARI
0.4
0.26
0.25
0.00
0.0
13
15
17
19
21
3
5
10
Slots
# Slots
+Ojjacts
(d) ARI (t) on CLEVR6
(e) ARI (f) on CLEVRIO
(f) ARI (t) on CLEVR
Figure: 3: Dependence of segmentation performance slot and object count. Models trained on CLEVR6
with 11 slots.
We study how performance depends on the number of objects in Subfigure Ic Here, we evaluate each model variant with 11 slot latents on the CLEVR10 test set and plot average foreground segmentation performance dependent the object count. Overall, observe that our proposed normalizations outperform the baseline, independently of the object count. Also note that, across all model variants, segmentation performance trends downwards as the number of objects in the scene increases.
Subfigures ld illustrate the behavior of the overall segmentation performance when background pixels are taken into considcration. appears that our proposed methods outperform the other two variants w.r.t. this metric, although the variablity across runs is large.
Excess Slots During Training While we have so far only discussed experiments in which we increase the number slot latents during inference, we will now outline an experiment in which the Slot Attention module is also provided with excess slots during training: Concretely, we train the autoencoders on the CLEVR6 dataset with 11 slot latents. We annotate the resulting variants with the tuple (6, 11) to underline that they were trained with 11 slots scenes consisting of at most 6 objects. To limit computational expenses, we perform only three runs per model variant. We observe object-centric behavior in all runs for the baseline, the layer norm variant, and the weighted sum variant. For the batch normalization, we encounter one run in which the scenes are deconstructed spatially in vertical stripes. As before, we exclude this run from our analysis and are therefore left with only two runs for this variant. Considering the breadth of our other experiments and the small variability observed in this experiment, we deem this loss of information acceptable.
comparably than before w.r.t. foreground again note that the performance of the bascline and layer norm variants starts to suffer we add additional slot latents during inference. In contrast, the performance of the two proposed variants remains more stable. In general, the foreground segmentation performance is lower than during training with few slots (Subfigures Ia Ich. All models trained with our proposed methods learn segment the background into single slot, leading high overall segmentation performance (Subfigures 3d 3f. One model using the baseline normalization exhibits this behavior, while
In this setting, the studied variants seem to perform segmentation performance (Subfigures 3a 3c). Notwithstanding,
none of the models using layer normalization do

---

Published in Transactions on Machine Learning Research (09/2024)
0.50
0.50
0.90
0.70
0.70
0.75
[-AR]
0.70
F-ARI
0.70
0,70
0.65
0.05
Sestiw (1C. TOI
Caate DE
Deae PACIA
PORTI SL, IEI
0.00
1.3. Nove 2TC.
0.55
0.55
0.53
13
15
17
18
21
13
15
17
19
21
9 10
SSots
Slots
Objects
(a) F-ART (f) on MOVi-C6, number of slots varies during inference.
(b) F-ART (f) MOVi-C10, num- ber of slots varies during inference.
(c) F-ART (f) MOVi-C, object count varies during inference. Eval- uated with 11 slots.
ARV
API
Sarr TU
JTC,
Wiighted Cumc
Batch
13
# Slots
Slots
# Objects
(d) ARI (1) on MOVi-C6
(e) ARI (f) on MOVi-C10
(1) ART (t) on MOVi-C
Figure 4: Dependence of segmentation performance on slot and object count. Modols are trained on MOVi-C10
with 11 slots.
5.2 Object Discovery on MOVi-C
Too further support the validity of our proposed normalizations, we run additional experiments, using the Dinosaur (Seitzer et al.) 2023) framework. As previously discussed, we train the MLP-based architecture described in (Seitzer et al. 2023) on the MOVi-C dataset. While it still is a synthetic dataset, this represents a signíficant step-up complexity compared to CLEVR, approaching the complexity of real-world scenes.
Training on MOVI-C10 In our first set of experiments, we closely follow the setup detailed in (Seiter et al. 2023) and train the autoencoders the full MOVi-C10 dataset, using 11 slots. As in the previous subsection, we investigate how the foreground segmentation performance develops as vary the nunber of slot latents during inference. For each model variant, we perform 5 training runs with different seeds. For sake of consistency with the other experiments, we evaluate the trained models both on the MOVi-C10 test. set and on the filtered MÓVi-C6 dataset.
In Subfigures La and 4b we observe that our proposed normalizations generally lead to improved foreground segmentation performance compared to the baseline and layer normalization across all slot counts. Tn particular, both proposed normalizations outperform the bascline and layer normalization variant with 11 slots on the MOVi-C10 datasel, can be observed in Subfigures 4a and 4b As in our previous experiments, we note that foreground segmentation performance starts to suffer as the baseline variant is provided with excess slots during inference. While our proposed methods exhibit. a similar behavior in these experiments, we note that the deterioration progresses at a slower rate. Additionally, we observe in Subfigure 4c that performance improved across smaller object counts.
The behavior of overall seginnttatio performance is shown in Subfigures 4d 4f observations, we note that performance suffers for all variants when excess slots are present during inference. While baseline, layer normalization and batch normalization yield comparable results w.r.t. this metric, the weighted sum variant performs noticably better.
In line with our previous
Training on MOVi-C6 While the authors of (Seitzer et al. 2023) trained autoencoders exclusively on the MOVi-C10 datasct, we will also investigate an approach that resembles the one described (Locatello ct al. p020), and which we adopted in Subsoction 5.1 Namely, we train models on the filtered MOVi-C6 dataset with 7 slots and subsequently evaluate them both MOVi-C6 and MOVi-C10. This approach may
10

---

Published in Transactions on Machine Learning Research (09/2024)
0.85
0.55
0.55
0.80
0.90
0.50
0.75
0.75
0.76
0.65
0.65
0.65
W
LOW Nore (CT)
Kpm (0-7)
0.05
0.50
0.55
13
15
17
18
21
9
11
13
15
17
19
21
10
(a) F-ARI (f) on MOVi-C6, number of slots varies during inference.
(b) F-ARI (1) on MOVi-C10, num- ber of slots varies during inference.
(c) F-ARI (t) on MOVi-C, ob- ject count varies during evaluation. Scenes are evaluated with 11 slots.
0..3
0.5
0.5
0.1
0.4
0.5
ARI
0.3
0.3
0.1
Batch
0.0
0.0
0.0
11
13
15
17
19
21
11
13
15
17
21
6
7
10
Objects
(d) ARI (1) on MOVi-C6
(e) ARI (f) on MOVi-C10
(I) ARI (t) on MOVi-C
Figure 5: Dependence of segmentation performance on slot and object count. Models are trained on MOVi-C6
with 7 slots.
be particularly interesting to practitioners, reducing the number of slot latents during training serve to greatly reduce computational effort. To limit computational expenses, we again only perform three runs per model variant.
We illustrate in Subfigures 5a the bohavior the foreground segmentation performance. As in our previous experiments, we observe that both of our proposed normalizations out the baseline when evaluated on the MOVi-C10 test with 11 slots. Interestingly, it can be noted that performance also improves over the models trained on the MOVi-C10 dataset with 11 slots (Figure 4 Again, we point out that excess slot latents during inference lead to substantial deterioration of foreground segmentation performance in the baseline and layer norm models.
In Subfigures 5d 51 we plot the behavior of overall segmentation quality. Compared to the previous experiment, varying slot count has a lesser impact on overall segmentation performance for all methods. While the differences seem unsubstantial, the bascline appears to perform best w.r.t. this metric.
Excess Slots During Training In line with the experiments on the CLEVR dataset, we turn to set of experiments that investigates the impact of excess slots during training. Similar to the corresponding setup in Subsection 5. train the models on the filtered MOVi-C6 dataset, but provide them with 11 slots during training. We generally observe that both the weighted sum normalization and the batch normalization lead to improved foreground segmentations compared to the baseline and layer normalization, especially at higher slot count, as can be concluded from Subfigures Oa and Gb. Subfigure bc additionally demonstrates once again that our proposed normalizations appear to perform
least as well as the baseline across all object counts. In Subfigures 6f we find that increased slot count during inference harms overall segmentation quality. The weighted sum variant performs best, although the variablity in ARI appcars large.
Evaluation on MOVi-D We now investigate how the previously described models (which were trained MOVi-C) transfer to the MOVi-D dataset. We recall that seenes of the MOVi-D dataset may contain up to 23 objects. Hence, the MOVi-D dataset allows us lo study how the trained models behave when slot- and object-count are increased significantly during inference. We evaluate the models with 24 slots. In Table [ we show the MOVi-D zero-shot performance of all model variants we trained on MOVi-C. The batch norm variant performs best w.r.t. our main metric, the F-ARI. This holds true both when comparing all 12 variants to each other, and when considering the subsets of models obtained by only considering variants with any fixed
11

---

Published in Transactions on Machine Learning Research (09/2024)
0.50
0.50
0.90
0.70
0.70
0.75
[-AR]
0.70
F-ARI
0.70
0,70
0.65
Cante DO
Deae 2e RCC
JEW NOTTIL
PORTIL
0.55
0.55
0.53
13
15
17
18
21
13
15
17
19
21
9 10
Slots
Slots
Objects
(a) F-ART (f) on MOVi-C6, number of slots varies during inference.
(b) F-ART (1) MOVi-C10, num- ber of slots varies during inference.
(c) F-ART (1) MOVi-C set, ob- ject count varies during inferences. Evaluated with 11 slots.
APH
Eayehooo 11
Star 91
Siighadd
Batch
13
# Slots
Slltts
@ Objects
(d) ARI (1) on MOVi-C6
(e) ARI (f) on MOVi-C10
(1) ART (t) on MOVi-C
Figure 6: Dependence of segmentation performance on slot and object count. Models are trained on MOVi-66
with 11 slots.
annotation (O,K). Compared to their batch normalization counterparts, the weighted sum models perform similarly w.r.t. foreground ARI, performing only slightly worse. Both the weighted sum normalization and the batch normalization produce markedly better foreground segmentations than the baseline and layer norm variants. With respect overall segmentation quality (ARI), none of the normalization variants consistently outperform the others, which is in line with our observations on MOVi-C.
For any fixed normalization method, the (6, 7) variant performs better w.r.t. F-ARI than the (10, 11) and (6, 11) variants. This illustrates that, firstly, training on filtered training sets with few objects can improve performance during inference on scenes with many objects; secondly, avoiding excess slots during training appears to be important, which is an observation that has been made in (Locatello et al. 2020 and which we also note when comparing Figures [ and 3 This effect underlines the importance of strong slot-count generalization capabilities. We posit that training at low object- and slot-counts reduces the number of 'unoccupied' slots during training, thereby tightening the representational blot-bottleneck, which has been postulated to encourage object-centrieness (Locatello et al. 2020 Stange et al. 2023
6 Related Work
Tn this work, we follow a longer tradition of using autoencoders to obtain semantic scene decompositions Greff
lt al.) 2016; 2017 2019 Crawford & Pineau 2019 Burgess et al. 2019 Lin et al.] 2020 Engelcke et al.)
2020
Locatello et al. 2020). An early work in this area is Neural-EM EMrral-EM& (Greff et al. 2017).
which performs expectation
maximization at the image level. Tt. is proceeded by TODINE (Greff et al. 2019), which learn variational autocncoders. Slot Attention (Locatello et al. 2020 has already been used many derivative works to scale object-centric learning increasingly complex datasets. In particular, motion
2019 and MONet (Burgess el al.
cues (Kipf et al. 2022 Elsayed et al.) 2022 Wu et al. 2023). high-level image features (Seitzer et al. 2023 and powerful decoder models (Singh et al. 2022a were found to be useful for obtaining desired behavior
on real-world datasets.
Several previous works have discussed generalization capabilities of object-centric representations (Dittadi ct al. 2022 Seitzer et al. (023). That the number of slot latents has influence on model performance has been noted by Seitzer et al. (2023) where the authors illustrate that varying the number of slots has a significant impact on segmentation quality, determining whether objects are split into constituent, parts discovered in
12

---

Published in Transactions on Machine Learning Research (09/2024)
Variant
F-ART (t)
( loss (4)
ART (1)
Bascline (10; 11)
0.647 =0.015
2.143
4000004
0.172
10.011
Layer Norm (10, 11)
0.660
2.146
0.171
Weighted Sum (10, 11)
0.722 -0.005
2.223
0.204
400004
Batch Norm (10, 11)
0.725 =0.006
2.149
#0.004
0.182
#0.011
Baseline (6, 11)
0.640 =0.009
2.220
#0.001
0.176
#00000
Layer Norm (6, 11)
0.639 =0.011
2.220
#0.004
0.176
#0.006
Weighted Sum (6, 11)
0.709
2.309
0.205
Batch Norm (6, 11)
0.711
2.219
0.175
Baseline (6, 7)
0.741 =0.005
2.370
20,000
0.253
20.018
Layer Norm (6, 7)
0.742 -0.019
2.365
40.007
0.253
+00000
Weighted Sum (6, 7)
0.797
=0.023
2,475
#0.038
0.242
#0.156
Batch Norm (6, 7)
0.809 20.005
2.374
#0.002
0.297
Table 1: Zero-shot performance on MOVi-D of models trained on MOVi-C. Evaluated with 24 slots, We show the median + maximum deviation across multiple runs. For each metric, we underline the most advantagous variant in each section and mark the most advantagous variant across all sections in bold.
their entirety. To date, only few works investigate the role of attention normalization in the performance of Slot Attention. The first ablation evaluated for Slot Attention in the original paper (Locatello et al., 2020 is almost identical to the approach we discuss in Subsection 4.1 differing from our method crucially in that the weighted sums are not scaled by a constant, which appears result in poor training behavior. In second ablation, the authors modify the first ablation by normalizing the update codes via layer normalization (Ba et al. 2016 Comparable performance is reported to the original method.
Normalization in Slot Attention has been investigated in (Zaagg et al. 2023). where the authors propose to use the Sinkhorn-Knopp iteration to normalize attention matrices. In contrast to our work, they do not investigate the impact on generalization capabilities and substantially increase the complexity of the SA module. More broadly, the use of the Sinkhorn-Knopp algorithm for normalization multi-head attention modules has been scrutinized before by Sander al. (2022).
7 Conclusion
Allowing models to dynamically find suitable levels of segmentation coarseness an important problem in unsupervised object-centric representation learning. In Slot Attention it has been found that excess slots oftentimes split objects into parts or distribute responsibility for individual pixels among several slots. Hence, finding reasonable number of slots has been crucial during training and inference. In this work, we discussed approaches for making the Slot Attention module more robust with respect to this choice. We studied normalizations of the slot-update vectors and analysed how they impact Slot Attention's ability to scale to different numbers of slots and objects during inference. On the theoretical side, we motivated this phenomenon via an analogy between Slot Attention and parameter estimation vMF mixture models, In experiments, we demonstrated that our proposed normalization schemes incrcase the generalization capability of Slot. Attention varying number of slots and objects during inference. With these insights, hope to contribute to increase performance of numerous existing and future applications of Slot Attention.
Acknowledgement
This work was supported by Max Planck Society and Cyber Valley. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Jan Achterhold.
13

---

Published in Transactions on Machine Learning Research (09/2024)
References
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/160s..../ 0bs/1607.06450, 2016. URL http://arxiv.org/ wwwp/b//////...o//////////////.e 064500
Arindam Banerjee, Inderjit S. Dhillon, Joydeep Ghosh, and Suvrit Sra. Generative model-based clustering of directional data. In Lisc Getoor, Ted E. Scnator, Pedro M. Domingos, and Christos Faloutsos (ods.),
Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 24 27, 2008, pp. 19-28. ACM, 2003. doi: 10.1145/956750.9560
URL https:///oi..... org/10 10. 1145/956750 956757
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew M. Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representation. CoRR, 2019. URL
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supervidd vision transformers. In International Conference on Computer Vision (ICCV), pp. 9630 9640. IEEE, 2021. doi: D0.1109/CCCV48922.202 200100091.. URL
attps://doi... mroi..org/1... 1ttg/10..109//C/44892. 202189922222.......2222222 00951 2021.00951
Michael Chang, Thomas L.. Griffiths, and Sergey Levine. Object representations as fixed points: Training iterative refinement algs with implicit differentiation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2022a. CRL httpps://pperrevier wwt.:/eeetteeete httee//orrmmnd---SPPOOO--
Michael Chang, Sergey Levine, and Thomas L. Griffiths. Object-centric learning as nested optimization. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022b. URL https://openreviiet
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of ncural machine translation: Encoder-docoder approaches. In Dekai Wu, Marine Carpuat, Xavier Carreras, and Eva Maria Vecchi (eds.), Workshop on Synlar, Semantics and Siructure in Statistical Translation (SSST@EMNLP), pp. 103 111. Association for Computational Linguistics, 2014. doi: 10.3115/v1/W1444012 URL httpps://aclanthoo..o. org/w14-4012/
Tim Cooijmans, Nicolas Ballas, César Laurent, Çaglar Gülçehre, and Aaron C. Courville. Recurrent batch normalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, Fronce, April 24-26, 2017, Conference Track Proceedings. OpenReview.nee neet, 2017. URL forum? Dorum?id=rVVdcHezz
net/
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In The Thirty-Third Third AAAT Conference on Artificial Intelligence, AAAT 2019, The Thirly-Firt Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pp. 3412-3420 3420. AAAT Press, 2019. doi: 10. (1609/AAAAVV33111 33013412.. URL https: org/ 10. 1609 aeai aaai 133101.33011411
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39(1):1 38, 1977. ISSN 00359246. URL http:/ //www/ jstor
Andrea Dittadi, Samuele S. Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 5221-5285, PMLR, 2022. URL httpp::/pprceeddiii mlr, wtt..prees/vi2. h2ml
dittadi22s html
14

---

Published in Transactions on Machine Learning Research (09/2024)
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B. McHugh, and Vincent. Vanhoucke. Google scanned objects: A high-quality dataset of 3D scanned household items. In International Conference on Robotics and Automation ) pp. 2553 2560. IEEE, 2022. doi: 9811998
URL https://doi. www.doi..orggg// wrg..o..o/....... Tt..:..1119///A4A.80100
Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world vidcos. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 28940- 28954. Curran Associates, Inc., 2022. URL httpps://proceeeinggg newriss..errpcccc paper http:////herrreeeeaoaaaoa//////r Confeeeeeeee
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: generalive scene inference and sampling with object-centric latent representations, In International Conference on Learning Representations (ICLR). OpenReview.net, 2020. URL
Ronald Aylmer Fisher. Dispersion sphere. Proceedings of the Royal Sociely of London. Series A. Mathematical and Physical Sciences, 217((130):295 305, 1953. doi: http://wwww..eeeooooooooo/ooaaaa prg//oi/abs//1...//// 1098//....19/pppppp 1953. 1064
URL https
//royalsocietypublliii
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and D. Mike Titterington (eds.), Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume 9 of JMLR Proceedings, pp. 249 256. JMLR.org, 2010. URL http://prroceedange..o gress/v9/glorotine htmml
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hotloo Hao, Harri Valpola, and Jürgen Schmidhu- ber. Tagger: Deep unsupervised perceptual grouping. In Daniel D. Lee, Masashi Sugiyama, Ulrike Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advonces in Newral Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4484-4492, 2016. URL httpps:/prroceedinng wewprediiiinnnnnnnisssree cc/paapr/2016//aate 2lee599922200660111963333066 Abstract. html
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximiza- tion. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Newral Information Processing Sys- tems (NewrIPS), pp. 6691-6701, 2017. URL (2ddee0000226622225540 www.eff-bbbtttttttttc.a
Klaus Greff, Raphaäl Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew M. Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pp. 2424-2433. PMLR, 2019. URL http:://proceedingg.. mlr. wttp:/sss........eseeese/ems wttps//////ggefff..aeoomm
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam H. Laradji, Hsuch-Ti Derck Liu, Henning Meyer, Yishu Miao, Derek Nowronzezahrai, A. Cengiz Oxtireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent. Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable datasct generator. In Conference on Computer Vision and Pallern Recognition (CVPR), pp. 3739 3751. IEEE, 2022. doi: 10. 1tt.:1110////20.0 2022. 00373
URL
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, 2(1):193-218, Dec 1985. ISSN 1432-1343. doi: 10. LL1007//BF01008075 http://pp///aa...r//////////////
15

---

Published in Transactions on Machine Learning Research (09/2024)
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, Prance, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 148-456. JMgR.org, 2015. URL http::/pproceedingee mlr wttp://nnnn........sss//ss3 ioffe15.hta .html.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Conference on Computer Viston and Pattern Recognition (CVPR), pp. 1988 1997. IEEE Computer Society, 2017. doi: 10.1109/ 101199//VR.2222 CVPR.2017.2155 URL https:://oo...o// http:///oo.orrg/////// 10. 1109/CVPR. 2017.
Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerehner. Multi-object datasets. htttps://github: http://twww..coo///eeeedd/nadeed wwwp///depm..mmmmmmmtooooooopppa oojeettttoottteeeeeeeeeeeeeeeett dateccdddtttts///// 2019. Accessed 2023-05-12.
Diederík P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Represenlations (ICLR), 2015. URL attp://arxiv org/ abs/ 1412. 1412.6980 6980
Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In International Conference on Learning Representations, 2022. URL https:://pperreveee..ee/
id=aD/uesX1GFF
Daniil Kirilenko, Alexcy Kovalev, and Aleksandr Panov. Object-centric learning with slot mixture models. http://eweeeeeeeeeeeeooooeiraaae AqX3oSb2yQ1
2023. URL https:///oeenrriieennet//eee net/forum?id
César Laurent, Gabriel Pereyra, Philemon Brakel, Ying Zhang, and Yoshua Bengio. Batch normalized recurrent neural networks. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2016, Shanghai, China, March 20-25, 2016, pp. 2667-2661. IEFE, 2016. doi: 10.1109/ 2CASSPP2016..472155.. URL https://doi.org::
1109/ICASSP 7472159
Zhixuan Lin, Yi-Fu Wi,, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decom- position. In 8th International Conference on Learning Representations, ICLR 2020, Addis Aboba, Ethiopia, April 26-30, 2020. ppenReview.net, 2020. URL
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Tn Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Mlriia-lorina Balcan, and Hsuan-Tiees Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 62, 2020, virtual, 2020. URL httttts::/ppoeeeeeeee www.ddiiinnnnnnnnneeeeeeeeeeeeee mewrips..c/ppppeee.eeeeeeeeeeeee / 2020/hash/85 /tt////1110880022000aaaaaa225200
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to atlention-based neural machine translation. In Lluis Mårquez, Chris Callison-Burch, Jian Su, Daniele Pighin, and Yuval Marton (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 1422-1421. The Association for Computational Tinguistics, 2015. doi: (L11653/v1/d15-111
URL hattts://///ot wtt.///0...//1////////...
Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. Tn Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pp. 1310 1318. JMLR.org, 2013. URL http://procceeding.. amwpaiiinnnmmm.m....mmmmmaee...m mmr http://ppcdoagg.g.oepeeeeeee/aap
html
William M. Rand. Objective critcria for the evaluation of clustering methods. Journal of the Americon Statistical Association, 66(336):846-850 1971. ISSN 01621459. URL http:://ww.. http://www.sttrrr/o jstor. org/stable/
2284239
16

---

Published in Transactions on Machine Learning Research (09/2024)
Michael E. Sander, Pierre Ablin, Mathicu Blondel, and Gabriel Peyré. Sinkformers: Transformers with doubly stochastic attention. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.), International Conference on Artificial Intelligence and Statistics, AISTATS 2022, 28-30 March 2022, Virtual Event, volume 151 of Proceedings of Machine Learning Reseorch, pp. 3513 3530. PMLR, 2022. URL https::/prooceedingg mlr. wttp:p//v1l//aaaeer22221 httml
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brex, and Francesco Locatello. Bridging the gap to real-world object-centric learning. In The Eleventh International Conference on Learning Represenlations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.nel, 2023. URL https //openreview http://wwwwe.neee/dddaa/aaaoeeee
aG
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-E learns to compose. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Evend, April 25-29, 2022. OpenReview.ner net, 2022a. URL
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, Novem- ber 28 December 9, 2022, 2022b. URL http://paperr: .ips. www.niiipsceeeeeee cc/paper www.ss/paper///22222 202pp//a///22/hhhaaaa #BB0111000EGd-BBccccc222233---1 Abb-442aaaa---sraacttt Coneertoooenoeeeeeeeeeeeeeeeeeee html
Andrew Stange, Robert Lo, Abishek Sridhar, and Kousik Rajesh. Exploring the role of the bottleneck in slot-based models through covariance regularization. CoRR, abs/2306.02577 2023.
Nicholas Watters, Loic Matthey, Christopher P. Burgess, and Alexander Lerchner. Spatial broadcast decoder:
A simple architecture for learning disentangled representations in VAEs. CoRR, abs/1901.07011 2019. URL http://wwwv..rr/aaa/1/1..000/
Ross Wightman. Pytorch image models. http:p///tttii..ocor///////////p Accessed: 2023-05-24.
2019.
Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. In The Eleventh International Conference on Learning Representations, TCLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:
//openreview. //opeeneviiio.ee///affaaaaaoeae. wwt.te/aaaf..1TTPo/iOOO..
Yan Zhang, David W. Zhang, Simon Lacoste-Julien, Gertjan Burghouts, and Cees G. M. Snoek. Unlocking slot attention by changing optimal transport costs. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett. (eds.), International Conference on Machine Learning, ICML 2023, 29-29 July 2023, Honolulu, Hawaši, USA, volume 202 of Proceedings of Machine Learning Research, pp. 41931 41951. PMLR, 2023. URL https://proceediii mlr press/ /
v202/ v202/zhang http///annnaaaaaaam html
17

---

Published in Transactions on Machine Learning Research (09/2024)
A Reconstruction Loss
In this subsection, we investigate how the e reconstruction loss is impacted by the choice of normalization. As in previous figures, we explore how this objective varies as object and slot-count is varied during inference. For each experiment provided in the main paper we provide a corresponding figure on the e loss.
CLEVR (6, 7) We first consider the experiment in which we trained a model with 7 slots on CLEVR6. We note Subfigures Ta and 7b that all normalization approaches suffer under excess slots during inference. The layer norm variant generally performs best in this context, while the baseline and weighted sum variants perform comparably to each other. In Figure Tc note that reconstruction quality decreases with increasing object count for all variants. It can be observed that this deterioration progresses fastest for the baseline, slowest with batch normalization, and at seemingly similar rate for the layer norm and weighted sum variants.
Recorstuckón Loss
Loye Nook
Recnsstuaton Loss
Reconciruction Loss
0.0008
oaer na Weighed Sarn 8.77
Weighed Sumis
Sarn
0.0075
0,003
0.6026
0.002
0.0000
13
Slots
Slots
# Objects
(a) Reconstruction lass
(4) on
(b)
Reconstruction
lass
(L)
(c) Reconstruction loss
(4)
on
CLEVR6
CLEVRI0
CLEVR
Figure Dependence of reconstruction quality on slot and object count. Models are trained on CLEVR6
with 7 slots. Note the non-zero y-intercept.
CLEVR (6, 11) Figure illustrates the behavior of reconstruction losses for models trained CLEVR6 with excess slots. We find that, analogous to our observations in Subsection 5.1 varying slot count during inference has lesser effect than in the previous experiment (compare Subfigures Sa and Sb to Subfigures 7a and 7b Generally, the weighted sum variant has the highest reconstruction loss, while the layer norm variant. has the lowest. The baseline and the batch norm variant perform comparably. As before, we observe in Subfigure Sc that reconstruction loss increases with increasing object count.
Danatine Ss 10
Reconstruction Loss
0.005
Rleonntruction.o Logs
Nore Wooettad 17)
Reconttruction Lose
0.020
Morptee (1)
0.004
hare
0.000
0.015
0.003
0.030
0.002
0.002
0.005
0.001
0,000
0.000
0.000
7
19
21
11 13
15
21
Slots
Slots
Objects
(a)
12 loss (L) on CLEVR6
(b) ( loss (L) on CLEVR10
(c) 82 loss (J) on CLEVR
Figure 8: Dependence of reconstruction quality on slot and object count. Models are trained on CLEVR6
with 11 slots.
Training MOVi-C We now shift our focus to the Dinosaur models. F the reconstruction loss for the models trained on MOVi-C10 with 11 slots. Contrary to previous observations, we note in Subfigures Da and 9c that reconstruction losses improve as the slot count increases. Generally, it appears that the weighted sum variant performs worst w.r.t. the loss, while the other variants perform comparably to each other. We observe analogous behavior in Figures 10 and
18

---

Published in Transactions on Machine Learning Research (09/2024)
Reconstruztion Loss
Repanstrutton Loos
2/2
222
2,0
2.0
2.0
KIG OEE 111
KIGE IHE SprVII
Liver
1.6
1.6
10.10
R.A. Mirr 238
13
16
17
19
21
7
9
11
13
15
17
19
21
(a) e loss (L) MOVi-C6
(b) e loss (L) MOVI-C10
(c) e loss (L) MOVi-C
Figure 9: Dependence of reconstruction quality on slot and object count. Models are trained on MOVi-C10
with 11 slots.
Recondtruucion Loss
2.4
Reconatruation Loss
2.1
2..
2,0
2.0
2.0
1.8
1.8
Sare
Winglad
2ach Ner
Bat
18
17
19
15
17
19
Slats
Slots
Objects
e loss (L) on MOVi-C6
(b) g
loss (L) on MÓVi-C10
(c) e loss (L) on MOVi-C
Figure 10: Dependence of reconstruction quality on slot and object count. Models are trained on MOVi-C6
with 7 slots.
Reconstructio Loss
Reconstruction Lose
2.2
2.2
2.2
2.0
2.0
2.0
BLTTI
1.6
1.6
1.8
Ban
11
13
15
17
19
21
7
13
15
19
21
9
Slotss
Objects
(a) e loss (1) MOVi-C6
(b) 8 loss (L) MOVi-C10
(c) e loss (L) MOVI-C
Figure 11: Dependence of reconstruction quality slot and object count. Models trained MOVi-C6 with 11 slots.
19

---

Published in Transactions on Machine Learning Research (09/2024)
B Lemmata on Layer Normalization
In this subsection, we will present some basic lemmata on the LayerNorm module which we use in Subsection D For on E R Pluput, LayerNorm(ū.) refers to the following operation:
LayerNorm(R,) = diag(a) In E 1
VVar TnJ +e +
(14)
where a E RD are learnable parameters, e tor, and B[]x] and Var[ax] are the expectation and variance of the entries of Tk. respectively:
0 constant,
the all-ones vec-
1 D
Var[an] = 1
1 D a E|
E B[Gn]
I
(15)
(16)
Given these definitions, we present the following lemma, giving expression for the affine hull of the image of the layer normalization module.
The veclor is [[Ex|] is the orthogonal projection of Tt onlo the orthogonal complement of the span of the all-ones vector (1) RDige Hence, the image of LayerNorm is contained in (D (Dinput 1)- dimensional affine subspace of R Doput More concretely, the offine hull of the image of LayerNorm module
Lemma 1.
with parameters a. R. Dirpue is given via:
aff (LayerNorm [IR Dupue diag(a) +
(17)
Here and in the following, the left- multiplication of the diagonal matric diag(a) with any set (e.g. the vector space 1-) refers to the image of that set under the left-mulliplication.
Proof. Define e1 == 1/VD1 and extend to an orthonormal basis c1...., Dimput via Gram Scham-. Then we have (1) = (e2, Dimpus) and for an arbitrary &n we have the orthogonal decomposition:
Disput
In
Σ
(ān: ed)ea
(18)
The orthogonal projection onto (1) is then given by
Diaput
Σ
(an ed)ed
En (ān, 01)
(19)
We note that we may rewrite
(in - -
(
1 B[
(20)
Hence, ã. [[ indeed realizes the orthogonal projection of o, onto (1)+ Clearly, (I)+ (Diput - 1)- dimensional linear subspace of R Piupat (being the span of Dimput many vectors). We note that the affine hull commutes with affine functions. Hence, we have:
aff (LayerNorm [R Dmpp)) = (aia aff (g Ptopot]] +
(21)
where the function g is given as:
g Var (.
(22)
We have already shown that ã - E is exactly the orthogonal projection onto 1 Since g differs from this projection via a multiplicative factor, the affine span of its image is exactly the affine span of the projection, which is + Thus, we have shown:
aff (LayerNorm [ Prouut) )) diag(a)1 +
(23)
20

---

Published in Transactions on Machine Learning Research (09/2024)
We can write these affine subspaces in particular fashion, as show in the following lemma. This result will be useful for explicitly writing down function satisfying Proposition 2
Lemma 2. For any affine subspace A C Pipur, there erists a unique vector a E and a unique vectorspace V c R Davua such that E V is orthogonal to V and we have A a aV..
Proof. Given any affine subspace A C IR Dipput let a E A be the orthogonal projection of the origin onto A. By definition of the orthogonal projection, for any other a' E A, we have (a - -) (a' a) = 0. Morcover, V == A a linear subspace of IR Pinpue with a + A. For any arbilrary e E V, we may write a' a for some a' E A and we have
(24) Hence, a V- holds. We have so lar shown the existence of some vector space V and E V with A V.
a a=v= - (a (a' a) =0
We now show that this decomposition A = + V is unique. Assume that there exists another vector space V' c Pupp and a' E T'L with a + V a' + V. We will show that V = V' and = a' must hold. Rearranging, we find V' = (a a') + V and V = V' - (a (a- a'). From the first equation, we may conclude a - a' E V.. Since a - a' a vector V', which is closed under subtraction, we find V (a a') = V V'. We now show that we must also have
Hence, substituting inlo the previous equation, we conclude a a'. We compute:
(a aa- (a d') a aa aa) (a-aa-) a"
(a a')
(25)
and recall that a a' V holds. By assumption, we have a, a' E V- and therefore a (a - a') = and at (a - d') 0. Hence, (a-a') aa) a') 0 and we conclude from the desitive-defmittles of the scalar product that = a' does indoed hold.
Finally, we show that for almost all parameters of the value map and the layer normalization module, the translation vector a from the previous lemma does not vanish. Again, this will be crucial to show that a lunction satisfying Proposition 2 exists almost always.
Lemma 3. Denote D == Dimput and let v Rº + RR be a linear map that is parametrized via matrir B E DXD which acts, say, via left-multiplication.. Consider perameters a, 8 E RRD of the layer normalization. Consider the offine hall of the image of the composition of v and the layer normalization:
A aff ((ro LayerNorm) [RRP])
(26)
As detailed in Lemma 2 we may write A uniquely as a + V, where a and V depend on the parameters of v and the layer normalization. In the following, we abuse notation by not making this dependence explicit. For almost all parameters (B, ax B3 (w.r.t. the Lebesgue measure on DDD x RDD RR.),, we have a #
Proof. Let the set
U== (BB,,,, EX x
R X
R
a a= 0}
(27)
represent the parameters for which vanishes. We will show that this is mullset w.r.t. Lebesgue measure. It is standard fact that the set of non-invertible matrices (RR) c DxD is a Lebesgue-mullset Hence, it is already sufficient to prove that the set
U' Un Un:Un UGLD() x D R
(28)
is a mullset. As we saw from the proof of Lemma 2 the vector is exactly the orthogonal projection of the origin onto A. Hence, = 0 holds iff 0 E A holds. Recalling Lemma [
we know
A == aff ((v LayerNorm) [IR v(fff LayerNorm[[e""]
(29)
= (ddia((a) +
21

---

Published in Transactions on Machine Learning Research (09/2024)
As may assume that B is invertible, it has trivial kernel. Hence, we have 0 A iff 0 diag(a)1 + B. From this requirement, we obtain an explicit expression for the set U":
U" = (R) x ((, ( 3 e (diag(())) (LD x {(a, diag(a)u) a u 1
(30)
We conclude by noting that ((a,-diag(a)) a E IR, E 1-3 is nullset in RD x RD, as it is the image of nullset under continuously differentiable function.
C Proof of Proposition
Proof. Assume by contradiction that such a function TR - exists. Consider now the setting which we have K 1 and 0 == 0. Denote the resulting attention matrix by r and the resulting update code
u1 by wi") Consider also the setting in which we have K := and O = =02 == 0. Denote the resulting
attention matrix by ( and the resulting update codes u1 U2 by w (2) (2) One may now easily verify
w2
that all entries of ( are 1 and that all entries of ((2) are 1/2. Hence, we have
(1)
(1) Wi
m=
7n Up
m=
Un
(31)
(1)
N
7n.1
and
w
(2)
ΣN
m=
N
Un
En Un
(32)
N
m=1
By equation 32
and our assumption (namely, that equation
holds), we have:
N = (w(²))
Σ
un/N
Un
(5(w(²)
1
(33)
2
At the same time, we also deduce from equation 31
Σ
f
Σ en/N / = (wf)))
n=
7n
(34)
N
This contradicts equation 133
7
D Proof of Proposition 2
Proof. We recall from Lemma that for almost all parameters of the value map v and the layer normalization module, we may choose a vector RR {0} and a vector space W c R such that a E W holds and for any input G,, the correspouding values v, lie in a + W. Given fixed parameters (that do not lie in the mullset outlined in Lemma 3/, fix such vector
Now, we define the function f RRD... - IR via:
f(u) = C Tuu
(35)
N| |
The values on may be written
fr + wn
(36)
22

---

Published in Transactions on Machine Learning Research (09/2024)
where wn W and a' wn 0 holds. Hence, recalling that we define Un g/ E compute:
Un may now
f(ux) C a
1
Σ
Yn + tw,)
N Σ nn (a + wp))
2llal
C
N
Σ
n aaa
(37)
M n=1
Σn Yo,k
which is what we wanted to show.
E Technical Details Regarding Object Discovery on CLEVR
While we closely follow the descriptions of (Locatello al. 2020 we use re-implementation of the method in our experiments.
Data As in [Locatello et al., 2020 we use the extended CLEVR dataset that provided in (Kabra et al, 2019). This version of the dataset consists of 100,000 images in total, each being of dimension 320 240. We follow (Locatello al.) [2200] Greff al. 2019 Burgess et al.) 2019) in the pre-processing of this data: We use 70,000 images for training and hold out 15,000 for validation and testing. We perform square center crop of size 192 to increase the space occupied by objects. The cropped images are then bilinearly scaled to shape 128 x 128. The corresponding ground-truth segmentation masks are pre-processed analogously, using nearest-neiighoor interpolation in place of bi-linear interpolation. In contrast to (Locatello el. al. 2020). we augment the data by performing random horizontal flips. Before feeding to the autoencoder, the RGB data is scaled to the interval [1.,
Architecture We follow the autoencoder architecture described in (Locatello et al., 2020) as closely as possible. Conceptually, we divide the autoencoder into three distinct entities: The encoder processes the input image and produces a set tokens. The Slol Atlention module processes these tokens and yields a latent slot representation. The decoder decodes the latent representation into reconstruction of the input.
The encoder consists of convolutional network, which we describe (as in (Locatello et al. 2020 in Table 2
Type
Shape
Out Shape
Activation
Comment
Conv 5 X 5
128 x 128 x 3
128 x 128 x 64
ReLU
stride:1
Conv 5 x
x 128 64
128 128 x
ReLU
stride:1
Conv 5 x 5
128 128 64
128 128 x 64
ReLU
stride:1
Conv 5 x 5
128 x 128 64
128 x 128 x 64
ReLU
stride:1
Pos. Embed
128 128 64
128 128 x
Spatially Flatten
128 x 128 64
(128 128) 64
Layer Norm
(128 128) x
(128 128) 64
per 64-dim token per 64-dim token
Affine
(128 128) x 64
(128 128) 64
ReLU
Affine
(128 128) x 64
(128 128) 64
per 64-dim token
Table 2: Encoder network for experiments CLEVR
We implement the positional embodding as (Locatello al. 2020 Namely, positionally embed a tensor X of shape W x C, we construct W H tensor in which each of the four channels is linear ramp spanning between 0 and 1, either progressing horizontally vertically and in either of the two possible
23

---

Published in Transactions on Machine Learning Research (09/2024)
and directions (e.g. left or right). In order to embed the feature Xtgas learn affine layer R RC compute the embedded feature:
Xi + afffin(P...f..
(38)
The resulting set of input tokens is then processed by the Slot Atlention module, which we implement described in (Locatello et al., 2020). The key, query, and value maps 9, k, the common dimension D = 64. Slots are also 64-dimensional. The residual MLP that is used to update the slot latents has single hidden layer of size 128.
We decode each slot separately, using a spatial broadcast decoder. Once again, we closely follow the approach (Locatello et al. 2020) and detail the decoder architecture Table
Тype
In Shape
Out Shape
Activation
Comment
Spatial Broadcast
64
x 8 x 64
for single slot
Pos. Embed
8 x 8 x 64
8 64
Transposed Conv 5 x
8 8 x 64
16 x 16 x 64
ReLU
stride:2
padding:2
out padding:
Transposed Conv 5 X 5
16 16 64
32 32 64
ReLU
as above
Transposed Conv 5 X 5
32 32 x 64
64 x 64 x 64
ReLU
as above
Transposed Conv 5 x 5
64 64 x 64
128 X 128 x
ReLU
as above
Transposed Conv 5 x 5
128 128 64
128 128 64
ReLU
stride:
padding:2
Transposed Conv 3 x 3 128 x 128 x 64
128 x 128 x
stride:1
padding:1
Table 3: Decoder network for experiments on CLEVR
The 4 channels of the output the decoder are split into RGB channels and an unnormalized alpha channel. The alpha channels are normalized via a soft operation across all slots, and the RGB reconstructions are blended produce an entire reconstruction.
Training We closely follow the training procedure of (oocatelo et al. 2020 Namely, we train the autoencoder with e² reconstruction loss, utilize 3 Slot Attention iterations during training, and use Adam Kingma & Ba 2015) optimizer. The models are trained for 500,000 steps. As the authors of (Locatello et al. 2020), we linearly warm up the learning rate over the course of the first 10,000 steps, after which it attains peak value of 4 (0.5)0.1, 10-4, Subsequently, we decay it over the course of the remaining steps, with a half life of 100,000 steps. We use a batch size of 64.
Evaluation During evaluation, we use Slot Attention iterations.
F Technical Details Regarding Object Discovery on MOVi-C
While we closely follow the descriptions of (Seitzer et al. 2023) we use a re-implementttiinn of the method in our experiments.
Data We use the MOVi-C datasct from (Greff et al. 2022 In total, it contains 10,986 vidco sequences. cach consisting of 24 frames. We hold out 250 of the sequence for validation and 999 for testing. Following (Seitzer et al. 2023), we pre-process the frames from the dataset by bicubicly resizing them to shape 224 224. We use a pretrained vision transformer to pre-compute image features from these resized frames. Spocifically, we use the model vit base aase_patch8_ 224 dino from the timm (Wightman, 2019) repository. After dropping the class token, this model provides 28 x 28 768 feature map for each frame. We save these feature maps at.
24

---

Published in Transactions on Machine Learning Research (09/2024)
half precision (16 bit floating point) and use them during training. Alongside, save the resized frames and analogously resized (via nearest-neighbor interpolation) ground truth segmentations.
Architecture We closely follow the MLP-based autocncoder architecture detailed in (Seitzer et al. 2023 The goal of this auloencoder is to encode and decode the pre-computed ViT features. Importantly, does not operate on RGB images. As before, we conceptually divide the autoencoder into the encoder, the Slot Atlention module, and the decoder.
The encoder processes each VIT feature separately via shared MLP. In contrast to the encoder we used in the experiments on the CLEVR dataset, no positional embedding is employed, as the ViT features still contain a sufficient amount of positional information (see the discussions in (eetterr et al. 2023 ). We provide a detailed description this architecture in Table
Type
In Shape
Out Shape
Activation
Comment
Layer Norm
28 x 28 768
28 x 28 768
per feature
Affine
28 x 28 768
28 x 28 768
ReLU
per feature
Affine
28 x 28 768
28 x 28 128
per feature
Table 4: Encoder network for experiments on MOVi-C
The set of tokens that produced by the encoder is processed by the Slot Attention module to produce a latent slot reprosentation. Slots, keys, values, and queries are 128-dimensional. The residual MLP that is used to updale the slot latents has single hidden layer of dimension 512.
In order to produce reconstruction of the ViT features, Conceptually, the decoder resembles the one we used in the experiments on the CLEVR dataset: Each slot is decoded separately into a partial reconstruction and an unnormalized alpha channel. A complete reconstruction is formed by normalizing the alpha channels across slots and blending the partial roconstructions. Each slot is broadcasted spatially before decoding and a positional embedding is added. Once again, the decoder consists of an MLP that operates on each spatial feature separately. We provide detailed description of the architecture in Table 5
MLP-based decoder architecture is used.
Type
In Shape
Out Shape
Activation
Comment
Spatial Broadcast
128
28 x 28 x 128
Pos. Embedding
28 x 28 128
28 x 28 x 128
Affine
28 x 28 x 128
28 x 28 x 1024
ReLU
per feature
Affine
28 x 28 x 1024
28 x 28 X 1024
ReLU
per feature
Affine
28 x 28 X 1024
28 x 28 x 1024
ReLU
per feature
Affine
28 28 x 1024
28 x 28 (768 + 1)
per fcature
Table 5: Decoder network for experiments on MOVi-C
The positional embedding used in Table 5 differs significantly from the one we used in the experiments on CLEVR. Namely, given an input tensor of shape W x H x C, we positionally embed the feature X by learning a tensor P shape W x H x and computing:
X + Pija
(39)
Training We follow the training procedure detailed in (Seitzer et al.; 2023 Namely, we train the autoencoder via an e reconstruction loss and use 3 Slot Attention iterations during training. The batch size is 64 and we employ an Adam optimizer. As before, a learning rate schedule consisting of a linear increase and exponential decay used. Here, the peak learning rate 4 10-4, which is reached after 10,000 steps. A Afterwards, the learning rate decays with half life of 100,000 steps. The models are trained for 500,000 steps in total.
25

---

Published in Transactions on Machine Learning Research (09/2024)
Evaluation During evaluation, also utilize 3 Slot Attention iterations. Since the alpha masks that are produced by our model are of shape 28 x 28, cannot directly compare them to ground truth segmentations, which are of shape 224 x 224. We follow the approach of (Seitzer et al. 2023) and bi-linearly upscale the alpha masks to shape 224 x 224 before deriving segmentations, which we then compare to the ground-truth.
G Pseudocode
In Algorithms 1 and we illustrate how the weighted sum and batch norm variants differ from the weighted mean variant in pseudo Py code. We illustrate this in diff format.
Algorithm Diff' of Weighted Sum Variant
bs, N, d_in inputs.shape
k, v = self wewf.key..aeeeeeeeeeeeeeeeeeeeee keekkeyapp www.kkyymipppppp (inputs), self. gallue_map(inputte
for idx in range
slots_prev slots
slots self hhomm_slossss/ll q self query_ satry__mppsslota_
dots = torch. ("didumm attn dots.softmax (dim=1)
k) / mp. sire (-1))
attn (attn + eps) / (attn eps). sum (dim=-1, keepdim=True)
10
updates torch.einsume siinumm""jd, bij->bid v, attn)
11
12
updates updates N
13
Algorithm Diff' of Batch Norm Variant
bs, N, d_in k, v = self.key_mn weekeeaamiipppeeeeeeee (inputs), self, saluvvale_maan (aalle__mp(innut)) var, mean None, None for idx in (aage(mumiitera) slots_prev slots
prppts.shap.
slots self hwrffnoomm_ooote mommmmsllssssssltts))
q self querq_ueeaaaa query_map(slots)
dots = torch. einsum (ind
9, k) / sp.sqrt(q.size
attn = dots.softmax (dim=1)
10
attn (attn + eps) / (attn eps) sum (dim=-1, keepdim=True)
11
12
updates torch. wind if idx and self. trafnirainin:
v, attn)
18
var, mean torch. var sar__eaannue (updates, correction=0)
self. update buffers (var, mean)
elif idx == and not self. training
var, mean = war_buffer, self mean buffer
mean_buffer
updates self. alpha (updates mean) / sqrt eps) self, beta
18
10
26

---

Published in Transactions on Machine Learning Research (09/2024)
H Visual Results
present visual results from object discovery the CLEVRIO and MOVi-C10 datasets, respectively. On CLEVR, we show both reconstructions and segmentations, while we only provide segmentations on MOVi-C, as the reconstructions are high-dimonsional ViT features. In both cases, wo perform inference with high slot count (21).
In Tables 6 and
Input
Baseline
Layer Norm
Weighted Sum
Batch Norm
Table 6: Visual results for object discovery CLEVR10, trained CLEVR6 with 7 slots. Showing reconstructions and segmentations. Evaluated with 21 slots.
27

---

Published in Transactions on Machine Learning Research (09/2024)
Baseline
Layer Norm
Weighted Sum
Batch Norm
Table 7: Visual results for object discovery MOViC10, trained MOViC10 with 11 slots. Evaluated with 21 slots.
28

---

Published in Transactions on Machine Learning Research (09/2024)
Property Prediction
We further illustrate the proposed normalizations on a property prediction task on the CLEVR1O dataset. We closely follow the experimental sctup detailed by Locatello al. (2020), using three sceds per variant. We train the models the CLEVRI0 dataset using 10 slots. In Figure 12 we plot how the foreground ARI varies the number of slots is modified during inference. Consistent with our observations discovery task, we find that the weighted mean and layer norm variants suffer from excess slots, while the proposed normalizations are robust to them. Overall, we find that the batch norm variant performs best w.r.t. foreground segmentation quality.
the object
0.6
0.4
0.2
0.0
11
15
19
Slots
Figure 12: F-ART (f) for property prediction on CLEVR10
additionally show the mean average prediction various distance thresholds, as defined in (ocatello et al. 2020). We observe similar behavior as before, namely that the proposed variants are robust to high slot counts during inference, while the weighted mean and layer norm variants are not. However, we generally find that the weighted mean variant appears to perform best at low slot counts.
In Figure 13
0.8
0.6
mAP-100
0.4
0.4
Norm
Loger nOrTS
0.0
11
15
11
15
19
# Slots
# Slots
(a) mAP@LO (t) on CLEVRIO
(b) mAP@0.5 (t) on CLEVRIO
0.020
mAP-0.25
0.015
0.010
Batch Num
Batch Num
0.00
0.000
11
13
15
17
19
21
13
15
17
Slots
Slots
(c) mAP@0.25 (1) CLEVR10
(d) mAP@0.1.. 12. (t) CLEVR10
Figure 13: Mean average precision different distance thresholds for property prediction CLEVR.
29

---

Published in Transactions on Machine Learning Research (09/2024)
J Ablation of Normalization Schemes
In this section, we study two alternative normalizations methods. Firstly, we consider the weighted sum variant with the scaling parameter chosen as = 1. We refer to this method as "unnormalized". Secondly, we consider a variant which ablates the normalization across the batch axis from the batch normalized variant. Le., we compute mean and variance for each instance separately across only the slot and layer axes, Hence, we also do not have to keep a moving average of the normalizing statistics for inference. Instead the normalization behaves identically during training and inference. As in the batch normalized variant, two scalar values and 3 are learned. We refer this variant "K-D-Layer Norm', to reflect that performing layer normalization across the slot (K) and layer (D) axes. We perform experiments the property prediction task on CLEVR with three seeds per variant.
The unnormalized variant fails to obtain object-centric behavior, as becomes apparent from the low foreground ARI across all slot counts, as shown Figure 14 In Figure [5] we observe that the mean average precision suffers correspondingly. This underscores the importance of scaling the update codes appropriately competitive performance. While the K-D-Layer Norm performs better, find that as the other baseline variants, it is not robust to high slot counts. Morcover, this specific task, it scems to generally underperform in terms of mAP when compared to the other object-centric variants.
achieve
0.6
0.2
Batch Rorm
0.0
Slots
Figure 14: F-ARI (t) for property prediction CLEVR10
0.6
mAP-10
0.6
0.4
Exyel Nom
Jol Sarm Weightod S1.
0.2
0.2
Dnnorrod 200
UnaEeIT 2011
0.0
0.0
11
13
15
17
19
21
7
11
13
15
17
19
21
Slots
(a) MAP@10 (t) CLEVRI0
(b) mAP00.5 (1) CLEVR10
0.20
IAAP-0.25
0.15
0.015
0.10
0.010
Losa
Layaa
0.005
0.000
15
# Slots
# Slots
(c) mAP30.25 (1) on CLEVR10
(d) mAP@0 125 (t) on CLEVR10
Figure 15: Mean average precision at different distance thresholds for property prediction on CLEVR.
30


Reviews:
[{"review_id": "wEUR0D2SH2", "reviewer": "Reviewer_48Hx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper is well-motivated in studying the generalization capabilities of Slot Attention. In particular, the normalization aspect of Slot Attention has not been scrutinized in prior research.\n2. The connection between the Slot Attention algorithm and EM in a mixture model of von Mises-Fisher distributions is interesting and seems sound, from my understanding.\n3. The experiments thoroughly study how varying the number of slots and objects affects segmentation performance across several datasets and methodologies, although I believe there could be more investigations into other aspects of the slot representations, as I describe below.\n\n**Weaknesses / Questions**\n\n1. Currently, all evaluations in the paper are with regard to segmentation quality. It would be very informative to investigate the quality of the representations by property prediction (e.g. as is done in [1, 2]) under the different normalization schemes. I understand that many recent works on object-centric learning only report FG-ARI, but in proposing a new normalization scheme, it is important to see its effect on the learned representation.\n2. In the Abstract, the authors write: “We argue that the original Slot Attention normalization scheme discards information on the objects’ sizes, which impairs its generalization capabilities.” However, object size is never investigated in the paper.\n3. The authors mention [3] as a related work that also investigates normalization in Slot Attention, but do not compare with it in their experiments. I believe at least comparing with SA-SH from their paper would be informative.\n4. The error bars seem larger with weighted sum normalization and the authors also state that for two seeds, the autoencoders decompose the input spatially instead of object-wise. Did the authors generally find these normalization schemes to be less stable to train?\n5. From Proposition 1, it seems one of the main takeaways is that with the weighted mean normalization that is normally done in Slot Attention, the column sums in the attention matrix will be different if the number of slots is changed at inference time. This is supported by their experiments showing worse segmentation quality when the number of slots during inference is different than the number of slots during training. It also makes sense intuitively because the scaling is different at inference time than at training time. What would happen if we use weighted mean normalization but scale the attention matrix by num_slots_inference / num_slots_training when evaluating with num_slots_inference?\n\n[1] Object-Centric Learning with Slot Attention (https://arxiv.org/abs/2006.15055) \n\n[2] Object-Centric Slot Diffusion (https://arxiv.org/abs/2303.10834) \n\n[3] Unlocking Slot Attention by Changing Optimal Transport Costs (https://arxiv.org/abs/2301.13197)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the normalization scheme in Slot Attention. The authors draw connections between Slot Attention and expectation maximization in a mixture model of von Mises-Fisher distributions and argue that weighted sum normalization preserves information on the fraction of input tokens assigned to the slots. In experiments, they demonstrate that weighted sum normalization and batch normalization show better generalization capability when scaling to varying number of slots during inference when compared to weighted mean normalization or layer normalization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I’ve listed several suggestions in the Weaknesses / Questions section above. While I believe addressing all of them would make a stronger paper, addressing #2 is critical since it is not supported in the paper. I am also especially curious about #5.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "52uXcvAE5A", "reviewer": "Reviewer_433y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. A simple normalization scheme on top of slot-attention to make it more robust to increase the number of slots during inference.\n\n2. Extensive and insightful experiments on object-centric video datasets. \n\n**Weaknesses**\n\n1. My major concern with this work is the lack of experimentation on complex scenes. In particular, even within the MOVi datasets, there are MOVi-D and E datasets that are more complex than MOVi-C in terms of camera motion and textured background so I'm curious as to how the results shown in 5 (a-b) generalize to more complex scenes. In addition, it would be good to see if there is any benefit of the proposed weighted-sum normalization over attention in the case of CATERTex [2] and MOVi-Tex scenes as well.\n\n\n2. The related works section needs to be elaborated and is poorly written in terms of lack of enough citations of prior object-centric works [7, 8] and prior slot-attention based works [3, 4, 5, 6] (I've only cited a few -- please refer to these works and their related works section to see more references). I find the related works section related to the normalization aspect of slot vectors to be sufficient however, the purpose of the related works section is to situate your work among the pool of existing works, and contrasting the aforementioned works would help the reader to gain some context of this work.\n\n----\n\n**References**\n\n[1] Generative Model-based Clustering of Directional Data, Arindam Banerjee et al., KDD 2003\n\n[2] CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning, Giridhar et al., ICLR 2020 \n\n[3] Slotformer: Unsupervised visual dynamics simulation with object-centric models, Wu et al., ICLR 2023\n\n[4] Conditional Object-Centric Learning from Video, Kipf et al., ICLR 2022\n\n[5] Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames, Biza et al., ICML 2023\n\n[6] Simple unsupervised object-centric learning for complex and naturalistic videos, Singh et al., NeurIPS 2022\n\n[7] Space: Unsupervised object-oriented scene representation via spatial attention and decomposition, Lin et al., ICLR 2020\n\n[8] Spatially invariant unsupervised object detection with convolutional neural networks, Crawford & Pineau, AAAI 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Slot attention in object-centric learning is a way to bind different \"slots\" to different objects in the scene. It involves an attention mechanism over different slots. This work proposes simple changes to slot attention, in particular, to the normalization scene of the attention vectors. The authors show that the current slot-attention normalization of the weighted mean of slot vectors is one factor that significantly impedes the generalization of slot-attention to OOD number of objects. To remedy this, authors propose 2 extensions of the normalization -- weighted sum and batch norm for attention vectors and show how these two normalization schemes can generalize to OOD number of slots, number of objects, and even OOD appearance in CLEVR and MOVi-C datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Why is it an appropriate choice to model the slots via a mixture of von Mises-Fisher distributions? I believe the manuscript needs a paragraph on what is it about this distribution that makes it amenable to object-centric slot attention-based methods.\n\n2. Section 3.1 also needs to cite and briefly discuss [1]. [1] discusses the relation of EM on mixture of von Mises-Fisher distributions.\n\n3. A pseudocode of the proposed normalization scheme similar to the slot-attention paper. In particular, the changes (highlighted in a different color) from the original slot-attention algorithm would be *incredibly* beneficial to readers who are already familiar with slot-attention.\n\n4. Please add visual results of the CLEVR and MOVI-C scenes, especially in the cases where there is a significant difference during inference with a higher number of slots.\n\n5. I'd also request the authors to address my **weaknesses** points appropriately with experiments for #1 and text for #2.", "broader_impact_concerns": "None required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wlYGyKeJot", "reviewer": "Reviewer_wHLB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- One major issue with SA is determining the number of $K$ during training. The proposed normalisation schemes generalise well even when the number of objects is misspecified during training, which is a very interesting result.\n\n- The proposed method appears to generalise to various numbers of objects, as demonstrated by its zero-shot capabilities.\n\n- Regarding Proposition 1, am I correct in stating that, in the case of SA with mean normalisation, it is impossible to find mixing coefficients $\\pi$? If not, what is the implication of Proposition 1?\n\n- For Fixed Scaling normalisation, ablations with $C=[1,<N,N,>N]$ would be useful to convey the point more effectively, given the wide bounds in Equation 13.\n\n- For Batch Scaling, I understand that normalising across the batch index aims to achieve scalar normalisation for a mini-batch. However, how does this approach compare to image-level normalisation (i.e., normalising only along the $K$ and $D$ axes)?\n\n- The authors claim that the SA discards object size information. Could you please provide more details? Given that the decoder reconstructs by generating slots in appropriate positions, it seems that slot representations do include positional knowledge.\n\n- What are the advantages of splitting the subspace $A=a+V$ and under what conditions does this hold true?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper primarily explores the effects of normalisation within the slot-attention architecture. To achieve this, the author suggests employing the Von-Mises-Fisher distribution to model the SA algorithm as an uncoupled iterative EM algorithm.\n\n- Two implementation schemes for the proposed method are provided, each offering distinct advantages.\n\n- The normalisation scheme is finally tested on the CLEVER 6 and MOVi datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The contribution is not clear from the abstract and introduction. The authors discuss different types of normalisation, so introducing the differences in the normalisation schemes earlier would help readers.\n\n- The authors mention the misalignment in EM steps, but its effects have not been discussed in detail. It would be interesting to see the performance differences between aligned and unaligned EM algorithms with vMF distributions, similar to the analysis in [1].\n\n- The concentration parameter $\\tau$ in the vMF distribution is somewhat analogous to variance in the normal distribution. How reasonable would it be to make $\\tau$ a learnable parameter? This could result in learning the true underlying slot distributions.\n\n- It would be valuable to include an ARD analysis on the mixing coefficients $\\pi$.\n\n- Please include visual results, demonstrating improvements in object discovery.\n \n\n[1] Kori, A., Locatello, F., Santhirasekaram, A., Toni, F., Glocker, B. and Ribeiro, F.D.S., 2024. Identifiable Object-Centric Representation Learning via Probabilistic Slot Attention. _arXiv preprint arXiv:2406.07141_.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were convinced by the paper and the authors' responses to their initial comments/questions.
Decision: Accept (poster)