Identifiable Object-Centric Representation Learning via Probabilistic Slot Attention
Authors: Avinash Kori
Francesco Locatello
Ainkaran Santhirasekaram
Francesca Toni
Ben Glocker
Fabio De Sousa Ribeiro
Venue: NeurIPS 2024 poster (2024)
Keywords: Object-centric learning
Probabilistic slot-attention
Identifiability
latent mixture models
Primary area: generative_models
TL;DR: We propose a method to learn identifiable object-centric representation up to a proposed equivalence relation.

Abstract: Learning modular object-centric representations is said to be crucial for systematic generalization. Existing methods show promising object-binding capabilities empirically, but theoretical identifiability guarantees remain relatively underdeveloped. Understanding when object-centric representations can theoretically be identified is important for scaling slot-based methods to high-dimensional images with correctness guarantees. To that end, we propose a probabilistic slot-attention algorithm that imposes an *aggregate* mixture prior over object-centric slot representations, thereby providing slot identifiability guarantees without supervision, up to an equivalence relation. We provide empirical verification of our theoretical identifiability result using both simple 2-dimensional data and high-resolution imaging datasets.

Identifiable Object-Centric Representation Learning via Probabilistic Slot Attention
Avinash Kori
Francesco Locatello
Ainkaran Santhirasekaram'
Francesca Toni!
Ben Glocker!
Fabio De Sousa Ribeiro
Imperial College London, UK
2 Institute of Science and Technology. Austria a mhori21@imperta ac..uk uk
Abstract
Learning modular object-centric representations is crucial for systematic general- ization. Existing methods show promising object-binding capabilities empirically, but theoretical identifiability guarantees remain relatively underdeveloped. Un- derstanding when object-centric representations can theoretically be identified is crucial for scaling slot-based methods to high- nigh-dimensional images with correctness guarantees. To that end, we propose a probabilistic Alot-attention algorithm that imposes aggregate mixture prior over object-centric slot representations, thereby providing slot identifiability guarantees without supervision, up to an equivalence relation. We provide empirical verification of our theoretical identifiability result using both simple 2-dimensional data and high-resolution imaging datasets.
1 Introduction
has been hypothesized that developing machine learning (ML) systems capable of human-level understanding requires imbuing them with notions of objectness [48, 65]. Objectness notions can be characterised as physical, abstract, semantic, geometric, or via spaces and boundaries [83, 17]. Humans can generalise across environments with few examples to learn from [70], and this has been attributed to our ability to segregate percepts into object entities [64, 25, 45, 4].
Obtaining object-centric representations deemed to be a key step for achieving true compositional generalization [5, 48, 22], and uncovering causal influence between discrete concepts and their environment [57, 19, 20, 65, 4]. Significant progress in learning object-centric representations has been made [15, 16, 44, 66, 10, 67], particularly in unsupervised object discovery settings using an iterative attention mechanism known as Slot Attention (SA) [53]. However, most existing work approaches object-centric representation learning empirically. leaving theoretical understanding relatively underdeveloped. Establishing the identifiability [31, 29] of representations is important as clarifies under which conditions object-centric representation learning theoretically possible [6].
well-known result shows that identifiability of latent variables is fundamentally impossible with- out assumptions about the data generating process [31, 51]. Therefore, understanding when ob- ject representations can theoretically be identified is important to scale object-centric methods to high-dimensional images. Recent works [6, 47] make important advances on this by explicitly
* Equal Contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

stating the set of assumptions necessary for pro- viding theoretical identifiability of object-centric representations. However, they restrict their atten- tion to properties of the mixing function, studying class of models with additive decoders. Although there are merits to this approach, there are prac- tical challenges with the so-called compositional contrast objective [6], as it involves computing Ja- cobians and requires second-order optimization via gradient descent. Consequently, satisfying the iden- tifiability conditions explicitly (e.g. compositional contrast must be zero) is computationally restrictive for moderately high-dimensional data. In this work, we present a probabilistic perspective that is not subject to the same scalability issues while still providing theoretical identifiability of object-centric representations without supervision. In Table 1, we list object-centric learning methods, their (sometimes implicit) modelling assumptions (see $ 5 for additional information and Appendix A for a detailed breakdown and discussion), and their respective identifiability guarantees of object representations. Most methods do not guarantee identifiability, and make the disentangglemen (1) and additive decoder (2) assumptions. Brady et. al. [6] do provide identifiability guarantees and additionally assume irreducibility (3) and compositionality (4). Our method provides identifiability guarantees by introducing latent structure (i.e. via a GMM prior) which generalizes to non-additive decoders. This is advantageous as the computational complexity of additive decoders scales linearly with the number of slots K- whereas our approach is invariant to K. Moreover, non-additive decoders have been found to significantly improve performance in prac- tice [66, 67, 69), though the theoretical basis underexplored. Finally, latent structure can reduce the complexity burden on the mixing function f (decoder), making it easier to learn in practice [18, 43).
Table I: Identifiability strategies (mixing func- tion f or latent dist. p(z)), and assumptions made by object-centric learning methods.
METHOD
ASSUMPTION
IDENTIE.
53,
1,2
N/A
[13,42, 731
1,2,5
N/A
[44]
1,2, 6
N/A
3,4,3,77
Proposed
1.8
p(z)
Contributions. Our main contributions are the following: (i) We prove that object-centric represen- tations (i.e. slots) are identifiable without supervision up to an equivalence relation ($ 5) under a latent mixture model specification. To that end, we propose a probabilistic slot-attention algorithm ($ 4) which imposes an aggregate mixture prior over slot representations. (ii) We show that our approach induces a non-degenerate (global) Gaussian Mixture Model (GMM) by aggregating per-datapoint (local) GMMs, providing slot prior which: (a) is empirically stable across runs (i.e. identifiable up to affine transformations and slot permutations); (b) can be tractably sampled from. (iii) We provide conclusive empirical evidence of our theoretical object-centric identifiability result, including visual verification on synthetic 2-dimensional data as well as standard imaging benchmarks ($ 6).
2 Related Work
Identifiable Representation Learning. Identifiability of representations stems from early work in independent component analysis (ICA) [31, 29], and making a resurgence recently [30, 32, 51, 37, 75, 46, 82]. Common strategies for tackling this identifiability problem are: (i) restricting the class of mixing functions: (ii) using non-i.i.d data, interventional data or counterfactuals; and (iii) imposing structure in the latent space via distributional assumptions. Regarding (i), restricting the transformations [81] has been found to produce identifiable models. For (ii), prior works [84, 52, 7, 2. 75] assume access to contrastive pairs of observations (x, x) obtained from either data augmentation, interventions, or approximate counterfactual inference. As for (iii), latent space structure is enforced via either: (a) using auxiliary variables to make latent variables conditionally independent [33, 37, 38]; or (b) distributional assumptions such placing a mixture prior over the latent variables in VAE [11, 80, 43]. In this work, we prove an identifiability result via strategy (iii) but within an object-centric learning context, where the latent variables are a set of object slots [53].
class of the mixing functions to conformal maps [8] or
Object-Centric Learning. Much early work on unsupervised representation learning is based on the Variational Autoencoder (VAE) framework [41], and relies on independence assumptions between latent variables to learn so-called disentangled representations [5, 24, 40, 12, 59]. These methods are closely linked to object-centric representation learning [9, 15, 21]. as they also leverage (iterative) variational inference procedures [58, 73, 50]. Alternatively, an iterative attention mechanism known as slot attention (SA) [53] has been the focus of much follow-up work recently [16, 67, 76, 68, 14].

---

Enstronment
Local GHIV
Aggrogato UNV
Samepled
Encoder
Figure 1: Probabilistic slot attention and the identifiable aggregate slot posterior. (Left) Slot posterior GMMs per datapoint (local) and the induced aggregate posterior GMM (global). (Right) Sampling slot representations from the aggregate slot posterior is tractable.
Although slot attention-based methods show promising object binding [22] capabilities empirically on select datasets, they do not provide identifiability guarantees the learned representations. Recently, [55, 56] assume the access to interventional data-generating process following [I] to demonstrate the identifiability of object-centric representations, while [6, 47] presented the identifia- bility results for object representations (i.e. slots), clarifying the necessary assumptions and properties of the mixing function (e.g. additive decoders). However, satisfying Brady et. al. [6]'s compositional contrast identifiability condition explicitly (must be zero) requires computationally restrictive second- order optimization. In contrast, shift the focus learning structured object-centric latent spaces via Probabilistic Slot Attention (PSA), bridging the gap between generative model identifiability literature and object-centric representation learning. Notably, our PSA approach is also related to probabilistic capsule routing (27, 63, 62, 61] since slots are equivalent to universal capsules [26], but like slot attention, offers output permutation symmetry and does not face scalability issues.
3 Background
Notation. Y CRHWXxC denote the input image space, where each image x is of size H x W pixels with C channels. Let fa X' 2 denote encoder mapping image space space 2 C xd where each latent variable z consists of N, d-dimensional vectors. Lastly, let fa S + X' denote a decoder mapping from slot representation space S g kxd to image space.
latent
Slot Attention. Slot attention [53] receives set of feature embeddings z E Nxd per input x, and applies an iterative attention mechanism to produce K object-centric representations called slots E kx Let WR. W, denote key and value transformation matrices acting on z, and W, the query transformation matrix acting on s. To simplify our exposition later on, let fa Z x S + S be shorthand notation for the slot update function, defined as:
Aij
qk
s s111 = fo(zs'' Av,
Ag =
A = softmax
E K
(1)
E An
where q = W E RR k= E. and W. E R correspond the query, key and value vectors respectively and A R is the attention matrix. Unlike self attenttent [74], the queries in slot attention are function of the slots s', and are iteratively refined over T iterations. The initial slots s!=0 are randomly sampled from standard Gaussian. The queries at iteration are given by q' W,s and the slot update process can be summarized as in Equation 1.
Compositionality. Compositionality as defined by Brady et al. [6] is structure imposed on the slot decoder mapping fa which implies that each image pixel is a function of at most one slot representation, thereby enforcing local sparsity structure the Jacobian matrix of fa.
3

---

1
u
s
0
Z
X
z
x
s
(O.I)
K
N
M
MK
N
(a) Slot Attention Encoder
(b) Probabilistic Slot Attention
(c) Aggregate Posterior Mixture
Figure 2: Graphical models of probabilistic slot attention. (a) Stochastic encoder of standard slot (b) Proposed model each image in the dataset {x; J.M is encoded into a respective latent representation z E IRN xd to which a (local) Gaussian mixture model with K components is fit via expectation maximisation. The resulting Gaussians serve as slot posterior distributions: S& ~ NV(ski Plx. o for k = I...... ................................ K. (c) Aggregate posterior distribution M
attention [53] with T' attention iterations.
obtained by marginalizing out the data: g(z) = -i=1 g(z x.)/M. We prove that g(z) is tractable, non-degenerate Gaussian mixture distribution which: (i) serves as the theoretically optimal prior over slots; (ii) is empirically stable across runs (i.e. identifiable up an affine transformation and slot permutation); (iii) can be tractably sampled from and used for scene composition tasks.
Definition 1 (Compositional Contrast). For a differentiable mapping fa Z + X, the compositional contrast of fa at z given by:
KKC ΣΣ
(f((z),
0fa(z)n Oz,
Coump (f
Σ jj=k+1
Oza
Brady et al. [6]'s main result (Theorem 1) relies on compositionality and invertibility of fa to guarantee slot-identiiiabiiity when both the compositional contrast and the reconstruction loss equal zero. However, using Coomp (fa, z) as a metric or as part of an objective function is computationally prohibitive?. Our method aims to minimize Cromp (Sa, 2) implicitly [47], without additive decoders.
4 Probabilistic Slot Attention
In this section, we present probabilistic slot attention framework which imposes mixture prior structure over the slot latent space. This structure will prove to be instrumental in establishing our main identifiability result in Section 5. We begin by approaching standard slot attention [53] from a graphical modelling perspective. As shown in Figure 2 and explained in Section 3, applying slot attention a deterministic encoding z = fe (x) E R Nxd yields set of K object slot representations $1:K = S SK. This process induces a stochastic encoder q(S1:K x). where the stochasticity
comes from the random initialization of the slots: $ ~ (s 0, I) E RR Kx Since each slot is deterministic function of its previous state s' fz s' 1) it possible randomly sample initial states 5d and obtain stochastic estimates of the slots.' However, since each transition depends on which in turn depends on the input x. we do not get a generative model we can tractably sample from. This can conceivably be remedied by placing a tractable prior over z and using the VAE framework along the lines of [77], however, here we propose an entirely different approach which does not require making additional variational approximations (see Appendix G for further discussion).
Local Slot Mixtures. Probabilistic slot attention augments standard slot attention by introducing per-datapoint (i.c. local) Gaussian Mixture Model (GMM) for learning slot distributions. Intuitively, a local GMM can be understood as way to cluster features within given image, encouraging the grouping of similar features into object representations. However, unlike regular clustering, here the clustered points are dynamically transformed representations of the actual data. Specifically,
FF.g. for a CNN with 500K parameters with hatch size 32, 2 125GB GPU memory is needed NNoe that we may use s' or s(t) interchangeably denote slot representations at slot attention iteration t.

---

we use an encoder function fe that maps each image x; E R in the dataset {x to latent spatial representation z; E R Nxd, The latent variable z may be deterministic or stochastic, and we consider the case where N HW to reflect a modest downscaling with respect to (w..t.) x. The goal is to dynamically map each of the N, d-dimensional vector representations in each z, to one-of- K object slot distributions within a mixture. local GMM can be fit to each posterior latent representation z ~ q(z xz)*
on the fly by maximizing likelihood:
NV.
K
II (zn Hikk oie),
p(z Te. pi: 0:) =
(2)
where 14: (P4....., (ak), of (
aix) and π. = (Ti1. Tik) are the respective means, diagonal covariances and mixing coefficients of the 8th K K-component mixture. Figure 2b illustrates the resulting probabilistic graphical model (PGM) in more detail.
To maximize the likelihood in Equation Algorithm 1 Probabilistic Slot Attention
per datapoint X,, present bespoke (EM) algorithm for slot attention, yielding closed-form up- date equations for the parameters as shown in Algorithm 1, and explained next.
ippectation-maxiiisate
Input: fe(x) E R k W.z E R
representation
> compute keys optional == k
V < W.z E R
Vk. T (0) + K 1(0)a ~ ~ (0) 1a
Probabilistic Projections. A powerful property of slot attention and cross-attentiin more broadly [74], is its ability to decouple the agreement mechanism from the repre- sentational content. That is, the dot-product is used measure agreement between each query (slot) vector and all the key vectors, to dictate how much of each value vector (con- tent from z,) should be represented in each slot's revised representation. To retain this flexibility and decouple the attention com- putation from the content, we incorporate key-value projections into our probabilistic approach. For brevity, the i subscript im- plicit in the following, keeping in mind that these are local quantities (per-datapoint x;). The parameters of the K Gaussian slot distributions are initialized (at attention iteration t 0) as follows:
for t = 0, 0,...,T--1
Ank + T())NN (kni (
Σ)=1 (()) (km Wap(t)s o (t)3)
Ank +- Ank/ /i AlR
D normalize
aut + 1)k + En= AntVa b update slots
at + 1)h + Em Ank (v,
pl + 1)
at + 1)k + >m= Ank/N update mixing return (TT),a(T))
K slot distributions
Ak: π (0)x K-1
(00), (100
(0) 1
(3)
The respective queries q. keys k, and values v are then given by: q(t) (,t(())
k W.z,
W.z.
(4)
where Wa. WR, W. E ddd whereas q(t) denotes the queries at attention iteration t. To measure agreement between each input feature (key) and slot (query), we evaluate the normalized probability density of each key under a Gaussian model defined by each slot:
K
1 Ank = (t) ((t))())) of (t)2), Z= Σ (t), q(t),, (t)))
(5)
J=1
where Ank corresponds to the posterior probability that slot (query) k is responsible for input feature (key) n. This process yields the slot attention matrix A E N As shown in Algorithm 1, the mixture parameters π (t), (t),, or (t)2 are then updated using the attention matrix and the values v. If the values are chosen to be equal to the keys v = k, then the procedure is more in line with standard EM, but the agreement mechanism and the content become entangled. After T probabilistic slot attention iterations, the resulting K Gaussians serve as slot posterior distributions:
s(T) N (μ(T)), (T))).
for k = 1,..., K,
(6)
where (T') and (T are the parameters of all the Gaussians in the mixture given a particular datapoint x. The slots ((T") are then used for input reconstruction, e.g. by maximizing (possibly Gaussian) likelihood p(x x ((T)): parameterized by a (possibly additive) decoder sa.
"The parametric form of g can be e.g. Gaussian Dirac delta.
5

---

GMS
Avy.
Auy
334M
0100
Figure 3: Aggregate Gaussian Mixture Density, Examples of aggregate posterior mixtures. For each plot, we compute the aggregate mixture (red line) based on three random bimodal Gaussian mixtures, and plot the respective densities. The three GMMs here are analogous to the local GMMs obtained from probabilistic slot attention (Algorithm I), and the aggregate GMM represents g(z).
Computational Complexity. Probabilistic slot attention (PSA) retains the O(TN compu- tational complexity of slot attention. The additional operations introduce for calculating slot mixing coefficients and slot variances (under diagonal slot covariance structure) have complexities of O(NK) and ((N K d) respectively, which do not alter the dominant term. When PSA is combined with an additive decoder, it can lower computational complexity by eliminating the need to decode inactive slots. In the following, we outline principled approach for pruning inactive slots.
Automatic Relevance Determination of Slots. An open problem slot-based modelling is the dynamic estimation of the number of slots K needed for each input [51, 44]. Probabilistic slot attention offers an elegant solution to this problem using the concept of Automatic Relevance Determination (ARD) [60]. ARD is statistical framework that prunes irrelevant features by imposing data-driven, sparsity-inducins
priors on model parameters to regularize the solution space. Since the output mixing coefficients π (T) IR are input dependent (i.e. local), irrelevant components (slots) will naturally be pruned after T attention iterations, i.e. π (T"); 2 + 0 for any unused slot k. We can either use probability threshold [0, prune unused slots place a Dirichlet prior over the mixing coefficients to explicitly induce sparsity. For simplicity, we take the former approach:
S, = {s())s k E [K], > 7).
(7)
where S, denotes the set of active slots with mixing coefficient greater than r, and each slot is (optionally) sampled from its Gaussian distribution: s(T); ~ N (M(T)s,o
Aggregate Posterior Gaussian Mixture. As previously explained, probabilistic slot attention goes beyond standard slot attention by introducing a per-datapoint (i.e. local) GMM to lear distributions over slot representations. This imposes structure over the latent space and gives access to posterior slot distributions after the attention iterations. Rather than constraining slot posteriors to be close to a tractable prior e.g. via the VAE framework [41, 76] which requires further variational approximations we leverage our probabilistic setup to compute the optimal (global) prior over slots.
We propose to compute the aggregate slot posterior by marginalizing out the data: g(z)= = Ei 4(2 x;) x,)MM, given pre-trained probabilistic slot attention model (Fig. 3). In $ 5, we prove that the aggregate posterior is tractable, non-degenerate Gaussian mixture distribution which: (i) Serves as the theoretically optimal prior over slots; (ii) Is empirically stable across runs (i.e. identifiable up affine transformation and slot permutation, 5); (iii) Can tractably sampled from and (optionally) used for slot-based scene composition tasks. Since GMMs are universal density approximators given enough components (even GMMs with diagonal covariances), the resulting aggregate posterior (zz) is highly flexible and multimodal. It often suffices to approximate using sufficiently large subset of the dataset, if marginalizing out the entire dataset becomes computationally restrictive, although we did not observe this to be the case in practice in our set of experiments.
5 Theory: Slot Identifiability Result
In this section, we leverage the properties of the probabilistic slot attention method proposed in Section 4 to prove new object-centric identifiability result. We show that object-centric representa- tions (i.e. slots) are identifiable without supervision (up to an equivalence relation) under mixture model-like assumptions about the latent space. This contrasts with existing work, which provides identifiability guarantees within specific class of mixing functions, i.e. additive decoders [47]. Our result unifies generative model identifiablity [32, 37, 43] and object-centric learning.

---

Definition 2 (Identifiailitty)) Given an observation space X' probabilistic model with parameters ② is said to be identifiable if the mapping 0 E O ++ Po (x) is injective:
(pe, (x) pe, (x), VX X)
O2.
(8)
Remark 1. Definition 2 says that if any two choices of model parameters lead to the same marginal density, they are equal. This is often referred to as strong identifiability [3], and it can be too restrictive, as guaranteeing identifiability up to a simple transformation (e.g. affine) is acceptable in practice. To reflect weaker notions of identifiability, we let ~~ denote an equivalence relation on θ such that model can be said be identifiable up to ~, or --iidentifiable
Definition equiivalaal,, Let fo S X' denote mapping from slot representation space S to image space X' (satisfying Assumption 8), the equivalence relation ~ w.r.t. to parameters O e θ defined bellow, where P P S {0, 1] RXX is slot permutation matrix, H RRdd is an affine transformation matrix, and R.:
Vx E X,
01 ~ O2 + S. (x) P(To
P( (x)
Lemma 1 (Aggregate Posterior Mixture). Given that probabilistic slot attention induces a local (per-datapoint x E {=) GMM with K components, the aggregate posterior qzz) obtained by marginalizing out x is non-degenerate global Gaussian mixture with M K components:
((z) M MΣΣs N (z: (.)
(10)
Proof Sketch. The full proof is given in Appendix B. The result is obtained by integrating the product of the latent variable posterior density q(z x) and the (local) GMM density given z, w.r.t. x. We then proceed by verifying that the mixing coefficients sum one over all the components in the new mixture (Corollary 4), proving aggregated posterior to be a well-defined probability distribution, this q(z) in our identifiablity result.
can be empirically confirmed in Figure 3. Next,
Theorem (Mixture Distribution of Concatenated Slots). Let S& denote a permutation equivariant PSA function such that [a(z, Pst) (fa(z,,'), where P E (0, 1] K is an arbitrary permutation matrix. Let = (s1....,8k) E Rdd be random variable defined as the concatenation of K individual slots, where each slot is Gaussian distributed within a K-component mixture: Sş ~ Npp, Σx) E RRd Vk E (1,... ..). Then, s is also GMM distributed with K! mixture components:
KI
p(s) Σ n Ns: Mp: Ep), where T K My E RR Σp E RkdxKd p=1
(11)
Proof Sketch. Sketch. The proof in Appendix We observe that the permutation equivariance of the PSA function Ju induces K! ways of concatenating sampled slots sg. where each permutation maps to different mode with block diagonal covariance structure in a GMM living in Rd (e.g. Fig. 6, 7).
Theorem 2 (dentiiiteee Slot Representations). Given that the aggregate posterior q(z) is an optimal, non-degenerate mixture prior over slot space (Lemma 1), f: X' piecewise affine weakly injective mixing function (Assumption 8), and the slot representation, s = (Si,.... .Sk) can be observed as a sample from a GMM (Theorem 1), then p(s) is identifiable as per Definition 3.
Proof Sketch. The proof is given in Appendix D. Lemma and Corollary 4 state that the optimal latent variable prior in our case is GMM distributed, non-degenerate and equates the aggregate posterior (z). This permits us to extend [43]'s result to show that if q(z) is distributed according to non-degenerate GMM and the mixing function l'a is piecewise affine and weakly injective, then the slot distribution representation, p(s) which is also GMM (Theorem 1) is identifiable up to an affine transformation and arbitrary slot permutation.
Corollary (Individual Slot Identifiabiliity. If the distribution over concatenated slots p(s), where s (Si...... 5K) E Kd is ~.-identifiable (Theorem 2) then this implies g(z) is identifiable up to an affine transformation and permutation of the slots Sy- Therefore, each slot distribution Sy N[H,, Σx) E Rd, Yk E [1.... K} is also identifiable up to an affine transformation.
7

---

Run #1
Run #2
Run #3
Run #4
Run #5
22
22
222
72
72
Figure 4: Aggregate posterior identifiability. Recovered (latent) aggregate posteriors g(z) across runs of our PSA model. As detailed in Section used 2D synthetic dataset with 5 total 'object' clusters, with each observation containing at most 3. This provides strong evidence of recovery of the latent space up affine transformations, empirically verifying our identifiability claim.
MONET
MONIT
SA NoA
SA NoA
104
PSA [Ours]
Shall Meusn Coreiaal om Covelicient
PSA (Duns)
PSA/Pig (Ours)
Compositionl ak
PSA Prij [Ours]
Slot Score
PSA-NiA (Ours)
PSA-KnA (Dund
Composttonal
24
(Dun))
2.0
20
VONET
MONET
MBA (Don))
PSA
Figure 5: Experiments comparing slot-identifiabiiity scores. Colour coding represents the level of slot identifiability achieved by each model as measured by the SMCC (left) and SIS (right). SMCC offers more consistent metric correlating with reconstruction loss.
6
Experiments
Given that the focus of this work is theoretical, the primary goal our experiments is to provide strong empirical evidence of our main identifiability result (ref. Figures 4, 5). With that said, we also extend our experimental study to popular imaging benchmarks to demonstrate that our method scales to higher-diimensonal settings (ref. Tables 2, 4).
Datasets & Evaluation Metrics. Our experimental analysis involves standard benchmark datasets from object-centric learning literature including SPRITEWOR WORLD [6], CLEVR [34], and OBJECT- SRoOM [35]. We report the foreground-adjusted rand index (FG-ARI) and FID [23] to quantify both object-level binding capabilities and image quality. Our main goal is to measure slot-identifiability so we use the slot identifiability score [6] and the mean correlation coefficient (MCC) across slot representations
call the latter Slot-MCC (SMCC). For two sets of slots {s. JMM and {š, J.M where s; IR xd š; E IR xxd extracted from M images {x; J. the SMCC between any s and š obtained by matching the slot representations and their order. The order is matched by mapping slots in & w.r.t assigned by followed by learned affine mapping A between aligned (i) and
K
SMCC s) = K
plsij. AS As,0))
(12)
x
1= 1=
For more details on the metrics please refer to Appendix F.
Models & Baselines. We consider three ablations on proposed probabilistic slot attention (PSA) method: (i) PSA base model (Algorithm 2): (ii) PSA-PROT model (Algorithm 1); and (iii) PSA-NoA model, which is equivalent to PSA-PROI but without additive decoder. We experiment with two types of decoders: (i) an additive decoder similar to [79]'s spatial broadcasting model; and (ii) standard convolutional decoder. In all cases, use LeakyReLU activations to satisfy the weak

---

Table 2: Comparing slot identifiability scores (SMCC and slot aver- aged R2) with existing object-centric learning methods.
METHOD
CLEVR
OBJECTS ROOM
SMCC t
R2
SMCC
R2 f
AE
0.43 .02
0.26 + 02
0.46 + .05
0.45 + .06
MONET
0.32 -01
0.39 - .09
0.43 + .04
0.41 - 10
(4
SA
0.56 .02
0.55 + .05
0.66 = .01
0.54 .00
SA-NoA
0.23 .03
0.24 .02
0.48 + .02
0.47 + .01
Ours:
PSA-NoA
0.56 .05
0.42 + .06
0.44 + .03
0.45 + .05
PSA
0.58
.06
0.48 + 02
0.66 + .01
0.64 + .02
PSA-PROJ
0.66 1 .06
0.62 1 00
0.71 1 .00
0.62 1 .02
Figure 6: Concatenated Slot Gaussian Mixture. Example of the higher-dim GMM (ID + 3D) induced by the permutation equivariance of PSA and the K! ways of concatenating sampled slots.
injectivity conditions (Assumption 8). In terms of object-centric learning baselines, we compare with standard additive autoencoder setups following [6], slot-attention (SA) [53], and MONET [9].
Verifying Slot Identifiability: Gaussian Mixture of Objects. To provide conclusive empirical evidence of our identifiability claim (Theorem 2), we set a synthetic modelling scenario under which it is possible to visualize the aggregate posterior across different runs. The goal is show that PSA is ~ ---identifeable (Theorem 2) in the sense that it can recover the same latent space distribution up to an affine transformation and slot order permutation. For the data generating process, we defined K= component GMM, with differing mean parameters (M),.... (M155..,,55)... (45), isotropic covariances. The 5 components emulate different object types in a given environment. To create single data point, we randomly chose 3 of the 5 components and sampled 128 points uniformly at random from each mode. Figure shows some data samples, where different colours correspond to different objects. We used 1000 data points in total for training our PSA model. As shown in Figure 4, the aggregate posterior is either rotated, translated, skewed, or flipped across different runs predicted by our theory- this contrasts with all baselines wherein the aggregate posterior is intractable. We observed an SMCC of 0.93 = 0.04, and R2-score of 0.50 + 0.08.
and shared
Case Study: Imaging Data. Although our focus is primarily theoretical, we now demonstrate that our method generalizes/scales to higher-dimensional imaging modalities. To that end, we first use the SPRITEWORLD [78] dataset to evaluate the SMCC and SIS w.r.t. ground truth latent variables. Figure presents our identifiability results against the baselines. Similar to [6], we observe higher SIS when compositional contrast and reconstruction error decreases. However, when the mixing function is not additive, the compositional contrast does not decrease drastically while maintaining higher SMCC and SIS this verifies our identifiability claim using only piecewise affine decoders. As shown in Figure 5, we also observe that PSA routing with additive decoder models achieves lower compositional contrast and reconstruction errors when compared with other methods. This indicates that stronger identifiability of slot representations is achievable when combining both slot latent structure and inductive biases in the mixing function. Unlike for the SPRITEWOR WOLL dataset, all the ground truth generative factors are unobserved for the CLEVR and OBJECTSROON datasets we use. Therefore, for evaluation in these cases, train multiple models with different seeds and compute the SMCC and SIS measures across different runs. This is similar to our earlier synthetic experiment and standard practice in identifiability literature. Table presents our main identifiability results on CLEVR and OBJECTSROOM, We observe similar trends in favour of our proposed PSA method as measured by both SMCC and (slot averaged) R2 score relative to the baselines.
Complex Decoder Structure. To empirically test slot identifiability using more complex non-additive decoders, we used transformer decoders following SLATE [67], and simply replaced the slot attention module with probabilistic slot attention. On the CLEVR dataset, we observed significantly improved SMCC of 0.73 + .01 and R2 of 0.55 + .06 relative to Table 2.
Case Study:

---

To demonstrate that probabilistic slot atten- tion can scale to large-scale real-world data we ran additional experiments on the Pas- cal VOC2012 2 [49] dataset, following the exact DINOS AUR strategies and setups de- scribed in [66, 36] for fairness, then sim- ply swapping out the slot attention module with probabilistic slot attention. As shown in Table 3, we find that probabilistic slot attention is competitive with standard slot attention on real-world data. We also tested more complex, non-additive decoders based on autoregressive transformers, following the DINOS AUR [66] setup. For our PSA TRANSFORMER (w/ DINO) model, we ob- served an MBO; of 0.447, and MBO, of 0.521 which is competitive with the DI- NOSAUR TRANSFORMER baseline [66] of 0.44 and 0.512 respectively. In this case, we found that a lower maximum learning rate of 10 was beneficial for stabilizing PSA training. In summary, our experiments corroborate our theoretical results and suggest why non-additive decoder structures can still work well given the appropriate latent structure and inference procedures are in place. With that said, there is trade-off between identifiability and expressivity induced by the choice of decoder structure [47], so depending on the use case, it may indeed be advantageous to combine both latent and additive decoder structures in practice.
Table 3: Pascal VOC2012 benchmark results using probabilistic slot attention (PSA), All baselines are standard results from [54]. SA MLP (w/ DINO) de- notes our replication of the DINOSAUR MLP baseline from [66], whereas (E) denotes the use of slot attention masks rather than decoder alpha masks for evaluation.
MODEL
MBO.t
MBO.T
BLOCK MASKS
0.247 - .000
0.259 1 .000
SA
0.222 = .008
0.237 + .008
SLATE
0.310 .004
0.324 = .004
ROTATING FEATURES
0,282 = .006
0.320 .006
DINO K-MEANS
0.363 = .000
0.405 + .000
DINO CAE DINOSAUR MLP
0.329 = .009
0.374 + .010
0.395 = .000
0.409 + .000
Ours:
SA MLP (W/ DINO)
0.384 = .000
0.397 + .00
SA MLP (W/ DINO)*
0.400 = .000
0.415 - 000
PSA MLP (W/ DINO)
0.389 = .009
0.422 1 .009
PSA MLP (W/ DINO)
0.405 + .110
0.436 011
7 Discussion
Understanding when object-centric representations can theoretically be identified is important for scaling slot-based methods to high-dimensional images with correctness guarantees. In contrast with existing work, which focuses primarily on properties of the slot mixing function, we leverage distributional assumptions about the slot latent space prove a new dottiidenifiaaiiityy result. Specif- ically, we prove that object-centric representations are identifiable without supervision (up to an equivalence relation) under mixture model-like distributional assumptions on the latent slots. To that end, we proposed a probabilistic slot-attention algorithm that imposes an aggregate mixture prior over slot representations which is both demonstrably stable across runs and tractable to sample from. Our empirical study primarily verifies our theoretical identifiability claim and demonstrates that our framework achieves the lowest compositional contrast without being explicitly trained towards that objective, which is computationally infcasible beyond toy datasets. In summary, we show how slot identifiability can be achieved via probabilistic constraints on the latent space and piecewise decoders. These piecewise decoders manifest as c.g. standard MLPs with LeakyReLU activations and are generally less restrictive than additive decoders. When coupling probabilistic and additive decoder structures, we observe further performance improvements relative to either one in isolation.
Limitations & Future Work. We recognize that our assumptions, particularly the weak injectivity of the mixing function, may not always hold in practice for different types of architectures (see Appendix C for sufficiency conditions). Although generally applicable, the piecewise-affine functions we use may not always accurately reflect valid assumptions about real-world problems, e.g. when the model is misspecified. Like all object-centric learning methods, we also assume that the mixing function is invariant to permutations of the slots in practice, which technically makes it non-invertibl We deem this aspect to be an interesting area for future work, as an extension to accommodate permutation invariance would strengthen and generalize the identifiability guarantees we provide. Additionally, we do not study cases where objects are occluded, i.e. when are shared or bordering other objects. This limitation is not unique to our work [53, 6, 14, 15, 44] and overcoming it requires further investigation by the community. Nonetheless, our theoretical results capture the important concepts in object-centric learning and represent valuable extension to the nascent theoretical foundations of the area. In future work, would be valuable to further relax slot identifiability requiremennt//asuumpiios and study slot compositional properties of probabilistic slot attention.
10

---

Acknowledgements
A. Kori supported by UKRI (grant number EP/S023356/1), as part of the UKRI Centre for Doctoral Training in Safe and Trusted AL B. Glocker and F.D.S. Ribeiro acknowledge the support of the UKRI Al programme, and the Engineering and Physical Sciences Research Council, for CHAI- EPPRC Causality in Healthcare Al Hub (grant number EP/Y028856/1).
References
[1] Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Prucessing Systems, 35:15516- 15528, 2022. [Cited page 3.]
[2] Kartik Ahuja, Yixin Wang. Divyat Mahajan, and Yoshua Bengio. Interventional causal repre- sentation learning. arXiv preprint arXiv:2209.11924 2022. [Cited on page 2.]
[3] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sonzaez-G Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.0126/ 2018. [Cited on page 1.]
[4] Timothy EJ Behrens, Timothy H Muller, James CR Whittington, Shirley Mark, Alon Baram, Kimberly L Stachenfeld, and Zeb Kurth-Nelson. What is cognitive map? organizing knowl- edge for flexible behavior. Neuron, 100(2):490-509, 2018. [Cited on page 1.]
[5] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798- 1828, 2013. [Cited on pages and 2.]
[6] Jack Brady, Roland Zimmermann, Yash Sharma, Bernhard Schölkopf, Julius Kügel-- gen, and Wieland Brendel. Provably learning object-centric representations. arXiv preprins 2rXhv:2305.142229 2023. [Cited on pages 1, 2, 4, 8, 9, 10, 17, 21, and 24.]
[7] Johann Brehmer, Pim Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319-38331, 2022. [Cited on page 2.]
[8] Simon Buchholz, Michel Besserve, and Bernhard Schölkopf. Function classes for identifiable nonlinear independent component analysis. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. [Cited
on page 2.]
[9] Christopher Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint arXiv: 11390, 2019. [Cited on pages and 9.]
[10] Michael Chang, Thomas I. Griffiths, and Sergey Levine. Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation. arXiv preprint arXiv:2207.00777. 2022. [Cited on page 1.]
[11] Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv: /6/ 1.02648, 2016. [Cited on pages 2 and 25.]
[12] Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018. [Cited on page 2.]
[13] Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. Advances Neural Information Processing Systems, 35:28940-28954 2022. [Cited
on page 2.]

---

[14] Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan. Slot order matters for com- positional scene understanding. arXiv preprint arXiv:2206.01370, 2022. [Cited on pages and 10.]
[15] Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gener- ative scene inference and sampling with object-centric latent representations. arXiv preprini 1XXiv:997.110533 2019. [Cited on pages 1, 2, and 10.]
[16] Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. Advances in Neural Information Processing Systems, 34:8085-8000 8094, 2021. [Cited on pages and 2.]
[17] Russell A Epstein, Eva Zita Patai, Joshua B Julian, and Hugo J Spiers. The cognitive map in humans: spatial navigation and beyond. Nature neuroscience, 20(1 (0(11):1104-10 15:1154-1513, 2017. [Cited on page 1.]
[18] Fabian Falck, Haoting Zhang, Matthew Willetts, George Nicholson, Christopher Yau, and Chris C Holmes. Multi-facet clustering variational autoencoders, Advances in Neural Informa- tion Processing Systems, 34:8676-8690, 2021. [Cited on page 2.J
[19] Tobias Gerstenberg. Noah D Goodman, David Lagnado, and Joshua Tenenbaum. A counterfactual simulation model of causal judgments for physical events. Psychological review, 128(5):936, 2021. [Cited on page 1.]
[20] Alison Gopnik, Clark Glymour, David M Sobel, Laura Schulz, Tamar Kushnir, and David Danks. A theory of causal learning in children: causal maps and bayes nets. Psychological review, 111(1):3, 2004. [Cited on page 1.]
[21] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object repre- sentation learning with iterative variational inference. In International Conference on Machine Learning, pages 2424-2433. PMLR, 2019. [Cited on page 2.]
[22] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05200 2020. [Cited on pages and 3.J
[23] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017. [Cited on page 8.]
[24] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations,
2017. [Cited on page 2.]
[25] Geoffrey Hinton. Some demonstrations of the effects of structural descriptions in mental imagery. Cognitive Science, 3(3):231-250, 1979. [Cited on page 1.]
[26] Geoffrey Hinton. How to represent part- whole hierarchies in neural network. Neural Computation, pages 1-40, 2022. [Cited on page 3.]
[27] Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In International conference learning representations, 2018. [Cited on page 3.]
[28] Matthew D Hoffman and Matthew Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1, 2016. [Cited on pages 18 and 25.]
[29] Hyvarinen and E Oja. Independent component analysis: algorithms and applications. Neural Networks, 13(45)):411 4300, 2000. [Cited on pages and 2.]
[30] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances neural information processing systems, 29, 2016. [Cited on page 2.]
12

---

[31] Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429 439, 1999. [Cited on pages 1 and 2.]
[32] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Tumnr. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019. [Cited on pages 2, 6, and 7.]
[33] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In Proceedings of the Twenty-Second International Conference Artificial Intelligence and Statistics, volume 89, pages 859-868. PMLR, 2019. [Cited on page 2.]
[34] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017. [Cited on page 8.]
[35] Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. https//gittubbco//eeenmndddatt object-datasets/, 2019. [Cited on page 8.]
[36] loannis Kakogeorgiou, Spyros Gidaris, Konstantinos Karantzalos, and Nikos Komodakis. Spot: Self-training with patch-order permutation for object-centric learning with autoregressive transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22776-22786, 2024. [Cited page 10.]
[37] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020. [Cited on pages 2, 6, 7, 17, and 24.]
[38] Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identi- fiable conditional emergy-based deep models based on nonlinear ica. In Advances in Neural Information Processing Systems, volume 33, 2020. [Cited on pages 2, 17, and 24.]
[39] lles Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Iden- tifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 33:12768 12778, 2020. [Cited on page 24.]
[40] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2018. [Cited on page 2.]
[41] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv: 14, 2013. [Cited on pages 2 and 6.]
[42] Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprint arXiv:2111.12594 2021. [Cited on page 2.]
[43] Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 35:15687-15701, 2022. [Cited on pages 2, 6, 7, 17, 21, and 22.]
[44] Avinash Kori, Francesco Locatello, Fabio De Sousa Ribeiro, Francesca Toni, and Ben Glocker. Grounded object centric learning. arXiv preprint arXiv:2307.09437, 2023. [Cited on pages 1, 2, 6, and 10.]
[45] Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolu- tional inverse graphics network. Advances in neural information processing systems, 28, 2015. [Cited on page 1.]
[46] Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies. arXiv preprint 241:02401.04490, 2024. [Cited page 2.J
13

---

[47] Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien. Additive decoders for latent variables identification and cartesian-produet extrapolation. Advances in Neural Information Processing Systems, 36, 2024. [Cited on pages 1, 3, 4, 6, 10, and 17.]
[48] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and bruin sciences, 40:e253, 2017. [Cited on page 1.]
[49] Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Fu. Tell me where to look: Guided attention inference network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2018. [Cited page 10.]
[50] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint arXiv:2001.02444 2020. [Cited on page 2.]
[51] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14- 4124. PMLR, 2019. [Cited on pages 1, 2, and 6.]
[52] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pages 6348 6359. PMLR, 2020. [Cited on page 2.]
[53] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:11525-11522 2020.
[Cited on pages 1, 2, 3, 4, 9, and 10.]
[54] Sindy Löwe, Phillip Lippe, Francesco Locatello, and Max Welling. Rotating features for object discovery. Advances in Neural Information Processing Systems, 36, 2024. [Cited on page 10.]
[55] Amin Mansouri, Jason Hartford, Kartik Ahuja, and Yoshua Bengio. Object-centric causal representation learning. In NeurlPS 2022 Workshop on Symmetry and Geometry in Neural Representations, 2022. [Cited on page 3.]
[56] Amin Mansouri, Jason Hartford, Yan Zhang, and Yoshua Bengio. Object-centric architectures enable efficient causal representation learning. arXiv preprint arXiv:2310. 19054, 2023. [Cited on page 3.]
[57] Gary F Marcus. The algebraic mind: Integrating connectionism and cognitive science. MIT press, 2003. [Cited on page 1.]
[58] Joe Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. In International Conference on Machine Learning, pages 3403-3412. PMLR, 2018. [Cited on page 2.]
[59] Emile Mathieu, Tom Rainforth, Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. In Proceedings of the 36th International Conference Machine Learning, 2019. [Cited on page 2.]
[60] Radford M. Neal. Bayesian Learning for Neural Networks. Springer-Verlag Berlin, Heidelberg, 1996. [Cited on page 6.]
[61] Fabio De Sousa Ribeiro, Kevin Duarte, Miles Everett, Georgios Leontidis, and Mubarak Shah. Learning with capsules: A survey, arXiv preprint arXiv:2206.0224 2022. [Cited on page 3.]
[62] Fabio De Sousa Ribeiro, Georgios Leontidis, and Stefanos Kollias. Capsule routing via variational bayes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3 3756, 2020. [Cited on page 3.1
[63] Fabio De Sousa Ribeiro, Georgios Leontidis, and Stefanos Kollias. Introducing routing uncer- tainty in capsule networks. In Advances Neural Information Processing Systems, volume 33, pages 6502, 2020. [Cited on page 3.]
14

---

[64] Irvin Rock. Orientation and form. 1973. [Cited on page 1.]
[65] Bernhard Schölkopf and Julius von Kügelgen. From statistical to causal learning. Proceedings of the International Congress of Mathematicians, 2022. [Cited on page 1.]
[66] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl- Johann Simon-Galee Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap to real-world object-centric learning. International Conference on Learning Representations, 2023. [Cited on pages 1, 2, and 10.]
[67] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns compose. International Conference Learning Representations, 2022. [Cited on pages 1, 2, and 9.]
[68] Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural block-slot representations. arXiv preprint arXiv:2211.01 177, 2022. [Cited on page 2.]
[69] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. Advances in Neural Information Processing Systems, 35:18181-18198 2022. [Cited on page 2.]
[70] Joshua B Tenenbaum, Charles Kemp, Thomas L Griffiths, and Noah D Goodman. How to grow a mind: Statistics, structure, and abstraction. science, (31(622))1279--2255 2011. [Cited on page 1.]
[71] Jakub Tomezak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics, pages 1214-1223, PMLR, 2018. [Cited page 25.]
[72] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. [Cited on page 25.]
[73] Sjoerd Van Steenkiste, Karol Kurach, Jürgen Schmidhuber, and Sylvain Gelly. Investigating object compositionality in generative adversarial networks. Neural Networks, 130:309-325, 2020. [Cited on page 2.]
[74] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in newral information processing systems, 30, 2017. [Cited on pages 3 and 5.]
[75] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451-16467, 2021. [Cited page 2.]
[76] Yanbo Wang, Letao Liu, and Justin Dauwels. Slot-vae: Object-centric scene generation with slot attention. arXiv preprint arXiv:2306. 06997, 2023. [Cited on pages and 6.]
[77] Yanbo Wang, Letao Liu, and Justin Dauwels. Slot-VAE: Object-centric scene generation with
slot attention. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 36020-3600 30035. PMLR, 23-29 Jul 2023. [Cited on pages 4 and 25.]
[78] Nicholas Watters, Loic Matthey, Sebastian Borgeaud, Rishabh Kabra, and Alexander Lerchner. Spriteworld: A flexible, configurable reinforcement learning environment. http://ggthhhhccommeeeepmmddpppp 2019. [Cited on page 9.]
[79] Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint /001::/00000000111110 2019, [Cited on page 8.]
[80] Matthew Willetts and Brooks Paige. don't need u: Identifiable non-linear ica without side information. arXiv preprint wwXiv:2106.052..0 2021. [Cited page 2.]
15

---

[81] Xiaojiang Yang. Yi Wang. Jiacheng Sun, Xing Zhang, Shifeng Zhang, Zhenguo Li, and Junchi Yan. Nonlinear ICA using volume-preserving transformations, In International Conference on Learning Representations, 2022. [Cited on page 2.]
[82] Dingling Yao, Danru Xu, Sébastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability. 2024. [Cited page 2.]
[83] Alan Yuille and Daniel Kersten. Vision bayesian inference: analysis by synthesis? Trends in cognitive sciences, 10(7):301-308, 2006. [Cited on page 1.]
[84] Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979-12990. PMLR, 2021. [Cited on page 2.]
16

---

On Sg
23- Alasoti
13-
Figure 7: Concatenated Slot Gaussian Mixture. Examples of the higher dimensional Gaussian mixture induced by the permutation equivariance of slot attention and the K! ways of concatenating sampled slots. Here we start with a random 1D GMM with K = 3 modes, each representing a different slot distribution, which then induces respective 3D GMM with K! 6 modes.
A List of Assumptions
Assumption 1 (B- Disentanglement, [47]). Let s = {Sp, VBE E B} be a set of features wrt partition set B. The learned mixing function fa is said to be B disentangled wrt true decoder fa if there exists a permutation respecting diffeomorphism uB = fa+ fa B E B which for a given feature s can be expressed Uy (s) (sB)-
Assumption 2 (Additive Mixing Function). A mixing function fa : S X, is said to be additive if there exist partition set B and functions f + R |2 such that: fa(s) = EBEB f (sg).
Assumption 3 (Irreducibility). Given an an object X E x, a model is considered as irreducible if any subset of an object, y S X5 is not functionally independent of the complement of the subset contained within the object, y n x as expressed by the Jacobin rank inequality in equation 5 in [6].
Assumption 4 (Compositionality) Compositionality as defined by is a structure imposed on the slot decoder fa which implies that each image pixel is function of at most one slot representation, thereby enforcing a local sparsity structure on the Jacobian matrix of fa [6].
guaranteed by explicitly minimising the compositionality contrast but was empirically observed to be implicitly satisfied in the case of additive decoders [6]. Later, [47] showed this as the property of additive decoder models. However, the additive decoders studied by [47] are not expressive enough represent the "masked decoders" typically used in object-centric representation learning, which stems from the normalization of the alpha masks. This means some care must be taken in extrapolating the results in [47] the models we use in practice. Additionally, additive decoders scale linearly in the number of slots K, so some less significant scalability issues remain relative to state-of-the-ard non-additive decoders (e.g. using Transformers).
Remark 2. The compositionality
Assumption 5 (u task). Conditioning latent variables on an observed variable to yield identifiable models. The main assumption is that conditioning on a (potentially observed) variable u renders the latent variables independent of each other [37].
Assumption 6 (Object Sufficiency). A model is said to be object sufficient iff there are no additional objects the original data distributions other than the ones expressed in training data.
Assumption (Decoder Injectivity). The function fa S + X mapping from slot space image space is non-linear piecewise affine injective function. That is, it specifies a unique one-to-one mapping between slots and images.
Remark 3. In practice, we monotonically increasing decoder with leakyReLU activation which should encourage injectivity behaviour [38, 37].
Assumption 8 (Weak Injectivity [43]). Let f Z - X mapping between latent space and image space, where dimn () < dim((2). The mapping fa is weakly injective if there exists x E X and > 0 such that |f ({x}) 1, V x E B(xg. 6) n f(2), and {x E X x |= oo) C f(z) has measure zero w.r.t. to the Lebesgue measure on f(2).
Remark 4. In words, Assumption 8 says that mapping fa is weakly injective if: (i) in small neighbourhood around a specífic point Xo E X' the mapping is injective meaning each point in this neighbourhood maps to exactly one point in the latent space 2; and (ii) while Sa may not be globally injective, the set of points in X' that map back to an infinite number of points 2 (non-injective points) is almost non non-existent in terms of the Lebesgue measure on the image of Z under fa-
17

---

B
Aggregate Posterior Proofs
Lemma 1 (Aggregate Posterior Mixture) Given that probabilistic slot attention induces local (per-datapoint x E {x, ) GMM with K components, the aggregate posterior g(z) obtained by marginalizing out x non-degenerate global Gaussian mixture with MK components given by:
K TgN Pike Rie)
((z)
(13)
M
Proof. We begin by noting that the aggregate posterior ((z) is the optimal prior p(z) so long as posterior approximation g(z x) is close enough to the true posterior p(z x), since for a dataset
(x: (ii we have that:
/ p(z x)p(x))xx
p(z)
(14)
Exxp(xx [p(z)
x)]
(15)
M
2
p(z xx)
(16)
M
t=1
M
2
(( x:)
(17)
M
(z).
(18)
where we approximated p(x) using the empirical distribution, then substituted in the approximate posterior and marginalized out x. This observation was first made by [28] and we use it to motivate our setup.
In our case, probabilistic slot attention (Algorithm 1) fits (local) GMM to each latent variable sampled from the approximate posterior: z q(z x:). for i I,..., M. Let f(z) denote the (local) GMM density, its expectation is given by:
/ʃ p(x)g(z) (x)(z)
e(ex)) [s(z)]
(19)
M 5(x xx x,)a(z (2)ddx
(20)
M
i=1
1 M
/
=
M Σ
g(z (z)d
(21)
M
K
=
N ((x)), o²)))) (x Σ
TeN (z; prkks dit) dz
(22)
M
k=1
M
K
8(z - u(xx))
Σ
TeN (z; pilss oih) dz
(23)
M
i=1
km1
M
K
TWN (u(x;); (aa o.eo
(24)
=: q2),
(25)
where we again used the empirical distribution approximation of p(x), and the following basic identity of the Dirac delta simplify: f 5(x xx)[(x)xx f(x').
For the general case, however, we must instead compute the product of g(z x) and f(z) rather than use Dirac delta approximation as in Equation 23. To that end we may proceed as follows w.r.t. to
18

---

each datapoint x,:
K
O x:) f(z) ((x o (xx)) Σ
TaN (z; pikks Rib)
(26)
tekk [V (z; prrks oite (
(27)
k=1
R&N (z: flite x
(28)
where the posterior parameters of the resulting mixture are given in closed-form by:
1 iik
[ux;) (x:)
Mikk
ik
Puu oik
(29)
(x
oik
which are the standard distributional parameters obtained from product of two Gaussians.
For the updated mixture coefficients 7u. propose principled way include posterior-weighted contribution of each mode to the new mixture coefficients. First, note that (TTi,, T422 Tik) are
parameters of multinomial distribution 2k=1 Ti for each datapoint x, Since the Dirichlet distribution is the conjugate prior of the multinomial distribution, we can place a Dirichlet prior over the mixing coefficients for each datapoint, then update it to posterior using the data. Concretely, we place a symmetric Dirichlet prior over the mixing coefficients as follows:
K
a1 T2: ..... kik Dirichlet (a4,1 0........ a.k), for i= 1,2, M,
(30)
where o; R the concentration parameters of the 7th Dirichlet distribution, and Vi, k Ogi
indicating uniformity over the open standard (K 11) >>-mmpleex To compute the posterior Dirichlet distribution we calculate 'pseudo-counts' by integrating the product the posterior density g(z x,) with each one of the K modes of the Gaussian mixture, thereby measuring a posterior-weighted contribution of each mode & to the new aggregate mixture:
Cik -
(Z: pak, ox))) (N(z (x:)) dz,
for 1 1,2..... M,
(31)
which we then pseudo-counts to compute the Dirichlet posterior:
(ft1, T2..................... ..., Axk (ca1 C2. .2,..,, GK) ~ Dirichlet (am, + C11, 042 + c2...., Q.K + C.K)
(32)
for = M. Each posterior probability is then readily given by the mean estimate
K
Olik + Gsk
n
= 1.
(33)
K Σド
(aij + Cig)
A=1
Putting everything together, the aggregated posterior is therefore given by:
q(z)
M Σ
RuN (z; (z; pirks R
where
2~ q(z x),
(34)
which concludes the proof.
Corollary 4. The aggregate posterior q(z) is a non-degenerate Gaussian mixture, in the sense that it is a well-defined probability distribution, as the updated mixture coefficients sum I over the number of components M x K.
Proof. Recall from Lemma that the aggregate posterior q(z) obtained by marginizaling out from probabilistic slot attention model is a mixture distribution of M x K components with the following parameters:
{ Tuk Pik. 03.
for i= 1... M, and k = 1,2,..... K.
(35)
19

---

To verify that (()) is non-degenerate mixture, we observe the following implication:
K
Tak
1,
for i= M,
(36)
k=1
due to the Dirichlet posterior update in Equation 33, and therefore
M K
M
1 M
Ri
11
M
(37)
M
M
k=1
M
K
R =1
(38)
M
which says that the scaled sum of the mixing proportions of all components in M GMMs must equal 1, proving that the associated aggregate posterior mixture ((z) is well-defined probability distribution.
Theorem (Mixture Distribution of Concatenated Slots). Let fa denote a permutation equivariant probabilistic slot attention function such that fs Ps*) Pf, (2, (²,s where P {0, _]xx is an arbitrary permutation matrix. Let (81, ................................ ,,k,,K) E IR Rdd be random variable defined as the concatenation of individual sampled slots, where each slot is Gaussian distributed within K Kc-compoent mixture: Sş (sss; M4, Σx) Rd, Ak E KK.. Then, it holds that is also Gaussian mixture distributed comprising K! mixture components:
K! Σ
p(s)
Ns: ppr Ep), where A
Mp E Rkd E, E Kdxkd,
(39)
Proof. Each slot Sy ~ (uux Ex) is sampled independently from s, ~ (N,,, E,), for any j A meaning they are conditionally independent given the latent mixture component assignment. Thus, the concatenated slots variable s = (81, . 8x) ), can be described by a K Kdddimensional multivariate Gaussian distribution with block diagonal covariance structure follows:
S+)1)
(a( (2)
Σx(1)
8x(2)
E Σx(2)
N
(40)
(x()))
[M((K)
Σ{(K)-
where π : [K] > [K] is a permutation function of the set [K] == f1, K}. Since the slot attention function ʃs permutation equivariant, there exist K! possible ways concatenate K slots, and each permutation induces a mode within a Gaussian mixture living in kdd space. Since each permutation of the slots is equally likely, the mixture coefficients given by:
π (T1, . TKI),
where
Tp
Vp E {1,2, ..., ..,
(41)
K!
R!
1,
(42)
which concludes the proof.
Remark Based on the above result, evident that concatenating > unique slots can be viewed as sample from GMM with K! components. Constructing scene requires at least two unique slots, one for the background and one for object, thus supporting our theory regarding slot composition.
20

---

C Injective Decoders: Sufficient Conditions
In this section, we provide a theoretical overview of the decoder architecture we use and offer sufficient conditions for a leaky-ReLU decoder to be weakly injective in the sense of Assumption 8 as shown and adapted from [43].
decodee
Definition (Piece-wise Decoders, [43]). Let = {s), $2,.... SK} denote a given set sampled ž; in each mixture component (K slots) in the GMM, P(Z). Let IR + R denote the leaky- Saky-ReLLU activation function, and let m = Rg. R1, 12.............................. 74 = n and H (n1, m2) denote the set of full full-rank . We consider piece- wise functions mapping each slot representation s E S RT x K to an image x E X E R" in the output space, Fmnk :S - N, of the form below:
affine functions h; I= + "
F Flamm
h4 0 1 1000.000000
ooh wo...o...ooooooooooooooooo...... h2 E mi)
(43)
The following Corollary, corollary and proofs are adapted from [43].
Corollary 5. Given fa E Fm- where fa = h4 oao 1 o............................... . o h1, fa is injective.
Proof. Each affine function h; has full column rank and is therefore both injective and invertible. Since the activation function a is also injective, we get that fa is injective and invertible.
Corollary 6. Let fa = h4 oa h w........oo...o................. oo Fm where no n1 nk Given by is affine and invertible, then for almost all x E fa(R.) there exists 8 such that Sa is dell-de affine function, h2 on ((x,8) O fa(
Proof. We know fa is piecewise affine function which is invertible. Therefore, it simply follows, vy E B(x, d) there exists such that f is an affine function in the domain B(x, 6). One can therefore deduce there exists some affine function h, where fa = h in the domain (xx, 6). 0
D Slot Identifiability Proof
Definition 5 (Slot Identifiability [6]). Given a diffeomorphic ground truth function ʃ S - X, and inference model 9 X + S, 9 correctly slot identifies every object X, E X with the ground slot Sy E S via 8₁ = (sx)) if there exists unique slot & E [K] for all x, E X, and there exists an invertible diffeomorphism, or in our case an affine transformation, such that Sy = h(S;)) for all
Sy S.
Remark 6. [6] established that compositionality (Assumption 4) and irreducibility (Assumption 3) of the decoder fa is required for slot identifiability in the sense of Definition 5.
Summary & Intuition. The following theorems and proof extend the identifiability results of [43] to slot-attention models, we include all the proof's and details for the sake of completion. In this work, we do not consider the irreducibility criteria in [6] and define slot identifiability only by an injective mapping of each slot to subspaces representing objects in a scene to satisfy compositionality without the use of computationally heavy methods such as additive decoders and compositional contrast. We to affine transformation by passing concatenation
show identifiability of each slot representation of samples from each mixture component which represents slots through piecewise function in Definition 4 representing the decoder, fa- The trick in this proof lies in our observation of the fact that a concatenation of samples from each slot mixture component is a sample from high dimensional GMM p(s), highlighted in Thcorem 1. We then use the identifiability results of [43] to show (s) identifiable up to affine transformation. This then implies identifiability up to affine transformation of the aggregate posterior q(z), non-degenerate GMM by Lemma where sample from a mixture component in (z)) represents an individual slot representation. This contrasts with [43] which does not consider identifiability of the aggregate posterior and its mixture components in the context of slot representation learning.
In order to proceed, we begin by stating three key theorems defined and proven in the work of [43] which are essential for our slot identifiability proof. First, we restate the definition of a generic point outlined by [43] below.
21

---

Definition 6. A pointx x E fa(R")) S R" is generic if there exists 6 > 0, such that fa B(s, 5) - R" is affine for every E fa ({x})
Theorem 7 (Kivva et al. [43]), Given fa : R + R" is a piecewise affine function such that {x E IR" : Ifz ([x})] = 00) C (R has measure zero with respect to the Lebesgue measure on [a(R""), this implies dim (ii((((("))) = m and almost every point in fa(lR") (with respect to the Lebesgue measure fa(IR") is generic with respect to fa-
Theorem 8 (Kivva et al. [43]). Consider a pair of finite GMMs in TR."
J Σ
y Σ NrNN MyNyyy poj Egh,
and
EaNNiy's E/).
(44)
Assume that there exists a ball B(x, 8) such that y and y' induce the same measure on B(x, 6). Then y =', and for some permutation T we have that TT; TT(() and (Mii 2i) - (H((3)) Σ (x)))
Theorem 9 (Kivva et al. [43]) Given z ~ E' TNN(2; p4: Eg) and z' ~ ES nI (2 v E
and fa(z) and ) are equally distributed. We can assume for x E R" and 6 > 0, fa is invertible on B(x, 26) n fR( This implies that there exists X1 BBx, 5) and 81 0 such that both fa and Ja are invertible on B(x1, 51) n fa(I..").
We next propose our slot identifiability result below.
Theorem 2 Ide-teentifiable Slot Representations). Given that the aggregate posterior q(z) is an optimal, non-degenerate mixture prior over slot space (Lemma 1), fa S + X' is a piecewise affine weakly injective mixing function (Assumption 8), and the slot representation, be observed as a sample from a GMM (Theorem 1), then p(s) is identifiable as per Definition 3.
(SI..... 8x) can
Proof. The proof extends from [43] to slot-based models. Given two piece- wise affine functions Sa, S + X. vk [K]. let ($1,.... sk), S& ((Es f4k; Σg) and s' (s',-... s'x) 3 S ~ ( 14 Σ/ be pair of slot representations constructed by sampling and concatenating each mixture component (i.e. slots) from two distinct GMMs. As proven in Theorem given individual sampled slots Sa are conditionally independent given the mixture component k, then a concatenated sample is from a higher dimensional GMM in RKdd Now, suppose for the sake of argument that fu(S) and fa(S') are equally distributed. We assume that there exists x E X' and d such that fa and fa are invertible and piecewise affine on Bxx, 5) n fa(S), which implies dim fa( = (S).
We now restrict the space (xx, 8) subspace B(x', 8') where x e Bxx', 88) such that fa and fa are now invertible and affine on (xx,, 8') n fa()). Next, we let L C X' be an |S| (S--dimensional affine subspace (assuming [ 2 [S)), such that B(x', f')
B(x', ((x)",")) n We also define h hy S > I. to be pair of invertible affine functions where h (B(x' 8') n L.) = ( (1B((x88') 88) n and 17 (B(x,, 8¹) n L) Ja (B(x', (
Therefore, this implies (s) and h (s') are finite GMMs which coincide on B(x', 8') n L and (s) = (s') based on Theorem 8. Given, h h hy and h (s) and (s'))) then h is an affine transformation such that h(ss) = s'.
Given Theorems 7 and 9, there exists a point x E fa(S) that is generic with respect fa and fa and invertible on (xx, 5) n fa(S). Having established that there is an affine transformation h(s) = s' and two invertible piecewise affine functions fa and fa on (xx, 8) n fa(S), this implies that p(s) is identifiable up to an affine transformation and permutation Sy s, which concludes the proof.
Corollary 3 (Individual Slot Identifiability). If the concatenated slot distribution p(s) is ~ identifiable (Theorem 2) then this implies g(2) is identifiable up to affine transformation and permuta- tion of the slots, sę and therefore each slot distribution Sy ~ N((s: pk. Σ₁) E Rd Y E (1.... K) is also identifiable up to an affine transformation.
Proof. Given Theorem 8, we know that each higher dimensional mixture component in p(s) induces the same measure on (xx, 6) and hence for some permutation we have that (Hx(()) En(r))
22

---

Algorithm 2 Probabilistic Slot Attention (no k or v)
Input: fc(() Rxxxd
input representation
Ak. π (0)x 1/K, p(0)x ((00 la
fort t ....., -1
T (1)aN (zn; Wau(DR,, (
Ank
Ej=1 ((1) (zas W ()), ( (1)3)
Ank t- Ank/ Zi=1 Au
P normalize attention
mlt 1)x Σ
n=1 Antoon
> update slot mean
att 1)3 Σ
n=1 Ânk (zn plt 1)6)
a(t 1)k- Emm1
Ank/N
> update mixing coeff.
return ((T), (T))
K slot distributions
Algorithm 3 Probabilisticc Slot Attention V.2 (PSA-PROJ)
Input: z = fe (x) E RN xd
b input representation
k + WiZ E IR.
D compute keys
Ak, π (0); 1/K m(0)x ~ LV. la
for t 0, ...... ,
Ank - T (t)xNN (kni Wam(() (
ΣS T(() (kni W
Ank < Ank/ Ei=1- Au
b> normalize attention
1(t 1)xx Σ n=1 And Wn
> update slot mean
al++1) +12 Σn Ank (W,zn, 11l 1)x)²
a1++1] En Anz/N
update mixing coeff.
return (
K slot distributions
(P((=())) )r Therefore, each mixture component S.(1) is identifiable up to affine transfor- mation, and permutation of slots representations in s. Now, given sampling Sy is equivalent to obtaining samples from the GMM, q(2) and concatenating, this makes q(2) identifiable up to affine transformation, h and permutation of slot representations in s.
now trivially follows that each slot representation Sy N(ss pkx E&) Rd, Vk fl...., K} is identifiable up to affine transformation, h based on the following observed property of GMMs:
K
K
The (N pk. Ex)) ~ hE Σ
TWN En))
(45)
which concludes the proof.
E Algorithm
As result of projection with separate query matrix is considered, the mean and variance updates are coupled with some non-linear affine transformation; for EM to have an exact solution, this transformation needs to be the identity (this can be seen when we derive the update equations for mean and variance). The resulting algorithms for these two cases are illustrated algorithms and
23

---

Table 4: Quality of compositional image generation with FID measure
METHOD
CLEVR
OBJECTS-
FG-ART
CFID +
RFID +
FG-ARI 1
CFID 4
RFID +
SA
0.96 0.01
41.81 2.82
0.79 + 0.06
16.49 1 3.34
SA-NoA
0.62 + 0.02
81.21 1 8.72
0.41 + 0.12
96.64 1: 21.33
PSA-NOA
0.84 = 0.01
36.42 + 8.53
68.02 + 10.21
0.78 + 0.04
54.55 + 1.43
21.37 + 1.03
PSA
0.85 1 0.02
28.50 1 4.27
52.70 + 1.74
0.78 1 0.02
20.49 1 2.36
21.00 1 1.43
PSA-PROJ
0.95 0.00
28.79 1 5.50
39.42 1 8.29
0.81 + 0.04
34.58 + 4.32
16.85 + 5.68
F
Metrics
SIS:
Slot identifiability score [6], mainly focus R2 between ground-truth and the estimated slot representations wrt to maximum R2 score from the models fit between each and every inferred slots. By design SIS requires model fitting every validation step compute this relativistic measure, due to which the metric seems to vary quite bit across runs, as observed in Figure 8.
Mean correlation coefficient is well studied metric in disentangled representational learning [39, 37], we extend this with additional permutational invariance on slot dimensions. SMCC measures the correlation between estimated and ground truth representations (or estimated represen- tations across runs, in the case when ground truth representations are unavailable) once the slots are matched using Hungarian matching. To compute identifiability up to affine transformation along the representational axis and permutation in slots (check definition 3), similar to weak identifiability as per the definition in the paper and in [38], use the MCC up to some affine mapping A, which learned by matching slot representations across runs. In summary, SMCC can be computed with the following three steps:
SMCC:
Matching the order of slots using Hungarian matching:
Affine transformation of slot representation;
Followed by computing mean correlation.
Apart from variations descrihed in Figure 8, we further analyse both the metrics by fixing the model and data; and by computing both the metries for 10 times. We then considered the mean and variance in the performance, which is reflected as follows: SIS: 35.26 + 6.46, SMCC: 77.83 + 0.36. Here, the resulting variation only the reflection of the metric, which clearly indicates the stability of SMCC over SIS.
Stot Mean Correlation Cocficient
0.75
Slot Ideutiflabillty Score
0.10
0.55
0.50
MONET SANAA
MONET
0.48
PSA (Dun))
(ound
20
62
50
102
20
63
100
Epoch
Epoch
Figure 8: Identifiability scores throughout training. Our proposed SMCC metric (top) is much more stable than SIS (bottom) in capturing identifiability, and better discerning differences between methods, showing substantial improvements for PSA.

---

Figure 9: Random samples from the 2D synthetic dataset used in our aggregate posterior identifiability experiments. As outlined in the main text, there are in total five 'object' clusters in the dataset, and each observation contains at most three of the clusters.
G Comparison with Autoencoding Variational Bayes
As explained in Section 3, applying slot attention to deterministic encoding fe(x) E Nxd yields a set K object slot representations S1:K == S....., In combination, this process induces stochastic encoder (($ | x), where the stochasticity comes from the random initialization of the slots in the first iteration: $1.K === (s 0) I) E Ig Since each slot is deterministic function of previous state s' := ʃa (z, s²-1) it possible randomly sample initial states s" and obtain stochastic estimates of the slots. However, since each transition depends 2, which in turn depends on the input x, we do not get a generative model we can tractably sample from.
This can be remedied by placing a tractable prior over and using the VAE framework along the lines of [77]. Specifically, Wang al. [77] propose the Slot-VAE: which is generative model that integrates slot attention and the VAE framework under two-layer hierarchical latent model. However, under their formulation, there is a key challenge in calculating the KI. term as the slot attention function is permutation equivariant meaning the slots have no fixed order across the posterior and prior. To compensate for this, the authors introduce a heuristic auxiliary loss and a parallel image processing path with additional slot attention operation which is computationally costly.
In contrast, our approach does suffer from such drawbacks. This achieved by designing the slot attention operation itself as probabilistic, and proving that having local (per-datapoint) GMM results in the aggregate slot posterior and the concatenated slots being GMM distributed (Section 4). We then proved new slot identifiability result using this insight (Section 5). Our use of the aggregate posterior inspired by but differs substantially both method and application from previous works on VAEs [28, 71, 72]. Our primary goal is not to learn better generative prior but to obtain slot identifiable representations. Lastly, since the concatenated slots are provably GMM distributed, our model is reminiscent of GMM-VAE [11], but with a unique slot-based structure.
H Automatic Relevance Determination of Slots
For evaluation, we calculate the MAE between the estimated and ground truth numbers of slots and measure the reduction in FLOPs achieved using the proposed ARD method. We observe MAE values 1.03 and 0.58, and significant savings of up to 41 % and 62% in FLOPs on the CLEVR and Objects-Rood Room datasets respectively, when compared to using a fixed number of slots K.
25

---

Recesotacion
SKC 06-500
SK02: 2--0.00
SBC 3:21-0:33
Seet pl
Inpu
Recesoontion
SMt
SB 3z2w8.22
Stc: pw0d:
Sie popm0.22
Recommettie
ske 3,316.2
Recossnation
2-2.00
2:200-33
Stt
Insu
Receseaation
S04::: 2-0.73
516 3:2m0-22
Sto 4 p=000
Sht 21:100:0
Siei pam0.00
Sol 210
Veeu
Revemmattiv
5642: =-.00
SBC 3,34--22
Sir pi4h03
Mar 3231-0112
Siar plerr.
Skat p4
Necommaaciin
Skell ai-0:000
Sie S.sho.11
Automatic Relevance Determination (ARD) of slots on the OBJECTSROOM dataset. As shown, when using our proposed probabilistic slot attention (Algorithm 1), the mixing coefficients π, E R vi automatically approach zero when slots are inactive.
Figure 10:
I Compositionality Results
26

---

2F:7
Compseed
S012
SKc.3
31X1
3M45
S
Str.7
Corpose]
SHL1
5012
3K..3
3154
3M45
S
St.77
Compxeed
SUEl
Sa12
SIE
sies
Sett
Corpxced
Figure 11: Aggregate posterior sampling for image composition on CLEVR dataset.
Sr.7
Corpsast
Carpore
9IE1
Sh12
91:1
Seth
Srr7
Corposel
Sit1
Sa12
S6:3
3845
Set6
91:7
Carosest
Figure 12: Aggregate posterior sampling for image composition on OBJECTSROOM dataset.
27

---

Figure 13: Visualizations of attention and alpha masks on the Pascal VOC2012 dataset are shown, with alpha masks on the left and attention masks on the right, for PSA-MLP model using DINO feature extractor. In the figure above, the images from left to right represent: the original image, ground-truth segmentation, alpha mask segmentation, individual entities grouped in the alpha mask, slot attention segmentation mask, and individual entities grouped in the slot attention mask.
Figure 14: Visualizations of attention and alpha masks on the Pascal VOC2012 dataset are shown, with alpha masks the left and attention masks on the right, for PSA-Transformer model using a DINO feature extractor. In the figure above, the images from left to right represent: the original image, ground-truth segmentation, alpha mask segmentation, individual entities grouped in the alpha mask, slot attention segmentation mask, and individual entities grouped in the slot attention mask.
28

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
We clearly state the limitations in existing literature and our proposed anal- ysis to overcome these limitation, which we further back with theoretical guarantees and experimental observations.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide a detailed discussion on limitations and explicitly list the assumptions made, which serves nicely for future works.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
29

---

Answer: [Yes]
Justification:
We provide ngorous mathematical proof all our theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We clearly list all the set of hyperparameters used in the analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
30

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide codebase with the described set of hyper-parameters for reprodu- cability.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbccccccceecce heabblicggiidee CodeeSbbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We discuss hyper-parameters in the paper along with the codebase for earier reproducability.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report all our analysis across 5 random runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
31

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics settps//neurrps... http://ppp....pppppcacccccccccee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
This paper proposes Probabilistic Slot Attention algorithm, whose goal is to achieve identifiable object-centric representations. The work primarily makes theoretical advancements in the field of object-centric learning, and as such it has little immediate societal or ethical consequences, Our method might be a step towards interpretable and aligned models which are desired properties of trustworthy AI.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
32

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
34

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35

---

36


Reviews:
[{"review_id": "piM6Sa4UmT", "reviewer": "Reviewer_11Sk", "summary": "This paper propose a probabilistic slot attention method that can learn identifiable object-centric representation. Compared with former work on identifiable object centric representation methods, the proposed model can scaling slot-based methods to high-dimensional images. Both theory analysis and experiments verify the effectiveness of proposed model.", "questions": "1. How much computational complexity has increased?", "limitations": "N/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper propose a novel probabilistic slot attention model, which can learn Identifiable Object-Centric Representation.\n\n2. the proposed model can scale to high-dimension image dataset. \n\n3. the proposed model is seem to be solid.\n\n4. the paper is well written and read well.", "weaknesses": "1. the experiments are only take on two toy datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their effort and overall positive outlook on our paper. We are encouraged to read that our work is perceived as solid, novel, and well-written. Please see our responses to the questions raised below.\n\n> **\"the experiments are only take on two toy datasets.\"**\n\nWe kindly remind the reviewer that the objective of our work is to study the theoretical identifiability of slot representations and the conditions that ensure this property, rather than to pursue state-of-the-art empirical results. Understanding when object-centric representations can theoretically be identified is crucial for scaling slot-based methods to high-dimensional images with correctness guarantees. To verify our theoretical results, we first conduct detailed experiments on controlled datasets and then extend our demonstrations to unstructured image data. We stress that the synthetic datasets we used are necessary for properly testing our identifiability hypotheses. \n\nBefore our work, there was a lack of explanatory theory for why state-of-the-art results were able to be obtained using non-additive autoregressive Transformers (DINOSAUR [60]) and/or diffusion-based decoders (Slot-Diffusion [r2]). We showed that by viewing slot attention through a probabilistic graphical modelling perspective it is possible to prove slot identifiability for non-additive decoders using proof techniques from the identifiable generative modelling literature. Given that vanilla slot attention can be seen as a simplified version of probabilistic slot attention, akin to the relationship between soft k-means and GMMs, our theoretical results suggest why non-additive decoder structures can work well given the appropriate latent structure and inference procedure are in place.\n\nNonetheless, **we have now evaluated our method on real-world large-scale datasets** and using more powerful decoders to demonstrate that our method also scales well - please find the details of our experiments in the general comment at the top. \n\n> **\"How much computational complexity has increased?\"**\n\nWe emphasise that probabilistic slot attention (PSA) retains the $\\mathcal{O}(TNKD)$ computational complexity of vanilla slot attention, where $T$ denotes the number of attention iterations, $N$ the number of input vectors, $K$ the number of slots and $D$ the slot/input dimension. The additional operations we introduce for calculating slot mixing coefficients and slot variances (under diagonal slot covariance structure) have complexities of  $\\mathcal{O}(NK)$ and $\\mathcal{O}(NKD)$ respectively, which do not alter the dominant term. Furthermore, when used in conjunction with additive decoder-based models, PSA can reduce computational complexity by pruning inactive slots via automatic relevance determination (ARD) as outlined in Section 4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hIc46hR3Id", "reviewer": "Reviewer_27BN", "summary": "Solving the problem of identifiability is necessary to find consistent and interpretable factors of variation. There are two approaches to do so: a) place restrictions on the decoders, and b) impose distributional constraints on the latent space. This work takes the second approach and aims to impose a GMM on the latent space. The paper does this by proposing a modification to the vanilla slot attention framework that they call probabilistic slot attention. Under this framework the papers shows theoretical and empirical identifiability of slots.", "questions": "* Could you elaborate on assumption 6 and why it is not necessary in the current work?\n* In Algorithm 1, line 6, where attention $A_{nk}$ is calculated, mean is calculated as $W_q \\mu(t)_k$ but variance is not calculated as $W_q \\sigma_k^2$ ?\n* Should we not return $\\pi (t)$ at the end of the algorithm?  \n* The work explicitly mentions that having additive decoders is not a necessity of the current work, but any experimental results are shown only with additive decoders. Perhaps it may be enlightening to see experiments akin to [45] for non-additive decoders (transformer based auto-regressive decoders). There are no results to go along with the choice of a convolutional decoder as the non-additive variant which is mentioned in L277. Did I miss something here?\n* R2 score is reported in Sec 6 without definition. I assume this is the correlation? This is defined as MCC in the paper.", "limitations": "The statistical distributional assumption on the latent space excludes for identification of any causal dependencies between objects which could be made explicit in the “Limitations and Future work” section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The proposed probabilistic slot attention framework is an intuitive extension of the vanilla slot attention with the updates in each iteration resembling the familiar EM algorithm.\n+ The proposed framework offers a possible solution to the problem of dynamically determining the required number of slots.\n+ This is the first work to experiment with imposing a distribution on the latent space where prior works either focus on the generator or the decoder.\n+ Recovering the latent space upto an affine transformation is shown in the synthetic modelling scenario with theoretical guarantees provided.\n+ The paper studies an important premise – the theoretical understanding of object-centric representations. The paper is written well, with clear motivations for the proposed contributions.", "weaknesses": "* The paper states that the framework allows for tractably sampling from the aggregate posterior distribution and using it for scene composition tasks; however, this is not empirically qualified anywhere.\n* Experiments only validate the theory on simple synthetic datasets. Testing on more diverse and realistic data would better demonstrate applicability, though the evaluation would also be more challenging. Generally speaking, I would be concerned about the scalability of such an approach leveraging GMMs. I understand however that the objective of this work is theoretically study the identifiability of object-centric representations under less-constrained settings as compared to previous work. Maybe, having a short discussion on how such an approach can be scaled would be nice to see.\n* Table 1 lists $\\beta$-disentanglement and weak injectivity as core assumptions. While the former is common to all other related methods, weak injectivity is newly introduced. The implications of this assumption is hence important, but is missing in the paper. \n* In fig. 3, it is not clear what experiment the latents (x-axis) correspond to.\n* It might be helpful to visit slot-identifiability in the related work section, considering its literature is the most closely related to this work.\n* Adequate details about the encoder and decoder in the synthetic modelling scenario have not been provided.\n\nMinor/editorial\n* Typo in 769, “Obsereved” -> observed; Typo in 234, “emphirically” -> empirically\n* Some references are repeated. Eg. 36-37. Please check carefully across the full list.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful, detailed comments and constructive feedback. We greatly appreciate the positive outlook and the fact that our work was found to be well-written, well-motivated and novel. \n\n \n> **\"The paper states that the framework allows for tractably sampling from the aggregate posterior ...\"**\n\nWe indeed show that it is possible (Lemma 1) to use the aggregate posterior for compositional tasks, but we felt that it does not meaningfully add to the paper's main theoretical identifiability contributions. We have now included some preliminary results for demonstration purposes (see pdf). Performing a more comprehensive compositional analysis of the aggregate posterior is certainly valuable and warrants dedicated investigation in future work.\n\n> **\"Experiments only validate the theory on simple synthetic datasets. Testing on more diverse and realistic ...\"**\n\nAs rightly pointed out by the reviewer, this work aims to study theoretical identifiability of slot representations and the conditions that ensure this property, rather than provide state-of-the-art results on large-scale datasets. To verify our theoretical claims, we first conduct detailed experiments on controlled datasets and then extend our demonstrations to unstructured image data. We stress that the synthetic datasets we used are necessary for properly testing our identifiability hypotheses. \n\nRegarding scalability concerns of probabilistic slot attention (PSA), we emphasise that PSA retains the $\\mathcal{O}(TNKD)$ computational complexity of vanilla slot attention, where $T$ denotes the number of attention iterations, $N$ the number of input vectors, $K$ the number of slots and $D$ the slot/input dimension. The additional operations we introduce for calculating slot mixing coefficients and slot variances (under diagonal slot covariance structure) have complexities of  $\\mathcal{O}(NK)$ and $\\mathcal{O}(NKD)$ respectively, which do not alter the dominant term. Furthermore, when used in conjunction with additive decoder-based models, PSA can reduce computational complexity by pruning inactive slots via automatic relevance determination (ARD) as outlined in Section 4.\n\nFinally, **we have now demonstrated the applicability of PSA to transformer-based decoders** - please refer to the general comment above for details.\n\n> **\"Table 1 lists -disentanglement and weak injectivity as core assumptions. While the former ...\"**\n\nWe have included a discussion on the weak injectivity assumption in the remark just below it - we’ll also include a similar discussion in the main text. In summary, weak injectivity ensures that a mixing function $f_d$: \n(i) in a small neighbourhood around a specific point $x_0 \\in \\mathcal{X}$ is injective – meaning each point in this neighbourhood maps to exactly one point in the latent space $\\mathcal{Z}$; and (ii) while $f_d$ may not be globally injective, the set of points in $\\mathcal{X}$ that map back to an infinite number of points in $\\mathcal{Z}$ (non-injective points) is almost non-existent in terms of the Lebesgue measure on the image of $\\mathcal{Z}$ under $f_d$. This assumption is generally satisfied when using Leaky-ReLU networks with randomly initialized weights (Appendix C). \n \n\n> **\"In fig. 3, it is not clear what experiment the latents (x-axis) correspond to.\"**\n\nOur apologies for the confusion. Figure 3 is a simple illustrative example of an aggregate Gaussian mixture density, it is there to provide the reader with a conceptual intuition and does not correspond to an experimental setting. \n\n> **\"Adequate details about the encoder and decoder in the synthetic modelling scenario have not been provided.\"**\n\nWe thank the reviewer for pointing this out as it escaped our attention. We have now added the architectural details for all our models in the appendix - all the code will be made available also. \n\n> **\"In Algorithm 1, line 6, where attention A is calculated, mean is calculated as $W_q\\mu$ but variance is not calculated as  $W_q \\sigma^2$?\"**\n\nThis suggestion is a valid design choice if either the weights $W_q$ are constrained or we use an activation function to ensure all entries in $W_q \\sigma^2$ remain positive. However, since $\\sigma(t)^2$, at attention iteration $t$, already has an indirect dependency on $W_q$ through $\\mu(t-1)$ we omit this style of projection of the variances for simplicity. \n\n\n> **\"The work explicitly mentions that having additive decoders is not a necessity of the current work, but any ...\"**\n\nThis is unfortunately incorrect as our \"NoA\" model variants do in fact correspond to a non-additive convolutional decoder as stated in Section 6. Switching these with more powerful autoregressive transformer-based decoders would possibly improve the results but would constitute an unfair comparison with our baselines. For new experiments please refer to the general comment.  \n \n> **\"R2 score is reported in Sec 6 without definition. I assume this is the correlation? This is defined as MCC in the paper.\"**\n\nR2 score is a coefficient of determination, it is proportional to correlation, which can be empirically observed even in our experiments. SIS score as introduced in [5] is a relative measure of R2. \n\n> **\"Should we not return  \\pi at the end of the algorithm?\"**\n\nYes thanks for pointing this out, we have now corrected it. \n\n> **\"Could you elaborate on assumption 6 and why it is not necessary in the current work?\"**\n\nThanks, we will add a remark explaining this in the paper. Object sufficiency is crucial when learning *grounded* object representations [43]. Here we do not focus on grounding so strict object sufficiency is technically not required.\n\n[r1] Yao, W., Sun, Y., Ho, A., Sun, C. and Zhang, K., 2021. Learning temporally causal latent processes from general temporal data. arXiv preprint arXiv:2110.05428.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "glsbH6vQCI", "reviewer": "Reviewer_PnmW", "summary": "This paper addresses the problem of identifiability of object-centric representations. In contrast to prior works which achieve identifiability via assumptions on the generator, this paper explores identifiability via assumptions on the latent distribution. To do this, the authors introduce a probabilistic variant of the popular Slot Attention algorithm and prove theoretically that this method identifies the ground-truth object representations. The authors verify their theory on toy data as well as test their method on high dimensional image data, showing improved performance over baseline methods.", "questions": "* Do the authors view their method as being a replacement for structured decoders in object-centric learning or do they view the method as being better used in tandem with structured decoders?\n\n* What is the key theoretical assumption/constraint which allows for identifiability to be possible? \n\n\n* What are the key differences between SMCC and SIS?\n\n\n* Do the authors have an explanation for the low SIS score on the toy data experiments?\n\n\n* Does “R2” refer to SIS when used or is this a different score? \n\n\n* Do the authors have any intuition about how well their method would perform for more complex models/datasets?", "limitations": "The authors include a limitation section which discusses some limitations of their theory. I would encourage the authors to discuss some limitations in their experiments, as discussed above in the “Weaknesses” section. I would also include some discussion of the limitations of purely probabilistic structure for object-centric learning as it pertains to compositional generalization, as discussed above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The authors address an important problem in representation learning. Namely, understanding when learning object-centric representations is theoretically possible.\n\n&NewLine;\n\n* The paper is well written, well structured, and, in general, easy to understand.\n\n&NewLine;\n\n* The authors position their contribution well within the broader representation learning literature and provide a good review of prior work.\n\n&NewLine;\n\n* Section 4 is well written. I particularly appreciated being able to look at Algorithm 1 while reading the section to guide my intuition.\n\n&NewLine;\n\n* Exploring probabilistic constraints for identifiability in object-centric learning is an important problem to understand, thus I think the author’s contribution is of interest for the representation learning community.\n\n&NewLine;\n\n* The authors achieve identifiability by proposing an adaptation to a widely used method, making their method potentially easy to adopt in practice.\n\n&NewLine;\n\n* The authors conduct a solid empirical study and achieve promising results, in particular when coupling their method with structured decoders.\n\n&NewLine;\n\n* The figures in the manuscript are conceptually helpful and aesthetically well done.", "weaknesses": "### __Paper Positioning/Storyline__\n\nOne of the main issues I have with this work is that I do not think that the paper’s storyline i.e. how the authors motivate and position their contribution, accurately reflects the actual contributions of this work. As I understand, the current pitch of the paper is the following: \n\n&NewLine;\n\n*Previous works from [1, 2], prove identifiability of object-centric representations by making assumptions on the decoder. Enforcing the assumptions in [1] is not tractable in practice, however, due to scalability issues with the compositional contrast in [1]. In this work, we remedy this by exploring probabilistic constraints for identifiability which yield identifiability theoretically and empirically but do not suffer from the same empirical scalability issues as prior works.*\n\nI think this pitch is problematic for the following reasons:  \n\n&NewLine;\n\n__Firstly__, it is important to note that the compositional decoders explored in [1] were proven to be a subset of the additive decoders explored in [2]. Consequently, if the ground-truth decoder is compositional and one uses an additive decoder for the inference model, then assuming the assumptions from [2] are met, the inference model is slot identifiable i.e. it will implicitly minimize compositional contrast. \n\n&NewLine;\n\nThis does not completely dismiss the scalability issues noted by the authors, since as mentioned in Lines 51-52, additive decoders also suffer from some scalability issues. I think, however, that the author’s current discussion of scalability, in particular wrt [1], misses the key nuance discussed above. I would suggest the authors incorporate this discussion into the paper by altering their writing and positioning of their contribution accordingly. \n\n&NewLine;\n\n__Secondly__, the storyline and writing in the manuscript give the impression that one could dispense with decoder structure all together in favor of probabilistic structure on the latent space. As the authors show empirically in Section 6, however, this is not exactly the case. While probabilistic structure gives identifiability gains relative to baselines, without incorporating decoder structure, identifiability drops non-trivially across all metrics.  \n\n&NewLine;\n\nMoreover, one of the core motivations for object-centric learning is learning representations which generalize compositionally [2, 3]. Such compositional generalization is only possible through decoder structure where additivity is one such structure (see [3] Section 2.). If one only focuses on enforcing structure on the latent distribution, it is not clear to me how such compositional generalization can be achieved.\n\n&NewLine;\n\nWith all of this being said, I think a more accurate and superior pitch for the contributions in this work should focus on the advantage of using probabilistic and decoder structure in tandem opposed to suggesting that probabilistic structure is somehow superior. Something like: \n\n&NewLine;\n\n*In this work, we show how identifiability can be achieved via probabilistic constraints on the latent space. We show how such constraints can be naturally and tractably incorporated into Slot Attention. We verify our theory and method on toy data. We then show on image data that our probabilistic structure leads to improved identifiability over unstructured baselines. Furthermore, when coupled with structured decoders, our method yields performance which outperforms both probabilistic structure and decoder structure in isolation.* \n\n&NewLine;\n\nFor the reasons stated above, I would encourage the authors to alter their writing in the manuscript to adhere closer to a storyline along the lines of the one presented above. As things currently stand, the messaging in the paper feels a bit misleading and over-claiming. \n\n&NewLine;\n\n$\\newline$\n\n&NewLine;\n\n### __Theory Explanation__\nI would have appreciated a short paragraph giving some intuition on how exactly the probabilistic constraints imposed by the authors sufficiently restrict the problem such that slot identifiability is possible. Specifically, unsupervised identifiability via probabilistic structure is challenging and, in most cases, impossible [4., 5.]. Thus, I think it would be helpful if the authors could explain which structure in their method is key towards overcoming these well known unidentifiability issues. For example, is it the permutation invariance, the complex aggregate posterior etc.?\n\n&NewLine;\n\n$\\newline$\n\n&NewLine;\n\n### __Metrics__\nThe authors use two main metrics to validate identifiability: the slot identifiability score (SIS) from [1] and a new metric, slot MCC (SMCC). I found the authors explanations of the differences in these metrics unclear, which made it difficult to interpret some of the empirical results. My understanding is that SMCC is just a linear/affine version of SIS in terms of the predictor fit between latents. Thus, it is a bit unclear to me why the values for SIS should be so much lower than SMCC in the experiments on toy data in Section 6.\n\n&NewLine;\n\n$\\newline$\n\n&NewLine;\n\n### __Experiments__\nThe authors experiments on image data focus primarily on simple decoders opposed to e.g. Transformers and visually complex datasets. To assess the scalability of the author’s method it would be important to test the method on more complex models/datasets. I view this as a more minor weakness of this work, however, given that the contribution is largely theoretical.\n\n&NewLine;\n\n$\\newline$\n\n&NewLine;\n\n### __Appendix__\nIn section A. of the Appendix, the authors review some definitions from prior works e.g. in [1]. These definitions, however, are presented in an informal, imprecise way. I would encourage the authors to be precise in this section if they are going to include definitions from prior works.\n\n&NewLine;\n\n$\\newline$\n\n&NewLine;\n\n## __Bibliography__\n\n1. Provably Learning Object-Centric Representations (https://arxiv.org/abs/2305.142290)\n\n&NewLine;\n\n2. Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation (https://arxiv.org/abs/2307.02598)\n\n&NewLine;\n\n3. Provable Compositional Generalization for Object-Centric Learning (https://arxiv.org/abs/2310.05327)\n\n&NewLine;\n\n4. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (https://arxiv.org/abs/1811.12359)\n\n&NewLine;\n\n5. Variational Autoencoders and Nonlinear ICA: A Unifying Framework (https://arxiv.org/abs/1907.04809)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their very detailed comments and constructive feedback which helped improve our paper significantly!\nWe also appreciate the fact that our paper was perceived to be well-written, easy to understand, and of interest to the community. \n\n> **\"One of the main issues I have with this work is that I do not think that the paper’s storyline ...\"**\n\nWe thank the reviewer for their very insightful and constructive suggestion regarding the messaging of the paper. We largely agree with the overall sentiment and will incorporate the suggested changes which will significantly improve the paper’s pitch. To clarify, it is not our intention to compete with or replace additive decoders as a model choice since they undoubtedly provide useful inductive biases for object-centric learning as ours and many other previous works show.  \n\nWe further remark that before our work there was a lack of explanatory theory for why state-of-the-art results were able to be obtained using non-additive autoregressive Transformers (DINOSAUR [60]) and/or diffusion-based decoders (Slot-Diffusion [r2]). We showed that by viewing slot attention through a probabilistic graphical modelling perspective it is possible to prove slot identifiability for non-additive decoders using proof techniques from identifiable generative modelling. Given that vanilla slot attention can be seen as a simplified version of probabilistic slot attention, akin to the relationship between soft k-means and GMMs, our theoretical results suggest why non-additive decoder structures can work well given the appropriate latent structure and inference procedure are in place. With that said, there is an identifiability and expressivity trade-off [45] induced by the choice of decoder structure, so depending on the use case it may indeed be advantageous to combine both latent and additive decoder structure!\n \n> **\"I would have appreciated a short paragraph giving some intuition on how exactly the probabilistic ...\"**\n\nWe would like to point out that we do discuss the intuition of our identifiability results in depth in the appendix, just before the proofs. As opposed to [4, 5], we use a more expressive GMM latent distribution followed by a weakly injective piece-wise linear decoder which in combination ensures that the slot representations are identifiable. \n\n\n> **\"The authors use two main metrics to validate identifiability: the slot identifiability score (SIS) from ...\"**\n\nSIS is the relative R2 (coefficient of determination) score-based measure, capturing the relative ratios of variance for a dependent variable explained by an independent variable with baseline model scores that are trained on the slot representations of a dynamically updating model. As for SMCC, it measures the linear/affine relationship between permuted slots and features. We provide more detailed explanations in appendix F.\n\n> **\"The authors experiments on image data focus primarily on simple decoders opposed ...\"**\n\nAlthough we agree that assessing the performance of probabilistic slot attention on more complex datasets would be useful, the controlled scenarios and datasets we used are necessary for properly testing our theoretical identifiability hypotheses, which is the objective of this work. \n\nRegarding scalability concerns of probabilistic slot attention (PSA), we stress that probabilistic slot attention PSA retains the $\\mathcal{O}(TNKD)$ computational complexity of vanilla slot attention and as such enjoys the same scalability properties. We did evaluate our model on large-scale datasets and with more complex decoders, we discuss them in the general comment.\n \n> **\"Do the authors view their method as being a replacement for structured decoders in object-centric learning or do they view the method as being better used in tandem with structured decoders?\"**\n\nFrom a theoretical identifiability viewpoint, latent distributional structure is a sufficient requirement as long as the decoder is piece-wise linear. However, additive decoders provide strong, useful inductive biases and may be easier to optimize relative to a probabilistic model with an arbitrary decoder. In our experiments, we observed that using our approach in tandem with an additive decoder tends to outperform other models.\n\n> **\"Do the authors have an explanation for the low SIS score on the toy data experiments?\"**\n\nSIS was proposed by [1] and it uses the relative R2 scores, where baseline models are trained on the slot representations of a dynamically updating model. Due to this, it tends to exhibit high variability as illustrated in Appendix F (we use the official implementation on GitHub for all our analysis). We believe this instability in the baseline model, which is also pointed out by others on their GitHub repository, could be due to the validation dataset size resulting in lower scores.  \n\n> **\"Does “R2” refer to SIS when used or is this a different score?\"**\n\nR2 is the coefficient of determination, while SIS is a relative R2 score. R2 is propositional to SIS but due to the instability of SIS, we use R2 for all imaging experiments.\n\n> **\"Do the authors have any intuition about how well their method would perform for more complex models/datasets?\"**\n\nThe method can be scaled to large-scale datasets using the approaches outlined in [60], with convolutional or transformer-based decoder since the computational complexity is the same as vanilla slot attention ($\\mathcal{O}(TNKD)$). **Please refer to the general comment for results on complex datasets using more powerful decoders.**\n\n[r2] Wu, Z., Hu, J., Lu, W., Gilitschenski, I. and Garg, A., 2023. Slotdiffusion: Object-centric generative modeling with diffusion models. Advances in Neural Information Processing Systems, 36, pp.50932-50958.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4spzDglzUl", "reviewer": "Reviewer_HxYt", "summary": "This paper studies the theoretical identifiability of object-centric representations (slots) in object-centric learning (OCL). Prior works study the same problem, but are limited to OCL models with an additive decoder. This work relaxes the constraint to non-additive decoders, which have shown important to scale up to more complex data in recent OCL works (e.g., TransformerDecoder, Diffusion Model). To tackle this generalized setting, authors propose Probabilistic Slot Attention (PSA), which applies the Gaussian Mixture Model (GMM) to produce slots from each data sample (e.g. an image). The authors prove the effectiveness of PSA both theoretically and empirically, using results on both low-dimensional data and 2D images.", "questions": "- Can the authors apply PSA to a better non-additive decoder? For example, the Transformer decoder proposed in [61]. Otherwise, it is hard to assess the contribution of this work.\n- What is the implementation detail of the non-additive decoder \"standard convolutional decoder\"? How to decode an image from K 1D slots using a CNN decoder?\n\nMinor:\n- How can we apply the theory proposed in this paper to more complex datasets, e.g., real-world COCO images? This is not required for a theory paper, but I'm curious about the authors' thoughts.", "limitations": "The authors have adequately addressed the limitations in Sec.7.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper tackles an important problem -- what assumptions are required to learn identifiable object slots?\n- The paper is generally clearly written and well presented.\n- I appreciate the efforts in experimenting with common OCL image datasets such as CLEVR and ObjectsRoom. They are missed in previous identifiable OCL works.\n- The experimental results with PSA and PSA-Proj (both using additive decoders) are solid.", "weaknesses": "I have only one big concern. However, this concern is closely related to the main contribution of this paper (correct me if I am wrong), and I cannot accept the paper if it is not well addressed.\n- This paper claims to be the first work learning identifiable slots with a non-additive decoder (NoA). This is great, as recent OCL works show impressive results using a Transformer-based or a diffusion-based decoder.\n- However, the paper does not really have experimental results supporting this claim. PSA-NoA seems to underperform PSA-Proj consistently on all datasets & metrics.\n- What's worse, even compared to vanilla SA, PSA-NoA still underperforms in FG-ARI (Table 3), and even on some identifiability-related metrics (Table 2). These results are concerning to me, as we are not sure if the proposed algorithm can really scale to recent OCL models with better decoders.\n\nMinor:\n- Please unify the citation format -- for papers that are published at conferences, please use their conference version, e.g., [5] (ICML), [45] (NeurIPS).\n- In Sec.4, the paper claims that PSA can dynamically determine the number of object slots. While Fig.10 in the Appendix shows a few results, I don't think that's enough to claim PSA \"offers an elegant solution to this problem\". [1] is a recent work that studies this problem and provides in-depth analysis. The authors can conduct experiments following [1]'s setting. But I understand that the dynamic slot number is not the main contribution of this work.\n- In line 51, the authors claim that the computation cost of non-additive decoders is invariant to the number of slots K. Is this true? In recent Transformer-based and diffusion-based decoders, they use cross-attention to condition the reconstruction on slots, and the computation cost of attention is quadratic to the token size, i.e., K in this case.\n\n[1] Fan, Ke, et al. \"Adaptive Slot Attention: Object Discovery with Dynamic Slot Number.\" CVPR. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments and constructive feedback. We appreciate the fact that our paper was considered to be clearly written and well presented, and we are glad that our results are perceived to be solid. \n\n> **\"However, the paper does not really have experimental results supporting this claim...\"**\n\nWe respectfully disagree that the results do not support this claim, PSA-NoA and PSA-Proj both include the probabilistic latent setting as proposed in our work. Notably, there is a known trade-off between identifiability and expressivity induced by the choice of decoder structure [45]. Depending on the use case, it may be beneficial to combine both latent and additive decoder structures in practice, particularly if the latter introduces useful inductive biases and/or simplifies the optimization problem. In our experiments, we observed that when using PSA in tandem with an additive decoder it is possible to outperform all other baselines. As for non-additive decoder-based experiments, PSA-NoA must be compared directly against SA-NoA for it be a fair comparison, and we observed it to perform better than SA-NoA while remaining competitive with the remaining baselines. \n \n> **\"Can the authors apply PSA to a better non-additive decoder? For example, the Transformer decoder proposed in [61]. Otherwise, it is hard to assess the contribution of this work.\"**\n\nWe agree that the applicability of our framework on large-scale datasets is a crucial evaluation of our theoretical results, but that does not reduce the contribution of our work, which is primarily theoretical. The main focus of this work is to investigate the theoretical identifiability of slot representations and the conditions that ensure this property, rather than provide state-of-the-art results on large-scale datasets. To verify our theoretical claims, we first conduct detailed experiments on controlled datasets, and then extend our demonstrations to unstructured image data. We stress that the synthetic datasets we used are necessary for properly testing our identifiability hypotheses. \n\nOne of the main assumptions necessary to prove slot identifiability in our setting is weak injectivity, which is achieved when we use piece-wise linear decoders. In the case of transformer decoders, this assumption is not guaranteed to hold because of the complexity of the attention mechanism (further theory is required here). With that said, we have conducted additional empirical evaluations on the identifiability of slot representations obtained with more complex transformer decoders, which result in an SMCC of $\\mathbf{0.73 \\pm 0.04}$ and R2 of $\\mathbf{0.55 \\pm 0.06}$ on the CLEVR dataset, which is significantly better than SA and all other baselines. ***For more details and additional experiments please see the general comment at the top.***\n\n> **\"What is the implementation detail of the non-additive decoder \"standard convolutional decoder\"? How to decode an image from K 1D slots using a CNN decoder?\"**\n\nThank you for pointing this out as missing as it escaped our attention. We simply concatenate all K slots together and upscale the resolution using four transposed convolutions until we reach the image resolution, we also use Leaky-ReLU activations. The architecture is very similar to the one used in the original SA work, we will be sure to add the remaining details to the paper, thanks again.\n\n> **\"How can we apply the theory proposed in this paper to more complex datasets, e.g., real-world COCO images? This is not required for a theory paper, but I'm curious about the authors' thoughts.\"**\n\nWe have included experiments on PascalVOC2012 datasets and also tested our model with more complex decoders, ***please refer to the general comment above for details and additional results.***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We extend our thanks to all the reviewers for their time and constructive feedback which has undoubtedly helped improve the paper. We are pleased that the work was perceived to be well-written, well-presented, and novel, with solid results and of interest to the community. 

In the following, we highlight the main clarifications of our work raised by multiple reviewers and ***provide additional large-scale experimental results*** addressing all requests (see Table below and the attached pdf).

**General Clarifications:**

As correctly noted by all reviewers, the primary focus of our work is theoretical. To the best of our knowledge, before our work, there was a lack of explanatory theory for why state-of-the-art results were able to be obtained using non-additive autoregressive Transformers (DINOSAUR [60]) and/or diffusion-based decoders (Slot-Diffusion [r2]). We showed that by viewing slot attention through a probabilistic graphical modelling perspective it is possible to prove slot identifiability for non-additive decoders using proof techniques from identifiable generative modelling. Given that vanilla slot attention can be seen as a simplified version of probabilistic slot attention, akin to the relationship between soft k-means and GMMs, our theoretical results suggest why non-additive decoder structures can work well given the appropriate latent structure and inference procedure are in place. 

However, there is a trade-off between identifiability and expressivity induced by the choice of decoder structure [45]. Depending on the use case, it may be beneficial to combine both latent and additive decoder structures in practice, particularly if the latter introduces useful inductive biases and/or simplifies the optimization problem.
 
We stress that probabilistic slot attention (PSA) retains the $\mathcal{O}(TNKD)$ computational complexity of vanilla slot attention (SA). The additional operations we introduce for calculating slot mixing coefficients and slot variances (under diagonal slot covariance structure) have complexities of  $\mathcal{O}(NK)$ and $\mathcal{O}(NKD)$ respectively, which do not alter the dominant term. Furthermore, when used in conjunction with additive decoder-based models, PSA can reduce computational complexity by pruning inactive slots via automatic relevance determination (ARD) as outlined in Section 4.

**Large-scale Experiments:** 

We empirically tested slot identifiability using more complex non-additive, transformer decoders, following the SLATE [61] implementation and simply replaced the sloat attention (SA) module with probabilistic slot attention (PSA). On the CLEVR dataset, we observed an SMCC of $\mathbf{0.73 \pm 0.04}$, and R2 of $\mathbf{0.55 \pm 0.06}$, which are significantly better than all other models listed in Table 2 in the paper.  

To demonstrate that PSA can scale to large-scale real-world data we ran additional experiments on the Pascal VOC2012 dataset, following the exact "DINOSAUR" strategies and setups described in [60, r4] for fairness, then simply swapping out SA with PSA . Note that SA MLP (w/ DINO) denotes our replication of DINOSAUR MLP from [60] as a baseline. The table below shows the obtained results (all baselines are standard results taken from [60, r3]):

| Models        | $\text{mBO}_i$        | $\text{mBO}_c$       | 
|---------------|----------------|---------------| 
| Block Masks   | $0.247  \ \small{\pm \ 0.000} $          | $0.259  \ \small{\pm \ 0.000}$         | 
| SA                    | $0.222 \ \small{\pm \ 0.008}$  | $0.237 \ \small{\pm \ 0.008}$ | 
| SLATE               | $0.310 \ \small{\pm \ 0.004}$  | $0.324 \ \small{\pm \ 0.004}$ | 
| Rotating Features | $0.282 \ \small{\pm \ 0.006}$ | $0.320 \ \small{\pm \ 0.006}$ |
| DINO k-means  | $0.363  \ \small{\pm \ 0.000}$ | $0.405  \ \small{\pm \ 0.000}$ |  
| DINO CAE      | $0.329  \ \small{\pm \ 0.009}$ | $0.374  \ \small{\pm \ 0.010}$ |  
| DINOSAUR MLP  | $0.395  \ \small{\pm \ 0.000} $  | $0.409  \ \small{\pm \ 0.000}$ | 
| **Ours:** |                |               | 
| SA MLP (w/ DINO)  |       $0.384 \ \small{\pm \ 0.000}$         |      $0.397 \ \small{\pm \ 0.000}$         | 
| SA MLP (w/ DINO)$^{\ddagger}$  |       **$0.400 \ \small{\pm \ 0.000}$**         |      **$0.415 \ \small{\pm \ 0.000}$**         | 
| PSA MLP (w/ DINO)  |       **$0.389 \ \small{\pm \ 0.009}$**         |      **$0.422 \ \small{\pm \ 0.009}$**         | 
| PSA MLP (w/ DINO)$^{\ddagger}$  |     **$0.405 \ \small{\pm \ 0.010}$**         |      **$0.436 \ \small{\pm \ 0.011}$**         | 
| PSA Transformer (w/ DINO)$^{\star}$  |     **$\mathbf{0.435} \ \small{\pm \ 0.01}$**         |      **$\mathbf{0.499} \ \small{\pm \ 0.01}$**         | 

$^{\ddagger}$ Using slot attention masks rather than decoder alpha masks for evaluation.

$^{\star}$ Trained for $\approx$15K steps only due to time constraints (250K are needed). 

The results show that PSA is competitive with SA at scale.

Finally, we have also included basic illustrations of compositional samples from the aggregate posterior on both CLEVR and Objects-Room datasets to verify our theory. Please note that these models are quite small and were not optimized for sample quality since they were used primarily to measure slot identifiability across runs in our main experiments.

[r2] Wu, Z., Hu, J., Lu, W., Gilitschenski, I. and Garg, A., 2023. Slotdiffusion: Object-centric generative modeling with diffusion models. Advances in Neural Information Processing Systems, 36, pp.50932-50958. 

[r3] Löwe, S., Lippe, P., Locatello, F. and Welling, M., 2024. Rotating features for object discovery. Advances in Neural Information Processing Systems, 36.

[r4] Kakogeorgiou, I., Gidaris, S., Karantzalos, K. and Komodakis, N., 2024. SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22776-22786).

Meta-review: The authors introduces a novel approach to solving the problem of identifiability in object-centric representations by proposing a probabilistic variant of Slot Attention that leverages a Gaussian Mixture Model (GMM) on the latent space. This method effectively shifts the focus from assumptions about the generator to the latent distribution, providing a robust solution for identifying ground-truth object representations. The authors support their approach with solid theoretical proof and validate it through both toy data and high-dimensional image experiments.

The main issue raised by the reviewers is the lack of additional large-scale experiments. While such experiments would further strengthen the paper, I believe the theoretical contributions on the slot identifiability and the current experimental results already represent a substantial and valuable contribution, warranting acceptance.
Decision: Accept (poster)