Grounded Object-Centric Learning
Authors: Avinash Kori
Francesco Locatello
Fabio De Sousa Ribeiro
Francesca Toni
Ben Glocker
Venue: ICLR 2024 poster (2024)
Keywords: object-centric representation learning
the binding problem
the grounding problem
slot attention
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose conditional slot attention model to learn specialized slots which bind to specific object types and remain invariant to identity-preserving changes in object appearance.

Abstract: The extraction of object-centric representations for downstream tasks is an emerging area of research. Learning grounded representations of objects that are guaranteed to be stable and invariant promises robust performance across different tasks and environments. Slot Attention (SA) learns object-centric representations by assigning objects to *slots*, but presupposes a *single* distribution from which all slots are randomly initialised. This results in an inability to learn *specialized* slots which bind to specific object types and remain invariant to identity-preserving changes in object appearance. To address this, we present *Conditional Slot Attention* (CoSA) using a novel concept of *Grounded Slot Dictionary* (GSD) inspired by vector quantization. Our proposed GSD comprises (i) canonical object-level property vectors and (ii) parametric Gaussian distributions, which define a prior over the slots. We demonstrate the benefits of our method in multiple downstream tasks such as scene generation, composition, and task adaptation, whilst remaining competitive with SA in object discovery.

Published as a conference paper at ICLR 2024
GROUNDED OBIECT-CENTRIC LEARNING
Avinash Kori 1, Francesco Locatello : i Fabio De Sousa Ribeiro Francesca Toni' t, and Ben Glocker
Imperial College London," Institute of Science and Technology Austria a. 1.kori211 221e211@20 imperial.aet ackkuk
ABSTRACT
The extraction of modular object-centric representations for downstream tasks is emerging area of research. Learning grounded representations of objects that are guaranteed to be stable and invariant promises robust performance across different tasks and environments. Slot Attention (SA) learns object-centric representations by assigning objects slots, but presupposes single distribution from which all slots are randomly initialised. This results in an inability to learn specialized slots which changes
bind to specific object types and remain invariant object appearance. To address this, we present Conditional Slot Attention (CoSA) using a novel concept of Grounded Slot Dictionary (GSD) inspired by vector quantization. Our proposed GSD comprises (i) canonical object-level property vectors and (ii) parametric Gaussian distributions, which define a prior over the slots. We demonstrate the benefits of our method in multiple downstream tasks such as scene generation, composition, and task adaptation, whilst remaining competitive with SA in popular object discovery benchmarks.
INTRODUCTION
A key step in aligning Al systems with humans amounts to imbuing them with notions of objectness akin to human understanding (Lake et al., 2017). It has been argued that humans understand their environment by subconsciously segregating percepts into object entities (Rock, 1973; Hinton, 1979; Kulkarni et al., 2015; Behrens et al., 2018). Objectness is a multifaceted property that can be characterised as physical, abstract, semantic, geometric, via spaces and boundaries (Yuille & Kersten, 2006; Epstein et al., 2017). The goal of object-centric representation learning is to equip systems with notion of objectness which remains stable, grounded and invariant different environments, such that the learned representations are disentangled and useful (Bengio et al., 2013).
The grounding problem refers to the challenge of connecting such representations real-world objects, their function, and meaning (Harnad, 1990). It can be understood as the challenge of learning abstract, canonical representations of objects.' The binding problem refers to the challenge of how objects are combined into a single context (Revonsuo & Newman, 1999). Both of these problems affect system's ability to understand the world in terms of symbol-like like entities, or true factors of variation, which are crucial for systematic generalization (Bengio et al., 2013; Greff et al., 2020).
Greff et al. (2020) proposed a functional division of the binding problem into three concrete sub-tasks: (i) segregation: the process of forming grounded, modular object representations from raw input data; (ii) representation: the ability to represent multiple object representations in a common format, without interference between them; (iii) compositionality the capability dynamically relate and compose object representations without sacrificing the integrity of the constituents. The mechanism of attention is believed to be crucial component in determining which objects appear be bound together, segregated, and recalled (Vroomen & Keetels, 2010; Hinton, 2022). Substantial progress in object-centric learning has been made recently, particularly in unsupervised objective discovery, using iterative attention mechanisms like Slot Attention (SA) (Locatello et al., 2020) and many others (Engelcke et al., 2019; 2021: Singh al., 2021; Chang et al., 2022; Seitzer al., 2022).
Work done when the author was a part of AWS.
Repprssentttinns learned by neural networks directly grounded in their input data, unlike classical notions of symbols whose definitions often subject human interpretation.

---

Published as a conference paper at ICLR 2024
Despite recent progress, several major challenges in the field of object-centric representation learning remain, including but not limited to: (i) respecting object symmetry and independence, which requires isolating individual objects irrespective of their orientation, viewpoint and overlap with other objects; (ii) dynamically estimating the total number of unique and repeated objects in given scene; (iii) the binding of dynamic representations of an object with more permanent (canonical) iden- tifying characteristics of its type. As Treisman (I (1999) explains, addressing the challenge (iii) is a pre-requisite for human- numan-like perceptual bind- ing. In this work, we argue this view of bind- ing lemporary object representations (so-called "object files") to their respective permonent ob- ject types, is a matter of learning a vocabulary of grounded, canonical object representations. Therefore, our primary focus is on addressing challenge (iii), which we then leverage help tackle challenges (i) and (ii). To that end, our proposed approach goes beyond standard slot attention (SA) (Locatello et al., 2020), as shown in Figure 1 and described next.
B1,
Grounded
Represertaiveel
Figure The leftmost block illustrates various scenes within environment, each featuring dif- ferent object instances. In the middle block, we depict our acquired grounded vocabulary of canon- ical object-centric representations, effectively cap- turing object types. The rightmost block displays collection of specialized slot distributions associ- ated with their respective canonical representations. These distributions are employed to sample initial slots for object instances within scene. This pro- cess, known as object binding, is elucidated by the placeholder slots 81 and $2. These slots are linked to specific object types in the environment and un- dergo further refinement. Notably, this differs from the SA, which relies random slot initialization and does not encourage slots to remain invariant in the face of identity- preserving changes in object appearance.
Contributions. Slot attention learns compos- able representations using dynamic inference- level binding scheme for assigning objects to slots, but presupposes a single distribution from which all slots are randomly initialised. This re- sults in an inability to learn specialized slots that bind to specific object types and remain invariant to identity- preeeervng changes in object appear- ance. To address this, we present Conditional Slot Attention (CoSA) A) using a novel Grounded Slot Dictionary (GSD) inspired by vector quan- tization techniques. Our GSD is grounded in the input data and consists of: (i) canonical object- level property vectors which are learnable in an unsupervised fashion: and (ii) parametric Gaussian distributions defining a prior over object slots. In summary, our main contributions are as follows:
single distribution for
We propose the Grounded Slot Dictionary for object-centric representation Iearning, which unlocks the capability of learning specialized slots that bind to specific object types;
(ii) We provide a probabilistic perspective unsupervised object-centric dictionary learning and derive a principled end-to-end objective for this model class using variational inference methods;
(iii) We introduce a simple strategy for dynamically quantifying the number of unique and repeated objects in given input scene by leveraging spectral decomposition techniques;
(iv) Our experiments demonstrate the benefits of grounded slot representations in multiple tasks such as scene generation, composition and task adaptation whilst remaining competitive with standard slot attention in object discovery-based tasks.
2 RELATED WORK
Object Discovery. Several notable works in the field object discovery, including Burgess et al. (2019); Greff et al. (2019); Engelcke et al. (2019), employ an iterative variational inference approach (Marino et al., 2018), whereas Van Steenkiste et al. (2020); Lin et al. (2020) adopt more of a generative perspective. More recently, the use of iterative attention mechanisms in object slotd models has garnered significant interest (Locatello et al., 2020; Engelcke et al., 2021; Singh et al., 2021; Wang et al., 2023; Singh et al., 2022; Emami et al., 2022). The focus of most of these approaches remains centered around the disentanglement of slot representations, which can be understood as tackling the segregation and representation aspects of the binding problem (Greff et al., 2020). Van Steenkiste
2

---

Published as a conference paper at ICLR 2024
et al. (2020); Lin et al. (2020); Singh et al. (2021) focus primarily tackling the composition aspect of the binding problem. either in a controlled setting or with a predefined set of prompts. However, unlike our approach, they do not specifically learn grounded symbol-like entities for scene composition. Another line of research tackling the binding problem involves capsules (Sabour et al., 2017; Hinton et al., 2018; Ribeiro al., 2020). However, these methods face scalability issues and typically used for discriminative learning. Kipf et al. 1; Elsayed et al. (2022) perform conditional slot attention with weak supervision, such as using the center of mass of objects in the scene or object bounding boxes from the first frame. In contrast, our approach learns specialized slot representations fully unsupervised. Our proposed method builds primarily upon slot attention (Locatello al., 2020) by introducing a grounded slot dictionary to learn specialized slots that bind to specific object types in an unsupervised fashion.
Discrete Representation Learning. Van Den Oord et al. (2017) propose Vector Quantized Vari- ational Autoencoders (VQ- VAE) to learn discrete latent representations by mapping continuous latent space to fixed number of codebook embeddings. The codebook embeddings are learned by minimizing the mean squared error between continuous and nearest codebook embeddings. The learning process
also be improved upon using the Gumbel softmax trick Jang al. (2016); Maddison et al. (2016). The VQ-VAE has been further explored for text-to-image generation (Esser al., 2021; Gu et al., 2021; Ding al., 2021; Ramesh al., 2021). One major challenge in the quantization approach is effectively utilizing codebook vectors. Yu et al. 2001);; Santhirasekaram al. (2022b;a) address this by projecting the codebook vectors onto Hyperspherical and Poincare manifold. Träuble et al. (2022) propose a discrete key-value bottleneck layer and show the effect of discretization on non-i.i.d samples, generalizability, and robustness tasks. We take inspiration from their approach in developing our grounded slot dictionary.
Compositional Visual Reasoning. Using grounded object- like representations for compositionality is said to be fundamental for realizing human-level generalization (Greff et al., 2020). Several notable works propose data-driven approaches to first learn object-centric representations, and then use symbol-based reasoning wrappers on top (Garcez et al., 2002; 2019; Hudson & Manning, 2019; Vedantam et al., 2019). Mao et al. (2019) introduced an external. learnable reasoning block for extracting symbolic rules for predictive tasks. Yi et al. (2018); Stammer et al. (2021) use visual question-answeringg (VQA) for disentangling object-level representations for downstream reasoning tasks. Stammer et al. (2021) in particular also base their approach object-centric representations, which are then further used for set predictions and rule extraction. Unlike most of these methods, which use some form of dense supervision either in the form of object information or natural language question answers, in this work, we train the model for discriminative tasks and use the emerging properties learned as a result of slot dictionary and iterative refinement for learning rules for the classification.
slot attention module to learn
BACKGROUND
Discrete Representation Learning. Let - =x E R.Nxd denote an encoded representation of an input datapoint x consisting of N embedding vectors in RRd, Discrete representation learning aims to map each z to set of elements codebook vectors are randomly initialized at the start of training and are updated to align with z. The codebook is initialised a specific range based on the choice of sampling as detailed below:
codebook G consisting of MM embedding vectors. The
(i) Euclidean:
The embeddings are randomly initialized between 1/M,,1/M] as described by Van Den Oord et al. (2017). The discrete embeddings ă are then sampled with respect the Euclidean distance as follows: ž arg ming, ||z G A G, E G.
(ii) Cosine: The embeddings are initialized on a unit hypersphere (Yu et al., 2021; Santhirasekaram et al.. 2022b). The representations z are first normalized to have unit norm: 2 = z/|lz|| The discrete embeddings are sampled following:
arg maxe, (z. 6,), Y G, , E 6, where (...) denotes vector inner product. The resulting discrete representations are then upscaled as ž |z|].
(iii) Gumbel: The embeddings are initialised similarly to the Euclidian codebook. The repre- sentations projected onto discrete embeddings in G by measuring the pair-wise simi- larity between the z and the M codebook elements. The projected vector is used in the Gumbel-Softman trick (Maddison al., 2016; Jang et al., 2016) resulting in the continuous
3

---

Published as a conference paper at ICLR 2024
Spectod
Oostttbtbtioo Selcctoon
Nipuon)
slots
((pa
Property
Eeceder
(q(
Prodictoo
Taak
Class
Varting
Figure 2: CoSA is an unsupervised autoencoder framework for grounded object-centric representa- tion learning, and composed of five unique sub-modules. all the distinct objects in a scene using spectral decomposition.
I The abstraction module extracts ② The grounded slot dictionary (GSD) module maps the object representation to grounded (canonical) slot representations, which are then used for sampling initial slot conditions.
3 The refinement module uses slot attention to itera- The discovery module maps the slot representations to observational space (used for object discovery and visual scene composition). (5) The reasoning module involves object property transformation and the prediction model (used for reasoning tasks).
tively refine the initial slot representations.
approximation of one-hot representation given by soft.axx(B5) 24)/T), where g, is sampled from Gumbel distribution, and r is a temperature parameter.
Slot Attention. Slot attention (Locatello et al., 2020) takes a set of feature embeddings z E Nxx, input, and applies iterative attention mechanism produce K object-centric representations called slots E IR K xx,, Let 2,, KCy and V denote query, key and value projection networks with parameters 8, y and respectively acting on z. To simplify our exposition later on, let and g be shorthand notation for the slot update and attention functions respectively, defined as:
g(q,
Mi
kq
f(A.v) = V Aij Em 96. k)2s
and g(q, k)
M =
(I)
E Ma
dg
where q = (2) RKKd, KG(z)) E RNdd and v -Vo(2) Rxd,, are the query, key and value vectors respectively. The attention matrix is denoted by A RNN Unlike atlf-attention (Vaswani et al., 2017), the queries in slot attention are function of the slots ~ A(ss; μt, o) E Kxx,, and are iteratively refined over T attention iterations (see refinement module in Fig. 2). The slots are randomly initialized at t 0. The queries iteration are given by q' 2, and the slot update process can be summarized as: g [(o k), v). Lastly, Gated Recurrent Unit (GRU), which we denote by Ho. applied the slot. representations g++ at the end of each SA iteration, followed by generic MLP skip connection.
4 UNSUPERVISED CONDITIONAL SLOT ATTENTION: FORMALISM
In this section, we present our proposed conditional slot attention (CoSA) framework using novel grounded slot dictionary (GSD) inspired by vector quantization. CoSA is an unsupervised autoencoder framework for grounded object-centric representation learning, and it consists of five sub-modules in total (as depicted in Fig. 2), each which we will describe in detail next.
Notation. Let D C X' x y denote dataset of images X' R and their properties y y x 012 R.Y There are properties in total, where ) corresponds to the space of image labels, and 312 consists of additional information about the images like object size, shape, location, object material, etc. Let &. : X + denote an encoder and 인 Z + X denote decoder, mapping to and from latent space 2. Further, let Ø Z y denote classifier for reasoning tasks.
A Astracciion Module. The purpose of the abstraction module is enable dynamic estimation of the number of slots required to represent the input. Since multiple instances of the same object can appear in a given scene, we introduce the concept of an abstraction function denoted as: Nxd, + N.Nx xx,, which maps N input embeddings to Ñ output embeddings representing N distinct objects. This mapping learns the canonical vocabulary of unique objects present in
4

---

Published as a conference paper at ICLR 2024
environment, shown in Fig. 1. To accomplish this, we first compute the latent covariance matrix C' = z z..z z² E RNN, where z = D. (x) R xd, are feature representations of an input x. We then perform spectral decomposition resulting in C = VAVT, where V and diag(A) are the eigenvector and eigenvalue matrices, respectively. The eigenvectors V correspond to the directions of maximum variation in C, ordered according the eigenvalues, which represent the respective magnitudes of variation. We project z onto the top N principal vectors, resulting in abstracted, property-level representation vectors in new coordinate system (principal components).
The spectral decomposition yields high eigenvalues when: (i) single uniquely represented object spans multiple input embeddings in z (i.e. a large object is present in x); (ii) a scene contains multiple instances of the same object. To accommodate both scenarios, we assume that the maximum area spanned by an object is represented by the highest eigenvalue X, (excluding the eigenvalue representing the background). Under this assumption, we can dynamically estimate the number of slots required to represent the input whilst preserving maximal explained variance. To that end, we first filter out small eigenvalues by flooring them A; = [A.]] then compute a sum of eigenvalue ratios مو -
w.r.t. A resulting in a total number of slots required: K = dynamically estimates the number of object instances, relative to the 'largest' object in the input. Note that we do not apply positional embedding to the latent features at this stage, as it encourages multiple instances of the same object to be uniquely represented, which we would like to avoid.
Dx.A>s] Intuitively, this ratio
GSD Module. To obtain grounded object-centric representations that connect scenes with their fundamental building blocks (i.e. object rypes (Treisman, 1999)), we introduce GSD, as defined 1.
Definition 1. (Grounded Slot Dictionary) grounded slot dictionary 6 consists of: (i) an object- centric representation codebook G¹; (ii) a set of slot prior distributions & The number of objects M in the environment is predefined, and the GSD G is constructed as follows:
6 = (?) M
= arg min (A)
6 = (SP:
(2)
6/eet
where ă E R xdd
xd, denotes the N codebook vector representations closest to the output of the abstraction function A(z) R Nxd= applied to the input encoding o. (x) E R Nxd.
As per Equation 2, the codebook 6 induces a mapping from input-dependent continuous repre- sentations z' = A(*x(x) to a set of discrete (canonical) object-centric representations ă via distance function d(-,)) (judicious choices for d((,-) are discussed in Section 3). Each slot prior
distribution p(s?) = N (sf: μú o?? in & is associated with one of the M object representations in the codebook & These priors define marginal distributions over the initial slot representations s We use diagonal covariances and the learn the parameters {M4, o 2 Jizl during training.
Iterative Refinement Module. After randomly sampling initial slot conditions from their
respective marginal distributions in the grounded slot dictionary s ~ 6(z) E Kxxd, we iteratively refine the slot representations using slot attention described in Section 3 and Algorithm 1. The subset of slot priors we sample from for each input corresponds to the K respective codebook vectors which are closest the output of the abstraction function x' (x))(x)), outlined in Equation 2.
Remark (Slot Posterior): The posterior distribution of the slots s at iteration T given x is:
T
p(s" x) =
II
Ho (s' (oold")..)vv)))
(3)
[=1
where 5(-) is Dirac delta distributed given randomly sampled initial slots from their marginals 8 ~ p(s") 2) = ITK N (s); ()) associated with the codebook vectors ž ~ q(ū x). The distribution over the initial slots s" induces distribution over the refined slots s"" One important aspect worth re-stating here is that the sampling of the initial slots {s? ~ p(s, } depends on the indices of the associated codebook vectors ž C 6', as outlined Definition 1, and is not conditioned on the values of themselves. In proposition in App. demonstrate the convergence of slot prior distributions to the mean of true slot distributions, using the structural identifiability of the model under the assumptions 1-5.
K @
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 Conditional Slot Attention (CoSA).
1: Input: inputs z = +x(x)= R.Vxd, Kz(2) E Nxd, and v Vo(2) E Rxddd
2: Spectral Decomposition:
3: 2 z² = =VAVT
V: eigenvectors, A = diag( A²): eigenvalues b z': project onto principle components Dynamically estimate number of slots
4: z' = VT
5: = Ei= [
6: for i O....,
> R Monte Carlo samples b Sample K initial slots from GSD b. Refine slots over I' attention iterations > Update slot representations
7: slots (z²) E Kxd,
8: for = I.....T
9: slots; = (g (Q.y (LayerNorm (slots, )), k) .v)
10: slots; += MLP (LayerNorm (Ho (slots, slots()))
D. GRU update & skip connection D MC estimate
11: return >, slots:/R
4 Discovery Module. For object discovery and scene composition tasks, we need to translate object-level slot representations into the observational space of the data. To achieve this, we use spatial broadcast decoder (?), as employed by IODINE and slot attention (Greff et al., 2019; Locatello al., 2020). This decoder reconstructs the image x via softmax combination of K individual reconstructions from each slot. Each reconstructed image x, = Φ d2(ss) consists of four channels: RGB plus a mask. The masks are normalized over the number of slots K using softmax, therefore they represent whether each slot is bound to each pixel in the input.
We can now define generative model of x which factorises as: p(x,s", 2) p(x s")p(s" E)p(Z), recalling that s := s..... 8% are the initial slots at iteration / = 0, and & are our discrete latent variables described in Definition For training, we derive a variational lower bound on the marginal log- regliikeihhodd of x, a.k.a. the Evidence Lower Bound (ELBO), as detailed in Proposition 1.
Proposition (ELBO for Object Discovery). Under a categorical distribution over our discrete latent variables and the object-leved prior distributions p(s)) 'N' (s); μsj of) contained in G?? we show that variational lower bound on the marginal log- likelihoodd of x can be expressed as:
log p(x) > (g (mxp(( ()) [(x s)] DKL (q (ž x)) p(z)) ELBO(x),
where s == II= Ho (s'-1 f(g( 1 k),v)) v)) denotes the output of the iterative refinement proce- dure described in Algorithm applied to the initial slot 5 representations.
The proof is given in App. B.
The reasoning tasks we consider consist classifying images into rule- based classes y E J4, while providing a rationale behind the predicted class. For example, class A might be something like "large cube and large cylinder", and rationale for a prediction emerges directly from the slots binding to an input cube and cylinder. Refer to App. C for more details.
Reasoning Module.
To achieve this, we define a set of learned functions G = {6%,.... & which map refined slot representations s'=T' to M rationales for each object type in the grounded slot dictionary G. This expands the dictionary to three elements: G = { @ 6²)] The reasoning task predictor Φ, combines K rationales extracted from slot representations an input, which we denote by p, = = E RDD,,, and maps them to the rule-based class labels. The optimization objective for this reasoning task is variational lower bound on the conditional lik-likelhhood, in Proposition 2.
Proposition (ELBO for Reasoning Tasks). Under a categorical distribution over our discrete latent variables and the object-level prior distributions p(s.) N (sf: ph, of) contained in G?? the variational lower bound on the conditional log- ligliikl of y given x is given by:
log ply xx Z (Ezag(ax)) ~>pis
[log ply
x)] DKL. (g(Z) x) p(Z))
(5)
noting that conditioning on xs is equivalent to conditioning the predicted rationales p, since the latter are deterministic functions of the former.
The proof is similar to proposition I and is given in App. B for completeness.

---

Published as a conference paper at ICLR 2024
Table 1: Foreground ARI on CLEVR, Tetrominoes, ObjectsRoom, and COCO datasets. for all baseline models and CoS A-cosine variant. For COCO we use the DINOSAUR variant (Seitzer et al., 2022) (with VYT-S16 as feature extractor) as baseline model and use the same architecture for IMPLICT and COSA (adaptation details in App. H.3).
METHOD
TETROMINOES
CLEVR
ORJECTSROOM
COCO
SA (Locatello et al., 2020)
0,99 1 0.005
0.93 1 0.002
0.78 1 0.02
BlockSlot (Singh et al., 2022)
0.99 1 0.001
0.94 1 0.001
0.77 0.01
SlotVAF (Wang et al., 2023)
0.79 0.01
DINOSAUR (Seitzer et al., 2022)
0.28 + 0.02
IMPLICIT (Chang et al., 2022)
0.99 + 0.001
0.93 0.001
0.78 = 0.003
0.28 + 0.01
COSA
0.99 + 0.001
0.96 + 0.002
0.83 + 0.002
0.32 + 0.01
(a) Embedding-7 7 (6!) (b) Embedding- 14 (61.) (c) Embedding 25 (61s) (d) Embedding-S5 (G)s)
Figure 3: GSD binding: we can observe that cheeks being bound to foreheud to 614, eyes to (G25). and facial hair to 64.5 illustrating the notion of object binding achieved in GSD, in the case
of bitmoji dataset for CoSA A model trained with cosine sampling stratergy.
5 EXPERIMENTS
Our empirical study evaluates COSA on variety of popular object discovery (Table 1, 2) and visual reasoning benchmarks (Table 3). In an object discovery context, we demonstrate our method's s ability to: (i) dynamically estimate the number of slots required for each input (Fig. 4); (ii) map the grounded slot dictionary elements to particular object types (Fig. 3): (iii) perform reasonable slot composition without being explicitly trained to do (Fig. 5). In terms of visual reasoning, the benefits of grounded slot representations for generalization across different domains, and for improving the quality of generated rationales behind rule-based class prediction (Table 3). We also provide detailed ablation studies on: (i) the on choice of codebook sampling (App. H); (ii) different codebook regularisation techniques to avoid collapse (App. G); (iii) the choice of abstraction function (App. I). For details on the various datasets we used, please refer to App. C. For training details, hyperparameters, initializations and other computational aspects refer App. M.
show
5.1 CASE STUDY 1: OBJECT DISCOVERY & COMPOSITION
In this study, we provide a thorough evaluation of our framework for object discovery across multiple datasets, including: CLEVR (Johnson et al., 2017), Tetrominoes (Kabra et al., 2019), Objects-Roon (Kabra et al., 2019), Bitmoji (Mozafari, 2020), FFHQ (Karras et al., 2020), and COCO (Lin et al., 2014). In terms of evaluation metries, we use the foreground-adjusted Rand Index score (ARI) and compare our method's 's results with standard SA (Locatello et al., 2020), SlotVAE (Wang et al., 2023), BlockSlot (Singh et al.. 2022), IMPLICIT (Chang et al., 2022) and DINO (Seitzer et al., 2022). We evaluated the majority of the baseline models on the considered datasets using their original implementations, with the exception of SlotVAE, where we relied on the original results from the paper. For DINOSAUR, we specifically assessed its performance on real-world COCO dataset, following the code adaptation outlined in App. H.3. Additionally, we measured the reconstruction score using MSE and the FID (Heusel al., 2017). We analyse three variants of CoSA: Gumbel, Euclidian, and Cosine, corresponding to three different sampling strategies used for the GSD. Detailed analysis and ablations on different types of abstraction function are detailed in App. I.
The inclusion of GSD provides two notable benefits: (i) it eliminates the need for a hard requirement of parameter K during inference, and (ii) since slots are initialized with samples from their grounded counterparts, this leads to improved initialization and allows us to average across multiple samples, given that the slots are inherently aligned. These design advantages contribute the enhanced
7

---

Published as a conference paper at ICLR 2024
Table 2: Object discovery results on multiple benchmarking datasets and previously existing methods. We measure both MSE and FID metrics to compare the results.
METHOD
CLEVR
TETROMINOES
SDEECTS-ROOM
BITMOJI
FFHQ
MSE
FID
MSE
FID 4
MSE 4
FID
MSE 4
FID
MSE
FID 1
SA
6.37
38.18
1.49
3.81
7.57
38.12
14.62
12.78
55.14
54.95
Block-Slot
3.73
34.11
0.48
0.42
5.28
36.33
10.24
11.01
52.16
41.56
IMPLICIT
3.95
38.16
0.59
0.41
5.56
36.48
9.87
10.68
47.06
49.95
COSA
3.14
29.12
0.42
0.41
4.85
28.19
8.17
9.28
33.37
36.34
Figure 4: Object discovery: reconstruction quality and dynamic slot number selection for COSA- COSINE CLEVR and Bitmoji, with an MAE of 2.06 over slot number estimation for CLEVR.
representational quality for object discovery, as evidenced in Table 1 and 2, while also enhancing generalizability which will be demonstrated later. To evaluate the quality of the generated slots without labels, we define some additional metrics like the overlapping index (OPI) and average slot FID (SFID) (ref. Section D, Table 9).
Fig. 4 showcases the reconstruction quality of CoSA and its adaptability to varying slot numbers. For additional results on all datasets and a method-wise qualitative comparison, see App. Regarding the quantitative assessment of slot number estimation, we calculate the mean absolute error (MAE) by comparing the estimated slot count the actual number of slots. Specifically, the MAE values for the CLEVR, Tetrominoes, and Objects-Roon Room datasets are observed to be 2.06, 0.02, and 0.32, respectively. On average, dynamic slot number estimation reduces 38 %, 26 % and 23% FLOPs on CLEVR, Objects-Room Room, and Tetrominoes opposed to fixed number of slots. In Fig. 3, we illustrate the grounded object representations within GSD, which are obtained by visualising the obtained slots result of sampling particular element in the GSD. Furthermore, generating novel scenes using the learned GSD, although out models were not designed explicitly for this purpose. To that end, sample slots from randomly selected slot distributions and decode them to generate new scenes. The prompting in our approach relies solely on the learned GSD, and do not need to construct prompt dictionary like SLATE (Singh al., 2021). To evaluate scene generation capabilities, we compute FID and SFID scores on randomly generated scenes across all datasets; please refer to App. Table 14. Fig. 5 provides visual representations images generated through slot composition on CLEVR, Bitmoji, and Tetrominoes datasets.
explore the possibility of
5.2
CASE STUDY 2: VISUAL REASONING & GENERALIZABILITY
As briefly outlined in the reasoning module description (Section 4), the visual reasoning task involves classifying images into rule-based classes, while providing a rationale behind the predicted class. To evaluate and compare our models, we use the F1-score, accuracy, and the Hungarian Matching Coefficient (HMC), applied to benchmark datasets: CLEVR- Hanss, SLEVR-Hand (Stammer et al., 2021), FloatingMNIST-2, and FloatingMNISTT-3 CLEVR-Hans includes both confounded validation set and non-confounded test set, enabling us to verify the generalizability of our model. FNNAtingMNIST (FMNIST) is variant of the MNIST (Deng, 2012) dataset with three distinct reasoning tasks, showcasing the model's adaptability across domains. Further details on the datasets given in App. C.
To facilitate comparisons, we train and evaluate several classifiers, including: (i) the baseline classifier; (ii) a classifier with vanilla SA bottleneck; (iii) Block-Slot; and (iv) our CoSA classifiers. The main comparative results are presented in Table 3, and additional results for FloatingMNIST3 and CLEVR-Hans datasets available in the App. (Tables 16-19). We conduct task adaptability
8

---

Published as a conference paper at ICLR 2024
Table Accuracy and Hungarian matching coefficient (HMC) for reasoning tasks on addition and subtraction variant of FMNIST2 dataset. Here, the first and third pair of columns correspond to the models trained and tested on the FMNIST2-Ad and FMNIST2-Sun datasets, respectively, while the second and fourth pair correspond to few-shot (k-1000))) 100) adaptability results across datasets.
METHOD
FMNIST2-ADDE
FMNIST2-SUU SUB
FMNIST2-SUB source
FMNIST2-ADE pargge
Acc
HMC J
Acc
F1
Acc f
HMC
Acc1 1
F1
CNN
97.62
10.35
10.05
98.16
12.35
9.50
97.33
0.14
11,06
09:40
97.41
0.13
08.28
7.83
Block-Slot
98.11
0.12
09.71
09.10
97.42
0.14
09.61
8.36
COSA
98.12
0.10
60.24
50.16
98.64
0.12
63.29
58.29
Figure 5: Top and bottom left illustrates the randomly prompted slots and their composition. Right demonstrates object discovery results of CoSA on COCO dataset.
experiments assess the reusability of grounded representations in our learned GSD. To that end, we create multiple variants the MNNSS dataset, introducing addition, subtraction, and mixed tasks. Initially, train the model with specific objective, which we consider the source dataset and assess its capacity to adapt to other target datasets by fine-tuning the task predictor layer through k-shot training. The adaptability results for FloatingMNIST--2 are provided in Table 3, and results for FloatingMNIST-3 are detailed in the App. in Table 19. Results for mixed objectives are also discussed in App. L. Fig. 28-30 depict the few-shot adaptation capabilities over multiple iterations on the FloatingMNISTZ, FloatingMNIST3, and CLEVR- datasets.
Overall we observe that GSD helps in (i) better capturing the rational for the predicted class, as illustrated by HMC measure; and (ii) learning reusable properties leading to better generalisability as demonstrated by accuracy and Fl measure in k-shot adaptability tasks; detailed in Table 3, 16- 18.
CONCLUSION
In this work, we propose conditional slot attention (CoSA) using a grounded slot dictionary (GSD). CoSA allows us bind arbitrary instances of object to a specialized canonical representation, encouraging invariance to identity-preservin
changes. This is in contract to vanilla slot attention which presupposes single distribution from which all slots are randomly initialised. Additionally, our proposed approach enables dynamic estimation of the number of required slots for a given scene, saving up 38% of FLOPs. We show that the grounded representations allow us perform random slot composition without the need of constructing a prompt dictionary as in previous works. We demonstrate the benefits of grounded slot representations in multiple downstream tasks such as scene generation, composition, and task adaptation, whilst remaining competitive with SA in popular object discovery benchmarks. Finally, we demonstrate adaptability of grounded representations resulting up to 5x relative improvement in accuracy compared to SA and standard CNN architectures.
The main limitations of the proposed framework include: (i) limited variation slot-prompting and composition; (ii) assumption of no representation overcrowding: (iii) maximum object area assumption. (App. N). An interesting future direction would be to use a contrastive learning approach to leam dictionary for disentangled representation of position, shape, and texture. resulting finer control over scene composition. From a reasoning point of view, it would be interesting to incorporate background knowledge in the framework, which could aid learning exceptions rules.

---

Published as a conference paper at ICLR 2024
7 BROADER IMPACT
Our proposed method allows us to learn conditional object-centric distributions, which have wide applications based the selected downstream tasks. Here, we demonstrate its working multiple controlled environments. The scaled versions with lot more data and compute can generalize across domains resulting in foundational models for object-centric representation learning; the negative societal impacts (like: realistic scene composition/object swapping from a given scene) for such systems should carefully be considered, while more work required to properly address these concerns. As demonstrated in the reasoning task, our method adapts easily to different downstream tasks; we can guarantee the human understandability of learned properties in the future, our method might be step towards learning interpretable and aligned AI models.
8 ACKNOWLEDGEMENTS
A. Kori is supported by UKRI (grant agreement no. EP/S023356/1), as part of the UKRI Centre for Doctoral Training in Safe and Trusted AI. F.D.S. Ribeiro and B. Glocker received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 757173, project MIRA, ERC-2017-STG).
REFERENCES
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-iivarranc--vvaraeee regularization for self- superviee learning. arXiv preprint arXiv:2105.04906. 2021.
Timothy EJ Behrens, Timothy H Muller, James CR Whittington, Shirley Mark, Alon B Baram, Kimberly L Stachenfeld, and Zeb Kurth-Nelson, What a cognitive map? organizing knowledge for flexible behavior. Neuron, 100(2):490-500 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1798)11798-1888, 2013.
Jack Brady, Roland S Zimmermann, Yash Sharma. Bernhard Schölkopf. Julius von Kügelgen, and Wieland Brendel. Provably learning object-centric representations. arXiv preprint arXiv:2305. 14229, 2023.
Christopher P Burgess, Loie Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv 2019.
preprint
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the IEFF/CVF international conference on computer vision, pp. 9650-9660, 2021.
Michael Chang, Thomas L Griffiths, and Sergey Levine. Object representations as fixed points: Train- ing iterative refinement algorithms with implicit differentiation. arXiv preprint arXiv:2207.00787 2022.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-144 2012.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34, 2021.
Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. Advances in Neural Information Processing Systems, 35:28940-28955 2022.
Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan. Slot order matters for compositional scene understanding. arXiv preprint arXiv:2206.01370 2022.
10

---

Published as a conference paper at ICLR 2024
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Gener- ative scene inference and sampling with object-centric latent representations. arXiv preprint arXiv:1907.13052, 2019.
Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. Genesis-v2: Inferring unordered object representations without iterative refinement. Advances in Neural Information Processing Systems, 34:8085-809) 2021.
Russell Epstein, Eva Zita Patai, Joshua Julian, and Hugo Spiers. The cognitive map in humans: spatial navigation and beyond. Nature neuroscience, 20(1
2017.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, (10066394::104-12220 2018.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Artur d' Avila Garcez, Marco Gori, Luis C Lamb, Luciano Serafini, Michael Spranger, and Son N Tran. Neural-symbolic computing: An effective methodology for principled integration of machine learning and reasoning. arXiv preprint arXiv: 06088, 2019.
Artur Avila Garcez, Krysia Broda, Dov Gabbay, et al. Neural-symbolic learning systems: foundations and applications. Springer Science & Business Media, 2002.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pp.
2433-2433. PMLR, 2019.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05200 2020.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. arXiv preprint arXiv:2111.1453 14822, 2021.
Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 33(1-3):33--346 1990.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Geoffrey Hinton. Some demonstrations of the effects of structural descriptions in mental imagery. Cognilive Science, 3(3):231 -20200 1979.
Geoffrey Hinton. How represent part- whole hierarchies in neural network. Neural Computation, pp. 1-40, 2022.
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In Interna- tional conference on learning representations, 2018.
Drew Hudson and Christopher Manning. Learning by abstraction: The neural state machine. Advances in Neural Information Processing Systems, 32, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:161 1. 1.01 144, 2016.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
11

---

Published as a conference paper at ICLR 2024
Rishabh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-object datasets. http://gtthb.ccm//eepmn///// object-datasets/, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 81 811088119 11,,, 2020.
Hlyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 33: 111112768-2777 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2655 PMLR, 2018.
Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. arXiv preprint arXiv:2111.12544 2021.
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. Advances in neural information processing systems, 28, 2015.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Tsung-Yu Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint arXiv:2001.02407, 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33: 11111112215333 2020.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00711 2016.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua Tenenbaum, and Jiajun Wu. The neuro-
symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904 12584, 2019.
Joe Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. In International Conference on Machine Learning, pp. 3403-3412. PMLR, 2018
Mostafa Mozafari. Bitmoji faces, Aug 2020. URL httph::///// www.weaggle.com cww.kaaggllecom//atssttt//// coostaaammzzaaari/iittm
faces.
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Lue Van Gool, Markus Gross, and Alexander Sorkine-Hornung. benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 724-732. 2016.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Antti Revonsuo and James Newman. Binding and consciousness. Consciousness and cognition, 8(2), 1999.
12

---

Published as a conference paper at ICLR 2024
Fabio De Sousa Ribeiro, Georgios Leontidis, and Stefanos Kollias. Capsule routing via variational bayes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3756, 2020.
Irvin Rock. Orientation and form. (No Title), 1973.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017.
Ainkaran Santhirasekaram, Avinash Kori, Andrea Rockall, Mathias Winkler, Francesca Toni, and Ben Glocker. Hierarchical symbolic reasoning in hyperbolic space for deep discriminative models. arXiv preprint arXiv:2207.01916, 2022a.
Ainkaran Santhirasckaram, Avinash Kori, Mathias Winkler, Andrea Rockall, and Ben Glocker. Vector quantisation for robust segmentation. In Medical Image Computing and Computer Assisted Intervention-MICCA. 2022: 25th International Conference, Singapore, September 18-22, 2022, Proceedings, Part IV, pp. 663-672. Springer, 2022b.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabries Tong He, Zheng Zhang. Bernhard Schölkopf, Thomas Brox, al. Bridging the gap to real-world object-centric learning. arXiv preprint arXiv:2209. 14860, 2022.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. arXiv preprint arXiv:2 10. arXiv:2110.114055 2021.
Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural block-slot representations. arXiv preprint a7X11:2211.01177, 2022.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference on machine learning, pp. 843-852. PMLR, 2015.
Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3619-3629, 2021.
Yuhta Takida, Takashi Shibuya, WeiHsiang Liao, Chich- Lai, Junki Ohmura, Toshimitsu Uesaka, Naoki Murata, Shusuke Takahashi, Toshiyuki Kumakura, and Yuki Mitsufuji. Sq-vae: Varia- tional bayes on discrete representation with self-annealed stochastic quantization. arXiv preprint 20X1::2205.07547, 2022.
Jakub Tomezak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics, pp. 1214-1223. PMLR, 2018.
Frederik Träuble, Anirudh Goyal, Nasim Rahaman, Michael Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf. Discrete key-value bottleneck. arXiv preprint arXiv:2007.11240. 2022.
Anne Treisman. Solutions to the binding problem: progress through controversy and convergence. Neuron, 24(1):105-122 -255, 1999.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Sjoerd Van Steenkiste, Karol Kurach, Jürgen Schmidhuber, and Sylvain Gelly. Investigating object compositionality in generative adversarial networks. Neural Networks, 130:309-325, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Ramakrishna Vedantam, Karan Desai, Stefan Lee, Mareus Rohrbach, Dhruv Batra, and Devi Parikh. Probabilistic neural symbolic models for interpretable visual question answering. International Conference Machine Learning, pp. 6428-6437. PMLR, 2019.
13

---

Published as a conference paper at ICLR 2024
Jean Vroomen and Mirjam Keetels. Perception of intersensory synchrony: tutorial review, Attention, Perception, & Psychophysics, 72(4):871-884 2010.
Yanbo Wang, Letao Liu, and Justin Dauwels. Slot-vae: Object-centric scene generation with slot
attention. arXiv preprint arXiv:2306.0699) 06997, 2023.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:21 10.04627. 2021.
Alan Yuille and Daniel Kersten. Vision as bayesian inference: analysis by synthesis? Trends in cognitive sciences, 10(7):301-308, 2006.
14

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Assumptions
16
B Proofs
16
Proposition (Object Discovery ELBO formulation:
B.2 Proposition 2: (ELBO formulation for reasoning task):
17
C Datasets
18
CLEVR (Johnson et al., 2017)
C.2 CLEVR-Hans3 (Stammer et al., 2021)
18
C33 CLEVR- Hans7 (Stammer et al.. 2021)
19
C.4 Tetrominoes (Kabra et al., 2019)
19
C.5 Objects-Rooe Room (Kabra al., 2019)
C.6 Bimmoji (Mozafari, 2020)
19 ................................
C.7 FFHQ (Karras et al., 2020)
19
C.8 COCO (Lin et al., 2014)
C.9 FloatingMNIST
D Metrics
20
D.1 Mean squared error (MSE)
20 .
D.2
20 ................................
index
D.3 Codebook divergence (CBD)
21
D.4 Codebook perplexity (CBP)
21
Frechet inception distance (FID)
21
D.6 Slot average Frechet inception distance (SFID)
................................
21
D.7 Hungarian Matching Coefficient (HIMC)
21
E Algorithm & Forward Pass
22
F Quantization
22
G Codebook Collapse
22
Object Discovery
23
H.l
Quantitative Analysis
23
H.2
Qualitative Analysis
23
H.3
.
DINOSAUR Adaptation
23
H.4
Codebook Size Analysis
"o..............................
32
H.5
Multiple Object Instances
35
Abstraction Function
36
Convergence CoSA
37
K Object Composition
39
K.1 Slot Dictionary Analysis
39
Scene Generation
40
L DiscrininativeRResoonnng Tasks
40
M Training and Computational Details
41
N I Limitations and Future work
43
15

---

Published as a conference paper at ICLR 2024
A ASSUMPTIONS
Now, we list all the assumptions for modeling our slot dictionary and also discuss the how some of these assumptions are implicitly made in previous baselines (ref. Table 4.
Assumption (Representation separation). Object level separation present in observational space can be observed the latent space of the model (i.e., if x = U [O:], then z = U [Ozz]. where O and Oz; correspond to individual object representation in observational and latent space respectively).
Remark 1. In practice, when features learned with end-to-end objective, object-level represen- tation emerges due to iterative attention.
Assumption 2 (Representation overerowding). The latent representation in RN can aceu- rately recover K slots when K S N. In other words, 2; E Rd is an element of that can only encode the knowledge of single object.
Remark 2. Verifying this property is difficult. However, the latent layer for CoSA can be empirically estimated to minimize this effect.
Assumption 3 (Object level sufficiency). We assume that there are additional objects in the original data distributions other than the ones expressed in training data.
Remark 3. The assumption object level sufficiency may not be required in SA. Here, this is required as the aim is to learn marginal distributions for every known object in the dataset.
Assumption 4 (Sufficient dictionary components). We assume the slot dictionary to have sufficient number M of (G), G²) pairs to capture the emergent behavior of objectness in the entire dataset, such that M 2 K 1.
Assumption (Injectivity Assumption). We assume the resulting decoder model is injective as by construction the architecture follows the properties as expressed in Khemakhem al. (2020).
Remark 4. The assumption is mainly used for the theoretical result in proposition 3, in we can observe non distinct object representations the generated scene if this property not satisfied.
Table 4: Comparison between methods and their implicit assumptions and tasks achieved.
METHOD
DATA
ASSUMPTION
TaSKS
GROUNDING
Engelcke et al. (2021; 2019); Burgess (2019); Wang al. (2023);
Representation. Segregation, und Generation
Image Data
A1, A2
No
Emami et al. (2022)
Locatello et al. (2020); Chung et al. (2022);
Representation, Segregation
Image Data
A1, A2
No
Seitzer (2022),
Image Data und conditioning info.
Representation, Segregation
Elsayed al. (2022); Kipl et al. (2021);
AI, A2, A3
No
Representation, Segregation, and Generation (constr- -ained environment or predefined prompts)
Van Steenkiste et al. (2020); Lin al. (2020);
Singh et al. (2021)
Image Data
A2, A3
No
Representation. Segregation, and Generution
CoSA
Image Data
AI, A3,
Yes
B PROOFS
B.
PROPOSITION 1: (Object Discovery ELBO formulation):
Under a categorical distribution over our discrete latent variables 2, and the object-level prior distributions p(s?) A' (sQ: (4, af) contained in & we show that variational lower bound on the
16

---

Published as a conference paper at ICLR 2024
marginal log-likelihood of x can be expressed as:
logp(x) 2 E _5 [lo s)] DKL (g (2 x)) || p()) =: Fx)O((),,
(6)
where = IT= Ho (s'-1 f(g(9"1 k), v)) denotes the output of the iterative refinement proce-
dure described Algorithm applied the initial slot s" representations.
Proof. For this proof, consider the data distribution p(x), and the aim is to maximize the log-likelihood of this distribution:
log p(x) log [. p(x, s, dddd
Consider variational distributions q x).
= log / p(x, s, 2)) ((2 x)
dsdz
q(2 x)
Eg x) Bao x)) log p(x s))(2)
g(2 x)
p(z)
= (2)g(2/x) E log
+ (x)x E. Bep logp(x s)
q(2 x)
p(z)
(g)((()x) log
+ Bappsss log log
g x)
logp(x s g(s))) DKL (g(E a (Ao 2x) (x))))(((p(()
B.2 PROPOSITION 2: (ELBO formulation for reasoning task):
Under a categorical distribution over our discrete latent variables 2, and the object-level prior distributions p(s?) - (sf; p4, of) contained in & the variational lower bound on the conditional
log-likelihood of y given x is given by:
logp(x 2 E2)g(()x Rg logp(y p) DRi. (g(2) x) p(z))
Proof. The proof very similar to the proof of proposition We include this for the sake of completion. For this, we consider categorical conditional distribution as p(y x). the model segregating slots into given categories. The aim maximize the log-likelihodd of this distribution:
logp(y | x) = log { ply, 8, x)dsdž
Consider variational distributions q(ž x).
log ply, s, x) q(2) x)
dsdž
q(ž x)
ply x, )(())
> Eang(R Be log
q(ž x)
p(z)
= (gxo((())) Bga
log
+ Ra Bgo logp(y p)
x)
17

---

Published as a conference paper at ICLR 2024
4
3
9
g
9
5
0
8
8
6
4
9
2
7
5
9
Figure 6: This image overviews all the datasets used this work, from left to right columns correspond CLEVR. -LnsR- CLEVR-Hans7, Tetrominoes, Objects-Room Bitmoji, FFHQ, FloatingMNIST-2,, FloatingMNIST-3, and COCO datasets respectively.
p(ž)
(g)o(((x)) log
Ex Pam logp(y p))
q( x)
(x/p)( (o p(y p) DxL ((( a (x)x))(x)) 11 p(z))
C DATASETS
In this work, we multiple datasets for every case studies. We make use of publically available datasets that are released under MIT Licence and that are open for all research works. We also created two variants of the MNIST dataset called FloatingMNIST2 (FMNIST2) and FloatingMNIST3 (FMNIST3) for evaluation reasoning/diserimnnatve tasks with the proposed method. We will release the data-generation scripts along with the source code of this project under the open-source MIT License. Table describes maximum number of objects per image the considered object discovery datasets. We details on the dataset specifications in this section:
C.1 CLEVR (JOHNSON ET AL., 2017)
CLEVR is diagnostic dataset usually used for benchmarking visual question answering tasks. Here, in our tasks we don't use the question answering part of the dataset, we make use this dataset for its object level compsitionality aspect. This dataset consists of 70000, 15000, and 15000 set of training, validation, and testing images respectively. In our tasks we resize all the images to 64x64 dimension, normalizing them bound pixel intensities lie between [0, 1].
C.2 CLEVR-HANS: (STAMMER ET AI.,, 2021)
CLEVR-Han is a dataset with multiple confouding factors, developed with the soul purpose of investigating reasoning possibilities within the network. The dataset was built on top of CLEVR dataset, where the CLEVR images were further categorized into multiple classes based on object attributes. Images a particular class are encoded with confounded information with respect some attributes. CLEVR-Hans has in total of 3 classes with the following rules for classl: Large cube and Large cylinder, class2: Small metal cube and Small sphere, and class3: Large blue sphere and small yellow sphere. This dataset contains total of 9000, 2250, and 2250 images in training, validation. and testing set respectively (equally split between different classes). our tasks we resize all the images to 64x64 dimension, normalizing them bound pixel intensities to lie between [0, 1].
18

---

Published as a conference paper at ICLR 2024
C.3 CLEVR-HANS, (STAMMER ET AL., 2021)
Similar to CLEVR-Hans3 CLEVR-Hans7 is built on top of CLEVR dataset, where the images are categorized into classes based on the object attributes. Here, are the rules used for classifying images: class1: Large cube and Large cylinder, class2: Small metal cube and Small sphere, class3: cyan object in front of 2 red objects, class4: image with small green, brown, and purple objects with two other small objects, class5: 3 spheres or 3 spheres with 3 metal cylinders, class6: 3 metal cylinders, and class7: large blue sphere with small yellow sphere. This dataset contains total of 21000, 5250, and 5250 images training, validation, and testing set respectively (equally split between different classes). In our tasks we resize all the images to 64x64 dimension, normalizing them to bound pixel intensities to lie between [0, 1].
C.4 TETROMINOES (KABRA ET AL., 2019)
Tetrominoes is a Tetris-like shapes dataset. This provides test bed for non-overlapping self- supervised object discovery. Here, we use around 100000, 10000, and 10000 images for training, validation, and testing. Each image is resized to 32x32 dimension and normalized to bound the intensity values between [O, Each image comprises 3 tetris shapes sampled from 17 unique shapes and orientations, where each object has of red, green, blue, yellow, magenta, eyan color.
C.5 OBJECTS-ROOD (KABRA ET AL.,, 2019)
Objects room is a direct extention of 3Dshapes dataset Kim & Mnih (2018), built using MuJoCo environment Eslami et al. (2018). While the actual dataset has 3 different variants, we make use of single variant with identical object and room variant. In this setting 4-6 objects placed in the room and have an identical, where the colors are randomly sampled. Here, in our work we use around 100000, 10000, and 10000 images for training, validation, and testing. Each image resized to 64x64 dimension and normalized to bound the intensity values between [0,1].
C.6 BITMOJI (MOZAFARI, 2020)
Bitmoji is cartoon faces dataset for bitmoji mobile application. The images in bitmoji dataset consists 5-point facial landmarks for both male and female faces. In our task we resize all the images 64x64 dimension and normalize them to bound pixel intensities to lie between [0, 1]. This dataset consists of 4084 images, divided into sets of 3500, 292, and 292 images for training, validation, and testing respectively.
C.7 FFHQ (KARRAS ET AL., 2020)
This dataset includes high quality human faces along with the attributes corresponding to facial features. FFHQ is real world extention of bitmoji datasets. This dataset consists of approximately 200k images of 128x 128 resolution with 40 different binary attributes, and the task is to categorize images based on gender (0 male; 1=female).
C.8 COCO (LIN ET AL., 2014)
COCO is a large-scale object recognition dataset with scene-level caption information. It consists of everyday scenes with common objects. The dataset consists of 328k images with varied resolutions. In this work, we resize the images to 224 x 224 resolution and perform our analysis for scene decomposition. The dataset consists of 91 different objects with around 2.5 million masks.
C.9 FLOATINGMNIST
MNNSS is inspired by the Moving MNIST dataset (Srivastava et al., 2015) and is used to benchmark reasoning tasks in the proposed model. FloatingMNIST is made by randomly combining MNIST digits Deng (2012). The dataset has three main objectives: addition, subtraction, and mixed. The addition task is to estimate the sum of all the digits present in the given image, the subtraction task is estimate the absolute difference between the digits (after ordering them in decreasing order),
19

---

Published as a conference paper at ICLR 2024
and finally, the mixed task is to perform addition if the digits are greater than five else performing subtraction.
Floating-MNISTT22: In the case of Floating.MNIST-2, combine 2 MNIST digits in a large canvas creating the image of 64x64, We propose three variants FloatingMMNIST-22Addd FloatingMMNIST2 Sub, and FloatingMMNISST--MMiieed datasets. The task in Flosting.MNIIST---AAd is estimate the add of digits present in the image. In the case of FloatingMNNIST-2-Suub the task is to estimate difference between 2 digits present in the image. While, In the case of FloatingMNIST-2-MMixs the task is to sum the digits if both the digits are less than else to estimate the absolute difference between them. We use 60000, 10000, and 10000 training, validation, and testing images respectively.
Floating.MNIST-3: Similar to FloatingMMNIST-2, this dataset is made of images with digits. Even this version of dataset consists of three variants of reasoning datasets, namely, Floating.MNIST----A FloatingMNIITT-3-- --b-Su,, and FloatingMNIST-SS SELIS-----ixe M-x-ed.ed. Where the task in FloatingMNISST---NCC and FloatingMNNIST-3--
to estimate sum and absolute difference beween all the digits. In the case of MloatingMNIST---MMiic. substracted from the final target. We use 60000, 10000, and 10000 training, validation, and testing images respectively.
all digits less than 5 are added while the digits greater than 5 are
Table 5: Maximum number of objects (including background) observed in single image per dataset.
DATASETS())
Number of Objects
CLEVR TETROMINOES OBJECTSROOM
11
D METRICS
All the task specific metrics are described in this section
D.1 MEAN SQUARED ERROR (MSE)
To compute the quality of the reconstructed image we compute pixel wise mean squared distance between the original image and the ground truth image. This metric doesn't ' provide any information about how good the image is decomposed into objects. As the task is unsupervised nature, we measure MSE with few other properties comment on the goodness of decomposition. MSE is formally described in equation 8, where N and x² correspond to a total number of images and particular image in the dataset.
N MSE
1 N
Σ
|x (c(CoSA(
(8)
D.2 OVERLAPPING INDEX (OPI)
OPI measures the mean overlap of particular slot with respect to every other slot across the given dataset. For computing the overlap we first rescale the pixel intensities in the estimated mask to lie between (0, 1) by applying min-max normalization and later thresholded at 0.5. The normalization and thresholding will result binary decomposed images, which can be used to compute average overlap. OPI is formally described in equation 9, where S, and rY correspond to total number of images, slots, and particular slot estimated for considered image x² respectively.
8
2w (x
1
OPI
(9)
J U
20

---

Published as a conference paper at ICLR 2024
D.3 CODEBOOK DIVERGENCE (CBD)
We use this metric to measure the distance between the codebook embeddings, higher distance indicates the effective usage of codebook embeddings. To measure this distance we first project all the embedding vectors onto the unit hypersphere and compute the average cosine distance between them. In practice this done by computing the inner product between the emhedding vectors, which formally described in equation 10, where K corresponds total number of codebook embeddings and cos applied to get the angular distance between the embeddings.
K Σ
CBD arccos
Σ fei ej)
(10)
K
K
D.4 CODEBOOK PERPLEXITY (CBP)
Perplexity measure a property that indicates the sampling efficiency of the codebook. The perplexity higher if all the codebook embeddings uniformly sampled. Lower perplexity is indicator of codebook collapse. Formally the perplexity is estimated as described in equation 11, where N, K, and qidx; correspond to total number of images a dataset, the total number of codebook embeddings and indicator value of jth codebook vector is sampled for image x².
K
CBP = exp
Σ
Σ qidx log
1 N Σ qidx;)
(11)
D.5 FRECIET INCEPTION DISTANCE (FID)
FID is computed to measure the quality of generated images with respect to real images, it's usually measured by computing the distance between the latent representations of real and generated images. For measuring this we usually use any pre- pre-trained model Dogg for computing latent representation; the
FID score is calculated using equation 12, where z" N Σ> Pugg (x]); Σ" Pogg (x") " "ogg (x")
similarly, 2 and " are defined of the generated image set.
FID(#", x0)) = || - 201 + Trace -+E3-22(E''))
(12)
D.6 SLOT AVERAGE FRECHET INCEPTION DISTANCE (SFID)
As FID measures the quality of the final reconstructed image, to measure the quality of individual slots, we compute FID score with respect original image and individual slot. The intuition behind doing this is if the slot contained object information is present in the original image, the FID for that slot would be less than the slot with random information. We compute average FID score for all the estimated slots with respect to the original image to obtain SFID, formally described in equation 13
K
SFID (x"
x²)
Σ
FID(a" (0,0",,2) x
(13)
K
D.7 HUNGARIAN MATCHING COEFFICIENT (HMC)
In the case of the reasoning objective, measure the implicit rationale provided by the model by comparing the emerging object properties with ground-truth object properties. To consider the permutation invariance property of slot properties, perform Hungarian Matching over all the properties with respect to the ground truth properties and compute the MSE cost between paired vectors.
21

---

Published as a conference paper at ICLR 2024
E ALGORITHM & FORWARD PASS
Our proposed object-level representation learning uses an encoder to map an image to its latent representation, followed by computing q. k, projection vectors. The obtained position-free en- coding z is further passed through image-level abstraction function A resulting cigenvectors V and corresponding eigenvalues A are used estimate principle components a as described in section As the obtained principle components are with respect to a particular image, to generalize them across the dataset, GSD sampling is used, k ~ G². Allowing us to sample slots from uniquely mapped conditioning distributions, ensuring slots to be sampled from the same conditioning distribution S for all instances of object in the case when the given image has more than instance of the same object. Based on the heuristic defined in section 4, K slots are sampled from their respective conditioning distributions, which are selected with respect to the obtained eigenvalues. Finally, the iterative attention mechanism is applied to obtain slot-level representations for the downstream tasks. Algorithm describes the proposed algorithm.
F QUANTIZATION
In the case of quantization, we first define codebook G E ggxxd, with N codebook embeddings, where individual embedding G; E IRd=, The objective in the case of quantization is to learn the categorical distribution over codebook features for given image In the case of deterministic sampling, as in the case Euclidean and Cosine sampling, the categorical distribution is modeled with one-hot probabilities determined by the mapping of each discrete vector the nearest codebook vector:
plz, = Gx x) =(1 for argming, ||z 6, or (z, G))
{1
(14)
otherwise
In this case, the probability estimates are usually non-differentiable. Due to this, in practice, back- propagation over this sampling block is achieved with the help of straight-through gradient ap- proximation, enabling end-to-end training as illustrated in Van Den Oord et al. (2017). Due to the deterministic nature of sampling and the uniform prior assumption, the DKL term in the objective drops to be constant.
While in the case of stochastic sampling, in Gumbel, the one-hot probabilities are approximated with the softmax distribution resulting in p(z x) so f ftmax(exp( + //)t)), where is sampled from Gumbel distribution, and is the temperature parameter as detailed Maddison et al. (2016); Jang et al. (2016).
G CODEBOOK COLLAPSE
Codebook collapse is a common challenge in training vector quantized models, in this case, only one selected few embeddings (>> total number of codebook embeddings) of codebook are being used repeatedly. Various engineering tricks have been proposed to address this issue previously Bardes al. (2021); Van Den Oord al. (2017); Träuble et al. (2022); Takida et al. (2022). this work, we first detach the inputs to the quantization block so that the object-centric representations are not affected by quantizer gradients. We also use the exponential moving average (EMA) over codebook embeddings which encourages the embedding vectors to converge at the mean of the representations. Along with EMA we also utilize random restart dead codebook vectors with the running representational statistics.
In terms of slot distributions, we initialize the distribution on a unit hypersphere, fix the mean of the distribution, and leam the slot-specific transformation functions which transform the fixed distribution into the required distribution. This was first proposed in Tomczak Welling (2018) to prevent overfitting of learnable priors.
22

---

Published as a conference paper at ICLR 2024
H OBJECT DISCOVERY
Here, we provide more analysis on sensitivity analysis of COSA on object discovery task. We first test our framework all three sampling methods using Gumbel, Cosine, and Euclidian codebooks. We use the convolutional encoder and decoder architectures with convolutional and transpose convolutional layers. Additionally, we provide the comparative results slot properties across different methods.
H.1 QUANTITATIVE ANALYSIS
Table 6 and demonstrates the sensitivity COSA with respect to selected sampling criteria. Based on the results, it can be observed that CoSA performs similarly better than its slot attention counterpart, illustrating the effectiveness of grounded representations. We measure the diversity in the codebook embeddingg and their variation in sampling by measuring CBP and CBD properties, and these properties illustrate the variability in sampling and differences in codebook representations. The resulting values for all five datasets described in table 8. These results demonstrate that there's 's no collapse in codebook representations and the sampling (ref. appendix section G for details on codebook collapse). To measure the goodness of the generated slots, we measure OPI and SFID which are tabulated in table 9.
Table 6: Sensitivity analysis of CoSA object discovery.
METHODS(L),
CLEVR
TETROMINOES
OBJECTS-ROOM
BITMOл
FFH
METRICS(->)
MSE()
FID(L)
MSEQ)
FID(J)
MSE(1)
FID(L)
MSE())
FID(J)
MSB(J)
FID(J)
38.18
1.49
7.57
38.12
14.62
12.78
55.14
54.95
COSA-GUMBEL
3,82
31.84
0,21
0.26
5.45
32.41
9.84
9.36
41.77
52.21
COSA-EUCLIDIAN
33.18
1.13
6.42
34.05
8.97
8.41
31,94
35.14
COSA-COSINE
3.14
29.12
0.42
0.41
4.85
28.19
8.17
9.28
33.37
36.34
H.2 QUALITATIVE ANALYSIS
Fig. 7(a). 11(a), 13(a), 15(a), and 9a) illustrate vanilla SA results. Fig. 7(b), 11 13(b), 15(b),
and 9(b) illustrate the results of CoSA-Cosinn While, Fig. 8(a), 12(a), 14(a), 16(a). and 10(a) illustrate the results of CoSA-Gumbor and finally, Fig. 8(b), 12(b), 14(b), 16(b), and 10(b) illustrates COSA COSA-Euclidian results.
H.3 DINOSAUR ADAPTATION
To extend slot attention to real-world images, we adopt DINOSAUR model Scitzer et al. (2022), which uses the Vision Transformer (ViT-S16) model trained with DINO training strategyCaron et al. (2021). The main objective of DINOSAUR to reconstruct latent representation rather than reconstructing the original image. To compare framework under similar specifica- tions, we use DINO as a feature extractor and use CoSA for the extracted latent features to reconstruct these features and the resulting attention maps as slots. We demonstrate the results of DINOSAUR and variants of DINO-COSA in Figures 17, 18. Table illustrates the quantitative results on COCO dataset; note that the results in the original work are based
Table 7: ARI on CLEVR6, Tetrominoes, ObjectsRoom, and COCO datasets, for SA baseline model and CoSA-cosioe A-cosine variant. In the case COCO we use DINOSAUR variant of SA Seitzer al. (2022) (ViT-S16) as baseline and use DINO feature extractor (ViT-S16) for CoSA.
METHOD
TETROMINOES
CLEVR6
OBJECTSROOM
COCO
SA/DINOSAUR
0.99 0.005
0.93 + 0.002
0.78 + 0.02
0.28 + 0.02
COSA-EUCLIDIAN
0.99 0.002
0.94 4 0.002
0.81 0.01
0.27 0.02
COSA-GUMBER
0.99 = 0.001
0.93 + 0.002
0.80 + 0.01
0.30 0.02
COSA-COSINE
0.99 + 0.001
0,96 + 0.002
0.83 + 0.002
0.36 + 0.01
23

---

Published as a conference paper at ICLR 2024
(a) Baseline
(b) CoS A-Cosine
Figure 7: Vanilla SA and CoSA-Cosime object discovery results on tetrominoes dataset.
24

---

Published as a conference paper at ICLR 2024
(a) CoSA-Gumbo
(b) CoSA Euclidian
Figure 8: CoSA-Euclidie and COSA A- A-Gumell object discovery results on tetrominoes dataset.
25

---

Published as a conference paper at ICLR 2024
(a) Baseline
(b) C-SA-Co
Figure 9: Vanilla SA and C-Ss-ne object discovery results on objects-room dataset.
26

---

Published as a conference paper at ICLR 2024
(a) AoSA-Gumbel
(b) CoSA-Euclidian
Figure 10: and A-Gubel object discovery results on objects-room dataset.
27

---

Published as a conference paper at ICLR 2024
8
8
(a) Baseline
(b) COSA A-Cosine
Figure 11: Vanilla SA and C-SA-Cosine object discovery results on CLEVR dataset.
28

---

Published as a conference paper at ICLR 2024
(a) CoSA-Gumbe
(b)
Figure 12: CoSA A-Fucliddia and CoSA-Gumbel object discovery results on CLEVR dataset.
29

---

Published as a conference paper at ICLR 2024
(a) Baseline
(b) COSA-Cosinn
Figure 13: Vanilla SA and COS A-Cosine object discovery results on bitmoji dataset.
30

---

Published as a conference paper at ICLR 2024
3
(a) G-umell
(b) -Euclidia
Figure 14: Souu-ddda and COSA- object discovery results on bitmoji dataset.
(a) Baseline
(b) C-Ss-Ce
Figure 15: Vanilla SA and COSA-Cosine object discovery results on ffhq dataset.
31

---

Published as a conference paper at ICLR 2024
Table Codebook properties across different sampling methods across datasets.
METHODS()),
CLEVR
TETROMINOES
OBJECTS- RRJECT--ROOM
BITмон
FFHQ
METRICS(-)
CBD(1)
CBP(T)
CBI(t)
CBP())
CBD(1)
CBP(1)
CBD())
CBP(T)
CBD(()
CBP(t)
COSA-COSIN COSINE
44.39
0.99
26.53
28.22
0.99
36.97
COSA-BUCLIDIAN
0.99
32.40
0.99
28.99
0.99
24.26
0.99
55.50
0.99
24.28
COSA-GUMBEL
1.0
29.62
1.0
22.35
1.0
23.05
1.0
37.61
1.0
24.69
Table 9: Slot properties.
METHODS(L).
CLEVR
TETROMINOES
OBJECTS-RooD
BITMОл
FFHQ
METRICS(-) -)
OPI(L)
SFID(J)
OPI(.)
SFID())
OPI())
SFID(()
OPI(L)
SFID(L)
OPI(L)
SFID())
SA
0.43
238.7
0.48
270.3
0.42
162.9
0.41
204.1
0.27
240.9
IMPLICIT
0.35
220.1
0.56
249.1
0.37
128.5
0.34
197.4
0.24
244.1
COSA-GUMBEL
0.35
192.7
0.43
231.2
0.42
119.4
0.30
201.2
0.28
213.1
COSA-BUCLIIIAN
0.34
236.7
137.5
0.29
190.3
0.33
232.3
COSA-COSINE
0.45
175.8
0.53
235.2
0.39
103.2
0.37
213.3
0.32
211.3
on the ViT-B16 feature extractor, while we use ViT-S16 due to computational resources. We base our implementation by considering base implementation from https:/ /tt:://gtthbbbtt com/ amazon-science / jbjeetccentrice wwwzzrii-eeaannnn-ee frameworrrrwrkkkr and use provided hyperpa- rameters in 'coco feat rec in small16 auto. yaml', with only changes in the number of GPUs and the batch size; we use a single GPU with batch size of 32.
H.4 CODEBOOK SIZE ANALYSIS
Given we make dictionary sufficiency assumption (refer to an assumption 4), we perform extensive ablations dictionary size CLEVR data for all three sampling strategies. Table 10, 11, and 12 correspond to Cosine, Euclidean, and Gumbel sampling strategies. As can be observed, a very low number of codebook embeddings (M=16) results in high MSE error and lower ARI scores, while increasing the embeddings drastically also does not help. While the number of embeddings around 64, 128 provide similar results without any collapse (indicated by higher perplexity scores). This indicates that the framework is not extremely sensitive to the codebook size, given the codebook does not collapse.
Table 10: Codebook size ablations with Cosine Sampling
METHODS(L), METRICS(-))
CLEVR
MSE(L)
FG-ARI(1)
CBD(f)
CBP(()
M=16
8.11
0.73
0.99
14.96
M=64
3.14
0.96
0.99
44.39
M=128
3.46
0.94
0.98
51.65
M=256
6.68
0.85
0.98
82.20
(a) A-Smeell
(b) CoSA-Euclidian
Figure 16:
and CoSA-Gumbe object discovery results ffhq dataset.
32

---

Published as a conference paper at ICLR 2024
Help
(a) DINOSAUR
(b) DINO- LANN--COS--achhicn
Figure 17: Results on COCO dataset.
33

---

Published as a conference paper at ICLR 2024
(a) DINO-COSA-Cosine
ASO
(b) DINO-COS..SO Caw.-OoSA-GGeeee
Figure 18: Results on COCO dataset.
34

---

Published as a conference paper at ICLR 2024
Obier may
Tecestactel mot
CAST
200
0001
Orignal mago
aaconttateed mmag
032 13
CEO-E
DS0-11
C50
CE0.3
6S0-42
DAOI
1003
0O01
Figure 19: Demonstration of object binding across instances in given image. The title of each figure corresponds to the sampled element index of GSD 6.
Table 11: Codebook size ablations with Euclidian Sampling
METHODS(L),
CLEVR
METRICS(-))
MSE())
FG-ARI(1)
CBD(f)
CBP(f)
M=16
4.81
0.93
0.99
8.33
M=64
4.04
0.94
0.99
32.40
M=128
3.64
0.94
0.95
20.69
M=256
4.34
0.89
0.96
19.79
Table 12: Codebook size ablations with Gumbel Sampling
METHODS(L), METRICS(-)
CLEVR
MSE(L)
FG-ARI(1)
CBD(1)
CBP())
M=16
6.64
0.88
1.0
11.06
M=64
3.82
0.93
1.0
29.62
M=128
3.89
0.93
0.99
27.16
M=256
5.46
0.92
0.99
30.28
H.S MULTIPLE OBJECT INSTANCES
In this section, demonstrate the effectiveness of CoSA binding multiple instances of object in given image to a same element in GSD. Figure 19, the title of each sub figure correspond to GSD index being selected for that particular object. As observed in frst-fifth row GSD elements 619, 64, 614. G1s. and Gy1 are being sampled twice, respectively.
the trained model and analyse the binding capabilities across multiple frames in a video sequence. For this we use the COCO trained model and perform zero-shot adaptation in DAVIS video dataset Perazzi al. (2016). Figure 20 reflects this analysis,
Similarly, previous analysis
35

---

Published as a conference paper at ICLR 2024
Original Imace
050:24
69029
050
Original
Original Imace
G9D:24
05031
GSD3
380164
Original Image
Figure 20: Demonstration of object binding across frames in video sequence. The title of each figure corresponds to the sampled element index of GSD G.
where each row correspond to a particular frame in a vide. We can observed that GSD elements 654- and G2 consistently map camel and the background, respectively.
ABSTRACTION FUNCTION
We tried multiple variants of spectral decomposition for abstracting object-level features including singular value decomposition, QR decomposition, vanilla quantization, and eigenvalue decomposition. Table 13 describes the results the CLEVR dataset for all different types of variants of abstraction functions. We use the principle components based decomposition approach as the default abstraction function because the results do vary too much among different types of spectral decomposition methods.
Table 13: Abstraction functions
METHODS(!), METRICS(
VANILLA QUANT.
PRINCIPLE VEC. QUANT.
QR VEC. QUANT.
SINGULAR VEC. QUANT.
MSE(1)
FID())
MSE(J)
FID(L)
MSE())
FID(J)
MSE())
FID (L)
COSA-GUMBEL
6.44
43.61
3.82
29.12
4.18
33.74
3.97
26.29
COSA-BUCLI
8.32
54.11
4.04
33.18
4.76
32.43
4.14
28.57
COSA-COSINE
7.68
48.72
3.14
31.84
34.12
30.22
36

---

Published as a conference paper at ICLR 2024
J CONVERGENCE OF COSA
0 Φ. 16) satisfies Eng (8 = (S) where is a representation of a true object, resulting the convergence of the learned canonical representation mean of true slot distrbutions.
Proposition 3. Under assumptions 1-5 the solution (,,,,77,8,
Proof. For proof, we make use of the slot posterior described in Equation 3
T
p(s" x) = s II Ho ((ol""
where 8()) is Dirac delta distributed given randomly sampled initial slots from their marginals associated with the codebook vectors 2 ~ g(ž x). The distribution over the initial slots s induces distribution over the refined slots s"
s p(s" 2) = I[ W
Let {ax 13,,, 14 o) set of all the parameters. Then, the joint distribution for slots can be described as:
I flo("" .k), v)) (& x)p((s 2)
(s S x) ST
By assumptions 2, 3, and 4, we know that the slot distribution is structurally identifiable. Structural identifiability property of the model explored in detail by Brady al. (2023).
Given structural identifiability of the model, to show ( = E. ( (8), under infinite
data regime, we need to show that the likelihood estimate follows:
lim Σ log (s xx) S lim
1 N
logp($s; x)
-0000
where corresponds to all the slots estimated for datapoint Xx.
By the law of large numbers, know that:
lim +
log Pw (s xx)
x)]
Now we can show:
Ep(a,)) Po x)] Ex(s.x) [log((5; x)]
log (s z {x(x)))
=pp(kx))
p(š x)
(s" === =xx]])
< Ep(xxx)
-1
(::
logt < 1)
p(s x)
= =s (s 2 ===)dd
0
This mainly indicates that the sampled from p(s 22) are closer to the mean of the tru distribution, given infinite data and under specified assumptions ELBO estimate recover this slot distribution.
Discussion.
Lemma (Convergence rate) Given the per iteration update SA, u' f(g(,, 4). v) and per iteration update of CoSA u² = fl(k, q), v). there exists A 2 such that d(u d(uu? 8) = A with being representation of true objects.
37

---

Published as a conference paper at ICLR 2024
SA
SA
Co5A-Bucildian CoSA-Cosine Gumble
CoGA-Euclidian
CoSA-Cosine
Gumble
MSE
Iterations
Iterations
(a) MSE
(b) FID
SFID
CoSA-uccidaa
CoSA-Cosine
CoSA-Gumble
Iterations
(c) SFID
Figure 21:
Iteration wise property comparison between methods with bull-backpropagation without Neumann's 's approximation for CLEVR dataset.
Remark 5.
In practice, the parameters in the update function and query, key, and value projection functions differ in both cases. Here, are interested in investigating the effect initializing slots concerning specific conditionals rather than using joint distribution.
Proof. The proof directly follows from the result of proposition 3 and we include the proof for completion.
For given the result of proposition 3 we know that d(s".5) 8) d(s".s) 2 O, where s", s correspond initial slots sampled in SA and CoSA respectively and d is convex distance functions.
Updates at iteration 0, up = f(g(k, Q(s")), v), uf = f(g(k, @(5")), v). since f(.) is convex combination function
elements of and with weights obtained from an attention map g()). The composite function is monotonic, resulting d(uu) $) d(u3, $) 2 0.
Similarly, h is also monotonic function with matrix product and tanh activation, resulting in general expression d(u', s) d(uu, 8) 2 O.
The easier way to understand the convergence rate in CoSA is that, here the slot initialization is based on the grounded (mean) object representation, rather than random, which requires fewer iterations to converge to the final object. To illustrate the correctness of Lemma 1. consider the object discovery task on the CLEVR dataset and compare results in terms MSE, FID, and SFID metrics by varying numbers of iterations in SA and CoSA variants. Here, compute the properties without Neumann's approximation (i.e.. without gradient truncation) to compare the vamilla version of SA and C. Fig. 21 illustrates the faster convergence, validating lemma.
38

---

Published as a conference paper at ICLR 2024
Algorithm 2 Scene composition by random slot prompting
1: Prompt Dictionary Items: index 440,
2: Sample Inputs: z = 6 =G"[index]
3: Compute: k- = (z),), and v (z)
4: for ...
> R Monte Carlo samples
5: slots; ~ 6 [index] E kxd
> Sample prompted slots from GSD Refine slots over T attention iterations b Update slot representations
6: for ,...., T
7: slots; = f (g (Q,, (LayerNorm (slots, 1)) k) v) 8: slots; += MLPP (LayerNorm (Ho (slots, slots.))
GRU update & skip connection
9:
return E,soots/RR
>> MC estimate
(a) Embedding-6
(b) Embedding-1
(c) Embedding-22 22
(d) Embedding-7
Figure 22: Gumbel dictionary embedding observation mapping for CLEVR dataset.
K
OBJECT COMPOSITION
Unlike SLATE Singh et al. (2021), our approach does not require test time prompt generation, as the dictionary behaves as a set of prompts; we do not need to learn prompt dictionary, as we learn object/property level distribution as part of our algorithm. The prompting in our approach is solely based on the learned slot dictionary. Algorithmically, and corresponding slots, followed by positional matting and iterative slot refinement for generating realistic object-level scenes. Algorithm describes the approach with pseudo-code.
first sample input features from dictionary
Object composition illustrates that the grounded representation (every embeddings in a GSD) are specialised to capture specific individual object in given environment, expected in proposition 3. We select K slots, sample an embeddings from the respective slot-distributions and refine them generating K channeled images, which we further combine by merging them using softmax distribution, providing us with the prompted image.
K. SLOT DICTIONARY ANALYSIS
Here, we describe the mapped dictionary elements, Fig. 3, 22, 23, and 24 illustrate randomly selected dictionary index to slot mapping. To generate these images, we list the sampled codebook index for given 100 images and group them with respect to uniquely sampled codebook indices. can be observed that the codebook kind groups similar- similar-looking object properties together. In Fig. 3, it can be seen that embedding-7 of the codebook corresponds to cheeks, embedding- 14 maps to head ware and forehead, embedding-25 maps eyes, and embedding-55 maps to facial hair. It is important to note that these semantics are completely subjective and assigned by observation. Similar observations can be made CLEVER and Object-Room models illustrated in Fig. 22 and Fig. 23, respectively. The model still struggles to disentangle in the case natural images like the FFHQ dataset as illustrated in Fig. 24, nevertheless, it is grouping some similar-looking regions together.
39

---

Published as a conference paper at ICLR 2024
(a) Embedding-I1 12
(b) Embedding-17 17
(c) Embedding-24
(d) Embedding-9
Figure 23: Cosine dictionary embedding to observation mapping for Objects-roon room dataset.
(a) Embedding-1
(b) Embedding-4 4
(c) Embedding-5
(d) Embedding 6
Figure 24: Gumbel dictionary embedding observation mapping for FFHQ dataset.
K.2 SCENE GENERATION
Finally, we also demonstrate the image composition objective by randomly sampling embeddings from the codebook and performing iterative attention to generate results. Fig. 25, 26, and 27 illustrate both randomly sampled slots and the resulting image composition.
L DISCRIMINATIV/REESSONIC TASKS
For validating discriminative tasks, we consider CLEVR-Hans and FloatingMNIST tasks, where the main objective is to segregate the image into different classes based on the contents of the image, while proving rational for the segregation without explicitly training for the rational. More details about the datasets are described in C. As the classification based on the object properties in the images, consider the properties emerging entities and model it as reasoning task by considering the emerging properties as rationale for prediction. The correctness of the generated rationale measured using Hungarian Matching Coefficient (HMC).
(a) CoSA-Cosine
(b) CoSA-Euclidia
(c) CoS A-Gumbel
Figure 25: Image composition with random slot prompt bitmoji dataset.
40

---

Published as a conference paper at ICLR 2024
Table 14: Compositional FID and SFID scores.
(ETHODS()),
CLEVR
TETROMINOES
OBJECTS-Roo RoOM
BITмол
FFHQ
METRICS(-)
FID(L)
SFID())
FID(.)
SFID()
FID()
SFID(L)
FID(L)
SFID(L)
FID(L)
SFID())
COSA-GUMBEL
116.1
257.7
32.1
256.1
194.1
184.2
127.1
262.6
225.7
237.1
COSA-BUCLIDIAN
122.2
236.2
36.3
278.4
194.1
184.2
114.3
215.8
209.3
241.3
COSA-COSINE
117.8
240.1
28.1
256.8
184.2
186.0
114.3
273.2
185.8
235.2
(a) CoSA-Cosine
(b) A-uulidiaa
(c) CoSA CoSA-Gumbel
Figure 26: Image composition with random slot prompt on CLEVR dataset.
To demonstrate the generalizability property of model due to grounded representations, we perform cross domain adaptability test. For adaptability test we train the model on one dataset and fine-tune the classification head on other datasets by controlling the data samples. The main hypothesis here that the grounded representations are meaningful environmental properties, which goes beyond the downstream tasks. Qualitatively, table 16 describes the performance of the model trained and and CLEVR-Hans, datasets in columns and while other two columns indicate the task adaptability results with few shot (k=100) learning performances with the different base model. Similarly, table 15, 17 and table 18 describe the results on FloatingMNISTT and FloatingMNIST3 datasets respectively with different variants of tasks (Add, Sub, and Mixed). Finally, table 19 demonstrates the performance on Mixed tasks on FloatingMNIST2 and FloatingMNIST33.
tested on CLEVR-
M TRAINING AND COMPUTATIONAL DETAILS
We train all the models with adam optimizer with a learning rate of 0.0004, batch size 16, with early stopping with the patience 5, and for a maximum of min(40epochs, 40000 steps). We use linear learning rate warmup for 10000 steps with a decay rate of 0.5 followed by reduce on the plateau learning rate scheduler. The experimental config files and training/inference scripts with be made available for reproducibility and further research.
We run all our experiments on a cluster with Nvidia Telsa T4 16GB GPU card with Intel(R) Xeon(R) Gold 6230 CPU. The processing speed of our framework in comparison to the baselines is tabulated
(a) CoSA-Cosine
(b) CoSA-Euclidia
(c) CoS A-Gubbll
Figure 27: Image composition with random slot prompt on Tetrominoes dataset.
41

---

Published as a conference paper at ICLR 2024
DefouliCAN
CoSA-Euciien Euci--uni
CoSA-Gumible
Accuracy
Iberations
Figure 28: Few shot task adaptability accuracy for all the methods.
DefoultCNN
SA
Fuclidian CoSA-Casine CoSi-Gumble
Accuracy
Figure 29: Few shot adaptability on FloatingMNISTTZddfffe to FloatingMNNIST--addd dataset.
Accuracy
DefaultCNN
SA
Casiie
CoGA Gumble
Iterations
Figure 30: Few shot adaptability on CLEVR-hanse CLEVR-hans7 dataset.
42

---

Published as a conference paper at ICLR 2024
Table 15: Accuracy and Hungarian matching coefficient (HMC) for reasoning objective on addition and subtraction variant of floatingMNIST2 (FMNIST2) dataset. Here, the first and third pair of columns correspond the models trained and tested on the FMNIST2-Add and FMNIST2-SM datasets, respectively, while the second and fourth pair correspond to few-shot(k=100) adaptability results across datasets.
METHODS(.). MET METRICS(-)
FMNIST2-ADA ADDsmms
AMNIST2-SUBBie
TMNIS12-SUBBa.
FMNIST2-ADDung
Acc(t)
HMC()
Acc(t)
FI(1)
Acc(1)
HMC()
Acc(f)
FI(t)
CNN
97.62
10.35
10.05
98.16
12.35
9.50
SA
97.33
0.14
11.06
09.40
97.41
0.13
08.28
7,83
COSA-GUMBEL
97.52
0.11
30.75
28.23
28.45
20.53
COSA-RUCLIDIAN
98.12
0.10
60.24
50.16
98.64
0.12
63.29
58.29
COSA-COSINE 98.33
0.10
37.47
36.00
98.81
0.12
50.43
44.47
Table 16: Accuracy and Hungarian matching coefficient (HMC) for reasoning objective on CLEVR- Hans dataset. Here, the first and third pair of columns correspond to the models trained and tested on CIEEVR-Hans3 and CLEVV- datasets respectively, while the second and fourth pair corresponds to few shot(k=100) adaptability results across models.
METHODS(
sounee
CLEVR-HANST,R 7
SLEVR-HANST.CM sounce
CLEVR-HANS3.L
METRICS(-)
Acc(t)
HMC()
Acc(f)
F1(f)
ACC(1)
HMC()
Acc(f)
FI(f)
CNN
67.89
28.14
16.66
80.06
58.49
55.11
68.53
31.69
26.21
81.12
0.41
68.28
COSA-GUMBEL
71.88
0.26
40.42
38.18
82.72
0.32
69.98
69.81
COSA-EUCLIDIAN
70.64
46.45
46.88
81.17
71.35
68.33
COSA-COSINE 70.28
0.28
43.23
45.05
82,28
0.31
72.82
70.41
below. Table 20 and 21 describe the object discovery and reasoning speed terms of iteration/sec.. is be noted that speed might differ slightly with respect to the considered system and the background processes.
All experimental scripts and scripts to generate datasets is available on GitHub at https:/ gi github. com//ooravinashh/ CooS..
N LIMITATIONS AND FUTURE WORK
Our proposed method also possesses few limitations; here we detail them and also propose a few possible ways address them:
Limited variation in slot-prompting: As the current prompting is solely based on dictionary size, the diversity in scene generation is limited. To address this, we can explore the possibility of stochastic VQVAE Takida et al. (2022). Additionally, contrastive learning could also prove useful to learn shape, colour, location, and texture codebooks separately, increasing the control and variability compositional scene generation.
Assumption representation overcrowding: the assumption that representation overerowd- ing can be controlled with the architectural design, but exploring the direction to enforce this behaviour with additional regularization would be interesting.
Heuristic for dot-distribution selection in the case of non-unique object composition: The current heuristic to handle repeated objects in scene is dependent on the scale of principle components (eigenvalues of the principle direction vectors): it would be interesting to explore the directions to have it implicitly within the sampling procedure.
43

---

Published as a conference paper at ICLR 2024
Table 17: Accuracy and Hungarian matching coefficient (HMC) for reasoning objective on addition and subtraction variant of floatingMNIST2 (FMNIST2) dataset. Here. the first and third pair of columns correspond to the models trained and tested on the FMNIST2-Add and FMNIST2-Sub variant of datasets respectively, while the second and fourth pair corresponds to k-shot adaptability results across models. Here, Base models corresponds to the models trained on source datasets and datasets described in the following row correspond to the target datasets, used in generalisability.
(-))
FMNIST2-MER MIXED
FMNIST2-MER
FMNIST2- MIXEE
FMNIST2-ADD
FMNIST2- SUB
METHODS(J).
FMNIST2-MIXED
FMNIST2-ADD
FMNIST2-SUB
MANIST2-MIXED
FMNIST2-MIXXI
METRICS(-)
ACC(f)
HMC())
Acc(1)
FI(T)
Acc(f)
F1(")
ACc(f)
F1(1)
Acc(t)
F1(")
14.01
SA
98.12
0.11
15.32
11.25
20.02
15.49
13.13
09.55
14.13
09.57
GUMA-GL
98.28
0.11
37.86
34.45
41.04
34.75
29.30
24.04
32.57
24.72
COSA-EUCLIDIAN
98.12
0.10
53.22
45.98
63.05
50.15
50.59
44,48
53.15
COSA-COSINE
98.40
0.08
56.21
49.30
64.60
$8.18
40.54
37.56
50.63
43.44
Table 18: Accuracy and Hungarian matching coefficient (HMC) for reasoning objective on addition and subtraction variant of floatingMNIST2 (FMNIST2) dataset. Here, the first and third pair of columns correspond to the models trained and tested on the FMNIST2-Ada and FMNIST2- variant datasets respectively, while the second and fourth pair corresponds to k-shot adaptability results across models.
BASE MODELS (-)
EMNIST3-MIXED
FMNIST3-MIXUD
FMNIST3-MIXE
FMNIST3-ADD FMNIST3-MIXED
FMNIST3-SUD
MCTHODD(().
FMNIST3-MIXDD
FMNIST3-ADD
FMNIST3-SUD
FMNIST3-MIXED
METRICS(-))
ACC(t)
HMC()
Acc(1)
FI(t)
Acc(t)
FI(?)
ACC(t)
FI(1)
ACC(t)
FI(1)
CNN
96.23
05.29
04.61
07.06
08.13
10.66
04.11
10.19
04.99
SA
96.29
0.18
08.83
04.77
06.68
03:40
07.16
03.87
13.81
06.28
DOSA-GGMBEL
0.14
08.53
12.28
COSA-EUCI EUCLIDIAN
97.07
0.15
16.32
11.26
35.46
22.09
23.37
18.97
20.50
11.76
COSA-COSINE 96.85
0.15
17.87
13.04
35.35
19.01
11.95
10.47
Table 19: Accuracy and Hungarian matching coefficient (HMC) for reasoning objective on addition and subtraction variant of floatingMNIST3 (FMNIST3) dataset. Here, the first and third pair of columns correspond the models trained and tested on FMNIST3-Add and FMNIST3-SU variant of datasets respectively, while the second and fourth pair corresponds to k-shot adaptability results across models.
BASE MODELS (+)
FMNIST3-ADD FMNIST3-ADD
FMNIST3-ADD
FMNIST3-SUL
FMNIST3-SUB
METHODS()), METRICS(-)
FMNIST3-SUR
FMNIST3-SUR
FMNIST3-ADD
Acc(1)
HMC())
Acc(t)
F1(f)
Acc(t)
HMC())
ACC(1)
F1(t)
CNN
96.42
15.92
08.10
97.62
06.80
04.46
SA
96.66
0.16
12.38
07.25
97.33
0.14
05.45
02.67
COSA-GUMBEL
98.76
0.14
18.92
10.11
97.52
0.11
09.59
08.01
COSA-EUCLIDIAN
98.11
19.82
11.72
98.12
0.10
17.91
11.03
COSA-COSINE
98.33
0.10
20.22
13.63
98.33
0.10
14.01
07.82
Table 20: Iteration per second details for all datasets on object discovery task.
Methods(L),
CLEVR
TETROMINOES
OBJECTS-ROOM
BITMOμI
FFHQ
Metrics(->)
it/s
it/s
it/s
it/s
it/s
SA
10
16
10
5
IMPLICIT
14
22
22
14
6
CoSA-Gumbel
18
18
7
4
CoSA-Euclidaan
20
5
CoSA-Cose
20
20
9
5
44

---

Published as a conference paper at ICLR 2024
Table 21: Iteration per second details for all datasets on reasoning task
METHODS(),
CLEVR-HANS
FLAOTINGMNIST
METRICS(->
it/s
it/s
SA
16
16
CNN
20
20
12
COSA-EUCLIDIA
15
15
COSA-COSINE
15
15
45


Reviews:
[{"review_id": "6WghMzWynD", "reviewer": "Reviewer_dFhu", "summary": "This paper introduces Conditional Slot Attention (CoSA) that uses a Grounded Slot Dictionary (GSD) to sample initial slots from a shared library of canonical object-level property vectors. Spectral decomposition is used to estimate the number of initial slots $K$ and vector quantization is used to select initial slot distributions from the GSD. The authors run experiments on object discovery, scene composition and generation, and downstream task adaptation showing benefits over previous methods.", "questions": "- The codebook size seems like a potentially important parameter. What size do you choose for the different experiments? How sensitive are the results to codebook size?\n- I want to confirm that the Abstraction Module is not differentiable and just uses the output of the encoder, which is trained through the Slot Attention path. Then, there are similarly no gradients flowing through the GSD and it is only updated with EMA (Appendix F). Is this understanding correct? If so, I think this could be stated more explicitly for clarity in the main text.\n- How are the dynamic number of slots $K$ actually implemented during training? From my understanding, different images in a batch may have different $K$, so this may need to be done with some masking of the softmax in Slot Attention, in which case a max number of slots still needs to be used. In that case, is the benefit in FLOPS only during inference for a single image?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper is well-motivated and the approach is novel, as far as I know. The authors show improvements over previous methods in terms of FG-ARI and downstream task performance and include additional experiments and analyses in the Appendix.", "weaknesses": "- The paper is missing some ablations that I think would be important in evaluating the significance of the method:\n    - How much of the improved performance is from just predicting the number of slots instead of using a fixed $K$? What if we use the predicted number of slots without the GSD?\n    - Conversely, how much is attributed to the GSD? What if we used a fixed $K$ with GSD?\n- What is the distribution of the number of objects for the different datasets? This would be important to interpreting the MAE values.\n- While I can appreciate the probabilistic interpretation of the model, I feel it does not add to the clarity of the paper and may be more appropriate for the appendix. Specifically, if I understand correctly, if $q(\\tilde{z}|x)$ is deterministic and $p(\\tilde{z})$ is uniform as in VQ-VAE, then the KL term is just a constant and not actually used to optimize the model?\n- The discussion of the different sampling strategies (Euclidean, Cosine, Gumbel) does not seem necessary for the main text since (from my understanding) the experiments in the main text are only done with the Cosine version? I do see additional experiments on other sampling strategies in the appendix, but if that is the case, this discussion can be removed from the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VPpM1tKl7F", "reviewer": "Reviewer_yF2M", "summary": "This paper focuses on grounded object-centric learning which can bind to specific object types. To achieve this goal, the authors take inspiration from Slot Attention, and introduce a Grounded Slot Dictionary to encode object properties and bind to different object types. This dictionary enables the model to conditionally sample the slots from different distributions. And the reasoning module with property transformation module enhances the interpretability of object property binding. The experiments show improvements on various object discovery benchmarks.", "questions": "The conventional slot attention based methods use reconstruction loss as the self-supervised objective to guide object decomposition. What is the relationship between the objectives used in this work and the recounstruction loss?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clear and makes sense. It takes inspiration from the recent research in binding problem and concludes three major challenges in unsupervised object discovery. And it takes the binding to canonical object properties and types as the primary problem to help simultaneously solve all three challenges.\n2. The design of conditional slot attention is novel. It employs the spectral decomposition for discrete mapping and enables the model to sample from different distributions corresponding to different object properties.\n3. The visualization of the separated object properties are interesting and shows interpretable Ground Slot Dictionary.", "weaknesses": "1. The experiments on more complex scenes are required. For example, multiple instances of the same object category, it would be interesting to show the property binding ability in this case.\n2. Does the model build on pre-trained backbones? Or a randomly initialized encoder $\\Phi$ is also sufficient to provide cues for spectral decomposition and discritization?\n3. I suggest authors to run the trained conditional slot attention with grounded slot dictionary on some video data, e.g., DAVIS-2017, to validate whether the slot dictionary can track objects across time and more vividly show the binding ability to object types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6J9m8cW1ED", "reviewer": "Reviewer_myKR", "summary": "The paper introduces Conditional Slot Attention (CoSA), a novel variant of Slot Attention that incorporates the concept of grounded representations. Unlike the original Slot Attention, CoSA utilizes a dynamic binding scheme using canonical object-level property vectors and parametric Gaussian distributions. This approach enables specialized slots that remain invariant to identity-preserving changes in object appearance. The proposed method is evaluated on multiple downstream tasks, including scene generation, composition, and task adaptation, and achieves competitive performance compared to Slot Attention in object discovery benchmarks.", "questions": "1. Besides performance gains, what evidence does the paper provide to show that the object-level property vectors are working effectively?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Unsupervised object discovery remains a challenge and an open question in the research community.\n2. The concept of the Grounded Slot Dictionary (GSD) module is logical, particularly the construction of a dictionary as outlined in Definition 1.\n3. The visualization of GSD binding in Figure 3 is both interesting and insightful for the community, providing evidence of the effectiveness of the GSD approach.", "weaknesses": "1. I would appreciate more ablation studies in the experiments section. The current version primarily presents the state-of-the-art (SOTA) performance for two case studies, but additional ablation studies would provide further insights into the specific contributions and the impact of different components or techniques employed in the proposed method.\n2. The author mentions that the method incorporates the object-level property vector, but there is a lack of evidence regarding how it functions. For instance, it is unclear whether the method can effectively discriminate between multiple instances with similar appearances. \n3. The visualization of the COCO results does not appear to be accurate, and it seems that the method may not be as applicable to real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)