Cycle Consistency Driven Object Discovery
Authors: Aniket Rajiv Didolkar
Anirudh Goyal
Yoshua Bengio
Venue: ICLR 2024 poster (2024)
Keywords: cycle consistency
object discovery
downstream RL
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Developing deep learning models that effectively learn object-centric representations, akin to human cognition, remains a challenging task. Existing approaches facilitate object discovery by representing objects as fixed-size vectors, called ``slots'' or ``object files''. While these approaches have shown promise in certain scenarios, they still exhibit certain limitations. First, they rely on architectural priors which can be unreliable and usually require meticulous engineering to identify the correct objects. Second, there has been a notable gap in investigating the practical utility of these representations in downstream tasks. To address the first limitation, we introduce a method that explicitly optimizes the constraint that each object in a scene should be associated with a distinct slot. We formalize this constraint by introducing  consistency objectives which are cyclic in nature. By integrating these consistency objectives into various existing slot-based object-centric methods, we showcase substantial improvements in object-discovery performance. These enhancements consistently hold true across both synthetic and real-world scenes, underscoring the effectiveness and adaptability of the proposed approach. To tackle the second limitation, we apply the learned object-centric representations from the proposed method to two downstream reinforcement learning tasks, demonstrating considerable performance enhancements compared to conventional slot-based and monolithic representation learning methods. Our results suggest that the proposed approach not only improves object discovery, but also provides richer features for downstream tasks.

Published as a conference paper at ICLR 2024
CYCLE CONSISTENCY DRIVEN OBJECT DISCOVERY
Aniket Didolkar Anirudh Goyal 2, Yoshua Bengio
ABSTRACT
Developing deep learning models that effectively learn object-centric representa- tions, akin to human cognition, remains challenging task. Existing approaches facilitate object discovery by representing objects as fixed-size vectors, called "slots" "object files". While these approaches have shown promise in cer- tain scenarios, they still exhibit certain limitations. First, they rely on architec- tural priors which can be unreliable and usually require meticulous engineering identify the correct objects. Second, there has been a notable gap in investigating the practical utility of these representations in downstream tasks, To address the first limitation, we introduce a method that explicitly optimizes the constraint that each object scene should be associated with distinct slot. We formalize this constraint by introducing consistency objectives which are cyclic in nature. By integrating these consistency objectives into various existing slot-based object- centric methods, we showcase substantial improvements in object-discovery per- formance. These enhancements consistently hold true across both synthetic and real-world scenes, underscoring the effectiveness and adaptability of the proposed approach. To tackle the second limitation, we apply the learned object-centric rep- resentations from the proposed method to two downstream reinforcement learning tasks, demonstrating considerable performance enhancements compared to con- ventional slot-based and monolithic representation learning methods. Our results suggest that the proposed approach not only improves object discovery, but also provides richer features for downstream tasks.
INTRODUCTION
Having object-based based reasoning capabilities important for solving many real-world problems. The world around us is complex, diverse, and full of distinct objects. Human beings possess the natural ability to parse and reason about these objects in their environment. Frequently, changing or ma- nipulating certain aspects of the world requires interacting with a single object combination of objects. For instance, driving a car necessitates maneuvering single object (the car) while avoiding collisions with other objects entities such as other cars, trees, and other obstacles. Developing object-hased reasoning capabilities is therefore crucial for improving the ability of deep learning models to understand and solve problems in the real world.
Unsupervised discovery of objects from a scene a challenging problem, as the notion of what an object refers to may be hard parse without any extra context. Many existing approaches (Greff et al., 2017; 2019; Burgess et al., 2019; Goyal et al., 2019; Locatello et al., 2020; Goyal et al., 2020; b021b; Ke et al., 2021; Goyal al., 2021a; Singh al., 2022) learn of slots represent objects, where each slot is fixed size vector. Most of these approaches use reconstruction loss coupled with certain architectural biases that depend on visual cues learn to segment objects into slots. There has been work that uses other auxiliary cues for supervision such as optical flow (Kipf et al., 2021) and depth prediction (Elsayed et al., 2022). However, architectural priors may not be always reliable and hence may not scale to real-world data while relying auxiliary information such as optical flow and motion not feasible since many datasets and scenes do not come with this information. To address these limitations, augment existing slot-based methods with two auxillary objectives called cycle consistency objectives.
Mila, University of Montreal, Google DeepMind Corresponding authors: adi dolkar kar 1kar1233 1 gma com

---

Published as a conference paper at ICLR 2024
SutAttioionn
Dooodor
5
S
Consistency
Siot Consistency
De D. Ny. D H
x [0x (Ng 80
oftmax ) #F
sofimant x 8 softmax x
Skat-FoluurwSit
Feature Sict Feature Si brity Maint
Cafe Croseertnayy Loca
Lpp Creaentoppy Losa
Figure 1: Cycle Consistency Objectives. Here show the general architecture of the model. The proposed approached augments two additional losses to existing object-centric methods (in this case, slot attention). These losses encourage object discovery directly in the latent space. We refer these losses as St.On-FEATURE-SI SLOT Consistency loss and FEATURE-SILOT- SO.--L.TTFFFFFFEEE consistency loss. The SLOT-FEATURE-SS SLOT consistency loss calculated as the cross-entropy between the slot- feature-slot similarity matrix and the identity matrix. The loss is computed the cross-entropy between the feature-slot-feature similarity matrix and custom matrix F, the details of which we provide in Section 2.
consistency
The proposed cycle consistency objectives operate directly on the latent representations of the slots and visual features (obtained from the neural encoder as shown in Figure 1). These objectives aug- ment the architectural priors used in slot attention style models with an extra layer of reliability by explicitly enforcing coherence and consistency between the representations of the visual features and the learned slots. To apply the objectives, we consider the visual features and slots as nodes in directed graph. The problem of object-discovery can be then formulated as that of adding the right edges into the graph such that (1) the outgoing edges from a set of features belonging to the same object should go into the same slot, and, (2) the outgoing edges from each slot should go into a distinct subset of features. Both these constraints are formulated into two cycle consistency ob- consistency loss and FEATURE-SL.OT- -----SS-OO--FFUUUEEEEE FEATURE consistency
jectives called SL.OT-FEATURE-SI.. loss. Further details regarding these objectives are elaborated in Section 2.
The proposed objectives simple and be integrated into any existing slot-based object- discovery method. We apply them to two object-discovery tasks and two downstream RL tasks. We find that augmenting slot-based methods with the proposed objectives improves object-discovery performance and exhibits stronger generalization to unseen scenarios. The learned slot- based rep- resentations also demonstrate strong transferrability downstreal RL tasks as compared to various baselines.
2 PROPOSED METHOD
In this section, we present the details of the proposed cycle consistency objectives and the un- derlying intuition behind them. The proposed method is designed to operate
set of N slots = (50. $1,...,8y } and a set M features = {foo fi...., fMr) which can be obtained using any suitable backbone such convolutional encoder. Similarly, the slots S can be obtained using any suitable object discovery or slot extractor method such as slot attention as shown in Figure 1:
S (oot_eetracto((F]
We denote the directed graph between the nodes and the features G. To find the correct edges to add to this graph we first score all possible edges. Next, we optimize the scores satisfy the following conditions
Preliminary Setup
2

---

Published as a conference paper at ICLR 2024
A set of features belonging to the same object must have outgoing edges to the same slot.
The outgoing edges from each slot should go to a distinct set features.
The score for an edge between a slot and a feature is computed by taking the dot product of their respective features. For example, outgoing edge from feature fi slot 8j is scored as
Se
([i,);)) ols +8))
T1
Here, Ti is the temperature which is generally set to 0.1 in our experiments. We convert these scores into probabilities by normalizing across all possible target nodes.
explols: - }8)))
p(f - 88) Et=N- exp(o(fi sk))
By computing these probabilities for all possible feature-slot pairs, we obtain feature-slot similarity matrix As-s E RMMNN Similarly, we compute slot-feature similarity matrix A.--f R.NxM. Note that, the scores in A,- normalized across all features.
We want both the similarity matrices to conform the above two conditions. For example, each row in Ag- which represents particular slot 8,, should assign highest probability to the features corresponding to that slot. Similarly, each row in A should assign highest probability to the slots that the feature corresponds to.
However, we cannot directly optimize these similarity matrices to satisfy the above conditions as do not have the ground truth slot feature assignments. Instead, consider cyclic paths consisting of two edges (1) SLOT-FEATURE-SUL -LOT paths Paths with an edge from slot to a paths Paths with
feature and another edge from feature to slot; (2) SAATURR-SLLOT--AATTUR an edge from a feature to a slot and another edge from a slot to a feature.
SLOT-FEATURE-SLOC Consistency Loss We want that for outgoing edge from slot s; to feature fee the outgoing edge from feature fü should cycle back to slot 8,. To develop an intuition about this, consider case where perfect object factorization has been achieved where each object represented by a distinct slot. In such case, for an edge going from a slot s; to a feature fex the outgoing edge from Ss will always come back s; because in the case of perfect factorization each feature belonging to a particular object will be represented by one slot only the slot that represents that particular object.
To achieve this constraint, we first calculate the scores of all possible i.c. paths with edges from a slot s; to feature fu and another edge from fx to slot sj follows- ø(s; + fk + sj) = p(s; > fk) p(fx + 8,). We convert this into probabilities by normalizing over possible paths beginning from slot s; and ending slot sj as follows
paths
exp(p(>: - fk) p(5s + +z)) Ag-ss-ooi,311 Σt= Ey= exp(p(s; + [w) plfx + s,))
(4)
A = softmax (aetmax(Agg//444 aris =1
(5)
We refer to As- E NxN as the slot-feature-slot similarity matrix as shown the Figure 1. Since want every path originating at slot to return back to slot 8;. we want the probabil- ities along the diagonal of A to be the highest hence we frame the consistency loss
SLOT
N-1 log(A (A 1og(As-A/asii
Lats =
(6)
1=0
3

---

Published as a conference paper at ICLR 2024
SAATURE-SLOT-FEATUUE Consistency Loss For path originating feature fi, belonging to an object o, going into a slot sІ. The outgoing edge from slot S₁ must go into any of the features f that represent Note that in this the outgoing path from slot 8 may not go back to the originating feature fi since multiple features belonging to one object map to the same slot.
To achieve this constraint, first calculate the caature-slot-feature similarity matrix which is shown in Figure as follows-
Af-s-1 -oftosss/smaaaass aris
Note that A E MMM In this case, we cannot optimize for the diagonals of A to have the highest prohabilities since FL.OT-FEATURE-SI. paths can have different source and target nodes. Therefore, we optimize by computing the cross-entropy between A and custom matrix F.
M-1
S - Σ il (o.0 1=0
(8)
Note that the above loss is only computed for the diagonal elements of A5-
FF is calculated as function of the features F output by the encoder as indicated in Figure 1. First we calculate the pairwise feature similarity values using and sparsify the feature similarity matrix based on threshold T. Consider two features from F fs and fj. The similarity score between these features is calculated as o.g fi The threshold value T computed T e (max({F= - min(F)) + min(F), where is a hyperparameter. to 0.8 unless specified otherwise. Once we obtain the sparse feature similarity matrix (denoted as F which is M x M matrix), we normalize across to convert the similarity scores into probabilities-
T2
our experiments, we set
f softmax aris =)
Training Details The proposed method can be applied on top any slot-basee object discovery method. To incorporate the proposed approach, add the the cycle consistency objectives to the original loss of the method. For example, slot attention (Locatello et al., 2020) utilizes a pixel-based reconstruction loss. On adding the proposed objectives, the final loss becomes:
Liinal = Crecon + Dafe' + Cfaf'
(10)
To set the hyperparameters for our approach, we select Aafs' and Afaf as 0.1 and 0.01, respectively, unless otherwise specified. We also employ an additional Exponential Moving Average (EMA) visual encoder. The EMA encoder is used for calculating F. This practice aligns with several self- supervised learning studies (He et al., 2019; Chen & He, 2020; Grill et al., 2020) and ensures that the features used in computing FF remain stable avoiding frequent changes caused by gradient updates in the visual encoder. Also, we detach the calculation of F from the gradient computation.
In methods involving multiple iterations of object discovery, such as slot attention, we apply the cycle consistency objectives to the slots obtained from each iteration of the method, unless explicitly stated otherwise.
We set Tj to 0.1 and to 0.01 unless specified otherwise.
Connections to k-means clustering Locatello et al. (2020) mention that slot attention can be seen as a form soft k-means clustering with dot product as the similarity function and a learned update rule. Taking this view, the proposed cycle consistency objectives can be seen further enforcing function on the clustering that acts by maximizing the similarity between features that belong to a cluster and minimizing the similarity of features that belong seperate clusters.
4

---

Published as a conference paper at ICLR 2024
Original
Rec.
Generated
Slot
Slot 2
Slot 3
Slot 4
Slot 5
Slot 6
Slate
Slate
Cyclic
Figure 2: Causal World Slot Masks In this figure, show the learned slot masks for both the pre- trained models. We can see that proposed approach captures obejcts more clearly than the baseline Slate model.
3 RELATED WORK
Unsupervised Object Discovery Our work addresses the problem object discovery in visual scenes. While this challenge has been tackled using supervised, semi-supervised, and unsupervised methods, our approach falls into the category of unsupervised object discovery techniques (Greff et al., 2017: 2019; Burgess et al., 2019; Eslami et al., 2016; Lin et al., 2020; Goyal et al., 2020; Crawford & Pineau, 2019; Zoran et al., 2021; Locatello et al., 2020; Ke et al., 2021; Engelcke al., 2019; Goyal et al., 2019 2021b). These papers mainly consider only synthetic datasets. They rely on set of vectors, known as slots, to represent objects, and use various architectural priors and objectives
group image features into slots that correspond distinct objects. Many recent works (Singh et al., 2022: Jia et al., 2022; Seitzer et al., 2023; Choudhury et al., 2021; Wang et al., 2022: 2023) have proposed improvements that have managed to scale these slot- blot-based methods to real-world datasets. While most of these datasets use pixel-wise reconstruction objectives, (Seitzer et al., 2023; Wang et al., 2023) are the only two works that utilize objectives in the latent space like the proposed approach. (Seitzer et al., 2023) introduce model called Dinosaur which applies slot attention on the features output by a pretrained self-supervised encoder such as DINO (Caron et al., 2021) and use feature reconstruction attention to the features output by pretrained encoder. They use an objective which is very similar the cycle consistency objective proposed in this paper. The main difference between work and theirs is that they mostly consider pretrained and frozen encoder backbones while we mainly consider encoders trained from scratch.
the training objective. (Wang et al., 2023) also apply slot
Cycle Consistency Cycle consistency is a concept in deep lcarning that enables the learning of a consistent mapping between two domains in cases where 1-to-1 data is not available. It relies the property of transitivity to enforce structure in representations, and has been successfully used for learning good representations images, videos, and language. Numerous studies (Wang al., 2013; 2014 4; Wilson & Snavely, 2013; Zach et al., 2010; Zhou et al., 2015a; 2016; 2015b; Hoffman al., 2018; Zhu et al., 2017) have employed cycle consistency in image-based tasks such as co- segmentation, domain transfer, and image matching. In these works, cycle consistency is typically used as an objective function that ensures the consistency of the mapping between the source and target domains, and the inverse mapping from the target domain back to the source domain. For example, in (Zhu al., 2017), the source and target domains are images from distinct styles. Cycle consistency has also been utilized Various studies (Dwibedi et al., 2019; Wang et al., 2019; Li et al., 2019; Lai & Xie, 2019; Jabri et al., 2020: Hadji et al., 2021) have used the cycle consistency objective enforce temporal consistency in videos, ensuring that there is a path forward from frame to frame k. and the path backward from frame & lands back on frame consistency objective for object discovery. Additionally, the cycle consistency objective is applied to the latent space consisting of the slots and features in this case while previous studies have primarily focused on applying the objective to the domains of language, images, or video.
self-supervised representation learning technique in videos.
Our work differs from previous works that apply the cycle
4 EXPERIMENTS
In our experiments, we assess the performance of the proposed objectives on object-discover tasks. Then evaluate the efficacy of the learned representation distinct downstream RI. tasks.
5

---

Published as a conference paper at ICLR 2024
Object Discovery Approaches We consider four unsupervised object discovery approaches as our base approaches: (1) Slot Attention (Locatello et al., 2020) Slot-attention uses a top-down iterative attention mechanism to discover slots from image features (2) SLATE (Singh et al., 2022) SLATE also uses slot-attention to discover slots but replaces the convolutional encoder in slot attention by dVAE (van den Oord et al., 2017; Ramesh al., 2021) and performs reconstruction
in vector-quantzed space using an auto-regressive transformer; (2) Dinosaur (Seitzer et al., 2023) -Dioosaur applies slot attention on the features from a pretrained encoder; (4) MoTok (Bao al., 2023) MoTokk applies slot-attention to discover objects in videos. It uses motion segmentation annotations supervise the attention maps in slot attention. We incorporate the proposed CYCLIC objectives in each of these models.
Datasets and Environments For object-discovery we consider both synthetic and real-world datasets. For the synthetic datasets, we use Shapestacks (Groth et al., 2018), ObjectsRoom (Kabra al., 2019), ClevrTex (Karazija al., 2021). We evaluate the segmentation performance on these synthetic datasets using the Adjusted Rand Index (ART) (Hubert & Arabie, 1985) and reconstruction performance using the mean- wean-squared error (MSE). Specifically. we calculate FG-ARI for all these datasets which is same as ARI but ignores the background information. For real-world datasets, we consider the task of multi-object segmentation in COCO (Lin et al.. 2014) and scannet (Dai al., 2017) datasets. For this task, we report the AP score (Everingham et al., 2014), precision score and recall score.
We also apply the proposed approach to object discovery in videos where we consider the Movi- video dataset (Greff et al., 2022). We use MoTok (Bao et al., 2023) as our base model for this experiment. For the experiments with Dinosaur (Seitzer et al., 2023), we use the Movi-C and Movi- datasets (Greff et al., 2022). We consider them as image datasets for these experiments rather than as video datasets.
For our downstream RL tasks we use the atari and causal world environments (Ahmed et al., 2020). In atari, we consider various different games and report the mean returns across 10 episodes similar to (Chen et al., 2021). ). In causal world, we consider variants of the object goal task where the agent is tasked with moving the robotic arm towards target object. We use success rate as the performance metric in causal world.
4.1 OВJECT DISCOVERY
Synthetic Datasets We study the object discovery performance of the proposed approach syn- thetic tasks. We follow the setup used in (Dittadi et al., 2022). We augment the slot-attention auto-encoder (Locatello et al., 2020) model with the cycle consistency objectives. Slot attention uses convolutional encoder to obtain image features and a mixture-basek decoder for reconstruc- tion (Watters et al., 2019). The encoder outputs features F R(xtxD, where t = 64. Before applying the cycle consistency objectives, we downsample the features to f = 16 and then project them using an MLP to have the same dimension as the slots. We also normalize the slots using their 1.2 norm before applying the cycle consistency objectives. More details regarding the architecture and
can be found in the Appendix.
Table 1: Synthetic Datasets Segmentation. In this table we compare the slot attention model augmented with the proposed CYCLIC objectives against the original slot attention model (Locatello al., 2020). As shown in the table, we observe that the proposed objectives result in performance gains across all the considered datasets. Results averaged across 3 seeds.
ObjoctsRoom
ShapeStacks
CleveTex
Model
Ayta' Azaf'
MSB
FG-ARI +
MSE
FG-ARI
MSE
FG-ARI t
Slot-Artention
0
0.0018 100000
0.7819
1.004
0.7738
0.007 200.00
0.6240 2.
+ CYCLIC
2>0
0
0.0019++0
0.7832
0.004
0.5491 +0.422
0.007=0.c0m
0.664002005
+ CYCLaC
0
0.8120
g.004
0.7755
0.4974
+ CYCLIC
>0 0
>
0.0015
0.8169
0.0037
20000
0.7838
0.007=2..0m:
0.7245
We took the result for SA from (Karazija et al., 2021), as we were unable to reproduce the same result in a statistically consistent manner. With our implementation, got FG-ART score of 0.586441001 To be fair, we have reported the score from the original paper (Karazija et al., 2021).

---

Published as a conference paper at ICLR 2024
Results. Table 1 presents the quantitative results on all three datasets. We observe that augmenting Slot Attention with the proposed objectives leads to better factorization (as measured by FG-ARI) and superior reconstruction (measured by MSE). We also observe that while the presence of only one of the objectives does not affect the reconstruction performance significantly, having both objectives crucial for achieving good factorization.
ClevrTex Generalization. The ClevrTex dataset provides two generalization splits which allow to probe the generalization capabilities the proposed approach: (1) CAMO Contains scenes where certain objects are camouflaged, and (2) OOD uses 25 new materials that were not seen during training. We train the models on the full training set of ClevrTex and transfer them to both the splits. Figure 3 presents the results, which demon- strate that the proposed method consistently outperforms the baseline the generalization splits.
6.5
SA. Cydlc
6.5 Fall
Camo
000
Figure 3: ClevrTex Generalization. Here we present transfer results on clevrtex. We find that the proposed approach outperforms the baseline on both the transfer splits. Results averaged across 5 seeds.
Table 2: Real World Datasets Segmentation. Here we present results for multi-object segmenta- tion on real-world datasets. We augment the improved Slate model presented in (Jia et al., 2022) with the proposed cyclic objectives. We can see that the proposed approach outperforms all base- lines on all metrics both datasets. Results averaged across seeds.
enco
Model
AP603 1
PQ
Procision
Recall
AP
Precioion
Recall
MONet (Burgess et 2019)
118.22
12.5-111
16. 16.1
21.9
24.8
24.6116
31.0g11
40.7.118
IODINE (Caeeff 2019)
40.11
6.221.1
9.
137422
1864442
Slx Attenion (Loctello
2020)
08:105
1.
1.
7.3
9.0
124
1833
Implicit Slot Attention (Chang al., 2022)
12.8
13.7
20.4
24.6
21.4
23:42:15
29.1
34.5.21
BO Slate (Ji al., 2022)
16.64
17.48
25.49 1.2
31
24.67
23.55
34.03
38.74.00
BO-Slate CYCLIC
29.20
26.09.
37.03-1144
Model
FG-ARI
MoTok
PG-ARI
mBO
MG-ARI
DINOSAUR (VIT-BW)
68.9.g0.4
38.0
65.1
65.
33.5
MoToak + Cyclic
72.48
CYCLIC
72.4
40.2
89.7
37.2
Table Cycle Consistency Objectives with mo- tion guidance. Here we incorporate the cycle consistency objectives into the MoTok model pre- sented in Bao et al. (2023). We find that incor- porating the proposed objectives into MoTok re- sults in improved performance on the considered video-based object discovery task. We consider the Movi-E dataset for this experiment.
Table Cycle Consistency Objectives with Pretrained Encoders In this table we demon- strate the improvements achieved by the cyclic objectives when added to the DINOSAUR model from (Seitzer et al., 2023) based on the pre- trained ViT-B/8 encoder. We can that pro- posed method achieves superior performance on both the datasets.
Cycle Consistency Objective for Video Datasets considering the MoTok model (Bao et al., 2023) as the base model. MoTok uses motion segmenta- tion annotations auxiliary information discover objects in the video. We incorporate the cycle consistency objectives in the slot attention module from the MoTok model presented in Bao et al. (2023). We present the results for this experiment Table We find that augmenting MoTok with the cycle consistency objectives result in improved performance on the Movi-E dataset compared to MoTok baseline.
We the Movi-E video dataset for this study
Cycle Consistency Objective with Pretrained Encoders We use the DINOSAUR model (Seitzer et al., 2023) as the base model. DINOSAUR applies slot attention to the pretrained features obtained from a DINO encoder (Caron et al., 2021). For the proposed method, we add the cyclic objectives to the DINOSAUR model, applying them only to the slots obtained from the last iteration of slot attention. In this case, we do not use an EMA encoder, as the DINO image encoder is kept frozen through training. We perform this comparison results in Table 4 show that DINOSAUR augmented with cyclic objectives outperform the base DINOSAUR model both the datasets. This further shows that the proposed cycle consistency
the MOVI datasets Greff et al. (2022). The

---

Published as a conference paper at ICLR 2024
objectives are agnostic to the underlying object discovery approach and only require the underlying approach to use slot attention.
Effect of Loss coefficients We study the effect of the loss coefficients (Anfs' and ffsf) ft) on the ClevrTex dataset in Figure 4. We can see that for the variation Aefx' (Figure (a)), the per- formance degrades rapidly for higher values. We conjecture that the reason behind this is that SLOT-FEATURE-SSLT consistency can be trivially satisfied if all features are mapped to one slot. Therefore, having a high Aafa' may bias the model towards the trivial solution thereby hurting per- formance. From Figure 4(b), we can see that the performance variation is much more stable in the case where vary the value of Asaf'. This shows that the model is fairly agnostic to the value of Afst'-
Real-World Datasets For these ex- periments, we use the BO-SLATE model (Jia et al., 2022) our base model. BO-Slate is an improved version of Slate (Singh et al., 2022) where the main improvements come from having learnable slot initializa- tions. For the proposed method, we add the cycle consistency objectives to the BO-SLATE model. We refer to this model as BO-Slate + Cyclic. We defer further details to the Appendix. The underlying task which con- sider for real-world datasets multi- object segmentations. The results for this task are presented in Table 2. We use the same versions of the datasets used in (Jia et al., 2022). We compare the proposed approach to various object-centric models that exist in literature. We observe that the BO-Slate model augmented with the proposed objectives outperforms all the baselines all the metrics. More information about these metrics is presented the appendix.
(a) A.pe' variation for Ag.s" = 0.01
(b) A variation for Axys' 0.1
Figure 4: Effect of Loss Coefficients (a) In this we vary the SLOT-FEATURE-SLOO weight (Aafst) and keep the FFEATURE-SS.OT-FFATTUI
weight (Afsp) fixed to 0.01. We can see that the model reaches peak performance for Aafa' = 0.1 and the performance degrades rapidly for higher values. (b) In this we vary Afsf' and keep Axfs' fixed to 0.1. We can that the performance is much more stable here as compared to varying Aufs'-
4.2
REPRESENTATION LEARNING FOR DOWNSTREAM RL TASKS
Atari One important aspect of any represen- tation learning method is that the learned rep- resentations should be useful in downstream tasks. this section, we explore the useful- ness of the proposed approach in the context of atari games. (Chen et al., 2021) introduced the decision transformer model which learns to play various games in the Atari suite by im- itating suboptimal trajectories from learned agent. We present more details of the decision transformer model in Appendix Section 9. In this work, we change the monolithic state rep- resentation used in decision transformer to an object-centric one.
Table 5: Atari Here we present results on vari- ous games from the Atari suite. Results averaged across 5 seeds.
Game
DT
DT + SA
DT + SA + Cyelic
Pong
11.0
7.4
14.8 2
70.6
93.4+24121
110.2-11100
Seaquest
1172.4
124.0
663/24111114
Qbert
$485.2 199..16
5275.2
7393.8 100
Asterix
523.333
471.667
785.0
Assault
387.333
430.667 4883003
462.0+
Boxing
78.0
77.333
78.667 209446720943
Carnival
486.0
814.0g.10.42
836.667 1.1.277
Frocway
26.667=04m
21.0::0166
23.0
The monolithic state representation of an obser- vation is D D-dimensional vector obtained by passing the atari observations through a convo- lutional encoder. Note that each observation is a stack of 4 frames. To obtain the correspond- ing object-centric version of this, convolutional encoder and the slot attention module from (Locatello et al., 2020) to encode each observation. Therefore, each observation is encoded into N slots instead of single vector.
Crazy Climber
765644.0
51490.0
94254.0.47500
BankIleist
11.4
105
105.0
144.8 116
Space Invaders
972
392.0
598.2 14.147
MsPacian
1461 44:0.s
1220.8 2:2.00
1900.0 1000.0+226.3644
the
To ensure that slots learn the correct object-centric representation we augment the decision trans- former loss with the slot attention loss L = LDT Entconstruction. Additionally, also add the
8

---

Published as a conference paper at ICLR 2024
cycle consistency objectives to the loss -= LDT Reconstruutionn + A + Afef Laff- We compare our method to the baseline decision transformer and an object-centric variant of deci- sion transformer (DT SA) where have the slot attention style reconstruction loss but omit the cycle consistency objectives.
The performance comparison in Table 5 reveals that the decision transformer, when augmented with object-centric representations solely obtained from slot attention (DT + SA) exhibits competi- tive performance across most games the original decision transformer (DT). However, when the object-centric decision transformer is combined with the proposed cycle consistency objectives, surpasses the baseline decision transformer in 10 of 13 games. This outcome showcases the
significance of the proposed cycle consistency objective in learning strong object-centric represen- tations capable of performing downstream tasks.
Causal World We consider the object goal task from the causal world environment (Ahmed et al., 2020). We follow the same setup (Yoon al., 2023). In the Object Goal task, the agent controls a tri-fing tri-finger robot placed in bowl. The bowl contains a target object and a few distractor objects. The task for the agent to move the trifinger robot towards the target objects without touching the distractors.
Similar to (Yoon et al., 2023), we first pretrain a Slate model augmented with the cycle con- sistency objectives on random rollouts from the object goal task. The baseline for this task is Slate model pretrained on the same dataset of random rollouts. We visualize the learned ob- ject masks in Figure 2. We can see that the the proposed objectives enable the model learn more accurate object masks as compared to the baseline.
Slate
80
Slate Cyclic
60
40
Next, we train a policy using proximal pol- icy optimization (PPO) (Schulman al., 2017). We use a transformer-base The inputs are the slot representations from the object-centric models along with a CLS token. The resulting output corresponding to the CL.S token is used to output the action distribution and the value. The object-centric model is kept frozen at this stage.
20
policy network.
Static Initiaa Robot Pos.
Dynamic Initia Robot Pos.
Task
Figure 5: Causal World RL In this figure, we present results for both causal world tasks. We can see that Slate models pretrained with the cyclic objectives achieve superior success rate com- pared to the baseline Slate Model. Results aver- aged across 5 seeds.
We consider two variants of the object-goal task (1) Static Initial Robot Position Initial posi- tion of the tri-finger robot is the same across episodes; (2) Dynamic Initial Robot Position Initial position of the tri-finger is different across episodes. We show results for this task in Figure 5. We can see that both cases, models pretrained with the eyclic objective outperform those without it. We present more details about the pretraining procedure and the policy in the Appendix Section 10.
FUTURE WORK AND LIMITATIONS
Significant research efforts have been dedicated learning object-centric representations. However, the evaluation of these representations has primarily focused on unsupervised segmentation perfor- mance using metries like ARI IoU. Regrettably, there is a dearth of studies demonstrating the practical utility of object-centric representations across diverse downstream tasks. To address this gap, our work takes a step forward by showcasing the effectiveness of our approach in the context of Atari and Causal World, We do not address all limitations of object-centric methods on downstream tasks, more work is still needed to showcase the effectiveness of these models on more complex and real-world downstream tasks. Moving forward, our objective is to shift our focus towards develop- ing object-centric representations that prove valuable a wide array of downstream tasks, spanning reinforcement learning visual tasks like visual question answering and captioning. We aim to explore the efficacy of cycle consistency objectives learning such representations and study what is lacking in building more pervasive object-centric representations.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENTS
This research was made possible in part by compute resources, software, and technical support pro- vided by Mila (mila.quebeec.. The authors extend their appreciation to Nanda Harishankar Krishna for assistance in refining Figure 1. Gratitude is also owed to Ayush Chakravarthy and Vedant Shah for their valuable contributions during brainstorming sessions. Additionally, the authors would like to thank Nanda Harishankar Krishna, Vedant Shah, and Jithendaraa Subramanian for their thoughtful reviews and suggestions to improve this manuscript.
REFERENCES
Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Manuel Wüthrich, Yoshua Bengio, Bern- hard Schölkopf, and Stefan Bauer. Causalworld: A robotic manipulation benchmark for causal structure and transfer learning, 2020.
Zhipeng Bao, Pavel Tokmakov, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Object discovery from motion-guided tokens, 2023.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew M. Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. CoRR, abs/1901.1 10011.13990 1390, 2019. URL http::/ http://aaxvvvvvo http:://axxwv..rr/abbb////////// /1lp/////// 1911.... 11390.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin. Emerging properties in self-super vision transformers. CoRR, abs/2104.14299 2021. URL
https: //arivv org/ abs//104/44.. 1.0004..4299 14294.
Michael Chang. Thomas L. Griffiths, and Sergey Levine. Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation. 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, 2021.
Xinlei Chen and Kaiming Hc. Exploring simple siamese representation lcarning. CoRR, URL http:::tt////att http://aawwvvv....eeee// wttp//ss///oooreeoeeeeeee 10566.
2020.
Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Unsupervised part discovery from contrastive reconstruction. CoRR, abs/21 1.06649, 2021. URL https://arxiz: org/ abstgg//////// abs/abss/12111 21111. 06349.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovarive Applications of Arrificial Intelligence Conference and Ninth AAAI Symposium Educational Advances in Artificial Intelligence. AAAI' 19/AAA 19. AAAI Press, 2019. ISBN 178-1-57735- http:///ooooogg/////
809-1. doi: 10.1609/aaai (1669//aai.v33111133033444 URL https: //tp///di../.. org wttp://./......../ v33101. 33013412.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
Andrea Dittadi, Samuele Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and rohustness implications in object-centrie learning. In International Conference on Ma- chine Learning, 2022.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanct, and Andrew Zisserman. Temporal cycle- consistency learning. CoRR, abs/ 1904.07466 2019. URL http::pttaaeeeeeeeeeeeeeeeeeeeet /ttpxa/xiivv////////////aa///aaa .org/ abs/19c4 /ttp//994..7740000 07846.
Gamaleldin Fathy Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Curtis Mozer, and Thomas Kipf. SAVi++: Towards end-to-end object-centric learning from real-world videos. In Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL httph:tt/////... wpwp/eveee..eeeeee//eeee. www.eeeeeeeeeeee.eeeeeeeeeeeeeee forum? wew.mnidd=fFss... LxKS..
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene infer- and sampling with object-centric latent representations. arXiv preprint 1907 13052, 2019.
10

---

Published as a conference paper at ICLR 2024
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, koray kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scone understanding with generative models. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https: /pppccediiggg....p nourips. cc/ haper_files/ httpllesspppeee/eeeeeeee httpp/apprr/////ee 20aer/2166. w/t.e//iilee///
3aee77000941133360090 33 7 .dd..
Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111:98-136 2014.
Saced Ghorbani, Kimia Mahdaviani, Anne Thaler, Konrad P. Körding, Douglas James Cook, Gunnar Blohm, and Nikolaus F. Troje. Movi: A large multipurpose motion and video dataset. CoRR, abs/2003.01 08888 2020. org/ /bss/003..... 01883.
URL wwwpt:tt///// / /ttp://aaavvv...///aeeeeeeiieeea
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv:/909. 2019.
Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blundell, Yoshua Bengio, and Michael Mozer. Object files and schemata: Factorizing declarative and procedural knowledge in dynamical systems. arXiv preprint arXiv:2006. 162222 2020.
Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, and Yoshua Bengio. Coordination among neural modules through a shared global workspace. arXiv preprint arXiv: 2103011000, 2021a.
Anirudh Goyal, Aniket Rajiv Didolkar, Nan Rosemary Kc, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael Curtis Mozer, and Yoshua Bengio. Neural production systems. In Beygelzimer, Dauphin, P. Liang, and J. Wortman Vaughan (cds.), Advances in Neural Information Processing Systems, 2021b. URL https: /
http//ppcceeiic.ceeeee/ee net/ / forum? ? www.idddxxccuaaeeeeeeeeeee
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. CoRR, abs/1708.03498, 2017. URL http:// hat/ivivvv.v.. wwtp/aa///////ooo////o/////.//// 1708103498.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew M. Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. CoRR, htt//1933.00455 2019. URL http: / arxiv. org/ abs/ 903. 00450.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Hermann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Is- sam Hadj Laradji, Hsueh-Ti Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour. Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitz- mann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable dataset generator. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3739 1771- 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec. Pierre Richemond, Elena Buchatskaya, Carl Docrsch, Bemaado Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self- supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 21271-21284, Curran Associates, neurips. cc/paper f wet.iee/pppee////000 Inc., 2020. URL bttts::/proceeddie Pawpee99---PPPPe.eeeeee Paper.pdds #33cca80d5c4ee71110 7b8192b2958
Oliver Groth, Fabian B. Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision-based physical intuition for generalised object stacking. ArXiv, abs/1804.08018, 2018.
Isma Hadji, Konstantinos G. Derpanis, and Allan Jepson. Representation learning via global temporal alignment and cycle-oonsistency. In Proceedings of the IEEE/CVF Conference on Compuder Vision and Pattern Recognition (CVPR), pp. 11068-11077, June 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. CoRR, abs/1911.05722, 2019. URL httt:://arxitt .org abs/ /1911. 05 722.
Judy Hoffmann, Erie Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrel. CyCADA:
consistent adversarial domain adaptation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1989-1998. PMIR, 10-15 Jul 2018. URL https: httpppa////rceeeeeeee procedd ngs. m1 .presss////eee / v80/ haf fman 8a html.
11

---

Published as a conference paper at ICLR 2024
Lawrence Hubert and Phipps Arabie. Comparing partitions, Journal Classification, 2:193-218, 1985.
Allan Jabri, Andrew Owens, and Alexei A. Efros. Space-time corresponneecee contrastive random walk.
CoRR, abs/2006. 2011/2006.14613, 2020. URL httpt:ttp///eet /ttps:///axiiie .rrg httrrg/bb///00... 14....
Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object-centric learning with query optimization, 2022.
URL hww.t:tt/////mm /ttp:s///aavvvv/ie wwwpaiss.rrr////o/oeooooobb /wtp..22......0 08990.
Rishahh Kabra, Chris Burgess, Loic Matthey, Raphael Lopez Kaufman, Klaus Greff, Malcolm Reynolds, and Alexander Lerchner. Multi-ohject datasets. https:/ggtthu..com/// wwtp:///eeeiiimmmmmiiiicceeeaaaa 2019.
Laurynas Karazija, Iro Laina, and C. Rupprecht. Clevrtex: A texture-rich benchmark for unsupervised multi- object segmentation. ArXiv, abs/2111. 2021.
Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Christopher Pal. Systematic evaluation of causal discovery in visual model based reinforcement learning. arXiv preprint arXiv: 20700:207,,0088,, 2021.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel datasct for fine-gras fine-grained image categorization Stanford dogs. 2012.
Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. CoRR, abs/2111.12599 2021. URL /taptppst//aiii..t.//p org/ abs 2111 12594.
Jonathan Krause, Michael Stark, Jia Deng, Li Fei-Fei. 3d object representations ane-grained catego- rization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Zihang Lai and Weidi Xie. Self-supervised learning for video correspondence flow. CoRR, abs/1905.00877 2019. URL http: //tp:i//////iv/// http://awxwwv....a//.o/oe/ wwtpg//////a.s....//////////
Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming Hsuan Yang. Joint-task self- supervised learning for temporal correspondence. CoRR, abs/1909. 2019. URL http://arxx... org/abs/ 909. 11895.
Tsung-Ys Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: unsupervised object-oriented scene representation via spatial attention and decompo- sition. CoRR, abs/2001 abs/2001.02207,, 2020. URL http:
/est:://arxiae /arxivv.vvv//v////////a////aaav/ org/ abs/ abs/2001. 2001. 02407.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy. and Thomas Kipf. Object-centric learning with slot at- tention. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 11525-11538. Curran As- sociates, Inc., 2020. URL https: pettes://pee /ttp////eeeddineeeeeeeeee 2al1af8c02abbsee112eee6
3bc0t- Fww.oot--aPpPpppeeee pdf.
Maria-Eloe Elena Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In IEEE Confer- ence on Computer Vision and Pattern Recognition, volume 2, pp. 1447 1454, 2006.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. CoRR, abs/2102.12092. 2021. URL https://arxiii org/abs / 2202.2111111 12092.
John Schulman. Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. CoRR, 01s/1707.06377, 2017. URL http:://taxiiii wwtp:/.iiorrrooooooooo..oo org/ abs 70707 707.07377
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. In The Eleventh International Conference on Learning Represen- tations, 2023, URL https:////// twtttps://opo.eeee www.rrvviowwwwoooooo/ooo..o.w..e net
Gautam Singh, Fei Deng, and Sungii Ahn. Illiterate D learns to compose. In International Conference on Learning Representations, 2022. URL https::::: opwnreviieeeeeeeeeeeo ew. mettfform? d=hoorvowe3ohh
12

---

Published as a conference paper at ICLR 2024
Aaron den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learn- ing. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, Cur- ran Associates, Inc., 2017. URL https: http///ddcc.c...cc//eeee/cec...c 7a 9 1999811700 1el1766335 weteeeaaaaaceeeeeeeeee3333 www.eeeeeeapape.eeeeeeeeeeeeeeee www.eccpinnns..uu........ www.ripp.ccpppper////eee/ pttps//////pp///////////
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongic. Caltech-uesd birds 200. Technical Report CNS- TR-201 California Institute of Technology, 2011.
Fan Wang, Qixing Huang, and Leonidas J. Guibas. Image segmentation via consistent functional maps. In 2013 TEEE International Conference on Computer Vision, pp. 849-856, 2013. doi: 10.1 109//CCV.2013110
Fan Wang, Qixing Huang, Maks Ovsjanikov, and Leonidas J. Guibas. Unsupervised multi-class joint image segmentation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3142-3149 2014. doi: 10.1109/C 2014.402.
Ning Wang, Yibing Song, Chao Ma, Wengang Zhou, Wei Liu, and Houqiang Li. Unsupervised deep tracking. CoRR, abs//194.0118m 01828, 2019, URL http://arxt: woog/rrgass.... abs/ 1904. C1828.
Yangtao Wang, XI Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufreydaz. Self- supervised transformers for unsupervised object discovery using normalized cut. 2022 IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 14523-14533, 2022.
Ziyu Wang, Mike Zheng Shou, and Mengmi Zhang. Object-centric learning with eyclic walks between parts and whole, 2023.
Nicholas Watters, Loic Matthey, Christopher Burgess, and Alexander Lerchner. Spatial broadcast decoder: simple architecture for learning disentangled representations in vaes. CoRR, abs/1901.07011 2019. URL. http: / arx xiv. wwtpwwwwwwwwg.////oo//o/a/baaa 1901 1011900000000 37017.
Kyle Wilson and Noah Snavely. Network principles for sfm: Disambiguating repeated structures with local context. In 2013 IEEE International Conference on Computer Vision, pp. 513-520, 2013. doi: 10.1109/ ICCV.2013.69.
Jaesik Yoon, Yi-Fu Wu, Heechul Bac, and Sungjin Ahn. An invostigation into pre-training object-centric representations for reinforcement learning, 2023.
Christopher 7ach, Manfred Klopschitz, and Mare Pollefeys. Disambiguating visual relations using loop con- straints. In 2010 IEFE Computer Society Conference Computer Vision and Pattern Recognition, pp- 1436-1433, 2010.
doi: 10.1109CCPR222000553300
Tinghui Zhou, Yong Jae Lee, Stella X. Yu, and Alexei A. Efros. Flowweb: Joint image set alignment by weaving consistent, pixel-wise correspondences. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1191-1200, 2015a. doi: 10. 1099CVPP.2215..29872..
Tinghui Zhou, Philipp Krähenbühh,, Mathieu Aubry, Qixing Huang, and Alexei A. Efros. Learning dense correspondence via 3d-guided eycle consistency. CoRR, Abs/1604.05383. 2016. URL http://arxiv:
org/abs/ / /1///160440 604. 05383.
Xiaowei Zhou, Menglong 7hu, and Kostas Daniilidis. Multi-image matching via fast alternating minimization. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 4032-4040, 2015b. doi: 10.1 ICCV.2015.459,
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consisee cele-consistent adversarial networks. CoRR, 1ts//7733.1599, 2017. URL http::/arxiv..ot org/ /bs/ 1703. 10593.
Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo J. Rezende. Parts: Unsupervised segmenta- tion with slots, attention and independence maximization. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10419 10427, 2021. doi: 10.110////C 202199922.220011111 1,0027.
13

---

Published as a conference paper at ICLR 2024
ObjectsRoom
ShapeStacks
ClevrTex
Movi-C
Movi-E
Figure 6: Here we show an example image from each synthetic dataset that we consider.
UbyectsRooes
ShapeStacks
Layer
C
Channels
Kernel Size Padking Strde C hannels Kernel Size Paaking Stride Channels Kernel Size Faoding
Stridd
Convolutional Encoder
Conv
32
Conv
32
32
64
Conv
32
32
Convolunccal Decoder
Trans
32
32
Trns
Com:. Trans
Table 6: Detailed architecture for the encoder and decoder used by the slot attention model used in synthetic dataset experiments. Note that we use relu activations after every layer except the last layer.
APPENDIX
SYNTHETIC DATASET EXPERIMENTS
We consider the Shapestacks (Groth et al., 2018), ObjectsRoom(Karra et al., 2019), ClevrTex (Karazija et al., 2021), and MOVi datasets (Ghorbani et al., 2020). Figure 6 shows an example image from each dataset.
Slot A Attention Implementation Details For ObjectsRoom, ShapeStacks, and ClevrTex, we use slot attention as our base model. Table 6 shows the detailed architecture of the convolutional encoder and decoder used by the slot attention model. Table 7 indicates values various hyperparameters used in these experiments. For each experiment, use RTX8000 GPU.
Ablation on Lyff application In our case, we calculate Lfsf' follows: Lfsf' log(P(f; fi)) = Therefore, instead of computing the loss for all pairs of and j, we only compute for the cases where i is equal to j. This design choice is made because the supervision signal, p² is function of the pairwise feature similarity values. Obtaining accurate pairwise feature similarity values for all i, in a model trained from scratch is challenging. Hence, we limit the loss calculation to only the diagonal elements the matrix, where = j. To assess the significance of this design choice, model that computes fef for all i, The results this study are presented Table 8. Notably, computing L/r solely for i yields considerably better performance compared to computing for all i.j.
compare the performance of the proposed model with
Effect of applying the objectives on all iterations of slot attention One implementation detail for the proposed method is that we apply the cycle consistency objectives to slots from all iterations of slot attention. We ablate this design choice by comparing against model where the cycle consistency objectives are only applied to the last iteration slot-attention. We present the results in Table 9. We can that the performance drops significantly when the eycle consistency objectives are only applied to the last iteration thus showing the importance of applying the objectives on all iterations.
14

---

Published as a conference paper at ICLR 2024
ObjectsRoom
ShapeStacks
ClevrTex
Num. Slots
7
7
11
Num. Iter
3
3
3
Slot size
64
64
64
MLP size
128
128
128
Batch Size
64
64
64
Optimizer
Adam
Adam
Adam
LR
0.0004
0.0004
0.0004
Total steps
500k
500k
500k
Warmup Steps
10k
10k
5k
Decay Steps
100k
100k
50k
Xafe'
0.1
0.1
0.1
Afaf'
0.01
0.01
0.01
T
0.1
0.1
0.1
Tafs'
1
1
1
Tfsf' 0,
0.01
0.01
0.01
0.8
0.8
0.8
Downsampled feature size
16 x 16
16 x 16
32 x 32
EMA Decay rate
0.995
0.995
0.995
Table 7: This table indicates all the values for various hyperparameters used in the synthetic dataset experiments.
ObjectsRoom FG-ARI
ShapeStacks
ClevrTex
Model
FG-ARI
FG-ARI
SA + Cyclic (Lyaf
i,j)
0.7341 2007
0.7161 10001
0.6349+0.11
SA + Cyclic (Cjag V i = j)
0.8169
0.7838
0.7245
Table Here we compare the performance of the model where Cya is computed for all the model which computes Lfs for all i =.j We can see that the latter performs better than former thus showing the importance of computing the Lysf' only for i =.. We perform this ablation on the ObjectsRoom, ShapeStacks, and ClevrTex datasets.
15

---

Published as a conference paper at ICLR 2024
BO-Slate
BO CO-Siste- Cyclic
Recon,
Mank
Mank Mark 5
Image
Recorn
Masi:
Music
Mask
CUB200 Birds
Recon.
Mask
Mask
Mask
Mask
Starfoord Cars
Image
Image
Recon
Stanford Dogs
Figure 7: Foreground Extraction Visualization. This figure showcases the reconstruction and segmentation masks produced by the baseline model (BO-Slate) and the proposed cyclic model. Notably, we observe that the baseline model tends to mix the foreground with parts of the back- ground, particularly for the Stanford Dogs and Stanford Cars datasets. In contrast, the use of our cyclic objectives leads to significantly clearer separation of the foreground and background, result- ing more accurate and refined representation the objects of interest.
Effect of EMA Encoder Another implementational detail of our approach that we use an EMA encoder to compute the feature-slot-featre supervision matrix F. We examine the importance of using EMA encoder by comparing the performance of the proposed approach on the shapestacks datasets with and without it. We present the results of this comparison in Table 10. We can see that the performance of the model with the EMA encoder is much better than the performance without it thus showing the importance of the EMA encoder.
Model
FG-ARI
Model
FG-ARI 0,7028
Cyclic iteration only)
=0066
SA Cyclic (No EMA Encoder)
SA + Cyclic (all iterations)
0.7245 10.01
Cyclic Encoder)
0.7838 2012
Table 9: Iteration Ablation We observe that the performance suffers significant drop when the cycle consistency objecives are only applied to the last iteration of slot attention. We run this experi- ment the ClevrTex dataset.
Table 10: EMA Encoder Ablation We can see that the performance of the model without the EMA encoder is much worse than with it thus showing the importance of the EMA encoder. We run this experiment on the shapestacks dataset.
Dinosaur Implementation Details We use DINOSAUR (Seitzer al., 2023) the base model to which we augment the cycle consistency objectives for our experiments on the MOVi-E and MOVi-C datasets. We use pretrained ViT-B/8 model pretrained using the approach presented in DINO (Caron et al., 2021). We use 10 slots for experiments on the MOVi-C dataset and 23 slots for experiments on the MOVi-E dataset. For both datasets, we slot attention iterations. this case, we apply the cycle consistency objectives on the slots obtained from the last iteration of slot attention. We set Aafa' run each experiment on RTXS000 GPU.
and A to 1. We use Adam optimizer with a learning rate of 4e-4. We
REAL WORLD DATASET EXPERIMENTS
For these experiments, the BO-St (Jia al., 2022) the base model to which augment the cycle consistency objectives. The values of all hyperparameters used for these experiments are shown in Table 11.
16

---

Published as a conference paper at ICLR 2024
batch size
warmup steps
10000
Training
learning rate
le-4
max steps
250k
Input image size
96 96
vocabulary size
1024
Gumbel-Softtax annealing range
1.0 to 0.1
dVAE
Gumbel-Softmax annealing steps
30000
Ir-dVAE (no warmup)
3e-4
layer
heads
4
Transformer Decoder
dropout
0.1
hidden dimension
256
slot dimension
256
Slot Attention Module
iterations
annealing steps
30000
Aafs'
10
Afef'
0.1
Cycle Consistency Objective
Tafs'
Tyef'
0,01
0.8
EMA Decay rate
0.900
Downsampled Feature size
24 24
Table 11:
This table shows various hyperparameters used in the real-world dataset experiments where we use BO-Slate as the base model.
Unsupervised Foreground Extraction For foreground extraction, we the Stanford dogs dataset (Khosla et al., 2012), Stanford cars dataset (Krause et al.. 2013), CUB200 Birds dataset (Wah al., 2011), and flowers dataset (Nilsback & Zisserman, 2006). We evaluate the performance using the loU (Intersection over union) and the Dice metries. IoU is calculated by dividing the over- lapping area between the ground-truth and predicted masks by the union area of the ground-truth and predicted masks. Dice is calculated twice the area of overlap between the ground-truth and predicted mask divide by the combined number of pixels between the ground-truth and predicted masks.
We use the BO-SLATE model (Jia et al., 2022) as our base model. BO-SLATE an improved version of Slate (Singh et al., 2022) where the main improvements come from having learnable slot initializations. We apply the cycle consistency objectives as auxillary objectives to BO-Slate.
The results for this task presented in Table 12. We observe that the proposed objectives helps the model achieve a superior performance compared to the baseline on all datasets. Additionally. in Figure 7, we visualize the reconstruction and segmentation masks from the model. We note that in certain cases, the baseline model tends to mix foreground and background information, whereas the same model augmented with the cyclic objectives is able to segregate the foreground and background information perfectly.
Table 12: Unsupervised Foreground Extraction. Here we present results for unsupervised fore- ground extraction the Stanford Dogs, Stanford Cars, and CUB 200 birds dataset. We augment cyclic objectives to the improved version of Slate (Singh et al., 2022) presented in (Jia et al., 2022). We can see that the performance of the Slate model improves when augmented with the proposed objectives thus showing the efficacy of our approach. Results averaged across 3 seeds.
Dogs
Cars
Birds
Flowers
Model
IoU 7
Dice
IoU
Dice
loU 1
Dice
IoU
Dice
BO-STate
0.6681 2m1
0.7686 in
0.8647.am
0.6129 200S
2008
0.746gamm
com
CYCLaC
0.84564ama
0.7462.4-os
0.8447 4-0m2
0.9145
0.6497 4.mm
0.7797 +4.000
0.7745
0.8525-00
17

---

Published as a conference paper at ICLR 2024
We follow the hyperparameters mentioned in Table 11. We use slots for all foreground extraction experiments except for Birds for which we use slots. To downsample the features obtained from the encoder for computing the cycle consistency objectives, we use 2 layered convolutional network in which each layer has kernel size 4, stride 2, and padding 1. We use a relu activation between the two layers. We run each experiment RTX8000 GPU.
Multi-Object Segmentation For multi-object segmentation, and scannet (Dai et al., 2017) datasets. We use the following metrics to evaluate performance-
the coco (Lin al., 2014)
AP@05: AP a metric used in object detection to measure the accuracy and relevance of detection results based on precision and recall values.
Panoptic Quality (PQ): PQ is a comprehensive metric for evaluating object segmentation that combines segmentation quality and instance-level recognition performance into a sin- gle score.
Precision score: Precision measures the proportion of correctly predicted foreground pixels among all the pixels predicted foreground, indicating the accuracy of the segmentation results.
Recall Score: Recall measures the proportion of correctly predicted foreground pixels among all the ground truth foreground pixels, indicating the completeness or coverage of the segmentation results.
We use the same hyperparameters as presented in Table 11. For calculating the cycle consistency objective, downsample the features output by the encoder using similar convolutional network as used in the Foreground Extraction task.
OBJECT CENTRIC MODELS IN ATARI
We follow the exact setup from decision transformer (Chen al., 2021) for this experiment. Decision Transformer models the offline RL problem as conditional sequence modelling task.
This is done by feeding into the model the states, actions, and the return-to-oo R. >e Pes where с denotes the timesteps. This results in the following trajectory representation: T = (R1, $1, 01, R2, 82, Q2, Ra, $3, az,...). where a₂ denotes the actions and & denotes the states. At test time, the start state 81 and desired return RG is fed into the model and it autoregressively gener- ates the rest of the trajectory.
The original state representations s; D-dimensional vectors obtained by passing the atari ob- servations through a convolutional encoder. Note that each observation is a stack of 4 frames. To obtain the corresponding object-centric version of this, and the slot attention module from (Locatello et al., 2020) to encode each observation. There- fore, each observation is encoded into N slots resulting a decision transformer trajectory sN T (Ri, {s} o7 .......,,. s a1, R2, {s/] ..,....s' 02, R3, {sg, s 1.......... a3....).
use the convolutional encoder
As mentioned in the main text, we augment the action preeoictinniioon loss from decision transformer with the reconstruction loss from slot attention and the proposed cycle consistency objectives.
We use 6-layered transformer and 8 attention heads with an embedding size 128. We use a batch size of 64. We use a context length of for Pong and context length 30 for Seaquest, Breakout, and Qbert. We keep all other hyperparameters same as mentioned in (Chen et al., 2021). For the slot attention implementation, we define the architecture of the encoder and decoder Table 13. The values of the other hyperparameters related to the slot attention module and the cycle consistency objectives are presented in Table 14. The input image size for the atari experiments is 84 x 84. We downsample the features to size 21 x 21 to compute the cycle consistency objectives. We use 2-layered convolutional network for this downsampling where each layer has kernel size 4, stride 2, and padding
18

---

Published as a conference paper at ICLR 2024
Channels
Layer
Kemel Size
Packing
Sirde
cutput padding
Conv.
64
Conv.
64
Convolutional Encoder
Cony.
64
Conv' Trans.
64
0
Cony Trans.
32
Convolutional Decoder
Trans.
mask)
Table 13: Architecture of the encoder and decoder used in the slot attention module for the decision transformer model.
Hyperparameter Value
slot dimension iterations
256
3
Xefa' Ages T
0.1
0.01
0.1
Tafs'
Tfsf' 0g
0.01
0.8
EMA Decay rate
0.995
Downsampled Feature size
21 21
Table 14:
Here present the values for the various hyperparameters used in the slot attention module for the decision transformer experiments.
10 OBJECT CENTRIC MODELS IN CAUSAL WORLD
We follow the same setup as (Yoon et al., 2023) for this experiment. We first petrain the object centric model on 1000000 trajectories from the object goal task of the causal world environment. The baseline object centric model in our case is Slate. We augment it with the proposed cycle consistency objectives for our model. Both the object-centric models utilize slots. We train them for 200k steps.
For training the policy, we use PPO. The agent transformer- based model which takes the slots as input along with a CLS token and outputs distribution over actions and a value. We train the agent for 1000000 interaction steps.
19


Reviews:
[{"review_id": "lDv9n1Spnk", "reviewer": "Reviewer_DLp5", "summary": "This paper proposes a cyclic training loss to improve the slot-attention-based, object-centric representation learning in neural networks. Specifically, it aims to make the mapping from features to slots and from slots to features more distinct. Additionally, it emphasizes applying the learned representations to downstream tasks.", "questions": "As there are already some works aiming to improve the slot attention-based method, the significance of this paper would be enhanced if it could show more compelling practical results, demonstrating that the overall direction is promising.\nWhy are the results limited to only 4 Atari games? Do the results generalize to more games?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It is a reasonable idea to use additional regularization terms in the training objective to make each slot in slot attention represent a more distinct concept. The paper presents this idea clearly.\nExperimental results show improved performance across various downstream tasks, including four Atari games, object discovery, and COCO/Scannet segmentation.\nFor segmentation tasks, it is interesting to see that additional cyclic losses are helpful with BO-Slate, as BO-Slate's optimization method should already aim to enhance disentangling between slots.", "weaknesses": "The experimental results show improvement, the overview accuracy level is low for real-world object discovery tasks. I doubt if adding more constraints on the disentanglement of representation is a promising direction. \n\nThe results on Atari games are a bit mixed. Also, why only evaluate it on four games?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lfrHJtqwpX", "reviewer": "Reviewer_SbRA", "summary": "This paper tackles object discovery and introduces additional constraints to existing slot-based methods. Specifically, two cycle consistency objectives, slot-feature-slot consistency, and feature-slot-feature consistency are explored. The authors applies the learned object-centric representations to downstream reinforcement learning tasks and demonstrates the effectiveness of the proposed method.", "questions": "How to determine the number of slots? \n\nAs shown in Fig. 2, in the bottom right, multiple semantics exist in slot 6. Why the model cannot depart them?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow. \n2. The motivation and the development of the two consistency losses are clearly conveyed. \n3. Experiments are extensively conducted to evaluate the proposed method.", "weaknesses": "1. The authors point out one of the limitations of existing methods that a notable gap exists for the learned object-centric representations to be used in the downstream tasks. However, it does not make sense to claim the proposed method overcomes this by achieving better performance on downstream tasks. The logic here is somewhat doubtful. \n\n2. The main contribution of this paper is the two proposed consistency losses which constrain the model to learn discriminative slots. The technical novelty is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DjsTPfFsL3", "reviewer": "Reviewer_7qi5", "summary": "- This work identifies two shortcomings with existing object discovery methods. The first one being excessive reliance on specific architectural priors and meticulous engineering efforts. The second shortcoming is the gap in investigating the real world application of representations learned using the discovery methods.\n- To mitigate the first shortcomings, authors propose an objective function based on cycle consistency that constraints features of a single or multiple instances of an object in a scene to belong to a single slot. \n- To mitigate the second limitation, authors demonstrate the effectiveness of learned representation in two downstream reinforcement learning tasks.\n- Authors demonstrate that these enhancements hold true consistently across both synthetic and real world datasets showcasing the effectiveness of the proposed approach.", "questions": "- In Eq. 8 the softmax is applied using $\\tilde F$ but only along the diagonal? Can the authors elaborate what happens if the full matrix is used for the loss? Isn't that a stricter case?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow.\n- Authors validate all the claims made in the paper through appropriate experiments\n- The proposed cycle consistency objectives are very simple and effective and I can foresee such objectives being useful for other q-former architectures like [1-2].\n[1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, Sergey Zagoruyko, End-to-End Object Detection with Transformers. \n[2] Junnan Li Dongxu Li Silvio Savarese Steven Hoi, BLIP-2:Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.", "weaknesses": "- I do not see any major drawbacks with the current work but I believe it misses a few more analysis to show the effectiveness of the cycle consistency objectiveness.\n- For example, does the Slot-feature-slot consistency objective reduce the total number of required slots creating a bottleneck? Does it have an effect on the size of the feature dimension of the slots?\n- Authors showed that increasing the value of $\\lambda_{s f s}$ results in a trivial solution but are there any other modes of failure?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SiSQ1fboI0", "reviewer": "Reviewer_nRCE", "summary": "The authors propose a novel regularization for object-centric learning methods which\ncluster features into slots, for example Slot Attention. The proposed loss terms\nregularize the cycle consistency between features and slots and vice versa:\nSlot-feature-slot similarities are regularized towards an identity matrix and\nfeature-slot-feature similarties are regularized towards feature-feature similarities.\nThis regularization is shown to improve the segmentation performance compared to the\noriginal Slot Attention model and other object-centric models. Moreover, the\nregularization is shown to lead to more useful features for reinforcement learning.", "questions": "- Slot Attention can be related to soft k-means clustering, as mentioned by Locatello\n  et al. 2020. From this perspective the proposed regularization terms can be\n  interpreted as enforcing compact, well separated clusters. In my view the paper would\n  profit from discussing the proposed loss terms from this angle.\n- The Dinosaur model, which is mentioned in the paper, shows improved scalability of\n  object-centric learning to real world data by applying Slot Attention in the feature\n  space of a large scale contrastive model (DINO). Do the proposed regularization terms\n  also improve the Dinosaur model?\n- I would find the result section more natural to read if object discovery came before\n  the experiments on downstream tasks.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed regularization is conceptually sound and leads to consistent improvements\n  across the considered datasets and tasks.\n- The proposed method is not only evaluated for unsupervised segmentation performance,\n  but the learned, object-centric representation is also tested on a downstream task\n  (reinforcement learning).", "weaknesses": "- It is claimed that the proposed losses address that existing approaches \"rely on\n  architectural priors\" for learning objects. In my view that's not true. Slot\n  Attention can be related to soft k-means clustering; the proposed losses enforce more\n  compact and better separated clusters. But what is considered as one cluster (i.e.,\n  an object) is still determined by architectural biases.\n- In my view it is not sufficiently motivated why training unsupervised object-centric\n  models on RGB images is the best approach for improving \"object-based reasoning\n  capabilities\":\n    - Segment Anything (Kirillov et al. 2023) suggests that generalizable object\n      segmentation can be learned from limited supervised data.\n    - Object-centric methods based on contrastive features such as DINO features are\n      very capable, e.g. Dinosaur (Seitzer et al. 2022) or CutLER (Wang et al. 2023).\n      It is argued that it \"limits the applicability of the method to domains that the\n      pretrained encoders are trained on\". The works mentioned earlier however show that\n      the resulting models work well on a range of datasets, including datasets that\n      were not used to train DINO (e.g., MOVi).\n    - Some works show that using additional data, such as optical flow or depth, allows\n      training strong object centric methods (e.g., Karazija et al. 2022). The paper\n      claims that \"relying on [...] optical flow and motion is not feasible since many\n      datasets and scenes do not come with this information\". In my experience however,\n      unlabelled video data is abundant for most practical settings.\n  In summary, it is not clear to me why the restriction to unsupervised, image based\n  methods trained from scratch is adequate for the goal of \"developing object-based\n  reasoning\".\n- Only FG-ARI is used as a metric for quantifying segmentation performance. It has been\n  pointed out several times in the literature that this metric is problematic since it\n  does not take into account whether object boundaries are accurate and favours\n  undersegmentation (e.g., Engelcke et al 2020, Karazija et al. 2021, Monnier et al.\n  2021). More established segmentation metrics, such as mIoU or AP, should be used that\n  do not share these problems.\n\n**Update:** The authors addressed the concerns in the rebuttal. In particular, additional experiments have shown that the improvements by the proposed method are orthogonal to other approaches, such as using contrastive features or videos. I therefore update my rating of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)