On the Transfer of Object-Centric Representation Learning
Authors: Aniket Rajiv Didolkar
Andrii Zadaianchuk
Anirudh Goyal
Michael Curtis Mozer
Yoshua Bengio
Georg Martius
Maximilian Seitzer
Venue: ICLR 2025 Poster (2025)
Keywords: representation learning
object-centric learning
object-centric representation learning
unsupervised learning
transfer
zero-shot
generalization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We study the transfer of object-centric representations and show that a finetuning strategy leads to state-of-the-art performance.

Abstract: The goal of object-centric representation learning is to decompose visual scenes into a structured representation that isolates the entities into individual vectors. Recent successes have shown that object-centric representation learning can be scaled to real-world scenes by utilizing features from pre-trained foundation models like DINO. However, so far, these object-centric methods have mostly been applied in-distribution, with models trained and evaluated on the same dataset. This is in contrast to the underlying foundation models, which have been shown to be applicable to a wide range of data and tasks. Thus, in this work, we answer the question of whether current real-world capable object-centric methods exhibit similar levels of transferability by introducing a benchmark comprising seven different synthetic and real-world datasets. We analyze the factors influencing performance under transfer and find that training on diverse real-world images improves generalization to unseen scenarios. Furthermore, inspired by the success of task-specific fine-tuning in foundation models, we introduce a novel fine-tuning strategy to adapt pre-trained vision encoders for the task of object discovery. We find that the proposed approach results in state-of-the-art performance for unsupervised object discovery, exhibiting strong zero-shot transfer to unseen datasets.

Published as a conference paper at ICLR 2025
ON THE TRANSFER OF OBJECT-CENTRIC REPRESENTA- TION LEARNING
Aniket Didolkar'.
Andrii Zadaianchuk?
Anirudh Goyal'
Mike C. Mozer
Yoshua Bengio'
Georg Martius
Maximilian Seitzer
'MILA& & University of Montreal
University of Amsterdam
3 Google Deepmind
MMPI for Intelligent Systems University of Tübingen
ABSTRACT
The goal of object-centric representation learning is to decompose visual scenes into a structured representation that isolates the entities into individual vectors. Recent successes have shown that object-centric representation learning can be scaled to real-world scenes by utilizing features from pre-trained foundation models like DINO. However, so far, these object-centrie methods have mostly been applied in-distribution, with models trained and evaluated on the same dataset. This is in contrast to the underlying foundation models, which have been shown to be appli- cable to wide range of data and tasks. Thus, in this work, we answer the question of whether current real-world capable object-centric methods exhibit similar levels of transferability by introducing benchmark comprising seven different synthetic and real-world datasets. We analyze the factors influencing performance under transfer and find that training complex natural images improves generalization unseen scenarios. Furthermore, inspired by the success of task-specific fine-tuning in foundation models, we introduce a novel fine-tuning strategy adapt pre- pre-trained vision encoders for the task of object discovery. We find that the proposed approach performance for unsupervised object discovery, exhibiting
results in strong zero-shot transfer to unseen datasets.
INTRODUCTION
In the past decade, deep learning-basse approaches have become ever more general, culminating in models that exhibit broad and flexible vision (Dehghani et al., 2023; Oquab et al., 2023) and language understanding (Brown et al., 2020; OpenAl Team, 2024). These so-called foundation models can be applied to variety of data and tasks in a zero-shot manner. An open challenge how to equip these models with the ability to robustly reason about visual inputs, that is, in a manner that supports compositional generalization and causal reasoning (Schölkopf et al., 2021: Goyal & Bengio, 2022). Evidence suggests that human cognition deals with these problems by dynamically binding raw perceptual features into symbol-like entities that can be flexibly composed together and reasoned over (Pinker, 1984; Spelke, 1990; 2000). Inspired by these findings, the field of object-centric representation learning aims to replicate these abilities deep learning models (Greff et al., 2020). By mirroring the compositional generative process of the world (Brady et al., 2023), these methods learn to decompose visual scenes into structured representations capturing the objects in the scene in fully unsupervised way. Not only can object-centric representations provably exhibit compositional generalization (Wiedemer et al., 2024), they also support a diverse set of downstream tasks such as world modeling (Ke et al., 2021; Wu et al., 2023a), robotic control (Zadaianchuk al., 2020; Driess et al., 2023; Didolkar et al., 2024; Haramati et al., 2024), visual question answering (Xu et al., 2024; Mamaghan et al., 2024), and compositional generation in 2D (Singh al., 2022a; Wu et al., 2023b; Jiang et al., 2023) and 3D (Sajjadi et al., 2022; Jabri et al., 2023).
"qqual contribution.
Correspondence:
maximilien Lian. seitzer3tue mpg. de,
adidolkarl220 k com.
Website: rw-oor1, .aa.r..gittuutt -ithhbb. / todiuruurpppeerr

---

Published as a conference paper at ICLR 2025
While long confined simplistic synthetic datasets (Eslami al., 2016; Greff et al., 2019; Engeleke et al., 2020; Locatello et al., 2020), recent progress has scaled object-centric representations to complex real-world image (Seitzer et al., 2023; Wu et al., 2023b; Jiang et al., 2023; Kakogeorgiou et al., 2024; Löwe et al., 2024) and video datasets (Zadaianchuk et al., 2023b; Aydemir et al., 2023). This success is enabled by the use of pre-trained features from self-supervised vision encoders such as DINO (Caron et al., 2021). While it has been shown that the representations produced by such self-supervised models are fairly robust to changes the training distribution, is unclear whether the same holds for an object- cefert-cetticc model trained on top of such representations. Thus, in this work, we study the question of how well object-centric representations using pre- trained features transfer new data. This comprises (1) the ability of models to discover objects in unseen scenarios and (2) the robust- ness of the object representation itself. particular, we focus on the "zero-shot" setting, where models are presented with object categories never seen during training. This setting is relevant as model with strong zero-shot abilities could serve as a "foundation model" for object-centric representations.
To this end, we introduce benchmark consisting of 7 datasets comprising diverse range of synthetic and weal-world scenes. Using this benchmark, we (1) seek to understand the zero-shot transfer capabilities of existing models, and (2) study the properties of training datasets that influence generalization. The general conclusion we draw from this benchmark is that object-centric models which are trained on naturalistic datasets consisting a variety of objects such Coco (Lin et al., 2014) - ussully exhibit decent zero-shot generalization.
Equipped with this knowledge, we aim build strong general-purpose object-centric model. To achieve this, we first make the observation that current approaches for real-world object-centric learning (Seitzer et al., 2023; Wu et al., 2023b; Jiang et al., 2023; Zadaianchuk al., 2023b; Aydemir et al., 2023) use fixed pre-trained encoders (e.g. with the DINO method (Caron et al., 2021)) to encode the imput. This may be limiting as, while the pre-trained encoders offer good general- purpose features, they may not be optimal for the task of object discovery. Instead, we propose to finetune the encoder parameters for the target task; to this end, we introduce a suitable training recipe as well as novel decoder that reduces the increased computational costs from finetuning. Building on the DINOSAUR model (Seitzer et al., 2023), our proposed finetuning approach sets a new statc-of-the-an for real-world object-centric learning on the Coco dataset, well as in the zero-shot setting. Our method shows zero-shot transfer across a multitude of diverse datasets, often achieving and even surpassing the in-distribution performance these datasets.
Our contributions are as follows:
We introduce benchmark to evaluate the transfer of real-world capable object-centric learning methods (Sec. 3.1).
Using the benchmark, (Sec. 3.2) and investigate dataset properties for training generalizable models (Sec. 3.3).
analyze the zero-shot capabilities of object-centric models
We propose finetuning approach applied to DINOSAUR, which allows the stable adaptation of the parameters of the pre-trained encoder for the task of object discovery (Sec. 4).
The resulting method, FT-DINOSAUR, achieves state-of sate-of-ther of-e-----h-httt art results across various in- distribution and -iitt---ditiiittinnn scenarios (Sec. 5).
2 RELATED WORK
Object-Centric Learning on Real-World Datasets Originally, object-centric methods were mostly applied to synthetic data with limited complexity (Johnson et al., 2017; Karazija et al., 2021; Greff al.. 2022) and trained from scratch (Eslami et al., 2016; Burgess Locatello et al., 2020; Traub et al.. 2023). Recently, there has been considrable interest (Elsayed et al., 2022; Singh et al., 2022b; Seitzer et al., 2023; Zadaianchuk et al., 2023b; Wu al., 2023b; Didolkar et al., 2024; Löwe et al., 2024; Kakogeorgiou et al., 2024) in scaling those methods to complex and unconstrained real-world image and video datasets like Coco (Lin et al., 2014) or YouTube VIS (Yang et al., 2019). Current state-of-the-ar techniques (Seitzer et al., 2023; Wu et al., 2023b; Jiang et al., 2023; Kakogeorgiou et al., 2024: Aydemir et al., 2023; Zadaianchuk et al., 2023b) rely applying slot attention (Locatello al., 2020) to frozen vision transformers (ViT) (Dosovitskiy et al., 2021) pre-trained with contemporary self-supervised representation learning methods (Caron et al., 2021; Oquab al., 2023). Approaches differ by their learning objective; one line of models
al.. 2019; Lin et al., 2020;
2

---

Published as a conference paper at ICLR 2025
Avarage
Average
MOVIC
EntlySeg
MOVI-C
EntlySeg
0.8
0.8
Model
DINOSAUR
SlatDiffusion
MOVI-E
Tlev
MOVIE
ClevrTex
SPOT
SAM (comp.)
SAM (best)
ScanNet
YCB
ScanNet
YCB
(a) Object discovery (FG-ARI).
(b) Linear position probing (R²2).
Figure 1: Evaluating zero-shot transfer of current object-centric models trained on the CoCo dataset. (a): object discovery performance in terms of FG-ARI. (b): linear probing of object position from slots in terms of R score. We report results for mBO and category probing in Fig. A.I.
based on DINOSAUR (Seitzer et al., 2023) and utilizes a feature reconstruction objective (Seitzer et al., 2023; Zadaianchuk et al., 2023b; Kakogeorgiou et al., 2024; Aydemir et al., 2023), whereas others apply diffusion objectives (Jiang et al., 2023; Wu et al., 2023b). Although these techniques confirm that object-centric representation learning is possible for complex real-world inputs, they are also limited by the quality of sulf--uuer encoders, as those encoders remain frozen during object-centric training. In contrast, our method, while starting from self-supervised features, adapts them through object-centric finetuning, making them more suitable for the task of object-centric scene decomposition.
Zero-shot Transfer A paradigm first introduced by Larochelle al. (2008), zero-shot transfer is the application of models to tasks or datasets not seen during training. Recently, zero-shot transfer has become more prevalent deep learning due to the availability of large foundation models (Oquab et al., 2023; Kirillov et al., 2023; OpenAl Team, 2024); these models utilize large-scale pre-training to develop robust purppse abilitics that can be applied zero-shot various tasks and datasets. In the context of object-centric learning, model transfer has been studied by testing models under various types of distribution shifts, changing factors like object color, texture, or the number of objects (Karazija et al., 2021; Dittadi et al., 2022; Wiedemer et al., 2024). Generally, these works only change a single factor of variation between training and test data. However, in real-world settings, distribution changes
much more uncontrolled than this: for instance, transferring to novel category of object might include (1) multiple simultaneous factor changes, and (2) completely new factors of variation. Thus, in this work, we study this more challenging setting by evaluating the zero-shot transfer to arbitrary object categories.
Task-Specific Finetuning While pre-training models on large and diverse datasets often offers good baseline performance for zero-shot transfer (Radford et al., 2019; Kirillov et al., 2023; Oquab et al., 2023), further improvement can be achieved by finetuning the model for the target task. For example, self-supervised vision representations (Caron et al., 2021) can be adapted by finetuning for the tasks of unsupervised semantic segmentation (Hamilton et al., 2022; Zadaianchuk et al., 2023a) or multi-object tracking in videos (Salchi et al., 2023; Tumanyan et al., 2024). In this work, we similarly propose adapt pre-trained vision representations by finetuning them for the task of object discovery.
3 WHAT MATTERS FOR TRANSFER OF
REPRESENTATIONS?
In this section, our goal is to (1) measure the degree to which real-world capable object-centric models support zero-shot transfer, and (2) understand the factors influencing the models' transfer ability. For (1), we introduce a benchmark for measuring zero-shot performance in Sec. 3.1 and compare models in Sec. 3.2. For (2), we study the impact of training data on transfer in Sec. 3.3.
3

---

Published as a conference paper at ICLR 2025
3.1 BENCHMARK FOR ZERO-SHOT TRANSFER
Our goal is to test the zero-shot transfer of real-world capable object-centric models utilizing pre- trained features. As this class of model is typically trained on the Coco dataset (Lin et al., 2014), we also use CocO as the training dataset for our benchmark. For our benchmark, we interpret zero-shot transfer to mean that model can successfully discover objects of novel categories not occurring in the training data. Objects of different categories (e.g. humans; cars) have at least one factor of variation that is not shared between them (e.g. eye color; number of wheels); thus, we use "category" as a criterion to make sure there is sufficient difference between train and test sets."
Evaluation Datasets To obtain a test bed that robustly measures zero-shot performance, we gather the evaluation splits of several datasets previously proposed by the object-centric community, with diverse properties and increasing complexity: CLEVRTEX (Karazija et al., 2021), SCANNET and YCB as used in Yang & Yang (2022), and Movi-C and Movi-E (Greff al., 2022). Additionally, we add the challenging ENTITYSEG dataset (Lu et al., 2023), consisting of in-the-wild real-world images with high-quality mask annotations. While ENTITYSEG includes images with categories also occurring in the CoCO dataset, it is an open world dataset without pre-defined classes; thus, we consider it adequate for the purpose of this benchmark. In total, we gathered datasets with a total of 18 874 images. For analysis. we also use the PASCAL Voc (Everingham et al., 2012) dataset, but do not include it in the zero-shot benchmark as its set of categories is fully included in the COCO dataset used for training. For further details on the datasets, we refer to App. E.
Metrics Similar to prior work (e.g. Dittadi et al., 2022), we evaluate the object representation in terms of object discovery (do the masks associated with each object align with the true object masks?) and downstream property prediction (are the representations informative about the objects?). To aggregate results over different datasets, we compute the per-sample average, normalizing by the dataset size (see Table E.9). We now provide a brief overview of the metrics; see App. F for details.
For object discovery, we compute the commonly used foreground ARI (FG-ARI) (Rand, 1971; Hubert & Arabie, 1985), measuring how well the discovered objects follow the separation prescribed by the reference mask annotations. In addition, we compute the mean best overlap (mBO) (Pont-Tuset et al., 2017), measuring how well the discovered masks fit to objects. To measure (in-distribution) performance on the Coco dataset, we additionally evaluate panoptic scene decomposition into "things" (objects) and "stuff" (background). This is sensible because on real-world images, there is no clear distinction between objects and background from the model's point-of-view. To this end, we compute panoptic ARI (P-ARI) and class-agnostic panoptic quality (PQ), where the latter measures both mask quality and precision/recall (Kirillov et al., 2019). For downstream property prediction, we closely stick to the protocol of Dittadi et al. (2022). In particular, we train linear predictors (probes) on top of the object representation to predict position and category (where available) of the objects. For position, use the R score metric; for classification, we use accuracy.
3.2 EVALUATING MODELS
Models We evaluate three recent methods capable of real-world object-centric learning, all using pre-trained DINO features as Dppuss:DNosAAU (Seitzer et al., 2023), which also uses DINO features as targets; SPOT (Kakogeorgiou et al., 2024), which builds upon DINOSAUR by improving the decoder; and SlotDiffusion (Wu et al., 2023b), which uses a diffusion decoder. We limit ourselves to pre-trained feature-based methods as other approaches have not shown scalability to real-world data. To showcase the gap between of-te--f-th--an supervised and unsupervised methods for object discovery, we evaluate the Segment Anything model (SAM) (Kirillov et al., 2023), supervised segmentation foundation model known for its zero-shot capabilities. We use two configurations: for SAM (best), use the largest available model (ViT-Huge), and pick the mask confidence threshold resulting in the best performance per-dataset; in contrast to current object-centric methods, this results in variable number of masks per-image. For better comparability, we also evaluate a baseline SAM (comp.), using a ViT-Base encoder and fixed number of masks. Please refer to App. D for details about the models.
Note that for our purposes, we are not going to take into account the data the utilized self supervupe pre-trained models (e.g. DINO) were trained on (e.g. ImageNet). This is hecause our primary interest is in the transfer of the object-centric components, which are trained on distinct set of images (c.g. Coco).
4

---

Published as a conference paper at ICLR 2025
Training Dataset
ClevrTex
MOVi-C
is
FGARI normalized by ID
0.8
Evaluation Datasel
(ID)
MOVi-E
FG-ARI
Moan OOD
ScanNet
0.6
EntiyySeg
1.0
ScanNet
YCB
YGB
PASCAL
0.4
MOVi-C
EntitySeg
0..5
MOVI-E
COCO
CleviTex
Glew
MOViC
MOV-E
ScanNel
YBB PASCAL Emiry
0.2
2
2
2
2
2
Size of Training Dataset
Evaluation Dataset
(a) Varying train datasets.
(b) Varying train dataset size.
Figure Effect of training data on transfer of the DINOSAUR model. Performance in FG-ARI, see Fig. A.2 for corresponding plots with mBO. (a): comparing transfer to in-distribution performance for different training datasets. (b): scaling behavior for training on differently sized subsets Coco.
Results We present the results in Fig. and Fig. A.I, with example predictions in App. G. For object discovery, DINOSAUR and SlotDiffusion are on par in terms FG-ARI, while SlotDiffusion achieves the best mBO. Unsurprisingly, SAM (best) outperforms the unsupervised methods; this is especially pronounced in terms of mBO - the difference can be explained by SAM's ability to output variable number of masks, which allows SAM discover smaller objects not captured by methods with fixed number of slots. If we remove that privilege, the SAM (comp.) baseline is inferior to the many datasets, with the exception of CLEVRTEX. For downstream property
unsupervised models prediction, we find DINOSAUR to perform best on average, allowing for a decent property readout on most datasets. We also evaluate property prediction for DINOSAUR models trained in-distribution on each dataset. Comparing the average performance of the zero-shot model to the in-distribution models, we see similar results for predicting categories (+1 rel. change), and only slight drops for predicting positions (-11% % rel. change). We conclude that object-centric models already exhibit decent zero-shot generalization to unseen datasets.
3.3 EVALUATING TRAINING DATA
So far, we used the same training data for comparing the models - natural question is how the training data affects transfer behavior. To answer it, we train DINOSAUR on different training datasets and evaluate the zero-shot performance in terms of object discovery. We organize our experiments into two groups: (1) varying the data distribution, and (2) varying the amount of samples from a particular data distribution. The former allows to identify properties of the data that influence zero-shot behavior, whereas the latter investigates how models scale with data.
Properties of the Data Distribution To obtain training datasets with different properties, utilize the training splits belonging to the benchmark datasets listed in Sec. 3.1. We characterize the training datasets along three dimensions: realism (CLEVRTEX), hybrid (Movi, SCANNET, YCB) and natural (PASCAL Voc, Coco, ENTITYSEG), spectrum from narrow broad (roughly CLEVRTEX SCANNET, YCB, PASCAL Voc < Movi < Coco & ENTITYSEG), and the amount of objects ranging from few (PASCAL Voc) to moderate (up 6; CLEVRTEX, SCANNET, YCB) many (Movi, Coco, ENTITYSEG)) The results are shown in Fig. 2a.
terms of the three categories synthetic
diversity
First, we find that training and evaluating di-distribution i.e. on matching datasets, unsurprisingly performs best in general. Training synthetic and hybrid datasets transfers well to datasets in those categories, but not to natural data; conversely, natural data transfers well to synthetic and hybrid data. Next, we find that the zero-shot performance is fairly similar when trained Coco and ENTITYSEG (high diversity, many objects), or PASCAL Voc (less diversity, few objects). This shows that having complex natural data is more important for zero-shot performance compared to data diversity. Moreover, even when trained on natural data with few objects (e.g. PASCAL VoC), the model transfers well to datasets with more objects such Movi, Coco, and ENTITYSEG.
5

---

Published as a conference paper at ICLR 2025
T Object-Centrie Finetuning
2 High-Res. Adaptation
Transfer
00co (D)
EritySeg
DINOV2 finM
p Tin
hiies
Iniiialize
Faecal VDC
CleeTT
FT-DINOSAUR
MOVIC
MOVIE
DINOV2
OO-K Decoder
Parzh
Sict
N' Peac
X Plota
BearNis
YG8
DINOV2
LOGs
starting from DINOv2, the encoder is finetuned for the task of object discovery on the Coco dataset. @ High-Res Adaptation: the model is further adapted to high-resolution images. Transfer: at test time, we transfer the trained model to different datasets from our proposed benchmark (Sec. 3.1).
Figure 3: Overview of our method "FT-DINOSAUR", Object-Centric Finetuning:
Overall, we can conclude that training on natural data leads to strong zero-shot performance of current object-centric models.
Effect of Data Scale We now investigate the effect of the number of training data points. To do so, we train DINOSAUR on differently sized subsets of the Coco dataset (up to 240k samples when including the "unlabeled" split). From the results in Fig. 2b, we find that in-distribution performance plateaus around 8 192 (223) samples, and zero-shot performance around 16 182 (2 samples. Intriguingly, this shows that current object-centric models can be very sample efficient in obtaining decent in in-distribution and competitive zero-shot generalization. However, we do not find evidence of favorable data scaling laws.
3.4
SUMMARY
Our initial question was whether object-centric models based pre-trained models inherit some of their generality. Our results show that this is indeed the case, the models we tested (DINOSAUR. SPOT, SlotDiffusion) exhibit decent zero-shot transfer to unseen object categories. Furthermore, our experiments show that training on complex natural data is an important component for zero-shot transfer, which can be attributed to the inherent complexities associated with such data. In addition, real-world datasets offer significantly larger catalog objects and instances to train on compared to synthetic or hybrid datasets.
Equipped with this knowledge, we shift our focus to enhancing the performance of unsupervised object-centric models. Specifically, the question ask is: Can improve object-centric represen- tations by finetuning pre-trained encoders for the task of object discovery?
4 EHECT-CENTRIC FINETUNING
Current methods for real-world object-centric learning (Seitzer et al., 2023; Jiang et al., 2023; Wu et al., 2023b; Kakogeorgiou et al., 2024; Zadaianchuk et al., 2023b; Aydemir et al., 2023; Löwe et al., 2024) are all based pre-trained self-supervised features (Caron et al., 2021; Oquab et al., 2023). While those features offer good performance for many downstream tasks out of the box, they are not explicitly designed for the task of object discovery. We conjecture that this gap between training and downstream objective leads to sub-optimal transfer performance. Thus, we propose to adapt the pre-trained features by task-specific finetuning Fig. shows our approach.

---

Published as a conference paper at ICLR 2025
Image
PCA (1-3)
Masks
Image
OC Ennetunn g D DINOSAUR RR
PCA (1-3)
Masks
OC Finetuning DINOSAUR
Figure 4: Visualization of encoder features in DINOSAUR (frozen DINOv2 features) and for fea- tures adapted with object-centric finetuning. We show the 1st to 3rd PCA components visualized by different RGB channels (second column). The last column shows scene decomposition masks by each method. More examples and additional PCA components are shown in Fig. A.4.
4.1 FINETUNED DINOSAUR
Finetuning We first deseribe how we adapt the DINOSAUR architecture (Seitzer et al., 2023) for finetuning. DINOSAUR uses pre-trained ViT as the encoder that is kept fixed during training. The original work reported that unfreezing the encoder leads to a collapse; this is because the encoder features are simultaneously used as the model's prediction targets. To sidestep this problem, we add target encoder that is initialized to be copy of the original encoder, but kept fixed throughout training. This allows us train the full model end-to-end without collapse.
We found that the encoder would initially drift away from its pre-trained initialization, likely induced by the noisy gradients from the randomly initialized slot attention module. To reduce the effect of this, we introduce blockwise exponentially decaying learning rates (Howard & Ruder, 2018) for the encoder. Furthermore, we found a better set of hyperparameters, namely a lower learning rate, switching to a cosine learning rate schedule (Loshchilov & Hutter, 2017), lower gradient clipping, weight decay on the encoder and a higher batch size. We detail the exact settings in App. C.4. The efficacy of this improved setup is shown by the fact that can now also train the model starting from a randomly initialized ViT encoder (42.3 FG-ARI, 27.3 mBO), a scenario which was previously reported leading collapse by Seitzer al. (2023). We further expand this setup in Section A. We also experimented with an EMA student-teacher setup to continuously adapt the targets throughout training, but found that this leads to worse results (see App. A.2).
High Resolution Adaptation A further way to make more effective usage of data is to increase the image resolution. Standard ViTs relatively low resolution of 224 224 pixels, leading to patch resolution of 16 x 16 when trained with patch size 14. This hides details, inhibits capturing smaller objects, and leads coarser objects masks. Thus, after training at 224 224 resolution, we add short second stage of training, in which the model is adapted to image resolution to 518 x 518 (i.e. 37 37 patches) over 10000 steps. This is similar DINOv2's training strategy (Oquab al., 2023), and adds significant improvements (Sec. 4.3) without high computational burden.
Efficient Top-K Decoding Finetuning the encoder and high resolution adaptation both significantly increase the costs in terms of computation and memory. To mitigate this, we introduce a novel efficient decoding approach based on the MI.P decoder introduced by Seitzer et al. (2023), which we call top-k decoding. For each of N patches, the MLP decoder produces an output by combining the predictions over slots using slot-wise weighted average, resulting in a computational cost of O(N C). Our insight is that most of this computation is wasted, as slots are localized and mostly sparsely distributed across the image - instead, suffices to decode the & most likely slots occupying patch, reducing the costs O(N) for constant k. While we do not have access to the true occupation probabilities apriori, empirically we found that the masks from slot attention can serve good proxy. We refer to App. C.3 for more details.

---

Published as a conference paper at ICLR 2025
Table 1: Ablation study on Coco. Starting from DINOSAUR (Seitzer et al., 2023) (first row), we ablate the impact of switching to DINOV2, finetuning the encoder (FT), improving general (G-HP) and encoder hyperparameters (E-HP), adding top-k decoding and high-resolution adaptation. Results averaged over 3 random seeds besides last two rows, which use 5 seeds.
Model
FT G-HP E-HP FG-ARI
mBO P-ARI
PQ
DINO
DINOv2
DINO VIT-B/16
40.3
27.2
37.1
14.4
DINOV2 VIT-S/14
42.5
28.8
39.5
16.3
X
42.9
29.1
39.8
16.8
/
X
46.5
29.8
42.2
17.9
/
V
48.0
30.6
42.8
18.8
48.5
30.7
42.6
19.0
+Top-k
46.4
32.0
43.5
19.5
+Top-k, +Hi-Res
V
V
46.6
35.6
49.6
23.6
4.2 ANALYSIS
Object-centric finetuning adapts the pre-trained encoder such that the original DINOv2 features can be predicted better, with the slot representations acting as a bottleneck. To better understand the effect of this procedure, we study how the encoder representations change after finetuning. In Fig. 4 and Fig, A.4, we show the first PCA components obtained from DINOv2 features (used by DINOSAUR) and from features after object-centric finetuning. DINOV2 features mainly exhibit semantic similarity, i.e. one component often corresponds to several different objects or parts of the same category (such as human heads). In contrast, after object-centric finetuning, PCA components are noticeably object- centric, splitting instances of the same category and grouping together different object parts into one component. To confirm this observation quantitatively, we apply per-image k-means clustering to the two types of features. On Coco, find that the clustering of features from object-centric finetuning corresponds better to object instances, reaching 34.0 FG-ARI and 28.7 mBO in contrast to 27.4 FG-ARI and 24.7 mBO for the original DINOv2 features.
4.3 ABLATIONS
In Table 1, we analyze the contribution of different components of our model on the Coco dataset, starting from the original DINOSAUR model and ending with our final model. First, we find that switching from DINO DINOv2 leads to moderate improvements (+2.2 FG-ART and +1.6 mBO). Adding finetuning results in strong improvement of FG-ARI (+4.0), demonstrating the importance of task-specific adaptation. evaluate our hyperparameter changes, we split them into two groups: general hyperparameters (cosine schedule, lower learning rate, lower gradient clipping), and encoder hyperparameters (blockwise learning rates, lower encoder learning rate, encoder weight decay). The changes to the general hyperparameters result in moderate improvements (+1.5 FG-ARI, +0.8 mBO, +0.9 PQ), with the changed encoder hyperparameters contributing further small improvements. Introducing top-k decoding reduces FG-ARI (-2.1), but increases the other metrics (e.g. +1.3 mBO). Finally, high-resolution adaptation results in further strong boosts (+3.6 mBO, +6.1 P-ARI, +4.1 PQ).
EVALUATION
To evaluate our approach, we use our benchmark answer the following three questions:
How does our proposed finetuning methodology work on diverse datasets (Sec. 5.1)?
How does method compare prior methods for real-world object-centric learning (Sec. 5.2)?
How does our method perform on the introduced zero-shot benchmark (Sec. 5.3)?
5.1 EVALUATION OF OBJECT-CENTER FINETUNING
We first validate our proposed finetuning approach as general methodology by training on diverse datasets. In particular, train DINOSAUR model using a DINOv2 backbone with and without
8

---

Published as a conference paper at ICLR 2025
FG-ARI norm.
1.2
Model
FG-ARI
mBO
P-ARI
PQ
mBO norm.
DINOSAUR
40.5
27.7
37.1
14.4
1.0
SlotDiffusion!
37.3
31.4
47.6
21.0
SPOT
37.0
34.8
52.4
21.3
0.8
FI-DINOSAUR
48.8
36.3
49.4
23,9
MOVI-C
YCB
EntitySeg
SAM (comp.)t
12.1
19.0
10.8
9.4
MOVI-E
Tex
COCO DINOSAUR
SAM (best.)*
44.9
56.9
54.4
10.9
ScanNet
PASCAL
Figure Normalized performance when adding finetuning to DINOSAUR for in- distribution training, using a ViT-S/14 DINOv2 encoder. Finetuning shows strong gains on all datasets. Numerical results in Table A.4.
Table 2: Comparison to prior work on Coco. We use ViT-B/14 encoder with top-k de- coding and hi-res. adaptation. Results for (FT-)DINOS AUR averaged over seeds. Results marked f evaluate official checkpoints, super- vised models in gray. We compare to more base- lines in Table A.5.
finetuning the training splits of the datasets used for analysis in Sec. 3.3, and evaluate in-distribution. The results are listed in Fig. 5. We find that adding finetuning results in strong improvements on both FG-ARI (up to points) and mBO (up points) across all 8 datasets. First, this demonstrates that finetuning, when using our training recipe, is general strategy to improve performance slot attention-based object-centric models with pre-trained backbones. This is in contrast to Seitzer et al. (2023)'s findings, who reported collapsing slots when finetuning the pre- trained ViT encoder. Second, this shows that while pre-trained features obtained from self-supervised methods like DINOV2 are powerful, it is possible Interestingly, even though the model's s objective is to predict DINOv2 features, the optimal input to slot attention not those exact features. Following analysis in Sec. 4.2, we conjecture that finetuning adapts the features to simplify grouping under the inductive biases of the model.
improve upon them with task-specific finetuning.
5.2 COMPARISON PRIOR WORK WORLD
LEARNING
Second, in Table 2, compare our full approach with prior work the Coco dataset. We find that our method sets a new state-of-the-an on Coco, achieving better results than all previous unsupervised object-centric methods, except being slightly worse than SPOT the panoptic ARI metric. Moreover, our method also outperforms the SAM (comp.) baseline (ViT-Base encoder, same number of masks) on all metries. In particular, our method has strongly improved FG-ARI (+9), indicating much better object discovery capabilities- - it even achieves higher FG-ART than the SAM (best.) baseline (ViT-Huge encoder, variable number masks). However, there still a large gap to SAM's a mBO, which we attribute to 1) S SAM's generally higher mask quality, and 2) its ability to capture a variable number of objects, which in particular leads to finding more small objects. We refer to Fig. G.6 for example predictions.
5.3 ZERO-SHIOT EVALUATION
Finally, evaluate our method in terms its zero-shot performance. First, in Fig, compare the zero-shot performance of our model finetuned on Coco (without top-k decoding or high-res. adap- tation) the performance our model finetuned in nn-distribution. We find that transferring from most datasets (SCANNET, ENTITY-
Coco yields comparable results to training in-distribution SEG), and even surpasses in-distribution training on some datasets (Movi-C, YCB) - surprisingly, object-centric finetuning does not hurt generalization (c.g. by overfitting), indicating that it adapts the model to the task rather than the data. Overall, this shows that tasc-fpec finetuning on diverse real-world data is viable path to obtain zero-shot object-centric models.
Second, in Fig. 7, we compare the zero-shot performance of our full model (including top-k decoding and high-resolution adaptation) to prior work. Averaged over all datasets, our approach achieves both the highest FG-ARI and mBO, while previous work generally trades off high FG-ARI with low mBO (DINOSAUR) or high mBO with low FG-ART (SlotDiffusion, SPOT). On top of finetuning,

---

important stepping stones Published as a conference paper at ICLR 2025
OOCO
COCO
MON.C
EralySeag
NOW
FG-ARI
mBO
0.6-
FG-AN
6,0
Clovroa
MBO
0,0
CHEV TOX
0.4
0.4-
VGB
Seart int
YCB
0.2
0.2
FT-DINOSAUR
SlolDilfusion
SAM (comp.)
In-distribution
Zero-shot from COCO
DINOSAUR
SPOT
SAM (best)
Figure 6: Comparing in-distribution training vs. zero-shot transfer from Coco for our fine- tuning approach. Overall, performance is simi- lar. Numerical results in Table A.4.
Figure 7: Zero-shot performance averaged over datasets. FI-DINOSAUR performs best both in FG-ARI and mBO. Results per datasets available in Table A.6.
ascribe this to our usage of the MLP decoder (higher FG-ARI) in combination with high-resolution training (higher mBO).
Last, compare our model to SAM. SAM (comp.) generally performs worse than our model, show- ing the difficulty of unsupervised scene decomposition in the absence of task-specific information. For SAM (best.), there is still large difference to our approach in terms of mBO (30.7 points), while the difference in terms of FG-ARI is much smaller (8.3 points). Taken together, these results show that unsupervised object-centric models are closing the gap supervised methods in terms of zero-shol object discovery. This is astonishing, given that SAM was trained on 10 million images with over billion mask annotations. Furthermore, principal advantage of object-centric models over SAM is that they come equipped with explicit object representations. While mask quality as measured by mBO is lacking behind SAM, we are hopeful that this gap is addressable by training on even higher resolution images and introducing innovations for variable number of slots. Finally, in the appendix, we present a comparison of the masks obtained from the proposed approach and all baselines (App. G), and comprehensively study failure cases of our model (App. B).
6 CONCLUSION
In this work, we have introduced benchmark of diverse real-world and synthetic datasets to study the zero-shot capabilities of object-centric representation learning models. Our findings indicate that object-centric models using mre-trained encoders already exhibit notable zero-shot capabilities when trained on real-world data. We then presented a finetuning procedure for adapting pre-trained encoders to the task of object discovery, demonstrating that this approach achieves state-of-the-att results across 8 datasets in both in-distribution and out-of-distribuutio scenarios. We believe that our contributed tools the zero-shot benchmark and stable finetuning towards an object-centric foundation model.
Our benchmark showed the importance of the type of training data for zero-shot transfer. Our experiments indicate that training on complex natural data is important, suggesting an exciting direction to design curated datasets for zero-shot object-centric learning. Moreover, our benchmark revealed that current object-centric models are highly sample-efficient but fail to leverage larger datasets to improve performance at current model sizes. This result is significant because it suggests that, unlike other deep learning domains, stronger object-centric models cannot be achieved simply by scaling up data alone. We hope our findings will encourage the community to develop object-centric models that scale effectively with both data and model size.
Finally, to build general-purposs object-centric models, it is crucial that the learned object-centric representation are useful for real-world downstream tasks. In this work, we evaluated downstream property prediction, which can already serve as a proxy for more complex tasks. While the down- stream applicability of object-centric representations has been explored with various tasks such as reinforcement learning (Yoon et al., 2023), dynamics prediction (Wu et al., 2023a), compositional image generation (Wu et al., 2023b), visual question answering (Xu et al., 2024; Mamaghan et al., 2024) - the zero-shot scenario has not been comprehensively studied so far. Thus, an exciting direction for future work extend our benchmark to include suite of zero-shot downstream tasks.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This work was supported by the ERC- 101045454 REAL-RL and funded by EXC number 2064/1 Project number 390727645. We acknowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 011S 8039B). Andrii Zadaianchuk is funded by the European Union (ERC, EVA, 950086). Views and opinions expressed are, however, those of the author only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Maximilian Seitzer. Aniket Didolkar was partly funded by the UNIQUE Excellence Scholarship. Aniket Didolkar would like to thank Mila for providing part of the compute resources used in this work.
CONTRIBUTIONS
This project was initiated by AD and MS, and MS had the role of project lead. AD and MS contributed equally. AZ joined the project from the start and had critical input all stages. AD, AZ. and MS shaped the project direction, with advise from AG, MM, YB, and GM. MS implemented most of the code, with contributions from AD. AD and MS performed the exploratory experiments. AD performed most of the final experiments and evaluations, with some experiments ran by MS. AZ. performed the analysis of encoder features and created the corresponding figures (Fig. 4, Fig. A.4), AD created the model figure (Fig. 3), and MS created the remaining figures. The first draft was written by AD, AZ and MS, with AG and GM contributing the final version.
REFERENCES
Görkay Aydemir, Weidi Xie, and Fatma Güney, Self-supervised object-centric learning for videos. In 0r11////330. 06907.
NeurlPS, 2023. URL
Jack Brady, Roland S. Zimmermann, Yash Sharma, Bernhard Schölkopf, Julius Von Kügelgen, and Wieland Brendel. Provably learning object-centric representations. In ICMI., 2023. URL mlr. press v22/2222////// 22202bbadyy33a .htmm...................
https: /ttpppppceengss.e/ee
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 2020. URL htt.t:ttt///aatt /ttp::///xxvvvvvee
wrgg attgg//b///// 2005 141 65.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv:1901.11399. 2019, URL https: www.:riivvv.orvaaaaaoo hbagg//bb/// 190.. 11390.
Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar, Benchmarking in manipulation research: Using the Yale-CMU-Benneley object and model set. IEEE Robotics Automation Magazine, 2015. URL http::tt////eett //ipe///////eeex///////e// ieee. org/
document /7243318.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. ICCV, 2021. URL
https: / httpx/xxvvvo//// www./oog///////ooooo//oooo/oooa/ /tbs//1100 14294.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. URL
httpss://axiyy org/ al/////s///// 1102 04405.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski. Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim M. Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnb,, Xiao Wang. Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evei, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander
11

---

Published as a conference paper at ICLR 2025
Kolesnikov, Filip Pavetić. Dustin Tran, Thomas Kipf. Mario Lučié, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In ICML, 2023. URL https:/:: www.iiii.ogg.oooooeoooooooo www./rg///////ooooooo//..../.... abs 302.. 05442.
Aniket Rajiv Didolkar, Anirudh Goyal, and Yoshua Bengio. Cycle consistency driven object discovery. In ICLR, 2024. URL
net /t/f//ormmmmmdlmmtfm
Andrea Dittadi, Samuele S Papa. Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Locatello. Generalization and robustness implications in object-centric learning. In ICML, 2022.
URL hwwphhtps/.cmmm http///axvvvv..e//ev org/ abs 2110 00637.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, 2021. URL https: /wtp///ppneeeieeeeee/ee/ee net wwwee//foommidddiiiiee cNNTTT......yy.y..mmm
Danny Driess, Fei Xia, Mchdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. PaLM-E: An embodied mul- timodal language model. In /CML, 2023. URL https::/ http/vaxvivvo... abs/abas// 03378.
Gamaleldin Fathy Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Curtis Mozer, and Thomas Kipf. SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos. In NeurlPS, 2022. URL
net fttt/foormmm dT= 9W5 331.1..... LXNS.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations. In ICLR, 2020. URL
https: wpwpnreveeeeeeeeeeeeeeeeeeeeeeee .nt/ forummiddBBxxfftVV.... http:///www..eeee.oooooooo/ooaaa
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E. Hinton. Attend, Infer, Repeat: Fast Scene Understanding with Gen- erative Models. In NeurlPS, 2016. URL httttss:/pproceeed//t
neuri cttppppppcc////////p///// 2016/hash/ 0ttp://5557700e55500050000 5ee4eeeel555000011e 38600866688888 23512327002222200444444336015553 Abstt html.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual
Object Classes Challenge 2012 (VOC2012), 2012. URL http: / /wwww www.ppascalneete network.
http://oocoow//ooooooooo// cechhhoppnndeee index.
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cog-
nition. Proceedings of the Royal Society A, 47802266):20210066, 2022. URL. https: //royalsocieie www.eltpppplhhhho.e
www.://.g/ooooooooooooooo / 1099/09//000 http://rppp.000 2021.0068
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-Object Representation Learning with Iterative Variational Inference. In ICML, 2019. URL hattp:://arxiv..tt abs/1903. 00450.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the Binding Problem Artifi- cial Neural Networks. arXiv:2012.05208, 2020. URL https: hta//axxvvvorro orggabs////12// 05208.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhoug, and Andrea Tagliasacchi. Kubric: A Scalable Dataset Generator. In CVPR, 2022. URL https: arxiiv.iiv..////i.//a..aaaaaiav. org/ abs 22203.2...2 0.2203..3 03570.
Jean-Basticn Grill, Florian Strub, Florent Altch'e e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your latent:
12

---

Published as a conference paper at ICLR 2025
A new approach to self- supersueeeeieed learning. In NeurlPS, 2020. URL URL.ttps://arxxl..oom abs/2006. 07733.
Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Unsupervised Semantic Segmentation by Distilling Feature Correspondences. In ICLR, 2022. URL https: /pwpnreeeeeeeeeeee/eeeeeee net forum? id= 6H1006666660000 6H10c.
Dan Haramati, Tal Daniel, and Aviv Tamar. Entity-centric reinforcement learning for object ma- nipulation from pixels. In ICLR, 2024. URL /ttp://////pereeee.... net forum? ?id= wdII.
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In ICCV, 2011. URL https:://ieeee https://ieexxplore ieeee org/ document/ 6126343.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Annual Meeting of the Association for Computational Linguistics (ACL). 2018. URL https:
//arxiv. org/ abs 1801. 06146.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 1985. URL https: //i::///nniii/... swrinprrrrrrre. com/ http://at.c.....//////e....
A. Jabri, Sjoerd van Steenkiste, Emiel Hoogeboom, Mehdi S. M. Sajjadi, and Thomas Kipf. DORSal: Diffusion for object-centric representations scenes et al. In ICLR, 2023. URL https://
arxiv.org/ei abs 2306.00000000
Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. In NeurIPS, 2023. URL https: //tp///aaavv////a///aa//eeeeeaaa org abs / 2303.10834
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. URL ht /tt.iaaxvvv///////////ea/////avv org/ abs 61 2 66990
Ioannis Kakogeorgiou, Spyros Gidaris, Konstantinos Karantzalos, and Nikos Komodakis. SPOT: Self- training with patch-order permutation for object-centric learning with autoregressive transformers. CVPR, 2024. URL https:://///.c.. aariivivvvv.////vv///aaa/aaaaiaa wwt./bs./aaaaaeaeoo..o..oe.. 2312. 00648.
Laurynas Karazija, Iro Laina, and Christian Rupprecht, ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi- Segmentation. In NeurlPS Track Datasets and Benchmarks, 2021.
URL https::///// /tap:p:://raaaae
Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Rezende, Yoshua Bengio, Michael Mozer, and Christopher Pal. Systematic evaluation of causal discovery in visual model based reinforcement learning. arXiv:2107.00846 2021. URL https://arxits org/ alss 2ww.700088.00 00848.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In CVPR, 2019. URL htt.s: /ttpivivvvvv..////////.////aaavv wttp:/v..or//bb//////e/ 000868.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV, 2023, URL /ttpt::://axxx...ot org/ abs 2304 02 043
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In AAAI, 2008. URL https: : / d1. acm. org doi//0.i10110/// http://11..5551111155553 /2015//66661116611 1620172.
Tsung-Yi Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zimick. Microsoft COCO: Common Objects in Context. In ECCV, 2014.
URL https:://// watpaaxvvvv...///..a//aaa org/ abs 1405 0312.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition. ICI.R, 2020. URL
022000 02407.
13

---

Published as a conference paper at ICLR 2025
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-Centric Learning with Slot Attention. In NeurlPS, 2020. URL https: http://tttteee.seeeeeeeepoecttst neurips cttpppprr/c22/ File/8511 /flle//511ffffeee (aee111ff90c2abbboe 1e23ecc2333c 30ccff00000000000ppeeee .ddf.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017. URL https: //wpivv.a.ivv.v//aa/iai http://wxwwwwo.og///// hbtp//a///ss5 http:///wwwga/a////e/eaaae/
03983.
Sindy Löwe, Phillip Lippe, Francesco Locatello, and Max Welling, Rotating features for object discovery. NeurlPS, 36, 2024. / wwweiiiiorrr.ooo abs/a3b6 20000.00000000
Qi Lu, Jason Kuen, Shen Tiancheng, Gu Jiuxiang, Guo Weidong, Jia Jiaya, Lin Zhe, and Yang Ming-Hsuan. High-quality entity segmentation. In ICCV, 2023. URL https: // //tp://.arxvo//o///oo abs 3bss2211.055 05776.
Amir Mohammad Karimi Mamaghan, Samuele Papa, Karl Henrik Johansson, Stefan Bauer, and Andrea Dittadi. Exploring the effectiveness of object-centric representations in visual question answering: Comparative insights with foundation models. arXiv:2407 2024. URL https:
wttp:/vv..grg/b////0000 15589.
OpenAI Team. GPT-4 technical report. arXiv:2303.087//4 2024. URL httstt/::/arxxioooo abs/2303. 08774.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Syafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Waer Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nico- las Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. TMLR, 2023. URL https::///// /wwpavaavvv///////////eaaaaawwaa org/ abs/ 2304. 07193.
Steven Pinker. Visual cognition: An introduction. Cognition, 1984. URL https:::://di. 0277 (84) 90021-0.
Jordi Pont-Tuset, Pablo Arbeláez, Jonathan T. Barron, Ferran Marques, and Jitendra Malik. Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2017. URL https::
dgee.orghe oieeeeorr///ocmeeeeeee dwwp///t/////////////mmmmm /7423791.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl Blog, 2019. URL https::/cdd..prttttttt openai. www. www.wotwtteeeeeeeeeeeeeeeeeewe.e langurgangggegeee maneuggemmmeelsssseesee 1anguage.. models laclelssare____le s__.are_wwsupprrrssseemmeateeeee mmldd__mllliii
.pf..
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American
Statistical association, 1971. URL https: /tww://www////////////////.. jstorr org/ wtt.ss/eabee2aa22
Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. In NewrIPS, 2022. UtLp aaa/vviivvv... orw.orrg//aao..o abs 22...2266000 06922.
Mohammadreza Salehi, Efstratios Gavves, Cees G.M. Snoek, and Yuki M. Asano. Time does tell: Self-supervised time-tuning of dense image representations. In ICCV, 2023. htts:::: //a www.wii.w............o.. a338
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Towards Causal Representation Leanning, IEEE Advances in Machine Learning and Deep Neural Networks, 2021. URL https:// http///axivv... wwwpwiiiiorrr.oooo / abs/
02. 11107.
Maximilian Seitzer, Max Horn. Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to roaldworld cbject-centric learning. In ICLR, 2023, URL https: //openrevie net d
14

---

Published as a conference paper at ICLR 2025
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-E Learns to Compose. In ICLR, 2022a.
URL httph::////.... /wwprrevee.eeeeeeeeeeee net tttpeeeeeeeeeeetttttotooooooooea
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple Unsupervised Object-Centric Learning for
Complex and Naturalistic Videos. In NeurIPS, 2022b. URL https: httttps://openrevieww
forum?id==YEIIBBB.
Elizabeth Spelke. Principles of object perception. Cognitive Science, 1990. URL https:
//on1 ine librrry.r... wi wiley cwm/cotooooooooo doi /tt////////// /10. 120.1220/151111 http:///o////a///....//////..1./ 6709c/1155500090000000
Elizabeth S. Spelke. Core knowledge. The American psychologist, 2000. URL https: //ooi. /doi. oxg/10.1037/000333
55. 55.11. 1233.
Manuel Traub, Sebastian Otte, Tobias Menge, Matthias Karlbauer, Jannik Thuemmel, and Martin Butz. Learning what and where: Disentangling location and identity tracking without supervision. In ICLR, 2023. URL https: https://openzeee http///pppeeeeeeeeeeeeeeeeeeeeee net/ forum? -d=-eed-Ak-H_.....
Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. Dino-tracker: Taming dino for self- supervised point tracking in a single video. arXiv:2403. 14548, 2024. URL prttps://arxot org/abs/ 2403. 14548.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Nicholas Watters, Loic Matthey, Christopher P. Burgess, and Alexander Lerchner. Spatial broadcast decoder: simple architecture for learning disentangled representations in vaes. 1XXiv:1901.00017 2019. URL https: arxiv org abs / 1 901 07017.
Thaddäus Wiedemer, Jack Brady, Alexander Panfilov, Attila Juhos, Matthias Bethge, and Wieland Brendel. Provable compositional generalization for object-centric learning. In ICLR, 2024. URL
https: epw.reviewwewweeeeeeeeeeee.eewwe net.
Ziyi Wu, Nikita Dvomik, Klaus Greff, Thomas Kipf, and Animesh Garg. SlotFormer: Unsupervised visual dynamics simulation with object-centric models. In ICLR, 2023a. URL https://
openreview wwwprreiewwneeteeee foret//foum22 -Fw.numdd--TTTEIILLILI
Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Object-centric generative modeling with diffusion models. In NeurIPS, 2023b. URL httss/tss://xxxi.orro abs/2305. 11281.
Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu. Slot-VLM: Slowfast slots for video-language modeling. arXiv:2402.1300 1308, 2024. URL /tt.ttt////xxx..o///////a/ 13088.
Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019. URL
https:/ /ttp///aavvvv///a///ee//eeee..av org/ barg/abs/// abs/ 905 04804.
Yafei Yang and Bo Yang. Promising or Elusive? Unsupervised Object Segmentation from Real- world Single Images. In NeurlPS, 2022. URL https::/openreviiee http:/wwwweee.o.oooeoeeeeeeeee iid=
DzPWIwfby5d.
Jaesik Yoon, Yi-Fu Wu, Heechul Bae, and Sungjin Ahn. An investigation into pre-training object- centric representations for reinforcement learning. In ICML, 2023. URL https: /tttps://arxiv.
org/abs/ 2302. 04419.
Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Self-supervised Visual Reinforcement Learning with Object-centric Representations. In ICLR, 2020. URL http::://oppeeeee/eee//// net/forum?ice batpzrumiid-xpppoee 2dtppumm2.dxxpppLXcooo1...
Andrii Zadaianchuk, Matthaeus Kleindessner, Yi Zhu, Francesco Locatello, and Thomas Brox. Unsupervised Semantic Segmentation with Self-supervised Object-centric Representations. In ICLR, 2023a. URL htwpt:ttt//tc. http::/://///eeeeeeeee//ee fett:/forrmmddd1_11111 07YC.
Andri Zadaianchuk, Maximilian Seitzer, and Georg Martius. Object-centric learning for real- world videos by predicting temporal feature similarities. In NeurIPS, 2023b. URL https:
//waxiv.orgg/aee 22306 022.2223666..22... 04829.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A ADDITIONAL EXPERIMENTS
A.1 ZERO-SHOT BENCHMARK
We show additional results complementary to the results in the main part. Figure A. shows benchmark results in terms of varying models, training data distribution, and training dataset size, but with the mBO metric instead of the FG-ARI metric. The results largely mirror those in Fig. 1; it can be seen that DINOS AUR generally has worse mBO than SPOT and SlotDiffusion, whereas with FG-ARI, this trend is reversed.
Figure A.3 shows the data scaling behavior of our FT-DINOSAUR method trained on different subsets of the Coco dataset, showing performance the individual datasets in Fig. A.3a, and comparing the aggregated performance to DINOSAUR in Fig. A.3b. While DINOSAUR is better in the very-low sample regime (less than 5 500 samples). FT- ST-DINOSAUR overall shows better scaling behavior. In particular, FT-DINOSAUR exhibits a slightly upward trending scaling curve for OOD evaluation with FG-ARI; while the effect is too weak to conclude that FT-DINOSAUR scales well with data, it would be interesting to extend this experiment to include 1-2 magnitudes more data.
A.2 OBJECT-CENTRIC FINETUNING
Targets from EMA teacher We can also frame our setup as variant of the student-teacher framework common self- supervised methods (Grill et al., 2020; Caron et al., 2021; Oquab et al., 2023). There, the weights of the teacher model continuously updated from the student's 's weights through an exponential moving average (EMA), with a momentum param- eter [o, 1] controlling the speed of adaptation. Through this lens, our approach uses r 1, corresponding to not updating the teacher. This view suggests use targets throughout training.
Table A.1: Analysis of targets. is the momentum for teacher up- dates.
FG-ARI
mBO
0.09
13.6
0.999
23.6
16.4
0.9999
37.8
21.0
improve the
48.5
30.7
In Table A.I, we analyze the effect of introducing student-teacher style EMA updates. Directly using the features of the student as the targets (T = 0) leads to collapse, as reported previously (Seitzer al., 2023). With momentum updates, we still find a high value for be necessary stabilize training. Using fixed targets (r = 1) gives the best results. We speculate this is because there is no missing information the auto-encoder setup, leading to gradual loss of information.
Training Encoder from Scratch Here, we build upon the approach mentioned in Section 4.1 where the model is trained with a ViT encoder initialized from scratch while still predicting DINOv2 features
Average
Average
MOVI-C
EntlySSg
MOVI-C
0.B
Entily Sog
Model
0.4g
4
DINOSAUR
SlorDiffusion
MOV
0.0
ClevrTex
0.0
SPOT
SAM (comp.)
SAM (best)
ScanNet
YCB
MOVIE
Tlew
(a) Object discovery (mBO).
(b) Linear probing of category (Accuracy).
Figure A.1: Evaluating zero-shot transfer of current object-centric models trained on the Coco dataset. Corresponds to Fig. 1, but shows (a) mBO instead of FG-ARI and (b) linear probing accuracy of object category from slots.
16

---

Published as a conference paper at ICLR 2025
Training Dataset
ClevrTex
MOVi-C
mBO normalized by ID
0.4
Evaluation Datasel
COCO
MOVi-E
1.5
Mean OOD
ScanNet
0.3
EntiyySeg
1.0
ScanNet
YCB
YCB
PASCAL
0.5
MOVI-C
EntitySeg
0.1
MOVI-E
COCO
CleviTex
Glev:
MOViC
MOV-E ScanNet
YoB PASCAL
Emtlyy
2'
2
2
2
2
Size of Training Dataset
Evaluation Dataset
(a) Varying train datasets.
(b) Varying train dataset size.
Figure A.2: Effect training data on transfer of the DINOSAUR model. Corresponds to Fig. 2,
but shows mBO instead of FG-ARI.
0.8
0.4
Evaluation Dataset
CDCO (ID)
FG-ARI
0.6
0.3
OOD
EntitySeg
0.4
0.2
ScanNet
YOB
0.2
0.1
MOVi-E
0.0
0.0
Clevr Tex
21
13
25
2
2a
2
2
Size of Training Dataset
Size of Training Dataset
Scaling behavior of FT-DINOSAUR, showing OOD performance individual datasets.
0.8
0.4
DINOSAUR
0.6
OOD
FG-ARI
COCO (ID)
mBO
TT-DINOSAUR
Mean OOD COCO
0.2
0.0
0.0
23
15
of Training Dataset
Size of Training Dataset
(b) Scaling behavior of FT-DINOSAUR vs. DINOSAUR.
Figure A.3: Scaling behaviour of FT-DINOSAUR on trained on differently sized subsets of Coco. Our method uses a ViT-S/14 with DINOV2 with finetuning, but no top-k decoding and hi-res adaptation.
17

---

Published as a conference paper at ICLR 2025
Table A.2: Zero-shot performance without a pre-trained encoder. Comparison of FG-ARI and mBO across multiple datasets. Scratch-DINOSAUR is identical to FT-DINOSAUR except that the ViT encoder initialized from scratch and trained to predict DINOv2 targets. The average is computed as weighted average, normalizing with the size of the evaluation datasets.
Movi-C
Movi-E
SCANNET
YCB
FG-ARI
FG-ARI
mBO
FG-ARI
mBO
FG-ARI
mBO
DINOSAUR SlotDiftusion
67.0
34.5
71.1
24.2
57.4
40.8
60.2
42.2
66.9
43.6
67.6
26.4
52.0
51.7
62.5
59.2
SPOT
63.0
40.8
47.8
21.5
48.6
43.2
52.9
45.1
FT-DINOSAUR (ViT-S/14)
71.3
44.2
71.1
29.9
54.8
48.4
67.4
54.5
Scratch-DINOSAUR (ViT-S/14)
68.3
31.9
67.9
23.1
50.5
36.9
57.3
35.8
CLEVRTEX
ENTITYSEG
COCO
Average
FG-ARI mBO
FG-ARI mBO
FG-ARI
mBO
FG-ARI mBO
DINOSAUR
82.5
35.2
43.5
19.4
42.4
29.4
68.4
32.7
SlotDiffusion
77.0
45.0
43.7
25.1
37.3
31.4
65.8
41.2
SPOT FT-DINOSAUR (VIT-S/14)
63.3
41.7
27.4
37.0
34.8
55.4
35.9
86.0
50.1
48.1
28.4
48.5
30.7
71.2
42.9
Scratch- Bretch-DINOSAUR (ViT-S/14)
78.8
31.4
46.6
22.7
42.3
27.3
67.1
36.4
targets. Note that, fully removing the pre-trained targets is not feasible real-world data because, in that case, the representations collapse and the model fails to discover objects. In this setting, the model is trained solely on the COCO dataset, and refer this variant
Table A.2 presents the results of zero-shot object discovery across the benchmark. Even though the encoder was not pre-trained, Scratch-DINOSAUT achieves 42.3 FG-ARI and 27.3 mBO on COCO, and it transfers to the other datasets with performance mostly on par with DINOSAUR in FG-ARI (even outperforming certain baselines in some cases), albeit with drop in mBO. Thus, a pre- traitri encoder is not strictly mandatory for robust zero-shot transfer of object-centric representations, but the combination of pre-trained features with slot attention remains advantageous. fact, adding our proposed finetuning strategy (FT-DINOSAUR) on top of high-quality encoder yields the best performance overall.
Zero-Shot Performance with Learnable Slot Initialization We investigate the impact of learnable slot initialization on zero-shot object discovery performance. Unlike our proposed approach, which random slot initialization, this variant employs learnable initial slots while keeping all other components unchanged.
On COCO, this model achieves 44.3 FG-ARI and 29.9 mBO, compared to 48.5 FG-ART and 30.7 mBO achieved by our proposed method. To further evaluate its effectiveness, we test the model under zero-shot conditions on multiple datasets. We denote this variant FT-DINOSAUR (learnable init.) and summarize its performance in Table A.3.
As shown in Table A.3, the learnable slot initialization approach performs competitively FG-ARI but significantly underperforms in mBO compared to our proposed FT-DINOSAUR method. This indicates that while the learnable initialization still allows for reasonable object discovery, produces less precise segmentation masks, leading to reduced mask quality.
Additionally, using learnable slots has major drawback in zero-shot settings: the number of slots is fixed at training time and cannot be adjusted during inference. In contrast, our proposed random initialization approach enables the model to generalize across different slot configurations at test time, which is particularly useful when dealing with novel datasets.
Overall, these results validate the importance of using random initialization for achieving both strong object discovery and high-quality segmentation masks.
Analysis of Finetuned Features Fig. A.4, we show additional examples for visualizing the PCA on the finetuned features compared DINOv2 features (similar to Fig. 4). Similar to the discussion
18

---

Published as a conference paper at ICLR 2025
Table A.3: Zero-shot performance with learnable slot initialization. We compare FG-ARI and mBO across datasets for FT-DINOSAUR DINOSAAUR variants. The learnable initialization approach performs competitively in FG-ARI but shows significantly lower mBO, indicating weaker mask sharpness.
Movi-C
Movi-E
SCANNET
YCB
FG-ARI
mBO
FG-ARI
mBO
FG-ARI
mBO
FG-ARI
mBO
FT-DINOSAUR (Icarnable init.)
72.2
30.8
65.6
18.2
54,9
42.9
67.3
42.8
FI-DINOSAUR (ViT-S/14)
71.3
44.2
71.1
29.9
54.8
48.4
67.4
54.5
FI-DINOSAUR (ViT-B/14)
73.3
42.9
69.7
27.9
55.8
48.6
70.1
54.1
CLEVRTEX
ENTITYŠEG
COCO
Average
FG-ARI mBO
FG-ARI mBO
FG-ARI mBO
FG-ARI
mBO
FI-DINOSAUR (learnable init.)
81.7
31.8
47.7
24.2
44.3
29.9
68.2
31.3
FI-DINOSAUR (ViT-S/14)
86.0
50.1
48.1
28.4
48.5
30.7
71.2
42.9
FI-DINOSAUR (ViT-B/14)
83.9
45.9
49.7
29.0
46.9
29.5
71.5
40.8
Sec. 4.2, we find that after finetuning, the encoder features are noticeably more object-centric. For example, in the first and last examples, DINOv2 features show a part-based split of the shown persons in the dominant PCA components; the finetuned features highlight the whole persons better. In the second example, DINOv2 features group semantic instances (human) together in the dominant components; after finetuning, the features clearly split the persons. However, note that is not necessary that the features highlight the instances in the dominant components derived instance- based grouping; in all examples, the masks discovered by DINOSAUR (last column) feature a correct instance split (while also splitting further into parts the last two examples). This may be because the necessary information for the correct split is contained in the less dominant components of the features (e.g. e.g. in PCA dimensions 4-6). However, we conjecture that the finetuned features simplify the grouping task for slot attention, leading to better and more consistent object discovery.
A.3 EVALUATION
include the following additional results for the evaluation of FT-DINOSAUR conducted Sec. of the main paper:
Finetuning In-Distribution (Sec. 5.1): we show the numeric values corresponding to Fig. in the main part in Table A.4. This table also shows the results for the in-distribution zero-shot comparison in Fig. 6.
Extended Comparison To Prior Work on Real-World Object-Centric Learning (Sec. 5.2): we conduct extended comparison prior work for real-world object-centric learning on the Coco dataset in Table A.5.
Zero-Shot Evaluation (Sec. 5.3): for object discovery, we show the full results all datasets of the zero-shot benchmark, corresponding to Fig. la and Fig. 7 in the main part in Ta- ble A.6. For downstream property prediction, we show the per-dataset results in Table A.7, corresponding to Fig. 1b in the main part.
B MODEL LIMITATIONS
Even though FT-DINOSAUR brings large improvements over DINOSAUR, the model still exhibits problems with certain types of scenes. In Fig. B.5, we show several examples of such failure cases, grouped into modes of failure. Two typical categories of failure are the overgrouping of semantically- related objects (Fig. B.5a), and the split objects into parts (Fig. B.5b). Both problems are primarily caused by the model using the wrong number of slots. But note that even having access to the "correct" number of slots per image can not resolve all problems, as the model may still allocate the slots in undesirable ways. Consider the last example in Fig. B.5a: here, the model could correctly split the two persons into individual slots if the motorbike is grouped as one object instead of as parts. A third category of failure broadly stems from difficult or unusual images (Fig. B.5c): for example, grouping
19

---

Published as a conference paper at ICLR 2025
Image
PCA (1-3)
PCA (4-6)
Masks
DINOSAUR
SPOKAY
OC Finetuning
Image
PCA (1-3)
PCA (4-6)
Masks
DINOSAUR
OC Finetuning
Image
PCA (1-3)
PCA (4-6)
Masks
DINOSAUR
OC Finetuning
Image
PCA (1-3)
PCA (4-6)
Masks
DINOSAUR
OC Finetuning
Figure A.4: Visualization of encoder features in DINOSAUR (frozen DINOV2 features) and for encoder features adapted with object-centric finetuning, similar Fig. 4 in the main paper. The second column shows 1st to 3rd PCA components, and the third column shows 4th to 6th PCA components grouped one image by using different RGB channels. The last column shows object discovery masks by each method.
20

---

Published as a conference paper at ICLR 2025
Table A.4: Evaluation of a ViT-S/14 DINOv2 backbone. Finetuning shows strong performance improvements on all eight datasets. We also show transfer when finetuning Coco, which performs comparable better to training in-distribution on 5 out of datasets, Results corresponding to experiment in Fig. 5 in the main paper.
to DINOSAUR when training in-distribution, using
Movi-C
Movi-E
SCANNET
YCB
FG-ARI
mBO
FG-ARI
mBO
FG-ARI
mBO
FG-ARI
mBO
DINOSAUR FFieetuning
63.1
33.3
74.0
25.5
52.8
38.8
67.5
28.9
71.9
36.8
82.0
28.1
63.8
43.8
71.1
27.5
Zero-shot (Coco)
76.4
34.9
75.1
24.5
55.4
42.8
69.9
44.4
CLEVRTEX
PASCAL Voc
ENTITYSEG
Coco
FG-ARI
mBO
FG-ARI
mBO
FG-ARI
mBO
FG-ARI
mBO
DINOSAUR
91.3
40.1
26.2
40.1
43.0
19.6
42.4
29.4
+Finctuning
95.8
40.1
29.7
40.8
45.7
20.9
48.5
30.7
Zero-shot (Coco)
87.5
38.2
31.7
40.4
47.7
21.8
48.5
30.7
Table A.5: Extended comparison of FT-DINOSAUR to prior work on the CocO dataset, corre- sponding to Table in the main paper. For our proposed approach, we average results across 5 seeds for FT-DINOSAUR, VIT-S/14 and across seeds for FT-DINOSAUR, ViT-B/14. Results marked with are from evaluating official checkpoints; results marked with * are taken from the respective papers. Supervised models (SAM) colored in gray.
Model
FG-ART
mBO
P-ARI
PQ
Slot Attention (Locatello et al., 2020), (Wu et al., 2023b)*
21.4
17.2
SLATE (Singh et al., 2022a), (Wu et al., 2023b)*
32.5
29.1
DINOSAUR (MLP Dec.) (Seitzer al., 2023)
40.5
27.7
37.1
14.4
DINOSAUR (TF. Dec.) (Seitzer et al., 2023)*
34.1
31.6
Stable-LSD (Jiang al., 2023)*
35.0
30.4
SlotDiffusion (Wu al., 2023b)
37.3
31.4
47.6
21.0
SPOT (Kakogeorgiou et al., 2024)t
37.0
34.8
52.4
21.3
FI-DINOSAUR, ViT-S/14
46.6
35.6
49.7
23.5
FT-DINOSAUR, VIT-B/14
48.8
36.3
49,4
23,9
SAM (comp.) (Kirillov et al., 2023)1
12.1
19.0
10.8
9,4
SAM (best.) (Kirillov et al., 2023)
44.9
56.9
54.4
10.9
21

---

Published as a conference paper at ICLR 2025
Table A.6: Per-dataset zero-shot performance, corresponding to Fig. 7 in the main paper. All unsupervised object-centric methods (DINOSAUR, SlotDiffusion, SPOT, FT-DINOSAUR) are trained on the Coco dataset. Furthermore, we compare with the supervised Segment Anything model (SAM). Average is computed as weighted average normalizing with the size of the evaluation datasets (cf. Table E.9), and only includes the zero-shot datasets (first columns). For completeness, we also include the performance when transferring from Coco PASCAL VoC in the last column. Results marked are from evaluating official checkpoints. Supervised models (SAM) colored in gray.
Movi-C
Movi-E
SCANNET
YC
FG-ARI
mBO
FG-ARI
mBO
FG-ARI
FG-ARI
mBO
DINOSAUR
67.0
34.5
71.1
24.2
57.4
42.2
SlotDiffusion'
66.9
43.6
67.6
26.4
52.0
51.7
62.5
59.2
SPOT
40.8
47.8
21.5
48.6
52.9
45.1
FI-DINOSAUR (ViT-S/14)
71.3
44.2
71.1
29.9
54.8
48.4
67.4
54.5
FT-DINOSAUR (ViT-B/14)
73.3
42.9
69.7
27.9
55.8
48.6
70.1
54.1
SAM (comp.))
57.6
45.3
38.5
27.4
45.8
45.5
46.9
40.9
SAM (best.)
79.7
73.5
84.7
69.7
62.2
64.7
69.4
69.8
CLEVRTEX
ENTITYŠEG
Average
PASCAL Voc
FG-ART
mBO
FG-ARI
mBO
FG-ARI
mBO
FG-ART
mBO
DINOSAUR
82.5
35.2
43.5
19.4
68.4
24.0
37.2
SlotDiffusion'
77.0
45.0
43.7
25.1
65.8
41.2
21.1
42.0
SPOT
40.0
41.7
27.4
55.4
35.9
50.6
FT-DINOSAUR (ViT-S/14)
86.0
50.1
48.1
28.4
71.2
42.9
24.0
37.6
FT-DINOSAUR (ViT-B/14)
83,9
45.9
49.7
29.0
71.3
41.1
25,9
37.8
SAM (comp.))
82.9
70.3
25.9
16.5
55.2
45.7
31.0
51.5
SAM (
94.0
90.0
53.4
51.0
79.5
73.2
31.1
64.2
Table A.7: Per-dataset downstream property linear prediction performance. We report in- distribution property prediction performance for each dataset (DINOSAUR (ID) and zero-shot perfor- mance of models trained on the COCO dataset (DINOSAUR, SlotDiffusion, SPOT, FT-DINOSAUR, FT-DINOSAUR Hi-Res). Results marked are from evaluating official checkpoints.
Movi-C
Movi-E
YCB
Acc (%)
H²
Ace (%)
R²
Acc (%)
H²
DINOSAUR (ID)
70.7
0.69
64.3
0.44
0.54
DINOSAUR
77.1
0.69
70.4
0.45
0.77
SlotDiffusion'
0.58
78.4
0.29
0.76
SPOT
68.2
0.55
66.8
0.35
0.76
FT-DINOSAUR (VIT-S/14)
75.9
0.65
67.4
45.5
0.83
FT-DINOSAUR (ViT-S/14) Hi-Res
74.6
0.65
69.3
0.47
0.83
CLEVRTEX
ENTITYSEG
SCANNET
Acc (%)
R
Ace (%)
R
Ace (%)
R²
DINOSAUR (ID)
90.7
0.90
25.9
0.53
DINOSAUR
83.4
0.76
24.1
0.54
0.80
SlotDiftusion'
77.8
0.67
21.5
0.36
0.82
SPOT
70.7
0.55
29.7
0.42
0.82
FT-DINOSAUR S/14
77.8
0.68
18.1
0.45
0.80
FT-DINOSAUR (VIT-S/14) Hi-Res
79.9
0.70
19.4
0.51
0.79
22

---

Published as a conference paper at ICLR 2025
tiny objects together with the background (cars on bridge); incorrect 3D inference due to unusual camera perspective (rail and light post); sub-optimal decompositions in OOD scenes (grass stalk in forest, sand patterns).
How could the failure modes regarding overgrouping and oversplitting be resolved? First, like all slot attention/DINoSS AUR-based methods, FI-DINOSAUR decomposes the scene into fixed number of regions/objects. However, especially on real-world images, the number of objects varies significantly from image image. Therefore, it important to develop methods that infer suitable number of objects for an image; however, further innovations are needed to deal with the slot allocation problem we have alluded before. Second, unsupervised scene decomposition is inherently an ill-defined data as scenes can be split in numerous ways (cf. Figs. B.5a and B. B.5b). Thus,
task on predicting only a single set of masks might ultimately be insufficient. Instead, may be beneficial to model the full part-whole hierarchy, producing various decompositions of different granularity. Such models could further allow control over the level of granularity through external conditioning variables text. However, the examples in Figs. B.5a and B.5b also demonstrate the limitations of current evaluation techniques. Arguably, these are not failures of the model, but are treated as such by the evaluation metries. This is because current datasets have annotations that prescribe single ground truth labeling for each image. Instead, datasets should be annotated with multi-level labelings, e-g. by including parts of objects, or further splitting the background into specific elements (e.g. splitting the background "tree" class into particular trees). To evaluate methods that model the full part-whole hierarchy, such annotations even become necessity.
C METHOD DETAILS
C.I DINOSAUR
DINOSAUR (Seitzer et al., 2023) proposes to use pretrained encoders for unsupervised object dis- covery. The image is processed using a pretrained DINO encoder (Caron et al., 2021) into set of feature vectors {fi, f2... J. J. The slot attention module (see App. C.2) is applied to the outputs of the pretrained encoder to produce set of slot vectors {81, ....... 8ar) ). The slots are decoded using broadcast MIP decoder (Watters et al., 2019) produce set of feature vectors fi, f2.... fN-
The MLP decoder decodes each slot 8 separately output mask logit riga and feature values fu- Here, i E f1....N' . indexes the features and k E {1...., K} indexes the slots. The mask logits are converted to probabilities miA by taking their softmax across all slots. Here, mik denotes the probability that slot 8 produces feature f. and fax denotes the feature value assigned feature f: by slot 8t. The final value for each feature is obtained by taking weighted sum across all slots: f. - fik. The training objective is the feature-reconstruction objective: L = || f1
Note that the mask my E R. is the mask obtained for given slot. This mask is resized to the original image size and overlayed over the image to produce the visualizations shown in Table 1.
C.2 SLOT ATTENTION
Slot attention (Locatello al., 2020) a differentiable clustering procedure that operates on a set of visual features (usually output by CNN) and outputs a set of slot vectors (21, 22, =x} where each slot represents an object. The differentiable clustering procedure is iterative in nature. We now describe the procedure in detail.
Consider that we have a set features [fi, JN where f: E RRP,, and a set of slots (= ... zx} where 2; R ? The features [fi,... fN } output by CNN encoder given input image and the slots are sampled from a Gaussian distribution. The attention mechanism computes the assignment of features slots. Ideally, the set of features belonging to a particular object will map a single slot and each slot will bind to a distinct object. To induce this behavior, the attention mechanism implements a competition between the slots to represent parts of the feature space.
First, following the general process of guery-key-vale (QKV) attention (Vaswani, 2017), the slots projected queries qi Linear,(z.) and the features are projected keys k; Linear (fi) and values v; Linear, (f.).. This results in queries q R keys k E RND, and values
23

---

Published as a conference paper at ICLR 2025
(a) Joining semantically-related ohjects together.
(b) Splitting objects into parts.
(c) Complex or unusual images: (1) tiny objects, (2) incorrect 3D inference, (3, 4) OOD scenes.
Figure B.5: Failure modes of FT-DINOSAUR. We show typical failure cases grouped into three categories: (a) joining semantically related objects into a single object; (b) splitting objects into parts; and (c) incorrect decomposition of complex or unusual scenes. Note that the model's 's decompositions in (a) and (b) arguably are correct but do not correspond to the labeling prescribed by the ground truth annotations; without knowledge of the intended downstream task, the "correct" grouping is ambiguous. We use the model from Table 2; it uses a ViT-B/14 encoder with hi-res adaptation and is trained on the Coco dataset. All images show zero-shot predictions the ENTITY SEG dataset.
24

---

Published as a conference paper at ICLR 2025
RN The dot dot-product attention then computes the affinities between the queries and the keys: attn = q x k", where attn E RN*K To induce a competition between the slots to represent the features, they take softmax across the queries opposed taking it across the keys which is what is usually done QKV attention. Therefore, the final attention values become attn softmax( gxk" are obtained by taking a weighted mean the values & W v, where Wij p attnngn Therefore, the attention computation used in each iteration of slot attention can be implemented as follows:
VD axis q), where vD is the softmax temperature. The final attention outputs
attney
attn softmax ((<) (()) k(f))
axis query)
(1)
VD
attn
attn Sum(attn, axis = keys)
(2)
2 attn (ifn-((f). vff).
The authors find useful from a stability viewpoint to process the inputs f and slots a using layer normalization. The output of the attention mechanism is used to update the current estimate of the slots in an iterative procedure which will describe next.
Iterative Procedure The iterative updates the slots carried out using a GRU and an MLP. The slots ...... 2K are first initialized as the hidden states of GRU. Given the updates from the attention mechanism, the slots are updated as follows:
= Ginput = state = 2,) 2,
= (L.((LaayNNrrm(()) - 2;-
These updates happen for fixed number of iterations T which is set hyperparameter.
This whole process can be viewed as soft k-means clustering. The slots are analogous to cluster centers and each attention computation computes the assignments of features slots. Next, the slots (cluster centers) are updated with the information from the features assigned to each slot.
C.3 TOP-K DECODING
We first recap the MLP decoder from DINOSAUR Seitzer al. (2023), For N patches and K slots, the MLP decoder produces reconstruction ý x as well as an alpha mask a E IR N that shows how active each slot is at each patch. The final reconstruction y E RR Nx then given by taking a weighted average over the slots, that is, the reconstruction y: for patch is given by
K
y
Úix & Mi
Mi,s
softmax fottmmaxxxs.) (:..)
(5)
With top-k decoding, only take the & E K., most active slots into account for each patch i, as determined by the slot attention mask a E [0, 1
yi Σ Di U Mi,Ks
Mi,x
softmax 06.) (
K.; = topk (a;,
jEK'
NEK:
where topk(æ,, k)) arg (11.. Eier x; is the function that selects the indices of the highest values of the vector R". In practice, we can efficiently implement the decoding step by first broadcasting slots to patches and adding the positional encoding, then packing the top-k slots for each position together using a gather operation, directly resulting in reconstructions ý E TR and alpha masks Nxk.
C.4 IMPROVED HYPERPARAMETRRR
As discussed in Sec. 4.1 in the main paper, we found an improved set of hyperparameters that work well for finetuning the pre-trained ViT encoder. We split these into general hyperparameters
25

---

Published as a conference paper at ICLR 2025
(G-HPs), affecting all modules of the model, and encoder hyperparameters (E-HPs), only affecting the finetuning of the encoder (see also Table C.8). We ablate the effect of these groups of hyperparameters in Table in the main paper.
The general hyperparameter changes are as follows:
Increasing batch size from 64 to 128.
Decreasing base learning rate from 0.0004 to 0.0003.
Switching from an exponential decay learning rate schedule to cosine schedule.
Lowering gradient clipping from 1.0 to 0.1.
The hyperparameter for encoder finetuning are as follows:
Lowering the base learning rate for the encoder by factor of 0.5 from 0.0003 to 0.00015.
Introducing blockwise learning rate decay with decay rate of 0.85.
Adding weight decay of 0.01 to the encoder parameters in conjunction with the AdamW optimizer.
Note that these changes resulted from joined hyperparameter search over all individual hyperparam- eters and it is highly likely that (I) not all of these parameters changes necessary, and (2) an even better set of hyperparameters can found.
D METHODS & HYPERPARAMETERS
DINOSAUR (Seitzer et al., 2023) DINOSAUR introduced the idea of applying slot attention pre-trained encoder and training the model by reconstructing the features of this pre-trained encoder. This also forms the base of our proposed approach. In DINOSAUR, the encoder is kept fixed while the slot attention and the decoder modules are trainable. While the original paper considers two kinds of decoders (I) Transformer Decoder and (2) MLP decoder in this work we mainly compare against DINOSAUR with the MLP decoder. We consider two variants of DINOSAUR, using DINO (Caron et al., 2021) and DINOv2 (Oquab al., 2023) pre-trained backbones respectively. We list the hyperparameters used for DINOSAUR in Table C.8 (first column).
FT-DINOSAUR Our method is implemented upon DINOSAUR and thus shares low-level implemen- tation details. In Table C.8, list the hyperparameters for the following models mentioned Sec. and listed in Table 1: (1) DINOSAUR + Training from Random Init., (2) DINOSAUR + FT w/G-HP's, (3) AIR + w/G-HP's E-HPP's (4) DINOSAUR + FT, + Top-k, + High-Res. Finetuning. While the models listed in Table Table C.8 all use DINOV2 with the ViT-S/14 backbone, the same hyperparamters are applicable for models using ViT-B/16 and ViT-B/14 backbones as well. For training model, we a single A100 GPU per run. Each training run of the proposed finetuning approach requires 2-3 days of training.
SlotDiffusion (Wu et al., 2023b) SlotDiffusion utilizes a latent diffision model as the decoder. The specific variant of the SlotDiffusion model which consider here is the which uses pretrained DINO encoder (ViT-B/16) to encode the images similar to DINOSAUR. We use the pre-trained checkpoint released by the authors² for all the comparisons in this work.
SPOT (Kakogeorgiou et al., 2024) SPOT uses two-stage training procedure. In the first stage, DINOSAUR model is trained similar to Seitzer et al. (2023). In the second stage, a student-teacher setup employed, where the model trained model from the first stage acts as teacher and the student is new model. During this stage, the model is trained with two objectives: (1) feature reconstruction loss, where the targets come from the teacher, and (2) attention distillation loss, where the teachers attention masks from slot attention are distilled into the student. Moreover, SPOT uses a Transformer encoder as opposed to an MI.P decoder. Similar to SlotDiffusion, for SPOT,
Phttps://git thub. ctt.b..co///zii/e/ 61 6/81 ctDi ctoi ffusion
26

---

Published as a conference paper at ICLR 2025
Table C.8: Hyperparameters for the DINOSAUR and FT-DINOSAUR models displayed in Table 1. The second column (DINOSAUR +Enc. Train (random init.)) also lists hyperparameters for training with random encoder initialization, as discussed in Sec. 4.1. Results in Fig. 5 use the settings in the fourth column (DINOSAUR +FT w/G-HP's w/E-HP's). s). Results in Table and Fig. 7 use the settings in the last column, but with ViT-B/14 encoder. See also App. C.4 for concise description of the improved hyperparameters for finetuning (G-HP's and E-HP's).
Models
DINOSAUR
DINOSAUR
DINOSAUR +FT. w/G-HP'ss
DINOSAUR
DINOSAUR
+Enc. Train (random init.)
w/G-HP's
+Top-k
w/E-HP's
High-Res.
Training Steps
300k
300k
300k
300k
10k
Batch Size
64
128
128
128
Imago/Crop Size
224
224
224
518
Cropping Strategy
Random
Random
Random
Random
Random
Augmentations
Image Tokens
784/256
256
256
256
1369
LR Warmup Steps
10000
10000
10000
10000
333
Base LR (Total LR Base LR
0.0004
0.0003
0.0003
0.0003
0.0001
LR
v( BBuuch
Batch Size
61
Exp. Decay Half-Life
100k
100k
100k Cosine
100k
10k
Schedule
Exponential
Cosine
Cosine
Cosine
Blockwise LR
x
/
LR
Factor
(Total LR
0.5
x
0.5
0.5
Encoder LR
Fnodeer Factor ((( (Baahh Size
Base
Layerwise LR Decay Factor (1)
0.85
x
(LRager! = 7 LRuyer 11)
Weight Decay
0.01
x
Type
VIT-B/16/ VIT-S/14
ViT-S/14
ViT-S/14
VIT-S/14
VIT-S/14
Pre-training
DINO/DINOV2
DINOv2
DINOv2
DINOv2
Encoder
Patch Size
16/14
Feature Dim. Diaa
768/384
384
384
384
384
Gradient Norm Clipping
1.0
0.1
0.1
0.1
0.1
Type
VIT-B/16 VIT-S/14
ViT-S/14
ViT-S/14
VIT-S/14
VIT-S/14
Target Encoder
Pre-training
DINO/DINOV2
DINOv2
DINOv2
DINOv2
DINOv2
Slots
7
7
Iterations
3
3
3
3
Slot Attention
Slot Dim. Datnss
256
256
256
256
256
MLP Hidden Dim.
1024
1024
1024
1024
1024
Type
MI.P
MLP
MLP
MI.P
MI.P
Layers
4
4
4
4
Decoder
MLP Hidden Dim.
2048
2048
2048
2048
2048
Top-k
X
X
X
X
3
27

---

Published as a conference paper at ICLR 2025
Table E.9: Number of images per dataset and the number of slots used for training and evaluating on each dataset.
Dataset
Num. Images
Num. Slots
Coco 2017 train
118 287
7
Coco 2017 validation
000
7
ENTITY train
789
7
ENTITYSEG validation
499
7
PASCAL VOC 2012 train
10582
7
PASCAL Voc 2012 validation
1449
7
Movi-C train
633
Movi-C validation
420
Movi-E train
87
Movi-E validation
4176
SCANNET train
10000
6
SCANNET validation
2000
6
YCB train
10000
6
YCB validation
20000
6
CLEVRTEX train
40000
11
CLEVRTEX validation
5000
11
also we use pre-trained checkpoint released by the authors for the evaluations in this work. The pre-trained checkpoint uses a ViT-B/16 encoder initialized with DINO weights.
Segment Anything (Kirillov al., 2023) The Segment Anything model (SAM) a large founda-
tion model for object detection and segmentation trained supervised. It has three stages of training: (1) a manual stage, where the model trained using 120k images annotated with 4.3M masks obtained from human labelers; (2) semi-automatic stage, where the model is trained 180k annotated with 5.9M masks partly annotated by human labelers and partly annotated by itself; and (3) a fully automatic stage, where the model is trained on 1 11M images with 1.1B masks annotated by the model itself. We consider 2 variants of SAM: comp. (Comparable) and best. Note that SAM includes an loU prediction MLP which outputs estimated IoU for each predicted mask. For the comp. variant, we use the ViT-Base model considering the top K masks by predicted IoU, where the value of K is based on the optimal number of objects for each dataset as listed Table E.9. For the best variant, we use the ViT-Huge model keeping all masks above a IoU threshold т. We evaluated values for 7 E (0.9, 0.95, 0.99) and found that T 0.9 works best across all datasets.
For inference, we use single A100 GPU for each of the baselines and the proposed approach.
E DATASETS
This section gives detailed description of the datasets we use in this work. See also Table E,9 for an overview over the number of images per dataset.
Note that current object-centric models sensitive to the number of objects. As a concession that, we evaluate the models with the number-of-slowe parameter matching the expected complexity of the target dataset (mostly following prior work, see Table E.9). We leave future work remove this limitation of the models.
Coco (Lin et al., 2014) This dataset contains complex images containing real-world objects in their natural context. For training. we use the Coco 2017 dataset which consists of 118 287 images. For evaluation, we use 5 000 images from the validation sets. Similar to Seitzer et al. (2023), we use instance masks to evaluate object discovery. Additionally, we also add the task of panoptic segmentation to our evaluation suite for the Coco dataset, using the panoptic labeling provided by Kirillov et al. (2019). Panoptic segmentation combines the task of instance segmentation, which
Phttps://ght thub. http://tthhbb..oogggkkkkkk//
spppt
28

---

Published as a conference paper at ICLR 2025
requires the model to segment each object/forgrooundthing instance, and semantic segmentation, which requires the model to segement each background/stuff class. The metrics we use for measuring panoptic segmentation
panoptic ARI and panoptic quality (see App. F). Following Seitzer et al. (2023), we evaluate square center crops, where the input images are resized to 224 x 224 pixels, and the targets masks are resized to 320 x 320 pixels.
ENTITYSEG (Lu et al., 2023) This dataset consists of complex real world images spanning a diverse range of entities. In contrast Coco, ENTITY SEG is an open-world dataset and does have pre-defined set of object classes. It consists of a large number of high-resolution images (71.25% and 86.23% of the images are of high resolution with at least 2 000px for the width and 1 (000px for the height). Each image is annotated with high-quality fine-grained mask annotations. The version of the dataset utilized in this work consists of 31 789 images for training and 1 498 images for evaluation. We evaluate the instance segmentation masks for object discovery. As in CocO, we evaluate square center crops, where the input images are resized to 224 x 224 pixels, and the targets masks are resized to 320 x 320 pixels.
PASCAL VOC (Everingham et al., 2012) Similar to Seitzer et al. (2023), we use the "trainaug" variant of the PASCAL Voc dataset for training. It consists of a total of 10 582 images for training, where 464 are from the segmentation train set and 9 118 are from the SBD dataset (Hariharan et al., 2011). For evaluating object discovery, we use the official instance segmentation validation split with 449 images. Following Seitzer et al. (2023), we evaluate square center crops, where the input images are resized to 224 x 224 pixels, and the targets masks resized 320 x 320 pixels.
Movi-C and Movi-E (Greff' et al., 2022) The Movi datasets are synthetically generated video datasets consisting of multiple objects per video. Each video is generated by placing 3D scanned objects on real-world backgrounds. Movi-C contains up to 11 objects per video and MOVI-E contains up to 23 objects per video. Additionally, Movi-E also features the camera moving in random directions. For our case, we treat these datasets as image datasets. We sample 9 frames per video which yields a total of 87 633 training images for Movi-C and 87 741 images on Movi-E, For evaluation, use 4 200 frames for Movi-C and 4 176 frames for Movi-E from the validation sets in each case. We use a resolution of 128 x 128 for both input images and target masks.
SCANNET and YCB (Yang & Yang, 2022) These datasets consist of real-world objects black backgrounds and were originally introduced to test limitations of object-centric learning methods (Yang& & Yang, 2022). SCANNET (originally from Dai et al. (2017)) consists of objects that can be typically be found in indoor scenes (e.g. furniture) and YCB (originally from Calli et al. (2015)) consists of 21 different classes of everyday objects (e.g. food items, kitchen items, tools, etc.). Each of these dataset consist of 10000 training images and 2 000 evaluation images. Both datasets consist of 2-6 objects per scene. We use a resolution of 128 128 for both input images and target masks.
CLEVRTEX (Karazija et al., 2021) This a synthetically constructed dataset where each scene consists of 3-10 simple geometric 3D shapes arranged in background sampled from catalogue of 60 different materials. The materials of the objects are also sampled from the same catalogue. This dataset contains 40 000 images for training and 10 for validation and test each. We use the 5 000 images from the validation set for our evaluation. CLEVRTEX also offers various OOD splits which utilize materials not seen during training. We do not these splits; for our zero-shot generalization evaluation, we can directly use the main split since it usually is not a part of the training set we use to train the object-centric model. We use resolution of 240 240 for both input images and target masks.
F METRICS
F.1 OBJECT DISCOVERY
FG-ARI The adjusted rand index (ARI) measures the similarity between two clusterings (Hubert & Arabic, 1985). We use the instance/object masks as the targets. We only compute this metric for pixels in the foreground (hence, FG-ARI). Unlabeled pixels are treated as background.
29

---

Published as a conference paper at ICLR 2025
mBO To compute the mBO (Pont-Tused et al., 2017), each predicted mask is assigned to the ground truth mask with highest overlap in terms of IoU. The mBO is computed as the average IoU of these mask pairs.
Panoptic ARI Panoptic ARI is computed as ARI, but uses panoptic mask annotations as ground truth targets. Panoptic masks (Kirillov al., 2019) provide more detailed mask annotations for an image by assigning different mask for separate instances of the same object ("things") and also segmenting background regions ("stuff""). We only compute the Panoptic ARI for those images which have at least two masks.
Panoptic Quality The Panoptic Quality (PQ) (Kirillov et al., 2019) is computed by first assigning each predicted mask to the ground truth mask with the highest overlap in terms of IoU, removing all matches that do not have an IoU overlap of at least 0.5: this results in a unique matching (Kirillov al., 2019). These mask pairs form the set of true positives (TP). Ground truth masks that were not assigned a predicted mask form the set of false negatives (FN). Similarly, predicted masks that were not assigned to a ground truth mask form the set of false positives (FP). Predicted masks that have an IoU overlap of more than 0.5 with pixels labeled as "void" or "crowd" are removed from the set of false positives. The panoptic quality is then computed as:
Σ EipglETP [oU(p.)) 9) PQ
TPP + 0.5 FP + FN
F.2 DOWNSTREAM PROPERTY PREDICTION
For downstream property prediction, we closely follow the protocol of Dittadi et al. (2022) and Seitzer et al. (2023). In particular, we train linear probes on top of the slots, training them using categorical cross entropy for discrete properties and MSE for continuous properties. Predicted properties for the set of slots are matched to the target set of properties using the Hungarian method, minimizing the total loss of the matching. For discrete properties, we report classification accuracy; for continuous properties, we report the -cooce We report metrics on the the same test sets used for object discovery.
We designate a portion of the training set as a validation set to measure overfitting; for this purpose, we sample a number of images equal to the size of the test set from the training set. We train for 15 000 steps, and evaluate the model with the lowest validation loss. Images without any objects (due to cropping) are filtered out. Following Seitzer et al. (2023), for each image, we only keep the N largest objects, where N equals the number of slots used on that dataset. This way, each object can in principle be matched a slot.
As properties, we use:
Location: x- and y-coordinate of the object, using the center-of-mass of the object mask. We report the average Re-score for and y-dimension.
Category: we use the class label with which the objects are annotated on most datasets; on CLEVRTEX, we use the object shape as the category. The used versions of the SCANNET and YCB datasets do not contain any class annotation; as such, we do not report category prediction on these datasets. Note that the number of classes differs per dataset and thus classification results between datasets are not directly comparable. In particular, CLEVRTEX has 4 categories; Movi-C and Movi-E have 17 categories; ENTITYSEG has 256 categories.
We report the property prediction for each individual dataset in Table A.7.
G EXAMPLES
In this section, we show example predictions for DINOSAUR, SPOT, Slot Diffusion, FT-DINOSAUR, and SAM, where all methods besides SAM were trained on the CoCO dataset. FT-DINOSAUR uses a ViT-B/14 encoder with top-k and hi-res adaptation, i.e. the model evaluated in Table 2 and Fig. 7.
Fig. G.6: in-distribution predictions Coco.
30

---

Published as a conference paper at ICLR 2025
Fig. G.7: zero-shot predictions on ENTITYSEG.
Fig. G.8: transfer predictions on PASCAL Voc.
Fig. G.9; zero-shot predictions on CLEVRTEX.
Fig. G. 10: zero-shot predictions on MOvi-C.
Fig. G.11: zero-shot predictions on Movi-E.
Fig. G. zero-shot predictions SCANNET.
31

---

Published as a conference paper at ICLR 2025
Image
DINOSAUR
SPOT
SlotDiffusion
FI-DINOSAUR
SAM
Figure G.6: In-distribution examples on Coco.
32

---

Published as a conference paper at ICLR 2025
Image
DINOSAUR
SPOT
SlotDiffusion
FT- TT-DINOSAUR
SAM
Figure G.7: Zero-shot examples ENTITYŠEG.
33

---

Published as a conference paper at ICLR 2025
Image
DINOSAUR
SPOT
SlotDiffusion
FT- T--DINOSAUR
SAM
Sterlingew
Steelingeu R
Storlng.eu
Steding eu
FEB
Figure G.8: Transfer examples PASCAL VOC.
34

---

Published as a conference paper at ICLR 2025
Image
DINOSAUR
SPOT
SlotDiffusion
FT- TT-DINOSAUR
SAM
FA EA M
Figure G.9: Zero-shot examples CLEVRTEX.
35

---

Published as a conference paper at ICLR 2025
Image
DINOSAUR
SPOT
SlotDiffusion
FT- DINOSAUUR
SAM
Figure G.10: Zero-shot examples on Movi-C.
Image
DINOSAUR
SPOT
SlotDiffusion
FT- DINOSAUR
SAM
Figure G.11: Zero-shot examples Movi-E.
36

---

Published as a conference paper at ICLR 2025
Image
DINOSAUR
SPOT
SlotDiffusion
FT- DI-DONSUUU
SAM
Figure G.12; Zero-shot examples on SCANNET.
37


Reviews:
[{"review_id": "9HfhJufQWo", "reviewer": "Reviewer_SC2i", "summary": "This paper studies the zero-shot transfer capabilities of object-centric representation learning models and proposes improvements to enhance their generalization. The key contributions are:\n- A benchmark comprising 7 diverse datasets to evaluate zero-shot transfer of object-centric models.\n- A novel finetuning approach that adapts pre-trained vision encoders for object discovery.\n- The method proposed in this paper achieves state-of-the-art results for object discovery tasks on both in-distribution and out-of-distribution scenarios.", "questions": "1. I'd like to discuss the \"blockwise exponentially decaying learning rates\". According to the ablation studies (Tab. 1),  blockwise learning rates don't bring significant improvement. The authors propose it because \"the encoder would initially drift away from its pre-trained initialization, likely induced by the noisy gradients from the randomly initialized slot attention module\". Since this phenomenon may be caused by the randomness of slot initialization, why not consider changing the initialization method of slots? Why not initialize slots with learnable queries like BO-QSA [1], OSRT [2], and SPOT [3], which have been proven to be effective? Instead, this paper introduces \"blockwise exponentially decaying learning rates\".  If initializing slots with learnable queries does not solve this problem, the problem may not be caused by random slot initialization. \n\n2. I'm interested in why these models don't scale well with dataset size, especially on real-world datasets such as COCO. What might help to improve scaling behavior? If the authors find out in their experiments what factors help enhance scaling behavior, I believe it is important to highlight them to provide more insight into the progress in this field, which can also increase the impact of this work.\n\n[1] Improving Object-centric Learning With Query Optimization.\n[2] OSRT: Object Scene Representation Transformer.\n[3] SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Technical Innovation:\n\t- First systematic study of zero-shot transfer in object-centric learning\n\t- Novel finetuning strategy that successfully adapts pre-trained encoders\n- Adequate experimentation and analysis：\n\t- Comprehensive empirical evaluation across multiple datasets and metrics\n\t- Thorough ablation studies validating each component\n- Clear presentation:\n\t- Well-structured and clearly written\n\t- Comprehensive appendix with implementation details", "weaknesses": "- The proposed method lacks novel and essential insights.  The fine-tuning strategy, high-resolution adaptation, and top-k decoding are engineering improvements that come easily to mind. \n\n- The paper shows that current models don't scale well with data size (Fig 2b), especially for real-world data, but doesn't propose any analyses or solutions for this limitation.\n\n- These datasets, while diverse, are still relatively small-scale for pre-training compared to modern vision datasets. The ScanNet and YCB datasets used in [1] provide images without background. Only COCO, PASCAL, and EntitySeg are more consistent with the natural image distribution. This may be one reason why no evident scaling law has been observed.\n\n[1] Promising or Elusive? Unsupervised Object Segmentation from Realworld Single Images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vyW07fLasl", "reviewer": "Reviewer_ixZS", "summary": "This paper discusses the transferability of self-supervised models, such as DINO, and the factors about the transferability performance. Then, the authors further explore the finetuning strategy to adapt the model to conduct object discovery task.", "questions": "See Weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "+ The topic about object discovery task is interesting.", "weaknesses": "The writing is confusing and difficult to understand. Not only the expression of writing is confusing, but also many important clarifications about technique can not be found in the paper. We list these issues as follows.\n+ The evaluation of zero-shot transfer. In the main body of paper, the authors only list the datasets and metrics used for evaluation, but don't mention why and how.\n+ For object centric finetuning, the authors don't mention how to conduct slot attention and top-k mlp decoder (the structure details) and why use these module. What is the loss function? Where do the output images come from in Figure 3? Are the output masked images just the output of DINOv2?\nIn summary, I am inclined to reject this submission in the current presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4uRY8G0Kth", "reviewer": "Reviewer_dN2g", "summary": "This paper studies the zero-shot performance of object-centric representation learning. The authors first combine 7 existing datasets to establish a comprehensive testbed, enabling them to evaluate the transferability of three recently proposed object-centric learning models. Based on the experimental findings, they propose finetuning the pre-trained encoders on the COCO dataset to enhance zero-shot performance on their testbed.", "questions": "Please see the weaknesses section. \n\nA small question: Would it be possible to provide a more detailed comparison of the similarities between COCO and the 7 datasets in testbed?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-motivated, beginning with a clear question and proposing a testbed to evaluate the performance of current methods. Based on the evaluation results, the authors propose method to further enhance the model.\n\n- The proposed finetuning method is straightforward yet effectively addresses model collapse problem", "weaknesses": "1. I would consider this as an analysis paper. However, many of the conclusions appear somewhat obvious. For instance, SAM already demonstrates strong zero-shot transfer capabilities in object-centric models, even extending well to domains like medical imaging. Additionally, it’s widely understood that training on complex natural datasets is crucial for zero-shot transfer performance, and using real data is key to enhancing this ability.\nThis paper would be more interesting with additional and more detailed experiments and insights provided by the testbed. For example, is it possible to identify some common failure cases and explore possible reasons behind these failures? I guess that could offer more insights.\n2. The paper proposes to finetune on an additional dataset (COCO) to enhance zero-shot performance. However, it doesn't have any specific design for zero-shot task. Additionally, task-specific finetuning on a general pretrained model typically leads to performance improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lGCTNSRUA9", "reviewer": "Reviewer_neGY", "summary": "This paper introduces a benchmark to evaluate the zero-shot generalizability of object-centric learning methods. The benchmark comprises seven synthetic and real-world multi-object datasets, allowing comprehensive testing across different scenarios. Additionally, the paper demonstrates that training on these diverse real-world images significantly enhances the transferability of existing methods to unseen contexts. Moreover, the authors propose an object-centric fine-tuning strategy, building on DINOSAUR’s pre-trained vision encoder, which achieves state-of-the-art performance in unsupervised object discovery with robust zero-shot transfer abilities.", "questions": "See the weaknesses section. Besides, I strongly encourage the authors to conduct experiments without the DINO encoder. Since DINO is pre-trained on ImageNet, which is significantly larger than those used in this paper’s experiments, it's hard to accurately evaluate the model's transferability when using DINO.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses an important topic of exploring zero-shot object-centric learning OOD settings.\n- The fine-tuning approach of DINOSAUR improves object-discovery performance and closes the gap between zero-shot and supervised learning.\n- Extensive experiments are conducted across various datasets, providing insights into the zero-shot generalizability and scalability of object-centric models.", "weaknesses": "My major concern on this paper is regarding the evaluation design:\n- The evaluation relies on pre-trained DINO features, which may already possess OOD capabilities due to large-scale pre-training. Since object-centric learning methods using pre-trained encoders only need to focus on feature grouping, these methods can easily transfer learned grouping techniques to new datasets supported by the powerful pre-trained features. As a result, it is difficult to ascertain whether the observed zero-shot generalization stems from the object-centric method itself or DINO’s inherent robustness.\n- The paper uses object segmentation as the primary evaluation metric, which might not fully capture the quality of the learned representations. The inclusion of additional downstream tasks, such as object classification or property prediction, would provide a more comprehensive evaluation of the generalizability.\n\nFurthermore, the method proposed in this paper offers limited novelty: the fine-tuning process relies on well-known techniques and does not introduce significant new insights beyond improved hyperparameter tuning.\n\nLastly, the claim that “training on diverse real-world images improves transferability to unseen scenarios” adds little to the current knowledge, as it is already widely accepted that increased data quantity and diversity generally lead to better model performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)