Object-Centric Pretraining via Target Encoder Bootstrapping
Authors: Nikola Đukić
Tim Lebailly
Tinne Tuytelaars
Venue: ICLR 2025 Poster (2025)
Keywords: Object-centric learning
bootstrapping
self-supervised pretraining
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Object-centric representation learning has recently been successfully applied to real-world datasets. This success can be attributed to pretrained non-object-centric foundation models, whose features serve as reconstruction targets for slot attention. However, targets must remain frozen throughout the training, which sets an upper bound on the performance object-centric models can attain. Attempts to update the target encoder by bootstrapping result in large performance drops, which can be attributed to its lack of object-centric inductive biases, causing the object-centric model’s encoder to drift away from representations useful as reconstruction targets. To address these limitations, we propose **O**bject-**CE**ntric Pretraining by Target Encoder **BO**otstrapping, a self-distillation setup for training object-centric models from scratch, on real-world data, for the first time ever. In OCEBO, the target encoder is updated as an exponential moving average of the object-centric model, thus explicitly being enriched with object-centric inductive biases introduced by slot attention while removing the upper bound on performance present in other models. We mitigate the slot collapse caused by random initialization of the target encoder by introducing a novel cross-view patch filtering approach that limits the supervision to sufficiently informative patches. When pretrained on 241k images from COCO, OCEBO achieves unsupervised object discovery performance comparable to that of object-centric models with frozen non-object-centric target encoders pretrained on hundreds of millions of images. The code and pretrained models are publicly available at https://github.com/djukicn/ocebo.

Published as a conference paper at ICLR 2025
OBIECT-CENTRIC PRETRAINING VIA TARGET EN- CODER BOOTSTRAPPING
Nikola Dukić
Tim Lebailly
Tinne Tuytelarrs
KU Leuven
(ni (nikola.jjukie, tim.lebailly. tinne.tuytel aars) @kuleuven. be
ABSTRACT
Object-centric representation learning has recently been successfully applied to real-world datasets. This success can be attributed pretrained non-object- centric foundation models, whose features serve reconstruction targets for slot attention. However, targets must remain frozen throughout the training, which sets upper bound on the performance object-centric models can attain. At- tempts to update the target encoder by bootstrapping result in large performance drops, which can be attributed to its lack of object-centric inductive biases, caus- ing the object-centric model's encoder to drift away from representations use- ful reconstruction targets. To address these limitations, propose Object- CEntric Pretraining by Target Encoder BOotstrapping, a self-distillation setup for training object-centric models from scratch, real-world data, for the first time ever. In OCEBO, the target encoder is updated as an exponential moving average the object-centric model, thus explicitly being enriched with object- centric inductive biases introduced by slot attention while removing the upper bound on performance present in other models. We mitigate the slot collapse caused by random initialization of the target encoder by introducing a novel cross- view patch filtering approach that limits the supervision to sufficiently informa- tive patches. When pretrained on 241k images from COCO, OCEBO achieves unsupervised object discovery performance comparable to that of object-centric models with frozen non-object-centric target encoders pretrained on hundreds of millions of images. The code and pretrained models are publicly available at https:/ / thub. com jukkiiiin /n/ec////oao cccbol
INTRODUCTION
In recent years, large-scale foundation models have become ubiquitous across many domains of deep learning. This is mainly thanks to the development of self-supervised learning (SSL) techniques that enable pretraining
massive amounts of unlabeled data. In computer vision, several families of
SSL techniques have emerged, focusing on (i) imposing cross-view invariance at the global (Grill
al.
2020: Caron et al. 2021 patch (Zhou al. 2022 Sttegüüller et al. 2023) cluster (Wen
et al.
2022 Lebailly et al. 20044) level, or (ii) reconstruction in the pixel (He et al. 2021) or latent
2022 Assran al.) (202)). On the other hand, research in cognitive psychol- ogy (Benjamin Peters 2021) suggests that human visual perception reasons about visual inputs
space (Zhou
by decomposing scenes into of objects with corresponding representations. In the past years, efforts have been made to mimic this behavior (Locatello et al. 2020 Seitzer et al. 2022; Kakoge- forgiou et al. 2024 Didolkar et al., 2024) and move past global and patch representations towards object-centric representations. Object-centric models are not only more biologically plausible (Ben- jamin Peters 202T), , but can provably achieve compositional generalization (Wiedemer et al.] (024) and have already proven useful several downstream tasks, such robotic control (Zadaianchuk ct al., 2021 Haramati et al. 2024), compositional generation (Singh et al.) 2022 Wu et al. 2023] Jiang al. 2023 Sajjadi al. 2002),, and visual question answering (Xu al. 2024). While in- credibly promising, object-centric models have not yet been successfully pretrained on a large-scale real-world dataset.
slot attention-baue object-centric model consists of an image encoder, slot attention encoder and slot decoder. Initially, these trained end-to-end with reconstruction objective in the pixel-

---

Published as a conference paper at ICLR 2025
space Locatello et al. 2020), resulting in good scene decomposition synthetic datasets but slot collapse on real-world data. Seitzer et al. (2022) introduce a reconstruction objective in the latent space of pretrained encoder (c.g., DINO (Caron et al. 202I), DINOv2 (Oquab et al. 2023), etc.), showing that the presence of informative reconstruction targets is crucial for avoiding slot collapse. As this encoder provides reconstruction targets, we refer to it target encoder. Moreover, [Seitzer iet al.] (2022) initialize the encoder of their object-centric model with the same pretrained encoder and keep it frozen alongside the target encoder, effectively training only the slot attention encoder and decoder.
While the framework with the frozen target encoder achieves unprecedented performance on real- world datasets (Seitzer et 2022), it has a significant limitation. Namely, the target encoder cannot be updated, meaning that there exists an upper bound on what the object-centric model can learn from it. In fact, despite the target encoder being pretrained on millions or even hundreds of millions of images (Caron et al.) 202T| Oquab et al. 2023), the object-centric model can efficiently consume all its knowledge and reach the upper limit. This is demonstrated in experiment by Didolkar ctt al. (2024). where the performance plateaus with 166 images from MS COCO (Lin et al.] 2015) and does not further increase, even with an order of magnitude more data. This clashes with the scalability trends observed in other suple--upe) representation learning families. A question arises wrises-how to update the larget encoder, thus removing the performance upper limil and enabling large-scale pretraining of object-centric models?
Kakogeorgiou et al. (2024) show for the first time that unfreezing a few final layers of the object- centric model's encoder is possible and further improves the performance. Didolkar et al. (2024) further extend this finding by fine-tuning the full encoder. A straightforward approach would be updating the target encoder as an exponential moving average (EMA) of the object-centric model's encoder. However, this has been shown to result in huge performance drops (Didolkar et al.) (024). We hypothesize that the reason for this is the lack of object-centric inductive biases in the target encoder. Most often trained for cross-view consistency, SSL. models organize their representations in terms of semanties, rather than object instances. For example, semantically similar parts of different objects will be close in the representation space and farther from less similar parts of the same object. When the object-centric model tries to reconstruct such features, its inductive bias focuses on assigning these semantically similar regions to different objects (i.e., instances), thus updating its encoder to focus slightly less semanties and more on positions in the image. EMA update of the target encoder then results in informative knowledge slowly leaking at the cost of more positional information, which ultimately degrades the performance.
In this work, we argue that a way to overcome the current limitations of object-centric learning and unlock its full potential is by large-scale pretraining from scratch. We pose object-centric learning as a self-distillation bootstrapping problem (Caron et al. 2021). in which the object-centric model distilled from the target encoder that is in turn updated as an EMA of the object-centric model's encoder. We propose the object-centric self-distillation loss, a patch-level loss that acts as a re- construction objective. We apply augmentation invariance. To prevent slot collapse stemming from the lack of informative reconstruc- tion targets due to random initialization of the target encoder, we propose a cross-view patch filtering procedure that uses eross-view correspondences to determine which patches to reconstruct at par- ticular training stage. The resulting model, named OCEBO allows pretraining object-centte models on real-world datasets for the first time. The EMA updates not only remove the upper bound from object-centric models but explicitly introduce object-centric inductive biases into the target encoder. By training from scratch, we allow the model learn informative instance- based features rather than trying to reorganize semantic features from pretrained non non-object-centrr encoder. When pre- trained on ~ 88 or 241k images from MS COCO (Lin et al. 2015). not only does OCEBO avoid slot collapse. but also achieves performance comparable to that of object-centric models with DI- NOv2 (Oquab et al.) 2023) target encoder pretrained on 142M images, proving the object-centric inductive biases highly beneficial for the target encoder. Moreover, OCEBO demonstrates scalabil- ity well beyond a few thousand training images, unlike other object-centric models. Altogether, we believe this work paves a way towards large-scale pretraining of object-centric foundation models.
cross-vies consistency fashion to enrich slot attention with
Slot collapse refers to scenario where slots learn to attach to positionally coherent blocks rather than actual objects.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Cross-view consistency has emerged as a predominant self-supervised learning (SSL) task in computer vision. In this framework, views of input image, obtained through data augmentations, are used to enforce cross-view representation consistency while avoiding rep- resentation collapse. While some works avoid collapse via the use of negative samples (Chen al. 2020 He et al.] 2020), self- oif-distillation approaches rely on positive samples only, in combination with mechanisms such as branch asymmetry (Grill et al. (020), stop-gradients (Chen He 2021 Caron et al., 2021), momentum encoders (Grill et al. 2020] Caron et al. [02T], etc. Moreover, self- distillation methods can enforce consistency different levels of granularity.
Self-distillation.
global or image- level self--upervision (Grill et al. 2020 Caron et al. 2021 consistency is enforced between global representations that capture the whole input image. In patch-level or local self-supervision (Zhou et al.) 2022 Stegmüller et al. 2023), consistency is enforced between pairs of corresponding patches in both views. Finally, consistency has been enforced between semantically coherent groups of patches (Wen al., [202 Lebailly et al. 2024), i.e., clusters (also referred to cluster-level or object-level seff-supervision).
Object-centric representation learning. Image-, patch- and cluster-level self-distillation methods have achieved impressive performance across computer vision. In parallel, attempts at object-centrie representation learning have been made. An object-centric model decomposes scene into set of objects with corresponding representations, which is in line with theories about underlying mech- anisms of human visual perception (Benjamin Peters 202I). Unlike object-level self-distillation methods that rely on soft clustering (Wen et al. 2022; Lebailly et al., 2024), which can often group semantically similar parts of different instances, object-centric models explicitly use inductive biases that enforce instance-level scene decomposition. Slot attention (Locatello et al.) 2020) has emerged as a particularly successful inductive bias. It introduces an attention mechanism that promotes com- petition among slots for different parts of input. In the original work (Locatello al. 2020), an autoencoding model consisting of an image encoder, slot encoder and slot decoder trained end-to- end has been proposed. It achieved impressive performance but resulted in slot collapse on real-world datasets. [Seitzer et al. (2022) replaced the pixel space reconstruction objective with reconstruction of features produced by frozen pretrained non-object- centric SSL model, enabling the training of object-centric models on real-world data. Subsequent works focused on different decoding strategies (Kakogeorgiou al., 2024) or fine-tuning the en- objectives Wu al. (2023); Diang al.) (2023) have been proposed as an alternative to reconstruction based ob- jective. However, these still rely on pretrained SSL models. To the best of our knowledge, direct pretraining of object-centric models on real-world datasets has not been achieved yet.
well-structured synthetic datasets
coder while keeping the target encoder frozen (Didolkar et al. 20024) Diffusion-
Note that SlotCon (Wen et al., 2022) also introduces the concept of slots. However, these serve as learnable queries whose assignment to patch representations of both views is used as a supervision signal. In this context, larger number of slots corresponding to the number of object types in the training dataset is initialized, such that each slot Iearns to represent particular object type. In scenes with multiple instances of same object type, SlotCon would not produce instance, but rather a single slot attached to patches of all instances. This is fundamentally different from OCEBO and other object-centric works, where instance-level decomposition is achieved.
representation per
METHOD
We provide the necessary background in Sec- tion 3.1 continuing with the training objective in Section 3.2 cross-view patch filtering approach in Section 3.3 and the optional mask sharpening stage in Section 3.4
An overview of OCEBO is provided in Figure [
3.1 PRELIMINARIES
Image views. Let œ E hxxuxx be an image from the training dataset. We obtain two views I1, 22 E khx by applying two randomly sampled sets of data augmentations to the original image x. When sampling augmentations, define the inverse augmentation function i invaug 4 xxxx + wxxd such that invaug(#1) and i (x2) cut the overlapping regions from views 21, 222 and interpolate them back to size
ensure that there exists some overlap between œ; and x22 We
3

---

Published as a conference paper at ICLR 2025
Encoder
Slot encoder and decoder
invaug
[CLS] a
M())
(a()
EMA
L
global
(..()
21,2
2,,2
Au(-)
encoder ft
Target
[CLS]
invang
vrews
patch fittering
[CLS]
Z1,1
Figure 1: Overview of OCEBO: View æi is processed by the object-centric model's encoder (top branch), producing global and patch representations 2 and 21, respectively. Patch representations are sent through the slot attention encoder and decoder, where the latter outputs a reconstruction of the input patch representations separately and produces their global and patch representations εt,1, 2t,1, Σt and ž4.2, respectively. Patch representations žt,1 and ž1,2 are used by the cross cross-view patch filtering approach to infer infor- mative target patches and produce the mask m. The inverse augmentation operation (invaug) is applied to the target features of z2 and reconstructions of æ to make them correspond to the over- lapping region (purple part of the image) before combining them with the mask m and applying the object-centric loss Loe- Global loss Lgiobal is applied to global representations ě1 and 21,2-
Target encoder (bottom branch) processes both views Iı and z2
h x w x e (see Wen et al. with a hat, e.g., invaug (+) Lt.
2022 for more details). We denote outputs of the invaug operation
Object-centric model. The encoder ʃ of our object-centric model is Vision Transformer
(ViT) (Dosovitskiy et al. 2021). For a view Tı, I outputs global image representation (i.e., the [CLS] token) E Rdtt and patch representations ø1 Nxd where d is the embedding dimension, p is the patch size, and N = h/p w/pis is the number patches. Patch representations 21 then processed by slot attention encoder, which maps them into dy-dimensiioal slots. These are finally decoded by a slot decoder, resulting in reconstruction qı E Nxd the original input patch representations, Similarly, the inverse augmentation operation for representations. For instance invaug(4) ) cuts the features of qı according to the corresponding overlapping region of æ; and 32 and interpolates them back to the original size of Applying this operation to representations ensures they represent only the overlapping region and are aligned.
define 22, #2 and q2 for the view Ø2- We also the define
Target encoder. Target encoder fi has the identical architecture as the encoder f of the object- centric model. Similarly to f. fo outputs global and patch representations ε,,1, 2,,, ž1,1 and 2,2- Initially, the parameters fe are the same as the parameters of f. After each update the object- centric model, the parameters of fe are modified as an exponential moving average (EMA) of the parameters of
3.2 OBJECT-CENTRIC TRAINING OBJECTIVE
Our training framework is formulated as a self-distillation bootstrapping problem (Caron et al. 2021), where the object-centric model can be viewed as the student and the target encoder as the teacher. To this end, we introduce projection heads h and h,, where ht is updated as an EMA h. We define the object-centric self-distillation loss
4

---

Published as a conference paper at ICLR 2025
N Σ
oop
H(P.I.
Σ H(P.,,
where Ha, b) Σ3 -a log by and Pt.1, Pr.,, Pt, are sharpened (and centered) L-dimensional
distributions obtained from outputs of the target encoder and the object-centric model subsequently mapped by projection heads, i.e.,
P(1.)) so softax (h(q(1,2))/T)
(t((1.] = softmax ((he(z - cx) /Te)
with temperature parameters T and controlling the scaling and controlling the centering.
Note that in most self-distillation scenarios, student and teacher have identical architectures. To avoid a trivial solution, cross-entropy is applied between features of two different views rather than within the same view as in equation [ In our case, this is possible because the target encoder does not have the slot attention encoder and decoder. Nonetheless, by imposing Coep across views, enforce augmentation invariance, which improves generalization the invaug operation to ensure that apply the loss only in the intersection of views æι and 22. Equations [ and 2 then become
the object-centric model. We
N
Lon
Σ
H(P.I.P2)
Σ H(P...P))
(3)
2
and
P(12) = sof softmax woe.ftmmxinnaaantt (invaug (h(()1,2)) /T)
(4)
PL,122) = so fmax (eftmax(iivaaa) ((invaug (h: (h+(21,112))) ci) /m))
Similarly to other patch-level self-distillation works (Zou et al. 2022 Stegmüller et al. 2023
Lebailly al., 2024), we apply additional cross-view global loss introduced by Caron et al. (2021) to ensure the training stability and improve the quality of representations. We define
N
N H p2) + Σ H(p...P1))
gioobal
Σ
(5)
2
im
where
P(1,2 softmax (k(2(1.2))/7)
(6)
PI,(1.2) = softmax (h+(2,,(12)) 21) /T).
The final training objective is f = Nachoc + Aglobal Loglobal.
3.3 CROSS-VIEW PATCH FILTERING
Previous works (Seizerr et al. 2022: Didolkar et al. 2024) have shown that avoiding slot collapse requires reconstruction targets good quality. However, the features from our randomly initialized target encoder do not satisfy this. We propose the cross-view patch filtering condition as a proxy for the quality target encoder features and apply the object-centric loss only to the patches that satisfy it. Intuitively, if take representation of particular patch in one view and compute its distance to all patches of the other view, its nearest neighbor should be the exact same patch in the other view. If the target encoder's representations do not capture this, we do not them in Loc. We allow some slack in the target encoder and consider nearest neighbors instead of just one. More formally, define binary mask {0, 1). whose i-th entry is defined
5

---

Published as a conference paper at ICLR 2025
m: [ if e x 1 A (22( 21,1)
otherwise,
where n 21,2); denotes indices of k nearest neighbors of the i-th patch of ε1, in 2,2 Note that _ = invaug and εt2 = (2,,
Incorporating the mask m into equation yields
N m. m. m.H((p..PPP)
Lox
Σ
m, H (p.1,. P2)
+
(8)
In the initial epochs training, the entries in m are mostly and they keep increasing throughout the training. This allows the model to learn informative features and prevents from collapsing (see Section 4.2
3.4 MASK SHARPENING STAGE
Due to a constantly changing target encoder, we sometimes observe lack of clear boundaries in the masks produced by OCEBO. We mitigate this by introducing optional shorter mask sharp- ening stage, in which we keep training the object-centric model with frozen target and an & reconstruction loss instead of the self-distillation loss. This can be viewed procedure similar to fine-tuning of FT-DINOSAUR (Didolkar et al.] 2024). but with a specialized target encoder en- riched with object-centric inductive biases instead of the non-object-centric DINOv2 target encoder. After the sharpening step, masks exhibit clearer boundaries, resulting in improved unsupervised object-discovery performance (see Section 4.2 for details).
EXPERIMENTS
We start by outlining the implementation details (training datasets, evaluation protocols, model chitecture and the training setup) of OCEBO in Section 4.1 In Section 4.2 we demonstrate that OCEBO can be pretrained from scratch on real-world data without slot collapse. We justify the design choices and demonstrate data scalability, further discussing the requirements for suitable pretraining datasets. Finally, we put the performance state-of-the-ar object-centric approaches that rely on non-object-centrie target encoders pretrained on orders of magnitude more data in Section 4.3 In addition, in Appendix A we demonstrate that OCEBO also produces high-quality patch representations, which is an important step towards ver- satile backbones capable of performing well in both object- and patch-level downstream tasks.
OCEBO in context by comparing to
4.1 IMPLEMENTATION DETAILS
Training data. OCEBO is trained MS COCO (Lin et al. 2015), the most common real-world dataset in the object-centric literature. We use the train2017 COCO split with approximately 118k images. Additionally, we construct a larger dataset of ~41k images named COCO+ by com- bining the train2077 and unl abe Aabeled200 017 splits. We follow the standard data augmentation procedure with random crops resized 224 224, horizontal flips, solarization, blurring and color jitter (Caron et al. 2021).
Evaluation protocols. Current trends object-centric learning define in-distribution unsuper- vised object discovery as the go-to evaluation task for object-centric models (Seitzer et al. 2022 Kakogeorgiou et al., (024).. This means that a model is trained on training split of a dataset and evaluated on the test split of the very same dataset. We believe this to be suboptimal it does not agree with other tasks in computer vision. Moreover, OCEBO is trained from scratch and pretrain- ing it on small synthetic dataset is not possibility. We believe the recently proposed zero-shot unsupervised object discovery (Didolkar et (024) to a more natural and fair way of evaluating. In this context, the models are trained COCO and evaluated on natural and synthetic datasets
MOVi-C and MOVI-E (Greff et al. 2022). ScanNet (Dai et al.] (017). YCB (Calli et al.) 2015). ClevrTex (Karazija al. 2021 Pascal VOC (Everingham al. 2010 and EntitySeg (Qi et al.

---

Published as a conference paper at ICLR 2025
For brevity, we report results on two natural and two synthetic datasets MOVi-C, MOVi-E, Pascal VOC and EntitySeg. We find the conclusions on the remaining three synthetic datasets iden- tical. We use validation splits of each dataset and 11, 24, 7 and 7 slots for MOVi-C, MOVi-E, Pascal VOC and EntitySeg, respectively. We report the foreground-Adjusted Rand Index (FG-ARI) and mean Best Overlap (mBO) as evaluation metrics. ARI treats ground truth and predicted masks as clustering assignments and measures their similarity. FG-ARI does the same, albeit on foreground pixels only. The mBO metric assigns to each ground truth mask predicted mask with the highest overlap and reports the mean Intersection over Union (IoU) between the assignments.
2023
Architecture. Both f and fe follow the VIT-S/16 (Dosovitskiy al. (02I) architecture. The slot attention encoder operates on = 7 slots during training and consists of layers with d, = 256. Each layer consists of slot attention (Locatello et al. 2020) with a single attention head, a Gated Recurrent Unit and a 2-layer MIP with hidden dimension of 1024. Slot decoder is a 3-layer MLP with the hidden dimension of 2048. The projection heads are identical to those of DINO (Caron et al. 202I with the exception of setting L = 8192 instead of the original 65536. Compared to the DINO head, ours projects every patch rather than just the global representations and we find that the gain in performance does not justify the computational cost.
Optimization. We train the model for 300 epochs with an additional mask sharpening stage of 100 epochs. As in DINO, target encoder updates are performed with momentum following cosine schedule between 0.996 and 1. Scaling temperatures are r = 0.1 and Tę 0.07, with the latter being linearly increased from the initial 0.04 during 30-epoch warmup stage. Learning rate is linearly ramped up to the base value of 0.0003 during the first 10 epochs and decayed following a cosine schedule. If not mentioned, a hyperparameter has the same value as in DINO. Finally, we set Aoc
Aglobal 1.
4.2 ANALYSIS
Table 1: FG-ARI and mBO for one synthetic (MOVi-E) and one real-world (EntitySeg) dataset attained by different variants OCEBO. The first row shows the base OCEBO model trained on COCO, while the subsequent rows represent modifications. Column "Collapse" refers to whether slot collapse has occurred (color green for No denotes that this is a desired scenario). We report results on two datasets for brevity but find the conclusions to be identical across other datasets in the zero-shot benchmark (Didolkar et al. 2024). We report the quantitative measure of slot collapse
described in Section 4.2
MOVi-E
EntitySeg
Model
Collapse
FG-ARI
mBO
FG-ARI
mBO
OCEBO
0.13
NO
54.8
25.8
41.5
15.3
(a) W/o patch filtering
0.02
YES
27.7
10.6
31.7
10.0
(b) Aoe = 0
0.02
YES
8.0
3.4
30.1
7.6
(c) Before sharpening
0.21
NO
44.0
20.8
39.4
12.8
(d) COCO+
0.22
NO
66.8
22.1
44.2
16.0
Measuring slot collapse. Slot collapse is the main mode of failure of object-centric models. When slot collapse occurs, slots do not attach to meaningful subparts of the scene, i.e., objects, but rather to positionally coherent regions of the image (e.g.. block-like structures or bands going from top importannee to bottom or left right). Hence, avoiding slot collapse is of the utmost importance when train- ing object-centric models. The simplest way to identify slot collapse is to qualitatively observe predictions made by the model. In addition, we introduce a quantitative measure of slot collapse. Consider a patch present in both views of the image and assume it is located at index i in view and index j in view 2, i.c., qı and q2.:- Consider also a patch in view at index i, i.c., 42 Intuitively, when no slot collapse occurs, the similarity of q1,; and q2 must be higher than that of qı and 42,5- When a positional collapse occurs, the opposite holds. We introduce a measure d = fiimqii,i 92.5) sim(qi, 42,;) averaged over the validation set measure of slot collapse.
Cross-view patch filtering. First and foremost, we analyze OCEBO's slot collapse avoidance ca- pabilities. The lack of informative reconstruction targets has been shown be the main reason
7

---

Published as a conference paper at ICLR 2025
for slot collapse in object-centric models Seitzer et al. (2022) Due to random initialization of our target encoder, reconstruction targets are noisy and not always informative in the initial stages of training. Cross-view patch filtering therefore serves collapse prevention mechanism by filtering the patches that should not be considered for self-distillation. As indicated Table I (a), omitting this mechanism immediately results in slot collapse. 'he importance of patch-filtering illustrated by Figure 2 In the first epoch of training, only ~ ~0%% of patches satisfy the patch filter- ing condition, meaning that the vast majority of reconstruction target not informative and should not be used. As the training progresses, the percentage of supervised patches increases drastically and finally starts plateauing at ~70% around epoch 200. We perform additional ablations of the cross-view patch filtering approach in Appendix B
further
70
% of supervised patches
60
COCO+
50-
COCO
40
30
20
10
50
100
150
200
250
Training epoch
Figure 2: The percentage of supervised patches, i.e., those that satisfy the cross-view patch filtering condition throughout the model training. Blue line corresponds to the model trained on COCO, while the orange line corresponds to that trained on COCO+.
Object-centric inductive biases. Second, we demonstrate the importance of injecting object- centric inductive biases into the target encoder by using the object-centric training objective and EMA updates of the target encoder. Table (b) shows the results achieved without the object- centric training objective, i.e., by setting Aor. and training slot attention encoder and decoder only during the mask sharpening stage. This reduces OCEBO to pretraining of a DINO model on COCO followed by FT-DINOSAUR fine-tuning with the frozen COCO pree--petd DINO target encoder. Interestingly, this also leads to collapse, although different mode of collapse in which all pixels are assigned to one slot. Hence, DINO pretraining on COCO is not sufficient to learn informative reconstruction targets.
Moreover, the importance of object- inductive biases can be clearly observed by comparing PCA visualizations (see Figure 3 of features produced by OCEBO and DINOv2. In the first three columns, OCEBO separates the bear and human instances, while DINOv2 groups them together. Another interesting observation code not only instances but also the whrt- hicrarchies. In the first three principal components, the food plate is treated as whole, while principal components 4-6 encode each food type as separate instance. Similarly, different groups of people are further divided in principal components 6.
be made from the remaining columns OCEBO learns to
Mask sharpening. Interestingly, segmentation masks produced by the slot decoder of OCEBO exhibit lack of clear boundaries between objects. We attribute this the constant change of reconstruction targets. Introducing a shorter mask sharpening stage in which the targets are kept frozen and the self-distillation objective replaced by a typical fg loss improves the quality of segmentation masks, indicated In Table (cc).
Scalability. Object-centric approaches that rely on pretrained encoders have been shown to ex- hibit poor scalability with added data Didolkar et al. (2024). In fact, when trained on COCO, their performance saturates with a subset of 16384 COCO images. To achieve large-scale pretraining of object-centric models, they must actually scale with data. We demonstrate the scalability of OCEBO by training it on COCO+, a dataset roughly twice the size of COCO. Table [d) indicates that the performance increases consistently across datasets in terms of FG-ARI. On MOVi-E, observe
8

---

Published as a conference paper at ICLR 2025
DINOVE (1-3)
Image
OURS (1-3)
DINOv2 (1-3)
Image
OURS (1-3)
OURS (4-6)
DINOV2 (4-6)
Figure 3: PCA visualizations of the representations produced by the target encoder of OCEBO and by DINOv2. RGB values correspond to principal components 1-3 or 4-6.
slight decrease in mBO but at the cost of high increase in FG-ARI. As we note in more detail in Section 4.3 trade-off between FG-ARI and mBO should always be taken into account. However, we argue that large increase in FG-ARI is highly beneficial as simple technigues. such as short high-resolution training stage (Didolkar al., 2024) can boost the performance terms of mBO.
We provide more detailed scaling plot in Appendix C
However, scale of the data not the only important consideration. COCO is a curated dataset containing complex scenes with several objects per scene. This is exactly the type of data required by object-centric models. For instance, ImageNet (Russakovsky et al., 2013) an order of magnitude larger than COCO 1.M images) but is not suitable pretraining dataset for object-centric models as mostly contains simple scenes with single object taking up a large and central part of the scene. In fact, attempt train OCEBO ImageNet results in a drastically lower performance (FG-ARI 46.0, mBO 16.0 on MOVi-E and 39.5 FG-ARI, 11.8 mBO on EntitySeg). Constructing a large-scale dataset suitable for pretraining of object-centric models remains open question.
4.3 COMPARISON
OBJECT-CENTRE MODELS
So far, have demonstrated that OCEBO enables pretraining of object-centric models from scratch on real-world data while avoiding slot collapse. To put its performance in context, we compare with state-of-the-ard object-centric models. The results reported in Table 2
Table 2: Performance comparison of OCEBO and state-of of-the-artttttt object-centric models. Numbers for models other than OCEBO are taken from Didolkar al. (2024).
MOViC
MONG-E
Pascal VOC
EntitySeg
Pretrained encoder
FG-ARI
mB0
FG-ARI
mBO
FG-ARI
mBO
FG-ARI
mBO
SlotDiffusion Wu 2023
DINO
66.9
43.6
67.6
26.4
21.1
42.0
43.7
25,1
SPOT (Kakogeirgnno et al. 2014
DINO
63.0
40.8
47.8
21.5
21.2
50.6
41.7
27.4
DINOSAUR (Siizerr
2022
DINOv2
67.0
24.2
24.0
43.5
19.4
FT-DINOSALE AUR Diboiial a
2024
DINOv2
71.3
44.2
71.1
29.9
24.0
37.6
48.1
28.4
OCERO
None
63.1
27.3
66.8
22.1
22.4
34.4
44.2
16.0
We note several things here. First, the models are not directly comparable. OCEBO is pretrained from scratch on COCO+, while the rest of the models rely on image encoders pretrained on 1.3M (DINO) or 142M (DINOv2) images. Second, in this work we do not focus on tuning the per- formance and achieving the highest possible numbers. For instance, the autoregressive decoding strategy of SPOT Kakogeorgiou et al. (2024) is known to significantly improve the mBO metric, while FT-DINOSAUR (Didolkar et al. 2024) uses top-k MLP decoder and a short high-resolution training stage that improve both FG-ARI and mBO. Rather than incorporating these additional com- ponents, we aim to demonstrate that pretraining is possible and scales well with data. Moreover, the results indicate that els, although OCEBO has seen orders of magnitude less images during training. This emphasizes the importance of object-centric pretraining rather than adapting encoder already pretrained in
achieve performance comparable that of state-of-the mod-

---

Published as a conference paper at ICLR 2025
non-object-centric way. However, we also predict that incorporating components from SPOT or FT-DINOSAUR (or subsequent works) will be of the utmost importance for reaching the highest possible performance. Finally, there is always trade-off between FG-ARI and mBO inherently present in every model. For instance, models such as DINOSAUR Seitzer et al. (022)) attain higher FG-ARI at the cost of mBO due to the use of MIPP-based decoder, which also use in OCEBO (we observe the same trend here). On the other hand, autoregressive decoders such as that of SPOT Kakogeorgiou et al. 2024) increase mBO the cost of FG-ARI. Therefore, comparing perfor- mance gains is further complicated by this as no metric is clearly better than others. This can be clearly observed by comparing methods in T
5 CONCLUSION
We propose OCEBO, the first ever pretraining scheme for object-centric models. The inspiration for this work stems from the limitations of state-of-the-art object-centric models caused by the use of frozen pretrained target encoders that cannot be meaningfully updated during training and thus impose an upper bound on attainable performance and result in poor data scalability. We demonstrate that through EMA updates of the target encoder we remove the upper limit and that training from scratch allows the target encoder to capture object-centric inductive biases, thus utilizing the data to its full extent. OCEBO scales well with dataset size and achieves comparable performance despite seeing only fraction of training data seen by pretrained non-object-centric models used as target encoders in other works.
A notable limitation of this work is the dataset size used for pretraining. Namely, for object-centric pretraining is an open question as datasets with simple scenes such as ImageNet prevent object- centric models from capturing the most important information objects. We hope that this work and the novel insights brings (most importantly, the possibility to scalably pretrain object-centric models) inspires the community explore object-centric pretraining larger datasets and ulti- mately pursue the goal of large-scale object-centric foundation models.
Reproducibility Statement. Code and models will be made available upon acceptance of the paper in a public repository. This will include a README file with instructions for setting up the environment and reproducing the experiments All datasets used are publicly available.
Acknowledgements. This research was supported by funding from the Flemish Government un- der the "Ondezzoeksprorramma Artificiele Intelligentic (AI) Vlaanderen". This work was also par- tially funded by the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation program (Grant Agreement No. 101021347). We acknowledge EuroCC Belgium for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium. The resources and services used in this work were provided by the VSC (Flemiis Supercomputer Center), funded by the Research Foundation Flanders (FWO) and the Flemish Govermment.
REFERENCES
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Ivana Balažević, David Steiner, Nikhil Parthasarathy, Relja Arandjelovié, and Olivier Hénaff. Towards in-context scene understanding. In Proceedings the Conference on Neural Information Processing Systems (NeurIPS), 2023.
Nikolaus Kriegeskorte Benjamin Peters. Capturing the objects of vision with neural networks. Na- ture Human Behaviour, 2021.
Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dol- lar. Benchmarking in manipulation research: Using the aale-cmu-berkeles object and model set. IEEE Robotics & Automation Magazine, 22(3):36-52, 2015. doi: 10. 1099MRR 2445.2448951.
10

---

Published as a conference paper at ICLR 2025
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the International Conference Computer Viston (ICCV), 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference Machine Learning (ICML), 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, 2021.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießsner. Scannet: Richly-annotated 3d reconstructions of indoor scenes, 2017.
Aniket Didolkar, Andrii Zadaianchuk, Anirudh Goyal, Mike Mozer, Yoshua Bengio, Georg Martius, and Maximilian Seitzer. Zero-shot object-centric representation learning, 2024. URL https:
69162
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Inf. J. Compul. Vis., 88(2): 303-338, 2010. URL httt:://bblp uni- trier de /b/ hbb/jourals/////// /jov//jcvvvs html #Bvellivennnhameee
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam. Florian Golemo, Charles Herrmann, Thomas Kipf. Ab- hijit Kundu, Dmitry Lagun, Issam Laradji, Hsuch-Ti, Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangeheng Zhong, and Andrea Tagliasacchi. Kubrie: A scalable dataset generator, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altch'e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to supe-supe learning. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2020.
Dan Haramati, Tal Daniel, and Aviv Tamar. Centry-centric reinforcement learning for object manip- ulation from pixels. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked toencoders are scalable vision learners. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. In Pro- ceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2023.
Ioannis Kakogeorgiou, Spyros Gidaris, Konstantinos Karantzalos, and Nikos Komodakis. Spot: Self-training with patch-order permutation for object-centric learning with autoregressive trans- formers. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2024.
Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevitex: A texture-rich benchmark for unsupervised multi-object segmentation, 2021.
11

---

Published as a conference paper at ICLR 2025
Tim Lebailly, Thomas Stegmüller, Behzad Bozorgtabar, Jean- Phin-Phil Thiran, and Tinne Tuytelaars. CrlBo: Self-supervised learning via cross- -rass object-level bootstrapping. In The Twelfth In- ternational Conference Learning Representations, 2024. URL n d= 3MOGXOJRZP
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitncck, and Piotr Dollár. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision (ECCV), 2015.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2020.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Po-Yao Yao Huang, Hu Xu, Vasu Sharma, Shang-Wei Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2023.
Lu Qi, Jason Kuen, Weidong Guo, Tiancheng Shen, Jiuxiang Gu, Jiaya Jia, Zhe Lin, and Ming- Hsuan Yang. High-quality entity segmentation, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (ICV), 1 115(3):211225 252, 2015. doi: 10. 1007/s 1263-015-0816-y.
Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučić, Leonidas J. Guibas, Klaus Greff, and Thomas Kipf. Object Scene Representa- tion Transformer. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2022.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Scholkopf, Thomas Brox, and Francesco Lo- catello. Bridging the gap to real-world object-centric learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Thomas Stegmüller, Tim Lebailly, Behzad Bozorgtabar, Tinne Tuytelaars, and Jean- Jean-Philippe Thiran. Croc: Cross-view online clustering for dense visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Xin Wen, Bingchen Zhao, Anlin Zheng. Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2022.
Thaddaus Wiedemer, Jack Brady, Alexander Panfilov, Attila Juhos, Matthias Bethge, and Wieland Brendel. Provable compositional generalization for object-centric learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Object- centric generative modeling with diffiusion models. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2023.
Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu. Slot-vim: Slowfast slots for video- language modeling, 2024. URL
/ arwiv.iiiv..... org. abs/2402 1104022110888
Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Self-supervised visual reinforcement learning with object-centric representations. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
12

---

Published as a conference paper at ICLR 2025
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
A ESTIMATING THE QUALITY OF PATCH-LEVEL. REPRESENTATIONS
Object-centric models that keep the pretrained encoder frozen throughout the training inherently keep the encoder as is and retain its patch-level representation capabilities. On the other hand, works like SPOT or FI-DINOSAUR fine-tune the encoder and thus change its weights and the quality of patch-level representations. Moreover, OCEBO is pretrained from scratch, it would ideally learn not only good object-level representations but also good patch-level representation, thus becoming versatile model useful for variety of downstream tasks. To verify this, we evaluate object- centric models in a dense downstream task of in-context semantic segmentation (i.c., retrieval-based scene understanding), as described in BBllažvić et al. (203);; Lebailly et al. (2024). We evaluate on Pascal VOC Everingham et al. (2010), subsampling its training set with factors of 128, 64, 8 and We nearest neighbors classifier with the number of neighbors set to and evaluate the performance the full validation set. We compare OCEBO to a state-of-the-art patch-level self-supervised approach CrOC (Stegmüller et al. 2023). For SPOT and FT-DINOSAUR, we report the performance of their original backbones and the fine-tuned versions. The results reported in Table OCEBO achieves performance comparable to that achieved by state-of-the-at patch-level model. Interestingly, fine-tuning the encoder for object-centric learning degrades the quality of patch representations, although not drastically. This is expected as a consequence non-object-centric model serving target encoder, thus forcing the object-centric model adjust its features towards slightly more spatially-oriented representations.
Table 3: Retrieval-based scene understanding performance. We report mIOU on the Pascal VOC dataset, where the training set is subsampled with factors of 128, 64, 8 and
Model
Backbone architecture
Pretraining dataset
1/128
1/64
1/8
1/1
OCEBO CrOC
ViT-S/16
COCO
28.5
32.3
39.6
46.2
VIT-S/16
COCO
27.1
31.4
40.3
47.1
SPOT
ViT-B/16
ImageNet ImageNet
25.9
31.9
41.9
50.2
DINO
VIT-B/16
29.2
34.7
47.2
54.9
FT-DINOSAUR
VIT-S/14 ViT-S/14
LVD142M
33.9
43.1
58.1
65.7
DINOv2
LVD142M
46.9
53.7
64.7
69.05
B ADDITIONAL ABLATIONS OF CROSS- VIEW PATCH FILTERING
In Section 4.2 we demonstrate that without the cross-view patch filtering strategy, OCEBO suffers from slot collapse. However, one could wonder whether the cross-view strategy necessary or if simpler heuristic would achieve the same effect. We design two simpler heuristics that rely on 1) su- pervising all patches but giving the object-centric objective low weight and increasing throughout the training or 2) randomly selecting patches to supervise with the drop ratio decreasing throughout the training. The results are shown in Table Although better than the case where no object-centric objective is used (see Table (b)), both versions fall short to the cross-view patch filtering strategy of OCEBO. Interestingly, the slot collapse measure significantly lower compared to OCEBO and qualitative inspection shows predictions not completely collapsed but closer to collapse in compar- ison with OCEBO. This further corroborates the need for selecting informative patches rather than just letting the global loss drive initial stages of training.
13

---

Published as a conference paper at ICLR 2025
Table 4: FG-ARI and mBO for synthetic (MOVi-E) and one real-world (EntitySeg) dataset tained by different variants of OCEBO's cross-view patch filtering approach. The first row shows the base OCEBO model trained on COCO, while the subsequent rows represent modifications. Col- umn "Collapse" refers to whether slot collapse has occurred (color green for No denotes that this desired scenario). We report results on two datasets for brevity but find the conclusions to be identical across other datasets in the zero-shot benchmark (Didolkar et al. 2024). We report the quantitative measure of slot collapse d described in Section
MOVi-E
EntitySeg
Model
Collapse
FG-ARI
mBO
FG-ARI
mBO
OCEBO
0.093
NO
54.8
25.8
41.5
15.3
(a)
All patches, increasing weight
-0,017
7
49.2
22.4
36.9
13.5
(b)
Random patches, decreasing drop ratio
-0.026
44.0
20.0
36.7
12.7
C SCALING PLOTS
From Section we determine that OCEBO scales well in the range of COCO and COCO+ dataset aim to better depict the scaling laws by providing few more data points. The plots indicate that the model still scales well and does not saturate near the size of COCO+ dataset, hinting that further scaling is possible. Note that due to the design of OCEBO (MLP decoder of slot attention), it is among the object-centric methods that favor FG-ARI rather than mBO, as discussed in Section 4.3
sizes. In Figure 4
PVCC
COCO
FrieySag
20-
2
2
Dasnctace
Daseatscs
20-
10
24
1180
Scaling plots for FG-ARI (left) and mBO (right) with dataset sizes of 15 26 2 and 2 sampled from COCO or COCO+. Bottom: FG-ARI vs. mBO plot where point sizes indicate the dataset size (the smallest point corresponds to 215 while the largest corresponds 218
Figure 4: Top:
14


Reviews:
[{"review_id": "VuURadrJ91", "reviewer": "Reviewer_7Zvh", "summary": "In the research background, large-scale foundation models are common due to self-supervised learning techniques in deep learning, especially in computer vision. Cognitive psychology research indicates human visual perception is object-centric, leading to object-centric representation learning, though such models lack successful pre-training on large-scale real-world datasets. The research purpose is to propose the OCEBO method for pre-training object-centric models from scratch on real data to overcome limitations and unleash potential. The research methods involve a model architecture with an image encoder, slot attention encoder, slot decoder, and a target encoder of the same architecture, and a training objective formulated as a self-distillation bootstrapping problem with defined object-centric self-distillation loss including cross-view patch filtering and an optional mask sharpening stage. The experimental results on the MS COCO dataset and evaluation on multiple datasets with different metrics show that OCEBO can avoid slot collapse and achieve comparable performance to existing models with pre-trained target encoders while demonstrating good data scalability.", "questions": "1. When updating the target encoder as an exponential moving average (EMA) of the object-centric model encoder, how can we ensure that the introduced object-centric inductive biases do not overly affect the model's learning of other features, thus maintaining good generalization ability in different downstream tasks?  \n2. When the cross-view patch filtering method determines which patches to use for training, although it considers the feature quality of the target encoder, is it possible that this method may overlook some patch information that is potentially helpful for the model's learning? How can the accuracy and comprehensiveness of patch selection be better balanced? \n3. The paper mentions that constructing a large-scale dataset suitable for pre-training object-centric models remains an open question. Do the authors have any preliminary ideas or directions on how to construct such a dataset?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. A new object-centric pre-training method, OCEBO, is proposed. It is the first self-distillation setup for training object-centric models from scratch on real-world data.\n\n2. Experiments prove that OCEBO can avoid slot collapse and achieve performance comparable to existing methods using a large number of pre-trained images on multiple datasets while demonstrating good data scalability.\n\n3. The importance of object-centric inductive biases is emphasized, and its positive impact on the target encoder is verified through experiments, providing new insights into the theory of object-centric learning.", "weaknesses": "1. Although good results have been achieved on the MS COCO dataset, the requirements for pre-training datasets are relatively high. Datasets containing simple scenes like ImageNet are not suitable for pre-training object-centric models, and a large-scale dataset suitable for pre-training object-centric models has not yet been found.\n\n2. When comparing with existing state-of-the-art object-centric models, due to different pre-training methods and datasets used, the models are not directly comparable, which, to some extent, affects the accurate evaluation of model performance.\n\n3. The experimental setup and evaluation system are still somewhat rudimentary and cannot fully demonstrate the scheme's advantages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LA1mkExh5j", "reviewer": "Reviewer_a9mq", "summary": "The authors propose an approach to train object-centric models from scratch using real-world data, rather than relying on pre-trained non-object-centric foundation models.\nThe method is based on cross-view teacher-student self-distillation, in a similar fashion to DINO, IBOT and DINOv2.\nThe model architecture incorporates a slot-attention bottleneck and the patch-level loss uses a filtering strategy to stabilize training.\nThe method is trained on COCO and evaluated on different datasets on the task of unsupervised object discovery, where it attains performance comparable to (but lower than) previous methods that leverage large-scale pre-trained models.", "questions": "**Equation 1:**\nI suggest renaming $\\mathcal{L}_{oc}$ to something else to avoid confusion with the actual loss used during training which is defined in equation 3.\n\n**Ablation of head design:**\nEquations 3 and 4, as well as the filtered version in 8, describe a cross-view teacher-student distillation loss.\nThis setup requires quite a few moving parts, especially the cropping strategy with overlapping parts and the inverse augmentation.\nWould it be possible to train the model without cross-view distillation, but only using the teacher's output on the same crop as the target?\n\n**Comparison with the DINO objective?**\nThe global loss in equations 5 and 6 is formulated exactly as in DINO, why does the paragraph above it cite other papers and not DINO?\n\n**Suggestion about notation:**\nParagraph 3.3 and line 269 \"where $nns_{n_n}(z_{t,1}, z_{t,2})_i$ denotes indices of nn nearest neighbors\".\n\nThere are too many \"n\" characters in the chosen notation and it's hard to read.\nI suggest trying to replace $n_n$ with $k$ if possible.\n\n**Where is SPOT in the introduction?**\nTo the best of my knowledge, SPOT is the first work that unfreezes the encoder during training, and it was published months before FT-DINOSAUR.\nHowever, in the introduction, FT-DINOSAUR is presented as the first and is discusses in depth, while SPOT is not mentioned. This is misleading and should be corrected.\n\n**Missing results:**\nL435 \"In fact, an attempt to train OCEBO on ImageNet results in a drastically lower performance.\" where are these results?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The main strength of the paper is succeeding in training an object-centric model from scratch on COCO, which is known from previous works to be challenging.\nThe architecture or training procedure per-se are not particularly novel, mostly resembling the global and patch losses of DINO, IBOT and DINOv2, with the addition of a slot-attention bottleneck in the architecture.\n\nWhat is novel is the idea of filtering noisy patches that could be detrimental to the object-centric objective, especially during the first stages of training.\nThis idea, albeit not well ablated, seems to be a strong contribution of the paper.\n\nThe paper is easy to follow, with a good balance between technical details, analogies, and high-level explanations.\nQuantitative results are presented clearly and accompanied by qualitative examples.", "weaknesses": "**Ablation on patch filtering:**\nFrom section 4.2, it appears that patch filtering is crucial to stabilize training.\nThe chosen strategy uses an heuristic to filter out patches, especially during the first stages of training, as show in figure 2.\nThe first question that comes to mind is: how sensitive is the method to the choice of the heuristic?\nIt could be that the chosen heuristic has no importance and what really matters is that initially the global loss drives the training and the object loss is introduced gradually later.\nIn my opinion, this is an important ablation study to perform in the paper.\nTwo alternatives that I would like to see tested are:\n- Keeping all the patches but gradually increasing $ \\lambda_{oc} $ from 0 to 1 during training.\n- Randomly dropping patches in $ \\mathcal{L}_{oc} $ as opposed to selecting them via nearest neighbors. The drop ratio could be gradually increased from 0 to 1 during training to mimic the proposed heuristic.\n\n**Measuring slot collapse:**\nAn important point of discussion is \"slot collapse\", defined in the footnote at L107.\nSince the authors claim that the proposed patch filtering strategy is crucial to avoid slot collapse, it would be helpful to have a quantitative and objective metric to measure slot collapse.\nThis could be, for example, the correlation between slots and spatial positions across images, to measure whether a slot encodes the \"bottom right corner\" or a category of objects.\nThe green/red results in table 1 would be more informative and convincing if accompanied by such a metric.\n\n**One model or several ones?**\nThe whole model is trained from scratch on COCO and evaluated on different datasets, each with a specific number of slots (L319).\nDoes it mean that a new model needs to be trained from scratch for each number of slots?\nIf so, this is highly impractical for real-world applications where a practitioner would like to sweep over the number of slots to find the best one.\nIn such a case, frameworks like DINOSAUR or SPOT are much less expensive to use.\nIf not, how is the number of slots changed in the model? Is it fixed before training or can it be changed at inference?\n\n**No evaluation of the learned representation:**\nAll evaluations focus on segmentation-based metrics (FG-ARI and mBO) on several datasets.\nThe task of \"object-centric learning\", however, implies that the model should learn a representation of objects, not just segment them.\nIt would be useful to include a section that evaluates the slot representation on downstream tasks in a quantitative manner. \n\n**Projection head design:**\nOn L328-331 it says \"The projection heads are identical to those of DINO (Caron et al., 2021), with the exception of setting L = 8192 instead of the original 65536. Compared to the DINO head, ours projects every patch rather than just the global representations and we find that the gain in performance does not justify the computational cost.\"\nHowever, both IBOT and DINOv2 use per-patch heads and find that a large number of heads, even up to 131072, is beneficial.\nIf time allows, I recommend running an ablation study on the design of the projection heads, possibly splitting the object and global heads.\n\n**Performance and usefulness:**\nWeaker performance when compared to other methods that leverage large-scale pre-trained models (table 2).\nThis is somewhat expected, since the model is trained from scratch on a smaller dataset.\nAt a high level, this paper demonstrates that training from scratch is possible, but fails to prove that is actually beneficial.\nIf a pre-trained model achieves better performance, why should one train from scratch?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z2J56XHAXA", "reviewer": "Reviewer_hsmN", "summary": "This work proposed an object-centric pretraining method that updates the target encoder by EMA. The experiment results show that the proposed method can successfully learn object-centric representation. When pretrained on 241k images from COCO, the proposed achieves unsupervised object discovery performance comparable to other models with frozen non-object-centric target encoders pretrained on hundreds of millions of images.", "questions": "1. What distance is used when calculating nearest neighbors?\n2. I don't understand what is the meaning of ``invaug(q1)''.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper is well written and easy to follow.\n2. The proposed method can achieve unsupervised object discovery performance comparable to other models with frozen non-object-centric target encoders pretrained on hundreds of millions of images.\n3. The proposed method demonstrates scalability well beyond a few thousand training images.", "weaknesses": "1. How exactly object-centric inductive biases are captured by the target encoder, it may be better to explain the mechanism more intuitively or theoretically.\n2. As the author mentioned, although the proposed method has achieved comparable results in COCO pre-training, its advantage still needs to be verified on a larger scale of pre-training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YADFW3KIx8", "reviewer": "Reviewer_pa7a", "summary": "This paper studies the problem of effectively updating the target encoder in object-centric pre-training. Previous works use frozen pre-trained encoders as the target encoder, resulting in a performance upper limit. While updating the pre-trained encoders causes a significant performance drop, the paper proposes to bootstrap the target encoder from scratch. To prevent slot collapse, a cross-view patch filtering technique is proposed. Experiments show that OCEBO can be trained from scratch and learn from more data.", "questions": "1. The authors are suggested to provide more evidence on scalability. Moreover, it would be better to provide an estimation of data amount required to achieve comparable performance with SOTA models.\n2. More benchmarks should be compared. This paper only reported MOVi-C, MOVi-E, Pascal VOC, EntitySeg results, while only two of them are real-world datasets. The authors are suggested to add more real-world datasets, especially the COCO dataset.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to follow and understand. \n2. The motivations for cross-view patch filtering and mask sharpening stage are straight-forward and these two techniques are proven to be effective.", "weaknesses": "1. The experimental evidence for scalability is too weak. A scaling plot, which shows how the model performs as training data increases, is more supportive. From only two data points, it's hard to tell the scaling trend. For example, what if the model is just in rapid growth on 100k images and has already plateaued on 200k images? The authors are suggested to provide a scaling plot instead of two data points.\n2. There still seem to be large gaps between the final results and previous methods, which can not support the claim that OCEBO is comparable to those with pre-trained encoders.\n3. Discussion on object-centric data and non-object-centric data should be added. While a frozen target encoder can be an upper limit, a feasible way is to use stronger target encoders, as shown in the comparison between DINOv2 and DINO. Stronger DINO can be trained using more data, where object-centric data and non-object-centric data can both be used. So what's the benefit of scaling object-centric data over scaling pre-trained data for target encoders?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)