CrIBo: Self-Supervised Learning via Cross-Image Object-Level Bootstrapping
Authors: Tim Lebailly
Thomas Stegmüller
Behzad Bozorgtabar
Jean-Philippe Thiran
Tinne Tuytelaars
Venue: ICLR 2024 spotlight (2024)
Keywords: self-supervised learning
representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce a cross-image object-level self-supervised learning method.

Abstract: Leveraging nearest neighbor retrieval for self-supervised representation learning has proven beneficial with object-centric images. However, this approach faces limitations when applied to scene-centric datasets, where multiple objects within an image are only implicitly captured in the global representation. Such global bootstrapping can lead to undesirable entanglement of object representations. Furthermore, even object-centric datasets stand to benefit from a finer-grained bootstrapping approach. In response to these challenges, we introduce a novel $\textbf{Cr}$oss-$\textbf{I}$mage Object-Level $\textbf{Bo}$otstrapping method tailored to enhance dense visual representation learning. By employing object-level nearest neighbor bootstrapping throughout the training, CrIBo emerges as a notably strong and adequate candidate for in-context learning, leveraging nearest neighbor retrieval at test time. CrIBo shows state-of-the-art performance on the latter task while being highly competitive in more standard downstream segmentation tasks. Our code and pretrained models are publicly available at https://github.com/tileb1/CrIBo.

Published as a conference paper at ICLR 2024
CRIBO 2: SELF-SUPERVISED LEARNING VIA CROSS- IMAGE OBJECT-LEVEL BOOTSTRAPPING
Tim Lebailly 1.
Thomas Stegmüller**
Behzad Bozorgtabar
Phan-Philippe Thiran" 2.3
Tinne Tuytelaars
KU Leuven
EEPFL
SCHUV
((firstname). wwwstaaaaaaeeeeeeteeee www.atte..ueeeeeea be
[(firstname). (Lastname))eptt....... ch
ABSTRACT
Leveraging nearest neighbor retrieval for self-supervised representation learning has proven beneficial with object-centric images. However, this approach faces limitations when applied to scene-centric datasets, where multiple objects within an image are only implicitly captured in the global representation. Such global bootstrapping can lead to undesirable entanglement of object representations. Fur- thermore, even object-centric datasets stand to benefit from a finer-grained boot- strapping approach. In response these challenges, Image Object-Level Bootstrapping method tailored to enhance dense visual rep- resentation learning. By employing object-level nearest neighbor bootstrapping throughout the training, CrIBo emerges as a notably strong and adequate candidate for in-context learning, leveraging nearest neighbor retrieval at test time. CrIBo shows otate-of-the-arr performance on the latter task while being highly compet- itive in more standard downstream segmentation tasks. Our code and pretrained models are publicly available at https: //t: thub. com/ t lebl TT....
introduce novel Cross-
INTRODUCTION
Over the past few years, the field of artificial intelligence has experienced significant advancements, primarily driven by the democratization of deep learning techniques (Krizhevsky et al. 2012). Self- supervised learning (SSL) stands out as one of the major factors contributing to the success of deep learning. Indeed, SSL has opened the door to the training of large foundation models pretrained on massive amounts of uncurated data. In natural language processing (NLP), this large-scale task- agnostic pretraining has been the key to unlocking general-purpose features that perform well on various downstream tasks. Conversely, in computer vision, task-specific models and finetuning remain the de facto choice, in part due to the lack of consensus around the pretraining pretext task, such as is the case in NLP with masked language modeling (MLM). Nonetheless, Balažević et al. (2023) showed that under the auspices of their contextual pretrain- ing, dense nearest neighbor retrieval performed comparably with task-specific models while being
denotes equal contribution.
a) croes-imaage
b) crose-image ocjectieve
c) objjcctHevel
Criso
Figure 1: Positioning of CriBO in the landscape of self-supervised learning. a) Illustration of the cross-image self-supervision concept. e) Depiction of object-level self-supervision. b) CrlBo benefits from both learning paradigms.

---

Published as a conference paper at ICLR 2024
significantly more efficient. The contextualization mechanism ensures the alignment of the pretrain- ing phase with dense nearest neighbor evaluation by leveraging within and across image attention. Nevertheless, this mechanism only partially fulfills the alignment objective as i) the cross-image attention mechanism occurs from local to global features and ii) consistency is only encouraged between contextualized image-level representations of views from the same image. To mitigate these limitations, we propose to explicitly enforce cross-image consistency between object-level representations as shown in Figure
Most recent self-supervised learning algorithms, e.g., Chen et al. 22020); : He et al. (2020); Grill
al.] (2020a); Caron al., (202T). revolve around the objective of learning image-lever representa- tions invariant to semantic- semantic-preserving preserving data augmentations. CrIBo departs from this paradigm in two ways. First, we encourage consistency at the granularity of the object in the images, and second, this consistency is enforced between objects from different images. In doing so, our pretraining aligns well with the objective of obtaining general-purpose representations tailored for dense near- est neighbor retrieval. Furthermore, by operating at the object level, CrIBo elegantly mitigates the pitfall of contextual bias Singh et al. (2020), and is therefore compatible with scene-centric images which constitute the bulk of web-scale datasets.
Our contributions are as follows. 1) To the best of our knowledge. CrlBo is the first end-to-end online SSL approach that explicitly enforces cross-image consistency the object/object-put level. 2) The resulting representations excel in in-context scene understanding tasks (Balažević et al.] 2023) without compromising the performance standard segmentation benchmarks. 3) Moreover, CriBo is compatible with scene-centric datasets, addressing a gap in the existing cross-image SSL literature.
2 RELATED WORKS
Image-level self-supervision. Seminal work SimCLR (Chen et al. 2020 has played a pivotal role in promoting the widespread adoption of cross-view consistency as fundamental principle for visual representation learning. However, such pretext task admits trivial solutions and contemporary works have yet agree on the ideal way to prevent them. One effective and intuitive approach to address this issue involves the use of negative samples (Chen et al., 2020 Hjelm et al. 2018). which effectively mitigates degeneracy but requires the use of large batchsizes. This can be alleviated with a memory bank (He et al., 2020). Self-distillation methods, on the other hand, do not use explicit negatives. They avoid trivial solutions by using asymmetry in the form of e.g., an additional predictor (Chen & He| 2020 Grill et al. 2020b) on one branch, using stop-gradients (Chen & Hej Caron et al. | 12 Li et al., 2021 Grill et al., 2020b), a momentum encoder (Caron et al, Grill et al. 2020b), etc. On the other end of the spectrum, clustering- methods avoid
2020
2021
trivial solutions by regularizing the assignment of the samples across a set of clusters (Asano et al,
2019
Caron et al. 2018 2020 2021 Zhuang al., 2019).
Localized self-supervision. A finer- finer-grained self-supervision can be obtained by leveraging the in- herent spatial structure of images and by enforcing cross-view consistency at localized level. This ensures that the resulting features are better aligned with dense downstream tasks. These methods can be broadly categorized based on the level of granularity which they enforce similarity. One
line of work involves enforcing similarity at the feature level, known as the local-level approach, where individual features (or s are directly contrasted (Wang al., [202T] Liu
et al. 2020: O Pinheiro et al., 2020: Xie et al. 2021b; Lebailly & Tuytelaars 2022 Bardes et al.]
2022). Another approach operates
coarser level. promoting similarity between semantically coherent groups of features, referred to as the object-level approach (Cho et al. 2021
Hénaff et al. 2022).
2021 2022, et al. a Seitzer al., 2022 Sttegüüller al. 2023 Wen et al.
Alternatively, there have been propositions of dense fine-tuning strategies (Hamilton et al.
2022
Ziegler & Asano| 2022 Yun et al. 2022. WWan et al., 2022] Zadaianchuk et al., 2022). These
techniques are designed to enhance models initially trained with an image-level objective Caron jet al. 2021) with localized seff-supervision. However, the former cannot be used as stand-alone pretraining methods. Another way of using the inherent spatial structure is by leveraging patch
location prediction (Caron et al. 2022) or by asking the model to reconstruct the input image from corrupted/masked version thereof (He al. 2022).
2

---

Published as a conference paper at ICLR 2024
Cross-image self-supervision. Moving beyond the conventional cross-view consistency inherent in numerous self-supervised methods, image-level bootstrapping harnesses nearest neighbors in the latent space for self-labeling and metric learning (Dwibedi et al., 2021 Koohpayegani et al. 2021 Azabou et al. 2021] Xie et al., 2022). Current bootstrapping methods for representation learning face two challenges. Firstly, the effectiveness of using distance metrics in the latent space proxy for semantic closeness is contingent on the quality of the learned representations, especially during the initial stage of pretraining when the encoder is randomly initialized. To address this limitation, adaptive bootstrapping methods have been proposed, aiming to avoid systematic nearest neighbor bootstrapping (Lebailly et al. 2023). Secondly, the context of scene-centric images, image-level bootstrapping reveals tendency to entangle object representations, which can have detrimental effects on dense downstream tasks. These issues motivate the introduction of the object- level cross-image self-supervised learning paradigm. To that end, ORL (Xie et al. 2021 introduced a three-stage procedure to identify pairs of region-of-interes (Rols) and encourage cross-image consistency between cropped objects at the image-level.
Recently, Balažević et al. 2023) proposed to leverage attention within and across images. The latter contextualizes the local representations of an image with the global ones from other images to adapt for in-context learning various scene understanding tasks at test time.
3 METHOD
3.1 PRELIMINARIES
Before diving into the method, we briefly introduce the terminology used in this paper.
Dense representation. A representation that explicitly preserves the spatial dependencies of the input image is referred to as dense representation. We denote such representations by #xxxx or RW where H and W are the height and width of the image or a downscaled version thereof and d refers to the dimension of the latent space. The dense representation produced by a convolutional neural network is a stack of output feature maps, whereas a vision transformer yields ordered sequence of tokens.
Local representation. Given spatial location out of the N HW possible ones, the asso- ciated slice the dense representation x" E Rd with n E {1,2, , is referred to as a local
representation.
The object representation c* Rd of the 1tth object within image is derived by aggregating the local representations associated with the corresponding object. We use the term object loosely to refer to a semantically coherent region within the image.
Object representation.
Global representation. The globul representation, denoted as E Rdd global aggregation over the spatial dimensions of the dense representation. As example, this corresponds to the [CLS] token in a ViT (Dosovitskiy et al., (2020) or the result of a global average pooling layer in case of a CNN.
Bootstrapping Here, the term bootstrapping refers the utilization of the model's current knowledge to enhance its performance further. In the scope this study, this materializes as lever- aging distances in the latent space find object-level nearest neighbors that can be used to form positive pairs for self-distillation.
3.2 OBJECT-LEVEL CROSS IMAGE BOOTSTRAPPING (CRIBO)
In Section discuss the need for generalist models capable of performing various downstream tasks without requiring any finetuning. A recent study (Balažević et al.) 2023) demonstrated that this could be achieved by leveraging nearest neighbor retrieval at test time and contextualization
The bootstrapping employed in the colloquial sense, as opposed the statistical sense.
3

---

Published as a conference paper at ICLR 2024
joint-space clustering
aatel 2068
refieved neares neigh
object representation one memory bank
femavel
c - mnte) objecttievel cross-maage loss
Figure 2: High-level overview of cross-image object-level bootstrapping (CrIBo). Given an encoder and pair of augmented views σ and ãz2 object representations cj (depicted colored object masks) from each view are computed. Using a memory bank, the nearest neighbors of each object representation et from the first view are retrieved. A self-supervised consistency loss (depicted as colored arrows) is then enforced between ci and its corresponding retrieved neighbor from the other view nn(c)).
throughout the training phase. Aligning with this aim, propose further reducing the gap be- tween the training and test phases by explicitly enforcing consistency between object-level nearest neighbors. The existing SSL studies exploiting nearest neighbors are built solely around global rep- resentations and are rooted in the assumption of datasets being object-centric. Hence, a simplistic extension of such frameworks to our proposed scenario is impracticable. To accomplish our goal, the following steps are required:
1. Identify semantically coherent regions within the image to create object-level representa- tions (Sec. 3.2.1).
2. Match pairs of object-level representations across images (Sec. 3.2.2
3. Apply cross-image consistency between pairs of matching object-level representations (Sec. 3.2.3).
high-level overview of our proposed method CrIBo can be found in Figure 2
3.2.1 SEMANTICALLY COHERENT IMAGE REGIONS
The first step towards cross- cross-image image object-level bootstrapping is to identify semantically coherent im- age regions within the images. We argue that it is beneficial to use pairs of augmented views for each input image and to have a cross-view correspondence between the objects in both views. Indeed, image-level SSL methods yield significantly worsened performance when training without photo- metric augmentations (Dwibedi et al. (021)). This is observed when positive pairs originate from the same image and a lesser extent (but still significant) when they do not. This drop in performance highlights the need to shed away low-level similarities between positive pairs of images with data augmentations. To prevent this from occurring with nearest neighbor (NN) positives, [Dwibedi et al. (022)) refrain from enforcing consistency between a retrieved NN and its query. At the image level, this can be easily implemented by leveraging the duplicity of the views. More specifically, given two augmented views T1 and 2h well their corresponding globall-epressentaton Æ1 and ø2, similarity constraint is enforced between ε and nn where nn(.) denotes the nearest neighbor operator. Conversely, the object level, this requires cross-view correspondence between object representations from both views. To that end, we rely on an online joint-space clustering algorithm from Stegmüller al. 2023).
Joint-space clustering. The clustering algorithm takes input the dense representations #1 N and #2 Nxd associated with the two augmented views and concatenates them along the token axis to obtain the dense representation of the joint-view, Zua R2Nxd The 2N tokens subsequently partitioned into K clusters to produce semantically coherent and spatially compact clusters. A hyperparameter Apos modulates the importance of the spatiality against that of the mantic (see Tab. 4.. The resulting assignment is encoded in matrix Q" E 222 which can
4

---

Published as a conference paper at ICLR 2024
smaw oud
B
batch
memory bonk
Figure 3: Illustration of cycle-consistent matchings. Such matchings are invariant to data aug- mentations and reciprocal, loosely speaking.
be split in two to obtain the view-wise assignments Qi, Q5 E RNNK Here (n, k) denotes a soft-assignment of token object-cluster k in view These can in turn be used compute the K object representations E I in each view i as follows:
cf
Σ Q (n, = E (1,2)
RE[N]
Note that the columns of Q undergo 11 normalization, rendering the object representations as affine combinations of local representations. It can be observed that the cross-view correspondence is provided for free the object representation of from the first view is matched to ci in the second one as a consequence the joint-space clustering. In practice, can be the case that some objects exclusive to of the two views i.e., ΣnE[N] Q (n, = 0. In that case, the object & is simply discarded from the view where it present. In Section 3.2.2 we show that such cross-view correspondence can be harnessed to improve the training stability through a bootstrapping criterion.
3.2.2 CROSS-IMAS MMOSS- OBJECT MATCHINGS
In Section 3.2.1
we detail the methodology for identifying objects within the views of given image and establish the strategy for their alignment from view to another. Subsequent to this step is the critical phase of aligning such objects across images. For reasons that will delineated in the next paragraph, rely pair of memory banks Mi and M2, which populated with object representations extracted from the first and second views during previous iterations. As such, the memory banks M; can be queried with object representation og to retrieve its nearest neighbor:
c
(n M.) A arg max
|cll, c
cEM;
Here, we consider nn and cf as cross-image object-level matches. visualization of cross-image object-level matches can be found in Appendix C.5 (Fig. and Fig. 5
Cycle-consistent matchings. the beginning of the pretraining phase, when the encoder ran- domly initialized, relying Equation (2) to identify cross-image object-level matches is not well grounded. Indeed, this straightforward approach assumes that the representations similar objects close in the embedding space, assumption that does not hold at the initialization. Therefore, enforcing consistency between such pairs of semantically distinct objects should be avoided. To that end, we design a semantic closeness condition. which only fulfilled when the matches established based on high-level features, thus enabling the discarding of invalid positive pairs that do not share semantic similarity.
At high-level, we consider a pair of objects as a valid positive pair if they are the nearest neigh- bors of each other. practice, the criterion is implemented as follows. Starting from an object representation cf in the batch of first views, its nearest neighbor nn(c); M,) in the queue is found using Equation (2). To ensure that the criterion is invariant to data augmentation, we then switch to the queue of second views via the jump(-) operator:
jump(ct) & cht
Jump cf
(3)
5

---

Published as a conference paper at ICLR 2024
From this step, obtain (immp and retrieve its NN in the batch of second views nn( jumjjnn(ee...ii. M1)). B2). The final step consists of switching back to the batch of first views and verifying if the resulting object is the same as the started from. More formally:
true false
= jumpmmmpns (umi (ump((nn(c
O(c,) -
i (1,2)i (4)
otherwise
A depiction of cycle-consistent matchings is provided in Figure
3.2.3 SELF-SUPERVISED TRAINING OBJECTIVES
At this point, we have all the necessary components to apply three distinct types of self-supervision: cross-view object-level, cross-image object-level, and global cross-image. All levels of self- supervision our method CrIBo are based on the self-distillation mechanism proposed by Caron et al. 2021), which employs student-teacher pair of siamese networks fa and fi, respectively. Given two augmented views #1 and 222 global representations and object representations can be obtained for each view and each network within the siamese pair.
Cross- View Object-Leeve Supervupervsi.n. The object representations from the teacher and stu- dent are fed to their respective head h2 and ht to output probability mass functions whose sharpness be modulated via temperature parameters (T, and T₁). For the teacher, this translates to:
p = softmax (he(ct,))TT)
i E (1,2)
(5)
The student's projections obtained analogously. The cross-vies view object-level loss is expressed as the averaged cross-entropy over all pairs of object representations and making the loss symmetric w.r.t. the student/teacher:
C.
Σ (H(Pi,..PE.)) + H(PE,. kE[K]
K
where H)a -- -t ai log(b:) denotes the cross-entropy.
Cross-Image Object-Level Self- Self-Supervisio. Once all object representations in the batch have been retrieved, the cross-image object-level loss is enforced in a manner analogous to its cross-view counterpart. The nearest neighbors are first fed to the projection head follows:
p softmax (Hx(nn(c))))/T))
E (1,
Note that these projections are only defined for the teacher since the nn(-) operator is differen- tiable. The formulation of the cross-image object-level loss aligns with that in Equation (6) up to the filtering of invalid positive pairs (see Sec. h.2.2
Ca
Hott P2.3) kE(K]
Σ
Σ ((o())) H H Zz RE(K)
1
where 1.,) denotes the indicator function and where 711 and 7/2 normalization constants iee, Ene[K] (oret)) and Σee(K] ((re))) respectively, Empirical evidence on the utility of the cycle consistency can be found in Appendix C.4
Global Cross-Vies Self-Supervision. Importantly, both the clustering and the bootstrapping steps are contingent on the quality of the representations. To avoid potential instabilities, we in- corporate global cross-view loss into framework, offering meaningful self-supervision at any stage of the pretraining. This loss is analogous to the one from Equation (6) except that it is applied to the global representation (21,a 22 21, and 22,1), which are fed to a dedicated head h:
Ait softmax (2.. I
i (1,2)
(9)

---

Published as a conference paper at ICLR 2024
Table I: Dense nearest neighbor retrieval. The quality of the learned spatial features is probed with a k-NN classifier and using different ratios of training data. The depicted mloU scores are derived from the validation sets of two scene-centric datasets. t refers our own reproduction from official GitHub repositories. If not specified, publicly available checkpoints are used. denotes results taken from Balažević et (2033)
Rackhone
Fpochs
1/138
1/8
Scone coutri
SlerCon Wen MI2
ResNes-00
25M
COCO
800
9.9
11.8
17.3
22.
37.3
42.8
52.9
57.1
ORL
XICEU FUMB
ResNet 50
25M
C000
900
9.1
10.3
13.6
16.6
30.2
33.3
42.0
45.6
S/16
6.9
18.4
Sagin
13.1
31.4
VT-8/16
CO00
10.9
12.8
18.4
44.0
Objert-vntric
SloOOnn (Wenet
M12
Re-Net50
25M
INIK
200
10.6
12.0
18.1
24.0
40.7
44.7
56.4
61.6
DINO
2049
MT S/16
21M
INIK
500
9.4
10.5
14.6
13.4
24.5
28.7
38.7
46.1
C:0
11 3035
VIT-8/16
21M
INIK
300
7.8
9.6
15.2
20.6
30.7
37.7
54.8
64.2
Time/1
Salett 31, 2025
VT-S/16
21M
INIK+YTVOS
800+30
11.7
13.6
19.7
24.6
38.9
44.3
56.0
62.7
WT-8016
Cmnnet
VIT-R/16
Woom (balakeric 6 2023
VIT-B/16
INIK
300
11.7
15.1
50.5
57.2
CrlBe
VIT-B/16
83M
INIK
400
132
16.5
23.6
30.0
50.5
60.3
70.8
74.9
where F. and T, are the corresponding temperature parameters for the L-dimensional output distri- bution. The global cross-view loss defined as:
C H (PI. D2.s) + H HP2,,,
(10)
where H(a.b) - - log((:))
Finally, we obtain the overall training objective CrlBo as sum of its individual components:
List L9 + Le. C3
An ablation study over the inclusion of the individual loss terms can be found Table As in Caron al. (2021), the student's parameters optimized to minimize the above loss whereas the parameters of the teacher are updated via an exponential moving average the student's weights.
4 EXPERIMENTS
We first verify that the proposed pretraining method aligns well with the objective of in-context learning via nearest neighbor retrieval. We then show that doing so, we do not compromise the performance standard evaluations. General details the experimental setup be found in Appendix
4.1 DENSE NEAREST NEIGHBOR RETRIEVAL
Inspired by Balažević et al. 2023 we probe the quality of the learned features with a dense nearest neighbor (NN) classification lask. The k-NN classifier fitted on the local representations of uniformly sub-sampled set of training images and evaluated on all the patches from the validation 128. The only preprocessing applied to the images is down-sampling the closest resolution which is a multiple of the patch size. Following that step, patch-level labels are obtained via majority voting. We report the mloU scores on Pascal VOC 2012 (Everingham et al.) and ADE20K (Zhou et al. 2017). When the sub-sampling ratio is greater than 1, the reported mloU scores result from averaging over 5 independent runs. The number of nearest neighbor patches is set to k: 50.
of images. The sub-sampling factor either 1,
In Table I we observe that using object-level nearest neighbor bootstrapping throughout training an adequate pretext task for this evaluation, which is not surprise due to the inherent similari- ties between the training and test phases. Overall, CrIBo outperforms all methods, but Humming- bird (Balažević al. 2023), by significant margin and particularly following the scene-centric

---

Published as a conference paper at ICLR 2024
Table Linear segmentation with frozen backbones. The linear decoder from Segmenter Strudell et al. (021) is trained on the frozen spatial features obtained with various self-supervised learning methods. We report the mloU scores achieved on the validation sets of different datasets. refers to our own reproduction from official GitHub repositories. If not specified, publicly available
checkpoints are used.
Epochs
Some-rratic
DINO!
Chion ctall 2023
VIT-S/16
21M
COCO
300
27.3
43.9
19.9
18.9
MAP
Fo 2019
VIT-5/16
21M
COCO
300
18.1
25.6
11.3
11.0
Croc
Seggmoner
2023
S/16
COCO
30,0
22.6
20.5
C210
S/16
21M
COCO
37.5
61,0
29.1
COCO
contric
S0le 21
INIK-YTVOS
800-10
67.9
CrIDo
VIT S/16
2IM
INIK
800
41.7
73.7
33.8
31.9
DINO
Caron et 2021
VIT B/16
SIM
INIK
500
38.9
64.6
31,9
30.5
MAE
HE 2744
VIT-B06
KSM
INIK
1600
27.9
40.9
14.4
17.5
CriBo
VIT-8166
S5M
INIK
600
42.9
74.9
36.0
34.7
Table 3: Finetuning evaluation with Segmenter. Backbones pre-trained with different self- supervised learning methods are finetuned using Segmenter (Strudel et al., [021]. We report the mloU scores achieved the validation sets of 4 different datasets. it refers from official GitHub repositories. If not specified, publicly available checkpoints are used.
reproduction
Method
Backhone
Params
Daraset
Epocts
Pascal Context
Pascal VOC
COCO-Stuff 164K
ADE20K
Scmee-contn contric
DINO!
MAB
Fk e DL AMM
VIT-S/16
2IM
COCO
300
32.5
61.6
35.4
35.4
C10C
Stepmmner
202
VIT-5/16
21M
COCO
300
38.7
70.5
38.0
37.9
VIT-S166
300
Objerk-amtiic
DINO
Caron et 2021
VT--/11
21M
INIK
800
46.0
80.3
43.2
43.3
CroC Stcemmner de 2023
VIT-S/16
21M
INLK
300
46.0
80.9
42.9
42.8
Time!
Sucent
VIT-5/16
21M
INIK+YTVOS
800+30
47.4
60.4
43.5
800
DINO
Caron el 2021
VIT-B/16
85M
INIK
400
45.8
82.2
45.0
MAE
[e 24 2002
VIT-B/16
85M
INIK
1600
47.9
82.7
45.5
46.4
CylRo
VIT-B/16
85M
INIK
400
49.2
83.4
44.6
46.0
pretraining. It is worth noting that the comparison with Balažević et al. (2023) is not exactly apple- to-apple, as do not use multiple augmentations per image and ii) we refrain from balancing the classes in the training set. These differences in the implementation of the evaluation reflect the divergence in our objectives: we place a higher emphasis on the assessment of the intrinsic quality of the learned representations rather than the absolute value of the reported results.
4.2 SEGMENTATION WITH A LINEAR HEAD
The frozen local representations are fed to the linear decoder head from Segmenter (Strudel et al. 2021] using the MMSSegmentation (Contributors) 2020) implementation. Herein, spatial tokens un- dergo a linear projection to the class space, followed by bilinear up-sampling to match the input image dimensions, enabling the application of pixel-wise cross-entropy loss. Performance metrics, in terms of mloU scores, are reported on the same 4 datasets (see Appendix A.2 for more details).
As reported in T CrIBo exhibits commendable performance in linear segmentation, following both scene-centric and object-centric pretrainings. Among the methods using ViT-B/16, CriBo strongly outperforms other baselines. In the case of ViT-S/16, CrIBo also outperforms all baselines but with smaller margin, due mostly to Time'T (Salehi et al. 2023) being particularly strong baseline. It noteworthy that as opposed to prevailing studies, e.g., Stegmüller et al. (2023); Bardes et al. (2022), we do not concatenate spatial tokens from multiple layers of the VIT. Indeed, this trick is typically used to compare on an equal footing with ResNet-50, which has much larger output dimension. As only and compare with ViTs, opt for evaluations incorporating as minimal parameters as possible, aspiring to present a more accurate reflection of the features' intrinsic quality.
8

---

Published as a conference paper at ICLR 2024
Table Ablation study on the hyperparameters of CrlBo. Here, ViT-S/16 undergoes pretrain- ing on COCO for 300 epochs with various combinations of hyperparameter values. The resulting features are then compared using dense nearest neighbors retrieval. We denote the size of the queues M; in terms of the number of images present by S. Unless explicitly indicated, the values of the hyperparameters are set to Apos 2.0, 25k, and K = 64.
Positional weighting (Aga)
Memory bank size (S)
Number of objocts (K)
Dataset
32
Pascal VOC
X
44,9
46.3
47.5
47.5
45.8
44,2
47.3
47.6
48.1
47.1
47,5
VOC
46.7
57.0
ADE20K ADE20K
16.9
18.0
18.9
18.4
17.8
16.3
17.7
18.0
18.9
18.5
18.4
21.6
21.8
23.1
23.4
22.6
22.5
18.1
18.4
23.4
18.1
19.4
20.3
21.7
23.4
4.3 END-TO-END FINETUNING WITH SEGMENTER
In a real-world scenario, self supervsee pretraining often precedes end-to-end finetuning on the target dataset with application-specifis decoder head. The ability of the different pretraining paradigms to cope with that setting is assessed with an end-to-end finetuning of the backbone topped with the transformer-basee decoder from Segmenter (Strudel et al. (02T). Here, the backhone's spa- tial features are fed to transformer encoder along with K learnable class tokens. The resulting class and spatial tokens are projected onto one another to obtain patch-level predictions, subsequently up- sampled to the input image's 's size to enforce the pixel-wise cross-entropy loss. We report the mloU scores achieved on the same datasets.
The insights gained from the preceding experiments (Tabs. D and underscore that models pre- trained with CrIBo have excellent generalists properties, iee. allowing them to perform various tasks without requiring finetuning. The results depicted in Table (3 indicate that, when required, these models can also evolve into remarkable specialists. All that aside, after fine-tuning, all models tend to perform comparably, highlighting that the finetuning may be suboptimal regime to compare SSL pretraining methods.
4.4 ABLATIONS
We report in Table 4 the results of grid search over various hyperparameters performed on COCO. Interestingly, we observe that CrlBo performs the best when operating in the overclustering regime, that is, with K much larger than the number of objects present in the images, but only when bootstrapping is used. We postulate that this is due to the fact that overclustering offers the possibility to find better matches. To illustrate, consider the image in Figure depicting a zebra, a giraffe, and an elephant. With K = 1, this would necessitate identifying another image displaying the identical trio animals; which is rare. With a larger value of K, the animals are isolated, expanding the number of good candidates. In the overelustering regime, the clusters represent object parts, which further facilitates the matchings. Similarly, a larger queue also amplifies the number of suitable candidates, trend reflected in the results until reaching saturation point at queue size of 25k. More ablations are available in Appendix
5 CONCLUSION
We introduce CriBo, the first end-to-end online cross-image object-level bootstrapping self- supervised learning method. CrIBo capitalizes the augmented sample diversity derived from image-level bootstrapping while harnessing the advantages of localized self-supervision inherent to object-level learning. We extensively evaluate CrIBo across multiple dense downstream tasks show- casing excellent performance on in-context scene understanding tasks and highlighting the merits of the cross-image object-level learning paradigm.
Limitations. This study is developed around the ViT architecture as it has become quite ubiquitous. As such, and for the sake of coherence, we only compare with other ViT-based methods. As we have to cope with a computational budget, we chose to invest it in producing well-grounded and motivated approach with extensive analysis, rather than in exploring larger datasets (Thomee et al.) 2016 Changpinyo et al. 2021) or models, e.g., ViT-Large or ViT-Huge.

---

Published as a conference paper at ICLR 2024
Acknowledgement. This project is funded by the European Research Council (ERC) un- der the European Union's Horizon 2020 research and innovation program (Grant Agreement No. 101021347). This work is also partially funded by the Personalized Health and Related Technolo- gies (PHRT)) grant number 2021/344; as well as the Fonds Wetenschappelijk Onderzoek (FWO), project G0A4720N. We acknowledge both EuroCC Belgium and the Swiss National Supercomput- ing Centre (project ID 606) for awarding this project access to the LUMI supercomputer.
REFERENCES
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. arXiv preprint arXiv:1911.05377 2019.
Mehdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C. Johnson, Kiran Bhaskaran-Nair, Max Dabagia, Keith B. Hengen, William R. Gray Roneal, Michal Valko, and Eva L. Dyer. Mine your own view: Self-supervised learning through across-sample prediction. CoRR, abbs/2102.11100 2021. URL httthtt::///raa wttp:////gg//b.....////// http://s///aaaaoo..raa//////.... 10106
Ivana Balažević, David Steiner, Nikhil Parthasarathy, Relja Arandjelović, and Olivier J Hénaff. Towards in-context scene understanding. arXiv preprint arXiv:2306. 01000,,0666,,,2500 2023.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicregl: Self-supervised learning of local visual features. arXiv preprint arXiv:2210.0157/1 2022.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209- 1218, 2018
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Mathilde Caron, Neil Houlsby, and Cordelia Schmid. Location-aware self-supervised transformers. arXiv preprint arXiv:2212.02400 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. htte//bb/221000 10566
CoRR,
0bs/2011.10566, 2020.
Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. Picie: Unsupervised semantic
segmentation using invariance and equivariance in clustering. In Proceedings of the IEEF/CVF
Conference on Computer Vision and Pattern Recognition, pp. 16794-16804, 2021.
MMSegmentation Contributors. MMSegmentation:: Openmmlab semantic segmentation toolbox and benchmark.
http://ggitubbb.oooppeeee com www.mmaaaammaaaa -wt.aaa/mmmmmmmmmtttttiooo 2020.
Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. Advances in neural
information processing systems, 26:2292-2300, 2013.
10

---

Published as a conference paper at ICLR 2024
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual represen-
IEEE Computer Society, October 2021. ISBN 978- 178-1-66542-812- tations. pp. 10.1099/CCCA499222211000009
URL https: www. wwwputer.p.c.................... org/ proceedings htt.:///cn..aacccceeec////////// 2tt.2///2200000/////00//... BmJDKW7JJgg
5. doi: 10.1
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal
Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object
detection. Advances in Neural Information Processing Systems, 34:26183-26197, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint vrXiv:2006.07733, 2020a.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning, 2020b.
Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint arXiv:2203.084144 2022.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Olivier Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, and João Carreira. Efficient visual pretraining with contrastive detection. arXiv preprint arXiv:2103.1 10957, 2021.
Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisser- man, João Carreira, and Relja Arandjelovié. Object discovery and representation networks. arXiv preprint arXiv:2203.0877
2022.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint rxiv:1808.06677. 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 44/26698, 2014.
11

---

Published as a conference paper at ICLR 2024
Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Mean shift for self- supervised learning. In Proceedings of the IEEE/CVF Interationaal Conference on Computer Vision (ICCV), pp. 10326-10335, October 2021.
Alex Krizhevsky, lly Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Tim Lebailly and Tinne Tuytelarrs. Global-local self-distillation for visual representation learning. arXiv preprint arXiv:2207. 2022.
Tim Lebailly, Thomas Stegmüller, Behzad Bozorgtabar. Jean-Philippe Thiran, and Tinne Tuytelaars. Adaptive similarity bootstrapping for celf-disilllation. arXiv preprint FrXiv:2303.13600 2023.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient supf-vsue vision transformers for representation learning. CoRR, abs/2106.09785, 2021. o 010016666775555
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Songtao Liu, Zeming Li, and Jian Sun. Self-emd: Self-supervised object detection without imagenet. arXiv preprint arXiv:2011. 2020.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Wine Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic seg- mentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 891-898, 2014.
Pedro O O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron Courville. Unsupervised learning of dense visual representations. Advances in Neural Information Process- ing Systems, 33:4489 4500, 2020.
Mohammadreza Salchi, Efstratios Gavves, Cees G. M. Snoek, and Yuki M. Asano. Time does tell: Self-supervised time-tuning of dense image representations. ICCV, 2023.
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap to real-world object-centric learning. arXiv preprint arXiv:2209. 14860. 2022.
Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, and Deepti Ghadiyaram. Don't judge an object by its context: Learning to overcome contextual bias. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11070- 11078, 2020.
Thomas Stegmüller, Tim Lebailly, Behzad Bozorgtabar, Tinne Tuytelaars, and Jean-Philippe Thiran. Croc: Cross-view online clustering for dense visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7000-7009, June 2023.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEESCVF international conference on computer vision, pp. 7262-7272. 2021.
Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):44-3, 2016.
Feng Wang. Huiyu Wang. Chen Wei, Alan Yuille, and Wei Shen. Cp2: Copy-paste contrastive pretraining for semantic segmentation. arXiv preprint 21071 2022.
Xinlong Wang. Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for supfrsupee visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3024-3033 2021.
12

---

Published as a conference paper at ICLR 2024
Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. supersspe visual representation learning with semantic grouping. arXiv preprint 22X::2205.112888 2022.
Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Unsupervised object- level representation learning from scene images. Advances in Neural Information Processing Systems, 34:28864-28876, 2021a.
Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew-Soon Ong. and Chen Change Loy. Delving into inter-image invariance for unsupervised visual representations. International Journal of Computer Vision, (90(11):2994-011 September 2022. ISSN 1573-1405, doi: 10. 1.200.10010 1007 /0007//111333 2111//111222-0222161111111
$11263-022-01681-
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consisteney for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16684-16693, 2021b.
Sukmin Yun, Hankook Lee, Jachyung Kim, and Jinwoo Shin. Patch-level representation learning for self-supervised vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8354-8363, 2022.
Andrii Zadaianchuk, Matthaeus Kleindessner, Yi Zhu, Francesco Locatello, and Thomas Brox. Unsupervised semantic segmentation with self-supervised object-centric representations. arXiv preprint ar arXiv:2207.05022 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633 633-641,22 2017.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmenta- tion, arXiv preprint ar 1t::2204.1310, 2022.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A EXPERIMENTAL SETUP
A.1 PRETRAINING
Pre-training datasets. Our pretraining datasets include COCO in et al. (014)) and ImageNet- Ik (Deng al.) 2009). The former includes about 1 118k scence-centric images while the latter includes about 1.3M object-centric images. Both are ubiquitous scene-centric self-supervvsed learning and object-centric self-supervised learning, respectively.
Network architecture. We employ vision transformers as backbone f. This selection aligns with its prevalent use in contemporary methodologies and allows for fair and easy comparisons with existing works. The configuration of the projection heads closely follows that of Caron et al. (2021). Both the image-level head h and object-level head h share their weights, except for the final linear layer. Both the image-level and the object-level output distribution have the same size i.e., L= 65, 536.
Optimization. The ViT-small (VIT-S/16) trained for 800 epochs, while the ViT-base (ViT-B/16) is trained for 400 epochs. The ViT-Base is only trained on ImageNet- Ik, while the ViT-Small is trained on both COCO and ImageNet- Ik. Pretrainings on COCO use a batchsize of 256 while pretrainings on ImageNet-1K Ik use batchsize of 1024. Learning rate, weight-decay, and other optimization-related hyperparameters are exactly the same as in DINO (Caron et al.) 202T).
Hyperparameters. Results reported tables using ViT-S/16 (apart from the gridsearch) are based on the following hyperparameters: (Apos S) (1.0, 25k, 32) and (Apos S, K) (2.0, 25k, 64) for pretrainings on ImageNet-IK and COCO, respectively. Results reported for ViT-B/16 are based using the following hyperparameters: (Apas, S, K) (1.0, 25k, 32).
A.2 EVALUATION PROTOCOLS
Linear segmentation. The frozen local representations are fed to the linear decoder head from Segmenter (Strudel et al. 2021). and we use the implementation available through MMSegmen- tation (Contributors) 2020). Herein, spatial tokens undergo a linear projection to the class space, followed by bilinear up-sampling to match the input image dimensions, enabling the application of pixel-wise cross-entropy loss. Performance metrics, in terms of mloU scores, are reported on four different datasets: Pascal Context (Mottaghi et al.) 2014), Pascal VOC 2012 (Everingham et al.) COCO-Stuff 164K (Caesar et al. 2018) and ADE20K (Zhou et al. 2017).
Regardless of the dataset used, the crop size is always 512x51 51 pixels, and the dataset configurations are kept as provided by MMSegmentation (Contributors) 2020). For ADE20K and COCO-Stuff the 160k _o0kiiterations..m schedule whereas for Pascal VOC 2012 and Pascal Context use the 80k jteratiors ieeeatioonssccccllle The only change we make to the default schedules configurations is that we use the Adam optimizer (Kingma & Ba 2014) instead of SGD. For each pretraining method and dataset, we run with four different learning rates (8e-4, 3e-4, le-4 4, and 8e-5) and report the highest mIoU score.
164K,
Finetuning with Segmenter. We perform an end-to-end finetuning of the backbone topped with the transformer-based decoder from Segmenter (Strudel et al.) 2021). Here, the backbone's 's spatial features are fed to a transformer encoder along with K learnable class tokens. The resulting class and spatial tokens are projected onto another obtain patch-level predictions and subsequently up-sampled to the input image's size to enforce the pixel-wise wiiee cross-entropy loss. We report the mloU scores achieved on 4 different datasets: Pascal Context (Mottaghi al. 2014), Pascal VOC 2012 (Everingham et al.), COCO-Sta Stuff 164K (Caesar et al. 2018) and ADE20K (Zhou et al. 2017).
Regardless of the dataset used, the crop size is always 512 512 pixels, and the dataset configu- rations are kept as provided by MMSegmentation (Contributors 2020). For ADE20K and COCO- Stuff 164K, use the 160k iterations schedule whereas for Pascal VOC 2012 and Pascal Context,
14

---

Published as a conference paper at ICLR 2024
the -scheduue.... Two changes are made w.r.t the default schedules configura- tions. First, we use the Adam optimizer (Kingma Ba 2014) instead of SGD and second, we use eta min 0.1 1r instead of eta min 1e-4. For each pretraining method and dataset, run with four different learning rates (8e-5, 3e-5, 5, 1 e-5, 5, and 8e-6) and report the highest mloU score.
B CLUSTERING ALGORITHM
We briefly detail the clustering algorithm, particularly the distinctions from its original implementa- tion proposed in Stegmüller (2023).
The overall objective of the clustering step is to assign the spatial tokens from both views of common image to semantically coherent groups. Formally, given the dense representations 21 and Z2 of two augmented views, the joint representation Zua E R2 is obtained by concatenation along the dimension of the tokens. The 1 centroid c* E IR is initialized by uniformly sampling without replacement one of the 2NV tokens. The clustering procedure relies the Sinkhorn-Knopp algorithm (Cuturi 2013) to solve the optimal transportation problem of assigning 2N tokens to K clusters/centroids. The transportation cost between tokens and centroids T E 2 K is given by the negative cosine distance:
T T
<
c
(12)
c
The Sinkhorn- Knopp algorithm efficiently computes a solution (an assignment matrix Q') to the following optimization problem:
Q = argmin < Q,T (sem)
(QQ)
(13)
QEQ
which aims to minimize the inner product of the transportation plan and the transportation cost, subject to entropy constraint on the assignments. The transportation polytope Q defines the set of valid assignments:
Q = [Q E
2NNK R
QIK
12N, Q 12NN
K 1k}
(14)
2N
We update the centroids by pooling over each cluster:
Σ RE[NN
(m z
(15)
At this point, we could stop, but we observed improved results when doing multiple iterations of the above steps, iee. restarting the procedure from Equation (12) and the centroids found in Equa- tion (15). Empirically, we found that 5 iterations is good trade-off between speed and accuracy. In practice, we use the "hard" version of the assignments obtained by ensuring that each token only assigned to single cluster. After a final i--oormalization of the columns of the herein obtained matrix, the view-wise object representations are computed as in Equation
Importantly, the above-defined algorithm diverges from the one proposed in CroC (Stegmüller et al., 2023) in multiple ways. The main difference is that we do not rely on an iterative procedure to find the optimal number of centroids K, and as such, the number of clusters is fixed. Another discrepancy that we use uniform distributions for both the sampling of the initial centroids and the marginals of the assignments (see Eqs. (13) and (4)). We observed that these modifications simplify the overall algorithm without incurring a drop in performance. the following paragraph, we discuss adjustments to the positional cues of the clustering algorithm.
An important property of the selected clustering algorithm is its ability to leverage positional cues to compensate for the lack of high-level semantics the features at the beginning of the training. The spatial guidance is incorporated the algorithm via an additive constraint on the transportation cost matrix:
Positional cues
Toot
T
Apas (pos)
(16)
15

---

Published as a conference paper at ICLR 2024
Table 5: Ablation study on the individual contribution of each loss. ViT-S/16 are trained on COCO for 300 epochs and evaluated with downstream tasks of different levels of granularity.
Paveh-level NN
Object Detection with YOLO-S
Image-level NN
4%
Pascal VOC (mloU)
ADE20k (mloU)
COCO (AP")
COCO (AP
COCO (AP))
INLK (top-1 Acc.)
23.9
34.3
47.1
18.3
27.1
44.2
27.6
35.8
58.1
23.4
30.2
48.4
31,0
38.2
Table 6: Object detection with YOLO-S. Backbones pre-trained with different self supervueee learning methods are finetuned with YOLO-S (He et al. 2017). We report the mean average preci- sion for the predicted bounding boxes (Aphe) for different IoU thresholds on COCO.
Method
Backbone
Params
Dataset
Epochs
AP
AP 50
AP
Object-centric DINO Caron ct al. 2021
VIT-S/16
21M
INIK
800
35.4
56.0
36.6
CrOC Stegmüiler ot al. 2023
VIT-S/16
21M
INIK
300
35.7
55.5
37.0
Time Salehi et al. 2023
VIT-S/16
21M
INIK+YTVOS
800+30
35.0
55.5
36.2
CrlBo
V/T-S/16
21M
INIK
800
35.7
Similar to CrOC, use the 2D coordinates (eatt) R22N2 of every patch/token in the joint dense representation Zea w.r.t. the upper left corner the image that generated the two views. Up to normalization constants, the distance between the jth token and the j"h centroid is given by:
(pos)
arg min E IES,
E
(17)
where Sy is the set of tokens assigned to the jth centroid from the previous iteration. It is noteworthy that the positional cost differs from that CrOC, which sets the position given cluster as the average position of the tokens assigned to it. In our approach, the distance between token and given centroid depends on the query token. This mitigates the issue of concentric objects sharing the same position.
In Table we ablate over the hyperparameter Apos. which modulates the importance of the positional cost the clustering algorithm.
C ADDITIONAL EXPERIMENTS
C.1 CONTRIBUTION OF INDIVIDUAL LOSS TERMS
Consecutive evaluations showcase the contribution of each loss in Table different downstream tasks. It follows that all losses contribute significant improvement the overall performance of CrlBo.
C.2 OBJECT DETECTION WITH YOLO-S
We consider an object detection downstream task to shed a different light on the quality of the learned embeddings. We perform end-to-end finetuning of the backbone using YOLO-S (Fang) et al. 2021) on COCO Lin et al. (2014). We assess performance based on the standard Average Precision metries, namely i.e., Apha AP and Aphh computed from bounding boxes predicted on the validation set. We strictly follow the finetuning protocol from Fang et al. (202I), with the sole modification being an adjustment to a batchsize of 32, as opposed to the originally used batchsize of 8.
Our choice of object detection method was guided by its simplicity and minimal requirement for additional parameters, aiming to capture the nuances of various pretraining methods. However, reveal that YOLO-S (Fang et al,] is relatively insensitive to the pretraining scheme and provides less information than anticipated. In hindsight, it might have been more judicious start
16

---

Published as a conference paper at ICLR 2024
query
clustering
object-level cross-Image matches
Figure 4: Visualization of cross-image object-level matchings on COCO. For a given query view (considered view here), object representations are computed via clustering. For each object- representation ci (highlighted in unique colors), its nearest neighbor an(c),M]] in the memory bank M2 1 visualized. In this visualization, K- 12.
with the parameters specific to YOLO-S already trained. Indeed, this approach involves appending hundred detection tokens to the image tokens before transmitting the resulting sequence through the transformer encoder. As such, earlier layers are at risk of receiving strong and biased gradients the beginning of the finetuning phase, potentially overriding the properties learned during pretraining and explaining the observed results. On the other hand, the results in Table 6 are typically in the ballpark of those reported the original paper (Fang et al. 202I).
C.3 SEGMENTATION WITH A LINEAR HEAD AND CONCATENATION
We replicate the same linear segmentation experiment from the main text (see Section 4.2 but we concatenate the spatial tokens from the last 4 layers of the ViTs in Table 7 This scenario yields improved overall performance, but it's not clear whether these improvements result from the fact that different layers of the ViT encode different information or from the increased number of trainable parameters.
C.4 CYCLE CONSISTENCY BRINGS STABILITY
We evaluate the effect of the cycle consistency criterion by comparing the results of various pretrain- ing settings with and without the bootstrapping condition. Training is performed on ImageNet- for 300 epochs for every combination of the following hyperparameters:
K: {4,8,12}.
Apas: 21..
Table Linear segmentation with frozen backbones. The linear decoder from Seg- menter (Strudel et al. (021) is trained on the frozen and concatenated spatial features from the last layers of ViTs pretrained with various self-supervised learning methods. We report the mloU scores achieved on the validation sets of 4 different datasets. Publicly available checkpoints are used when possible, and our reproduced results are indicated with a f symbol.
Methed
Ruckhane Parame
Frechs
Context
VDC
COCO-Stuf 164K
Sceme-rontic
DINO
Aenon cl 2021 2021
VIT S/16
21M
300
COCO
28,9
47.4
21.6
20.3
MAC
S/16
21M
380
COCO
29.4
13.0
13.5
CUCO
25.9
Object centric
Caron
Time/
Si00 20 2M0
S/16
21M
300+30
INIK-YTVOS
41.9
71,9
34.5
32.3
CrIBo
VIT-S/16
21M
800
INIK
42.4
75.2
34.6
33.4
DINO
Capos at 2021
VIT-Bn6
K3M
400
INTK
74,2
29.9
34.5
MAF
Inrerain 2054
VIT-R/16
85M
1690
INIK
32.7
53.0
19.7
23.5
CriBo
V/1-8116 B/16
S5M
400
INTK
430
75.7
36.5
35.4
17

---

Published as a conference paper at ICLR 2024
query
clustering
object-level cross-image matches
Figure 5: Visualization of cross-image object-level matchings on ImageNet-Ik. For a given query view (considered as view here), object representations are computed via clustering, For each objecttrepresentation of (highlighted in unique colors), its nearest neighbor nn(c),M,) in the memory bank Mi is visualized. In this visualization, K 12.
S: (25k, 50k3.
We then evaluate the resulting models with the dense nearest neighbor retrieval task (see Sec- tion[4.1] In Table we report the aggregated mloU results for two datasets (Pascal VOC 2012 (Ev- eringham al.) and ADE20K (Zhou et al. 2017)) and the settings, i.e., with and without the cycle consistency condition. The aggregation is either the minimum, the maximum, or the average of all the results. As can be observed Table & per se, but stabilizes the training procedure. Indeed, under the systematic bootstrapping regime, the performance can be unpredictable, as exemplified by the large variations of mloU scores. This happens when nearest neighbors are determined based on low-level cues, scenario that adaptive bootstrapping strives to prevent through employment cycle consistency.
the proposed criterion does not bring improvements
C.5 VISUALIZZATIN OF CROSS-IMAGE ORJECT-LEVEL MATCHINGS
Figure and Figure show the cross-image object-level matchings during a pretraining on COCO and ImageNet-Ik, respectively. Even though ImageNet- Ik is object-centric dataset, classes can always be decomposed into finer grained classes. For example, the first image with the bird has tree branches present, which CrlBo does not fail to recognize.
C.6 BOOTSTRAPPING RATIO
The evolution of the bootstrapping ratio over the training epochs is illustrated in Figure Using the notations from the paper, the bootstrapping ratio corresponds to approximately Zi+Za averaged over the whole dataset. Zı and Zg are the normalization constants from Equation 88. In practice, the number of object representations in image is slightly smaller than K as discard the ones that do not span both views. At the initial epoch, the encoders are randomly initialized, making it very hard for the object representations to satisfy the cycle consistency condition. However, learning progresses, representations become more semantically driven, and more bootstrapping is allowed. It
Table 8: Ablation study on the adaptive bootstrapping using the cycle consistency condition. Results are aggregated over multiple hyperparameters. Trained on ImageNet-Ik for 300 epochs and evaluated using dense nearest neighbor retrieval (Tab.
Pascal VOC
ADE20k
Bootstrap criterion
min. mloU
max. mloU
avg. mloU
min. mloU
max. mlnUi
avg. mloU
None
49,1
71.2
68.8
16.4
27.2
25.7
cyce-coonsstenccy
68.0
71.5
69.9
24.8
27.6
26.0
18

---

Published as a conference paper at ICLR 2024
Sootstrapping ratio [%)
70-
40
Km4
30
K=16
20
K-32
K-64
300
Epoch
Figure 6: Bootstrapping ratio over the epochs. More bootstrapping is allowed as the training progresses, and the representations become more semantically driven.
Table 9: High-level profiling of CriBo. Experiments are run on a single node with 4x AMD MI250x (2 compute die per GPU ie., worldsi ze = 8) with a memory usage of 43.5 GB per compute die. The backbone is a ViT-B/16 and the batchsize is set to 128 per compute die i.e. 1024 total and 12. CrIBo specific operations are highlighted (operations specific to object-level nearest neighbors).
Description
Operation
Absolute time per iteration [ms]
Relative time [%]
Forward pass
f((-) + f.(+) +h.()
453.4
41.7
Clustering
Q' Q'= C())
82.2
7.6
NN retrieval
nn(.) + (pdate(M))
47.5
4,4
Object projections
he(.) + ha(:)
11.2
Weights update
backprop. + RMA
493.1
45.3
total
1087.4
100
worth noting that a higher results in a lower bootstrapping ratio as both the batch and the queue are larger (in terms of object representations), which imposes more stringent condition to meet.
C.7 HIGH--EEVEL PROFILING
detailed timing analysis of CrlBo is performed Table 9 worth noting that CriBo-specific operations take less than 15% of the total time.
C.8 RUNTIME COMPARISON
runtime comparison with other methods can be found in Table TO CrIBo is faster than DINO and slower than MAE. Note that the quantitative comparisons in our paper are done with MAE trained for 1600 epochs and DINO/CrIBo trained for 400 epochs. This leads to similar total computational budget.
Table 10: Runtime comparison. Experiments are on single node with 4x AMD MI250x (2 compute die per GPU i.c worldsize = 8).
Method
Time per epoch [minutes::econds]
Memory per compute die [GB]
Batchsize
MAE
04:45
36
4096
DINO
29:43
45
1024
CrIBo 12)
23:58
1024
CrlBo (K 32)
28:22
~ 60
1024
19

---

Published as a conference paper at ICLR 2024
Table 11: We investigate the usage of supervision in CriBo. To get sense of how supervision could help CrlBo, we ablate over the usage of labels for the clustering step and for the bootstrapping
criterion. The hyperparameters (where applicable) are set (Apos: S, K) (2.0, 25k, 64).
Supervision
Patch-level NN
Clustering
Bootstrapping
Pascal VOC
ADE20k
52.5
21.2
/
X
52.3
20.2
X
/
59.5
23.4
X
X
58.1
23.4
C.9 SUPERVISED ORACLES
ablation using different supervised oracles is shown in Table [1 The supervision takes place at two different levels: 1) the clustering and 2) the bootstrapping criterion.
1. Instead of using an unsupervised clustering algorithm for computing object representations, the semantic segmentation labels from COCO and COCO-Stuff are used. The masks are downsampled (16x) to match the dense representation output resolution. Given an image, the object representation associated with given mask label is the average pooling of the patches assigned to the label.
2. Instead of using the yyclecconsistency condition for discarding supposed invalid pairs of NNs, the downsampled (16x) semantic masks of COCO and COCO-Stuff are used. A pair NN object representations is deemed valid if the union of all patch labels is the same in both object representations.
Surprisingly, using unsupervised clustering results in better performance than using the labels from COCO. In hindsight, this confirms again that the overclustering regime is where CrIBo is most at ease. Indeed, the granularity the mask annotations in COCO is not particularly high. The results would have probably been different using finer-graines annotations. However, replacing the cycle- consistency condition with supervised oracle yields modest improvements.
C.10 CLUSTERING ALGORITHM
The clustering algorithm plays pivotal role in the overall performance of CrlBo. We compare the clustering algorithm, detailed Appendix with a well-established alternative, i.e., the K-Means algorithm. Consequently, pretrain VIT-S/16 for 300 epochs COCO (Lin al. 2014), using memory bank of size = 25k, various numbers of clusters K along with the K-Means algorithm.
Table 12 depiets the performance achieved by each clustering algorithm on the dense nearest neigh- bor retrieval task (see Sec. 4.J). can observed that using the Sinkhorn- clustering from improvement over the K-Means algorithm. This difference is only partially explained by the lack of positional cues in K-Means as shown by the row Apos 0.0. Indeed, the Sinkhorn-basee -bsed method, deprived of positional cues, still yields better scores, confirming the utility of the non- sta-dstad clustering algorithm.
Appendix B leads
Table 12: Ablation study on the clustering algorithm.
Clustenng
Pascal
ADEZUK
Pascal
ADU2OK
VOC
ADEZUK
YOC
ADBZUK
Pascal
ADE2OK
428
125
47.6
Sintham Ch... 0.50
42.
16.7
901
173
51.4
2011
559
31.6
Suntham 2.00
40.9
I8.
$2.9
19.4
$4.
28A
$7.0
21.7
SK.1
23.4
20

---

Published as a conference paper at ICLR 2024
D DATASETS
Pascal VOC 2012 (Everingham et al.) A dataset composed of a training set includes 10,582 im- ages distributed across 21 classes, with being a background class. The validation set incorporates 1,449 images.
scene-centric dataset comprises 4,998 training images spanning 60 semantic classes (including the background). The validation set contains 5,105 images.
Pascal Context (Mottaghi al., 2014)
COCO-Stuff 164K Caesar et al. 2018 A scene-understanding dataset, featuring labels span- ning 91 "stuff" categories and 80 "things" categories. The training set composed of 118K images, while the validation set contains 5K images.
ADE20K (Zhou et al. 2017) A dataset that encompasses scenes featuring fine- fine-grained labels across 150 distinct semantic categories, making it one of the most demanding available semantic segmentation datasets. The training set comprises 20,210 210 images, and the validation set consists of 2,000 images.
21


Reviews:
[{"review_id": "XfXFzsgSZj", "reviewer": "Reviewer_FBUq", "summary": "The authors propose a shift in paradigm for self-supervised learning (SSL) methods; most SSL methods consider global (image-level) representations and are trained on object-centric datasets but the majority existing real-world images are instead composed of multiple objects (scene-centric). Their proposal is to add to the traditional cross-view consistency with cross-view object-level consistency and cross-image object-level consistency, approaching every image as a collection of objects that may also be present in other images. They show state-of-the-art performance on dense nearest neighbors retrieval and segmentation with a linear head and have SoTA-comparable performance when fine-tuned for segmentation.", "questions": "1. Given that a lot of the design comes from DINO, did the authors make use of the large-crop/small-crop protocol for teacher and student respectively? \n2. On a similar note, DINO benefits considerably from using smaller patch sizes (S/8, B/8, ...); did you experiment with this setup?\n3. A global representation is also trained but not used in any of the downstream tasks, what do the authors know of the usefulness of this representation? Is it comparable to object-centric SSL representations?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The manuscript is easy to follow and compelling to read. Preliminaries are helpful and the organization of the design into three steps is easy to follow.\n- The authors present a direct improvement on the method of Hummingbird with a more general framework for scene-centric SSL\n- The method is the SoTA on dense nearest neighbors retrieval without any fine-tuning steps\n- Empirical evidence shows that each addition was beneficial to the final performance; I have found the B.2 and B.4 experiments to be particularly beneficial to support the implementation of the cycle consistency criterium, one of the \"riskier\"/less intuitive ideas from the paper.", "weaknesses": "- While most of the paper is easy to follow, the section on cycle-consistency matching (3.2.2) is particularly convoluted; on a similar note, Figure 3 is not very intuitive on a first glance and it is hard to determine \"who is being compared to whom\" to determine consistency.\n- Experiments on linear segmentation fail to comment on direct backbone comparisons; this is important since it seems that the model is only better than TimeT (Salehi et al., 2023) -- a ViT-S/16 model when using a larger ViT-B/16 backbone; this is probably due to the larger training set used but still deserves a mention.\n- Opinion: I disagree that the fine-tuning regime is suboptimal for comparing SSL models since this is the regime that is going to be used by partitioners when enough training data is available; I also do not think lower/comparable performance in one of the downstream tasks detracts from the manuscript. \n\nExtra notes:\n- Table 6 has swapped headers (dataset and epochs)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LRhIs3EHs0", "reviewer": "Reviewer_sFy5", "summary": "This paper works on self-supervised learning from scene-centric images, and locates in between the use of object-level contrastive learning and cross-image nearest neighbours. Compared with prior related work Hummingbird, it proceeds on object regions discovered by the network online instead of pixels, enabling strong representations for both in-context learning evaluation and traditional evaluations (linear probing & fine-tuning). A novel cycle-consistency strategy, which requires NN's another view to be of consistent semantic with another view's NN, creates positive pairs with reasonable variation, and helps improve representation. Experiment results are supportive.", "questions": "Typo: \"Finetuning with Segmnenter\" -> segmenter (page 14)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*Originality*: The core idea of this paper is novel and original. Though the overall framework still fits in common practices of object-level contrastive learning, the integration of nearest neighbours and the design of cycle consistency are elegant and useful.\n\n*Quality*: The proposed method is well-motivated and extensively evaluated. Its relationship to prior works on different topics is also clearly discussed. I also like the discussions on over-clustering and how it helps scene-centric learning (compositionality), which provided some new insight.\n\n*Clarity*: The delivery is very clear and easy to understand, I did not find issues in understanding.\n\n*Significance*: This work is inspiring and could be helpful for both in-context scene understanding and representation learning.", "weaknesses": "- (minor) It would be better if a comparison in GPU memory / time cost in both pre-training and in-context inference stages. \n- (minor) The comparison of scene-centric learning is limited to those with ViT-based architectures. If possible, other closely related works on this topic should also be discussed and compared (eg, ORL, SlotCon, COMUS). For instance, I find from SlotCon similar visualizations as fig.3, and also similar model-discovered compositional concepts (eg, parts). Is it possible to apply in-context learning evaluation to these prior works?\n\nReferences:\n- ORL: Unsupervised object-level representation learning from scene images, NeurIPS'21\n- SlotCon: Self-supervised visual representation learning with semantic grouping, NeurIPS'22\n- COMUS: Unsupervised semantic segmentation with self-supervised object-centric representation, ICLR'23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OU0aaJNnFW", "reviewer": "Reviewer_w5sb", "summary": "This work introduces Cross-Image Object-Level Bootstrapping (CrIBo), a self-supervised visual representation pre-training approach that enforces object-level representation consistency across different images. Different from existing self-supervised pre-training strategies that only considers image-level representation consistency or cross-view object-level representation consistency, this work also aligns representations of semantically similar objects from different images. By introducing this cross-image object-level self-supervision, CrIBo pre-trains Vision Transformers (ViT) with improved visual representation quality, as demonstrated by the semantic segmentation experiments in various settings, including nearest neighbor retrieval, linear probing, and fine-tuning.", "questions": "- Sec 3.1: In the description of object representation, what does “aggregating” exactly mean? Average pooling?\n\n- Sec 3.2.2: When the memory bank is full, are the oldest samples replaced? Or is there a more sophisticated replacement strategy?\n\n- Table 3: In this fine-tuning experiment, CrIBo pre-trained ViT-B underperforms ViT-S. This trend is different from previous experiments. Why?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- For the first time, this method introduces representation consistency of semantically similar objects from different images into self-supervised pre-training. This would be helpful for performing pre-training on image datasets of complex scenes.\n\n- By leveraging the proposed cycle-consistency condition, trivial matchings between objects can be avoided and the visual encoder is enforced to learn more abstract semantic representations. This design can be helpful for future self-supervised pre-training approaches.\n\n- The writing and figure illustrations are clear. I enjoyed reading this paper.", "weaknesses": "- Inadequate experiments: The experiments only consider the semantic segmentation task, which is just one example of object-centric visual recognition tasks. It is strongly suggested to show the generalizability of CrIBO to object detection, instance segmentation, panoptic segmentation, etc.\n\n- Efficiency comparison: Since CrIBO introduces additional operations like clustering and NN retrieval, it is suggested to compare CrIBO and previous methods in terms of the training efficiency. For example, time per epoch may be listed in Table 1.\n\n- Fixed-$K$ clustering: The clustering algorithm (Sec 3.2.1), which comes from a previous work CrOC, is adopted here to identify $K$ objects and $K$ object-level representations in each view. My concern with this clustering algorithm (and other similar methods like K-means as in ODIN [1]) is that it always generates a fixed number of objects per image, regardless of the contents in the image under consideration. One simple image may contain less than $K$ objects; another complex scene may consist of much more than $K$ objects. Thus, it may be complicated to decide an optimal $K$ for a given pre-training dataset. Furthermore, even though an optimal $K$ is found, it can be sub-optimal for some images in the dataset. Would it be possible to improve over such fixed-$K$ clustering algorithms?\n\n- Early-stage training signals: As introduced in Sec 3.2.2, a cycle-consistency criterion is applied to decide whether a pair of object representations is adopted in training. I would assume that at the early training stage, the representations are nearly random since the visual encoder is just initialized. Therefore, there may be only few or even zero object pairs that can pass the test. If I understand Figure 5 correctly, it supports this assumption. Would this phenomenon significantly reduce the available training signals and slow down the early-stage learning?\n\n[1] Olivier J. Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, Relja Arandjelović. Object discovery and representation networks. In ECCV, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "73DVcZtK7o", "reviewer": "Reviewer_4jZr", "summary": "The authors propose a self-supervised cross-image object-level bootstrapping method for dense visual representation learning. Specifically, the authors first use clustering algorithms to cluster dense features to obtain object features. Afterwards, the authors use object features from one view to retrieve object-level nearest neighbors from the memory bank, and then enforce consistency between each retrieved neighbor and its corresponding object feature from another view. Extensive experiments on multiple dense downstream tasks demonstrate the superiority of the proposed cross-image object-level learning paradigm.", "questions": "I have some additional questions and suggestions:\n\n-\tThe authors use the clustering algorithm to group dense features in the feature space to obtain object representations. What if using some heuristic algorithms (e.g., Multiscale Combinatorial Grouping [6]) to directly produce object masks on the input images first and then obtaining the corresponding object representations in the feature space (like what is done in DetCon [7]). Will the grouping in the feature space be better than the grouping in the image space? Furthermore, the authors may also want to consider using ground-truth object masks (e.g., mask annotations from COCO) to replace the clustering algorithm to see whether the supervised object annotations are the upper bound of the proposed method or the unsupervised grouping could even exceed the supervised counterpart. These experiments can add more insights to the paper.\n\n-\tAccording to Table 4, using a large memory bank size tends to improve the performance. What if using a memory bank size larger than 25k? Given the current ablation results, it seems that the performance does not saturate and may be further improved with a larger memory bank size.\n\n-\tApart from ViT, is CrIBo applicable to other backbone architectures (e.g., CNN)?\n\n\n**References:**\n\n[1] Unsupervised Object-Level Representation Learning from Scene Images. In NeurIPS, 2021.\n\n[2] Local Aggregation for Unsupervised Learning of Visual Embeddings. In ICCV, 2019.\n\n[3] Delving into Inter-Image Invariance for Unsupervised Visual Representations. In IJCV, 2022.\n\n[4] Mine Your Own View: Self-Supervised Learning Through Across-Sample Prediction. In arXiv, 2021.\n\n[5] CrOC: Cross-View Online Clustering for Dense Visual Representation Learning. In CVPR, 2023.\n\n[6] Multiscale Combinatorial Grouping. In CVPR, 2014.\n\n[7] Efficient Visual Pretraining with Contrastive Detection. In ICCV, 2021.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe paper is well-motivated. The authors focus on two kinds of self-supervision in SSL: (i) cross-image self-supervision, and (ii) object-level self-supervision. Compared with cross-view self-supervision, cross-image self-supervision can provide more natural and diverse inter-image invariance. Compared with image-level self-supervision, object-level self-supervision is finer-grained and more suitable for scene-centric datasets. Based on this motivation, the authors extend existing cross-image SSL from the image level to the object level.\n\n-\tThe paper is generally well-written and easy to follow.\n\n-\tThe authors conduct extensive experiments and demonstrate promising results on various dense downstream tasks.", "weaknesses": "-\tThe authors claim that CrIBo is the first SSL approach that explicitly enforces cross-image consistency at the object/object-part level. However, some prior SSL works (e.g., [1]) have already performed cross-image SSL at the object/object-part level. Specifically, [1] enforces cross-image object-level consistency with KNN-retrieved cross-image object-instance pairs. The main difference is that [1] uses region proposal algorithms to produce region-level object features, whereas CrIBo uses clustering algorithms to produce pixel-level object features. Therefore, I suggest the authors discuss the differences with [1] and modify the claim accordingly.\n\n-\tFor cross-image self-supervision, there are several closely related works (e.g., [2, 3, 4]) that should also be discussed in Sec. 2.\n\n-\tIt seems that CrIBo does not perform very well in the fine-tuning evaluation. For example, as shown in Table 3, CrIBo performs worse than MAE when pre-trained on IN1K. Using a larger ViT-B/16 backbone even degrades the performance. Could the authors provide some explanations on this?\n\n-\tThe authors use the clustering algorithm in [5]. Could the authors provide the justification on this? What if other clustering algorithms (e.g., k-means) are used? Since the quality of object representations depends on the clustering algorithms, an ablation study on different clustering algorithms would be an interesting experiment to explore.\n\n-\tIt seems that CrIBo tends to induce heavy computational costs. A computational cost comparison (e.g., training time, GPU memory) with previous methods is preferred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)