Self-Supervised Visual Representation Learning with Semantic Grouping
Authors: Xin Wen
Bingchen Zhao
Anlin Zheng
Xiangyu Zhang
XIAOJUAN QI
Venue: NeurIPS 2022 Accept (2022)
Keywords: self-supervised learning
representation learning
object discovery

Abstract: In this paper, we tackle the problem of learning visual representations from unlabeled scene-centric data. Existing works have demonstrated the potential of utilizing the underlying complex structure within scene-centric data; still, they commonly rely on hand-crafted objectness priors or specialized pretext tasks to build a learning framework, which may harm generalizability. Instead, we propose contrastive learning from data-driven semantic slots, namely SlotCon, for joint semantic grouping and representation learning. The semantic grouping is performed by assigning pixels to a set of learnable prototypes, which can adapt to each sample by attentive pooling over the feature and form new slots. Based on the learned data-dependent slots, a contrastive objective is employed for representation learning, which enhances the discriminability of features, and conversely facilitates grouping semantically coherent pixels together. Compared with previous efforts, by simultaneously optimizing the two coupled objectives of semantic grouping and contrastive learning, our approach bypasses the disadvantages of hand-crafted priors and is able to learn object/group-level representations from scene-centric images. Experiments show our approach effectively decomposes complex scenes into semantic groups for feature learning and significantly benefits downstream tasks, including object detection, instance segmentation, and semantic segmentation. Code is available at: https://github.com/CVMI-Lab/SlotCon.

Self-Supervised Visual Representation Learning with Semantic Grouping
Xin Wen'
Bingchen Zhao
Anlin Zheng!:4
Xiangyu Zhang'
Xiaojuan Qi'
Universiity of Hong Kong
Universityy of Edinburgh
LunarAI *MEGVII Technology
{wenxin, xjqi)@eee.hku hku.
zhaobc. gm@gmail.com
(zhenganlin, zhangxiangyu) www.eii.eeiiiiooooeeeeeeeeeeeeee
Abstract
In this paper, we tackle the problem of learning visual representations from un- labeled scene-centric data. Existing works have demonstrated the potential of utilizing the underlying complex structure within scene-centric data; still, they commonly rely on hand-crafted objectness priors or specialized pretext tasks build learning framework, which may harm generalizability. Instead, we pro- pose contrastive learning from data-driven semantic slots, namely SlotCon, for joint semantic grouping and representation learning. The semantic grouping is performed by assigning pixels to set of learnable prototypes, which can adapt each sample by attentive pooling over the feature and form new slots. Based on the learned data-dependent slots, a contrastive objective is employed for representation learning, which enhances the discriminability of features, and conversely facilitates grouping semantically coherent pixels together. Compared with previous efforts, by simultaneously optimizing the two coupled objectives of semantic grouping and contrastive learning, our approach bypasses the disadvantages hand-crafted priors and is able to learn object/groupp-Ket ages. Experiments show our approach effectively decomposes complex scenes into semantic groups for feature learning and significantly benefits downstream tasks, including object detection, instance segmentation, and semantic segmentation. Code is available https: / giiggtuu........ com/CVMI-Lab/ Lab/SlotCSn....
representations from centric im-
Introduction
Existing self-supervised approaches have demonstrated that visual representations can be learned from unlabeled data by constructing pretexts such transformation prediction [22], instance dis- crimination [70, 31], and masked image modeling [2, 30, 67], etc. Among them, approaches based instance discrimination [28, 6, 7, 11], which treat each image single class and employ contrastive learning objective for training, have attained remarkable success and are beneficial to boost performance on many downstream tasks.
However, this success is largely built upon the well-curated object-centric dataset ImageNet [15], which has large gap with the real-word wal-worldd data for downstream applications, such city scenes [13] or crowd scenes [46]. Directly applying the instance discrimination pretext to these real-world data by simply treating the scene as whole overlooks its intrinsic structures (e.g., multiple objects and complex layouts) and thus will limit the potential of pre-training with scene-centric data [61]. This leads focus: learning visual representations from unlabeled scene-centric data.
Recent efforts to address this problem can coarsely categorized into two types of research. One stream extends the instance discrimination task to pixel level for dense representation learning [54,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Pooling
Grouping
Objoct-cantric
Consistency
Consistency
Consistency
Pooling
Grouping
Scene-centric
Views
Image-level
Pixel-level
Object-level (Ours)
Somantic Grouping
Figure I: Left:
Unlike object-centric images that highlight one iconic object, scene-centric images often contain multiple objects with complex layouts. This adds to the data diversity and increases the potential of image representations, yet it challenges previous learning paradigms that simply treat an image as a whole or individual pixels. Middle: Contrastive-lcarinng objectives built upon different levels of image representations. in which object-level contrastive learning is viewed as the most suitable solution for scene-centric data, Right: We jointly learn a set of semantic prototypes in order to perform semantic grouping over the pixel-level representations and form object-centric slots.
66, 73], which shows strong performance downstream dense prediction tasks. Yet, these methods still lack the ability to model object-level relationships presented in scene-centric data, which is crucial for learning representations. Although another stream of works attempts perform object- level representation learning, most of them still heavily rely on Nomainn-seciif priors to discover objects, which include saliency estimators [62, 56], unsupervised object proposal algorithms [68, 72], hand-crafted segmentation algorithms [81, 34] or unsupervised clustering [35]. However, if the representation supervised by hand-crafted objectness priors, it will be discouraged from learning objectness from the data itself and prone to mistakes from priors. Therefore, the capability and generalizability of the representation will limited. In this work, we aim at a fully learnable and data-driven approach to enable learning representations from scene-centric data for enhanced effectiveness, transferability and generalizability.
We propose contrastive learning from data-driven semantic slots, namely SlotCon, for joint semantic grouping and representation learning. Semantic grouping is formulated as feature-space pixel-level deep clustering problem where the cluster centers are initialized as a set of learnable semantic prototypes shared by the dataset, and grouping is achieved by assigning pixels to clusters. The cluster centers can then adapt to each sample by softly assigning pixels into cluster centers and aggregating their features via attentive pooling to form new ones, also called slots. Further, upon the learned slots from two random views of one image, contrastive objective, which attempts pull positive slots (i.e., slots from the same prototype and sample) together and push away negative ones, is employed for representation learning. The optimized representations will enhance the discriminability of prototypes and slots, which conversely facilitates grouping semantically coherent pixels together. Compared with previous efforts. by simultaneously optimizing the two coupled objectives of semantic grouping and contrastive representation learning, our method bypasses the disadvantages of hand-crafted objectness priors' and is able to discover object/groupp-lved representations from scene-centric images.
We extensively assess the representation learning ability of our model by conducting transfer learning evaluation on COCO [46] object detection, instance segmentation, and semantic segmentation on Cityscapes [13], PASCAL VOC [20], and ADE20K [83]. Our method shows strong results with both COCO pre-training and ImageNet-1K pre-training, bridging the gap between scene-centric and object-centric pre-training. As byproduct, our method also achieves notable performance in unsupervised segmentation, showing strong ability in semantic concept discovery.
In summary, our main contributions this paper are: 1) We show that the decomposition of natural scenes (semantic grouping) can be done in a learnable fashion and jointly optimized with the representations from seratch. 2) We demonstrate that semantic grouping can bring object-centric representation learning to large-scale real-world scenarios. 3) Combining semantic grouping and representation learning, we unleash the potential of scene-centric pre-training, largely close its gap with object-centric pre-training and achieve state-of-the-arr results in various downstream tasks.
TTinns be different if the data not reliable, Section G in the appendix for details.

---

2
Related work
Our work is in the domain of self-supervised visual representation learning, where the goal is to learn visual representations without human annotations. We briefly review relevant works below.
Image-level self-supervised learning aims at learning visual representations by treating each image one data sample. To this end, a series of pretext tasks are designed in which the lahels readily available without human annotations. Early explorations range from low-level pixel-wise reconstruction tasks that include denoising [64], inpainting [55], and cross-channe prediction [80] to higher-level instance discrimination [17], rotation prediction [22], context prediction [16], jigsaw puzzle [52], counting [53], and colorization [79]. Modern variants of instance discrimination [17] equipped with contrastive learning [60, 36] have shown strong potential in learning transferable visual representations [70, 8, 31, 58, 84, 82]. Other works differ in their learning objectives but still treat an image as a whole [28, 10, 78]. To further utilize the complex structure in natural images, some works exploit local crops [6, 71, 74, 61] while others either step to pixel- or object-level, detailed as follows.
Pixel-level contrastive learning extends the instance discrimination task from image-level feature vectors feature maps [54, 47, 66, 73]. Their main differences lie in the way positive pixel-pairs are matched (spatial adjacency [54. 73], feature-space centure--pace nearest-neighbor [66], sink-horn matching [47]), and the image-level baseline they build upon (MoCo v2 [54, 66], BYOL [47, 73).. Their pixel- level objective naturally helps learn dense representations that are favorable for dense prediction downstream tasks but lacks the grasp of holistic semantics and commonly require an auxiliary
image-level loss to attain stronger performance [66, 73].
Object-level contrastive learning first discovers the objects in images and applies the contrastive objective over them, achieving a good balance in fine-grane tne-grained structure and holistic semantics, yielding strong empirical gains with both object-centric [34, 35] and scene-centric data [72]. The key issue lies in finding objects in an image without supervision. Current works, however, still heavily rely on heuristic strategies that include saliency estimators [62, 56), selective-search [68, 72], hand-crafted segmentation algorithms [81, 34), or k-means clustering [35]. In contrast, our semantic grouping method is fully learnable and end to end, ensuring transfcrability and simplicity.
Unsupervised semantic segmentation is emerging task that targets addressing semantic segmen- tation with only unlabeled images. The first attempt of IIC [40] maximizes the mutual information of augmented image patches, and later works [38, 62] rely on saliency estimators as a prior to bootstrap semantic pixel representations. Recently, PiCIE [39] adopt pixel-level deep clustering [5] to cluster the pixels into semantic groups, which SegDiscover [37] further improves by adopting
sapperpixels... On the other hand, Leopart [85]. STEGO [29], and FreeSOLO [65] exploits pre-trained networks' attention maps for objectness. Still, they commonly rely for initialization, while our method trained fully from scratch.
supervis))) pre- pre-trained network
Object-centric representation learning is viewed robust and interpretable machine learning algorithms [27]. Towards unsupervised object-centric representation learning, a series of works have been proposed [26, 3, 19, 48, 25, 24, 23]. Directly extracting objects from images is challenging due to the lack of supervision and thus prior works have long been restricted synthetic data [42, 51, 1]. Recent works try step toward roal-world videos; however, they either adopt motion [75, 41] or depth [18] as a cue for objectness to solve this problem. Instead, our method shows that we can first explicitly optimize for desirable properties of clusters (that can describe an object) over the dataset, then retrieve the objects from an image with the learned prototypes. More importantly, first shows the possibility of learning object-centric representations from large-scale unlabelled scene-centric natural images.
essential component of data-efficient,
3 Method
3.1 Semantic grouping with pixel-level deep clustering
Given a dataset DD of unlabeled images, we aim learning set of prototypes that classifies each pixel into meaningful group, such that pixels within the same group semantic-consistent (have
3

---

Imamz Augmerration (RolAlias Fipi
Prefictor
Aug.
Sy
View
S
EMA
Pos. Neg. Allat
Image
Aug.
f
%
Imvere Augmeraation (RelAligs)
Deroc
Scaled Dut
Encoder Puojocos
Prijeciion
Product
Assignment
Sleta
Figure 2: Overview of our proposed framework. Based on a shared pixel embedding function, the model learns classify pixels into groups according their feature similarity in pixel-level deep clustering fashion (Sec. 3.1) 3.1); the model produces group-level feature vectors (slots) through attentive pooling over the feature maps, and further performs group-level contrastive learning (Sec. 3.2). We omit the symmetrized loss computed by swapping the two views for simplicity. (best viewed in color)
similar feature representations), and pixels between different groups are semantic-incoheren We find that this problem can be viewed as unsupervised semantic segmentation [22, 39], and solved with pixel-Ievel deep clustering [5, 76, 6, 7].
Intuitively, a mcaningful semantic grouping should be invariant to data augmentations. Thus, for different augmentations of the same image, enforce the pixels that lie in the same location to have similar assignment scores w.r.t. the same set of cluster centers (prototypes). Besides consistent grouping, the groups should be different from each other ensure that the learned representations discriminative and avoid trivial solutions, e.g., identical features. Together with standard techniques used in self-supere self-supervised learning (e.g., non-linear projector and momentum teacher [28, 7], etc.), this leads to the following framework.
Specifically, as illustrated in Figure 2, method contains two neural networks: the student network is parameterized by and include an encoder fa. a projector ge and a set of K learnable prototypes ; the teacher network holds the same architecture with the student,
.......... S. = [&). sy....,, $ E xx but uses a different set of weights & that updates as an exponential moving average of 8. Given applied to produce augmented views v
an input image æ, two random augmentations and v². Each augmented view v' E {v', v²} then encoded with an encoder f into a hidden feature map h' R#xWxD, and then transformed with a multilayer perceptron (MLP) g to get z = g g (h') E HRHxWxD, We then compute the assignment P/ of the projections z/ with the corresponding prototypes So, and enforce it to match the assignment O produced with another view o" by the teacher network. More precisely, with fz-normalized projections Σ = = |=| and prototypes $/||s||
d softmax
( -c) P/- softmax 2,,33,/7, RRHWwx (1)
with T., > O temperature parameters that control the sharpness of the output distribution of the two networks. The c can be omitted and will be explained later. Note that the scale or layout of the two feature maps z// and =! can be inconsistent due to geometric image augmentations, e.g., random crop, scale or flip. So we perform the inverse augmentation process (including RolAlign [32] and an optional flipping, details provided in Section A.l in the appendix) the predicted assignments to
align their spatial locations: Qi = invaug (6)) (o)). Pb invaug (P%). The inverse augmentation is
performed on the assignments Q rather than the projections z to keep the context information out of the overlapping area for slot generation, which will be detailed in Section 3.2.
Based on the aligned assignments. we apply the cross-entropy loss C. (qE, (g) - a log p to enforce the consistency in assignment score between spatial-aligned pixels from different views. The cross-entropy loss is averaged over all spatial locations to produce the grouping loss:
Grop
CE Co.
@ Pbl.ii) + C P.l..))
(2)
H W

---

Directly optimizing the above objective resembles an unsupervised variant of Mean Teacher [57], which collapses as shown in [28]. In order to avoid collapsing, we follow [7] to maintain mean logir E IR& and reduce it when producing the teacher assignments De. as indicated in Eq. 1. The mean logit stores an exponential moving average of all the logits produced by the teacher network:
c c x.c (1- Ao)
1
Σ 1.7.
(3)
B
where B stands for the batch size. Intuitively, reducing the mean logit amplifies the difference in assignment between different pixels, preventing all pixels from being assigned to the same prototype. Besides that, the teacher temperature is smaller than the student temperature т, to produce sharper target and avoid uniform assignments. Both operations help avoid collapse and force the network to learn meaningful semantic grouping.
Discussion with DINO. The resulting solution for semantic grouping may seem like a naive extension of DINO [7]. However, this is not the whole picture. DINO is an image-level representation learning approach that adopts a large number of prototypes (e.g., 65536), while our objective is built on pixel-level representations and is tailored to learn meaningful semantic groups. Intuitively, DINO captures scene semantics that can be more complex and thus requires more prototypes to represent all scene-level variations, while our method learns object-level semantics which can be composed to depict a complicated scene and thus only requires a small number of prototypes (e.g.. 256 for COCO). Besides, the representation learning objective of DINO is built on prototypes that are shared by the whole dataset, while our method instead adapts prototypes for each image and performs contrastive learning over groups (detailed in Section 3.2). In summary, we present a novel view for the decoupling of pixel-level online clustering (i.e., semantic grouping) and object-level representation learning.
3.2 Group-level representation learning by contrasting slots
Inspired by Slot Attention [48], we then reuse the assignments computed by the semantic grouping module in Eq. 1 to perform attentive pooling over the dense projections to produce group-level feature vectors (rephrased as slots), as shown in Figure 2. Intuitively, as the softmax normalization applies to the slot dimension, the attention coefficients sum vector. As a result, the soft assignments AA of the dense projections z w.r.t. the corresponding prototypes S can also be viewed as the attention coefficients that describe how the prototypes decompose the dense projections into non-overlapping groups This inspires us to decompose the dense projections with attentive pooling following [48]. Specifically, for a dense projection z/p produced from view v', we extract K slots:
for each individual input feature
S/ = Abb. A.
Adl.do 2f ERXDD .A4- =softmax (z6.3,/7) 37) Hxwx
K
(4)
where denotes the Hadamard product, and a similar operation applies for the teacher network to produce S Since the initial slots are shared by the whole dataset, the corresponding semantic may be missing in a specific view v', thus producing redundant slots. Therefore, we compute the following binary indicator 1' to mask out the slots that fail to occupy dominating pixel:
k.l I b.
3 such that argmax (Ab) [ K. K
(5)
and n is computed similarly. Following the literature in object-level SSL [62, 34, 72, 35], we then apply a contrastive learning objective to discriminate the slot that holds the same semantic across views from distracting slots with the InfoNCE [60] loss:
CInfONCE Cl
(5
1 K
log
1 k
R.l'
exp
qo
k.l
86
k.l'
/Te
(6)
Σn
k. M 1
exp
go
8
3
/Te
This objective helps maximize the similarity between different views of the same slot, while minimiz- ing the similarity between slots from another view with different semantics and all slots from other
5

---

images. Note that here an additional predictor with the same architecture as the projector ge is applied to the slots Sn as empirically it yields stronger performance [11, 68, 35]. And the resulting slot-level contrastive loss also follows a symmetric design like Fq. 2:
Slot
cInfoNCE (S), + La,
Cla (5).
L,g
3.3 The overall optimization objective
We jointly optimize the semantic grouping objective (Eq. 2) and the group-level contrastive learning objective (Eq. 7), controlled with balancing factor Xgi
Overall 5,5
Group + A
At each training step, the student network is optimized with gradients from the overall loss function:
0 < optimizer Vo ) where ry denotes the learning rate; and the teacher network updates as an exponential moving average of the student network: & <- Agg + (1 X²)0, with X denoting the momentum value. After training, only the teacher encoder fs is kept for downstream tasks.
4 Experiments
4.1
Implementation details
Pre-training datasets. We pre-train models on COCO train2017 [46] and ImageNet-IK [15], respectively, COCO train2017 [46] contains 1118 images of diverse scenes with objects of multi- ple scales, which is closer to real-world scenarios. In contrast, ImageNet- IK is curated object-centric dataset containing more than ~1. .28 images, which is better for evaluating model's potential with large-scale data.
Table 1: Details of the datasets used for pre-training.
Dateset
#lmg.
#Obj./Img.
#Class
ImageNet- 1K [15]
1.28M
1.7
1000
COCO [46]
118K
80
COCO+ [46]
241K
N/A
N/A
Besides, we also explore the limit of scene-centric pre-training on COCO+, i.e., COCO train2017 set plus the mnlabeled2011 set. See details in Table 1.
Data augmentation. random resized crop with random horizontal flip, followed by random color distortion, random grayscale conversion, random Gaussian blur, and solarization. The crop pairs without overlap are discarded during training.
The image augmentation setting is the same as BYOL [28]: 224 x 224-pixel
Network architecture. We adopt ResNet-50 [33] as the default encoder for fo and fe. The projector go- Is and predictor qo are MLPs whose architecture are identical that in [7] with hidden dimension of 4096 and an output dimension of 256.
Optimization. We adopt the LARS optimizer [77] pre-train the model, with batch size of 512 across eight NVIDIA 2080 Ti GPUs. Following [73], we utilize the cosine learning rate decay schedule [50] with base learning rate of 1.0, linearly scaled with the batch size (LearningRate 1.0 X BatchSize/ 256), a weight decay of 10-5, and a warm-up period of 5 epochs. Following [72, 66, 731, the model pre-trained for 800 epochs on COCO(+) and 100/200 epochs on ImageNet, respectively. Following the common practice of [72, 73]. the momentum value X of the teacher model starts from 0.99 and is gradually increased following a cosine schedule. Synchronized batch normalization and automatic mixed precision are also enabled during training.
Hyper-parameters. The temperature values т, and T. in the student and teacher model are set to 0.1 and 0.07, respectively. Besides, the center momentum xe is set to 0.9. The default number of prototypes
set to 256 for COCO(+) and 2048 for ImageNet, according to our empirical finding that setting the number of prototypes close to the number of human-annotate categories can help downstream performance. The temperature value T. for the contrastive loss set to 0.2 following []] and the default balancing ratio A is set to 0.5.

---

4.2 Evaluation protocols
Following the common practice of previous self-supervised works [66, 73, 72], we evaluate the representation ability of the pre-trained model by taking it as the backbone of downstream tasks. Specifically, we add a newly initialized task-specific head to the pre-trained model for different downstream tasks, i.e., object detection and instance segmentation on COCO [46]. and semantic segmentation on PASCAL VOC [20], Cityscapes [13], and ADE20K [83].
Object detection and instance segmentation. We train a Mask R- R-CNN [32] model with R50- FPN [45] backbone implemented in Detectron2 [69]. We fine-tune all layers end-to-end on COCO train2017 split with the standard x schedule and report AP, AP50. AP75 on the va12017 split. Following [66, 73, 72]
we train with the standard 1x schedule with SyncBN.
Semantic segmentation. The evaluation details of PASCAL VOC and Cityscapes strictly fol- low [31]. We take network to initialize the backbone of colloyyaovlluiional network [49] and fine-tune all the layers end-to-end. For PASCAL VOC, we fine-tune the model on train_ train_aug2012 aug2012 union (mloU) the va12012 set. For
set for 30k iterations and report the mean intersection Cityscapes, we fine-tune on the train_ fine set for 90k iterations and evaluate on the val fine For ADE20K, follow the standard 80k iterations schedule of MMSegmentation [12].
Unsupervised semantic segmentation. We also evaluate the model's ability of discovering se- mantic groups in complex scenes, which is accomplished by performing unsupervised semantic segmentation on COCO-Stuff [4]. We follow the common practice in this field [40, 39, 37] to merge the labels into 27 categories (15 "stuff" categories and 12 "thing" categories), and evaluate with a subset created by [40]. We perform inference with resolution 320 and number of prototypes 27 following the standard practice. The predicted labels are matched with the ground truth through Hungarian matching [43], and evaluated on mloU and pixel accuracy (pAcc).
4.3 Transfer learning results
COCO pre-training. In Table 2, we show the main results with COCO pre-training. There have been steady improvements object-level pre-training with COCO, in which the top performance methods are DetCon [44] and ORL [72], which still rely on objectness priors like selective-search [59] or hand-crafted segmentation algorithms [21], yet they still fail beat the pixel-level state-of-the-art
Table 2: Main transfer results with COCO pre-training. We report the results in COCO [46] object detection, COCO instance segmentation, and semantic segmentation in Cityscapes [13], PASCAL VOC [20] and ADE20K [83]. Compared with other image-, pixel-, and object-level self-supervised learning methods, our method shows consistent improvements over different tasks without leveraging multi-crop [6] and objectness priors. (f: re-impl. w/ official weights; t: full re-impl.)
Multi Obj. crop
COCO detection
COCO segmentation
Semantic seg. (mloU)
Method
Epochs
Prior
APb
AP
AP
Ap
AP
APm
City.
VOC
ADE
random init.
X
x
32.8
50.9
35.3
29.9
47.9
32.0
65.3
39.5
29.4
Image-level approaches
MoCo v2 [9]
38.5
42.1
34.8
55.3
37.3
Revisit. [61]
800
/
x
40.1
60.2
43.6
36.3
57.3
38.9
75.3
70.6
37.0
Pixel-level approaches
Self-EMD [47]
800
X
x
39.3
60.1
42.8
DenseCL [66]
800
X
X
39.6
59.3
43.3
35.7
56.5
38.4
75.8
71.6
37.1
PixProt [73]
800
X
X
40.5
60.5
44.0
36.6
57.8
39.0
75.2
72.0
38.3
Object Group-level approaches
DetCon 1 [34]
1000
X
39.8
59.5
43.5
35.9
56.4
38.7
76.1
70.2
38.1
O [72]
800
/
/
40.3
60.2
44.4
36.3
57.3
38.9
75.6
70.9
36.7
Ours (SlotCon)
800
X
41.0
61.1
45.0
37.0
58.3
39.8
76.2
71.6
39.0

---

PixPro [73].2 Our method alleviates such limitations and significantly improves over current object- level methods in all tasks, achieving consistent improvement over the previous approaches and even several methods that were pre-trained on the larger dataset ImageNet- K (Table 4). It is also notable that our method can achieve better performance on the largest and most challenging dataset for segmentation, ADE20K, adding to the significance of this work.
COCO+ pre-training. Table we report the results with COCO+ pre-training. The COCO+ is COCO train2017 plus unlabeled2017 set, which roughly doubles the number of training images and greatly adds to data diversity. Our method further sees notable gain in tasks with extended COCO+ data, and even shows comparable results with our best-performing model pre-trained on ImageNet- 1K (5x large of COCO+), showing the great potential of scene-centric pre-training.
Table 3: Pushing the limit of scene-centric pre-training. Our method further sees notable gain in all tasks with extended COCO+ data, showing the great potential of scene-centric pre-training.
COCO detection
COCO
segmentation
Semantic seg. (mloU)
Method
Dataset
Epochs
APh
AP
AP
Apm
AP
APT:
City.
VOC
ADE
SlotCon
COCO
41.0
61.1
45.0
37.0
58.3
39.8
71.6
39.0
SlotCon
ImageNet
100
41.4
61.6
45.6
37.2
58.5
39.9
75.4
73.1
38.6
SlotCon
ImageNet
200
41.8
62.2
45.7
37.8
59.1
40.7
76.3
75.0
38.8
ORI[772)
COCO+
800
40.6
60.8
44.5
36.7
57.9
39.3
SlotCon
COCO+
800
41.8
62.2
45.8
37.8
59.4
40.6
76.5
73.9
39.2
ImageNet-1K pre-training. In Table 4, we also benchmark our method with ImageNet- -K pre- training and show its compatibility with object-centric data. Without selective-search for object proposals and without pre-training and transferring the FPN head, our method still beats most of the current works and largely closes the gap with the detection sreection-peecializedd method SoCo.
Table 4: Main transfer results with ImageNet-Il -K pre-training. Our method is also compatible with object-centric data and shows consistent improvements over different tasks without using FPN [45] and objectness priors. (f: re-impl. w/ official weights; $: full re-impl.)
w/ FPN
Obj. Prior
COCO detection
COCO segmentation
Semantic seg. (mloU)
Method
Epochs
APh
AP
APt.
AP
AP
AP
City.
VOC
ADE
random init.
X
x
32.8
50.9
35.3
29.9
47.9
32.0
65.3
39,5
29.4
supervised
100
X
X
39.7
59.5
43.3
35.9
56.6
38.6
74.6
74.4
37.9
Image-level approaches
MoCo v2 [9]
X
X
40.4
60.1
44.2
57.2
39.2
76.2
73.7
36.9
DetCo' [71]
200
X
X
40.1
61.0
43.9
36.4
58.0
38.9
76.0
72.6
37.8
InsLoc [74]
200
/
X
40.9
60.9
44.7
36.8
57.8
39.4
75.4
72.9
37.3
Pixel-level approaches
DenseCL [66]
200
x
x
40.3
59.9
44.3
36.4
57.0
39.2
76.2
72.8
38.1
PixPro 1 [73]
100
x
x
40.7
60.5
44.8
36.8
57.4
39.7
76.8
73.9
38.2
Object Group-lever approaches
DetCon [34]
200
X
/
40.6
36.4
75.5
72.6
SoCot [68]
100
/
/
41.6
61.9
45.6
37.4
58.8
40.2
76.5
71.9
37.8
Ours (SlotCon)
100
X
X
41.4
61.6
45.6
37.2
58.5
39.9
75.4
38.6
Ours (SlotCon)
200
X
X
41.8
62.2
45.7
37.8
59.1
40.7
76.3
75.0
38.8
4.4 Unsupervised semantic segmentation results
Given our approach's 's consistent improvement in representation learning, further analysis of how well our semantic grouping component can parse scenes quantitatively and qualitatively. It should be
PixPro aggregates the global context with self attention [63], so each pixel can also be viewed as an object-level embedding.

---

Table 5: Main results in COCO-Stuff unsupervised semantic segmentation.
Imngg
murkting
Method
mloU
pAcc
Ous PAC1B
MaskContrast [62]
8.86
23.03
Gooas)
PiCTE + H. [39]
14.36
49.99
SegDiscover [37]
14.34
56.53
Ours (SlotCon)
18.26
42.36
Parmuss
clarified that this is just to help understand the learned prototypes, and not to propose new SOTA. Unlike current SSL. approaches that exhaustively enumerate the massive object proposals and report the best score 35], we follow the common practice of unsupervised semantic segmentation [40, 39] match the predicted results with the ground-truth using the Hungarian algorithm [43], where each ground-truth label is assigned to prototype mutual-exclun mutual-exclusively. For fair comparisons, the model used for evaluation is trained with K 27 match the number of categories of COCO-Stuff. As shown in Table 5, our method can surpass the previous works PICIE [39] and SegDiscover [37] with 4 points higher mIoU. Meanwhile, the pAce is lower since we train the model with lower resolution (7 x 7 v.s. 80 x 80 feature map). Besides Table 5, we also depict the visualization results, in which our method distinguishes confusing objects apart (4th column) and successfully localizes small objects (5th column). Since we only need to separate pixels with different semantics within the same image, the errors category prediction can be ignored.
4.5 Ablation study
Table 6: Ablation studies with COCO 800 epochs pre-training. We show the AP on COCO objection detection and mloU Cityscapes, PASCAL VOC, and ADE20K semantic segmentation. The default options are marked with gray background.
(a)
Number of prototypes
(b) Loss balancing
(c) Teacher temperature
K
COCO
City
VOC
ADE
Xg
COCO
City
VOC
ADE
Tt
COCO
City
VOC
ADE
128
40.7
76.4
71.9
38.5
0.3
41.0
76.1
72.1
37.9
0.04
40.4
75.5
70.2
37.9
256
41.0
76.2
71.6
39.0
0.5
41.0
76.2
71.6
39.0
0.07
41.0
76.2
71.6
39.0
512
40.9
75.6
71.6
38.9
0.7
40.5
75.2
71.5
38.4
1024
40.7
75.8
70.9
1.0
40.4
74.2
70.1
38.6
Number prototypes K. Table 6a ablates the number of prototypes, we observe that the most suitable K for COCO detection is 256, which is close its real semantic class number 172 (thing + stuff) [4]. Besides, the performance on Cityscapes and PASCAL VOC have consistent tendency to drop, while the performance ADE20K is consistently good if K is big enough. We hypothesize that a suitable
can encourage learning data data-specific semantic features, which are only helpful when the pre-training and downstream data alike (from COCO to COCO); increasing K produces fine-ge aineggrined features that may lack discriminability in semantics but hold better transferability to ADE20K that require fine- inn-grained segmentation [14].
Loss balancing weight Ay. Table 6b ablates the balancing between the semantic grouping loss and the group-level contrastive loss, where the best balance achieved with both losses treated equally. is notable that when Ag = 1.0, only the semantic grouping loss is applied, and the performance drops considerably, indicating the importance of our group-level contrastive loss for learning good representations.
Teacher temperature Tę. Table 6e ablates the teacher model's 's temperature parameter, indicating that a softer teacher distribution with 7. 0.07 helps achieve better performance.
4.6 Probing the prototypes
Finally, we analyze whether the prototypes learn semantic meanings by visualizing their nearest neighbors in COCO va12017 split. We first perform semantic grouping on each image to split them into non-overlapping groups (segments), then pool each group to a feature vector, and retrieve the

---

Figure 3: Examples of visual concepts discovered by SlotCon from the COCO va12017 split. Each column shows the top 5 segments retrieved with the same prototype, marked with reddish masks or arrows. Our method can discover visual concepts across various scenarios and semantic granularities regardless of small object size and occlusion. (best viewed in color)
top nearest-neighbor segments for each prototype according to cosine similarity. As shown in Figure 3, the prototypes well bind to semantic meanings that cover a wide range of scenarios and semantic granularities from animals, foods, and sports, to furniture, buildings, etc., localizing them well regardless of small object size and occlusion; and notably, without any human annolation.
5
Discussion on the emergence of objectness
should be clarified that the discovered objectiveness is at the semantic level, and object instances with identical semanties can be indistinguishable. Concerning the emergence of objectness, impose geometric-covariane and photometric-invaiian to decompose large complex dataset into small number of clusters through optimizing feature space and cluster centers. The emergence of objects/parts that are compositional and thus occupy reasonable number of prototypes. can be viewed as natural consequence under such conditions. Concerning granularity, our intuition is that the prototype number and the dataset distribution generate a bottleneck for the granularity of groups. For example, in the COCO dataset, with 256 prototypes, the model finds that splitting animals into cats, dots, elephants, etc., is enough and won't 't further separate them. In contrast, for humans (the most occupying category of COCO), as shown in the appendix, the model discovers not only human parts (Figure 7) but also different types of activities (Figure 8), indicating that parts are more helpful this scenario and deserve more prototypes.
as guiding cues which force the model
6 Conclusion
This work presents unified framework for joint semantic grouping and representation learning from unlabeled scene-centric images. The semantic grouping is performed by assigning pixels to a set of learnable prototypes, which can adapt to each sample by attentive pooling over the feature map and form new slots. Based on the learned data-dependent slots, a contrastive objective is employed for representation learning, enhancing features' discriminability and facilitating the grouping of semantically coherent pixels together. By simultaneously optimizing the two coupled objectives of semantic grouping and contrastive learning, the proposed approach bypasses the disadvantages of handcrafted priors and can learn object/group-level representations from scene-centric images. Experiments show the proposed approach effectively decomposes complex scenes into semantic groups for feature learning and significantly facilitates downstream tasks, including object detection, instance segmentation, and semantic segmentation.
Acknowledgments
This work has been supported by Hong Kong Research Grant Council Early Career Scheme (Grant No. 27209621). HKU Startup Fund, and HKU Seed Fund for Basic Research. The authors acknowledge SmartMore, LunarAI, and MEGVII for partial computing support.
10

---

References
[1] Federico Baldassarre and Hossein Azizpour. Towards Self-Supervised Learning of Global and Object- Centric Representations. arXiv:2203.0590 [cs], March 2022. 3
[2] Hangho Bao, Li Dong, and Furu Wei. BEIT: BERT Pre- Tre-nrain of Image Transformers. arXiv:2106.08233 [cs]. June 2021.
[3] Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised Scene Decomposition and Representation. arXiv: 190011913990 [cs, stat], January 2019. 3
[4] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari, COCO-Stuff: Thing and Stuff Classes in Context. In 2018 IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 1209-1218, June 2018 8. 7.9
[5] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep Clustering for Unsupervised Learning of Visual Features. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision ECCV 2018, pages 139-156, Cham, 2018. Springer International Publishing.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M. E. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9912-9924. Curran Associates, Inc., 2020. 7,7
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 9650-9660, 2021. 5, 6,9
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings the 37th International Conference on Machine Learning, pages 1597-1607. PMLR, November 2020." 3
[9] Xinlei Chen, Haoqi Fan. Ross Girshick, and Kaiming He. Improved Baselines with Momentum Contrastive Learning. arXiv:2003.0422 [cs], March 2020. 7,
[10] Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. In 2021 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15745- 15753, Nashville, TN, USA, June 2021. IEEE. 3
[11] Xinlei Chen, Saining Xie, and Kaiming He. Empirical Study of Training Supervisdd Vision Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640 9649, 2021. 1,
[12] MMSegmentation Contributors. MMSegmennaation: Openmmlab semantic segmentation toolbox and benchmark. https://gitee wtw.co/ooooooo open- - mml ab/ mbmmaa/mgmmmtttttiioooo. 2020. 7
[13] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler. Rodrigo Benen- son, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urhan Scene Understanding. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 3213-3223. IEEE Computer Society, June 2016. 1,2,7
[14] Quan Cui, Bingehen Zhao, Zhao-Min Chen, Borui Zhao, Renjie Song, Jiajun Liang, Boyan Zhou, and Osamu Yoshie. arXiv:2203.0580 [cx], March 2022.
Transaaiiittttaaaaaaaaiillli Trade-Off: An Information--Teerrtti Perspective.
[15] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hierarchical image database, 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, June 2009. 1,6 6
[16] Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised Visual Representation Learning by Context Prediction. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1422-1430, December 2015. 3
[17] Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. Dis- criminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Transactions Pattern Analysis and Machine Intelligence, 38(9):1734 1747, 2015. 3

---

[18] Gamaleldin F Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. +avi++: Towards end-lo-end object-centric learning from real-world videos. arXiv preprint urXiv:2206007664 2022.
[19] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative Scene Inference and Sampling with Object- Centrrc Latent Representations. In International Conference on Learning Representations, September 2019. 3
[20] Mark Everingham, S. M. Ali Eslami, Lue Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The Pascal Visual Object Classes Challenge: A Retrospective. International Journal of Computer Vision, (11(1))98-11))))) 136, January 2015. 2,7
[21] Pedro Felzenszwalb and Daniel P. Huttenlocher. Efficient Graph- Based Image Segmentation. Interna- tional Journal of Computer Vision, 59(2):167-18) 181, September 2004. 7
[22] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised Representation Learning by Predicting Image Rotations. In International Conference on Learning Representations, February 2018. 1, 3.3,4
[23] Anirudh Goyal, Aniket Rajiv Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess. Michael Curtis Mozer, and Yoshua Bengio. Neural production systems. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[24] Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Charles Blundell, Sergey Levine, Yoshua Bengio, and Michael Curtis Mozer. Factorizing declarative and procedural knowledge in structured, dynamical environments. In International Conference Learning Representations, 2021.
[25] Anirudh Goyal, Alex Lamb, Jordan Hoffmaann Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Berhhard Schölkopf. Recurrent independent mechanisms. In International Conference on Learning Representations, 2021. 3
[26] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-Object Representation Learning with Iterative Variational Inference. In Proceedings of the 36th International Conference on Machine Learning. pages 2424-2433, PMLR, May 2019.
[27] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the Binding Problem in Artificial Neural Networks. arXiv:2012.055 [es], December 2020. 3
[28] lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent a new approach to self- supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21271 --1284. Curran Associates, Inc., 2020.
1, 1, 5,6
[29] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan. Noah Snavely, and William T. Freeman. Unsu- pervised semantic segmentation by distilling feature correspondences. In International Conference on Learning Representations, 2022.
[30] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners, arXiv:2111.06377 [cs]. December 2021.
[31] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsuper- vised Visual Representation Learning. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9735 June 2020. 1,3,7
[32] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R R-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, October 2017. 4,
[33] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, June 2016. 6
[34] Olivier J. Hénaff, Skanda Koppula, Jean- Maan-Bapiiste Alayrac, Aaron van den Oord, Oriol Vinyals, and João Carreira. Efficient Visual Pretraining With Contrastive Detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10086-10096, 2021. 2, 7,
12

---

[35] Olivier J. Hénaff, Skanda Koppula. Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović. Object discovery and representation networks. 2rXi::203.08877 [cs]] March 2022. 2, 2,
[36] R. Devon Hjelm, Alex Fedorov, Samuel Marcie-Mar,hildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations. September 2018. 3
[37] Haiyang Huang. Zhi Chen, and Cynthia Rudin. SegDiscover: Visual Concept Discovery via Unsupervised Semantic Segmentation. arXi::204.10922 [cs), April 2022. 3, 7,9
[38] Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang. Xiao Zhang, and Liang-Chieh Chich Chen. Segsort: Segmentation by discriminative sorting of segments. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7334-7344, 2019.
[39] Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. PICIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering. In 2021 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). pages IEEE. 3, 7,9
Nashville, TN, USA, June 2021.
[40] Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference on Computer Vision,
pages 9865-9874, 2019. 3, 7,9
[41] Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. arXiv:2111. [cs, star), March 2022.
[42] Thomas Kipf. Elise van der Pol. and Max Welling. Contrastive Learning of Structured World Models. In International Conference on Learning Representations, September 2019. 3
[43] H. W. Kuhn, The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(122):33-97, 1955. 7.9
[44] Chunyuan Li, Jianwei Yang. Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan. and Jianfeng Gao. Efficient Self-supervised Vision Transformers for Representation Learning. arXiv:2106.099) 099885 [cs], June 2021. 7
[45] Tsung-Ye Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature Pyramid Networks for Object Detection. In 2017 JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 936 944, July 2017. 7.8
[46] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects Context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014, pages 740-755, Cham, 2014.
Springer International Publishing. 1, 2. 6,7
[47] Songtao Liu, Zeming Li, and Jian Sun. Self-EMD: Self-Supervised Object Detection without ImageNet. 1. 1.3677 [cs], March 2021. 3,
[48] Francesco Locatello, Dirk Weissenbom, Thomas Unterthiner, Aravindh Mahendran, Georg Hcigold, Jakob Uszkoreit, Alexcy Dosovitskiy, and Thomas Kipf. Object-centric lcarning with slot attention. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11525 1538. Curran Associates, Inc., 2020. 3,5
[49] Jonathan Long, Evan Shelhamer, and Trevor Darell Fully convolutional networks for semantic segmenta- tion. 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 3431-3440. June 2015. 7
[50] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. arXiv: 03983 [cs, math), May 2017.
[51] Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning Object- Centric Vidco Models by Contrasting Scts. arXiv:201 10287 [cs], November 2020. 3
[52] Mehdi Noroozi and Paolo Favaro. Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016, pages 69 84, Cham, 2016. Springer International Publishing.
13

---

[53] Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation Learning by Learning to Count. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5899-5907, October 2017. 3
[54] Pedro O O. Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Unsupervised learning of dense visual representations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4589- Curran Associates, Inc., 2020. 1,3
[55] Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexci A. Efros, Context Encoders: Feature Learning by Inpainting. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2536-2544, June 2016. 3
[56] Ramprasaath R. Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik. CASTing Your Model: Learning to Localize Improves Self-Supervised Representations. In 202/ /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11053-1 1062, June 2021. 2,3
[57] Anti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. 5
[58] Yonglong Tian, Chen Sun, Ben Poolc, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6827-6839 Curran Associates, Inc., 2020. 3
[59] J. R. R. Uijlings, K. E. A. van de Sande, T.. Gevers, and A. W. M. Smeulders. Selective Search for Object Recognition. International Journal of Computer Vision, 104(2):154-171, Septembbr 2013. 7
[60] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. 1807v:187.00377 [cs, stat], January 2019. 3,5
[61] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue V Gool. Revisiting contrastive methods for unsupervised leamnnng of visual representations. M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan. editors, Advances in Neural Information Processing Systems, volume 34, pages 16238 16250,, Curran Associates, Inc., 2021. 3,
[62] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsupervised Se- mantic Segmentation by Contrasting Object Mask Proposals. In Proceedings of the IREE/CVF International Conference on Computer Vision, pages 10052-10062, 2021. 2, 3, 5, 9
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31 International Conference Neural Information Processing Systems, NIPS' 17, pages 6000-6010, Red Hook, NY, USA, December 2017. Curran Associates Inc.
[64] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Fierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning ICML '08, pages 1096-1108 Helsinki, Finland, 2008. ACM Press. 3
[65] Xinlong Wang. Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. Frecsolo: Learning to segment objects without annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4176-14416, 2022. 3
[66] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense Contrastive Learning for Self-Supervised Visual Pre-Training. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3023-3032, Nashville, TN, USA, June 2021. IEFE, 2,3,6, 6,7,
[67] Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked Feature Prediction for SuperSupe Visual Pre- Treiningg arXiv:2112.09133 [cs]. December 2021. 1
[68] Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin. Aligning Pretraining for Detection via Object-Level Contrastive Learning. In Advances in Neural Information Processing Systems, volume 34, pages 22682 -22944. Curran Associates, Inc., 2021. 2, 3, 6, 8
[69] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2, https: // //github.aa h/tptttb.cco///eeeeeeeeeee http:///www....oeeeee/////t/ttee 2019.
14

---

[70] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised Feature Learning via Non- parametric Instance Discrimination. In 2018 IEEE/CVE Conference on Computer Vision and Pattern 1.3
Recognition, pages 3733-3742, June 2018.
[71] Enze Xie, Jian Ding, Wenhai Wang. Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luoo, DetCo: Unsupervised Contrastive Learning for Object Detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8392-84988 -0228, 2021. 3, 8
[72] Jiahao Xie, Xiaohang Zhan, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Unsupervised object-level representation learning from scene images. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages
28864 28876. Curran Associates, Inc., 2021. 2, 6,
[73] Zhenda Xie, Yutong Lin, Zheng Zhang. Yue Cao, Stephen Lin, and Han Hu. Propagate Yourself: Exploring Pixcl-Level Consisteney for Unsupervised Visual Representation Learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16679 16688, Nashville, TN, USA, June 2021. IEEE. 2,3,6,1 8
[74] Ceyuan Yang, Zhirong Wu, Bolei Zhou, and Stephen Lin. Instance Localization for Self-supervised Detection Pretraining. 202/ IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3986-3995, June 2021. 3,8
[75] Charig Yang, Hala Lamdouar, Erika Lu,, Andrew Zisserman, and Weidi Xie. Self-Supervised Video Object Segmentation by Motion Grouping. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 7177-7188, 2021. 3
[76] Asano Ym, Rupprecht C, and Vedaldi A. Self-labelling via simultaneous clustering and representation learning. In International Conference on Learning Representations, September 2019. 4
[77] Yang You, Igor Gitman, and Boris Ginsburg. Large Batch Training of Convolutional Networks. arXiv: 038888 [es], September 2017. 6
[78] Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow Twins: Learning via Redundancy Reduction. In Proceedings of the 38th International Conference on Machine Learning, pages 12310-12320. PMLR, July 2021.
Self-Supervised
[79] Richard Zhang, Phillip Isola, and Alexci A. Efros, Colorful Image Colorization. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision ECCV 2016, pages 649-666, Cham, 2016. Springer International Publishing. 3
[80] Richard Zhang, Phillip Isola, and Alexei A. Efros. Split-Brain Autocneoders: Unsupervised Learning by Cross-Channel Prediction. 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 645-654, July 3017.
[81] Xiao Zhang and Michael Maire. Self-supervised visual representation learning from hierarchical grouping. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Curran Associates, Inc., 2020. 2,
P Processing Systems, volume 33, pages
[82] Bingchen Zhao and Xin Wen. Distilling Visual Priors from Self-Supervised Learning. In Adrien Bartoli and Andrea Fusiello, editors, Computer Vision ECCV 2020 Workshops, pages 422-429, Cham, 2020. Springer International Publishing. 3
[83] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic Understanding of Scenes Through the ADE20K Dataset. International Journal of Computer Vision, 127(3)302-3211 March 2019. 2.7
[84] Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving Contrastive Learning by Visualizing Feature Transformation. arXiv:2108, 02982 [cs], August 2021. 3
[85] Adrian Ziegler and Yuki M. Asano. Sell-Supervised Learning of Object Parts for Semantic Segmentation. arXiv:2204.133 13101 [cx], April 2022. 3
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] The limitations discussed in Section G in the appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] The potential negative societal impacts are discussed in Section G in the appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] Link to the code is provided in the abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "_mLQ_283L-w", "reviewer": "Reviewer_oHQN", "summary": "The paper proposes a method for learning objectness from visual data by combining deep clustering in feature space and a contrastive objective that is made invariant to geometric augmentations.\n\nThe method involves two losses:\n- $L_\\text{group}$ which operates per-pixel and that encourages each pixel to be clustered according to the clusters produced by an EMA-updated teacher,\n- $L_\\text{slot}$ which operates at the slot level and ensures consistency between slot representations in different augmentations of the same image through a contrastive objective.\n\nAfter self-supervised pre-training, the encoder is used as the backbone for several vision tasks, namely object detection, semantic segmentation and instance segmentation. In all cases, fine-tuning the pre-trained model yields better results than starting from a random initialization, provided that the number of fine-tuning epochs is equal for the pre-trained and from-scratch models.", "questions": "It would be interesting to discuss statistics about the binary indicator in eq. 5 and 6:\n- How many slots are active on average for each image? This is mentioned in the supplementary for estimating FLOPS but not discussed from the perspective of representation learning.\n- How often is one slot active over the whole dataset?\n- How many terms are excluded in eq. 6 because either $1_\\text{teacher}$ or $1_\\text{student}$ are 0?\n- Is there any regularization to ensure that all slots are used evenly? Or to ensure a min/max number of slots per image?\n\nL226: Why the number of prototypes is 256 for COCO and 2048 for ImageNet? Intuitively COCO contains more diverse objects than ImageNet and should use more prototypes.\n\nL281: the text mentions that the method \"successfully localizes small objects\" w.r.t. column 5 of the figure in table 5. However, that colum depicts a laptop on a bed that are recognized as a small piece of \"sky\" on the \"floor\". How come the method has learned to distinguish small portions of pixels but hasn't learned much about semantics? Shouldn't the two losses in eq. 8 optimize for semantic consistency?\n\nIs there anything in the method that explicitly biases the optimization towards learning object-centric representations as opposed to parts?\nIf not explicitly, why do you think the model learns to group together entire objects rather than parts? Alternatively, do you have evidence that no preference is made between objects and parts?", "limitations": "Yes, but only in the supplementary material.\nIf possible, I would appreciate a mention of limitations in the main text with a link to the corresponding section in the appendix.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and easy to follow, and I am overall positive about it.\nThe structure of the text introduces the method with gradual complexity, which helps navigating the multitude of symbols and variables.\nAlthough figure 1 appears confusing at first because it has \"too much going on\", it becomes progressively clearer with the contents of section 3.1 and 3.2.\nMost details are not given in the main text, but the supplementary material goes into great detail about datasets and training schedules.\n\nIn terms of originality, the paper fits into the popular category of deep clustering methods.\nThis category has recently seen a shift from image-centric clustering to a finer level of detail.\nThe strength of this method is to explicitly optimize for desirable properties of clusters, rather than trying to extract clusters from models trained for another objective.\n\nIndeed, the formulation of $L_\\text{group}$ resembles a per-pixel extension of DINO because of the prototype-based unsupervised clustering.\nThus, I am happy that the authors anticipated this discussion at the end of Section 3.1.\nI agree with the distinction that DINO uses a large number of prototypes while this method uses a smaller number of per-object prototypes.\nMy intuition is that DINO prototypes capture scene semantics that can be more complex than patch semantics due to compositionality, hence the need for more prototypes.\nI am not sure, however, how the prototypes of this method can be \"adaptive to each image\" (L164).\nEq. 1 clearly shows that the assignments for $L_\\text{group}$ are obtained by computing a similarity with learned dataset-wide prototypes, same as DINO. It's only in eq. 4 that slots are extracted by pooling image features using prototype-based attention, these slots can be said to be \"adaptive\". It would be good to clarify the terms \"prototypes\", \"assignments\" and \"slots\".\n\nRegarding the experiments, I appreciate their exhaustiveness and the choice of datasets.\nMy main complaint is directed at the lack of confidence intervals for the reported metrics.\nConsidering how much results can vary due to randomness it is important to report (at least) the mean and std of 3 runs to be sure that improvement is due to the proposed method.\nIf the number of pages is a constraint at least provide extended versions of tab. 2-6 in the appendix.\n\nLast, a critique for the choice of ablation studies.\nOf course, it is interesting to study how the method behaves with a different number of prototypes or different weights for the losses.\nHowever, I do not think the chosen ablations target the most important points of the method.\nIt would be more interesting to ask: are geometric augmentations (and the subsequent inversion) necessary to learn object-centric representations? Would the model learn something different if only color augmentations were employed on two identical crops of each image? What happens when the binary indicator is not used and all slots contribute to the loss? Would an image-level contrastive objective improve or hinder learning object-level prototypes?\n\nMinor:\n- L132 revise \"two-layer multilayer perceptron\"\n- L162 \"much less prototypes\" -> \"fewer\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZrwrI6s0sdP", "reviewer": "Reviewer_NYjs", "summary": "The authors design a method to learn visual representations in a self-supervised fashion from scene-centric data. The work is built upon the choice of exploiting intrinsic scene-centric characteristics of the data, and contrasting with existing literature that exploits object-related priors to guide the learning process. \n\nThe method includes a pixel-level semantic grouping learning mechanism based on learning a set of prototypes to which the pixels are assigned. The set of prototypes can adapt to each sample by an adaptive pooling mechanism.", "questions": "I would revisit the statements about the use of prior knowledge as a guidance to the training process, and relate this work better with existing streams of work that show successful use of priors both for performance improvement and data-efficiency (e.g. in the VIPriors workshop series). \n\nThe results to not clearly show superiority of the proposed approach, lacking a full support of some statements in the introduction. Neither the results show complementarity, in the way they are presented. An error and overlap prediction analysis wrt other methods would contribute to a better understanding of the cases in which one method is better than another. Also, marginal improvements might be checked with statistical tests of significance. If not done, these marginal improvements are bound to be interpreted as a result of randomness.", "limitations": "The authors do not state limitations explicitly. They could discuss more extensively in which cases this approach would be better than others and in which cases not. From the results it is not clear why and when this method would be better to be used.\nI do not find direct implications as potentially negative societal impact. It is a general approach to learn CV models, and its uses are very broad.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strenghts*\n- novelty: the work approaches the self-supervised learning strategy in a novel way favoring pixel-level semantic grouping rather than image-level or object-level representation learning pre-text tasks. The design of the learning procedure has its merit, and the authors state clearly similarity, inspirations from and differences with other methods (e.g. DINO)\n- technical soundness: the description of the method is clear and well-argumented, From my perspective, it has a good balance of mathematical formulation, motivations and textual explanation of the several concepts proposed.\n- experiments on benchmark datasets: the experimental analysis is done on ImageNet1K and COCO(+), which guarantees that the results can be analysed and commented in full\n- the paper is very well-written, clear and concise\n\n*Weaknesses*\n- results: results are comparable with those of existing approaches, which have different working principles. Most of the time the slight improvements are marginal (or in some cases the results are also marginally lower than others). This does not allow to fully appreciate the quality and usefulness of the method. \n- motivation/background: the main difference with other works is stated to be the fact that no priors related to instance discrimination are used in the pre-text task for self-sup training as they will limit the learning potential. This moves the authors to focus on only exploiting the data intrinsic properties and characteristics. I find this a stretched motivation, as priors related to human knowledge about any problem have been demonstrated to steer the learning process in favor of good performance and efficient use of the data (avoiding to re-learn that prior knowledge somehow from the data).  The fact that this argumentation is weak, is reflected also in the results, which do not show that using only knowledge from the data would contribute to overall better performance (see point above).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KJn_YdmDjoO", "reviewer": "Reviewer_LB2h", "summary": "This paper advances a self-supervised representation learning strategy, dubbed SlotCon, that can learn from scene-centric data by reasoning and processing visual information at the pixel level. This is in contrast with the large majority of the self-supervised methods designed for object-centric data (e.g., ImageNet) where image-level reasoning is sufficient. \nRecent methods have addressed scene-centric data, however the authors argue that such approaches are limited by the hand-crafted priors (e.g., objects as superpixels) or too specialized pretext tasks as they can limit their generalization.\nSlotCon builds upon the SlotAttention method and uses slots as learnable prototypes for pixels that are thus grouped via the assignments to their corresponding slots. SlotCon is composed of a teacher and student network. The pixel grouping supervision resembles both SWaV (aligning the pixel assignments between teacher and student spatial-aligned pixels from different views) and DINO (student + EMA teacher, centering of teacher logits, different temperatures on student and teacher). This loss encourages grouping of pixels into object-like structures via the learned slots.\nIn order to discriminate slots that carry the same visual or semantic information across views from other non-informative or redundant slots, SlotCon has also a contrastive objective (InfoNCE loss) at the slot level that encourages similarity between different views of the same slot and discourages similarity of slots from different views and slots from other images.\n\nSlotCon is evaluated on a number of pre-training settings (ImageNet, COCO, COCO+) and downstream tasks (object detection, instance segmentation, semantic segmentation, unsupervised segmentation) and datasets (COCO, Cityscapes, Pascal VOC, ADE20k), with nice performance and results.", "questions": "\nHere are a few questions and suggestions to help improve the paper, that could be potentially addressed in the rebuttal\n\n\n1) Conduct a comparison of the computational cost for training SlotCon compared to related methods, MOCOv2, PixPro, etc.\n\n2) Clarify results and implementations for baseline methods\n\n3) (Optional) How is SlotCon performing on more complex scene-centric datasets, e.g., autonomous driving datasets?\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Post-rebuttal update\n\nThe detailed rebuttal addresses all my questions and offers convincing responses and additional insisghts (across the responses to all reviewers).\nI don't have any other questions for the authors and I confirm my positive recommendation for this submission.\n\n==========================================\n\n\n### Recommendation\nThis paper advance a nice and effective idea for self-supervised learning from scene-centric images. The approach looks sound and the results are encouraging. I'm overall positive about this work and leaning towards recommending it for acceptance.\n\n### Paper strengths\n\n- _Clarity_: I find this paper is mostly well written and argued. The intuition, reasoning and formalism of the method are rather well explained and overview figure 1 is very nice. Then the authors conduct a number of ablation and sensitivity studies (number of prototypes, loss balancing weights, temperature of the teacher) for better understanding. There are also some interesting qualitative results (Figure 2) visualizing pixels grouped by slots and their nearest neighbors. The approach is convincing.\n\n- _Significance_: The paper addresses a challenging problem related to self-supervised representation learning: how to learn from scene-centric datasets that are not as well balanced as ImageNet, more cluttered and usually with fewer images to learn from. The results reached by SlotCon are good.\n\n\n- _Quality_: The approach seems technically sound. There are several experiments on different settings and different relevant baselines considered. The authors do not provide the code in the supplementary, but provide rich information about the implementation -- it would be better to release the code though. The supplementary is abundant in additional experiments, ablations and implementaiton details. \n\n- _Originality_: This work combines various good practices from the recent literature (SWAV, DINO, Slot Attention, see my comments from the summary), however the output method does still have originality and is effective. The masking of uninformatives slots (eq. 5) is an interesting idea.\n\n\n- _Misc_: I appreciate that the authors stick to the lower epoch regime (100-200 epochs) which is more reasonable and allows comparison with a larger spectrum of methods\n\n\n### Paper weaknesses\n\nMostly minor concerns:\n\n#### Originality:\n- as mentioned above, to me SlotCon is a fairly original work aggregating in an effective manner various good practices and approaches from the literature.\n- from the text it can be understood that using SlotAttention for unsupervised pixel-level representation learning is a finding of the authors. However other prior methods have used it for weakly-supervised or  self-supervised learning [a], [b]. Those are different approaches and I don't think a discussion is necessary, however the text should be adjust to better delimitate the contributions and prior works\n- the approach is similar in spirit with Odin [31] as both aim to learn representations without human object priors. The qualitative examples in both papers are similar. I think it would be good to acknowledge the relatedness of these concurrent works\n\n\n#### Scope of experiments:\n- COCO is indeed a challenging scene-centric dataset and it's great to see pre-training and evaluation done on it, moving beyond ImageNet\n- I think this work would benefit from showing results on more complex settings, like autonomous driving data where the amount of objects per scene can be larger (BDD100K, Cityscapes). Some options could be:\n    + self-supervised pretraining on an autonomous driving dataset, e.g., BDD100K[56]\n    + self-supervised pretraining on smaller autonomous driving datasets, e.g., Cityscapes, and evaluation on unsupervised semantic segmentation on these datasets, see PiCIE, STEGO[c]\n\n#### Computational cost:\n- the authors discuss the computational cost in appendix D and give exact number of FLOPS for forward operation compared to a standard ResNet-50 backbone. It appears SlotCon is 12.3% more expensive in terms of FLOPS.\n- However this does not tell us much on how fast are forwards and backward passes and how does SlotCon fare against other methods. It's nice that SlotCon can be trained in fewer epochs, but it would be great to be able to compare also the training times\n- To get an idea of more straightforward and easy to visualize and understand format I recommend the time and memory cost studies from MoCov2, OBoW[d] and iBOT[e]\n\n\n#### Baseline results and implementations:\n- In Table 3 (transfer results with ImageNet-1k pre-training), it's not clear where to the scores from DetCon (200 epochs) come from. In the original paper there are some result on a few plots and maybe the authors eye-balled them (e.g., COCO instance segmentation), but not all results can be found in the original paper. The authors don't mention another reference for them. Are these scores reproduced by the authors, taken from a different paper (if so, please cite the corresponding source).\n- PixPro results in Table 3 seem lower than originally reported, about 0.8-1.0 for COCO. Do the authors know why?\n\n- The common protocol for COCO downstream as proposed in MoCO uses the default `ROI_BOX_HEAD` configuration with 2 FC layers. PixPro seems to have modified the default configuration of 2FC to 4 convolution layers + 1 FC layer that seems to improve scores. Was this setting used for PixPro or SlotCon?\n\n\n#### Related work:\n- this work mentions a whole lot of relevant works from the literature\n- however for the unsupervised semantic segmentation section there are several key methods in addition to the now usual PiCIE and IIC and the very recent ones mentioned by the authors (SegDiscover).\n- here are a few suggestions for this area of the literature:[f],[g],[h],[c]\n\n\n**References:**\n\n\n[a] T. Kipf et al., Conditional Object-Centric Learning from Video, ICLR 2022\n\n[b] Z. Bao et al., Discovering Objects That Can Move, CVPR 2022\n\n[c] M. Hamilton et al., Unsupervised Semantic Segmentation by Distilling Feature Correspondences, ICLR 2022\n\n[d] S. Gidaris et al., Online Bag-of-Visual-Words Generation for Unsupervised Representation Learning, CVPR 2021\n\n[e] J. Zhou et al., iBOT: Image BERT Pre-Training with Online Tokenizer, ICLR 2022\n\n[f] J. Hwang et al., Segsort:Segmentation by discriminative sorting of segments, ICCV 2019\n\n[g] M. Chen et al., Unsupervised object segmentation by redrawing, NeurIPS 2019\n\n[h] X. Wang et al., FreeSOLO: Learning to Segment Objects without Annotations, CVPR 2022\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CltN9Irkkz", "reviewer": "Reviewer_MdX5", "summary": "The paper proposes a self-supervised learning framework, SlotCon, from unlabeled scene-centric data. The method adopts joint semantic grouping -- softly assigning pixels to learnable prototypes shared by the datasets, and contrastive learning -- first perform attentive pooling on the prototypes to from slots, and then conduct contrastive learning on slots from two different views. In experiments, the authors evaluated on COCO, ImageNet-1K and COCO+ dataset. On transfer learning tasks (COCO detection, COCO segmentation, cityscape & Pascal VOC & ADE20k semantic segmentation), the proposed method achieves performance on par or surpassing previous methods. It also achieves better mIoU on unsupervised semantic segmentation results compared with previous methods.", "questions": "- How does the proposed method perform on image classification tasks? e.g., ImageNet.\n", "limitations": "The authors addressed the limitations and negative societal impact in the supp.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The method is purely data-driven without the need of hand-crafted priors or specialized pretext tasks.\n- The proposed method perform self-supervised learning on pixel level, which is shown to perform better on dense prediction downstream tasks compared with image-level self-supervised learning methods.\n- Extensive experiments are conducted to show the effectiveness of the proposed methods.\n- The paper is relatively easy to follow.\n\nWeaknesses:\n- Limited novelty and lack of citation on related works. Pixel-level clustering with contrastive learning are already studied in previous works, e.g., [1]. Prototype-based semantic segmentation are also already studied, e.g., in [2]. But many related works are not discussed. \n\n- The representation in the method section is not very clear. e.g., is $A^l_{\\theta}$ (Eq. 4) identical to $P^l_{\\theta}$ (Eq. 1)? Eq.6 is unclear, since $q_\\theta$ is not introduced. Whether it's contrastive learning among two views of a single image, or among a batch is not clear from the equation.\nSome symbols in the Method Section are not explained.\n\n- The experiments only show performance on ResNet50, whether the proposed method scales to larger backbones is not clear.\n\n- The COCO detection task is only trained and compared with on 1x schedule, while recent study show that 1x schedule is far from convergence. Hence the results are not very conclusive, it's likely that the proposed method only converges faster, but after training longer, the performance will become similar with comparison methods.\n\n- Type: L211 \"solarization\".\n\n[1] Ke, Tsung-Wei, Jyh-Jing Hwang, and Stella X. Yu. \"Universal weakly supervised segmentation by pixel-to-segment contrastive learning.\" ICLR 2021.\n[2] Zhou, Tianfei, et al. \"Rethinking Semantic Segmentation: A Prototype View.\" CVPR 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept