Unsupervised Semantic Segmentation with Self-supervised Object-centric Representations
Authors: Andrii Zadaianchuk
Matthaeus Kleindessner
Yi Zhu
Francesco Locatello
Thomas Brox
Venue: ICLR 2023 notable top 25% (2023)
Keywords: unsupervised semantic segmentation
object segmentation
object-centric learning

Abstract: In this paper, we show that recent advances in self-supervised representation learning enable unsupervised object discovery and semantic segmentation with a performance that matches the state of the field on supervised semantic segmentation 10 years ago. We propose a methodology based on unsupervised saliency masks and self-supervised feature clustering to kickstart object discovery followed by training a semantic segmentation network on pseudo-labels to bootstrap the system on images with multiple objects. We show that while being conceptually simple our proposed baseline is surprisingly strong. We present results on PASCAL VOC that go far beyond the current state of the art (50.0 mIoU), and we report for the first time results on MS COCO for the whole set of 81 classes: our method discovers 34 categories with more than 20% IoU, while obtaining an average IoU of 19.6 for all 81 categories.

Published as a conference paper at ICLR 2023
UNSUPERVISED SEMANTIC SEGMENTATION WITH SELF-SUPERVE OBJECT-CENTRIC REPRESENTA- TIONS
Andrii Zadaianchuk Matthaeus Kleindessner', Yi Zhu?, Francesco Locatello", Thomas Brox Maxx-Plnnc Institute for Intelligent Systems, Tübingen, Germany, AAmazon Web Services, Department of Computer Science, ETH Zürich, *University of Freiburg
ABSTRACT
In this paper, we show that recent advances self-supervised representation learn- ing enable unsupervised object discovery and semantic segmentation with a per- formance that matches the state of the field on supervised semantic segmentation 10 years ago. We propose methodology based on unsupervised saliency masks and self-supervised feature clustering to kickstart object discovery followed by training semantic segmentation network on pseudo-labels bootstrap the sys- tem on images with multiple objects. We show that while being conceptually sim- ple our proposed baseline surprisingly strong. We present results PASCAL VOC that go far beyond the current state of the art (50.0 mloU). and we report for the first time results on MS COCO for the whole of 81 classes: our method discovers 34 categories with more than 20% IoU, while obtaining an average IoU of 19.6 for all 81 categories.
Label
MeskContrast
COMUS (ours)
Figure 1: Unsupervised semantic segmentation predictions PASCAL VOC (Everingham al., 2012). Our COMUS does not use human annotations to discover objects and their precise local- ization. In contrast the prior state-of-the-art method MaskContrast (Van Gansbeke et al., 2021), COMUS yields more precise segmentations, avoids confusion of categories, and is not restricted to only one object category per image.
INTRODUCTION
The large advances in dense semantic labelling in recent years were built on large-scale human- annotated datasets (Everingham et al., 2012; Lin et al., 2014; Cordts et al., 2016). These supervised
Work done during internship Amazon.

---

Published as a conference paper at ICLR 2023
semantic segmentation methods (e.g.. Ronneberger et al., 2015; Chen al., 2018) require costly human annotations and operate only on a restricted set of predefined categories. Weakly-supp -ccally-upervised segmentation (Pathak et al., 2015; Wei et al., 2018) and semi-supervised segmentation (Mittal al., 2019; Zhu et al., 2020) approach the issue of annotation cost by reducing the annotation to only a class label or subset of labeled images. However, they still bound to predefined labels.
In this paper, we follow a recent trend to move away from the external definition of class labels and rather try to identify object categories automatically by letting the patterns in the data speak. This could be achieved by (1) exploiting dataset biases to replace the missing annotation, (2) a way to get the learning process kickstarted based on "good" samples, and (3) a bootstrapping process that iteratively expands the domain of exploitable samples.
A recent method that exploits dataset biases, DINO (Caron et al., 2021), reported promising ef- fects of self-supervised feature learning in conjunction with a visual transformer architecture by exploiting the object-centric bias of ImageNet with a multi-crop strategy. Their paper emphasized particularly the object-centric attention maps some samples. We found that the attention maps of their DINO approach are not strong enough on a broad enough set of images to kickstart unsu- pervised semantic segmentation (see Fig. 4), but their learned features within an object region yield clusters of surprisingly high purity and align well with underlying object categories (see Fig. 3).
Thus, we leverage unsupervised saliency maps from DeepUSPS (Nguyen al., 2019) and BAS- Net (Qin et al., 2019) to localize foreground objects and to extract DINO features from these fore- ground regions. This already enables unsupervised semantic segmentation images that show dominant object category together with an unspectacular background as they are common PAS- CAL VOC (Everingham al., 2012). However, on other datasets, such MS COCO (Lin al., 2014), most objects are in context with other objects. Even on PASCAL VOC, there are many images with multiple different object categories.
For extending to more objects, we propose training regular semantic segmentation network on the obtained pseudo-masks and to further refine this network by self-training it on its own outputs. Our method, dubbed COMUS (for Clustering Object Masks for learning Unsupervised Segmentation), allows us to segment objects also in multi-object images (see Figure 1), and it allows us for the first time to report unsupervised semantic segmentation results on the full 80 category MS COCO dataset without any human annotations. While there are some hard object categories that are not discovered by our proposed procedure, we obtain good clusters for many of COCO object categories.
Our contributions can be summarized as follows:
1. We propose a strong and simple baseline method (summarized in Figure 2) for unsupervised discovery of object categories and unsupervised semantic segmentation in real-world multi- object image datasets.
2. We show that unsupervised segmentation can reach quality levels comparable to supervised segmentation 10 years ago (Everingham al., 2012). This demonstrates that unsupervised segmentation is not only an ill-defined academic playground.
3. We perform extensive ablation studies to analyze the importance of the individual components in proposed pipeline, well as bottlenecks to identify good directions to further improve the quality of unsupervised object discovery and unsupervised semantic segmentation.
2 RELATED WORK
There are several research directions that try to tackle the challenging task of detecting and segment- ing objects without any, or with only few, human annotations.
Unsupervised Semantic Segmentation The first line of work (Van Gansbeke et al., 2021; He et al., 2022; Cho et al., 2021; Ji et al., 2019 Hwang et al., 2019; Ouali et al., 2020; Hamilton et al., 2022; Ke et al., 2022) aims to learn dense representations for each pixel in the image and then cluster them (or their aggregation from pixels in the foreground segments) to get each pixel label. While learning semantically meaningful dense representations is an important task itself, cluster- ing them directly to obtain semantic labels seems to be a very challenging task (Ji et al., 2019; Ouali et al., 2020). Thus, usage of additional priors or inductive biases could simplify dense rep-
2

---

Published as a conference paper at ICLR 2023
Self-supervised
Unsupervised Object Categories Discovery
Iterative Seif-training
pretraining
Zalegorizaiin Prior (What?)
Noley pseudo maska
Object- centric
Salf- supervised representation
delaset
Images
Object proposals
Self-supervised object representations
Object Clusters
Noley pseudo masks
labels)
leaming
Localization Prior
Satf-ttaaining itteraiins)
(Where?)
Sallency Detaset (no labols)
Self- supervised sallency detector
Semantic segmentation
Object
proposale
network
masks
Self-supervised representation leaming Self-supervised saliency detector
Crop and resize
Spectral Clustering
Multiply mask diuster
Figure Overview of our self-supervised semantic segmentation framework. First, the self- supervised representation learning network (e.g., DINO (Caron al., 2021)) and the unsupervised saliency detector (c.g., DeepUSPS (Nguyen et al., 2019)) are trained without manual annotation on object-centric and saliency datasets (e.g., ImageNet (Deng et al., 2009) and MSRA (Cheng al., 2015)). Next, we use the saliency detector to estimate object proposal masks from the original se- mantic segmentation dataset. After this, the original images cropped to the boundaries of object proposal masks and resized. We compute feature vectors within these regions and cluster them with spectral clustering discover different object categories. We filter the clusters by removing the most uncertain examples. The cluster IDs are combined with the saliency masks to form unsupervised pseudo-masks for self-traa self-training of semantic segmentation network (e.g.. DeepLabv3).
resentation learning. PICIE (Cho et al., 2021) incorporates geometric consistency as an inductive bias to facilitate object category discovery. Recently, STEGO (Hamilton et al., 2022) showed that DINO feature correspondences could be distilled to obtain stronger bias for category discov- ery. MaskContrast (Van Gansbeke et al., 2021) uses a more explicit mid-level prior provided by an unsupervised saliency detector learn dense pixel representations. To obtain semantic labels in unsupervised way, such representations are averaged over saliency masks and clustered. We show that better representations for each mask could be extracted by using off-the-shelf self-supervised representations from DINO (Caron et al., 2021) encoder. Recently, DeepSpectral (Melas-Kyriazz al., 2022) proposed use spectral decomposition of dense DINO features. They suggested over- cluster each image into segments and afterward extracting and clustering DINO representations of such segments while using heuristics to determine the background segment. Those segments repre- sent object parts that could be combined with over-clustering and community detection to improve quality of pseudo-masks (Ziegler & Asano, 2022). contrast, we show that starting from object- centrie saliency priors discovered categories (see App. D.I). In contrast to estimating pscudo-mase on the full dataset, using only good quality reliable object proposals for each category combined with iterative self-training expanding datasets additionally decrease biases over the initial pseudo-masks.
simpler dataset provides large benefits for discovering object
Unsupervised Object Discovery (UOD) UOD another research direction that also aims dis- cover object information such as bounding boxes or object masks from images without any human annotations. Recent works on UOD (H' enaff al., 2022; PeriszKyriazi al., 2022; Wang al., 2022; Simeoni et al., 2021; 1; Vo et al., 2021; Zhang et al., 2020; Vo et al., 2020) showed the poten- tial benefit using the embeddings of pretrained networks (supervised or self-supervised) for both object localization (to the level of the object's bounding box) and object clustering. First, rOSD (Vo et al., 2021; 2020) showed that supervised features could be used to localize single objects in the image. Next. LOST (Simconi et al., 2021) proposed a heuristic that relies self-supervised fea- tures to localize the most salient object in the image. In contrast to those methods we consider the challenging task of object segmentation, not only object detection. Finally, Melas-Kyrian et al.
3

---

Published as a conference paper at ICLR 2023
Core Filtered
Custer train
Arrone Clugse
Figure 3: Visualization of unsupervised pseudo-mash masks on PASCAL VOC val set. (left) 2D t-SNE projection of object proposal features. Colors correspond to cluster IDs. (right) Pseudo-masks from different clusters. The pseudo-masks were randomly sampled for each cluster from both cluster core pseudo-masks (green columns) and filtered pseudo-masks (red columns).
(2022); Wang et al. (2022) propose do spectral decomposition of dense DINO features and use of sign of Fiedler cigenvector as criteria for object localization mask.
Unsupervised Object-centric Representation Learning Object centric learning assumes that scenes are composed different objects and aims to learn sets of feature vectors, where each of them binding to one object. Unsupervised methods based on single images (Burgess et al., 2019; Greff et al., 2019; Engelcke et al., 2020; Locatello al., 2020; Singh et al., 2022a) suffer from single- view ambiguities, which one tries to overcome by exploiting the information in multiple views of a static scene (Chen et al., 2021), in a single view of dynamic scene (i.c., a video) (Hsich et al., 2021; Kipf et al., 2021; Singh et al., 2022b) or multiple views of dynamic scene (Nanbo et al., 2021). In contrast to previous methods and similar to DINOSAUR method (Seitzer et al., 2023), our method exploits unlabeled object-centric datasets to extract object masks and representations.
SELF-SUPERVISED SEMANTIC SEGMENTATION
3.1 INITIAL DISCOVERY OF OBJECT CATEGORIES
Unsupervised decomposition of complex, multi-object scenes into regions that correspond to the present objects categories is hard and largely ill-defined as it is possible to decompose scene with different levels of granularity obtaining several valid decompositions for the same scene (e.g.. a person could be considered is unnecessary to correctly decompose all images of dataset to kickstart unsupervised object discovery. In this paper, we show that is sufficient exploit simple images in dataset in order to discover some objects and their categories. This works particularly well due to intrinsic properties of natural images and photos made by humans. One such property is that the most salient region of the image often corresponds to single distinct object.
object additionally decomposed to body parts). However,
Self-supervised Object Localization Similar to MaskContrast (Van Gansbeke et al., 2021), we propose to retrieve a set of object mask proposals for the images in our dataset by using an un- supervised saliency estimator. In particular, we were using the DeepUSPS (Nguyen et al., 2019) model as an unsupervised saliency estimator. DeepUSPS was trained on MSRA (Cheng et al., 2015) in unsupervised way exploiting the bias towards simple scenes with often homogeneously textured background of the MSRA dataset, as well as the hard-coded saliency priors of classical -non-learning based) saliency methods. To further improve the estimator's transfer ability more complex datasets like PASCAL VOC and MS COCO, we trained another saliency model, Bas- Net (Qin et al., 2019), the saliency masks generated by DeepUSPS. Some examples of saliency

---

Published as a conference paper at ICLR 2023
Algorithm 1: Object Categories Discovery for Unsupervised Pseudo-Masks Estimation
Given: N images 2i, self-supervised salient regions' segmentation network L with binary thresh-
old 0, self-supervised representation learning method C. percentage of proposals to filter p.
Step 1: Obtain binary object proposal masks mn by s; L(x,) 0 and object proposal regions
0, crop(Ez, 8,).
Step 2: Compute object representations r; of object proposal regions 0g, r; = C(o,)
Step 3: Cluster object proposal representations using spectral clustering to assign cluster ID 4
for each object proposal 0₁-
Step 4: Filter p percents of the most uncertain object proposals for each discovered cluster (pro- posals with the largest distance to the cluster center in the eigenvalue embedding).
Step 5: Combine cluster IDs f, with object proposal masks 8; to obtain initial pscudo-masks m,,
Return: Noisy object segmentation pseudo-mase masks rit.-
detector masks are presented in Section 4.3; see Figure 4. In addition, we studied performance of method with original DeepUSPS masks and with DeepSpectral saliency masks in App. D.
Self-supervised Representation Learning The self-supervised feature learning technique DINO (Caron et al., 2021) exploits the dataset bias of ImageNet, which mostly shows a single object in the center of the image. DINO uses, among other transformations, the multi-crop strategy to link local patterns of the same object instance to its global pattern in the embedding. This leads to feature representation that tends to have similar embedding for patterns from the same category.
We start extraction of the object representation by cropping the image to the borders of the saliency mask and resizing the obtained crop to 256 256 resolution. Next, we feed the object proposal into the Vision Transformer (ViT) (Dosovitskiy et al., 2021) architecture pretrained a self-supervised way with DINO. The feature vector of the CLS token from the last layer is used object rep- resentation. As CLS token attention values from the last layer of DINO were shown to attend to foreground objects (Caron al., 2021), the obtained CLS token features are implicitly aggregating object related information from the object proposal.
Discovery of Semantic Categories We cluster the feature vectors obtained per image with spectral clustering (von Luxburg, 2007). Thanks to the saliency masks, most of the feature vectors are based on foreground patterns and disregard the background, i.e., they become object-centric. Even though this is clearly not the case for all images, either because there are salient regions in the background or because the image shows multiple objects from different categories, there are enough good cases for spectral clustering to yield clusters that are dominated by a single object category: see Figure As we show in Table 5, this clustering of features within the saliency masks already yields unsupervised object discovery results beyond the state of the art.
Filtering of Cluster Samples Since neither the salient regions nor DINO features are perfect, we must expect several outliers within the clusters. We tested the simple procedure to filter the most uncertain samples of each cluster and discard them. We measure uncertainty by the distance to the cluster's mean in the spectral embedding of the Laplacian eigenmap (von Luxburg, 2007). In Figure we show that such examples are often failure cases of the saliency detector, such as parts of background that are not related to any category. In addition, we study sensitivity of COMUS algorithm App. B, showing that COMUS performs comparably well when the percentage of filtered examples varies from 20% to 40%. We refer the reader to the Algorithm I for the detailed pseudocode of object categories discovery part of the COMUS method.
3.2 UNSUPERVISED ITERATIVE SELF-TRAINING WITH NOISY PSEUDO-MASKS
As discussed above, the clustering of feature vectors extracted from within saliency masks makes several assumptions that are only satisfied in some of the samples of dataset. While this is good enough to get the object discovery process kickstarted, it is important to alleviate these assumptions in order to extend to more samples. In particular, objects from one category (otherwise the feature vector inside the saliency mask comprises patterns from different categories) and on the correct localization of the object boundaries.
implicitly relied on the image to show only
5

---

Published as a conference paper at ICLR 2023
Algorithm 2: Self-training with Noisy Pseudo-Masks
Given: N images x; with clustering pseudo-makks my external M images zy for self-training
Step 1: Train Teacher network 9, (with prediction function f) on images with unsupervised pseudo-masks by minimizing the total loss L for object segmentation:
N 0: arg min E(my, 8t)).
Step 2: Generate new pseudo-masks M, for all unlabeled images xy (c.g., images from PASCAL VOC trainaug set). Step 3: Train a Student network 8g images and new pseudo masks (2z. my):
N+M
arg min
Σ
((2,,,,
N
Return: Semantic segmentation network 0*.
To extend also multi-object images and improve the localization of object boundaries. we propose using the masks with the assigned cluster IDs as initial pseudo-labels for iterative self- training of a semantic segmentation network. Self-training originally semi-supervised learning approach that uses labels to train teacher model and then trains student model based the pseudo-labels generated by the teacher on unlabeled data (Xie et al., 2020). Similar self-training methods were shown be effective also in semantic segmentation (Chen al., 2020; Zhu et al., 2020). In this paper, we use the unsupervised pseudo masks from Sec. 3.1 to train the teacher. In our experiments, we used the network architecture of DeepLabv3 (Chen et al., 2017), but the method applies to all architectures. Since large architectures like DeepLaby3 are typically initialized with an ImageNet pretrained encoder, also pretrained encoder for initialization. However, since want to stay in the purely unsupervised training regime, we use self-supervised DINO pretraining.
Once the semantic segmentation network trained on the pseudo-masks, it can predict its own masks. In contrast to the saliency masks, this prediction is not limited to single-object images. Moreover, the training can consolidate the information of the training masks and, thus, yields more accurate object boundaries. Since the masks of the segmentation network are on average better than the initial pseudo-masks, we use them as pseudo-masks for a second iteration of self-training. In addition, if such masks are obtained from unseen images of an extended dataset, the predictions of the segmentation network are not overfitted to the initial pseudo-masks and thus are an even better supervision signal. We refer to the pseudocode in Algorithm 2 for overview of iterative self-training. addition, Table in Sec. 4.3 shows that both the initial self-training (Step 1 in Algorithm 2) and the second iteration (Step 3 in Algorithm of self-training improve results.
4 EXPERIMENTS
Evaluation Setting We tested the proposed approach on two semantic object segmentation datasets, PASCAL VOC (Everingham et al., 2012) and MS COCO (Lin et al., 2014). These bench- used the ground truth seg-
marks are classically used for supervised segmentation. In contrast, mentation masks only for testing but not for any training. We ran two evaluation settings. For the first, we created as many clusters as there are ground truth classes and did one-to-one Hungarian matching (Kuhn, 1955) between clusters and classes. For the second, we created more clusters than there are ground truth classes and assigned the clusters to classes via majority voting, i.e, for each cluster we chose the class label with most overlap and assigned the cluster to this class. In both cases we used IoU as the cost function for matching and as the final evaluation metric.
Hungarian matching is more strict, as it requires all clusters match ground truth class. Hence, reasonable clusters are often marked as failure with Hungarian matching; for instance, the split of the dominant person class into sitting and standing persons leads to cluster creating IoU of 0. This is avoided by majority voting. where clusters are merged to best match the ground truth classes. However, in the limit of more and more clusters, majority voting will trivially lead to perfect result.

---

Published as a conference paper at ICLR 2023
Table 1: Comparison prior and iterative improvement via self-training (evaluated by IoU after Hungarian matching) on the PASCAL 2012 val set. The results for SwAV and IIC methods are taken from MaskContrast paper. COMUS results are mean standard dev. over runs.
Method
mloU
Colorization (Zhang et at,
4,9
ICC (Ji et al., 2019)
9.8
SwAV (Caron al., 2020)
4.4
MaskContrast (Van Gansbeke al., 2021)
35.1
DeepSpectral Kyriazi et al., 2022)
37.2 +. 3.8
DINOSAUR (Seitzer al., 2023)
37.2 1.8
Leopart (Ziegler Asano, 2022)
41.7
Pseudo-masks (Iteration 0)
43.8 0.1
COMUS (Itration 1)
47.6 1:: 0.4
COMUS (Iteration 2)
50.0 0.4
Table 2: COMUS performance on PASCAL VOC 2007 test (evaluated by loU after Hungarian matching). The test data was never seen during self-learning or validation.
6
8
mloU
DeepSpectra 77.3
40.2
0.0
76.2
25.0
6.0
66.7
90.7
82.7
0.0
43.6
24.5
56A
63.5
31.5
20.6
2.3
0.0
9.4
75.0
0.1
35.8
COMUS
84.3
38.3
30,9
47.1
39.
66.3
S4.7
67.7
0.0
60.2
213
563
$1.9
62.7
45.4
13.8
81.5
43.4
47.7
When not noted otherwise, we used Hungarian matching in the following tables. We report mean intersection over union (mloU) evaluation metric.
Implementation Details We used pretrained DINO features with the DINO architecture released in DINO's official GitHub'. In particular, we used DINO with patch size that was trained for 800 epochs ImageNet-Il 1k without labels. For the saliency masks, used the BasNet weights pretrained on predictions from DeepUSPS released by MaskContrast's official GitHub² (see folder saliency). All parameters of spectral clustering and self-training described in App. G.
4.1 PASCAL VOC EXPERIMENTS
PASCAL VOC 2012 (Everingham al., 2012) comprises 21 classes 20 foreground objects and the background. First, to qualitatively validate that the obtained clusters correspond to true object categories, visualize the t-SNE embedding (van der Maaten & Hinton, 2008) of DINO repre- sentations showing that clusters correspond to different object categories (see Fig. 3). Further, we quantitatively confirmed that saliency masks with assigned cluster ID psseudo-masks) produce state- of-the-art unsupervised semantic segmentation on PASCAL VOC and outperforms the MaskCon- trast method that learns dense self-supervised representations; see Table (Iteration row).
For eeff-training the DeepLabv3 architecture, we initialized the encoder with ResNet50 pretrained with DINO (self-supervised) on ImageNet and finetuned the whole architecture on the aseudo-maske we computed on the PASCAL 2012 truin set. This increased the performance from 43.8% mloU to 47.6% mloU, see Table (Iteration 1), which supports our consideration of bootstrapping from the original pscudo-masks In particular, allows to segment objects in multi-category images.
Successively, we added one more iteration of self-learning on top of the pseudo-masks on the PAS- CAL 2012 trainaug set. The PASCAL 2012 trainaug set (10582 images) is extension of the original train set (1464 images) (Everingham et al., 2012; Hariharan et al., 2011). was used by previous work on fully-supervised (Chen et al., 2018) and unsupervised (Van Gansbeke al., 2021) learning. The second iteration of self-training further improves the quality to 50.0% mloU; see Ta- ble 1 (Iteration 2). In particular, it allows
make multi-category predictions on images from the validation set unseen during self-supervised training (Fig. 1). Accordingly, the method also yields good results on the PASCAL VOC 2007 official test set; see Table
Phttps: //t:::///g/ http://hhhh..cc//aaaggaaaaa////a
sed Semanti e- Segment on
7

---

Published as a conference paper at ICLR 2023
Table 3: Unsupervised semantic segmentation before and after self-learning evaluated by mloU after Hungarian matching on the MS COCO val set. As discovered object category we count those categories with an loU > 20% from all 81 categories. Also, we show IoU for categories that have corresponding cluster (i.e., with IoU larger than zero).
all
discovered (with 1oU2 20%)
have cluster (with IoU> 0%)
mloU
number
mloU
number
mloU
Pseudo-masks
18.2
33
36.6
73
20.2
COMUS
19.6
34
40.7
60
26.5
Table 4: Transfer from PASCAL VOC to MS COCO for the 21 PASCAL VOC classes. Training on the simpler PASCAL dataset yields better performance on COCO than learning on COCO itself while both COMUS runs perform better than DeepSpectral.
86
6
R
W
A
36
71.6
4.2 MS COCO EXPERIMENTS
We further evaluated our method on the more challenging COCO dataset (Lin et al., 2014). It focuses on object categories that appear context each other and has things categories. We transform the instance segmentation masks to category masks by merging all the masks with the same category together. Our method able discover 34 categories with more than 20% IoU. Among those categories, we obtained an average IoU of 40.7%; see Table 3.
Additionally, we studied the transfer properties of COMUS under a distribution shift. To this end, we self-trained our COMUS model the PASCAL VOC dataset and then tested this model on the same 20 classes on the MS COCO dataset. The results in Table show that the transfer situation between datasets is quite different from supervised learning: training the PASCAL VOC dataset and testing on COCO yields better results than training in-domain on COCO (see Fig. 10 in Appendix). This because the PASCAL VOC dataset contains
single object images than MS COCO, which makes self-supervised learning on PASCAL VOC easier. This indicates that the order in which data is presented plays role for unsupervised segmentation. Starting with datasets that have more single-object bias is advantageous over starting right away with a more complex dataset.
4.3 ANALYSIS
Ablation Study To isolate the impact of single components of our architecture, conducted various ablation studies on PASCAL VOC; see Table 5. All proposed components have a positive effect the result: spectral clustering that additionally computes Laplacian eigenmap before k- means clustering yields better results than k-means clustering (see App. B for detailed analysis on clustering method choice and sensitivity of its parameters); self-training is obviously important to extend to multi-category images; filtering the most distant samples from cluster followed by the second iteration of self- ceff-training on the much larger trainaug gives another strong boost.
Choice of Categorization Prior Next, we investigated how COMUS works with different self- supervised representation learning methods. COMUS performs best with VIT based features ex- tractors such as DINO (Caron al., 2021) and iBOT (Zhou al., 2022), while performance significantly worse for SwAV method (Caron et al., 2020) based on ResNet architecture. We fur- ther show that clustering ability of categorization method could be evaluated on ImageNet images clustering where self-supervised methods were originally trained on (see App. C).
Quality and Impact of Saliency Masks In Table 7, we compare the quality of the used unsuper- vised saliency mask detector with other recently proposed detection methods. We report the loU for the foreground class while using different methods for foreground object segmentation. In particu- lar, we evaluated segmentation masks proposed in LOST (Simeoni et al., 2021) that uses DINO keys correlation between features, and DINOSeg (Simeoni al., 2021; Caron al., 2021), which uses the
8

---

Published as a conference paper at ICLR 2023
Table 5: Ablation experiment identify the effect of indi- vidual components of the unsupservised learning process.
Table 6: Comparison of COMUS performance with different feature
extractors on PASCAL VOC.
Laplacian Eigenmap
Self- training
Filtering
2nd self- training
mloU
mIoU
X
X
X
X
42.8
COMUS with SwAV
28.6
x
X
43.8
COMUS with iBOT
43.8
V
X
X
47.1
47.6
COMUS with DINO
50.0
50.0
Figure 4: Visualization of foreground masks obtained with different foreground segmentation methods.
Table 7: Comparison between dif- ferent class-agnose foreground seg- mentation methods.
IoU
Unsupervised saliency
51.0
LOST
34.8
DINOSeg
24.5
Supervised saliency
60.5
attention weights from the last layer of the DINO network, to construct foreground object segmen- tation masks (see Figure 4 for examples of predictions by foreground segmentation methods that we consider). The chosen unsupervised saliency based on DeepUSPS and BASNet outperforms both LOST and DINOSeg with large margin showing the importance of additional localization prior in contrast with relying only on DINO as both categorization and localization prior. In addition, we show how the quality of saliency masks proposal affects COMUS performance in App. D.
Limitations and Future Work Directions Although COMUS shows very promising results on the hard tasks of unsupervised object segmentation, there expected. First, although we reduced the dependency on the quality of the saliency detector via suc- cessive self-trae self-training, the approach still fails to segment objects that are rarely marked as salient (see Figure 11 in App. E.2). Second, while the bootstrapping via self-learning can correct some mistakes of the initial discovery stage, it cannot correct all of them and can be itself biased towards self- training data (see App. E. Third, we fixed the number of clusters in spectral clustering based on the known number of categories of the dataset. While COMUS works reasonably with larger num- ber of clusters (we refer to App. A for over-clustering experiments), in fully open data exploration scheme, the optimal number of clusters should be determined automatically.
number of limitations,
be
CONCLUSION
In this work, we presented procedure for semantic object segmentation without using any human annotations clearly improving requires some biases to be assumed or learned from data, we propose to use object-centric datasets on which localization and categorization priors could be learned in self-supervised way. We show that combining those priors together with an iterative self-training procedure leads to significant improvements over previous approaches that rely on dense self-supervised representation learning. This combination reveals the hidden potential of object-centric datasets and allows creating a strong baseline for unsupervised segmentation methods by leveraging and combining learned priors.
previous work. As any unsupervised segmentation method
While research on this task is still in its infancy, our procedure allowed us to tackle a significantly more complex dataset like MS COCO for the first time. Notably, on PASCAL VOC we obtained results that match the best supervised learning results from 2012, before the deep learning era. Hence, the last ten years of research not only have yielded much higher accuracy based on supervised learning, but also allow us to remove all annotation from the learning process.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We would like to thank Maximilian Seitzer and Yash Sharma for insightful discussions and practical advice. In addition, we thank Vit Musil for PASCAL VOC class icons.
REPRODUCIBILITY STATEMENT
Algorithm 1 and Algorithm contain pseudocode for both parts of the COMUS method. Ap- pendix G contains detailed information about the COMUS architecture and all hyperparameters used for COMUS training. Appendix H contains information about the datasets used the paper. The COMUS implementation code is located here: http://ggiabb..omm/aaaaaantt/tuu
REFERENCES
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint arXiv:1901.1133 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9922,, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-96660 2021.
Chang Chen, Fei Deng, and Sungjin Ahn. Roots: Object-centric representation and rendering of 3d scenes. J. Mach. Learn. Res., 22:259-1, 2021.
Liang-Chied Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous 2017.
convolution for semantic image segmentation. ArXiv,
Liang- Chhih Chen, George Papandreou, lasonas Kokkinos, Kevin P. Murphy, and Alan Loddon Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions Pattern Analysis and Machine Intelligence, 40: 834-848, 2018.
Liang- Chighh Chen, Raphael Gontijo Lopes, Bowen Cheng, Maxwell D. Collins, Ekin Dogus Cubuk, Barret Zoph, Hartwig Adam, and Jonathon Shlens. Leveraging semi-supervised learning in video sequences for urban scene segmentation. In ECCV, 2020.
Ming-Ming Cheng, Niloy J. Mitra, Xiaolei Huang, Philip H. S. Torr, and Shi-Min Hu. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3):569-555 2015. doi: 10.1 2AW.TTPA....014.22244..
Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16794-16804, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 321 2223-32 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2009. doi: 10.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: / wtppp//eview.nee////ee foreee//foorome//eo d
10

---

Published as a conference paper at ICLR 2023
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. International Confer- Learning Representations, 2020. URL https: httppss//ppnnneeeee...tteeeee BkxfaTVFwF.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.
The pascal visual object classes challenge 2012 (voc2012) results (2012). In URL http://www. pascal-network. org/challenngss/////voe20 //worsshop/idde html, 2012.
Mark Everingham, SM Eslami, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98-136) 2015.
Klaus Greff, Raphaël Lopez. Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-obbect representation learning with iterative variational inference. Proceedings of the 36nd International Conference on Machine Learning, 2019.
Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Un- supervised Semantic Segmentation by Distilling Feature Correspondences. In International Con- ference on Learning Representations, 2022. URL https: / wttp://iwwweee/oouuneeeee fformmmidd
SaK062 6H10c.
Bharath Hariharan, Pablo Arbeláez, Lubomir D. Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. 2011 International Conference on Computer Vision, pp. 991-998, 2011.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 9726-9735, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Olivier H' enaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zis- serman, João Carreira, and Relja Arandjelovi' c. Object discovery and representation networks. In ECCV, 2022.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning to de- compose and disentangle representations for video prediction. In Advances Neural Information Processing Systems, 2021.
Jyh-Jing Hwang, Stella X. Yu, Jianbo Shi, Maxwell D. Collins, Tien-Ju Yang, Xiao Zhang, and Shang-Chieh Chen. Segsort: Segmentation by discriminative sorting of segments. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7333-7343, 2019.
Xu Ji, Andrea Vedaldi, and João F. Henriques. Invariant information clustering for unsupervised image classification and segmentation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp.
2019.
Tsung-War Ke, Jyh-Jing Hwang, Yunhui Guo, Xudong Wang, and Stella X. Yu. Unsupervised hierarchical semantic segmentation with multiview cosegmentation and clustering transformers. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2561- 2571, 2022.
Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff, Conditional Object-Centric Learning from Video. arXiv preprint arXiv:2111.12594, 2021.
Harold W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2:83-97, 1955.
11

---

Published as a conference paper at ICLR 2023
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft coco: Common objects in context. ECCV, 2014.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 33:11525-11533 2020.
Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In CVPR, 2022.
Sudhanshu Mittal, Maxim Tatarchenko, and Thomas Brox. Semi-supervised semantic segmenta- tion with high-and low-level consistency. IEEE transactions on pattern analysis and machine intelligence, 43(4):1369-1379, 2019.
Li Nanbo, Muhammad Ahmed Raza, Hu Wenbin, Zhaole Sun, and Robert B. Fisher. Object-centric representation learning with generative spatial-temprral factorization. In Advances in Neural Information Processing Systems, 2021.
Tam Nguyen, Maximilian Dax, Chaithanya Kumar Mummadi, Nhung Ngo. Thi Hoai Phuong Nguyen, Zhongyu Lou, and Thomas Brox. DeepUSPS: deep robust unsupervised saliency pre- diction with self-supervision. In Advances in Neural Information Processing Systems, 2019.
Yassine Ouali, Céline Hudelot, and Myriam Tami. Autoregressive unsupervised image segmenta-
tion. ArXiv, 02./2007.02277 2020.
Deepak Pathak, Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional multi- class multiple instance learning. ICLR Workshop, 2015.
Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand. Basnet: Boundary- salient object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.
Chaitanya Ryali, David Schwab, and Ari S. Morcos. Learning background invariance im- proves generalization and robustness in self-supervised learning on imagenet and beyond. In NeurlPS 2021 Workshop on ImageNet: Past, Present, and Future, 2021. URL https: / seapeerrviewweet/ffoor... wtw.ormmmmd222n0neeffooo
Maximilian Seitzer, Max Hom, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel. Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. In International Confer- ence Learning Representations, 2023. URL httpt:ttp//sec.. httttp:://pennrviie..ees. forum?id b9tUk-E.aG.
Oriane Simeoni, Gilles Puy, Huy Vo, Simon Roburin, Spyros Gidaris, Andrei Bursue, Patrick P'erez, Renaud Marlet, and Jean Ponce. Localizing Objects with Self-Supervised Transformers and no Labels. In BMVC, 2021.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-e leams compose. In Interna- tional Conference on Learning Representations, 2022a. URL https::/ /ttttsss///penrevieew.eeet forum=1d=hhorrvwee333
Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for com- plex and naturalistic videos. ArXiv, abs/2205. 14065, 2022b.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 9866):559-2603 2008. URL http: http://lllooo/ppaeeeee////////// papprs//e///////p////// vandermaateen0aa.kee
12

---

Published as a conference paper at ICLR 2023
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis. Mare Proesmans, and Lue Van Gool. Scan: Learning to classify images without labels. In European Conference on Com- puter Vision, pp. 268-285. Springer, 2020.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsuper- vised semantic segmentation by contrasting object mask proposals. In International Conference on Computer Vision, 2021.
Huy V. Vo, Patrick P Preez, and Jean Ponce. Toward unsupervised, multi-object discovery in large- scale image collections. In ECCV, 2020.
Van Huy Vo, Elena Sizikova, Cordelia Schmid, Patrick Pérez, and Jean Ponce. Large-scale unsuper- vised object discovery. Advances in Neural Information Processing Systems, 34:16764-16778, 2021.
Ulrike von Luxburg. A tutorial on spectral clustering. Statistics and Computing, 17:395-416, 2007.
Jingdong Wang, Huaizu Jiang, Zejian Yuan, Ming-Ming Cheng, Xiaowei Hu, and Nanning Zheng- Salient object detection: A discriminative regional feature integration approach. International Journal of Computer Vision, 123(2):251-268, 2017.
Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufreydaz. Self-Supervised Transformers for Unsupervised Object Discovery Using Normalized Cut. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
pp. 14543-14553, June 2022.
Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jie, Jiashi Feng. and Thomas S. Huang. Revisiting dilated convolution: A simple approach for weakly- and semi semi-supervised semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018.
Qizhe Xie, Eduard H. Hovy, Minh- Minh-Thang Luong, and Quoe V. Le. Self-training with noisy student improves imagenet classification. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695. 2020.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18134-18144, 2022.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer-
ence on computer vision, pp. 649-666. Springer. 2016.
Runsheng Zhang, Yaping Huang, Mengyang Pu, Jian Zhang, Qingji Guan, Qi Zou, and Haibin Ling. Object discovery from a single unlabeled image by mining frequent itemsets with multi- scale features. IEEE Transactions on Image Processing, 29:8606-8621, 2020.
Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representa- tions (ICLR), 2022.
Yi Zhu, Zhongyue Zhang, Chongruo Wu, Zhi-Li Zhang, Tong He, Hang Zhang, R. Man- matha, Mu Li, and Alex Smola. Improving semantic segmentation via self-training. ArXiv, abs/2004, 14960, 2020.
Adrian Ziegler and Yuki M. Asano. Self-supervised learning of object parts for semantic segmen- tation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14482-14491, 2022.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
A OVER-CLUSTERING
Table Over-clustering results on PASCAL VOC evaluated with mloU after majority voting. We present the results for 30 clusters, whereas also include the results for 50 clusters for comparison with MaskContrast (Van Gansbeke et al., 2021).
clusters
50 clusters
MaskContrast
41.4
COMUS (Iteration 1)
49.3
46.9
COMUS (Iteration 52.6
51.0
As the number of discovered clusters could be different from the number human-defined cate- process with larger number of clusters than there are ground truth categories (over-clustering). Each cluster was matched to the ground truth category with the highest IoU, i.e., each category have multiple clusters being assigned to it. This kind of matching avoids penal- ization reasonable subcategories (see Fig. 5). Discovery for using unsupervised methods. Table 8 shows that also under this evaluation protocol obtain better results. For comparison to MaskContrast, we also included results with 50 clusters, showing that COMUS outperforms MaskContrast in the clusters setting.
gories,
subcategories is a strong motivation
Self-traine trainiriinn from pseudo-masks with larger number of categories decreased the performance slightly. Potentially the segmentation network has difficulties learning patterns when several clusters have the same semantic interpretation. In case of fixed saliency masks, this evaluation setting yields better numbers and becomes trivial as the number of clusters becomes very large.
Clusher Cluster
Cluster Cluster
Cluster 6
Cluster
Cluerer
Cluster
Figure 5: Visualization of discovered subcategories on PASCAL VOC val set after clustering of self-supervised representations into 30 clusters. The pseudo-masks were randomly sampled for each cluster. Each row shows two clusters of the same category. The clusters have clear semantic interpretation, such as different dog breeds, flying or staying on land airplanes.
14

---

Published as a conference paper at ICLR 2023
Figure 6: (a) COMUS (Iteration 1) performance with different clustering methods. (b) Effect of " filtered" on final performance of COMUS. The results are mean + standard dev. over 5 runs.
52
51
mloU
45
49
40
48
kmeans kmedaids Clustering
50
% filtered
B SENSITIVITY OF THE COMUS PARAMETERS
First, we compare COMUS performance with different clustering methods. In Figure 6a we show that SC provides the best supervision signal for COMUS. Additionally, we investigate the sensitivity of COMUS to the choice of % of filtered examples (see Fig. 6b), showing that [20%-40%] of filtered examples leads to comparably good performance. We observe that larger % of filtered examples leads drop in performance, potentially due to small size of the obtained dataset for training the segmentation network.
Next, look the sensitivity of COMUS to the choice of SC parameters. In Table 9 show that SC is not sensitive to the choice of the number of neighbors and n iinit parameters. SC is sensitive to the number of eigenvectors, however, the default value (equal to the number of clusters) shows very good performance. Similar to other works (e.g. Van Gansbeke et al. (2021); I); Melas-Kyriar al. (2022)), we used the same number of clusters annotated categories (nceded for quantitative evaluation).
Finally, we also study the effect of additional iterations of self-training iterations on trainaug part of PASCAL VOC dataset. We find that trained for two or three iterations COMUS performs similarly, while for even more self-training iterations we observe slow decrease of the performance (Fig. 7). While we were performing additional self-training iterations on the same dataset, in the future work, it interesting to study how our method performs the open-ended regime where new data is available for each iteration.
Figure 7: Number of self-training iter- Table 9:
Spectral Clustering parameters study, perfor-
ations in COMUS training. The results are mean + standard dev. over runs.
mance after the first iteration.
50
n.neighbors
n.components
n-init
Range
20
50]
[10 40)
[10 60)
mlou
48
mloU
46.0 + 1.3
38.8 6.2
47.6 1.0
46
0
4
Itteration of self-training
C SELF-SUPERVISED FEATURES QUALITY
As was showed in the original paper (Caron et al., 2021), DINO features are demonstrating excellent performance for k-NN classification (78.3% top-1 ImageNet accuracy), which reveals the quality of the feature space for clustering. In contrast, other self-supervised methods require fine-tuning of last layer (Caron et al., 2020; He et al., 2020) or several last layers (He et al., 2022). We further confirmed (see Table 10) that DINO performs significantly better than SwAV (Caron et al., 2020)
15

---

Published as a conference paper at ICLR 2023
Table 10: Clustering of random subsets of ImageNet classes.
Top-1 I Accuracy,
50 Classes
100 Classes
200 Classes
SCAN
76.8 81.6 + 0.5
68.9
58.1
SWAV
71.5 + 0,4
59.2 + 0.8
Supervised
91.2 + 0.9
87.5 + 0.3
82.2 + 0.4
DINO
91.3 + 5.1
88.0 = 0.2
83.1 + 0.4
and SCAN method (Van Gansbeke et al., 2020) (based on MoCo (He et al., 2020) features) for image clustering. For this, similar to SCAN methods ( (Van Gansbeke et al., 2020), we picked random subsets of ImageNet categories, consisting of 50, 100 and 200 classes. For this experiment, were using validation images of ImageNet (50 images per category). The results show that DINO features could be used for image clustering with performance comparable with supervised ResNet- 50 features.
D SALIENCY MASKS QUALITY
In this section, we study how COMUS works with different self-supervised and supervised saliency detectors. Overall, we observe that improving original object proposal masks important for both category discovery and further iterative self-training. Also, we note that the proposed iterative self- training from filtered pseudo masks is effective for all of the studied choices of saliency detectors.
D.1 CHOICE OF UNSUPERVISED SALIENCY MASKS DETECTOR
First, we compare COMUS combined with supervied BasNet saliency detector (Qin et al., 2019) (i.e., BasNet pretrained with DeepUSPS masks on MSRA-B dataset) with COMUS that is using recently proposed Spectral Decomposition saliency masks from DeepSpectral (Melas-Kyriazi al., 2022). While original predictions from Spectral Decomposition are performing worse than unsupervised semantic segmentation proposed in DeepSpectral (see the first row of the Table 11), using them in COMUS as objects proposal performs better than DeepSpectral showing the impor- tance of other COMUS components, such the initial discovery of object categories from object proposals, not from clustered segments (as those are not always object-centric and could cover only parts of objects), filtering of the most uncertain pseudo-masks masks and several iterations of self-train self-training on previously unseen data. This way, we are able to outperform DeepSpectral semantic segmenta- tion on more than mloU points, while using only DINO features for both object proposal masks and object representation extraction.
Next, additionally show how COMUS performs if we use original DeepUSPS masks (Nguyen et al., 2019) as object proposals. As we discussed in the main text, DeepUSPS seems to be less robust than self-supervised BasNet more complex OOD images from the PASCAL VOC dataset, that was not used in the training (see the first row of the Table 11). Additional self- self-training iterations are improving the quality of the original object proposals similarly. showing that COMUS can operate with originally lower quality mask proposals in case of successful object categories discovery.
While the original DeepUSPS initialize its backbone weights from supervised pretraining (Nguyen et al., 2019), similar to other areas, it was recently shown that this is not necessary for DeepUSPS good performance (Ryali et al., 2021). They show that DeepUSPS" (Ryali et al., 2021) model that does not use any annotations during backbone pretraining still performs comparable or better than the original USPP Thus, similar to the original DSePUSPS saliency masks could be discovered from architecture where no labels were used. As DSePUSPS2 implementation is not publically avail- able and for better comparison with previous methods in our work, we were using self supervued BasNet from MaskContrast (Van Gansheke et al., 2021).
16

---

Published as a conference paper at ICLR 2023
Table 11: Choice of the unsupervised salient object detector. We compared COMUS performance with three different unsupervised saliency masks detectors: self-supervised BasNet model (Qin al., 2019), original DeepUSPS (Nguyen al., 2019) and spectral decomposition saliency masks from DeepSpectral (Melas-Kyrial et al., 2022). All the models are evaluated with by IoU after Hungarian matching on the PASCAL 2012 val set.
Self-supervised BasNet
Spectral Decomposition
DeepUSPS
COMUS (Iteration 1)
47.6
43.8
45.5
COMUS (Iteration 2)
50.0
45.9
47.5
D.2 SUPERVISED SALIENCY MASKS AS LOCALIZATION PRIOR
In addition to using fully self-supervised saliency masks as localization prior, we also consider using BasNet saliency detector trained with supervision on MSRA-B dataset as localization prior. Supervised training of saliency masks leads to even better masks, but drops the property of the method being fully unsupervised. Table 12 shows that supervised saliency masks also improve the final results, as to be expected.
Table 12: Effect of object proposals from supervised saliency detector.
Unsupervised Saliency
Supervised Saliency
MaskContrast
35.1
38.9
COMUS (Iteration 1)
47.6
50.4
COMUS (Iteration 2)
50.0
52.3
E EXTENDED LIMITATIONS AND FUTURE WORK
E NUMBER SEMANTIC CATEGORIES BIAS
As we are starting our iterative self-training from pseudo-masks restricted to one foreground se- mantic category per image, it is natural to study how well COMUS can work on more complex images where it is more than one semantic category per image by using discovered categories as an additional signal. Thus, we study COMUS performance in comparison with DeepSpectral (Melas- Kyriazi et al., 2022) on two subsets of PASCAL VOC val: the first one is the subset where is only one foreground semantic category (subset 1) and the second one is the subset of images with two or more foreground semantic categories (subset 2). We compare several iterations of self-training with DeepSpectral performance using two metrics: first is the standard mIoU showing the overall quality of the predictions. In addition, we compared the precision of recognizing each group. For example for the first subset, is equal the percentage predictions that contain only one foreground semantic category prediction).
For each method, expected, we observe that the quality of the predictions the subset is worse than on the subset 1. Iterations of self-training are improving COMUS performance, and allowing reaching better quality (measured by mloU) on more complex images from subset than overall DeepSpectral predictions. While improving overall prediction quality, the second iter- ation of COMUS shows bias towards predicting one foreground semantic category per image. This is potentially due to bias in the PASCAL VOC t i nnuug dataset itself. In contrast, DeepSpectral tends to have the opposite bias toward predicting more than one category per image (i.c., has lower precision for the first task and higher precision for the second task). This could be due to the image features clustering task that is used by DeepSpectral for segment extraction. Interestingly none of the methods predicts well the number of foreground masks in both subsets. Thus determining the right number of semantic categories for each image is still a challenging problem for unsupervised semantic segmentation and interesting direction for future work.
17

---

Published as a conference paper at ICLR 2023
Table 13: Number of semantic categories bias. Performance of studied methods on two subsets of PASCALVOC val dataset.
semantic category
I semantic category
overall
mloU
precision, %
mloU
precision, %
mloU
precision, %
DeepSpectral
39.9
50.9
34.5
65.7
37.1
56.2
COMUS (Itteration 1)
52.0
50.5
39.9
61.8
47.6
54.6
COMUS (Itteration 2)
55.2
67.7
41.2
46.5
50.0
60.1
E.2 FAILURE MODES ANALYSIS
We present illustration of several failure cases in Figure 11. We observe that COMUS still failures to discover some categories, such as table category that is treated background by saliency method. Also, when an object fills all the background our method fails to fully recover from initial saliency mask assumption, that objects appears only in the foreground. Finally, as our method has only one category for the background, extending COMUS to additionally split backgrounds (e.g.. COCO-Stuff semantic segmentation masks) an interesting direction for future work.
F MORE DETAILED QUANTITATIVE AND QUALITATIVE RESULTS
F.1 PASCAL VOC
In this subsection, present additional analysis of COMUS performance on PASCAL VOC dataset. First, in Table 14, we show COMUS performance for each PASCAL VOC category. COMUS performs better than MaskContrast and DeepSpectral on most of the categories. In ad- dition to fully unsupervised semantic segmentation methods, we compare COMUS with recently proposed, weakly-supervised GroupViT method (Xu al., 2022). GiT uses text descrip- tions as weak supervision signal to group image regions into progressively larger arbitrary-shaped segments. While does not require any pixel-level annotations, GroupViT still relies on large an- notated datasets containing millions of image-text pairs. On average COMUS performance is worse than GrT performance, COMUS performs better on 9 from 21 categories while using no text annotations. In addition. in Figure 12, visualize COMUS predictions random images from PASCAL VOC dataset for two stages of COMUS as well as MaskContrast predictions. Finally, for exploring Interactive Demo that visualizes clustering of the whole PASCAL VOC val set, visit COMUS project website: https: / tes.gossgggoo...oo c.com/ /vi ew comuspaper home.
Table 14: More detailed comparison prior and iterative improvement via self-training (eval- uated by IoU after Hungarian matching) on the PASCAL 2012 val set. Our method results are averaged over 5 runs.
Mehae
sial
Draptty feor openooee 2002)
151
314
754
161
965
161
169
0.11
76.6
517
25.5
510
151
Deepppecaal plrias Kystal 200))
461
714
43.4
4B6
364
536
Pandoorzaio (handin 2i
S25
481
236
913
407
25A
838
32.7
232
S8.3
37.2
54.7
362
25-5
66.8
17.6
523
339
435 261
2t9
600
A16
66-
471
169
11.9
18.9
961
=64
361
535
6.5
618
(66
193
611
56.1
M6.
117
16.
635
F.2 COCO
In this subsection, we present additional analysis of COMUS performance on COCO dataset. Fig- ure 9 shows the performance of COMUS on COCO for each of 80 COCO categories. We note that COMUS achieves better performance for animal and vehicle categories, as well as salient object categories such stop sign and traffic light categories. We also observe that most of the undiscovered categories have small relative object's size (e.g., spoon, remote and mouse). For additional analysis of the connection between relative object's size and COMUS performance,
18

---

Published as a conference paper at ICLR 2023
Figure 8. In addition, in Figure 13, visualize COMUS predictions random images from PASCAL VOC dataset for two stages of COMUS as well as MaskContrast predictions.
06
05
02
0.0
000
5:10
0.15
0.00
0.%
0:00
Relative size of object
Figure 8: Connection between mean relative size of objects and IoU for each COCO category. The Spearman's rank correlation between relative size and loU equal 0.43.
G IMPLEMENTATION DETAILS
G.1 SPECTRAL CLUSTERING
We spectral clustering implementation from skleam2. In particular, affinity matrix obtained from the nearest neighbors graph. Number of cigenvectors and number of clusters is the same as number GT categories. We refer Table 15 for the parameters of spectral clustering.
Table 15: Spectral clustering parameters for COMUS on PASCAL VOC and MS COCO datasets.
Hyper-parameter
PASCAL VOC
MS COCO
Number of clusters
20
80
Number of components Affinity
20
80
nearest ne neightbors nearest neightbors
Number of neighbors
30
30
G.2 TELININGGN
During self-training. DeepLabv3 (Chen et al., 2018) with standard cross-entropy loss is chosen to make training set up as comparable to previous research (e.g. to MaskContrast (Van Gansbeke et al., 2021)) as possible. We CropRes and Hori werrizontaFFln. For PASCAL VOC, we perform two iterations of self-training on train and trainaug sets. For COCO dataset, we perform one iteration of self-training on train See Table 16 for the parameters of the self-training.
data augmentation methods.
G.3 COMPUTATIONAL REQUIREMENTS
Similar other unsupervised segmentation methods (Van Gansbeke al., 2021; Melas-Kyriat et al., 2022; Hamilton et al., 2022), the most expensive part our pipeline, is the training self- supervised representation learning method. In particular, DINO with Vision Transformers training takes 3 days on two 8-GPU servers and is comparable with other self-supervised representation learning methods. However, learned features could be transferred without further fine-tuning on new natural data images, such as scenes from PASCAL VOC and MS COCO.
The other parts of method much faster train: DeepUSPS could be trained 30 hours of computation time on old four Geforce Titan X (Nguyen et al., 2019), while BasNet could be trained with four GTX 1080ti GPU (with 11GB memory) in around 30 hours (Qin et al., 2019), while the
19

---

Published as a conference paper at ICLR 2023
Table 16: Self-training parameters for COMUS on PASCAL VOC and MS COCO datasets.
Hyper-parameter
PASCAL VOC
MS COCO
Optimizer
Adam with default settings
Adam with default settings
Learning rate
0.00006
0.00006
Batch size
56
Input size
512
256
Crop scales
[0.5, 2]
[0.2, 1.0]
Number of iterations
1
Number of epochs. Iteration
10
1
Number of epochs. Iteration
inference for 256x256 image only takes 0.040s (25 fps). For more details, we refer the reader to the main papers for these methods.
Finally, the main parts of our method that do require training on new data are object proposals clustering and self-training iterations. Spectral Clustering complexity depends on the sample size. For the full COCO train dataset Spectral Clustering with amg solver took 20 minutes on 96 core node. Thus, for large datasets, is recommendable to use its subset for the initial discovery of object categories and then use self self-training on the full dataset. For semantic segmentation self- training, we used the standard in supervised semantic segmentation set up for training DeepLabv3 architecture. While all the models could be trained on single GPU, for convenience we perform all the experiments on one node with NVIDIA T4 GPUs, where 2 iterations of self-training took around one hour.
H DATASETS (DIRECTLY OR INDIRECTLY) USED IN THE PAPER
The PASCAL Visual Object Classes (VOC) project (Everingham al., 2015) provides different datasets challenges for the years from 2005 to 2012. We apply our pro- posed method to the datasets from 2012 and 2007, which come with semantic segmentation masks. All datasets and detailed descriptions are available on the PASCAL VOC homepage htctp://hos..robot..om
PASCAL VOC:
The dataset and informations are available on wwtp://aaaaaaaaaa....a/ocooa
MS COCO:
We also apply our method to the Microsoft (MS) COCO dataset (Lin et al., 2014). htw.ee...r..o.......... #home.
ImageNet:
For feature extraction, we use vision transformers pretrained with the self-supervised (no labels!) DINO method (Caron et al., 2021) on ImageNet (Deng et al., 2009), The pretrained checkpoint can be found on https: : htt////ttttt/coco wwt.:/ccc//aacbbooo reeac /dino. Informa- tions about ImageNet are provided on https: htag:///mmmmeeetooggg/
MSRA-B: For computing saliency mask, we use BasNet Qin et al., 2019) pre- trained on pscudo-labels generated with the unsupervised DeepUSPS (Nguyen et al., 2019) outputs on the MSRA-B dataset (Wang et al., 2017). The pretrained checkpoint can be found on
http://gthhub.co//wwgggggg wewrgggnnbbeee/huuuurrerr (Unsupervised Semaatiinnn ic- hatmennttt..o//rree/aaa/aaaaaa on https: h httpa:////mmmeeennnnessssssl...
The dataset and informations about are available
20

---

Published as a conference paper at ICLR 2023
fire hydrant
bed
bus
refrigerator
zebra
tennis racket
sheep
sink
teddy bear
skateboard
bear
COW
carrot
stop sign
tie
pizza
dining table
motorcycle
kite
clock
donut
snowboard
dog
sports ball
elephant
baseball glove
giraffe
chair
train
cup
hot dog
frisbee
airplane
bottle
horse
backpack
Category
Category
cat
skis
toilet
handbag
traffic light
baseball bat
boat
fork
cell phone
knife
bird
spoon
bicycle
apple
sandwich
parking meter
cake
truck
couch
umbrella
potted plant
orange
mouse
banana
remote
broccoli
keyboard
laptop
microwave
bench
surfboard
toaster
person
book
bowl
scissors
suitcase
hair drier
wine glass
toothbrush
50
100
50 loU
100
loU
Figure 9: IoU for COCO categories after Hungarian matching the cluster IDs to ground-truth categories.
21

---

Published as a conference paper at ICLR 2023
Immgge
Label
COMUS
Figure 10: Predictions of the COMUS method trained on PASCAL VOC on COCO val set. We
notice that the predictions from models trained on PASCAL VOC transfer reasonably well to COCO.
Label
COMUS
Figure 11: Several failure samples of the COMUS method on PASCAL VOC val set. The failures show the limitation and biases of our method, such as bias towards salient objects and misclassifica- tions in multi-object images.
22

---

Published as a conference paper at ICLR 2023
Image
Label
MaskContrast
COMUS (Iteration 1)
COMUS
Image
COMUS
Figure 12: COMUS and MaskContrast predictions on ran- dom images from PASCAL VOC val set.
Figure 13: COMUS predictions on
random images from COCO val set.
23


Reviews:
[{"review_id": "XsbK100ZqV", "reviewer": "Reviewer_6jsH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of unsupervised semantic segmentation from images, which is relatively new in the literature. The methodology is based on saliency detection, clustering of self-supervised features (e.g., DINO) within the salient regions to obtain pseudo-masks, followed by a few iterations of self-training. This method achieves state-of-the-art results on unsupervised semantic segmentation on PASCAL-VOC and MS COCO.  ", "summary_of_the_review": "Although this paper achieves state-of-the-art results on the task of unsupervised semantic segmentation by a large margin, these results may be (to a large degree) due to the superior saliency estimation method that is used in this paper in contrast to prior work. The rest of the pipeline is very similar to that of (Melas-Kyriazi et al., 2022). As a result, I find that the novelty of this paper is not significant enough to warrant acceptance to ICLR.  ", "strength_and_weaknesses": "\n**Strengths**\n\nThe proposed method is simple, yet effective as it achieves over 10 points mIoU above prior work. Results are also shown for the first time on the challenging MS COCO dataset (80 classes).   \n\nThe method directly builds on pre-trained representations, with optional self-training. As such, and in contrast to some other approaches, it does not require costly self-supervised pre-training at object level (i.e., dense representations) to achieve a pixel-wise task, such as semantic segmentation.  \n\n\n**Weaknesses**\n\n(1) One major issue with the paper is its close resemblance to DeepSpectral (Melas-Kyriazi et al., 2022), although the authors do not discuss this explicitly. DeepSpectral uses a similar pipeline for (unsupervised) semantic segmentation: unsupervised extraction of image regions, clustering for categorizing these regions, and self-training. The most notable difference appears to be the choice of method for the extraction of relevant image regions, that is DeepUSPS for this paper and spectral clustering for DeepSpectral. The rest of the pipeline seems to be quite similar with minor changes (spectral vs k-means clustering, or the number of iterations during self-training). This limits the technical contribution of the paper, despite its strong performance. Therefore, the authors should provide an extensive discussion of the differences to DeepSpectral and highlight their significance. \n\n(2) The approach relies on a saliency detection method (DeepUSPS/BASNet) to get an initial set of object masks. Despite common belief, I would argue that DeepUSPS may not be considered *truly* unsupervised since:  \n - it distills an ensemble of *handcrafted* priors\n - it uses *supervised* pretraining (e.g., on ImageNet or Cityscapes according to (Nguyen et al., 2019)) \n\nThis fact gives an inherent advantage to DeepUSPS vs fully unsupervised approaches for saliency detection. This is also clear from Table 7, where DeepUSPS performs significantly better than approaches such as LOST or DINOSeg which only rely on self-supervised features to estimate saliency. This in turn gives a clear advantage to the object proposal masks used by the rest of the method. \n\nNotably, DeepSpectral also relies on self-supervised features (e.g., DINO) for object proposal masks. Therefore, the question that arises is whether the biggest part of the performance boost of this paper could be simply attributed to the advantage of DeepUSPS over DINO-based saliency. The authors could investigate this further, e.g., by swapping DeepUSPS for the object proposal part of DeepSpectral (namely, spectral decomposition). \n\n(3) In most figures, it is notable that this method still struggles with images containing more than one semantic category. This is due to the initial assumption that there is only a single object per image (i.e., assigning a single cluster ID to the entire salient region in an image). Despite self-training, the model does not seem to fully recover from this assumption, especially in PASCAL VOC where the majority of images do indeed contain a single object, so there is little training signal. This likely can be measured, e.g., by measuring the performance and predicted object count on images containing more than one object \n\n(4) The authors could also consider running experiments on COCO-Stuff, to facilitate comparisons with other state-of-the-art methods, such as STEGO (Hamilton et al., 2022) and PiCIE (Cho et al., 2021).  \n\n(5) The authors should also consider citing, discussing, and (where possible) comparing to the following related methods:\n\n[1] Ziegler and Asano, “Self-Supervised Learning of Object Parts for Semantic Segmentation.” (CVPR 2022)\n\n[2] Ke et al. \"Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering Transformers.\" (CVPR 2022)\n\n[3] Henaff et al., “Object discovery and representation networks.” (ECCV 2022)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BRP8Rlto_E", "reviewer": "Reviewer_7EBx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper addresses the problem of unsupervised semantic segmentation with self-supervised object-centric representations. The authors use object-centric datasets on which localization and categorization priors are learned in a self-supervised way, Combining these priors with an iterative self-training procedure allows to obtain highly performing semantic segmentation framework.\nThe method has been validated on 2 public datasets (MS COCO and PASCAL VOC). The abblation study shows that the method is able to identify individual components of the unsupervised learning process for semantic segmentation. Final results on PASCAL VOC achieved 47,3 of mIoU and on MS COCO - 40,7. ", "summary_of_the_review": "This paper addresses an important problem - unsupervised semantic segmentation. The paper proposes a well justified and technically sound pipeline composed of straightforward steps. Validation is extensive and final results are looking very promissing. However, unfortunately the authors do not compare their method with any state of the art supervised and unsupervised semantic segmentation method.", "strength_and_weaknesses": "The strengths are as follows:\n1. The paper is addressing a very important problem - unsupervised semantic segmentation.\n2. The pipeline is composed of straighforward techniques with solid backgrdound.\n3. The final pipeline is very well justified.\n4. State of the art is updated and complete.\n5.  Validation is showing the high performance of the method.\n6. Public datasets are used to validate the method.\n7. Abblation study is presented.\n7. Appendices additionally  give insight on the performance of the method.\n\nWeaknesses:\n1 Although the pipeline is composed of technically sound techniques, in its essence the novelty is limited to how to combine these techniques. Still, this weakness is considered not so substantial given the good justification of the pipeline and the final segmentation results.\n2. My main concern is that the method is not compared to any state of the art unsupervised semantic segmentation technique. In the paper it is said that the performance of the proposed method is comparable to the supervised semantic segmentation but the paper does not report any state of the art (superivesd neither unsupervised segmentation) method performance.\n3. It is highly recommendable to apply the same methodology (datasets, statistical measures, etc.) of other state of the art methods for unsupervised semantic segmentation and compare them. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6A_5mUlKNiH", "reviewer": "Reviewer_m7o9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper makes a solid contribution toward unsupervised multi-object segmentation in images. The COMUS method leverages unsupervised saliency detectors to initially estimate object proposal masks (for accurate object localization). It then uses self-supervised feature representation networks for feature extraction from the designated region. These representations are clustered into different categories for different object discovery. The cluster IDs are combined with the saliency masks to form an initial set of pseudo-labels used for object segmentation, not just category discovery. ", "summary_of_the_review": "Overall the paper is good and the contribution is solid. The experiment that was particularly interesting was COMUS under a distribution shift - how well it performs out-of-distribution and the implications of starting with easier datasets and gradually increasing the difficulty. The reviewer has no grounds for rejection. ", "strength_and_weaknesses": "Strengths:\n* The tackled problem is challenging and of great interest to the research community.\n* State-of-the-art performance (by a large margin compared to very recent published work) on a well-known benchmark PASCAL VOC.\n* The first to report results on a more challenging (more object categories) dataset - MS COCO.\n* The paper presentation and experimental analysis is well thought out and executed - if it proves the method's efficiency.\n* Good ablation studies.\n* The dedicated section with the author's discussion regarding the method's limitations is a plus. \n\nWeaknesses: \n* The design decision of using another saliency model BasNet on the saliency masks from DeepUSPS is not argued. What are the implications of not doing this step and just using the initial saliency maps provided by DeepUSPS - why is this distillation procedure needed? \n\nMinor comments:\n* Algorithm 1 is not referenced in the paper.\n* Table 1 caption - MaskContrast is misspelled.\n* Figure 4 is first referenced on Page 4, but you need to scroll down to Page 9 to actually see it.\n\nQuestion: How can this work be extended (if) to semantic segmentation not just object-centric, to be comparable to methods such as STEGO (referenced in the paper)? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0j14la4Stj", "reviewer": "Reviewer_hDe9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work develops a pipeline for unsupervised semantic segmentation. The distinguishing feature of this approach is its generalization from training on binary classification examples to multiple classes. This is achieved by clustering feature embeddings extracted with the help of unsupervised saliency detection methods and a self-training formulation of multi-class model training.", "summary_of_the_review": "Unsupervised semantic segmentation is an emerging problem domain within the realm of unsupervised learning. This paper makes a notable leap and I can see its potential in inspiring a large following.", "strength_and_weaknesses": "I like this work: it is clearly presented, the pipeline is carefully designed in that it leverages some of the latest advances in unsupervised learning, and achieves a substantial improvement of the segmentation accuracy over previous methods in this problem domain.\nThe experiments also extend to COCO and include some analysis of different design choices for the feature extractor and the saliency detection method. This is very nice.\n\nOn the downside, this multi-stage pipeline may look a bit involved, as it requires pre-training of feature embeddings and of a saliency detection network, clustering and multiple rounds of self-training. It would be great if the paper provided more detail on the computational requirements of these individual steps, since such considerations (model complexity) are often part of the game.\n\nTable 2,4 lack some reference. Why not evaluate MaskContrast or DeepSpectral on VOC/COCO for a comparison? Pre-trained models are either publicly available or could be surely requested from the respective authors.\n\nWhat is the strategy for model selection? (validation data and metric)\nWhat is the effect of more self-training rounds? Would test-time augmentation help (or already used)?\n\nSome analysis of failure modes would be helpful both for understanding the inherent biases in the approach and as inspiration for future work.\n\n(Apologies if I overlooked any answers to these questions — I would be grateful for the pointers.)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%