Boosting Unsupervised Semantic Segmentation with Principal Mask Proposals
Authors: Oliver Hahn
Nikita Araslanov
Simone Schaub-Meyer
Stefan Roth
Venue: Accepted by TMLR ()

Abstract: Unsupervised semantic segmentation aims to automatically partition images into semantically meaningful regions by identifying global semantic categories within an image corpus without any form of annotation. Building upon recent advances in self-supervised representation learning, we focus on how to leverage these large pre-trained models for the downstream task of unsupervised segmentation. We present PriMaPs – Principal Mask Proposals – decomposing images into semantically meaningful masks based on their feature representation. This allows us to realize unsupervised semantic segmentation by fitting class prototypes to PriMaPs with a stochastic expectation-maximization algorithm, PriMaPs-EM. Despite its conceptual simplicity, PriMaPs-EM leads to competitive results across various pre-trained backbone models, including DINO and DINOv2, and across different datasets, such as Cityscapes, COCO-Stuff, and Potsdam-3. Importantly, PriMaPs-EM is able to boost results when applied orthogonally to current state-of-the-art unsupervised semantic segmentation pipelines. Code is available at https://github.com/visinf/primaps.

Published in Transactions on Machine Learning Research (09/2024)
Boosting Unsupervised Semantic Segmentation with Principal Mask Proposals
Oliver Hahn-
Nikita Araslanoy23 Department of Computer Science, TU Darmstadt Munich Center for Machine Leorning (MCML)
Simone Schaub-Meyer Stefan Roth 1.4
Department of Computer Science, TU Munich Aessian.Al
Reviewed on OpenReview: https: // openreview. net/ net/forum? id= Uawa fuwffy
Abstract
Unsupervised semantic segmentation aims to automatically partition images into semanti- cally meaningful regions by identifying global semantic categories within an image corpus without any form of annotation. Building upon recent advances in self-supervised rep- resentation learning, we focus on how to leverage these large pre-trained models for the downstream task of unsupervised segmentation. We present PriMaPs Principal Mask Proposals decomposing images into semantically meaningful masks based their feature representation. This allows us to realize unsupervised semantic segmentation by fitting class protolypes to PriMaPs with stochastic expeclation Despite its conceptual simplicity, PriMaPs-FM leads to competitive results across various pre-trained backbone models, including DINO and DINOv2, and across different datasets, such Cityscapes, COCO-Stuff, and Potsdam-3. Importantly, PriMaPs-EM is able to boost results when applied orthogonally to current state-of-the-art unsupervised semantic segmentation pipelines. Code is available at httpt::/ //tp://gtttttt www/wviiinfpprmaaaaeeeeeeeeeeeee
algorithm, PriMaPs-EM
1 Introduction
Semantic image segmentation is dense prediction task that classifies image pixels into categories from a pre-defined semantic taxonomy. Owing to its fundamental nature, semantic segmentation has broad range of applications, such as image editing, medical imaging, robotics, or autonomous driving (see Minace et al. 2022. for an overview). Addressing this problem via supervised learning requires ground-truth labels for every pixel [Lnng et al. 2015 Ronneberger et al., 2015 Chen et al. 2018b). Such manual annotation is extremely time and resource intensive. For instance, a trained human annotator requires an average of 90 minutes to label up to 30 classes a single MP image (Cordts et al. 2016). While committing significant resources to large-scale annotation efforts achieves excellent results (Kirillov et al. 2023), there is natural interest in more economical approach. Alternative lines of research aim to solve the problem using cheaper so-called "weaker" variants of annotation. For example, image-level supervision describing the semantic categories present in the image, or bounding-boo annotations, can reach impressive levels of segmentation accuracy
DDai et al. 2015 Araslanov & Roth 2020 Oh et al. 2021 Xu et al. 2022 Ru et al. 2023
As an extreme problem scenario toward reducing the annotation effort, unsupervised semantic segmentation aims to consistently discover and categorize image regions in a given data domain without any labels, know- ing only how many classes to discover. Unsupervised semantic segmentation is highly ambiguous class boundaries and the level of categorical granularity are task-dependent However, that typical image datasets have homogeneous underlying laxonomy and exhibit invariant domain char- acteristics. Therefore, it is still feasible to decompose images in such datasets in a semantically meaningful and consistent manner without annotations.
can leverage the fact
While assigning actual semantic labels regions without annotation generally infeasible, the assumption is that the rategories the discovered segments will strongly correlate with human notions of semantic meaning.

---

Published in Transactions on Machine Learning Research (09/2024)
Image
Mask
Mask 2
Mask 3
PriMaPs (all)
Pseudo Label
Figure 1: PriMaPs pseudo-label example. Principal mask proposals (PriMaPs) are iteratively extradted from an image (dashed arrows). Each mask is assigned semantic class, resulting in a pseudo label. The examples are taken from the Cityscapes (top), COCO-Stuff (middle), and Potsdam-3 (bottom) datasets.
Despite the challenges of unsupervised semantic segmentation, we have witnessed remarkable progress on this task in the past years (Ji et al. 2019 Cho et al. 2021 Van Gansbeke al. 2021
2022 Ke et al.) 2022
Yin et al. [022 Hamilton et al.) 2022, Karlsson et al. 2022 Li et al. 2023 Seong et al. 2023 Seitzer et al.
2023 Deep representations obtained with self-supervised learning (SSL), such as DINO
(Caron et al. 2021
have played a critical role in this advance. However, it remains unclear whether previous work leverages the intrinsic properties of the original SSL representations. or merely uses them for "bootstrapping" and learns a new representation on top. Exploiting the inherent properties of SSL features preferable for two reasons. First, training SSL models incurs a substantial computational effort, justifiable only if the learned feature extractor is sufficiently versatile. In other words, one amortize the high computational cost over many downstream tasks, provided that task specialization is computationally negligible. Sccond, studying SSL representations with weight lools, such linear models, leads to more interpretable empirical analysis than with the use of more complex models, as evidenced by the widespread use of linear probing in SSL evaluation. Such interpretability advances rescarch on SSL models toward improved cross-task generalization.
Equipped with issential tools of linear modeling, e. Principal Component Analysis (PCA), we generate Principal Mask Proposals, or PriMaPs, directly from the SSL representation. Complementing previous findings object-centric images (Tumanyan et al. 2022 Amir al.) 2022). show that principal com- ponents of SSL features tend to identify visual patterns with high semantic correlation also in scene-centric imagery. Leveraging PriMaPs and minimalist post-processing, we construct semantic pseudo labels for each image as illustrated in Fig. [ Finally, instead of learning a new embedding on top of the SSL representation (Hamilton et al. 2022 Scong et al.) 2023, Seitzer et al. 2023 Zadaianchuk et al.] 2022), we employ a moving average implementation of stochastic Expectation Maximization (FM) (Chen et al. 2018a) assign a con- sistent category to each segment the pscudo labels and directly optimize class prototypes in the feature space. Our experiments show that this straight approach not only boosts the segmentation accu- racy of the DINO baseline, but also that of more advanced toat--of-thh-att approaches tailored for semantic segmentation, such as STEGO (Hamilton et al.) 2022 and HP (Seong et al. 2023).
We make the following contributions: (i) We derive lightweight mask proposals, leveraging intrinsic properties of the embedding space, e. g., the covariance matrix, provided by an ofe-the-shelf SSL approach. (it) Based on the mask proposals, we construct pscudo labels and employ moving average stochastic EM to assign a consistent semantic class to each proposal. (it) We demonstrate improved segmentation accuracy across a wide range of SSL embeddings and datasets.
2

---

Published in Transactions on Machine Learning Research (09/2024)
2
Related Work
Our work builds upon recent advances in self-supervised representation learning, and takes inspiration from previous unsupervised semantic and instance segmentation methods.
The goal of self-supervised representation learning (SSL) is to provide generic, task-agnostic feature extractors (He et al.) 2020; Chen et al. 2020 Grill et al. 20200. A pivotal role in defining the behavior of self-supervised features on future downstream tasks is taken by the self-supervised objective, the so-called pretent task. Examples of such tasks include predicting the context of a patch (Doersch et al. 2015 or its rotation Gidaris et al. 2018). image inpainting (Pathak et al., 2016). and "solving" jigsaw puzzles [Nroozi & Favaro 2016). Another family of self-supervised techniques is basod on contrastive learning (Chen et al, 2020 Caron et. al.
) More recently, Transformer networks (Dosovitskiy el. al., 2020) revived some older
pretext tasks, such as context prediction (Caron et al. 2021 He et al. 2022). in more data-scalable fashion.
While the standard evaluation practice in SSL (c. 9., linear probing, transfer lcarning) offers some glimpse into the features' properties, understanding the embedding space produced by SSL remains an active terrain for research Ericsson et al.
2021 Naseer et al. 2021 In particular, DINO fcatures (Caron et al.) 2021 Oquab et al. 2024) are known to encode accurate object-specific information, such as object parts (Amir p022p. However, remains unclear to what extent DINO embeddings allow for semantic representation of the more ubiquitous multi-object scenes. Here, following previous work (e. 9-: Hamilton et al.
et al. 2022] Tumanyan et al.
2022: Scong ct al. 2023 provide further insights.
Early techniques for unsupervised semantic segmentation using deep networks (Cho et al. 2021
Van Gansbeke et al. 2021 approach the problem in the spirit of transfer learning and, under certain nomenclatures, may not be considered fully unsupervised. Specifically, starting with supervised ImageNet pre-training (Russakovsky et al. 2015), a network obtains fine-tuning signal from segmentaation-orientel training objectives. Such supervised "bootstrapping" appears to be crucial in the ill-posed unsupervised for- mulation. Unsupervised training of deep model for segmentation from scratch is possible, albeit sacrificing accuracy (Ji et al.) 2019: Ke et al. 2022). However, training new deep model for each downstream task contradicts the spirit of SSL of amortizing the high SSL training costs over many computationally cheap specializations of the learned features (Bommasani et al. 2021
Relying on self-supervised DINO pre-training, recent work (Hamilton et al. 2022 Li et al. 2023 Seong et al, 2023 has demonstrated the potential of such amortization with more lightweight fine-tuning for semantic segmentation. Nevertheless, most of this work has treated the SSL representation learning a new embedding space over the SSL features (e. g., Hamilton et al.) 2022; Seong et al.) 2023). In contrast, following SSL principles, use the SSL representation in more direct and lightweight fashion by extracting mask proposals using linear models (PCA) with minimal post-proccssing and learning a direct mapping from feature to prediction space.
an inductive prior by
Mask proposals have an established role in computer vision Arbelaez et al. 2011 Uijlings et al. 2013 and
remain highly relevant in deep learning (Hwang al. [209 Van Gansbeke ct al. 2021 Yin et al. 2022). Different. from previous work, we directly derive the mask proposals from SSL. representations. Our approach is inspired by the recent use of classical algorithms, such normalized cuts (Neut (hh & Malik, 2000), in the
context of self-supervised seginnntation (Wang et al. 2023a b). However previous approaches (Van Gansbeke
et al. 2021 2022 Wang et al.) 2023a b) mainly proposed foreground object masks on object-centric data, utilized in a multi-step self-training. In contrast, we develop a straightforward method for extracting dense pseudo labels for learning unsupervised semantic segmentation of scene-centric data and show consistent benefits in improving the segmentation accuracy across a variety of baselines and state-of-the-arl methods
(Hamilton al. 2022 Seong al. 2023
3 PriMaPs: Principal Mask Proposals
In this paper, we leverage recent advances in self-supervised representation learning (Caron et al.) 2021 Oquab al. 2024) for the specific downstream task of unsupervised semantic segmentation. Our approach is
3

---

Published in Transactions on Machine Learning Research (09/2024)
based on the observation that such pre-trained features already exhibit intrinsic spatial similarities, capturing semantic correlations, thus providing guidance to fit global pseudo-class representations.
A simple baseline. Consider a simple baseline that applies K-means chustering to DINO VIT features (Caron et al.) 2021p. Surprisingly, this already leads reasonably good unsupervised semantic segmentation results, e. 9., around 15 % mean IoU to segment 27 classes on Cityscapes (Cordts et al., 2016), see Tab. [ However, supervised linear probing between the same feature space and the ground-truth labels the the- oretical upper bound - leads to clearly superior results of almost 36 % mean ToU. Given this gap and the simplicity of the approach, we conclude that there is valuable potential in directly obtaining semantic seg- mentations without enhancing the original feature representation, unlike in previous work (Hamilton et al.
2022
Seong et al. 2023).
From K-means to PriMaPs-EM. When examining the K-means baseline as well as state-of-the-art methods (Hamilton et al. 2022 Seong et al. 2023), see Fig. 4 it can be qualitatively observed that more local consistency within the respective predictions would already lead to less mis-classification. We take inspiration from (Drineas et al. 2004 Ding & He 2004). who showed that the PCA subspace, spanned by principal components, is a relaxed solution to K-means clustcring. We observe that principal components have high semantic correlation for object- as well as scene-centric image features (ef. Fig, D. We utilize this by iteratively partitioning images based on dominant feature patterns, identified by means of the cosine simi- larity of the image features to the respective first principal component. We name the resulting class-agnostic image docomposition PriMaPs Principal Mask Proposals. PriMaPs stem directly from SSL representa- tions and guide the process of unsupervised semantic segmentation. Shown in Fig. 3 our optimization-base approach, PriMaPs-EM, operates over an SSL feature representation computed from a frozen deep neural network backbone. The optimization realizes stochastic EM a clustering objective guided by PriMaPs. Specifically, PriMaPs-EM fits class prototypes to the proposals in a globally consistent manner by optimizing over two identically sized vector sets, with one of them being an exponential moving average (EMA) of the other. We show that PriMaPs-EM enables accurate unsupervised partitioning of images into semantically meaningful regions while being comparatively lightweight and orthogonal to most previous approaches to unsupervised semantic segmentation.
3.1 Deriving PriMaPs
We start with frozen pre-trained self-supervised backbone model F R3xxx + RCXHXW which embeds an image I E 33xxx into a dense feature representation f E RCCHHW as
f F
(1)
Here, C' refers to the channel dimension of the dense features, and H = h/p, W = "/p with corresponding to the output stride of the backbone. Based on this image representation, the next step is to decompose the image into semantically mcaningful masks to provide a local grouping prior for fitting global class prototypes.
Initial principal mask proposal. To identify the initial principal mask proposal in image we analyze the spatial statistical correlations of its features by means of PCA. Specifically, consider the empirical feature covariance matrix
Σ
ΣV 1=15=1
(fras (fass
HW
where Jiis E C are the features at position (i,j) and JERC is the mean feature. To identify the feature direction that captures the largest variance in the feature distribution, we seek the first principal component of Σ by solving
Ev Av.
We obtain the first principal component as the eigenvector in to the largest eigenvalue Åι, which can be computed efficiently with Singular Value Decomposition (SVD) using the flattened features f.
To identify a candidate region, our next goal to compute a spatial feature similarity map to the dominant feature direction. We observe that doing directly with the principal direction does not always lead
4

---

Published in Transactions on Machine Learning Research (09/2024)
Image
Peatures f Map M
Mask p
Peatures f Map M
Mask p
Peatures f Map M Mask pa
PriMaPs (oli)
Figure 2: PriMaPs process. Given the dense feature embeddings f of an image 1, compute the cosine- similarity map M of all features f to their first principal component's nearest neighbor feature. The first PriMaP pl is obtained by thresholding M To obtain P the features assigned to Pl are masked out, and the process is repeated with the remaining features f We repeat the PriMaPs process until the majority of features have been assigned to masks. Finally, all masks P are upsampled and refined using a CRF.
to sufficietlly good localization, i. c., high similarities arise across multiple visual concepts in an image, claborated in more detail in Appendix A.I This can be circumvented by first anchoring the dominant feature vector in the feature map. To that end, we obtain the nearest neighbor feature f E RC of the first principal component on by considering the cosine distance in the normalized feature space f as
f= f.mma
where (m, n) = arg max (vl f).
(4)
i,j
Given this, we compute the cosine-similarity map M E HR of the dominant feature w. r. t. all features as
M = (Mig)ig where
(5)
Next, threshold vi E (0, 1) is applied to the similarity map in order to suppress noise and further localize the initial mask. Accordingly, elements of binary similarity map pl E {0, H are set to 1 when the similarity is larger than a fraction vp of the maximal similarity, and 0 otherwise, i. e.,
pl = MM > 20 max M J.3
(6)
b.
where denotes the Iverson bracket. This binary principal mask pa gives rise to the first principal mask proposal in image I.
Further principal mask proposals. Subsequent mask proposals result from iteratively repeating the described procedure. To that end, it is necessary to suppress features that have already been assigned to a pseudo label. Specifically iteration z, given the mask proposals pe, 8 = ,, 1,, extracted in previous iterations, we mask out the features that have already been considered as
f.a. = fine [ P = P2
(7)
Applying Eqs. to 6 principal mask proposal Pa, and so on. We repeat this procedure until the majority of features (e. g., 95%) have been assigned to a mask. In a final step, the remaining features, in case there are any, are assigned to an "ignore" mask
top of the masked features fz yields the cosine-similarity map M and the
2 1j = - Σ
P
(8)
This produces tensor P (0,1)2xHxW of Z spatial similarity masks, decomposing a single image into Z non-overlapping regions.
Proposal post-processing. To further improve the alignment of the masks with edges and color-correlated regions in the image, a fully connected Conditional Random Field (CRF) with Gaussian edge potentials Krähenbühl & Koltun 2011 applied to the initial mask proposals P (after bilinear upsampling to the image resolution) for 10 inference iterations. The process for obtaining PriMaPs is visualized Fig. 2
PriMaPs pscudo-label generation. In order form a pseudo label for semantic segmentation out of the Z class-agnostic mask proposals, each mask has to be assigned out of K pseudo-class labels. This
5

---

Published in Transactions on Machine Learning Research (09/2024)
Image
Peatures
Image: I
Pseudo label p
y
PriMaPs Pseudo Label
Generation
EMA
Feat. Mask.
1st PC NN
ID Assign
Llocil
P
Similarity Map
CRF
Aug. Image
(a) PriMaPs-EM architecture
(b) PriMal's pseudo-label generation
Figure 3: (a) PriMaPs-EM architccture. An image I and its augmented version I' are embedded by the frozen self-supervised backbone F, resulting in the dense features f and f". The segmentation prediction y by the momentum class prototypes 0Nr arises via the dot product with f. Likewise, y' arises from the dot product of the running class prototypes On with f'. Pscudo labels P are constructed using PriMaPs, I, and y. We use the pseudo labels to optimize OR, applying a focal loss. On is gradually transferred to Os by means of an EMA. (b) PriMaPs pseudo-label generation. Masks P are proposed by iterative binary partitioning based on the cosine similarity the features of any unassigned pixel to their first principal component's ncarest neighbor feature. Gray indicates these iterative steps. Next, the masks P are aligned to the image using CRF. Finally, a per-mask pseudo-class IDD is assigned using majority voting based the segmentation prediction y, resulting in the pseudo label Pa.
is accomplished using segmentation prediction of our optimization process, called PriMaPs-EM. detailed in Sec. 3.2 Given segmentation prediction y, we assign the pseudo-class ID that is most frequently predicted within each proposal, yielding the final pseudo-label map p E {0, K a one-hot encoding of a pleudo-class ID. The entire PriMaPs beeudo-label generation process is illustrated Figure 3b
3.2 PriMaPs-EM
Shown Fig. 3 PriMaPs-EM an iterative optimization technique for fitting class prototypes for semantic segmentation across a dataset utilizing pscudo labels p based on PriMaPs (cf. Sec. 3.1. PriMaPs pro- vide guidance through local per-image consistency for fitting the global class protolypes across dataset. PriMaPs-EM leverages frozen pre-trained self-supervised backbone model F and optimizes two identically sized vector sets, the running class prototypes On and their moving average, the momentum class prototypes 0M.- The class prototypes Og and ON are the K pseudo-class representations in the feature space, project- ing the C-dimensional features linearly to K semantic pseudo classes, which equates to segmenting. More precisely, we use the segmentation prediction the momentum class prototypes Owr to assign consistent se- mantic pseudo-class IDs to the class- agnssii PriMaPs. We optimize the running class prototypes 0R using the pseudo labels pa and gradually transfer On to Oss using F.M.A
PriMaPs-EM performs the optimization in two stages, since in our case, meaningful initialization of the class prototypes is vital to provide a reasonable optimization signal. This can be traced back to the pseudo- label generation, which utilizes a segmentation prediction to assign globally consistent classes to the masks. Initializing the class prototypes randomly leads to a highly unstable and noisy signal.
We initialize the class prototypes On with the first K principal components using vanilla PCA computed over the feature embeddings of a large number of images across the respective dataset. Next, a cosine-distance batch-wise K-mcans [MacQueen 1967) loss
Initialization.
L L--means (0m) = Σ max max(0Tr (r.s)
is minimized with respect 0 for fixed number of epochs. This minimizes the cumulative cosine distances of the image features fay their respective closest class prototype. 0R initialized with the same prototypes.

---

Published in Transactions on Machine Learning Research (09/2024)
Moving average stochastic EM. Visualized in Fig. 3a every optimization iteration starts with com- puting the segmentation prediction y by the dot product of the dense image features ʃ of the image 7 with the momentum class prototypes Oar. Applying the arg max results in the dominant prototype for each fea- ture location and, accordingly, in prototype-based binary masks y E {0, 13 Kxxxx, Note that we bilincarly upsample the Teatures to the image resolution before the dot product. Using y, S, and 1, we gel PriMaPs pseudo labels P* as described in Sec. 3.1 We obtain the semantic prediction s/ by the dot product of the running class prototypes OR with the dense image features f' of the augmented image I'. OR is optimized by applying batch-wise focal loss (Lin et al. 2020) with respect to the pseudo labels P The focal loss Ltocal is a weighted version of the cross-entropy loss, increasing the loss contribution of less confident classes, i.e.
(0x; --00)------- (1) (og(Vkk)))
(10)
k,L.J
where sk.., (".. are the predictions and X is the class-wise confidence value approximated by averaging Vi... spatially, The rumming class prototypes 0₂ are optimized with an augmented input image I. We employ photometric augmentations (Gaussian blur, grayscaling, and color jitter), introducing a controlled noise, thereby strengthening the robustness of our class representation. The momentum class prototypes Bas are the EMA of the running class prototypes dn. This is utilized in order to stabilize the optimization, accounting for the noisy nature of the unsupervised signal used for optimization. We update as every K iterations with a decay y as
03~~= = 700 + -1-
(11)
where is the iteration index of the previous update. This optimization approach resembles moving average stochastic FM. Hereby, the E-step amounts to finding pseudo labels using PriMaPs and the momentum class prototypes. The M-step optimizes the rumning class prototypes with respect to their focal loss Liocal Stochasticity arises from performing EM in mini-batches of images.
Inference. At inforence time, we obtain the sogmentation prediction y via the momentum class proto- types fM, refined using a fully connected CRF with Gaussian edge potentials (Käähenüthl & Koltun 2011)
following previous approaches (Van Gansbeke et al., 2021 Hamilton et al. 2022 Seong al.
2023 This
is the identical CRF as already used for refining the masks in the PriMaPs pseudo-label generation using
the same CRF parameters as previous work Van Gansbeke et al. 2021
Hamilton et al. 2022 Seong et al.
2023
4 Experiments
To assess the efficacy of our approach, we compare it to the current state-of-the-art in unsupervised semantic segmentation. For fair comparison, we closely follow the overall setup used by numerous previous works
(Ji et al. 2019 Cho et al. 2021 Hamilton et al. 2022 Scong et al. 2023)
4.1 Experimental Setup
Datasets.
Following the practice of previous work, conduct experiments on Cityscapes (Cordts et al. 2016), COCO-Stuff (Caesar et al. 2018) and Potsdam-3 (ISPRS). Cityscapes and COCO-Stufe are evaluated using 27 classes, while Potsdam is evaluated on the 3-class variant. Adopting the established evaluation protocol (Ji et al.) 2019 Cho et al.) 2021 Hamilton et al.) 2022 Seong et al., 2023), we resize images to 320 pixels along the smaller axis and crop the center 320 x 320 pixels. This is adjusted to 322 pixels for DINOv2. Different from previous work, we apply this simple scheme throughout this work, thus dispensing
with claborate multi-crop approaches of previous mothods (Hamilton et al. 2022 Yin et al. 2022 Scong
el al.
2023
Self-supervised backbone. Experiments are conducted across a collection of pre-trained self-supervised feature embeddings: DINO (Caron et al. 2021 based on VIT-Small and ViT- Base using 8 x patches; and DINOv2 (Qquab et al.] 2024) based on ViT-Small and ViT-Base using 14 x 14 patches. In the spirit of SSL principles, we keep the backbone parameters frozen throughout the experiments. We use the output from the last network layer our SSL feature embeddings. Since PriMaPs-Fi EM is agnostic to the used embedding
7

---

Published in Transactions on Machine Learning Research (09/2024)
Table I: Cityscapes PriMaPs-EM (Ours) comparison to existing unsupervised semantic seg- mentation methods, using Accuracy and mean IoU (in %) for unsupervised and supervised probing. Double citations refer to a method's origin and the work conducting the experiment.
Unsuperoised
Supervised
Method
Backbone
Ace
mIoU
Ace
mIoU
IIC (Ji et al. 2019
Cho et al.
2021
47.9
6.4
MDC
Caron et al.
2018
Cho al.
2021
R8sNet18 +FPN
40.7
7.1
PiCIE
Cho et al.
2021
65.5
12.3
VICE Karlsson
2022
31.9
12.8
86.3
31.6
Baseline (Caron el al. 2021
61.4
15.8
91.0
35.4
+ TransFGU Yin et al.
2022
77.9
16.8
DINO VIT-S/8
+ HP (Song el al.
2023
80.1
18.4
91.2
30.6
+ PriMa EM
81.2
19.4
91.0
35.4
+ HP (Song et al.
2023
PriMaPs-EM
76.6
19.2
91.2
30.6
Buseline (Caron el al. 2021
49.2
15.5
91.6
35.9
+ STEGO (Hamilton et al.
2022
Koenig el al.
2023
73.2
21.0
89.6
28.0
DINO VIT-B/8
+ HP (eong et 2023
79.5
18.4
90.9
33.0
+ PriMal's s- EM
59.6
17.6
91.6
35.9
+ -STEGO (Hamilton al.
2022
PriMaPs-EM
78.6
21.6
89.6
28.0
Baseline Oquab al.
2024
DINOv2
49.5
15.3
90.8
41.9
+ PriMMal'-EM
VIT-S/14
71.5
19.0
90.8
41.9
Baseline (Quaab al.
2024
DINOv2
36.1
14.9
91.0
44.8
+ PriMaPs-EM
ViT-B/14
82.9
21.3
91.0
44.8
space, we can also apply it on top of current state-of-the-art unsupervised segmentation pipelines. Here, we consider STEGO (Hamilton et al.) (2022) and HP (Seong et al. 2023). which also use DINO features but learn target domain-specific subspace.
Baseline. Following Hamilton et al. 2022 Seong et al. 2023 we train a single linear layer as a baseline with the same structure as OR and 0M by minimizing the cosine-distance batch-wise K-Means loss from Eq. Hereby, parameters, such as the number epochs and the learning rate, are identical to those used when employing PriMaPs-FM.
PriMaPs-EM. As discussed in Sec. 3.2 the momentum class prototypes 0 are initialized using the first K principal components; use 2975 images for PCA, this is the largest number of training images shared by all datasets. Next, 0x is pre-trained by minimizing Eq. (9) using Adam (Kingma & Ba 2015). We use a learning rate of 0.005 for 2 epochs on all datasets and backbones. The weights are then copied to 0n- For fitting the running class prototypes using EM, OR is optimized by minimizing the focal loss from Eq. (10) with Adam (Kingma & Ba 2015) using learning rate of 0.005. The momentum class prototypes 6.r are updated using an EMA according to Eq. [1] every % = 10 steps with decay Tv 0.98. We set the PriMaPs mask-proposal threshold to 0.4 and provide detailed ablation experiments in Appendix A.2 We use batch size of 32 for 50 epochs Cityscapes and Potsdam-3, and use 5 epochs larger size. Importantly, the same hyperparameters are used across all datasets and backbones. Moreover, note that fitting class prototypes with PriMaPs-FM quite practical, e. g., about hours on Cityscapes. Experiments are conducted on a single NVIDIA A6000 GPU.
COCO-Stur due to its
Supervised upper bounds. To assess the potential of the SSL features used, we report supervised upper bounds. Specifically, train linear layer using cross entropy and Adam with learning rate of 0.005. Since PriMaPs-FM uses frozen SSL features, its supervised bound is the same as that of the underlying features. This is not the case, however, for prior work (Hamilton et al. 2022 Seong et al. 2023), which project the feature representation affecting the upper bound.
Evaluation. For inference, we use the prediction from the momentum class prototypes 8.. CRF refinement 10 inference iterations and standard parameters 4, b 3, 0., 67, 0g 3, 0, = from prior work
8

---

Published in Transactions on Machine Learning Research (09/2024)
Table 2: COCO-Stuff PriMaPs-EM (Ours) comparison to existing unsupervised semantic segmentation methods, using Accuracy and mean IoU (in %) for unsupervised and supervised probing. Double citations refer to a method's origin and the work conducting the experiment.
Unsuperoised
Supervised
Method
Backbone
Ace
mIoU
Acc
mIoU
IIC (Ji et al. 2019
Cho et al.
2021
21.8
6.7
44.5
8.4
MDC Caron et al.
2018
Cho al.
2021
32.2
9.8
48.6
13.3
ResNet18 +FPN
PICIE
Cho et al.
2021
48.1
13.8
54.2
13.9
PICIE
11 Cho el
2021
50.0
14.4
54.8
14.8
VICE Karlsson et al.
2022
28.9
11.4
62.8
25.5
Baseline CCron et al. 2021
34.2
9.5
72.0
41.3
+ TrausFGU Yin et al. 2022)
52.7
17.5
+ STEGO (Hamlton
2022
48.3
24.5
74.4
38.3
DINO VIT-S/8
+ ACSeg (Li el al.
2023
16.4
+ HP (Seong et 2023
57.2
24.6
75.6
42.7
+ PriMal's-
46.5
16.4
72.0
41.3
+ HP (Song et al.
2023
PriMaPs-EM
57.8
25.1
75.6
42.7
Buseline (Caron el al. 2021
38.8
15.7
74.0
44.6
+ STEGO Hamilton et al.
2022
DINO VIT-B/8
56.9
28.2
76.1
41.0
+ -PriMaPs-
48.5
21.9
74.0
44.6
+ -STEGO Hamilton et al. 2022
PriMaPs-EM
57.9
29.7
76.1
41.0
Baseline Oquab et al. 2024)
DINOv2
44.5
22.9
77.9
52.8
PPriMal'-EEM
VIT-S/14
46.5
23.8
77.9
52.8
Baseline Oquab al.
2024
DINOv2
35.0
17.9
77.3
53.7
+ PrriMal's
VIT-B/14
52.8
23.6
77.3
53.7
(an Gansbeke et al. 2021 Hamilton et al. 2022 Seong et al, 2023 We evaluate common metrics unsupervised semantic segmentation, specifically the mean Intersection over Union (mIoU) and Accuracy (Acc) over all classes after aligning the predicted class IDs with ground-truth labels by means of Hungarian matching Kuhn 1955
SotA PriMaPs-EM. To explore our method's potential, additionally employ PriMaPs-EM top of STEGO (Hamilton et al. 2022) and HP (Seong et al. 2023). For each backbone-dataset combination, apply it on top of the best previous method in terms of mloU, To that end, the training signal for learning the feature projection of (Hamilton et al.] 2022 Seong et al. 2023) remains unchanged. We apply PriMaPs-FM fully orthogonally, using the DINO backbone features for pseudo-label generation and fit. a direct connection between the feature space of the state-of-the-art method and the prediction space.
4.2 Results
We compare FMM against prior work for unsupervised semantic segmentation (Ji et al. 2019 Cho
et al.] 2021; Hamilton et al. 2022 work, we use DINO (Caron et al. 2021 as the main baseline. Additionall,, we also test PriMa EM on top of DINOv2 (Oquab al.
Yin et al.) 2022 Li et al. 2023 Seong et al. 2023). As in previous
2024], STEGO (Hamilton et al. 2022), and HP (Seong et al., 2023 Overall, we observe that the DINO baseline already achieves strong results (cf. Tabs. to DINOv2 features significantly raise the supervised upper bounds in terms of Acc and mloU, the improvement in the unsupervised case remains more modest. Nevertheless, PriMaPs-EM further boosts the unsupervised segmentation performance.
In Tab. I we compare to previous work on the Cityscapes dataset. PriMaPs-EM leads to a consistent improvement over all baselines terms of unsupervised segmentation accuracy. For example, PriMaPs- EM boosts DINO ViT-S/8 by +3.6% and +19.8% in terms of mIoU and Acc, respectively, which leads state-of thee-af-hhe-rtt performance. Notably, we find PriMaPs-EM be complementary other state-of thate-of-the-rt unsupervised segmentation methods like STEGO (Hamilton et al. 2022) and HP (Seong et. al. 2023 the
9

---

Published in Transactions on Machine Learning Research (09/2024)
Table 3: Potsdam-3 PriMaPs-EM (Ours) comparison to existing unsupervised semantic seg- mentation methods, using Accuracy and mean IoU (in %) for unsupervised and supervised probing. Double citations refer to a method's origin and the work conducting the experiment.
Unsuperoised
Supervised
Method
Backbone
Ace
mloU
Acc
mIoU
RandomCNN (Cho et al.
2021
38.2
K-Means Pedregosa cl
2011
et al.
2021
45.7
SIFT Lowe 2004 Cho et al. 2021
38.2
VGG 11
ContexTPredeei Prediction Doersch
2015
Cho el al.
2021
49.6
CC Isola et al. 2015 Cho et al.
2021
63.9
DeepCluster Caron el al. 2018
Cho el al.
2021
41.7
IIC Ji et 2019 Cho
2021
65.1
Baseline (Caron
2021
56.6
33.6
82.0
69.0
+ +STEGO Hamiloo el al.
2022
Koenig el al.
2023
DINO VIT-S/8
77.0
62.6
85.9
74.8
+ +PriMaPs-EMM FM
62.5
38.9
82.0
69.0
+ STEGO (Hamilton al.
2022
PriMaPs-EM
78.4
64.2
85.9
74.8
Baseline Caron
2021
66.1
49.4
84.3
72.8
+ HP (Seong el al.
2023
DINO
82.4
69.1
88.0
78.4
+ PriMaPs EM
VIT-B/8
80.5
67.0
84.3
72.8
+ HP (Soong et
2023
PriMaPs-EM
83.3
71.0
88.0
78.4
Baseline (Quuab et al. 2024
DINOv2
75.9
61.0
86.6
76.2
+ PriMaPs-EM
ViT-S/14
78.5
64.3
86.6
76.2
Baseline Oquab et al. PPriMal'sEE.
2024
DINOv2
82.4
69.9
87.9
78.3
ViT-B/14
83.2
71.1
87.9
78.3
corresponding backbone model. This suggests that these methods use their SSL representation only to limited extent and do not fully leverage the inherent properties of the underlying SSL embeddings. Similar observations can be drawn for the experiments on COCO-Stuff in Tab. 2 PriMaPs-EM leads to a consistent improvement across all four SSL baselines, as well as an improvement over STEGO and HP. For instance, combining STEGO with PriMaPs leads to +14.0% and +19.1% improvement over the baseline in terms of mIoU and Ace for DINO ViT-B/8. Experiments on the Potsdam-3 dataset follow the same pattern (ef. Tab. 3). PriMaPs-EM leads to consistent, gain over the baseline, e. g- +17.6% and +14.4% in terms of mloU and Acc, respectively, for DINO ViT-B/8. Moreover, it also boosts the accuracy of STEGO and HP. In some cases, the gain of PriMaPs-EM is limited. For example, in Tab. for DINO ViT-B/8 + PriMaPs-EM, the class protolype for "sidewalk" is poor while the classes "road" and "vegetation" " superimpose smaller objects. For DINO ViT-S/8 + PriMaPs-EM in Tab. 3 the class prototype "road" is poor. This limits the overall performance of our method while still outperforming the respective baseline both cases.
Overall, PriMaPs-EM provides modest but consistent benefits over a wide range baselines and datasets and reaches competitive segmentation performance u.. f. the state-of-the-art using identical hyperparameters across all backbones and datasets. Recalling the simplicity of the techniques behind PriMaPs, believe that this is significant result. The complementary effect of PriMaPs EM on other state-of-the-art methods (STEGO, HP) further suggests that they rely on DINO features for mere "bootstrapping" and learn feature representations with orthogonal propertics to those of DINO. We conclude that PriMaPs-EM constitutes a straightforward, entirely orthogonal tool for boosting unsupervised semantic segmentation.
4.3 Ablation Study
To untangle the factors behind PriMaP=-FM, we examine the individual components variety of ablation experiments access the contribution.
PriMaPs pseudo-label ablations. In Tab. 4a we analyze the contribution of the individual sub-steps for PriMaPs pseudo-label generation by increasing the complexity of label generation. We provide the DINO
10

---

Published in Transactions on Machine Learning Research (09/2024)
Table 4: Ablation study analyzing design choices and components in the PriMaPs pseudo-label generation (a) and PriMaPs-EM (b) for COCO-Stuff using DINO ViT-B/8.
(a) PriMaPs pseudo-label ablation
(b) PriMaPs-EM ablation
Method
Acc
mIoU
Method
Acc
mIoU
Baseline (Caron et al. 2021
38.8
15.7
Baseline (Caron et al. 2021
38.8
15.7
Similarity Masks
46.3
19.8
PriMaPs pseudo label
38.8
18.0
+NN
44.9
20.0
+ +EMA
45.0
20.2
+ P-CRRF (= PriMaPs-EMI EM)
48.4
21.9
Augment
46.0
20.1
PriMaPs- EM(unnniie. PC)
47.9
21.7
CRF (= PriMaPs-EM)
48.4
21.9
Table 5: Oracle quality assessment of PriMaPs pseudo labels for Cityscapes, COCO-Stuff, and Potsdam-3 by assigning oracle class IDs to the masks. "Pseudo" refers to evaluating only the pixels contained in the pseudo label, "All" to evaluating including the "ignore" assignments of the pseudo label.
Cityscapes
COCO-Stuff
Potsdam-3
Method
Ace
mloU
Ace
mIoU
Ace
mloU
Pseudo
92.4
54.0
93.4
82.4
95.2
90.9
All
73.2
32.4
74.1
55.9
67.4
48.9
DINO VIT-B/8 Baseline
(Caron et al. 2021)
49.2
15.5
38.8
15.7
66.1
49.4
baseline, which corresponds to K-means feature clustering, for reference. In the most simplified case, we directly use the similarity mask, similar to Eq. 44. Next, we use the nearest neighbor (+NN in Tab. [aa] of the principal component to get the masks as in Eq. (5), followed by the full approach with CRF refinement (+P-CRF). Except for the changes in the pseudo-label generation, the optimization remains as described in Sec. 4.1 We observe that the similarity masks alrcady provide good staring point, yet we identify a gain from every single component step. This suggests that using the nearest neighbor improves the localization of the similarity mask. Similarly, CRF refinement improves the alignment between the masks and the image content. We also experiment with using the respective next principal component (non-iter. PC) instead of iteratively extracting the first principal component from the masked features. This leads to slightly inferior results. Naively using the K leading eigenvectors and simply assigning the masks based the arg max of their cosine similarity to the features without any iterations would lead to significantly worse results with pixel Accuracy of 43.1 % and mloU of 19.9 %. Note that nearest neighbor anchoring and thresholding are used in both experiments. Additionally, we ablate the nearest neighbor anchoring, the threshold vb and the stop criterion in Appendix A
PriMaPs-EM architecture ablations. In a similar vein, we analyze the contribution of the different architectural components of PriMaPs-EM. Optimizing over single set of class prototypes using the proposed PriMaPs pseudo labels already provides moderate improvement (+PriMaPs pseudo label in Tab. 4b), despite the disadvantage of an unstable and noisy optimization. Adding the FMA (+EMA) leads to more stable optimization and further improved segmentation. Augmenting the input (+Augment) results in a further gradual improvement. We provide a more detailed breakdown of the augmentations in Appendix A.4. Similarly, refining the prediction with a CRF improves the results further (+CRF).
Assessing PriMaPs pseudo labels. To estimate the quality of the pseudo labels, respectively the principal masks, decouple those from the class ID assignment by providing the oracle ground-truth class for each mask in Tab. 5 To that end, we evaluate all pixels included in our pseudo labels ("Pseudo"), corresponding to the upper bound of our optimization signal. Furthermore, we evaluate "All" by assigning the "ignore" pixels to a wrong class. The results indicate a high quality of the pscudo-label maps. We show qualitative examples of the PriMaPs mask proposals and pseudo labels in Appendix B.5
Qualitative results. We show qualitative results for Cityscapes, COCO-Stuff, and Potsdam-3 3 in Fig. We observe that PriMaPs-FM EM leads to less noisy results compared to the baseline, showcasing improved
11

---

Published in Transactions on Machine Learning Research (09/2024)
Cityscapes
COCO- Stuff
Potsdam-3
Imnage
Ground Truth
Basellne
PriMaPs-ER
STEGO + PriMaPa-EMM STEGO
Figure 4: Qualitative results for the DINO ViT-B/8 baseline, PriMaPs-FM (Ours), STEGO (Hamilton et al. 2022), and STEGO+PiMaPP-EMM (Ours) for Cityscapes, COCO-Stuff, and Potsdam-3. Our method produces locally more consistent segmentation results reducing overall misclassification compared to the corresponding baseline.
local consistency of the segmentation and reduced mis-classification The comparison with STEGO as a base- line exhibits a similar trend. For further examples and comparisons with HP, please refer to Appendix B.2
Limitations. One of the main challenges is to distinguish between classes that happen to share highly similar SSL feature representations. This is hardly avoidable the feature representation fixed, as was the case here and previous work (Hamilton et al. 2022: Seong et al. 2023 Another limitation across existing unsupervised semantic segmentation approaches is the limited spatial resolution. This limitation comes from the SSL training objectives (Caron et al.) 2021 Oquab al.] 2024), which are image-level rather than pixel-level. As result, observe difficulties in sogmenting very small, fincly resolved structures.
5 Conclusion
We present PriMaPs, novel dense pseudo-label generation approach for unsupervised semantic segmen- tation. We derive lightweight mask proposals dircctly from off-tho-shelf self-supervised learned features, leveraging the intrinsic properties of their embedding space. Our mask proposals can be used as pseudo la- bels to effectively global class prototypes using moving average stochastic EM with PriMaPs-EM. Despite the simplicity, PriMaPs-EM leads to a consistent boost unsupervised segmentation accuracy when applied variety SSL features or orthogonally to current stato-of-the-ar unsupervisod semantic segmentation pipelines, shown by results across multiple datasets.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Acknowledgments
This project is partially funded by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 866008) as well as the State of Hesse (Germany) through the cluster projects "The Third Wave of Artificial Intelligence (3AI)" and "The Adaptive Mind (TAM)".
References
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep ViT features as dense visual descriptors. In ECCVW, volume 13804, pp. 39-55, 55, 2022.
Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In CVPR, pp- 4253 4262, 2020.
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE T. Pollern Anal. Mach. Intell., 33(5):898 916, 2011.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, and Simran Arora et al. On the oppor- tumities and risks of foundation models. arXiv:2108.072: 07258 Lcs. 2021.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stuff: Thing and stuff classes in context. CVPR, pp. 1209 1218, 2018.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In ECCV, volume 11218, pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. In NeurIPS 20200, volume 33, pp- 9912-9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-superiseed vision transformers. In ICCV, pp. 9650 9660, 2021.
Jianfei Chen, Jun Zhu, Yee Whye Teh, and Tong Zhang. Stochastic expectation maximization with variance reduction. In NeurIPS$2011 volume 31, pp. 7967-7977, 2018a.
Liang-Chiel Chen, George Papandreon, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE T. Pattern Anal. Mach. Intell., (0(4):8348845 2018b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pp. 1597 1607, 2020.
Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. PICIE: Unsupervised semantic seg- mentation using invariance and cquivariance in clustering. In CVPR, pp. 16794-16804, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Be- nenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, pp. 3213 3223, 2016.
Jifeng Dai, Kaiming He, and Jian Sun. BoxSup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In ICCV, pp. 1635 1643, 2015.
Chris Ding and Xiaofeng He. K-means clustering via principal component analysis. In JCML, 2004.
Carl Doersch, Abhinav Kumar Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In ICCV, pp. 1422-1430) 2015.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostala Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
Petros Drineas, Alan M. Frieze, Ravi Kannan, Santosh S. Vempala, and V. Vinay, Clustering large graphs via the singular value docomposition. Machine Learning, 56(1-3):9-33, 2004.
Linus Ericsson, Henry Gouk, and Timothy M. Hospodales. How well do self-supervised models transfer? CVPR, pp. 5414 5423, 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In JCLR, 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchalskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent A new approach to self-supervieed learning. In NewrIPS#2020 volume 33, pp. 21271-21284, 2020.
Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Unsupervised semantic segmentation by distilling feature correspondences. In ICLR, 2022.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast. for unsupervised visual representation learning. In CVPR, pp. 9729 9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pp. 16000-16009, 2022.
Jyh-Jing Hwang, Stella X. Yu, Jianbo Shi, Maxwell D. Collins, Tien-Ju Yang, Xiao Zhang, and Liang-Chicl Chen. SegSort: Segmentation by discriminative sorting of segments. In ICCV, pp. 7333 7343. IEEE, 2019.
Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward Н Adelson. Learning visual groups from co- occurrences in space and time. In arXiu:1511.0681 [cs. CVJ, 2015.
ISPRS. ISPRS 2d semantic labeling contest.
https: http::///wwww.t isprs httg//dducatioon/eenmmaaar
UrbanSemLab Accessed: 2024-04-19.
Xu Ji, Joao F. Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, pp. 9865-9874, 2019.
Robin Karlsson, Tomoki Hayashi, Keisuke Fujii, Alexander Carballo, Kento Ohtani, and Kazuya Takeda. VICE: Improving dense representation learning by superpixeliaatio and contrasting cluster assignment. In BMVC, 2022.
Tsung- Wei Ke, Jyh-Jing Hwang, Yunhui Guo, Xudong Wang, and Stella X. Yu. Unsupervised hicrarchical semantic segmentation with multiview cosegmentation and clustering transformers. In CVPR, pp. 2561- 2571, 2022.
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In ICCV, 2023.
Alexander Koenig, Maximilian Schambach, and Johannes Otterbach. Uncovering the inner workings of STEGO for safe unsupervised semantic segmentation. In CVPRW, 2023.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected CRFs with Gaussian edge potentials. In N2P1* volume 24, 2011.
Harold W. Kuhn. The Hungarian method for the assignment, problem. In NVL, volume 52, 1955.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Kehan Li, Zhennan Wang, Zesen Cheng, Runyi Yu, Yian Zhao, Guoli Song, Li Yuan, and Jie Chen. Dynamic clustering network for unsupervised semantic segmentation. In CVPR, pp. 7162-7172, 2023.
Tsung-Yi Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. IEEE T. Pattern Anal. Mach. Intell,, 42(2):318-327, 2020.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, pp. 3431-3440, 2015.
David G. Lowe. Distinctive image features from scale-invariane keypoints. Int. J. Comput. Vision, 60(2): 91-110, 2004.
James MacQueen. Some methods for classification and analysis of multivariate observations. In Berkeley Symp. on Math. Statist, and Prob., volume 1, pp. 281-297. Oakland, CA, USA, 1967.
Shervin Minacc, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning; A survey. IEEE T. Pattern Anal. Mach. Intell., 44(7):3523 3542, 2022.
Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Hsuan Yang. Intriguing properties of vision transformers. In NewrIPS#2021, pp. 23296 23308, 2021.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. Tn ECCV, volume 9910, pp- 69 84, 2016.
Youngmin Oh, Beomjun Kim, and Bumsub Ham. Background-aware pooling and noise-aware loss for weakly- supervised semantic segmentation. In CVPR, pp. 6909 6918, 2021.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin Fl-Nouby, Russell Howes, Po- Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Syn- naeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labalut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. In TMLR, 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. Tn NewrIPS+2019, pp. 8024 8035, 2019.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, pp. 2536-25660 2016.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthien Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine learning in python. In JMLR, volume 12, pp. 2825 2825-2830, 2011.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Not: Convolutional networks for biomedical image segmentation. In MICCAI, volume 9351, pp. 234 241, 2015.
Lixiang Ru, Heliang Zheng, Yibing Zhan, and Bo Du. Token contrast for weakly-supervised semantic segmentation. In CVPR, pp. 3093-30 3302, 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Sathoesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. Trd. J. Comput. Vision, 115(13):211-22) 2015.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. In JCLR, 2023.
Hyun Seok Seong, WonJun Moon, SuBeen Lee, and Jac-Pil Heo. Leveraging hidden positives for unsupervised semantic segmentation. In CVPR, pp. 19540-19549, 2023.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE T. Pattern Anal. Mach. Intell., 22(8))8888998) 905, 2000.
Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing VIT features for semantic appearance transfer. In CVPR, pp. 10738 10747, 2022.
Jasper R. R. Uijlings, Koen E. A. van de Sande, Theo Gevers, and Arnold W. M. Smeulders. Selective search for object recognition. Inl. J. Compul. Vision, 104(2):154-171, 2013.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue Van Gool. Unsupervised se- mantic segmentation by contrasting object mask proposals. In ICCV, pp. 10052211065 2021.
Wouter Van Gansbeke, Simon Vandenhende, and Luc Van Gool. Discovering object masks with transformers for unsupervised semantic segmentation. In orXiv:2206. 06363 CV,, 2022.
Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In CVPR, pp. 3124-3134, 2023a.
Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L. Crowley, and Do- minique Vaufreydaz. TokenCut: Segmenting objects in images and vidcos with self-supervised transformer and normalized cut. In IEEE T. Pattern Anol. Mach. Intell., volume 45, pp. 15790-15801 2023b.
Lian Xu, Wanli Ouyang, Mohammnd Benuamoun, Farid Boussaïd, and Dan Xu. Multi-class token trans- former for weakly supervised semantic segmentation. In CVPR, pp. 4300 1309, 2022.
Zhaoyuan Yin, Pichao Wang, Fan Wang, Xianzhe Xu, Hanling Zhang, Hao Li, and Rong Jin. TransFGU: A top-down approach to fine-grained unsupervised semantic segmentation. ECCV, volume 13689, pp. 73 89, 2022.
Andrii Zadaianchuk, Matthäus Kleindessner, Yi Zhu, Francesco Locatello, and Thomas Brox. Unsupervised semantic segmentation with self-supervised object-centric representations. In ICLR, 2023.
16

---

Published in Transactions on Machine Learning Research (09/2024)
A Further Analysis
In this appendix, we provide more detailed insights into PriMaPs-EM beyond the scope of the main paper.
A.1 Nearest- Netesbbor Anchoring
As described in Sec. of the main paper; PriMaPs are anchoring the first principal component in each iteration of the mask proposal generation. Corresponding to the quantitative findings (cf. Tab. Aap, here we additionally analyze this qualitatively. Fig. 5 shows similarity maps of all features to the iteratively computed first principal component (Similarity 1st PC) as well as to the respective nearest-neighbor feature (Similarity 1st PC NN) for example images from Cityscapes (Cordts et al. 2016), COCO-Stuff (Cacsar et al. 2018), and Potsdam--3 (ISPRS). We observe that the originally computed principal direction can have high similarities with multiple semantic concepts in an image. Hence, finding a suitable threshold that isolates a single main concept is difficult.
However, using the nearest image feature to the principal component as an anchoring element helps to circumvent high similarity values to multiple visual concepts. For instance, in the first example for Cityscapes, the similarity map for the first principal component has high similarities to both "building" and "vegetation". In contrast, the similarity map for the nearest neighbor of the first principal component results in high similarity to the class "vegetation" " only. Consequently, a suitable mask proposal can be obtained through thresholding. We observe this particularly for the Cityscapes dataset and some examples of COCO-Stuff. Further, in cases where the similarity map is already localized to one visual concept, anchoring merely leads to change in the similarity values without changing the shape of the thresholded proposal.
A.2 Ablation of the Threshold
In the spirit of unsupervised learning, our method effectively has only a single additional hyperparameter the threshold v. Furthermore, this parameter can be set simply by examining the mask proposal as detailed next. In addition, it should be noted that we keep this parameter unchanged for all backbone models and datasets, which further emphasizes the generalizability of our method. As described in Sec. 3 of the main paper, the threshold & used to remove noise in the similarity masks and to localize the optimization signal. In Fig. 6 qualitative examples of PriMaPs mask proposals are visualized for DINO ViT-B/8 on Cityscapes, COCO-Stuff, and Potsdam-3. We show mask proposals for se = 0.3, 1/ = 0.4, and y 0.5. Visually, it can be observed that meaningful masks are produced for all thresholds. Especially those for threshold 0.3 and 0.4 align very well with the semantic content in the images. While & 0.3 seems to provide better mask proposals for COCO-Stuff and Potsdam-3, a problem arises with Cityscapes. Here, the mask proposals contain several semantic concepts and spatially small objects in a large mask (cf. the second Cityscapes example, where the mask of the bush in the left half of the image covers both the traffic sign in the foreground, parts of the house in the background, poor optimization signal and the masks of the other two datasets using th 0.4 seem visually appcaling, the threshold is set to 0.4 in all other experiments.
well as the sky). Since this would lead to a
To further shed light on these qualitative observations, we apply PriMaPs-EM for the scenario described above and vary the threshold from & = 0.2 to th = 0.6 with a step size of 0.1. We perform this with the DINO VIT-B/8 backbone for Cityscapes, COCO-Stuff, and Potsdam-3 and show the mloU in Fig. 7 The quantitative results reflect our qualitative conclusion of the threshold well, and show the trade-off between better segmentation accuracy on COCO-Stuff and Potsdam-3 for lower threshold and vice versa for Cityscapes. In numbers, threshold of s = 0.5 instead of & = 0.4 for Cityscapes would lead to an additional gain of 1.4 % mloU, but also to slight losses on COCO-Stuff of 0.8 %. Additionally, this results in more mask proposal itcrations. We conclude that the determined threshold 1 = 0.4 appears to be reasonable. Even if better results could be achieved for some backbone-datasel combinations with individually set thresholds, we consider a fixed threshold that generalizes well across all scenarios to be sensible. We would like to emphasize that this experiment was conducted solely for this single backbone and serves only to validate the qualitative judgement from above. Importantly, we did not determine the hyperparameters based on the evaluation sets.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Image
Similarity 1st PC
Similarity 1st PC NN
Ground truth
Cityscapes
COCO Stuff
Potsdam-3
Figure 5: Nearest neighbor anchoring the principal direction in PriMaPs. Image, ground-truth label, and the first. three similarity maps with respect the principal direction (left) and their nearest neighbor (right) for all three datasets using DINO VIT-B/8. Anchoring localizes the signal for principal directions with high similarities to multiple visual concepts.
A.3 Ablation of the Stopping Criterion
Our method iteratively decomposes images into mask proposals. Hence, a stopping criterion is essential to decide whether a sufficient number of masks has been generated for a particular image. This allows us divide different images into different nunber masks. We use the percentage of features assigned masks. Specifically, the iterative PriMaPs process stops when 95% of image features are assigned to masks, While setting this value as high possible to cover the entire image with masks might seem intuitive, not all features can be assigned reasonably due to the inherent noise in the self-supervised feature representation. Setting the value too high would result in masks with very few features in the final iterations. We chose a stopping criterion of 95%, which we confirmed quantitatively to yield good results across all datasets as shown in Fig. g The relatively low mIoU for the Potsdam-3 dataset with smaller stopping criteria results from a few initial mask proposals covering large areas, while the proposals the final iterations represent. finer details.
A.4 Ablation the Image Augmentations
In addition the Tab. 4b we analyze the photometric augmentations used PriMaPs-EM greater detail. As can be seen, the additional augmentation leads to slightly improved performance terms of metrics. We used combination of standard augmentations well as the standard forchvision implementation and did
18

---

Published in Transactions on Machine Learning Research (09/2024)
Cityscapes
COCO-Stuff
Potsdam 3
Imnnge
=05.5
PrMMPs
0.
Figure 6: Qualitative threshold ablation with DINO ViT-B/8 using different values for PriMaPs
mask-ppopoose generation all three datasets. The hyperparameter exhibits favorable stability properties.
Cityscapes COCO-Stuff
60
60
Cityscapes
mloU (%)
COCO-Stuff
Potsdam-33
40
Potsdam-3
mloU
40
20
20
0.2
0.3
0.4 Threshold vb
0.5
0.6
85
99
Stop Criterion (%)
Figure 7: Quantitative threshold & ablation with DINO ViT-B/8 using different &o values for PriMaPs generation on all three datasets. The single hyperparameter of our method exhibits favor- able stability properties.
Figure 8: Quantitative stopping-criterion abla- tion with DINO ViT-B/8 using differont stopping criteria for PriMaPs mask proposal generation on all three datasets.
not tume any parameters. Tab. 0 provides a contribution breakdown of the used augmentation techniques (grayscaling, Gaussian blur, color jitter) COCO-Stuff using DINO ViT-B/8. Each of the three photometric augmentations contributes to the method's performance in terms of downstream metrics.
A.5 Ablation on the Number of Pseudo Classes K
Following previous works (Ji et al. 2019 Cho al.) 2021 Hamilton al. 2022 Seong al. 2023) in unsupervised semantic segmentation, generally the mumber of pseudo classes K is to the number of amnotated semantic classes in the dataset. This is done to evaluate the performance in terms of downstream metrics every semantic class after matching the predicted pscudo-class IDs with the ground-truth classes using Hungarian matching. Using a K smaller than the number of ground-truth classes is hard to evaluate and compare as it results in ground-truth classes that need to be ignored by the metric. However, is possible have more pscudo classes than ground-truth classes by assigning multiple pscudo classes a single ground-truth class. We conduct an experimented in this ver-seementttion setup, e. predicting K categories, where K is larger than the number of ground-truth classes. We realize the multi-to-one matching
19

---

Published in Transactions on Machine Learning Research (09/2024)
Table 6: Ablation study analyzing design choices and components of PriMaPs-EM for COCO-Stuff using
DINO ViT-B/8 including breakdown of the image augmentations.
Method
Acc
mloU
Baseline (Carnn et al.
2021
38.8
15.7
+ PriMaPs pseudo label
38.8
18.0
EMA
45.0
20.2
+ Augment (grayscaling)
45.3
20.4
+ Augment (grayscaling, Gaussian blur)
45.8
20.2
+ Augment (grayscaling, Gaussian blur, color jitter)
46.0
20.4
+ CRF (= PriMaPs-EMM
48.4
21.9
Table 7: Over clustering with Pr for COCO-Stuff using DINO ViT-B/8.
Method
Acc
mIoU
PriMas--EM (K=27; 100%)
48.5
21.9
PriMaPs-EM (K=40; 150%)
53.1
23.2
by applying Hungarian matching first and subsequently matching the remaining pseudo-class IDs based on their highest correspondence to the ground-trute classes, Tab. 7 we report the unsupervised semantic segmentation results for COCO-Stuf with DINO ViT-B/8 once using of ground-truth classes, and for K 40. Over clustering leads to better performance in terms of metrics, multiple pseudo classes represent single ground-truth class. Overall, estimating the number of pseudo classes semantic concepts in dataset in an unsupervised manner represents an intriguing direction for future research.
27, which matches the number
B Further Experiments
This appendix provides insights beyond the experiments and ablations shown in Sec.
B.1 Class-Level Quantitative Analysis
To gain deeper understanding of PriMaPs-EM, we assess the segmentation accuracy in terms of IoU for individual classes. Additionally, we present the confusion matrices among the semantic classes for the DINO ViT-B/8 Baseline, PriMaPs-EM, STEGO, and STEGO+PriMaP-ER for the COCO-Stuff dataset in Fig. g Generally, we can observe that for both the DINO and STEGO baseline, for certain classes (e. g-, "Appliance", "Indoor", "Kitchen") the discovered unsupervised class concept does not correlate with human- defined semantic classes. This suggests that the respective backbone feature representation may already be hard to separate. Furthermore, some of the 27 intermediate COCO-Stuff classes merge visually distinct concepts. For instance, the class "indoor" combines "hairbrush", "toothbrush", "hair dryer", "teddy bear". "scissors", "vase", "clock", and "book". We see this assumption partially confirmed by analyzing the class loUs of linear probing of the DINO features. For the problematic classes, the linear probing IoUs are in the range of approx. 16 %-30 %, whereas for the other classes the ToU is % and higher. We conclude that if it is already difficult lincarly distinguish the classcs basod on the backbone features, our method can hardly improve upon this.
However, for classes meeting this requirement, our method clearly boosts class LoUs. In rare cases (c. g., STEGO comparison to STEGO+PriMaP-EN, for classes "Outdoor" and "Sports"), there is decrease in one class loU with PriMaPs-EM while another class loU increases. This can occur if the change in the protolype representation results a change the Hungarian matching for evaluation, though this rarely observed. In terms of class confusion, the model's prodictions align with the ground-truth labels. Furthermore, the existing confusions are reasonable. For instance, when using STEGO, confusions of the two "food" mid- level classes emerge. Overall, Fig. 9a indicates that our method either enhances at least maintains the
20

---

Published in Transactions on Machine Learning Research (09/2024)
Figure 9: COCO-Stuff Comparison the segmentation performance for individual classes terms of IoU (in %) (a) and class confusion for the DINO ViT-B/8 Baseline (b), PriMaPs-EM (c), STEGO (d), and STEGO + PriMaPs-EM (e). Overall, PriMaPs-EM preserves or boosts the individual class IoU across most classes and moderately reduces class confusion.
(a) Class loUs (in %) for DINO VTT-B/8 Baseline, PriMaPs-EA EM (Ours), STEGO, and PTEGO+PriMaPs (Ours),
Rawana Rawantterial terial
Eleetronte
Applisnes
Parnitun
Kitchen
Funihus
Tixtlle
Window
Bullding
Ground
Food
Floor
Food
Sky
Baseline
0.0
82
8.0
15.3
3
23.7
2.2
6-3
0.2
30-7
77.4
18.0
27.1 45.1
Chrr
08.7
61.4
40.7
36.3
64.3
(b) Confusion matrix for the Baseline
(c) Confusion matrix for PriMaPs-BB
Eleetronic
Eleccronic
Apphance
Applaance
Food
Puriiture
Puraiture
Indoce
Indoor
Kitchen
LE
Kischem
Ausowwry
Ground-trut -anntt labels
labels
Oundcor
Outioor
Ponson
Pensnn
Sports
Sports
Vehicle
Ground-trui tr-truth
Vehirle
Creling
Floos
Pood
Food
Purniture
Furniture
Rawmuberiial
Ravmeterial
Pertile
Teottle
Winrdowe
Duilding
Duilding
Ground
Ground
Planet
Plane
Sky
Ske
Straeturdl
Strectural
Water
Witer
Elettooicc
Applaame
Funalue
Indoe
Kildion
Accesorry
Oudor
Prsmn
Spocts
Velicls
Flour
Food
Rasounlliin
Tescilee
Wall
Window
Grondd
Pant
Sky
Sirudtund
Water
Eletbonic
Applicnoe
Furzinuze
Indovr
Kirhon
Acossrry
Anmed
Poman
Spotts
Voliele
Geiling
Floor
Tocrdle
Wall
Beildngg
Grond
Pant
Sky
Solte
Predicted labels
Predicted labela
(d) Confusion matrix for STEGO
(e) Confusion matrix for STEGO PriMaPs-MM
Elastemni
Elestronic
Applianne
Appuinnce
Food
Food
Funitture
Furaiturs
Kitchen
Kitchen
Asssonscy
Assonnatyy
Bround-truth labels
Antina
Antinal
Outdicor
Outioor
Sports
Sports
Vehicie
Vehitle
Celinng
Ground-trut)
Orclingg
Plase
Flace
Food
Purniture
Rawemilarrdd
Rowmusterial Rostlle
Wall
Will
Windowr
Window
Huildline
Huildting
Plane
Plaze
Sky
Sky
Suiid
Soikd
Stractural
Stractural
Wasser
Walter
Eleetronic
Fanituue
Indooe
Kilchan
Ashtol
Onlion
Pesson
Sports
Veliide
callin
Food
Fumiluus
Testull
Wissoo
Duulingg
Groand
Plass
Stenttun
Water
Eleetronkc
Apyliscon
Durniture
Indoor
Kichen
Accnssrry
Antmal
Colling
Flooe
Rumiiture
Rowmattri
Tootale
Will
Windoo
Dullinng
Gromn
Plair
Slor
sald
Predicted labels
Predicted labels
21

---

Published in Transactions on Machine Learning Research (09/2024)
Cityscapes
COCO-Sta Stuff
Potsdam-3
Image
Ground truth
Easellie
PriMaPs- PMMM
HP
PriMaP= BBM
+
Figure 10:
Qualitative results for the DINO ViT-S/8 Bascline, PriMaPs-EM (Ours), HP (Scong et al. 2023), and HP (Scong et al. 2023)+ +PriMaPs-EMM (Ours) for all three datasets. Our method produces locally more consistent segmentation results, reducing misclassification
segmentation performance per class in terms of ToU regardless of the backbone model. Additionally, our method aids in reducing the confusion among classes.
B.2 Qualitative Comparison to HP
Similar to the qualitative comparison in Fig. in the main paper, we aim to compare PriMaPs-EM with HP (Seong et al. 2023) We present qualitative examples for the baseline, PriMaPs-EM, HP, and the combination of HP and PriMaPs-Hi EM in Fig, [0. These qualitative examples align with the findings from the quantitative results the main paper (ef. Tabs. to 3 It evident that our method produces locally more consistent and less noisy results compared both baselines. Despite the already impressive qualitative results of HP on COCO-Stuff, method excels in correcting misclassifications and achieving better segmentation of object boundaries. We observe one limitation, particularly with the DINO VIT-S/8 baseline, both independently and in conjunction with PriMaPs-EM for the Potsdam-3 dataset. In this case, the semantic concept of "street" " is not recognized and is consequently rarely predicted.
B.3 Comparison to HP using DINOv2
Current state-of-the-arl methods Hamilton et al. (0022 Seong et al. (2023) for unsupervised segmentation do not provide experiments using DINOv2 features. To be able to compare with previous methods, we train HP using DINOv2 for the comparison in Tab. g We strictly follow the training schodule and hyperparameters used the original implementation for all respective datasets. HP does not generalize well to the DINOv2 features and hardly keeps up with the strong baseline. PriMaPs-EM moderately but consistently improves
22

---

Published in Transactions on Machine Learning Research (09/2024)
Table 8: Comparing PriMaPs-EM (Ours) to HP using DINOv2 for the Cityscapes WiT-B/114) COCO-Stuff (ViT-B/14), and Potsdam-3 (ViT-S/14) datasets. We report Accuracy and mean IoU (in %) for unsupervised probing.
Cityscapes
COCO-Stuff
Potsdam-8
Method
Acc
mloU
Ace
mloU
Ace
mIoU
DINOv2 Baseline Oguab al. 2024
49.5
15.3
44.5
22.9
82.4
69.9
+ HP
Seoug el al.
2023
67.9
15.9
48.9
19.8
79.4
65.7
PriMal'
EM
71.6
19.0
46.4
23.8
83.1
71.0
HP
Scong et al.
2023 + PriMaPs-EM
74.3
16.6
49.3
20.2
79.6
66.0
Cityscapes
COCO-Stuff
Potsdam-3
PMorran
Imaae
Ground Trush
DINO
DINO PriMaPs-EM
PMcrrmm
DINOv2 Baseline
DINOv2 + PriMaPs EM
Figure Qualitative comparison of DINO and DINOv2 using VIT-B/8 and ViT-B/14 respectively. We show the DINO Baseline, DINO+PHMMaPs-EM (Ours), DINOv2 Baseline and DINOv2-PriMaPP-EE. (Ours) for Cityscapes, COCO-Stuff, and Potsdam-3.
upon both DINOv2 and HP across all datasets and metrics using the identical PriMaPs-EM hyperparameters we use across all other experiments in this work.
B.4 Qualitative Comparison of DINO and DINOv2
Our experiments could not identify any clear differences between DINOv1 and DINOY2 and the different transformer architectures terms of the downstream task performance of unsupervised semantic segmenta- tion. Both methods provide excellent feature embeddings and similar quantitative results as basclines. When applying PriMaPs-FM, DINOv2 often provides slightly better quantitative results but coarser segmentation
23

---

We visualize both the DINO and DINOv2 baselines and Published in Transactions on Machine Learning Research (09/2024)
Cityscapes
COCO-Stul
Potsdam-3
Image
PriMaPs Colonee
PriMaPs Orscie IDs
Ground truth
Figure 12:
Qualitative PriMaPs examples using DINO ViT-B/8 for Cityscapes, COCO-Stuff, and
Potsdam-3. PriMaPs Colored each mask proposal is visualized in a different color. PriMaPs Oracle class IDs cach mask is colored in the corresponding ground-truth class color.
masks due to the larger patches (ef. Tabs. with PriMaPs-EM in Fig. I1
T to
3).
B.5 Qualitative PriMaPs Pseudo- Examples
Following the quantitative assessment of the pseudo labels in Sec. 4.3 Fig. 12 visualizes qualitative examples of the PriMaPs mask proposals and pscudo labels. We visualize individual masks, cach in different color (PriMaPs Colored). We also display oracle pseudo labels, assigning each mask color based the ground- truth label (PriMaPs Oracle class IDs). We observe that the mask proposals align well with the ground-trute -ttuh labels across all three datasets, generalizing across three distinct domains. PriMaPs effectively partitions images into semantically meaningful masks.
B.6 Failure Cases
Finally, would like to discuss observed failure cases of PriMaPs-EM. Fig. 13 shows examples of failure cases occurring in the segmentation predictions as well as failure examples for PriMaPs pseudo labels. For PriMaPs-EM, we observe misclassifications, such as in Cityscapes where buses are often partially segmented as the class "car" or that cobblestone is misclassified as "sidewalk", For COCO-Stuff, shadows and structures influence the sogmentation predictions and confusions also occur (sce "ground" and "floor" in example two). For Potsdam-3, we observe that vehicles and road markings are sometimes erroneously attributed to the class "building" instead of "road". For PriMaPs pseudo labels, identify two main sources of error. First, small objects in cluttered images sometimes assigned to larger neighboring masks, phenomenon that be attributed to the limited backbone feature resolution. This is particularly noticcable for Cityscapes, where the center horizontal area is often detailed (ef. Fig. 12 "Pole", "Traffic Light", "Traffic Sign"). Second, PriMaPs sometimes oversegment images containing a large foreground object as seen in the COCO-Stufi example. Similarly, different visual appearances of the same semantic class can lead to multiple masks (Potsdam-3). Despite these observations, the simple PriMaPs provide promising mask proposals that correspond well with the ground-truth label.
24

---

Published in Transactions on Machine Learning Research (09/2024)
Cityscapes
COCO-Stuff
Potsdam 3
Cityscapes COCO-SI Stufff Potsdam 3
Iraage
Imaage
Ground truth
PriNat's Colored
Bazeline
PriMal's Oracle IDs
Ground truth
PrMaPs-ENN
Figure 13: Failure cases for the PriMaPs-EM segmentation (left) as well as PriMaPs pseudo labels (right)
using DINO ViT-B/8 for Cityscapes, COCO-Stuff, and Potsdam-3.
C Implementation
Since all significant high-level implementation details and hyperparameters have been addressed in the main paper, this section addresses only few remaining details. Both the code and models are publicly available
at https://gitthb. www.coomiiiiieeeeeeeeooeeeceeeie caam/visinf/// hooff//rrmpppss Our work is implemented in PyTorch (Paszke et al. 2019). We
build up on the code of Ji et. al. (2019), Van Gansbeke el al. (2021) and Hamilton el. al. (2022
C.1 Backbone Models
For cach backbone model, we use the corresponding original implementation. Specifically, for DINO (Caron et al. 2021) and DINOv2 (Oquab et al., 2024), we utilize the PyTorch Hub implementation. In the case of STEGO (Hamilton et al. 2022) and HP (Seong et al. 2023), integrate the respective original implemen- tations into framework.
C.2 Computational Requirements
We perform all experiments on single NVIDIA A6000 GPU. The PriMaPs-EM optimization runtime varies based on dataset size and the ViT architecture used as the backbone, with ViT-B/8 exhibiting the longest runtime and highest memory consumption. Specifically, using DINO ViT-B/8, PriMaPs-EM requires about 2.5 hours for Cityscapes and Potsdam, and approximatcly 4 hours for COCO-Stuff, utilizing about 30GB of memory. For the lightest backbone, DINOv2 VIT-S/14, optimization times decrease to about 1.5 hours for Cityscapes and Potsdam-3, and around 2.5 hours for COCO-Stuff with memory usage of about 20GB, DINO VIT-S/8 and DINOv2 VIT-B/14 lie in between.
C.3 Datasets
We close with some further details regarding the datasets used.
Cityscapes Cordts et al. 2016) an ego-centric street-scene dataset containing 5000 high-resolution images with 2048 1024 pixels. split into 2975 train, 500 val, and 1525 images. Following previous work (Ji et al. 2019: Cho et al. 22021] Yin al.) 2022 Hamilton al. 2022 Scong al. 2023), evaluation is conducted on the 27 classes setup using the val split.
25

---

Published in Transactions on Machine Learning Research (09/2024)
COCO-Stuff Caesar et al. 2018) is datasct of everyday life scenes containing 80 things and 91 stuff
classes. Following previous work (Ji et al. 2019 Cho et. al. 2021: Hamilton et al. 2022 Yin et al. 2022: Li et al. 2023 Seong et al. 2023). we use a reduced variant by Ji et al. (2019) containing 49629 train and 2175 test images. Hereby, all images consist of at least 75 % stuff pixels. The dataset evaluated on the 27 classes upt
remote sensing dataset consisting of 8550 RGBIR satellite images with 200 200 pixels, which is split into 4545 train and 855 test images, well 3150 additional unlabeled images. In experiments, the 3-label variant of Potsdam is evaluated and the additional unlabeled images are not used.
Potsdam-3 (SPRRS)
26


Reviews:
[{"review_id": "5YitOfzZ2A", "reviewer": "Reviewer_v3pd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- The paper proposes a novel method for generating semantic image segments using only SSL representations alone.\n- Unlike other unsupervised semantic segmentation methods, PriMaps introduces lightweight post-processing of SSL image representations to create segmentation masks. Since the representations are directly used, without any adaptation/finetuning, the method is comparatively more practical.  \n\n### Weaknesses:\n- The method section is hard to follow. I would encourage authors to add some visualisations in method section. To elaborate more on my concerns, I have asked questions for further clarifications, kindly consider adding additional explanations where necessary.\n\n\n### Questions/Clarifications:\n- How expensive (time and compute-wise) is the linear optimisation (post-processing) of representations, anchor selection and EM optimization?\n\n- Since the method is completely unsupervised (performance upper bounded by linear probing classification performance), can some sort of fine-tuning or adaption be done to  improve the performance of masks generation, and align masks with images semantically?\n\n- How do models pre-training distributions influence the inferred semantics of segmentations on a new target dataset? As DINOv1 is pre-trained on imagenet and DINOv2 is pretrained on a large curated dataset, do these pre-training datasets induce different models in behavior?\n\n- Heuristics for selecting K (unknown classes) within a dataset? What happens with large K and small K? \n- Did you ablate image augmentations? What are the guidelines on selecting optimal augmentation setting?\n- What are the heuristics behind using the nearest neighbour feature (to first principal component) as spatial anchor?\n- Why/How does dominant feature (a spatial anchor) ensure consistency for localising multiple concepts within an image?\n- Regarding reliability of using dominant feature ($\\tilde{f}$) for spatial localisation of concepts within image, how often do the semantically same concept get split across multiple $\\tilde{f}$ i.e., the same concept is explained by possibly two or more $\\tilde{})?\n- For “Further mask proposal”, do you still consider the first principal component and select second, third and so forth similar vectors as spatial anchors? or second, third, forth… principal components are used? Please mention this explicitly.\n- As far as I understood, selection of Z is based on how much image region is left to be segmented. In the paper it says 95%, is this threshold based on some statistical distribution of segments within natural images?\n- Instead of SSL method, how would a supervised, pre-trained method behave? Did authors try that?\n- Could you please shed some light the selection of K and induced Z? and how intricate is there relationship?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces PriMaps (Principal Mask Proposal), a method to generate semantically meaningful image segmentation masks using only image representations, learned by self-supervised learning models like DINO, DINOv2. The method operates by performing PCA over learned representations and then iteratively decomposing the image into segments by computing the affinity between image features and the respective first principal component. The method further leverages conditional random fields (CRF) to refine the predicted masks around edges, and use expectation maximization alogorithm to create pseudo labels for predicted masks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- First, kindly address the questions mentioned above section. Add additional details where necessary.\n- nitpick: - Repeated references in the related work section. Line with “Nevertheless, most of this work… “ \n- Try to add visualizations for the method section.", "broader_impact_concerns": "The paper proposes a new method from generating segmentation masks of an image using only pre-trained self-supervised embeddings. There are no broader impact concerns related to the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iqAcgb8Bjo", "reviewer": "Reviewer_DBXQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the paper lies in its generalizability, as it can be applied to the output of other patch-based methods, whether supervised or not, and enhance semantic segmentation.\n\nAs far as I know, the method is novel. The PCA-based labeling method they propose is very interesting, and I believe it can be generalized to other tasks as well.\n\nThe results are impressive. While the improvement on top of supervised methods is marginal, it is consistent and far more impressive than the larger improvement over the SSL method.\n\nThe main weaknesses are around the clarity of the paper. While most of the paper is clear, the method section is quite confusing.\n\nIn section 3.1:\n\n1.\tI would try to focus on the difference between standard PCA and the method you are using. As it is written now, it is unclear where PCA ends and your addition begins.\n2.\tThe sub-subsection titled “Proposal post-processing” seems more fitting for section 3.2, as the CRF method is orthogonal and commonly used in different methods to improve segmentations.\n\nIn section 3.2:\nGeneral question: You moved to batches. Are these batches of images or batches of patches inside a single image? Is the method used on an entire training set (with the same semantic classes) at once, or only valid for a single image at a time?\n\n1.\tInitialization: Are you using standard PCA or your PriMaPs method? If it’s the former, please elaborate on why. If it’s the latter, please make it clearer.\n2.\tMA: This entire part is unclear: “We use the backbone features and momentum class prototypes θT to yield a segmentation prediction y from which pseudo labels are generated as described in Sec. 3.1.” This is not described in 3.1. In 3.1, you describe how to use feature maps, but I am not sure how the prototypes are used. Focal Loss: How exactly do you use it? I understand it is optimized according to it, but where exactly? The entire optimization process of the prototypes (apart from updating using the MA) and the way you use them is unclear.\n\nAnother question: When assigning classes, do you still use the 95% rule from the first part, or do you continue until you have k classes? Why do you have psi as a subscript in the decay factor? Is it related to the previous psi you use in 3.1?\n\nAside from the clarity issue, some ablations are missing for the first part. Specifically, how would using the standard K leading eigenvectors instead of the current process to get the single eigenvector at a time work? Additionally, ablating the threshold psi would be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors have proposed a method for unsupervised semantic segmentation that can work with the feature outputs from standard SSL-trained networks, such as DINO, or even previously supervised semantic segmentation networks.\n\nInitially, the authors proposed PriMaPs, which iteratively identifies the first principal component, finds its 1-NN patch, and sets that as a class. It assigns all cosine similarities above a certain threshold to the same class. This process is repeated, removing these patches until 95% of the image is masked.\n\nThe second contribution builds on the first by following a process similar to EM. They generate K initial class prototypes. Using these prototypes and the backbone features, they create predictions, which are then used as pseudo labels in a batch-wise focal loss to optimize the prototypes. The new prototypes are used in a moving average to update the prototypes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Address the clarity issues and provide the missing ablations.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WPB083uXCg", "reviewer": "Reviewer_BQDo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Important field of study: Unsupervised semantic segmentation is important in the study of scene understanding.\n2. Simple method: The proposed method is simple, enhancing its accessibility and implementation.\n3. Impact of proposed solution: The proposed solution can be potentially impactful as it is an efficient strategy to improve existing models.\n\nWeaknesses:\n1. Writing issues: Certain details are missing from the paper, make it not less straightforward to follow. For instance, does $y$ refer to the prediction in pixel space or feature space. Why dot product is applied to the encoded feature and prototype, isn't the prototype the eigenvectors of the encode features. The paper doesn't clearly state how their method can be used orthogonally with existing method, for current version, it seems like the method itself can be a standalone method for unsupervised segmentation.\n2. Class ambiguity: The approach may struggle to distinguish between classes that share similar SSL feature representations. This can lead to misclassifications, particularly in complex scenes with multiple similar objects.\n3. Generalisation ability: The method relies on fixed SSL feature representations, which may limit its ability to fully adapt to specific segmentation tasks. If the SSL features are not well-suited for a particular dataset (For example, DINO for CityScape dataset), the segmentation performance could be suboptimal.\n\nIn general, the paper writing requires further reviews for more details and clarification. While I think the idea might have certain limitations, the proposed solution is interesting and novel, therefore I would like to see authors' reponses towards the aforementioned concerns.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method called Principal Mask Proposals (PriMaPs) for unsupervised semantic segmentation. The method involves using an iterative approach to identify semantic masks using features from a pretrained self-supervised model. Based on the masks, the authors construct pseudo labels and employ a moving average stochastic expectation-maximization algorithm called PriMaPs-EM. This technique assigns a consistent semantic class to each mask proposal, ensuring global consistency across the image regions. The authors demonstrate that proposed approach is a general approach that can be applied orthogonally to existing unsupervised semantic segmentation pipelines, thereby boosting their performance. Experimental results show consistent improvements in segmentation accuracy on multiple datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness part.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers vote for acceptance. This work has a clear scope at the intersection of self-supervised learning and unsupervised segmentation and shows a clear and simple method for direct inference of masks. The experiments cover timely and relevant methods and datasets, which are also standards for the topic, and therefore can inform the community. This work demonstrates that a simple linear method based on PCA and EM can do well, and without needing to (re-)learn the image representation, and that is worth signaling.

The review process has yielded revisions to the paper that clarify its contents, provide supplemental results and visualizations, and further analyze and ablate the proposed method.

As the action editor, I thank the authors for engaging in the TLMR process and congratulate them on the acceptance of their paper.
Decision: Accept (poster)