Unsupervised Semantic Segmentation by Distilling Feature Correspondences
Authors: Mark Hamilton
Zhoutong Zhang
Bharath Hariharan
Noah Snavely
William T. Freeman
Venue: ICLR 2022 Poster (2022)
Keywords: Unsupervised Semantic Segmentation
Unsupervised Learning
Deep Features
Contrastive Learning
Visual Transformers
Cocostuff
Cityscapes
Semantic Segmentation

Abstract: Unsupervised semantic segmentation aims to discover and localize semantically meaningful categories within image corpora without any form of annotation. To solve this task, algorithms must produce features for every pixel that are both semantically meaningful and compact enough to form distinct clusters. Unlike previous works which achieve this with a single end-to-end framework, we propose to separate feature learning from cluster compactification. Empirically, we show that current unsupervised feature learning frameworks already generate dense features whose correlations are semantically consistent. This observation motivates us to design STEGO ($\textbf{S}$elf-supervised $\textbf{T}$ransformer with $\textbf{E}$nergy-based $\textbf{G}$raph $\textbf{O}$ptimization), a novel framework that distills unsupervised features into high-quality discrete semantic labels. At the core of STEGO is a novel contrastive loss function that encourages features to form compact clusters while preserving their association pattern. STEGO yields a significant improvement over the prior state of the art, on both the CocoStuff ($\textbf{+14 mIoU}$) and Cityscapes ($\textbf{+9 mIoU}$) semantic segmentation challenges.  

Published as a conference paper at ICLR 2022
UNSUPERVISED SEMANTIC SEGMENTATION BY DISTILLING FEATURE CORRESPONDENCES
Mark Hamilton MIT, Microsoft ma rwt.hhitttttt... edu
Zhoutong Zhang MIT
Bharath Hariharan Cornell University
Noah Snavely
William T. Freeman
Cornell University, Google
MIT, Google
ABSTRACT
Unsupervised semantic segmentation aims to discover and localize semantically meaningful categories within image corpora without any form of annotation. To solve this task, algorithms must produce features for every pixel that are both se- mantically meaningful and compact enough to form distinct clusters. Unlike pre- vious works which achieve this with a single end-to-end framework, we propose separate feature learning from cluster compactification. Empirically, we show that current unsupervised feature learning frameworks already generate dense fea- tures whose correlations are semantically consistent. This observation motivates to design STEGO -Self-supervised Transformer with Energy- -bssed Graph Optimization), a novel framework that distills unsupervised features into high- quality discrete semantic labels. At the core of STEGO is novel contrastive loss function that encourages features to form compact clusters while preserving their relationships across the corpora. STEGO yields a significant improvement over the prior state the art, on both the CocoStuff (+14 mIoU) and Cityscapes (+9 mIoU) semantic segmentation challenges.
INTRODUCTION
Semantic segmentation the process of classifying each individual pixel of an image into a known ontology. Though semantic segmentation models can detect and delineate objects at much finer granularity than classification or object detection systems, these systems are hindered by the dif- ficulties of creating labelled training data. In particular, segmenting image can take more effort for human annotator than classifying or drawing bounding boxes (Zlateski et al, 2018). Furthermore, in complex domains such as medicine, biology, or astrophysics, ground-truth segmen- tation labels may be unknown, ill-defined, or require considerable domain-expertise to provide YYu al. 2018)
100x
Recently, several works introduced semantic segmentation systems that could learn from weaker forms labels such as classes, tags, bounding boxes, scribbles, or point annotations (Ren et al., 2020 Pan et al. 2021 Liu et al., 2020 Bilen et al.). However, comparatively few works take up the challenge
semantic segmentation without any form of human supervision or motion cues. Attempts such as Independent Information Clustering (IIC) (Ji et al. (019)) and PICIE (Cho et al.) 2021] aim to learn semantically meaningful features through transformation equivariance, while imposing a clustering step to improve the compactness of the learned features.
In contrast to these previous methods, we utilize pre-trained features from unsupervised feature learning frameworks and focus distilling them into compact and discrete structure while pre- serving their relationships across the image corpora. This is motivated by the observation that cor- relations between unsupervised features, such as ones learned by DINO (Caron et al. 2021). already semantically consistent, both within the same image and across image collections.
As result, introduce STEGO (Self-supervised Transformer with Energy-based Graph Optimization), which is capable of jointly discovering and segmenting objects without human super- vision. STEGO distills pretrained unsupervised visual features into semantic clusters using novel

---

Published as a conference paper at ICLR 2022
Inane
Label
PICEE (Beseline) STEGO (Ours)
Ground Sky Building Plant Vehicle
Animal Wall
Floor
Furniture
Water Structural
Figure 1: Unsupervised semantic segmentation predictions on the CocoStuff (Caesar et al. 2018) 27 class segmentation challenge. Our method, STEGO, does not use labels to discover and segment consistent objects. Unlike the prior state of the art, PICIE (Cho al. 2021). STEGO's predictions are consistent, detailed, and do not omit key objects.
contrastive loss. STEGO dramatically improves prior and is considerable step towards closing the gap with supervised segmentation systems. We include a short video detailing the work https:: /aaka. ms/ wwsssttegoviide
Specifically, make the following contributions:
Show that unsupervised deep network features have correlation patterns that are largely consistent with true semantic labels.
Introduce STEGO, a novel transformer-based architecture for unsupervised semantic seg- mentation.
Demonstrate that STEGO achieves state the art performance on both the CocoStuff (+14 mloU) and Cityscapes (+9 mloU) segmentation challenges.
Justify STEGO's design with an ablation study on the CocoStuff dataset.
2 RELATED WORK
Self-supervised Visual Feature Learning Learning meaningful visual features without human annotations is longstanding goal of computer vision. Approaches to this problem often optimize surrogate task, such as denoising (Vincent al.) 2008), inpainting (Pathak et al. 20166), jigsaw puz- zles, colorization (Zhang et al., 2017). rotation prediction (Gidaris et al. 2018). and most recently. contrastive learning over multiple augmentations (Hjelm et al., 2018 Chen et al. 2020a a c Oord ct al. 2018). Contrastive learning approaches, whose performance surpass all other surrogate tasks, assume visual features are invariant under a certain set of image augmentation operations. These ap- proaches maximize feature similarities between image and its augmentations, while minimizing similarity between negative samples, which are usually randomly sampled images. Some notable examples of positive pairs include temporally adjacent images in videos (Oord al. 2018), image
augmentations (Chen et al. 2020a c), and local crops of single image (Hjelm et al. 2018). Many
works highlight the importance of large numbers of negative samples during training. To this end Wu et al.) (2018) propose keeping a memory bank of negative samples and Chen et al. (2020c) propose momentum updates that can efficiently simulate large negative batch sizes. Recently some works have aimed to produce spatially dense feature maps opposed to a single global vector per image. In this vein, VADeR (Pinheiro et al.) 2020) contrasts local per-pixel features based on random com- positions of image transformations that induce known correspondences among pixels which act as positive pairs for contrastive training. Instead of trying to learn visual features and clustering from scratch, STEGO treats pretrained sele-super features input and is agnostic to the underlying feature extractor. This makes it easy to integrate future advances in self-supervised feature learning into STEGO.
2

---

Published as a conference paper at ICLR 2022
Unsupervised Semantic Segmentation Many unsupervised semantic segmentation approaches use techniques from self-supervised feature learning. IIC (Ji et al. 2019) maximizes mutual in- formation of patch-level cluster assignments between an image and its augmentations. Contrastive Clustering (Li et al.) 2020), and SCAN (Van Gansbeke et al. 2020) improve on IIC's image cluster- ing results with supervision from negative samples and nearest neighbors but do not attempt semantic segmentation. PiCIE (Cho et al.) 202I) improves on IIC's semantic segmentation results by using invariance photometric effects and equivariance geometric transformations as an inductive bias. In PICIE, a network minimizes the distance between features under different transformations, where the distance is defined by an in-the-loop k-means clustering process. SegSort (Hwang et al.) 2019) adopts a different approach. First, SegSort learns good features using superpixels as proxy seg- mentation maps, then uses Seppetation-Maxiiizaa embedding space. In a similar vein, Mask (Van Gansbeke et al. (02T) achieves promising results on PascalVOC by first using an off-the-shelf saliency model to generate a binary mask for each image. MaskContrast then contrasts learned features within and across the saliency masks. In contrast, our method focuses refining existing pretrained self-supervised visual features to distill their correspondence information and encourage cluster formation. This is similar to the work of Collins et al. (2018) who show that low rank factorization of deep network features can be useful for unsupervised co-segmentation. We are not aware of any previous work that achieves the goal of high- Suaa--uualtyy, pixel-level unsupervised semantic segmentation on large scale datasets with diverse images.
to iteratively refine segments over a spherical
Visual Transformers Convolutional neural networks (CNNs) have long been state of the art for many computer vision tasks, but the nature of the convolution operator makes hard to model long- range interactions. To circumvent such shortcomings, Wang et al. (0188; Zhaag et al. (2019) use self-attention operations within a CNN to model long range interactions. Transformers /Vaswani et al. a 2017). or purely self-attentive networks, have made sigmificant progress in NLP and have re- cently been used for many computer vision tasks (Dosovitskiy et al. 2020, Touvron et al. 2021
Ranfil et al., 202T Caron et al. 202T). Visual Transformers (VIT) (Vaswani et al.) 2017) apply self-attention mechanisms to image patches and positional embeddings in order to generate features and predictions. Several modifications of ViT have been proposed to improve supervised learning, unsupervised learning, multi-scale processing, and dense predictions. In particular, DINO (Caron et al. (2021) uses ViT within a self-supervised learning framework that performs self-distillation with exponential moving average updates. Caron et al. (2021) show that DINO's s class-attention can produce localized and semantically meaningful salient object segmentations. Our work shows that DINO's features not only detect salient objects but can be used to extract dense and semantically meaningful correspondences between images. In STEGO, we refine the features of this pre- trained backbone to yield semantic segmentation predictions when clustered. We focus on DINO's 's embed- dings because of their quality but note that STEGO can work with any deep network features.
METHODS
3.1 FEATURE CORRESPONDENCES PREDICT CLASS CO-OCCURRENCE
Recent progress in self-supervised visual feature learning has yielded methods with powerful and semantically relevant features that improve variety of downstream tasks. Though most works aim to generate a single vector for an image, many works show that intermediate dense features are semantically relevant (Hamilton al.) 2021 Collins et al.) 2018 Zhou al. (016).. To use this information, we focus on the "correlation volume" " (Teed & Deng 2020) between the dense feature maps. For convolutional or transformer architectures, these dense feature maps can be the activation map of a specific layer. Additionally, the Q, K or V matrices in transformers can also serve as candidate features, though we find these attention tensors do not perform as well in practice. More formally, let f E ICf .9 E RRCJ be the feature tensors for two different images where C represents the channel dimension and W), (I, represent spatial dimensions. We form the feature correspondence tensor: Fhowij := Σ Schw Jaj |frwol [gis!
whose entries represent the cosine similarity between the feature at spatial position (h, w) of feature tensor and position (i.j) of feature tensor g. In the special case where = g these correspon-
3

---

Published as a conference paper at ICLR 2022
Image and Query Points
Self Correspandence
KNN Correspondence
38-
Precision
0.4
88% STEGO [Durs]
10-
DING
AP-024 63% MoCov2
AF 60% CRF
0.
Figure 2: Feature correspondences from DINO. Correspon- dences between the source image (left) and the target images (middle and right) are plotted over the target images in the respective color of the source point (crosses in the left im- age). Feature correspondences can highlight key aspects of shared semantics within a single image (middle) and across similar images such as KNNs (right)
Recall
Figure 3: Precision recall curves show that feature self-correspondences strongly predict true label co- occurrence.
DINO outperforms MoCoV2 and a CRF kernel, which shows its power as unsupervised learning signal.
dences measure the similarity between two regions of the same image. We note that this quantity appears often as the "cost-volume" within the optical flow literature, and et al. (202) show this acts higher-order generalization of Class Activation Maps (Zhou al. (206)) for contrastive architectures and visual search engines. By examining slices of the correspondence tensor, F. at given (h, w) we are able visualize how images relate according the featurizer. For example, Figure shows how three different points from the source image (shown in blue, red, and green) are in correspondence with relevant semantic areas within the image and its K-nearest neighbors with respect to the DINO (Caron et al.) 2021) as the feature extractor.
This feature correspondence tensor not only allows us visualize image correspondences but strongly correlated with the true label co-occurrence tensor. In particular, we can form the ground truth label co-occurrence tensor given a pair of ground-truth semantic segmentation labels CHW, e C.J where C represents the set of possible classes:
Lhwrij 1, if law kij
0, if lhw kaj
By examining how well the feature correspondences, F. predict the ground-truth label co- occurrences, L, we can measure how compatible the features are with the semantic segmentation labels. More specifically we treat the feature correspondences as probability logit and compute the average precision when used as a classifier for L. This approach not only acts as a quick diag- nostic tool to determine the efficacy of features, but also allows us to compare with other forms of Koltun,
supervision such the fully connected Conditional Random Field (CRF) (Krähenbühl 2011), which uses correspondences between pixels to refine low-resolution label predictions. In Figure we plot precision-recall curves for the DINO backbone, MoCoV2 backbone, CRF Kernel, and our trained STEGO architecture. Interestingly, lar predictor of label co-occurrence within the Coco stuff dataset despite never seeing the labels. In particular, DINO recalls 50% of true label co-occurrences with a precision of 90% and significantly outperforms both MoCoV2 feature correspondences and the CRF kernel. One curious note that our final trained model is a better label predictor than the supervisory signal it learns from. We tribute this to the distillation process discussed in Section which amplifies this supervisory signal and drives consistency across the entire dataset. Finally, truth labels within this section is solely to provide intuition about the quality of feature correspon- dences as a supervisory signal. We do not use the ground truth labels tune any parameters of STEGO.
find that DINO is already spectacu-
stress that our comparison to ground
3.2 DISTILLING FEATURE CORRESPONDENCES
In Section 3.1 we have shown that feature correspondences have the potential to be a quality learning signal for unsupervised segmentation. In this section explore how to hamnes this signal to create pixel-wise embeddings that, when clustered, yield quality semantic segmentation. In particular, we seek to leamn dow-dimensional embedding that "distills" the feature correspondences. To achieve
4

---

Published as a conference paper at ICLR 2022
this aim, we draw inspiration from the CRF which uses undirected graphical model refine noisy or low-resolution class predictions by aligning them with edges and color-correlated regions in the original image.
More formally, let N IR''HWW + RCHW represent deep network backbone, which maps an image z with C' channels and spatial dimensions (H', W to a feature tensor f with C' channels and spatial dimensions (H, W). In this work, we keep this backbone network frozen and focus on training a light- weight segmentation head S RCHW + IR KWW.W that maps our feature space to code space of dimension K, where K C. The goal of S is to learn nonlinear projection, S(f) =: E I2 kKW that forms compact clusters and amplifies the correlation patterns of f.
To build our loss function let f and g be two feature tensors from a pair of images x, and y and let : :(= RCHW and t = Sg)) E IRJ be their respective segmentation features. Next, using Equation we compute feature correlation tensor F E R from f and g and segmentation correlation tensor S E RTWWJJ from s and t. Our loss function aims to push the entries of s and t together if there is a significant coupling between two corresponding entries of and g. As shown in Figure we can achieve this with simple element-wise multiplication of the tensors F and S:
Leimple- corr (z, y. == Σ (Fhwrrs - b) S)Shwis
(2)
hwnij
Where b is hyper-paaaeee parameter which adds uniform "negative pressure" to the equation to prevent collapse. Minimizing with respect to S encourages elements of S be large when elements of F b are positive and small when elements of F b are negative. More explicitly, because the elements of F and are cosine similarities, this exerts an attractive or repulsive force on pairs of segmentation features with strength proportional to their feature correspondences. We note that the elements of S are not just encouraged to equal the elements of F but rather to push to total anti-alignment (-1) or alignment (1) depending the sign of F b.
In practice, we found that Csimple-con corr sometimes unstable and does not provide enough learning signal to drive the optimization. Empirically, we found that optimizing the segmentation features towards total anti-alignment when the corresponding features do not correlate leads to instability, likely because this increases co-linearity. Therefore, we optimize weakly-correlated segmentation features to be orthogonal instead. This can be efficiently achieved by clamping the segmentation correspondence, S, at O, which dramatically improved the optimization stability.
Additionally, we encountered challenges when balancing the learning signal for small objects which have concentrated correlation patterns. In these cases, Pharkj b is negative in most locations, and the loss drives the features to diverge instead of aggregate. To make the optimization more balanced, we introduce a Spatial Centering operation on the feature correspondences:
F
SC huij
= FFwrrk
Σ 3'3'
Frwey s.
(3)
IJ
Together with the zero clamping, our final correlation loss is defined as:
Leorr Coorr(1, y, == Σ
(Press -
0).
(4)
hwij
We demonstrate the positive effect of both the aforementioned "O-Clamp" and "SC" modifications in the ablation study of Table
3.3 STEGO ARCHITECTURE
STEGO uses three instantiations of the correspondence loss of Equation 4 to train segmentation head to distill feature relationships between an image and itself, its K-Nearest Neighbors (KNNs), and random other images. The self and KNN correspondence losses primarily provide positive, attractive, signal and random image pairs tend to provide negative, repulsive, signal. We illustrate this and other major architecture components of STEGO in Figure 4
STEGO is made up of a frozen backbone that serves as a source of learning feedback. and as an input to the segmentation head for predicting distilled features. This segmentation head is simple
5

---

Published as a conference paper at ICLR 2022
Image
Image 2
Training
Tensor
Product
Frozen Visual Backbone
Frozen Visual
Backbone
Random Image Corr.
Corr.
Self
Segmentation
Segmentation Head
Head
Prediction
Feature
Segmentation Correspondences
Correspondences
F
S
Vis. Irans
Sag Head
Cluster
CRF
L = - - SS
Correspondence Distillation Loss
Figure 4: High-level overview of the STEGO architecture at train and prediction steps. Grey boxes represent three different instantiations the main correspondence distillation loss which is used to train the segmentation head.
feed forward network with ReLU activations (Glorot et al. 2011). In contrast to other works, our method does not re-train fine-tune the backbone. This makes method very efficient train: it only takes less than 2 hours on a single NVIDIA V100 GPU card.
We first use our backbone extract global image features by global average pooling (GAP) our spatial features: GAP(f). We then construct a lookup table of each image's K-Nearest Neighbors according to cosine similarity the backbone's s feature space. Each training minibatch consists of collection of random images r and random nearest neighbors kknn In our experiments sample knn randomly from each image's top KNNs. We also sample random images, x zond by shuffling and ensuring that no image matched with itself. STEGO's full loss is:
L xeeff Ceorr (z, z, byelf) + Aknn (2, (z, kmn bknn) + Arand (I, x rand brand)
(5)
Where the X's and the b's control the balance of the learning signals and the ratio of positive to negative pressure respectively. In practice, we found that a ratio of Xeetf Xrand 2Agun worked well. The parameters tended to be dataset and network specific, but we aimed to keep the system rough balance between positive and negative forces. More specifically we tuned the bs to keep mean KNN feature similarity at <z 0.3 and mean random similarity at z 0.0.
Many images within the CocoStuff and Cityscapes datasets are cluttered with small objects that are hard to resolve at feature resolution of (40, 40). To better handle small objects and maintain fast training times we five-crop training images prior to learning KNNs. This not only allows the network to look closer details of the images, but also improves the quality of the KNNs. More specifically, global image embeddings are computed for each crop. This allows the network to resolve finer details and yields five times many images find close matching KNNs from. Five-cropping improved both our Cityscapes results and CocoStuff segmentations, and we detail this in Table
The final components of our architecture the clustering and CRF refinement step. Due to the feature distillation process, STEGO's s segmentation features tend to form clear clusters. We apply a cosine distance based minibatch K-Means algorithm (MacQueen al. 1967) to extract these clus- ters and compute concrete class assignments from STEGO's continuous features. After clustering, we refine these labels with a CRF to improve their spatial resolution further.
3.4 RELATION TO POTTS MODELS AND ENERGY-BASED GRAPH OPTIMIZATION
Equation 4 can be viewed in the context of Potts models continuous Ising models from statistical physics (Potts 1952 Baker Jr & Kincaid 1979). We briefly overview this connection, and point detailed discussion. To build the general Ising model, let = (V, w) be fully connected, weighted, and undirected graph on [V] vertices. In our applications take V to be the of pixels in the training dataset. Let
interested readers to A.ction|A.8Joo
y IR represent edge

---

Published as a conference paper at ICLR 2022
Table 1: Comparison of unsupervised segmentation architectures on 27 class CocoStuff validation
set. STEGO significantly outperforms prior art in both unsupervised clustering and finear-probe style metries.
Unsupervised
Linear Probe
Model
Accuracy
mIoU
Accuracy
mloU
ResNet50 (He et al.
2016)
24.6
8.9
41.3
10.2
MoCoV2 Chen el
2020c
25.2
10.4
44.4
13.2
DINO Caron ct al
2021
30.5
9.6
66.8
29.4
Deep Cluster (Caron
2018
19.9
SIFT (owe
1999
20.2
Doersch al. (2015)
23.1
Isola et al. 2015)
24.3
AC
(Oall et al. 2020)
30.8
InMARS (Mirsadeghı al.
2021
31.0
IIC
[J1 ct al.
2019)
21.8
6.7
44.5
8.4
MDC Cho al.
2021
32.2
9.8
48.6
13.3
PÍCIE
(Cho et al.
2021
48.1
13.8
54.2
13.9
PICIE H (Cho al. 2021
50.0
14.4
54.8
14.8
(TEGO(Ours
56.9
28.2
76.1
41.0
weighting function. Let Φ -> be vertex valued function mapping into a generic code space such the probability simplex over cluster labels P(L). the K-dimensional continuous feature space K The function & can be parameterized neural network, or a simple lookup table that assigns code to each graph node. Finally, define compatibility function μ C x C - R that measures the cost of comparing two codes. We can now define the following graph energy functional:
E(d)= = Σ
w(t, ())n(d(s)
(6)
EEREEV
Constructing the Boltzmann Distribution (Hinton 2002) yields a normalized distribution over the function space क;
exp(-F((g)
p(olw, ()) Sn exp(-E(d'))dd
In general, sampling from this probability distribution is difficult because of the often-intractie intractable normalization factor. However, is easier to compute the maximum likelihood estimate (MLE),
arg max +c p(p/w, μt). In particular, if o is smoothly parameterized space of functions and é and yu are differentiable functions, one can compute the MIEE using stochastic gradient descent (SGD) with highly highly-optimized automatic differentiation frameworks (Paszke al.) 2019 Abadi et al. 2015).
In Section A.3 of the supplement we prove that the finding the MLE Equation i equivalent to minimizing the loss of Equation 4 when [V] is the set of pixels in image training set, N. w is the cosine distance between features, and is cosine distance. Like STEGO, the CRF is also a Potts model, and use this connection to re-purpose the STEGO loss function create continuous, minibatch, and unsupervised variants of the CRF We detail this exploration in Section A.9 of the Supplement.
EXPERIMENTS
We evaluate STEGO standard semantic segmentation datasets and compare with current state-of- the-art. We then justify different design choices of STEGO through ablation studies. Additional de- tails datasets, model hyperparameters, hardware, and other implementation details can be found in Section A. of the Supplement.
4.1 EVALUATION DETAILS
Datasets Following Cho et al. 2021 we evaluate STEGO the 27 mid-level classes of the CocoStuff class hierarchy and on the 27 classes of Cityscapes. Like prior art, we first resize images to 320 pixels along the minor axis followed by (320 x 320) center crops of each validation image. We use mean intersection over union (mloU) and Accuracy for evaluation metrics. Our CocoStuff evaluation setting originated in li et al. (2019) and is common in the literature. Our Cityscapes

---

Published as a conference paper at ICLR 2022
Predicted labels
Memmonk
Hester
Road
Image
= Sidewalk
app rce
Parking
Eumiuure
B Car
kiccher
IN Vegetation
pertonn
Label
Building
True label
Wall
Pole
Person
nattro
STEGO (Ouns)
Bicycle
K.30
Rider
bullcing
Traffiic Sign
Traffiic Light
pk
ignored
Figure 6: Confusion matrix of STEGO
Figure 5: Comparison of ground truth labels (middle row)
cluster probe predictions on CocoStuff. Classes after the "vehicle" class are "stuff" and classes before are "things". Rows are normalized sum to
and cluster probe predictions for STEGO (bottom row) for images from the Cityscapes dataset.
evaluation setting is adopted from Cho et al. (2021). The latter is newer and more challenging, and thus fewer baselines are available. Finally we also compare on the Potsdam-3 setting fro et al. (0199) in Section A.2 of the Appendix.
Linear Probe The first way evaluate the quality of the distilled segmentation features is through transfer learning effectiveness. As in Van Gansbeke et al. (2021); Cho et al. (2021); Chen al. (2020b). we train a linear projection from segmentation features to class labels using the cross entropy loss. This loss solely evaluates feature quality and is not part of the STEGO training process.
Clustering Unlike the linear probe, the clustering step does not have access to ground truth super- vised labels. As in prior art, we use Hungarian matching algorithm to align unlabeled clusters and the ground truth labels for evaluation and visualization purposes. This measures how consistent the predicted semantic segments are with the ground truth labels and is invariant to permutations of the predicted class labels.
4.2 RESULTS
We summarize our main results on the 27 classes of CocoStuff in Table I STEGO significantly outperforms the prior state of the art, PiCIE, both linear probe and clustering (Unsupervised) metries. particular, STEGO improves by +14 unsupervised mloU, +6.9 unsupervised accuracy, +26 linear probe mIoU, and +21 linear probe accuracy compared to the next best baseline. In Table we find a similarly large improvement of +8.7 unsupervised mloU and +7.7 unsupervised accuracy on the Cityscapes validation set. These experiments demonstrate that fine-tune the backbone for these datasets, DINO's self-supervised weights on ImageNet (Deng et al.) 2009) are enough to simultaneously solve both settings. STEGO also outperforms simply clustering the features from unmodified DINO, MoCoV2, and ImageNet supervised ResNet50 backbones. This demonstrates the benefits of training a segmentation head distill feature correspondences.
though we do not
We show some example segmentations from STEGO and our baseline PICIE on the CocoStuff dataset Figure I We include additional examples and failure cases in Sections A..4 and A.5 We note that STEGO is significantly better at resolving fine-grained details within the images such as the legs of horses in the third image from the left column of Figure [ and the individual birds the right-most column. Though the PICIE baseline uses a feature pyramid network to output high reso- lution predictions, the network does not attune to fine grained details, potentially demonstrating the limitations of the sparse training signal induced by data augmentations alone. In contrast, STEGO's predictions capture small objects and fine details. In part, this can be attributed to DINO backbone's 's higher resolution features, the 5-crop training described in 3.3) and the CRF post-processing which helps to align the predictions to image edges. We show qualitative results on the Cityscapes dataset in Figure 5 STEGO successfully identifies people, street, sidewalk, cars, and street signs with high
8

---

Published as a conference paper at ICLR 2022
Table 2: Architecture ablation study on the CocoStuff
Dataset (27 Classes).
Table 3: Results on the Cityscapes
0- A-Clamp
Dataset (27 Classes). STEGO improves
5-Crop
significantly over all baselines in both ac- curacy and mloU.
CRF
SC
Unsup.
Linear Probe
Arch.
Acc.
m
Acc.
mloU
MoCoV2
48.4
20.8
70.7
26.5
Unsup.
ViT-S
34.2
7.3
54.9
15.6
Model
Acc.
mloU
VIT-S
44.3
21.3
70.9
36.8
IIC (Ji et al. 2019)
47.9
6.4
VIT-S
47.6
23.4
72.2
36.8
MDC
Chho et al.
2021
40.7
7.1
VIT-S
47.7
24.0
72.9
38.4
PICIE
Cho et al.
2021
65.5
12.3
VIT-S
48.3
24.5
74.4
38.3
(OurO
73.2
21.0
VIT-B
54.8
26.8
74.3
39.5
VIT-B
56.9
28.2
76.1
41.0
detail and fidelity. We note that prior works did not publish pretrained models or linear probe results on Cityscapes so we exclude this information from Table 3 and Figure
To better understand the predictions and failures of STEGO, include confusion matrices for CocoStuff (Figure ( and Cityscapes (Figure [T of the Supplement). Some salient STEGO errors include confusing the "food" category from the CocoStuff "things". and the "food" category from CocoStuff "stuff". STEGO also does not properly separate "ceilings" from "walls", and lacks con- sistent segmentations for classes such "indoor". "accessory". "rawmaterial" and "textile". These errors also draw our attention to the challenges of evaluating unsupervised segmentation methods: label ontologies can be arbitrary. In these circumstances the divisions between classes are not well defined and it is hard to imagine system that can segment the results consistently without additional information. In these regimes, the linear probe provides more important barometer for quality be- cause the limited supervision can help disambiguate these cases. Nevertheless, we feel that there is still considerable progress to made on the purely unsupervised benchmark, and that even with the improvements of STEGO there is still measurable performance gap with supervised systems.
4.3 ABLATION STUDY
To understand the impact of STEGO's architectural components we perform an ablation analysis on the CocoStuff dataset, and report the results in Table 2 We examine the effect of using several different backbones in STEGO including MoCoV2, the ViT-Small, and ViT-Base architectures of DINO. We find that ViT-Base is the best feature extractor of the group and leads by significant margin both in terms of accuracy and mloU. We also evaluate the several loss function and architec- ture decisions described in Section 3.3 In particular, we explore clamping the segmentation feature correspondence tensor at to prevent the negative pressure from introducing co-linearity (O-Clamp), five-cropping the dataset prior mining KNNs to improve the resolution of the learning signal (5- Crop), spatially centering the feature correspondence tensor to improve resolution of small objects (SC), and Conditional Random Field post-procsssing to refine predictions (CRF). We find that these modifications improve both the cluster and linear probe evaluation metrics.
5 CONCLUSION
We have found that modern self-supervised visual backbones can be refined to yield state of the art unsupervised semantic segmentation methods. We have motivated this architecture by show- ing that correspondences between deep features are directly correlated with ground truth label co- occurrence. We take advantage of this strong, yet entirely unsupervised, learning signal by intro- ducing a novel contrastive loss that "distills" the correspondences between features. Our system, STEGO, produces low rank representations that cluster into accurate semantic segmentation pre- dictions. We connect STEGO's loss to CRF inference by showing it is equivalent to MLE in Potts models over the entire collection improvement over the prior state of the art, both the CocoStuff (+14 mloU) and Cityscapes (+9 mloU) semantic segmentation challenges. Finally, we justify the architectural decisions of STEGO with ablation study the CocoStuff dataset.
pixels in our dataset. We show STEGO yields a significant

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We would like to thank Karen Hamilton for proofreading the work and Siddhartha Sen for spon- soring access to the Microsoft Research compute infrastructure. We also thank Jang Hyun Cho for helping rum and evaluate the PICIE baseline. We thank Kavital Bala, Vincent Sitzmann, Mare Bosch, Desalegn Delelegn, Cody Champion, and Markus Weimer for their helpful commentary on the work.
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. 2021323067. Any opinion, findings, and conclusions or recommenda- tions expressed in this material are those of the authors(s) and do not necessarily reflect the views of the National Science Foundation. This research based upon work supported in part by the Office of the Director of National Intelligence (Intelligence Advanced Research Projects Activity) via 2021- 201 1000006. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U S Government. The US Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. This work is supported by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org)
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving. Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vin- cent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Watten- berg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. BtLLhttp:/ttensoe tensorflow.org.
org/ Software available from
Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. CoRR, abs/1904.05044, 2019. URL httt:://axxv...rgg// 1904. 05044
George A Baker Jr and John M Kincaid. Continuous-spin ising model and A: 4: d field theory. Physical Review Letters, (2(22))1431, 1979.
Hakan Bilen, Rodrigo Benenson, and Seong Joon Oh. Eeev 2020 tutorial weakly-supervised learning computer vision. URL httpp::/ com/ http//bblee/wwwet eccv20.
github.ic.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1209- 1218, 2018
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint OrXiv:2104. 14294, 2021.
Liang-Chich Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Se- mantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint 14Iiv:1412.7000 2014.
Liang-Chie Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint 1oXiv:/706.0550 2017.
10

---

Published as a conference paper at ICLR 2022
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi cemi-supervised learners. arXiv preprint urXi:2006.11002, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.044: 044997 2020b.
Xinlci Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. arXiv preprint arXiv:2003. 0429977 2020c.
Jang Hyun Cho, U. Mall, K. Bala, and Bharath Hariharan. Picie: Unsupervised semantic segmenta- tion using invariance and equivariance in clustering. ArXiv, abs/2103.17077 2021.
Edo Collins, Radhakrishna Achanta, and Sabine Susstrunk. Deep feature factorization for concept discovery. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 336-352, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. leee, 2009.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp- 1422-1430, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprini arXiv:2010. 1929, 2020.
William Falcon et al. hwwp:////...........iiii/n///nee
Pytorch lightning.
GitHub.
Note:
3, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint
2018.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp- 315-323. JMLR Workshop and Conference Proceedings, 2011.
Mark Hamilton, Scott Lundberg, Lei Zhang, Stephanie Fu, and William Freeman. Model-agnostic explainability for visual search. arXiv preprint (tti::2/030007000 2021.
Yufei Han and Maurizio Filippone. Mini-batch spectral clustering. In 2017 International Joint Conference on Neural Networks (IJCNN), pp. 3888-3895, IEEE, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800 2002.
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint /00iv:880..06000 2018.
Karan Grewal, Phil Bachman, Adam
Jyh-Jing Hwang, Stella X. Yu, Jianbo Shi, Maxwell D. Collins, Tien-Ju Yang, Xiao Zhang, and Liang-Chieh Chen. Segsort: Segmentation by discriminative sorting of segments. CoRR,
abs/1910.06962, 2019. http:://////a org/ abs / 1910. 06962
Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H Adelson. Learning visual groups from co-occurrences in space and time. arXiv preprint arXiv: 1,
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. arXiv preprint arXiv: 150..0000000222 2015.

---

Published as a conference paper at ICLR 2022
Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9865-9874, 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected erfs with gaussian edge potentials. Advances in neural information processing systems, 24:109-117, 2011.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Proba- bilistic models for segmenting and labeling sequence data. 2001.
Omer Levy and Yoav Goldberg, Neural word embedding as implicit matrix factorization. Advances in neural information processing systems, 27:2177-2185, 2014.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng. Joey Tianyi Zhou, and Xi Peng. Contrastive cluster- ing. arXiv preprint arXiv:2009.09633 2020.
Yun Liu, Yu-Huan Wu, Peisong Wen, Yujun Shi, Yu Qiu, and Ming-Ming Cheng. Leveraging instance- image- and dataset-level information for weakly supervised instance segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020. doi: 10.1 10.1909T 2020.3023152.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference computer vision, volume 2, pp. 1150-1157. Iece, 1999.
James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of fifth Berkeley symposiun on mathematical statistics and probability, volume 1, pp. 281-21121 281-297. Oakland, CA, USA, 1967.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representa- tions of words and phrases and their compositionality. arXiv preprint 100 2013.
S Ehsan Mirsadeghi, Ali Royat, and Hamid Rezatofighi. Unsupervised image segmentation by mutual information maximization and adversarial regularization. IEEE Robotics and Automation Letters, 6(4):6931-693) 2021.
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. arXiv preprint arXiv:1707.05005. 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.037/ 2018
Yassine Ouali, Céline Hudelot, and Myriam Tami. Autoregressive unsupervised image segmenta- tion. In European Conference on Computer Vision, pp. 142-158. Springer, 2020.
Shun-Ya Yi Pan, Cheng- You Lu, Shih-Po Lee, and Wen-Hsiao Peng. Weakly-supervised image seman- tic segmentation using graph convolutional networks. In 202/ IEEE International Conference
Multimedia and Expo (ICME), pp. 1-6. IEBE, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bi Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
12

---

Published as a conference paper at ICLR 2022
Deepak Pathak, Philipp Krähenbühh, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context en- coders: Feature learning by inpainting. In CVPR, 2016.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier
Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Seikit-learn Machine learning in python. the Journal of machine Learning research, 12:2825-2833 2011.
Pedro O Pinheiro, Amjad Almahairi, Ryan Y Benmalek, Florian Golemo, and Aaron Courville. Unsupervised learning of dense visual representations. arXiv preprint (tfi::00110059999 2020.
Renfrey Burnard Potts. Some generalized order-disorder transformations. Mathematical pro- ceedings of the cambridge philosophical society, volume 48, pp. 106-109. Cambridge University Press, 1952.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. ArXiv preprint, 2021.
Zhongzheng Ren, Zhiding Yu, Xiaodong Yang, Ming- Yu Liu, Alexander G Schwing, and Jan Kautz. Ufo2: unified framework towards omni-supervised object detection. In European Conference on Computer Vision, pp. 288-313. Springer, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way prevent neural networks from overfitting. The journal of machine learning research,
2014.
Zachary Teed and Jia Deng. RAFT: recurrent all-pairs field transforms for optical flow. CoRR, abs/2003. 12039, 2020. URL
Marvin TT Teichmann and Roberto Cipolla. Convolutional erfs for semantic segmentation. arXiv preprint vrXiv:1805.04777 2018.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis. Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. European Conference on Com- puter Vision, pp.
Springer, 2020.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Un- supervised semantic segmentation by contrasting object mask proposals. arxiv preprint arxiv:2102.06191, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
Xiaolong Wang, Ross Ginshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 7794-7803, 2018.
Zhirong WU, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3733-3742, 2018
Donghui Yan, Ling Huang, and Michael Jordan. Fast approximate spectral clustering, In Pro- ceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 907-916, 2009.
13

---

Published as a conference paper at ICLR 2022
Hongshan Yu, Zhengeng Yang, Lei Tan, Yaonan Wang, Wei Sun, Mingui Sun, and Yandong Tang. Methods and datasets on semantic segmentation: review. Neurocomputing. 304:82-103, 2018.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning, pp. 7354-7363. PMLR, 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. CVPR, 2017.
Xuewen Zhang, Selene E Chew, Zhenlin Xu, and Nathan Cahill. Slic superpixels for efficient graph-based dimensionality reduction of hyperspectral imagery. In Algorithms and Technolo- gies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXI, volume 9472, pp. 947209. International Society for Optics and Photonics, 2015.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Aleksandar Zlateski, Ronnachai Jaroensri, Prafull Sharma, and Frédo Durand. On the importance of label quality for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1479-1487, 2018.
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 VIDEO AND CODE
We include short video description of our work allttss///ka..tt/tttto-vvt
We also provide training and evaluation code at
A.2 ADDITIONAL RESULTS ON THE POTSDAM-3 DATASET
In addition to our evaluations in Section 4.1 we compare STEGO to prior art on the Potsdam 3-class aerial image segmentation task presented in Ji al. (2019). In Table ?? We find that STEGO is able to achieve +12 accuracy compared to the previous state of the art, IIC. We show example qualitative results in Figure 7
Table 4: Additional results the Potsdam-3 aerial image segmentation challenge
Model
Unsup. Acc.
Random CNN (Ji et al. 2019)
38.2
K-Means (Pedregosa et al.
20T1
45.7
SIFT Lowe 1999)
38.2
Doersch al. (2015)
49.6
Isola et al. 2015)
63,9
Deep Cluster Caron et al
2018
41.7
IIC (Ji ct al. 2019
65.1
STEGO
77.0
Image
STEGO
Ground Truth
Roads
Vegetation
Buildings
Figure 7: Qualitative comparison of STEGO segmentation results on the Potsdam-3 segmentation challenge.
15

---

Published as a conference paper at ICLR 2022
A.3 ADDITIONAL ABLATION STUDY
In addition to the ablation study of Table 2 we investigate the effect of each major architectural decision in isolation. We find that in most metrics, removing each architectural component hurts performance.
Table 5: Additional architecture ablation study on the CocoStuff Dataset (27 Classes).
KNN-Loss
Rand-Loss
0-Clamp
Pointwise
Self-Loss
5-Crop
CRF
Unsupervised mIoU
Linear Probe
Backbone
Acc.
Acc.
mIoU
ViT-Small
48.3
24.5
74.4
38.3
MoCoV2
43.1
19.6
65.9
26.0
ViT-Small
42.8
10.3
59.3
19.3
ViT-Small
48.0
23.1
73.9
38.9
ViT-Small
50.2
22.3
73.7
37.7
ViT-Small
47.7
24.0
72.9
38.4
ViT-Small
43.0
20.2
73.0
36.2
ViT-Smal
47.0
22.2
74.0
37.7
ViT-Smal
39.8
12.8
65.5
29.9
16

---

Published as a conference paper at ICLR 2022
A.4 ADDITIONAL QUALITATIVE RESULTS
Ground
L2b
Sky
Building
Plant
Vehicle
Person
Animal
70
Wall
Floor
Furniture
Water
Indoor
Solid
Ceiling
Structural
Libal
Window
Sports
Appliance
Outdoor
Food (Stuff)
Food (Thing)
Ignored (white)
Figure 8: Additional unsupervised semantic segmentation predictions the CocoStuff 27 class segmentation challenge using STEGO (Ours) and the prior state of the art, PICIE, Images are not curated.
17

---

Published as a conference paper at ICLR 2022
A.5 FAILURE CASES
Unsupervised Segmentation is prone to a variety of issues. We include some of the following to segmentations demonstrate cases where STEGO breaks down. In the first column of Figure 2 we can see that STEGO improperly segments ground from trees and backgrounds. see that STEGO makes an understandable eπror and assigns the barn floor the "outdoor" class and the barn wall to the "building" class. In the third column STEGO misses the boundary between wall and ceiling. The fourth column demonstrates the challenge between food (thing) and food (stuff) characterization. Interestingly PICIE makes the same type of error both here, and in the barn case. The last column shows an example STEGO missing a human in the lower left. In this image it is challenging to spot the person, probably because it is grayscale.
the second column
Ground
mang
Building
Plant
1BX
Arimal
Label
Floor
Furniture
STEGO (Ours)
Indoor
Coling
Structunal
Window
Sports
PICIE (Beseline)
Outdoor
Food [Scuff]
Food (Tring)
Ignored
(white)
Figure STEGO failure cases.
18

---

Published as a conference paper at ICLR 2022
A.6 FEATURE CORRESPONDENCES PREDICT STEGO's ERRORS
Section demonstrates how unsupervised feature correspondences serve as an excellent proxy for the true label co-occurrence information. In this section we explore how and where DINO's feature correspondences systematically differ from the ground truth labels, and show that these insights allow us to directly predict STEGO's final confusion matrix.
More specifically we consider the setting of Section 3.1 Instead of computing precision-recall curves from our feature correspondence scores we can instead threshold these scores, select the strongest couplings between the images, and evaluate whether these couplings are between objects of the same class or objects of different classes. In particular, Figure I0 shows confusion matrix capturing how well DINO feature correspondences between images and their K-Nearest Neighbors align with the ground truth label ontology in the CocoStuff27 dataset. We find that that this analysis that
predicts many of the areas where the final STEGO architecture fails. In particular, DINO conflates the "Food (things)" and "Food (stuff)" and this error also appears in STEGO's confusion matrix in Figure 12 Likewise both visualizations show confusion between "appliance" and "furniture", "window" and "wall", and several other common errors.
This analysis demonstrates that many of STEGO's 's errors originate from the structure of the DINO features used to train STEGO as opposed to other aspects the architecture. However we note that the question of whether whether this is issue with the DINO features, due ambiguities in the CocoStuff label ontology is still outstanding. Finally we note that this analysis is able to predict the results of fully-trained STEGO architecture, and could be used as a way select better backbones without having to training STEGO.
KNN Labels
rawmateria
electronic
appliance
fumiture
indoor
kitchen
accessory
animal
outdoor
person
sports
vehicle
celling
floor
food
fumiture
textile
wall
window
building
ground
plant
sky
solid
structural
wwaker
electronic
appliance
Food
furniture
indoor
kitchen
accessory
animal
outdoor
Image labels
person
sports
vehicle
ceiling
floor
food
furniture
rawmaterial
textile
wall
window
building
ground
plant
sky
solid
structural
water
Figure 10: Normalized matrix of predicted label co-occurrences between an Images and KNNs. This analysis shows where unsupervised supervisory signal, the DINO feature correspondences, fails to align with the CocoStuff27 label ontology.
19

---

Published as a conference paper at ICLR 2022
A.7 HIGHER RESOLUTION CONFUSION MATRICES
Predicted labels
roa
sidewalk
park king
track
building
wall
fence
-uuard rail
bridge
tunnel
pole
polegroup
traffic light
traffic sign
vegetation
terrain
sky
person
car
truck
bus
caravan
traller
train
motorcycle
blcycle
road
sidewalk
parking
building
wall
fence
guard rail
bridge
tunnel
True labels
pole
polegroup
trafflc light
traffic sign
vegetation
terrain
sky
person
rider
truck
bus
trailer
train
motorcycle
bicycle
Figure 11: Confusion Matrix for Cityscapes predictions
Predicted labels
electrenic
furiiture
indoor
kitchen
accessory
animal
outdooor
person
sports
vehicle
furniture
rawmaterial
boxtie
window
building
ground
structural
food
floor
food
wall
plant
solid
electronic
appliance
food
furniture
indoor
kitchen
accessory
outdooor
True labels
sports
vehicle
celling
floor
food
furniture
rawmaterial
textile
wall
window
building
ground
plant
sky
salid
structural
water
Figure 12: Confusion Matrix for CocoStuff predictions
20

---

Published as a conference paper at ICLR 2022
A.8 RELATIONSHIP WITH GRAPH ENERGY MINIMIZATION
In section 3.4 we briefly mention that STEGO's feature correlation distillation loss defined in Equa- tion 4 can be seen as a particular case of Maximum Likelihood (ML) estimation on a undirected graphical model or Ising model. In this section we demonstrate this connection in greater detail using the formalism defined 3.4 In particular, we recall the energy for Potts model:
E(o) == Σ
(llo,,)))(l(ss)..o owy))
(8)
We then construct the Boltzmann Distribution (Hinton 2002 the function space Φ:
yields a normalized distribution over
exp( exp(-E(o))
plol ()
J exp(- F.(d')
In general, sampling from this probability distribution is difficult because of the often-intractable normalization factor. However, it is easier to compute the maximum likelihood estimate (MLE);
1 7 exp((-E())
arg max plbl (4) arg max SED
(10)
de
Where Z is the unknown constant normalization factor. Simplifying the right-hand side yields:
arg max p(o (4) arg min F(Ø) arg min
Σ EVReEV
w(e, u; ())
(11)
dE
We are now in the position connect this the STEGO loss function. First, we take our nodes V to be the set of all spatial locations across our entire dataset of images. For concreteness we can represent e by the tuple (n, h, w) where h, w represent height and width represents the image number. We now let ((x;) be the output of the segmentation head, 80;, at the image and spatial as the compatibility function, μe, yields
location 44- Using cosine distance, deos (2. y) the following:
argmin
Σ B.PyE
-w U3))
S0z
Sey
(12)
[sv.
(syy
Wherte the argmin now ranges over the parameters of the segmentation head S. We can now observe that the sum over all pairs Ue, Bj E V can be written sum over pairs of images z, X and pairs of spatial locations (h, w). j) where we note that (i.j) i) in this context refers to the spatial coordinates of image y as in 3.1 and not the indices of the vertices.
arg min
Σ Σ
(w yewws S(z, y)
(13)
x,yEX kwij
Where we define S(x, y) to be the segmentation feature correlation tensor images x, y as defined
in Section 3.2 Finally letting W (xx, y)mwin = Fhanij b we recover our loss:
arg max p(olw, (1) = arg min Seb
Σ Z
Lsimple- corr (z. v,
(14)
Finally note that in practice approximate the minimization using minibatch SGD, and our inclusion KNN and Self-coorespondence distillation changes the weight function w, but does not change its functional form.
21

---

Image Published as a conference paper at ICLR 2022
Switching to the ML formulation of this problem allows to solve this optimization for ó by gradient descent on the parameters of the segmentation head, S, and makes this computationally tractable. For large image datasets that can contain millions of high-resolution images, the induced graph can contain billions of image locations. Other graph embedding and clustering approaches such as Spectral methods require solving for eigenvalues of the graph Laplacian, which can take ((1V))) time Yaan et al., 2009). More recent attempts to accelerate Spectral clustering such as Yan al., 2009) and (Han & Filippone, 2017) further assume "Nonparametric" structure on the function Ø. where a separate cluster assignment is learned for each vertex. This assumption of "nonparametric" function & can be undesirable as one cannot cluster embed new data without recomputing the entire clustering. In contrast, STEGO's backbone and segmentation head act as parametric form for the function Φ allowing the approach to output predictions for novel images.
A.9 CONTINUOUS, UNSUPERVISED, AND MINI-BATCH CRF
= P (20)) C
R
Figure 13: Unsupervised CRF solutions for discrete (middle) and continuous (right) code spaces. In the discrete case we mark the boundaries between classes, in the continuous case we visualize the top 3 dimensions of the code space.
Fully connected Gaussian Conditional Random Fields (CRFs) (Lafferty et al. 2001) are an ex- tremely popular addition to semantic segmentation architectures. The CRF has the ability to improve initial predictions of locations, and can "sharpen" predictions to make them consistent with edges and areas with consistent color in the original image. CRF post-processing for refining supervised and weakly supervised semantic segmentation predictions is ubiquitous in the literature (Lafferty et al. 2001 Chen al. 2014: Long al. 2015 Liu et al.) 2020 Ahn et al. 2019). Recently, connections between CRF message passing and convolutional networks have allowed CRFs to be embedded into existing models (Chen al. 2017 Teichmann & Cipolla 2018) and trained jointly for better performance. By connecting the STEGO correspondence distillation loss to the energy of undirected model image pixels we can use the same minibatch MLE strategy estimate other similar graphical models. For example, in the fully connected Gaussian edge potential CRF, one forms a pairwise potential function potential function for the pixels of single image:
[p: P2
- 1,1
(-1p: hP:--pl" p3l1 202
|p:
Wery(Dr, vy) exp
+ bexp
(15)
202
202
Where p: represent the pixel coordinates associated with node v and I, represents pixel colors sociated with node v; The parameters a, b, 0., 0g, f, are hyperparameters and control the behavior of the model. These parameters balance the effect of long- and short-range color similarities against
22

---

Published as a conference paper at ICLR 2022
smoothness. The CRF directly learns pixel-wise array of probabilistic class assignments over labels corresponding to the probability simplex code space P()) and non-parametric clus- tering function f. For compatibility function j the CRF chooses the Potts Model (Potts) 1952); (potts((((s)) ((())) = (P((())) * o(U)))
With this setting of the weights and compatibility function, directly recover the binary potentials of the fully connected Gaussian edge potential CRF (Krähenbühl & Koltun 2011). We can also add the unary potentials which are often the outputs of another model. However, for our analysis we explore the case without unary potentials which yields an "unsupervised" variant of the CRF. However, without external unary potential terms, the strictly positive similarity kernel encourages the maximum likelihood estimator (MLE) of the graph to be the constant function. To rectify this, we can add small negative constant, -b, to the weight tensor to push unrelated pixels apart. This negative force is the direct analogue of the negative pressure hyper-paramen parameter in STEGO and can be interpreted through the lens of negative sampling (Mikolov et al., (2013). This negative shift also appears in the word2vec and graph2vec embedding techniques (Narayanan al. 2017 Levy Goldberg 2014). Our shifted CRF potential encourages natural clusters to form that respect the structure of the potentials that capture similarities in pixel colors and locations. In the discrete case, solutions to this equation resemble superpixel algorithms such as SLIC (Zhang et al., 2015). Additionally lifting this the continuous code space and provide natural continuous generalization of superpixels and seems to avoid challenging local minima. We illustrate these solutions to just the unsupervised CRF potential in Figure [3 Finally, we note that the second term of Equation [5] referred to as the smoothness kernel, matches IIC's notion of local class consistency. However, found that adding these CRF terms to the self-correspondence loss of STEGO did not improve performance.
23

---

Published as a conference paper at ICLR 2022
A.10 IMPLEMENTATION DETAILS
Model STEGO uses the "ViT-Base" architecture of DINO pre-trained on ImageNet. This back- bone was trained using self-supervision without access to ground-truth labels. We the "teacher" weights when creating our backbone. We take the final layer of spatially varying features and ap- ply small amount (p 0.1) of channel-wise dropout (Srivastava et al. 2014) before using them throughout the architecture during training. Our segmentation head consists of a linear network and a two-layer ReLU MIP added together and outputs a 70 dimensional vector. We use the Adam optimizer (Kingma & Ba, 2014) with learning rate of 0.0005 and batch size of 32. To make our losses resolution independent we sample 121 random spatial locations in the source and target implementations and use grid sampling (Jaderberg al. 2015) to sample features from the back- bone and segmentation heads. Our cluster probe trained alongside the STEGO architecture using minibatch k-means loss where closeness is measured by cosine distance. Cluster and linear probes are trained with separate Adam optimizers using a learning rate of .005
Datasets We the training and validation sets of Cocostuff described first in al. (2019) and used throughout the literature including in Cho et al. (2021). We note that the validation set used in Ji al. (2019) is subset of the full CocoStuff validation set and we use this validation subset be consistent with prior benchmarks. We note that using the full validation set does not change results significantly. When five-cropping images we use a target size of (.5h, .5w) for each crop where h,, w are the original image height and width. Training images are then scaled to have minor axis equal to 224 and are then center cropped to (224, 224), validation images are first scaled to 320 then are center cropped to (320, 320). All image resizing uses bilinear interpolation and resizing of target tensors for evaluation uses nearest neighbor interpolation.
CRF
We PyDenseCRR (Krähenüühl & Koltun 2011 with 10 iterations with parameters
4, = 3, la = 67, 03 = 3, 0, as written in Section A.9
Compute All experiments use PyTorch (Paszke et al. 2019) v1.7 pre-trained models, on an Ubuntu 16.04 Azure NV24 Virtual Machine with Python 3.6. Experiments use PyTorch Lightning for distributed and multi-gpu training when necessary (Falcon et 2019).
Hyperparameters
We use the following hyperparameters for results in Tables and
Table 6: Hyperparamtters used in STEGO
Parameter Cityscapes
CocoStuff
Arand
0.91
0.15
Aknn
0.58
1.00
Aself
1.00
0.10
brand
0.31
1.00
bken
0.18
0,20
byelf
0.46
0.12
24

---

Published as a conference paper at ICLR 2022
A.11 A HEURISTIC FOR SETTING
Setting hyperparameters without cross-validation on ground truth data can be difficult and this is an outstanding challenges with the STEGO architecture that we hope can be solved in future work. Nevertheless we have identified some key intuition to guide manual hyperparameter tuning. More specifically, we find that the most important factor affecting performance is the balance of positive and negative forces. Too much negative feedback and vectors will all push apart and clusters will form well, too much positive feedback and the system will tend towards small number of clusters. To debug this balance, we found it useful to visualize the distribution of feature correspon- dence similarities as a function of training step as shown in Figure [4 A balanced system (Orange distribution) will tend towards bi-modal distribution with peaks alignment or orthogonality at O. This bi-modal structure is indicative that there is some clustering within images, but that not everything is assigned to the same cluster. Pink and blue distributions show too much positive and negative signal respectively. We find that given a reasonable balance of the A's, this balance can be achieved by tuning the bs to achieve the desired balance.
Unsupervised mloU
Balanced Positive and Negative Signal
Step 1000
9
0.0
Learned Feature Similaaity Distribution
Step
Nat Enough Positive Signal
Top Much Positive Signal
Step
0.6
2
0.2
0.0
1.0
-0.8
0.8
1.0
Learned Feature Similarity Distribution
Learned Feature Similarity Distribution
Figure 14: Distributions of feature correspondences between an image and itself across three dif- ferent hyper- perammeeetttrr settings. The orange curve and distribution shows proper balance between attractive and repulsive forces allowing some pairs features to cluster together (the peak at 1) and other pairs of features orthogonalize (the peak at 0)
25

---

Published as a conference paper at ICLR 2022
A.12 A NOTE ON 5-CROP NEAREST NEIGHBORS
We found that pre-processing the dataset by 5-cropping images was a simple and effective way to improve the spatial resolution of STEGO and the quality of K-Nearest Neighbors. We consider each resulting 5-crop as a separate image when computing KNNs and patches from the same image are valid KNNs. Figure 15 shows the distribution of these self self-matches for the CocoStuff dataset. We note that the majority of patches do not have any nearest neighbors from the same image.
70000
60000
Frequency
30000
20000
10000
Number of 5-crop Patches in 7 Nearest Neighbors
Figure 15: Number of patches from the same image found within each patch's nearest neighbors
26


Reviews:
[{"review_id": "WhfrKjHmedJ", "reviewer": "Reviewer_XoJq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The authors have properly addressed related work. The paper is well-structured and the method is sound. Design decisions are backed up by ablation studies. The method is feature extractor agnostic, which still makes STEGO relevant in the future as these technologies advance. \n\nWeaknesses: I haven't rigorously checked the mathematical formulation, but at the moment I do not find any reason to reject this paper. The pretrained backbone (DINO). used as a starting point (not fine-tuned) is quite strong and it would have been interesting to see what is the gain of using STEGO on weaker feature extractors. Also, I am a big advocate for highlighting the limitations of the method and also showing some failure cases to clearly state, from the authors' perspective, what other issues should the research community address in the future.  \n\nMinor comment: Found minor typos throughout the manuscript (difficult to pinpoint them due to the current paper format, lines without number) that do not necessarily impact the quality, but proofreading is highly recommended. (e.g. \"an an\", missing a/to, \"O-clamp\" should be \"0-clamp\")", "summary_of_the_paper": "The paper introduces STEGO (Self-supervised Transformer with Energy-based Graph Optimization), a novel feature correlation refinement method that builds on top of modern self-supervised visual backbones (visual-transformers) that generate dense semantically-correlated features in an effort to improve scene semantic segmentation without any type of labels (unsupervised). Different from previous works, the authors decouple the feature learning from cluster compactification and introduce a novel contrastive loss function (a combination of three correlation factors: KNN, self and random images) in order to further constrain the features to form compact semantic clusters without damaging their consistency throughout the dataset. The authors demonstrate state-of-the-art results on two popular benchmarks CocoStuff and Cityscapes. The results are quite striking in both a quantitative (method improves segmentation by a large margin (+14mIoU and +9mIoU) compared to recently published work) and also a qualitative manner (clearly distinguish the structure of the objects in the scene).", "summary_of_the_review": "The contribution is solid - the problem is relevant and quite difficult. There is still so much work to be done in order to close the gap between unsupervised and supervised methods for scene semantic segmentation, but nonetheless, STEGO makes a step in the right direction. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SPZVUQoJpBt", "reviewer": "Reviewer_2hfj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method builds on top of DINO (Caron et al., 2021). The features extracted by DINO are further refined with a small segmetation head that aims to boost performace by balancing Knn, self-correlation and random image correlation loss. The training batch consisnt of random images and random nearest neighbours (precomputed for the whole dataset). The segmentation head is a simple feed forward network - by avioding the backbone retraining (and implicitly knn recomputation), the method is very efficient to train. Hovewer, there are a nuber of problems related to the backbone - e.g., 40x40 feature resolution, which means small objects are difficult to resolve; the authors address this issue by five-cropping the dataset (corners+center crop), resulting in five times more images to find knns and better details (no rescaling to 224x224- the original transformer input size). The final steps involve clustering the resulting features and further CRF refinement for finer details in object edges. Additionally, the authors propose spatial centering on the feature correspondences, to improve segmentation of small features and clamping the segmetation correspondence at 0, to improve the optimization stability. \n\nOne does note that most performance improvement strategies are related to resolving small details that the original transformer network is incapable of doing. Therefore, I belive a future, better transformer that has fine detail capabilies would render those tricks obsolete. On the other hand, the performance is signinficantly boosted by the transformer. One would assume that future work would boost the proposed method, which has a significant performance boost over the baseline (DINO). Therefore, there are a number of plausible options of future research based on this work.\n\nAfter training STEGO on Coco (ViT-small, no CRF[the code has some issues]), I also also some practical aspects that have not been raised in the paper. Looking at the initial and final confusion matrix, the performance improves on several classes and worsens on others; I believe this is mostly due to the features extracted by DINO, but sometimes the errors are amplified - no investigation on this aspect.\n\nThere is an ablation study for clamping the segmentation feature correspondence tensor at 0, five-cropping the dataset (corners+center), spatial centering on the feature correspondences and CRF. The three term loss gets no ablation study (except for SC). At the very least, I would like to see numbers with knn only, self-correlation only and random correlation only. Can we improve the performance by designing better training pairs? I believe so, and this additional ablation study would have shed light on this issue. Otherwise, we have to manually tune the $\\lambda$s and the $b$ for each dataset (see page 20), and this doesn't look very appealing to me. Can the authors share some insights on this issue? I see the values are wildly different on Coco/Cityscapes. I'm willing to improve my rating, provided this issue is properly addressed. \n\nAlthough well written and illustrated, there are a small number of issues with the content of the paper:\n- 'typical results' images; 28 mIoU is pretty bad and most results from the paper look amazing; I have a folder full of truly random images, if anyone is interested :)\n- page 5, after Eq 4, shoud read 0-clamp, not O-clamp\n- page 15, In The >> In the", "summary_of_the_paper": "The paper proposes a novel transformer-based unsupervised feature distillation method for semantic segmetation- STEGO (Self-supervised Transformer with Energy-based Graph Optimization). The pipeline has two separate stages, feature learning and cluster compactification (feature distillation). The feature network is DINO (Caron et al., 2021) - pretrained, frozen. \n\nThe main contribution is the cluster compactification(distillation) network. It features a novel 3-term loss function: Knn, self correltation and random image corelation. The authors also propose a number of strategies for performance improvement: clamping the segmentation feature correspondence tensor at 0, five-cropping the dataset (corners+center), spatial centering on the feature correspondences and conditional random fields for better segmentation at the object edges. Training the segmentation head is fast (a couple of hours on a recent GPU with the ViT-small transformer backbone) and the results are state-of-the art on semantic segmentation on CocoStuff and Cityscapes (+14mIou@CocoStuff, +7mIoU@Citiscapes).\n", "summary_of_the_review": "This paper could have been outstanding, but as it stands, it's only acceptable. The best part is the unsupervised performance boost over prevoius methods (+14mIou@CocoStuff, +7mIoU@Citiscapes) and the ability to use the proposed model on top of most feature extractors. That being said, the authors have shown that the larger the transformer, the better the unsupervised segmentation, but have failed to provide a method that 'just works', regardless of the dataset. Instead, we rely on a number of tricks for performance boosting (mostly related to small objects) that could have been incorporated in the original transformer stage (fair enough, except for CRF, which is pretty much a popular solution for squeezing 2%-ish in accuracy) and some variables we need to manually set - the three-pronged loss that is at the core of the paper does not get a proper ablation study, each term is weighted empirically. Apart from the state-of-the-art performance, the training itself is kind of confusing - some classes do not benefit at all from this stage, sometimes the feature noise is happily propagated in the final segmentation, sometimes amplified - the limitations of the feature extractor and loss have not been thoroughly investigated, IMHO. This is not shown in the main paper, but I have tried the provided code on CocoStuff and those are my findings. Nevertheless, being feature extractor agnostic, future work could expand and improve the performance of this method. Furthermore, the authors suggest that label ontologies can be arbitrary - future work could benefit from a hierarchical ontology, for example. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KGwDmMIwpR", "reviewer": "Reviewer_GKTw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nOverall, writing of this paper is clear and straightforward. STEGO achieves state of the art performance on both the CocoStuff and Cityscapes segmentation challenges. \n\nWeakness:\n1)\tNovelty: \nThe main contribution is that it trained the additional segmentation head with the proposed contrastive loss over the frozen visual backbone. The novelty of utilizing the contrastive learning to form feature clusters is not so obvious. \n\n2)\tImplementation: \nFirst, the approach seems complex to implement. For example, there are too many hyperparameters (eg, b and lamba) in the contrastive loss definition and it seems that the tunning of the parameters is complex.\nSecond, as is pointed out by the authors, the optimization is sometimes unstable. Adding clamping step seems not completely solves this problem.\n\n3)\tExperiment：\nThe state-of-the-art segmentation performance may be due to many factors such as strong DINO backbone features, CRF processing etc. Hoping more ablation studies are performed on each component of the proposed contrastive loss to verify the effect. For example, more ablation studies based on the ViT-Base architectures of DINO are preferred. The ablation studies on the hyperparameters are also preferred to be added. \n\n\n\n\n\n", "summary_of_the_paper": "In this paper, the author proposes an unsupervised semantic segmentation approach STEGO. The core of STEGO is a novel contrastive loss function that encourages features to form compact semantic clusters. ", "summary_of_the_review": "I tend to weakly reject the paper based mainly based on the concerns about implementation and experiments mentioned above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2IMTyVtvhUk", "reviewer": "Reviewer_cF6T", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I like the idea of this paper and vote for acceptance. The idea of distilling frozen and general self-supervised features for segmentation specific features is neat. Though as expected, there are several regularisations and 'tricks' to really make the idea work in practice, they are well-motivated and valitdated in the ablation experiments.\n\nMy major concern is about more clarifications of details and some additional ablation models (see cons below). Hopefully the authors can address my concern in the rebuttal period. \n\nPros:\n- The paper is well written and easy to follow. The key design choices in the paper are well motivated and described in the paper.\n- The idea of pushing features from self-supervised representation learning towards segmentation task-specific ones is really interesting, simple and effective.\n-Extensive experiments are done to validate the system and promising segmentation quality is achieved.  \n\nCons/Suggestions:\n\n(1) The discussion in Sec 3.3 could be better arranged in my opinion and it is easier for readers to follow the main idea by jumping from Sec 3.2 to Sec. 3.4. I agree the discussion is helpful to connect the idea of this method to undirected graphical model and can be put into later part of paper.\n\n(2) Is the choice of distance very improtant generally? In addition to consine distance, do normalised l2-distance or RBF/Gaussian perform similarly? I think the choice of distance or similarity metric is quite important and worth more discussions.\n\n(3) I would like to see more qualitative results of STEGO without CRF step as CRF usually smooths out the predictions. It is more informative for readers to know the \"raw\" segmentations before CRF to better understand the systems. Do segmentations before CRF usually become noisy or have coarse boundries aligned with images? \n\n(4) More clarifications:\n- Since 5 crops are used before KNN sampling, is there a tendency to select other crops from the same image instead of different images? If it is true, is it intended to do so ? More clarifications would be good. \n\n- Similar to last point,  in Table 2, the effectiveness of 5-crop and SC are not properly adjustified. How about the performance with a full  STEGO model exluding 5-crop or SC only, like the ablation of CRF. I am wondering, instead of adding on top of vanilla baselines, if the contribution of these designs will be less prominent when other modules already exist?\n\n- I was a bit concerned about the huge computational cost in constructing dense correspondence tensors F and S at the beginning. In supplement it is mentioend that 121 samples are taken per step per training image, is this used to alleviate the cost of building these dense matrix? Will STEGO work better given larger training images with more closer details?\n\n\nSome typos: \n\n(1) Sec 2 Unsupervised Semantic Segmentation: between an an-> between an.\n\n(2)  Sec 4.2: We show some examples segmentations -> We show some example segmentations.\n", "summary_of_the_paper": "This paper proposed STEGO, an unsupervised approach for semantic clustering/segmentation using feature refinement on top of self-supervised neural networks. A distilled version of self-supervised features is learned segmentation specifically through a feed-forward network via SGD.\n\nThe main contribution comes from an idea of distilling powerful deep features from strong self-supervised backbones to further improve their semantic discriminativeness. Though the idea is intuitive, several loss functions and regularisations are proposed to avoid trivial solutions and make the idea really work in practice. Extensive qualitative and quantitatve results are done to demonstrate the performance of STEGO and its design choices.", "summary_of_the_review": "I have carefully read the paper and like the idea of this paper in making self-supervised representation leanring towards dense predicton tasks like segmentation/clustering without any explicit mannual supervision.\n\nI am aware of pros and cons in this works and would like to vote for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received two accept and two marginally accept recommendations. All reviewers find value in the proposed supervised semantic segmentation methodology (making self-supervised representation learning towards dense prediction tasks like segmentation or clustering without explicit manual supervision) and appreciate the experimental gains, but had (mostly practical) criticism that was reasonably well addressed in the rebuttal.
Decision: Accept (Poster)