SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentation
Authors: Mengcheng Lan
Xinjiang Wang
Yiping Ke
Jiaxing Xu
Litong Feng
Wayne Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: unsupervised semantic segmentation; self-supervised learning; smoothness prior
TL;DR: This paper presents a novel  unsupervised semantic segmentation approach based on the smoothness prior within an energy minimization framework.

Abstract: Unsupervised semantic segmentation is a challenging task that segments images into semantic groups without manual annotation. Prior works have primarily focused on leveraging prior knowledge of semantic consistency or priori concepts from self-supervised learning methods, which often overlook the coherence property of image segments. In this paper, we demonstrate that the smoothness prior, asserting that close features in a metric space share the same semantics, can significantly simplify segmentation by casting unsupervised semantic segmentation as an energy minimization problem. Under this paradigm, we propose a novel approach called SmooSeg that harnesses self-supervised learning methods to model the closeness relationships among observations as smoothness signals. To effectively discover coherent semantic segments, we introduce a novel smoothness loss that promotes piecewise smoothness within segments while preserving discontinuities across different segments. Additionally, to further enhance segmentation quality, we design an asymmetric teacher-student style predictor that generates smoothly updated pseudo labels, facilitating an optimal fit between observations and labeling outputs. Thanks to the rich supervision cues of the smoothness prior, our SmooSeg significantly outperforms STEGO in terms of pixel accuracy on three datasets: COCOStuff (+14.9\%), Cityscapes (+13.0\%), and Potsdam-3 (+5.7\%).

SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentation
Mengcheng Lan Xinjiang Wang'
Yiping Ke²
Jiaxing Xu²
Litong Feng' Wayne Zhang*
S-Lab, Nanyang Technological University
2 SCSE, Nanyang Technological University
SenseTime Research
(lanm0002, jiaxing003) ntu. edu. sg (wangxinjiang, fenglitong, wayne zhang) com
ypke@ntu.
https:/ https://github. htt///tttuuu....//// wwmmmm-ooeeeooooooooooooooooo lan a
Abstract
Unsupervised semantic segmentation is a challenging task that segments images into semantic groups without manual annotation. Prior works have primarily fo- cused on leveraging prior knowledge of semantic consistency or priori concepts from self- supervised learning methods, which often overlook the coherence prop- erty of image segments. In this paper, we demonstrate that the smoothness prior, asserting that close features in metric space share the same semantics, can signifi- cantly simplify segmentation by casting unsupervised semantic segmentation energy minimization problem. Under this paradigm, we propose novel approach called SmooSeg that harnesses self-supervised learning methods to model the closeness relationships among observations as smoothness signals. To effectively discover coherent semantic segments, we introduce novel smoothness loss that promotes piecewise smoothness within segments while preserving discontinuities across different segments. Additionally, to further enhance segmentation quality, we design an asymmetric teacher-sture student style predictor that generates smoothly updated pseudo labels, facilitating optimal fit between observations and labeling outputs. Thanks to the rich supervision cues of the smoothness prior, our SmooSeg significantly outperforms STEGO in terms of pixel accuracy three datasets: COCOStuff (+ (+14.9%) Cityscapes (+ (+13.0%), and Potsdam-3 (+5.7%)
Introduction
Semantic segmentation is crucial task in computer vision that allows for better understanding of the visual content and has numerous applications, including autonomous driving [I] and remote sensing imagery (2). Despite advancements in the field, most traditional semantic segmentation models heavily rely on vast amounts of annotated data, which can be both arduous and costly to acquire. Consequently, unsupervised semantic segmentation 3 4 promising alternative. Prior knowledge is fundamental to the success of unsupervised semantic segmentation models. One key prior knowledge is the principle of semantic consistency, which stipulates that an object's semantic label should remain consistent despite photometric or geometric transformations. Recent advances (3 9 [0] use contrastive learning achieve consistent features or class assignments. Another essential prior knowledge is the priori concepts implicitly provided by self-supervised learning techniques, e.g., DINO [II] and precedent arts [12 g 6) whose learned features can be employed to partition each image into different segments. Despite their effectiveness,
5
6
7 g has emerged
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Sky
Sports
Building
Ground
Plant
Water
Person
Animal
Solid
Outdoor
Structural
Image
Label
SmooSeg
STEGO
TransFGU
Figure A case study of our SmooSeg with state-of-the-art, STEGO and TransFGU
[6]. on the COCOStuff dataset. Our observations reveal that the segmentation maps generated by STEGO and TransFGU for regions such the sand beach (first row) and the grassland (second row) are incomplete and lack smoothness and coherence. In contrast, our SmooSeg exhibits improved segmentation results for all these regions by considering the smoothness prior.
these methods often overlook the coherence property of image segments, resulting in predicted segments that are incomplete and lacking in coherence, shown in Fig.
Real-world images often demonstrate a natural tendency towards piecewise coherence regarding semantics, texture, or color. Observations close to each other, either in the form of adjacent pixels in the coordinate space close features in metric space, are expected share similar semantic labels, and vice versa. This essential property, known as the smoothness prior, plays crucial role in various computer vision tasks [I3 [5] Surprisingly, still under-explored in the field of unsupervised semantic segmentation.
In this paper, we attempt tackle unsupervised semantic segmentation from the perspective of smoothness prior. As dense prediction task, semantic segmentation aims at finding a labeling e F that assigns each observation (pixel, patch. features) could be formulated within an energy minimization framework [6]: E(f) = Emmooth (.f) + (f). Emmosth a pairwise smoothness term that promotes the coherence between observations, and Esaa represents pointwise data term that measures how well (p) fits the observation p. However, directly applying smoothness prior to unsupervised semantic segmentation faces several obstacles. 1) Due to the large cara-class variations in appearances within an image, it is difficult to define well-suited similarity (dissimilarity) relationship among low-level observations. This makes it challenging to discover groups of complex observations coherent segments. 2) Exmosth can lead to trivial solution where f becomes smooth everywhere, a phenomenon known as model collapse. 3) Optimizing Edata without any observed label can be challenging.
semantic category f(p), which
In this study, we propose a novel approach called SmooSeg for unsupervised semantic segmentation address the aforementioned challenges, By leveraging the advantages of self-supervised representation learning in generating dense discriminate representations for images, we propose model the closeness relationships among observations by using high-level features extracted from frozen pre-trained model. This helps capture the underlying smoothness signals among observations. Furthermore, we implement novel pairwise smoothness loss that encourages piecewise smoothness within segments while preserving discontinuities across image segments to effectively discover various semantic groups. Finally, we design an asymmetric teacher-student style predictor, where the teacher predictor generates smooth pseudo labels to optimize the data term, facilitating good fit between the observations and labeling outputs.
Specifically, our model comprises frozen feature extractor, a lightweight projector, and predictor. The projector serves to project the high high-dimensional features onto a more compact, low-dimessional embedding space, and the predictor employs two sets of learnable prototypes generate the final segmentation results. We optimize our model using a novel energy minimization objective function. Despite its simplicity, our method has demonstrated remarkable improvements over state-of-the- approaches. In particular, our method significantly outperforms STEGO [ in terms of pixel accuracy on three widely used segmentation benchmarks: COCOStuff 14.9%)) Cityscapes (+13.0 %), and Potsdam-3 (+5.7%).

---

2 Related work
Unsupervised semantic segmentation has gained increasing attention for automatically partitioning images into semantically meaningful regions without any annotated data. Early CRF models [17 [8] [8] incorporate smoothness terms that maximize label agreement between similar pixels. They define adjacency for a given pixel in the coordinate space, e.g., using 4-connected or 8 connected grid, which relies heavily on the low-level appearance information and falls short in capturing high-level semantic information in images. Recently, many methods [ 4 [Q] have attempted to learn semantic relationships the pixel level with semantic consistency as supervision signal. For example, IIC R is clustering method that discovers clusters by maximizing mutual information between the class assignments of each pair of images. PICIE [4] enforces semantic consistency between an image and its photometric and geometric augmented versions. HSG [TO] achieves semantic and spatial consistency of grouping among multiple views of an image and from multiple levels of granularity. Recent & 20 have benefited from self-supervised learning techniques, which provide priori concepts as supervision cues. For instance, InfoSeg [99 segments images by maximizing the mutual information between local pixel features and high-level class features obtained from self-supervised learning model. The work in [2] directly employs spectral clustering on an affinity matrix constructed from the pre-trained features. TransFGU [6] generates pixel-wise pseudo labels by leveraging high-level semantic concepts discovered from DINO [I] Additionally, STEGO (8) utilizes knowledge distillation to learn a compact representation from the features extracted from DINO based on correspondence distillation loss, which also implies a smoothness regularization through the dimension reduction process. However, the utilization of smoothness prior in STEGO is implicit and entails separate post-process, such as min-batch K-Means, for the final semantic clustering. Besides, MaskContrast [2I] and FreeSOLO [2 leverage mask priors and primarily focus on foreground object segmentation. In contrast, we propose to leverage the smoothness prior as a supervision cue to directly optimize the generated semantic map, achieving more coherent and accurate segmentation results.
advances 19 12
Self-supervised representation learning (SSL) aims learn general representations for images without additional labels, which has offered sigmificant benefits to various downstream tasks, including detection and segmentation (6 8). One main paradigm of SSL is based on contrastive learning 26 27 28 29]. which seeks to maximize the feature similarity between an image and its augmented pairs while minimizing similarity between negative pairs. For example, MoCo 24 trains a contrastive model by using a memory bank that stores and updates negative samples in a queue-based fashion. SimCLR (23 proposes learn a nonlinear transformation, i.e., a projection head, before the contrastive loss, to improve performance. Notably. DINO M. built upon Vision Transformer (ViT) 300, has nice property of focusing on the semantic structure of images, such as scene layout and object boundaries. Features extracted by DINO exhibit strong semantic consistency
23
24
25
and have demonstrated significant benefits for downstream tasks 1 6. g) Another mainstream belongs to the generative learning approach [31 32 33 MAE [32 and SimMIM (31] propose to predict the raw masked patches, while MaskFeat (32) proposes to predict the masked features of images. Our work also leverages recent progress in SSL for unsupervised semantic segmentation.
3 Method
Problem setting, Given a set of unannotated images 1 = II....,, Ip] E RBX3xXXW where B denotes the number of images, and 3, H1, W represent the channel, height, and width dimensions respectively, the objective of unsupervised semantic segmentation is to learn a labeling function f F that predicts the semantic label for each pixel in each image. We represent the predicted semantic maps as Y = [Y,...., YB! E [1,-.. K] BXBxXXM HxW where K refers to the number of predefined categories.
Architecture. To achieve this goal, we introduce the SmooSeg approach, which capitalizes on self- supervised representation learning and smoothness prior within an energy minimization framework, as illustrated in Fig. 2 SmooSeg comprises three primary components: a feature extractor fo. a projector hag and a predictor go- Initially, for each image I,, we employ a pre-trained backbone network, such as a frozen version of DINO, to acquire feature representations X; = fo(T:) E RCNN where C' and N denote the number of feature channels and image patches, respectively. Subsequently, the projector he maps these features onto a low-dimensional embedding space, resulting in a set
3

---

Extractor
Projector
Predictor
A
X
E
data
P
argmax
1;
fo
h
yo
EMA
A
P
Stalled
Producr
smoothh
W & W
Figure 2: Overview of our SmooSeg framework, showing the application of the smoothness prior within image I, and across images I and Iv. sg denotes the stop-gradient operation.
of compact features Z; ho (X,) RDX,, where D denotes the reduced feature dimensionality. Finally, the predictor Do generates the label assignments A (1.1) E KxN by computing the similarity scores between the compact features Z; and the prototypes p(a,t, Here, Ps and p represent student and teacher prototypes. respectively. The semantic map Y; for image can be obtained by reshaping the output Y of the teacher branch.
3.1 Smoothness Prior
Real-world images typically exhibit inherent continuity and coherence in terms of semantics, texture, and color. Within single object, semantic labels tend to demonstrate smoothness and consistency, ensuring a cohesive representation the object. In contrast, labels between distinct objects manifest discontinuity and divergence, facilitating the separation of different object instances. This essential property, known as the smoothness prior, is expected to play a critical role guiding unsupervised semantic segmentation tasks toward more accurate and meaningful segmentation results. We therefore consider the following pairwise smoothness term:
B
N
Esmorth
Σ
Σ
W
5(Y Yi.g):
i=1 p.gml
where W E N is the closeness matrix of image I, 5(Y ip: Yi.g) is the penalty that takes the value of if .,p * Vi.. and otherwise. By minimizing this smoothness term, two close patches with different labels will be penalized. In other words, the segmentation model is encouraged to assign similar labels to close patches, thereby promoting the coherence within objects.
Closeness matrix. It is worth noting that the large intra-class variation in appearances within the raw pixel space renders the discovery of well-suited closeness relationships among low-level observations challenging. We therefore propose to model the closeness relationships by the cosine distance in the high-level feature space. Specifically, W can be calculated by:
Xi.p X
Win pq
where Xi, and Xi, represent the feature vectors for patches and of image I,, respectively. Theoretically. a large element value in the closeness matrix, i.e., a high cosine similarity, suggests high possibility of a close patch pair, and vice versa. We apply a zero-mean normalization to this matrix: W W 1/4 E, W pq' This normalization balances the negative and positive forces during optimization, which prevents excessive influence from either the negative or positive components of the closeness matrix and ensures that the optimization process is more stable.
Label penalty. Directly minimizing Eq. to optimize our segmentation model is not feasible due to the non-differentiable property of 8(.,-) and the hard label assignment Y. As result, have to

---

resort to another form of penalty cost. Suppose we have the soft label assignment A E RKxN of image I, (which will be introduced later), by which we can redefine the penalty cost function as:
A A 8(A B(A Alag)= =1
| 11.
Because the non-negative property of the softmax output, i.e., 0 S. A*, 0 < -(--)) < always holds. larger value of -)--) denotes greater dissimilarity between two labels, thereby indicating a higher penalty cost, and vice versa.
Smoothness prior within and across images. To prevent the model from converging to a trivial solution where the labeling function becomes smooth everywhere, we also apply the smoothness prior across images, acting as a strong negative force, by introducing another image Ly that is randomly selected from the current batch. We then obtain the final smoothness term:
B
N
Esmooth
smooot
smooth
Σ
Σ P.q=1
{( bb)- + - -2) A: (4)
Here, we introduce scalar by adjust the threshold for applying the penalty. That is, when W -1 > O, indicating that two patches p, with a high closeness degree are nearby patches in the embedding space, patches p. q with different labels will be penalized, encouraging the piecewise smoothness within segments; otherwise, they are rewarded to assign different labels, leading to the discontinuities across segments. By doing so, SmooSeg capable of finding globally coherent semantic segmentation maps.
Discussion with CRF and STEGO. CRF methods I7: 18] model the closeness relationship of pixels using their spatial coordinates, emphasizing the local smoothness within each image. On the contrary, our SmooSeg encodes the global closeness relationship of image patches based on the cosine distance in the feature space, which discover the high-level semantic groups of images. Our smoothness term appears to be similar to the correlation loss in STEGO; Zeor (F 6 ma (S, 0), but essentially the two losses model different things. In STEGO, S denotes the feature correlation, by which STEGO aims to learn low-dimensional compact representations for images through learnable projection head. A separate clustering algorithm, e.g., k-means, is required to obtain the final segmentation maps. However, even with the learned compact representations, the coherence of image segments is not guaranteed in STEGO as slight differences in features may lead to inconsistent labels in the clustering stage. In contrast, our SmooSeg aims to directly learn labeling function (projector + predictor) based on the smoothness prior, which encourages piecewise smoothness within segments and preserves disparities across segments, leading to more coherent and semantically meaningful segmentation maps. Additionally, the negative part of S contradicts the learning intention of STEGO and therefore requires a O-clamp via max 0), which however, represents discontinuities between image patches and should be preserved. In contrast, our label penalty 0 < 8(.,)) < 1 has desirable property compared to S.
3.2 Asymmetric Predictor
desirable labeling function learnt through energy minimization should on the one hand produce piecewise smooth results, and on the other hand be well fit between the observations and labeling outputs. For semantic segmentation, we expect the labeling output of an image to align well with its semantic map. In other words, the labeling output should accurately predict category for each individual pixel with high confidence or low entropy. However, this goal is a nutshell in unsupervised semantic segmentation as there is no observed semantic map.
Self-training (34 29 emerges as a promising solution for tasks involving unlabeled data. To address the above challenge, we design asymmetric student-teacher style predictor to learn the labeling function through a stable self-training strategy. The student branch employs set of K learnable prototypes (class centers) Ps = [rf ... pkkl
E R to predict the semantic maps of images. The teacher branch holds the same number of prototypes pt as the student, and pd is updated as an exponential moving average of pa. We then compute the soft assignment (5.1) of the embeddings
Z, with the prototypes ple.t) by computing their cosine similarity. With Eo-oomaleeeed embeddings
5

---

Algorithm SmooSeg: PyTorch-like Pseudocode
f, extractor, projector
Ps, Pt, tau, atpha: student prototypes, teacher prototypes, temperature, momentum
for img in dataloader:
f f(igg)
feature extraction
2 hxx)
feature projection
{x, 2, Ps, Pt) = F. normalize(fx, 2, Pa, Pt}, dim=1) detaach() Ps) (z. (2.
As matmul
st student label assi assignment
At softmax (matmul (z.T, Pt. detach () /tu, dim-1)
teacher label ass ass snnment
loss_data F. c.cross_entropp (As, argmax(At))
At F. normalize dim=1)
{x_p, At_p) shuffle((x, At})
shuffle along the datch dimension
i = matmul (x. x) matmul T, x) mean (dim=-1)
R_p matmul (x. T, x_p) matmul (x.T, T, x_p) mean (dim--1)
delta matmul (At.T, T, At)
delta_p Loss_smooth sum((W b1) delta (W_p b2) delta_p)
matmul (At.T, At At_p)
(loss_smooth loss_data) backward update(h, Ps) Pt alpha Pt alpha)
update projector and student prototypes
momentum update teacher prototypes
2, = Z:/1 and prototypes Pla.t p(a.) /II |].
we have
A = softmax Pa sg( A softmax((sg(P")) 2)/r) E RRN
(5)
where temperature parameter t > controls the sharpness of the output distribution of the teacher branch. The teacher branch is responsible for generating smoothly updated pseudo labels to supervise student prototypes' learning. By using a patch-wise cross-entropy loss, we have the data term as
B
N
Edata Σ
I vi, & log A
(6)
where I. is an indicator that outputs the argument is true, and O otherwise. Y argmax A the hard pseudo label for patch p of image I, By minimizing Edaus the segmentation model is expected to generate label assignments for each patch with high confidence, thus ensuring a better fit between the observations and their predicted labels.
3.3 Overall Optimization Objective
Our final optimization objective function for training SmooSeg is obtained by incorporating the smoothness term and the data term follows:
B
N
L
(
b() Ain)) ) + (Wi'" - -22 50( Ab,)))
1-
(7)
K
Σ ΣΣ 1x log Aog
In practice, L could be approximately minimized using Stochastic Gradient Descent (SGD). During each training iteration, the projector optimized using gradients from the smoothness loss, while the student prototypes are optimized using gradients from the data loss. The teacher prototypes are updated as an exponential moving average of the student prototypes: p at + (1 - a)) Ps, with denoting the momentum value. After training, we use the output from the teacher branch as the segmentation results. The overall procedure in pytorch-like pseudocode of SmooSeg is summarized in Algorithm

---

Experiments
4.1 Experimental Setup
Datasets. Our experimental setup mainly follows that in previous works 8 4 in datasets and evaluation protocols. We test on three datasets. COCOStuff [35] is a scene-centric dataset with total of 80 things and 91 stuff categories. Classes merged into 27 categories for evaluation, including 15 stuff and 12 things. Cityscapes [36] a collection of street scene images from 50 cities, with classes merged into 27 classes by excluding the "void" class. Potsdam-3 []] is remote sensing dataset with 8550 images belonging to 3 classes, in which 4545 images are used for training and 855 for testing.
Evaluation metries. For all models, we utilize the Hungarian matching algorithm to align the prediction and the ground-truth semantic map for all images. We also use CRF 17 the post- processing to refine the predicted semantic maps. Two quality metrics including mean Intersection over Union (mIoU) and Accuracy (Ace) over all the semantic categories are used in the evaluation.
RTX 3090 GPU. To ensure a fair comparison with previous works 8), we use DINO | with a ViT-small 8 x 8 backbone pre-trained model training. Our projector consists of a linear layer and a two-layer SiLU MLP whose outputs summed together. The predictor contains two sets of prototypes with the same initialization. The exponential moving average (EMA)
Implementation details. Our experiments were conducted using PyTorch 33
ImageNet as our default feature extractor, which is frozen during
set to a = 0.998. The dimension of 0.1. We use the Adam optimizer
the embedding space is D 64. The temperature is set with learning rate of x 10-1 and 5 x 10-1 for the projector and predictor, respectively.
38
Table I: Performance on the COCOStuff dataset (27 classes).
We set a batch size of 32 for all datasets. For Cityscapes and COCOStuff datasets, we employ a five-crop technique to augment the training set size. We train our model with total of 3000 iterations for Cityscapes and Potsdam-3 datasets, and 8000 iterations for the COCOStuff dataset.
Methods
backbone
Acc.
mloU
ResNet50
39
ResNet50
24.6
8.9
IIC [3]
R18+FPN
21.8
6.7
MDC
RI8+FPN
32.2
9.8
PICIE 4
R18+FPN
48.1
13.8
4.2 Comparison with State-of-the-Arts
PiCIE+H
R18+FPN
50.0
14.4
SlotCon 29
ResNet50
42.4
18.3
Quantitative results. We summarise the quan- titative results on three datasets in Tables T 2 and respectively. Results of baselines, ResNet50[39], MoCoV2(40) and DINO(IT] are directly cited from the paper (8) while the re- sults of DINOV2 41 (Table 3) are obtained by our implementation. For these baselines, we first extracted dense features for all images, then utilized a minibatch k-means algorithm to per- form patches grouping, which resulted in the final segmentation maps. Our SmooSeg sig- nificantly outperforms all the state-of-the-arr methods in terms of both pixel accuracy and the CO-
MoCoV2 40
ResNet50
25.2
10.4
+ STEGO
8
ResNet50
43.1
19.6
SmooSeg
ResNet50
52.4
18.8
DINO [|]]
ViT-S/8
29.6
10.8
+ TransFGU 6
ViT-S/8
52.7
17.5
+ STEGO g
ViT-S/8
48.3
24.5
+ SmooSeg
VIT-S/8
63.2
26.7
Table 2: Performance on the Cityscapes Dataset (27 Classes).
Methods
backbone
Acc.
mIoU
mIoU on all datasets. In particular, COStuff dataset in Table with DINO ViT-S/8 as backbone, SmooSeg gains 14.9% improve- ment in pixel accuracy and 2.2% improve- ment in mloU over the berf-oorngg baseline STEGO.
IIC 3
RI8+FPN
47.9
6.4
MDC
RI8+FPN
40.7
7.1
PICIE
R18+FPN
65.5
12.3
DINO
ViT-S/8
40.5
13.7
+ TransFGU 6
ViT-S/8
77.9
16.8
+ STEGO 8
VIT-S/8
69.8
17.6
We observe that TransFGU outperforms STEGO terms of accuracy, but is inferior mloU on both COCOStuff and Cityscapes. This is due to
+ SmooSeg
VIT-S/8
82.8
18.4
7

---

Image
Label
SmooSeg
STEGO
TransFGU
Figure 3: Qualitative results on COCOStuff (left) and Cityscapes (right) datasets.
Table 3: Performance the Potsdam-3 Dataset.
Amad
Methods
backbone
Acc. mloU
Random CNN 3
VGG11
38.2
K-means 42
VGG11
45.7
Label
SIFT
45
VGG11
38.2
ICC
3
RI8+FPN
65.1
Deep Cluster 44
R18+FPN
41.7
InfoSeg 19
CNN
71.6
DINO 11
VIT-B/8
62.2
43.3
STEGO
VIT-B/8
77.0
62.6
SmooSeg
V/T-B/8
82.7
70.3
STEGO
DINOV2 41
VIT-B/14
81.8
69.0
SmooSeg
VIT-B/14
86.3
75.7
Figure 4: Qualitative results on Postdam-3 dataset.
the fact that TransFGU adopts pixel-wise cross- cross-entropy loss, which focuses more on the overall accuracy of pixels, while STEGO achieves better balaneed segmentation results through mini- batch k-means. Our SmooSeg significantly outperforms both TransFGU and STEGO in both accuracy and mloU, We attribute this superiority our energy minimization loss, which optimizes both the smoothness term and the data term simultaneously.
Same conclusions be drawn the Postdam-3 dataset, shown in Table We SmooSeg, with DINO ViT-B/8 as the backbone, significantly outperforms STEGO, with gains of 5.7% in accuracy and 7.7% in mloU. The improvement is particularly significant in terms mloU. This is not surprising as Potsdam-3 is a remote sensing image dataset that contains only 3 classes, so segments the Potsdam-3 are often relatively large. In such scenario, the smoothness prior becomes even more important in ensuring coherent segmentation maps.
that
Qualitative results. We present qualitative examples of SmooSeg, STEGO and TransFGU on three datasets in Figs. and Additional qualitative results, along with color maps, can be found in Appendix [C] As shown Fig. 3 SmooSeg produces high- high-quality fine- mne-grained segmentation maps that outperform those obtained by STEGO and TransFGU, Though STEGO uses feature correspondence loss encourage features to form compact clusters, its segmentation maps still suffer from incoherence problems. It can be observed that, slight differences in features can result in inconsistent labels in STEGO. The second image from the left column Fig. 3 shows such an example: the slight variations in light the surface results in differences in features, leading an

---

Tedtars 1ESI-EL
Enbedding Accedd mesondarr
Endaaciing
Centers:
Centers:
Centers:
visualization of feature embeddings for DINO, STEGO, and our SmooSeg on the Potsdam-3 dataset. The large-scale image is partitioned into 25 sub-images with resolution of 224 x 224 before being input into each algorithm. We highlight the distribution of three sub-images using square, star and triangle_up markers, respectively, and plot the class centers of each algorithm using three pentagon markers with different colors, where teal, yellow and purple represent buildings + clutter, roads cars, vegetation + trees, respectively. Best viewed color and zoomed in.
Figure 5: t-SNE 445
incomplete and incoherent water segment. Similar phenomenon can be observed in the segmentation maps Cityscapes and Potsdam-3 too. Besides, although TransFGU is top-down approach, it still overlooks the relationship between image patches in its top-down approach, and therefore achieves much worse segmentation results. In contrast, SmooSeg with the aim of generating smooth label assignments within segments while preserving differences across different segments by leveraging the smoothness prior, the semantic maps produced by SmooSeg show more coherent and semantically meaningful results. In Fig. 4 we can see that SmooSeg outperforms the other methods in terms of accurate boundaries.
4.3 Analyses
Visualization. Feature visualizations of DINO, STEGO and SmooSeg are illustrated in Fig. 5 We can see that the feature distribution of DINO with ViT-base/8 the backbone exhibits some semantic consistency, with compact clusters within each image but disperse across images. The embeddings of STEGO, which are distilled from DINO features using feature correspondence loss, show higher semantic consistency than DINO, with more compact clusters across images, such as the yellow markers, and improved performance. However, STEGO still suffers from the label incoherence problem due to the large intra-class variation embeddings, indicating that feature distillation alone insufficient to capture the high-level semantic coherence of segments. Our SmooSeg leverages the smoothness prior to encourage smooth label assignments, measured by the cosine distance between patch embeddings and prototypes (centers), and achieves remarkable improvement in the semantic consistency of feature embeddings. As shown in the right part of Fig. [5] SmooSeg produces highly semantically compact and coherent clusters with clear class boundaries for all images, and the performance, at 87.4% Acc and 77.8% mloU, significantly higher than STEGO. These results further prove the effectiveness of our SmooSeg using smoothness prior for unsupervised semantic segmentation.
Objective function. To assess the effectiveness of our energy minimization objective function, we conduct an ablation study on the COCOStuff dataset by comparing SmooSeg with four variants of the objective function, each with different term removed. For the variant of w/o Ednta we only keep the prototypes in the teacher branch to generate the label map for the smoothness term. The results are shown in Table 4 where Eacross denotes the smoothness term across different images. We can see that the performance drops by 10.2% on Ace and 1.2% on mIoU when removing the data term Edata- which highlights the importance of the data term in promoting the fitting of the labeling function. Besides, removing Egcross results in a much larger drop in performance, with decrease of 27. 1 on Acc and 16.3% on mloU. Moreover, our segmentation model fails when removing the entire Fsmooth- The smoothness term utilizes a closeness matrix constructed from high-level features of pre-trained model, acting as strong supervision signals to guide the label learning for all

---

Table 4: Analysis of objective function on the COCOStuff dataset.
80
mieu
60
60
88.
Methods
Acc.
mloU
SmooSeg
63.2
26.7
20
20-
miou
w/o Edata
53.0
25.5
0.07
0.1
0.7
0.4
0.6
0.8
0.1
0.5
0.9
0.99
0.995 0.998
w/o Eueruss
36.1
10.4
Temperature
Momentum
smooth
w/o Smmothh
16.8
0.6
Figure 6: Sensitivity study on the temperature parameter
Figure Sensitivity study on the momentum parameter a.
image patches. Therefore, it is reasonable performance. On the contrary, the data term operates in a self-training fashion with pseudo labels derived from the teacher branch, which alone cannot generate accurate segmentation maps. These findings demonstrate the crucial role of both the data and smoothness terms for optimal performance of SmooSeg in unsupervised semantic segmentation.
that Esmooth contributes significantly the overall
Temperature parameter We investigate the effect of the temperature parameter the performance SmooSeg on COCOStuff dataset, and report the results in Fig. 6 Theoretically, smaller sharpens the softmax output, providing greater gradients and supervision signals for model training. Fig. shows that plays critical factor in the success of SmooSeg. Specifically, SmooSeg achieves good results when S 0.1, while performance drops considerably when 5 2 0.2 because the softmax output tends to become uniformly distributed.
Momentum parameter a. We also study the impact of the on SmooSeg. a controls the smoothness of the update of the teacher predictor from the student predictor. We plot the performance on the COCOStuff dataset as changes from 0. to I in Fig. 7 The performance of SmooSeg gradually improves increases, and reaches stable when 0.99 a.
Limitation. Setting hyper-parameters without cross-validation is always a challenge for unsuper- vised learning methods. The main limitation of method is that it involves two dataset-specific hyper-parameters in the smoothness term. We present a feasible strategy Appendix A to alleviate this issue.
Conclusions
In this paper, we propose SmooSeg, simple effective unsupervised semantic segmentation approach that delves into the potential of the smoothness prior, emphasizing the coherence property of image segments. In particular, we implement pairwise smoothness loss to effectively discover semantically meaningful groups. We also design an asymmetric teacher-student style predictor to generate high-quality segmentation maps. SmooSeg comprises frozen extractor, well as a lightweight projector and a predictor which could be optimized using our energy minimization objective function. Experimental results show that SmooSeg outperforms state-of- soffthe-art approaches on three widely used segmentation benchmarks by large margins.
Acknowledgement. This research is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well cash and in-kind contribution from the industry partner(s), by the National Research Foundation, Singapore under its Industry Alignment Fund Pre- Pre-positionin (IAF-PP) Funding Initiative, and by the Ministry of Education, Singapore under its MOE Academic Research Fund Tier (STEM RIE2025 Award MOE-T2EP20220- 0006). Any opinions, findings and conclusions or recommendations expressed in this material are those the author(s) and do not reflect the views National Research Foundation, Singapore, and the Ministry of Education, Singapore.
10

---

References
[1] Mennatullah Siam, Mostafa Gamal, Moemen Abdel-Razeke Senthil Yogamani, Martin Jagersand. and Hong Zhang. A comparative study of real-time semantic segmentation for autonomous driving. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 587-597, 2018.
[2] Xiaohui Yuan, Jianfang Shi, and Lichuan Gu. A review of deep learning methods for semantic segmentation remote sensing imagery. Expert Systems with Applications, 169:1 14417, 2021.
[3] Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsuper- vised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9865-9874, 2019.
[4] Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16794-16804, 2021.
[5] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Labels4free: Unsupervised segmentation using stylegan. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19979, 2021.
[6] Zhaoyuan Yin, Pichao Wang, Fan Wang, Xianzhe Xu, Hanling Zhang, Hao Li, and Rong Jin. Transfgu: a top-down approach to fine-grained unsupervised semantic segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27,
2022, Proceedings, Part XXIX, pages 73. Springer, 2022.
[7] Shanghua Gao, Zhong- Yu Li, Ming-Hsuan Yang. Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[8] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint arXiv:2203.08414, 2022.
[9] Yassine Ouali, Céline Hudelot, and Myriam Tami. Autoregressive unsupervised image segmen- tation. In Computer Mision-ECCh 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part VII 16. pages 142-158. Springer, 2020.
[10] Tsung- Wei Ke, Jyh-Jing Hwang, Yunhui Guo, Xudong Wang, and Stella X Yu. Unsupervised hierarchical semantic segmentation with multiview cosegmentation and clustering transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2571-2581, 2022.
[11] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[12] Luke Melas-Ky Perias,,riizzi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral meth- ods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8364 0011-837,,22 2022.
[13] Meng Tang, Federico Perazzi, Abdelaziz Djelouah, Ismail Ben Ayed, Christopher Schroers, and Yuri Boykov. On regularized losses for weakly-supervisee enn segmentation. Proceedings of the European Conference on Computer Vision (ECCV), pages 507-522, 2018
[14) Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia. Underexposed photo enhancement using deep illumination estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1849-6851) 2019.
[15] Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2613-2622, 2021.

---

[16] Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. IEEE Transactions on pattern analysis and machine intelligence, 23((1):1222-12)) 2001.
[17] Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected erfs with gaussian edge potentials. Advances in neural information processing systems, 24, 2011.
[18] Xide Xia and Brian Kulis. W-net: A deep model for fully unsupervised image segmentation. arXiv preprint 1rXiv:1711.08000 2017.
[19] Robert Harb and Patrick Knöbelreiter. Infoseg: Unsupervised semantic image segmentation with mutual information maximization. In Pattern Recognition: 43rd DAGM German Conference, DAGM GCPR 2021, Bonn, Germany, September 28-October 1, 2021, Proceedings, pages 18-32. Springer, 2022.
[20] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl- Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap to real-world object-centric learning. arXiv preprint arXiv:2209. 14860, 2022.
[21] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsu- pervised semantic segmentation by contrasting object mask proposals. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10052-10062, 2021.
[22] Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. Freesolo: Learning to segment objects without annotations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14176-14186. 2022.
[23] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729- 9738, 2020.
[25] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 13:21271-21288 2020.
[26] Ishan Misra and Laurens der Maaten. Self-supervised learning of pretext-invariant rep- resentations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6707-6717, 2020.
[27] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1807.03747 2018
(28) Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3024-3033, 2021.
[29] Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. arXiv preprint arXiv: /5288, 2022.
[30] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 2020.
arXiv:2010. 19999 1929,
[31] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2022.
12

---

[32] Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14668-14678. 2022.
[33] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[34] Qizhe Xie, Minh-Thame Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
[35] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1209-12: 1218, 2018.
[36] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2223-3223, 2016.
[37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412 2014.
[39] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[40] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint #xXiv:2003.04297 2020.
[41] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[42] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning python. the Journal of machine Learning research, 12:2825-2830 2011.
[43] David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1150-1157. leee, 1999.
[44] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132-149. 2018
[45] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1)), 2008
13

---

Appendix
A Setting Hyper-parameters
Tuning hyperparameters without cross-validation on labels is particularly challenging for unsupervised learning methods. For example, STEGO [8] contains six parameters that are both dataset- and network-specific. Inspired by their hyperparameter tuning methods, we introduce a similar strategy for manual hyperparameter tuning. Recall that our SmooSeg introduces two hyperparameters and by in the smoothness loss, which are used to adjust the threshold for applying the penalty. Ideally, a segmentation model should promote smoothness within segments while maintaining discontinuity between different segments. To achieve this, we can monitor the distribution of ö, referred to as the smoothness degree, during the model training (see Fig. 8
balanced setting, the smoothness degree distribution should exhibit bimodality, as demonstrated in the first column of Fig. g Otherwise, the segmentation model will generate semantic maps that are either too smooth or too discontinuous. Specifically, The hyperparameters used in SmooSeg with DINO as the backbone are summaried in [5
Balanced
Too smooth
Too discontinuous
mluu
19
mlou
7.2
2.5c
Performance
step
step
step
1000
2000
2000
5
Smoothness degree distribution
Figure Distributions of the label penalty d 1-cos(A, (A, (A, A)) within an image and their corresponding performance under three different parameter settings. If the distribution of tends toward single- modal with peaks at 0.0 (or 1.0), the semantic maps will be overly smooth (or discontinuous).
Table 5: Hyperparameters used in SmooSeg.
Parameter COCOStuff Cityscapes Potsdam-3
0.5
0.5
0.5
b2
-0.02
-0.02
0.1
B The impact of CRF
CRF postprocessing is a common practice in both supervised and unsupervised semantic segmentation g and that the use of CRF does not overshadow the contribution of the smoothness term in our work. The smoothness prior in this work performs on the high-level feature maps and mainly contributes to semantic smoothness, while CRF operates on pixels to refine the fine details and remedy the resolution loss caused by the final upsample operation that exists in most semantic segmentation models (a normal upsample rate 8x8). Therefore, the application of a CRF serves as a supplement to our smoothness prior to further refine low-level smoothness.
14

---

Table 6: Experimental results of the impact of CRF SmooSeg and STEGO.
COCOStuff
Cityscapes
Potsdam-3
|
Avg.
Acc
mloU
Acc
mloU
Acc
mloU
Acc
mloU
STEGO w/o CRF
46.5
22.4
63.5
16.8
74.1
58.9
61.4
32.7
STEGO CRF
48.3
24.5
69.8
17.6
77.0
62.6
65.0 (+3.6)
34.9 (+2.2)
SmooSeg w/o CRF
60.6
25.2
79.8
18.0
81.4
68.4
73.9
37.2
SmooSeg CRF
63.2
26.7
82.8
18.4
82.7
70.3
76.2 (+2.3)
38.5 (+1.3)
Table demonstrates that SmooSeg still achieves state-of-the-art without CRE. Overall, the perfor- mance degradation in STEGO is notably more pronounced compared to SmooSeg. Importantly, the application of a CRF serves an effective supplement our smoothness prior. Moreover, present qualitative visualizations with and without CRF in Figures 9 and 10) It is found that CRF able refine the quality fine details on both STEGO and SmooSeg. However, SmooSeg is consistently more semantically coherent than STEGO either with or without CRE
also
C Additional Qualitative Results
To provide further evaluation, we have included some difficult samples from the COCOStuff dataset that were predicted by our Smooseg and STEGO in Figure []] In these cases, Smooseg and STEGO tend to generate inaccurate semantic maps. However, it is noteworthy that even in challenging scenarios, SmooSeg consistently generates to STEGO. This observation underscores the advantages of incorporating our smoothness prior in semantic segmentation tasks.
semantically coherent segmentation maps compared
We provide the visualization of more results Fig. 12 for the COCOStuff dataset, Fig 13 for the Cityscapes dataset, Figs [5) for the Potsdam-3 datasets. Overall, SmooSeg consistently produces more coherent segmentation maps when compared to both STEGO and TransPGU. These visual results provide further evidence of the efficacy of the smoothness prior in enhancing the label coherence and the overall segmentation quality in unsupervised settings.
15

---

Image
Label
SmooSeg
SmooSeg w/o CRF
STEGO
STEGO
w/o CRF
Figure 9: Qualitative results of STEGO and SmooSeg with and without CRF on the COCOStuff dataset.
16

---

Image
Label
SmooSeg
SmooSeg
w/o w/oCH CRF
STEGO
STEGO
w/o CRF
Figure 10: Qualitative results of STEGO and SmooSeg with and without CRF on the Cityscapes and
Potsdam-3 datasets.
Image
Label
SmooSeg
STEGO
Figure 11: Difficult examples predicted by SmooSeg and STEGO on the COCOStuff dataset.
17

---

Image
Label
Smooses
STEGO
TransFGU
Image
Label
SmooSeg
STEGO
TransFGU
Sky
Sports
Building
Ground
Plant
Water
Person
Animal
Solid
Vehicle
Textile
Structural
Floor
Window
Ceiling
Electronic
Appliance
Foo
Furniture
Indoor
Kitchen
Accessory
Rawmaterial
Outdoor
Wall
Figure 12: Additional qualitative results on the COCOStuff dataset. It's evident that SmooSeg produces higher-quality segmentation maps compared to STEGO and TransFGU.
18

---

Image
Label
SmooSeg
STEGO
TransFGU
Road
Traffic sign
Traffic light
Car
Sky
Trailer
Building
Train
Fence
Rider
Bicycle
Vegetation
Wall
Person
Parking
Terrain
Sidewalk
Ignore
Figure 13: Additional qualitative results on the Cityscapes dataset.
19

---

Image
Label
STEGO
SmooSeg
Buildings clutter
Roads + cars
Low vegetation trees
Figure 14: Additional qualitative results on the Porsdam-3 dataset. The large-scale image have resolution of 4800 4800, which will be partitioned into 225 sub-images with resolution of 320 x 320 before being input into each algorithm.
20

---

Image
Label
STEGO
SmooSeg
Buildings clutter
Roads + cars
Low vegetation + trees
Figure 15: Additional qualitative results on the Potsdam-3 dataset.
21


Reviews:
[{"review_id": "jDyqH1bUAi", "reviewer": "Reviewer_HQSk", "summary": "The paper tackles the difficult task of unsupervised semantic segmentation at the level of the scene (dense prediction). The authors exploit the piecewise coherence regarding the semantics, texture, or color that similar objects naturally have (termed as smoothness prior). The problem formulation raises some challenges the author solves by using a pre-trained (frozen) feature extractor to model the closeness relationships among observations and introduce a novel pairwise smoothness loss, and a teacher-student style predictor. Results are showcased on popular benchmarks: COCOSttuff, Cityscapes, and Potsdam, on which the authors report state-of-the-art performance.", "questions": "Please see weaknesses above.", "limitations": "The authors have not properly addressed the limitations of their proposed method. I highly encourage them to do so in a dedicated section in the main submission. Ideas for future work are a plus.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality - the problem formulation (energy minimization objective function) and the addition of the smoothness prior counts as novel work.\n2. Quality - the paper is decently structured, method and experimental analysis are sound and convincing. \n3. Clarity - some aspects of the paper could be improved. Some details are left out, making it be challenging to reproduce the results based on the information provided in the paper.\n4. Significance - the topic is indeed relevant to the research community. ", "weaknesses": "* L112 - The architecture description paragraph is tough to follow and the dedicated figure (Figure 1 in the main submission) is overcomplicated. Also, the use of the term \"prototypes\" for the teacher-student paradigm is confusing. Please consider rephrasing this part.\n* L62 vs. L312 - minor contradiction.\n* L307 - Type \"We\" - no capital letter\n* The main submission has an appendix document but there is no reference in the main submission regarding the contents of the supplementary material.\n* Important details are in the supplementary material and not mentioned in the current submission - such as the use of CRF for further refining the final segmentation maps.\n* In the experiments section there is no mention of how the authors produced the segmentation maps from SSL feature extractors such as ResNet50, MoCoV2, DINO (Table 1), DINO (Table 2), or DINO, DINOV2 (Table 3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work. \n\n### Architecture description is tough to follow, the dedicated figure is overcomplicated，and the term \"prototypes\" is confusing.\n\n1\\) We apologize for the confusion on the architecture description and the figure (we guess you are referring to Fig.2 instead of Fig.1 here?). To improve clarity, we will itemize main modules/steps in the architecture description. \n- Step 1: feature extraction: $X = f_{\\theta}(I)$, where $f_{\\theta}$ denotes the extractor, a pre-trained model.\n- Step 2: feature projection: $Z = h_{\\theta}(X)$, where $h_{\\theta}$ denotes the projector, a non-linear projection head.\n- Step 3: label prediction: $A^{s,t} = Z^TP^{s,t}$ in both teacher and student branches in the predictor.\n\nWe will also simplify Fig.2 by e.g., removing unnecessary annotations.\n\n2\\) We thank the reviewer for pointing out the confusion on \"prototypes\". We will clarify the term \"prototypes\" as \"class centers\" in the paper for better clarity.\n\n### Minor contradiction and typos\n\nWe greatly appreciate your careful reading and pointing out these errors. We will update the description in L312 to eliminate the inconsistency and correct any typos.\n\n### No reference regarding the supplementary material\n\nThank you for bringing this to our attention. We will make a reference for each Appendix in the corresponding section of the revised main paper as:\n- *L 215: “Implementation details can be found in Appendix A.”*\n- *L 249-251: “Additional qualitative results, along with color maps, can be found in Appendix C.”*\n- *L 304: “However, we present a feasible strategy in Appendix B to alleviate this issue.”*\n\n\n### The use of CRF not mentioned in the submission\n\nPlease see the general response.\n\n### Implementation detail of baselines\n\nWe will add the following details in the experiments section. \n\n*Results of ResNet50, MoCoV2, DINO (Table 1), DINO (Tables 1,2,3) are directly cited from the paper [8], while the results of DINOV2 (Table 3) are obtained by our implementation. For these baselines, we first extracted dense features for all images. We then utilized a minibatch k-means algorithm to perform patches grouping, which resulted in the final segmentation maps.*\n\n### Limitations:\nThanks for your suggestions. We will reorganize the materials and discuss more in the main manuscipt about the setting of hyper-parameters in the smoothness term. It is observed that the pre-calculation on the statistics for estimating the feature similarities could be a good prior in setting the hyper-parameters, which is still under investigation in our further study. "}, {"review_id": "8t4miokUSS", "reviewer": "Reviewer_1NSo", "summary": "The paper introduces a new approach called SmooSeg for unsupervised semantic segmentation, which aims to segment images into semantic groups without manual annotation. SmooSeg is based on the idea of smoothness: adjacent features in a metric space should share the same semantics. Specifically, it formulates unsupervised semantic segmentation as an energy minimization problem (Eq. 1,4). Training is performed using teacher-student style self-training.", "questions": "### Removing E_{data} in Table 4\n* For Table 4, I’m surprised that the method still works when the data term is removed in Table 4. In that experiment, are there any losses apart from the smoothness term? Without the data term, should there not be degenerate solutions?", "limitations": "Limitations are discussed adequately in the paper. One main limitation (how to set hyperparameters) is mentioned clearly at the end of the paper and addressed in the Appendix.", "rating": 5, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "### The paper is well-structured.\n* The introduction is well-written and motivates the problem well. The methods section breaks down the contribution into provides helpful preliminaries (S3.1) in a good amount of detail. It also describes the contribution (S3.2-3.4) precisely without unnecessary complications.\n\n### The visualizations provided are nicely done\n* The t-SNE plot in Figure 5 is quite illustrative, and the qualitative examples in Figure 3 are also helpful.\n\n### Algorithm 1 does a good job of communicating the proposed method.\n * It is very helpful to be able to refer to the PyTorch pseudocode alongside the equations in the paper.\n\n### The “Discussion with CRF and STEGO” section is very helpful\n* My natural first question upon reading the introduction and the beginning of the methods section was about the relationship between SmooSeg and CRFs. This section answered some of my questions, as it gave an analysis of their relationship.", "weaknesses": "### It would help to have further analysis of why the method works.\n* As discussed in the paper’s introduction, the idea of smoothness is image segmentation has a long history, having been explored since well before deep learning. This is especially true of energy minimization approaches, for which many different types of smoothness losses have been proposed over the years. Empirically, it seems that the performance of the smoothness loss proposed in this paper is good. However, I do not really understand _why_ it should be better than any other approach to enforcing image smoothness, such as the approach applied in CRFs. What would make this paper really useful to the community would be if it tried to understand why this particular formulation works well, so that the community can learn some generalizable lessons/insights (which could then be applied to other problem domains as well). \n* Is it something particular about the combination of your smoothness term and the student-teacher approach that works well, or is it each of them individually? \n* What would happen if you took STEGO (exactly as it is) and added your smoothness loss to their method? Would it be good, or does your smoothness loss work particularly well when combined with your student-teacher method? \n\n### Why do you also need to apply a CRF? \n* The supplement states: “We also use a CRF as the post-processing to refine the predicted semantic maps.” This confuses me. If you already have a smoothness loss (which is your contribution), why do you need a CRF? \n* How does your performance compare without a CRF? \n\n_Meta note on the score (because there is nowhere else to put this in the review)_: I am between 4 and 5. I gave a preliminary score of 4, but I am certainly willing to raise my score when the above questions/points are answered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work. Some of the concerns are responded in the \"General Response\" thread. Please kindly refer to the response regarding \"**why this method works**\" and \"**CRF**\". \n\n### Roles of the smoothness term and the student-teacher predictor\nWe apologize for any confusion caused by the lack of detailed information of our ablation study in the main submission. We will make it clear in our revision. The student-teacher predictor is specifically designed for the data term. The smoothness term requires a predictor but not necessarily a student-teacher one to generate the label map for label penalty. \nIn order to justify our choice of the student-teacher predictor and the effectiveness of the smoothness term, we remove the student branch in the variant of “w/o $E_{data}$” (Table 4) and the data term in the loss function. That is, we only keep the prototypes in the teacher branch to generate the label map for the smoothness term. The prototypes $P^t$ and the projector are then updated simultaneously by SGD from the smoothness term. Our ablation study shows that the smoothness term with a normal predictor has some performance downgrade (see w/o $E_{data}$ in Table 4). On the other hand, the student-teacher predictor alone, without considering any smoothness prior, could not obtain reasonable results (see w/o $E_{smooth}$ in Table 4). Combining the smoothness term and the student-teacher predictor leads to the best performance.\n\n### Add smoothness loss to STEGO\nThe smoothness loss in our work is similar in form to the correlation loss in STEGO. However, the main difference is that the correlation loss aims to perform feature correspondence, while our smoothness loss works on the label penalty and requires a predictor to generate the label map. From this point, there is no straightforward way to add our smoothness loss to STEGO.\n\n### (Question) Removing $E_{data}$ in Table 4\nIn Table 4, we only used the smoothness term as the sole loss when we removed the data term. The smoothness term utilizes a closeness matrix constructed from high-level features of a pre-trained model, acting as strong supervision signals to guide label learning for all image patches. Therefore, it is reasonable to see that the smoothness term contributes significantly to the overall performance. On the contrary, the data term operates in a self-training fashion with pseudo labels derived from the teacher branch, which alone cannot generate accurate segmentation maps, as evidenced by the extremely poor results of \"w/o $E_{smooth}$\" in Table 4. When combined with $E_{smooth}$, which provides enhanced pseudo labels, the data term aiming to minimize the entropy of the predicted segmentation maps, yields performance improvements. We will update the descriptions in the ablation study to provide better clarity."}, {"review_id": "xcENCLELdF", "reviewer": "Reviewer_gyDm", "summary": "This work addresses the problem of unsupervised semantic segmentation. In contrast to STEGO (baseline), the approach learns semantic embeddings directly in a student-teacher regime without the need for the K-means. The objective is entirely unsupervised and is reminiscent of a CRF energy formulation with data and a smoothness term. The pseudo labels from the teacher embeddings provide the signal for the data term. The work achieves improved segmentation accuracy on standard benchmarks over the state of the art and offers an interesting analysis. Overall, the work is compelling in quality and contribution.", "questions": "- l. 140-141: Why is optimisation only stable with such normalisation?\n- Eq. 5: Why is there a need to stop the gradient flow in this fashion?\n- l. 212: I would be curious to understand how the predictions and the ground truth are aligned in a bit more detail.\n- How were the other parameters (e.g. the momentum, the temperature) chosen/fine-tuned?\n- I do not quite follow the sentence in l. 172: “which represents discontinuities between image patches that should be preserved.”", "limitations": "The limitations are discussed in the main text. The supp. material provides further analysis.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The application of the smoothness constraint is natural and simple.\n- The learning problem is well-designed and executed. The text is generally well-written (albeit not without minor typos).\n- The experiments have sufficient scope and provide valuable insights. The empirical results are compelling.", "weaknesses": "There are technical similarities to STEGO, such as in the loss formulation in Eq. (4). There are differences, though, as discussed in 158-174. However, it does take away a bit from the novelty.\n\nEmpirically, the approach appears at its best with DINO features using ViT architecture; the improvement over prior art may not translate to other SSL representations and model architectures (MoCo, ResNet, see Tab. 1). Nevertheless, the approach remains competitive.\n\nThe method introduces a number of hyperparameters, which can be challenging to fine-tune in an unsupervised setup. I like that this is acknowledged in the work, however, and Appendix B provides some interesting analysis. \n\nTypos:\ne.g. l. 178, 307", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Technical similarities to STEGO\nWe acknowledge some technical similarities between our smoothness loss and the correlation loss in STEGO. These loss formulations are not entirely novel and have been seen in various dimensionality reduction methods (e.g., [a]). Essentially, STEGO boils down to an unsupervised dimensionality reduction method [b], followed by a kmeans grouping of learned embeddings for patches. In contrast, our method goes beyond dimensionality reduction and utilizes the smoothness prior to facilitate the learning of the labeling function, which encourages piecewise smoothness and leads to more coherent and semantically meaningful semnentation maps. Additionally, our smoothness loss directly constrains the label assignment, which brings a desirable property when compared to the correlation loss that operates on the reduced feature correspondence.\n\n*[a] He, Xiaofei, and Partha Niyogi. \"Locality preserving projections.\" Advances in neural information processing systems 16 (2003).*\n\n*[b] Koenig, Alexander, Maximilian Schambach, and Johannes Otterbach. \"Uncovering the Inner Workings of STEGO for Safe Unsupervised Semantic Segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3788-3797. 2023.*\n\n\n### Backbone choice\nDINO features using ViT architecture indeed yield significant benefits for downstream dense prediction tasks when compared to other SSL representations. This is evident from its wide adoption in the literature, including STEGO, TransFGU, and Deep spectral methods [12].\n\n### Hyper-parameter choice\nFine-tuning hyper-parameters is a common requirement in state-of-the-art unsupervised methods, including STEGO and TransFGU. Therefore, we strive to find a practical solution to address this challenge, which leads to the analysis in Appendix B.\n\n### Why is optimisation only stable with such normalisation\nBy treating the closeness matrix as a weight matrix between nodes in a graph, the zero-mean normalization balances the negative and positive forces during optimization. This balance ensures that the optimization process is more stable, preventing excessive influence from either the negative or positive components of the closeness matrix. Consequently, this normalization contributes to a smoother and more consistent learning process. We will clarify this in the revision.\n\n\n### Why is there a need to stop the gradient flow in this fashion\nThe stop-gradient operation in Eq. 5 is an essential step in our asymmetric student-teacher style predictor. In our approach, there is no observed semantic map available for the data term, so we adopt a self-training strategy to minimize the data loss. The self-training relies on the teacher branch to generate enhanced pseudo labels, which are then used to supervise the learning of the student prototypes. To ensure stability and prevent rapid updates during each training batch, the stop-gradient operation is necessary. On the other hand, we stop the gradient flow from the data loss to the projector’s parameters. This decoupling mechanism for the projector and the predictor will make the learning easier and more stable, as we can avoid the need to trade-off between these two losses during model training. \n\n### how the predictions and the ground truth are aligned\nThe Hungarian matching algorithm is widely used to align the predictions and the ground truth in the unsupervised setting. The objective is to find an assignment function that maps each predicted class to a ground truth class with the minimum total cost. One needs to first compute a cost matrix with each entry indicating the cost of an assignment and uses the Hungarian matching algorithm to find the best assignment function. In practice, we utilize the linear_sum_assignment function from the scipy package to calculate the assignment. This function takes the cost matrix as input and outputs a list of pairs, such as $[(m_0, n_0), (m_1, n_1), ...]$, where each predicted class $m_i$ is aligned with a ground truth class $n_i$. \n\n### How were the other parameters chosen/fine-tuned\nWe have conducted experiments to evaluate the sensitivity of the momentum parameter and the temperature parameter. Empirically, we found that using a large momentum parameter and a small temperature parameter consistently leads to good results on all datasets. Throughout the experiment, we set these two parameters as follows: momentum parameter $\\tau = 0.1$ and temperature parameter $\\alpha = 0.998$, unless stated otherwise.\n\n### Explanation of “which represents discontinuities between image patches that should be preserved.”\nWe apologize for any confusion caused. In STEGO, a correlation tensor $S$ (with each entry within $[-1, 1]$ when using cosine similarity) is used to measure the similarities between two image patches. A large value of $S_{ij}$ indicates that patches $i$ and $j$ have a high similarity and should belong to the same class. Conversely, a small value of $S_{ij}$, especially when $S_{ij} < 0$, indicates a low similarity between patches $i$ and $j$, suggesting they may be located at the boundary of segments and tend to belong to different classes. We refer to these differences between patches as \"discontinuities\". Under the smoothness assumption, an effective model should not only encourage piecewise smoothness but also maintain the discontinuities between image segments. Preserving these discontinuities is crucial to avoid trivial solutions. However, in STEGO, the negative part of the correlation tensor $S$ is discarded using a 0-clamp, potentially overlooking the significance of these discontinuities. In contrast, our label penalty function $\\delta(·, ·)$, which satisfies $0 ≤ \\delta(·, ·) ≤ 1$, possesses a desirable property compared to $S$. It allows us to properly handle and preserve these crucial discontinuities in the smoothness term, contributing to improved performance and meaningful segmentation maps in our approach."}, {"review_id": "64vsbgGsXo", "reviewer": "Reviewer_iJZU", "summary": "The paper presents an unsupervised training method for semantic segmentation models. It trains the network on a loss with two terms: one encouraging smoothness in the segmentation labels, and a \"data\" term based on self-training. It assumes a neural net architecture with a) an extractor that produces per-pixel features $X$, b) a projector that produces lower-dimensional features, to be compared against \"prototypes\" and used in self-training, and c) the \"predictor\" that outputs the per-pixel segmentation labels from the prototype/feature comparisons. The backbone feature extractors are primarily based on DINO [11], though some experiments appear to use MoCo [24] in its place. This is evaluated in terms of its accuracy in semantic segmentation tasks including COCOStuff, Cityscapes, and Potsdam-3.", "questions": "Why the focus on semantic segmentation? If the labels aren't used, then I'd assume the method should equally apply to instance, panoptic, or salient-foreground segmentation. Especially given it does seem to generalize across different kinds of semantic segmentation. Is it because the prototypes $P$ are expected to somehow relate to semantic classes of \"stuff?\"\n\nThe stated goal of the smoothness prior is that \"the segmentation model is encouraged to assign similar labels to adjacent patches, thereby promoting spatial coherence within objects.\" What determines which pixels are \"adjacent\" when constructing E_smooth? I.e. is it 4-connected, 8-connected, or fully connected? The pairwise weights don't seem to include spatial distance. Toward the end of Section 3.1 it's stated that the authors \"also apply the smoothness prior across images.\" Which pairs of pixels across pixels are included as terms in the sum in Eq (1)?\n\nMinor comments:\n\n  * Typos in Algorithm 1 comments: \"updata\" and \"prototyeps\"\n  * Citation to [20] is repeated on line 94", "limitations": "The sensitivity of the results to hyperparameter choices is mentioned as a limitation. This does seem likely to be the major drawback, with bad results like mode collapse being a concern. In Appendix B of the supplement, it is described in qualitative terms that one can monitor the distribution of the differences in soft label assignments (the \"smoothness degree\") to adjust the parameters. They don't give a fully reproducible method or algorithm for determining b1 and b2 to achieve the desired $\\delta$ distribution, but it does seem clear what one would be looking for to tune this manually.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "i) Experiments consider variety of semantic segmentation tasks: street, general, and aerial image datasets.\n\nThis is a good demonstration that the method can be applied generally, even if it is sensitive to some choice of hyperparameters.\n\nii) Uses a backbone (DINO) also trained in an unsupervised (self-supervised) way.\n\nUses DINO ViT-S/8 [11] or MoCo [20] as backbones in different experiments on the segmentation head & training. These were also trained without hand labelling, wtih a self- or unsupervised method; as opposed to using, for example, a standard ImageNet backbone from supervised training. This is a correct and principled way to make sure the proposed segmentation models are truly and completely unsupervised.\n\niii) Precise presentation\n\nUses both an algorithm, written in plain code, along with diagrams and the math to describe the method quite completely. Those parts of the method that are unclear from the definition of the loss can be deduced quite unambiguously from Algorithm 1, and vice versa.", "weaknesses": "iv) Not a lot of experiments or analyses explaining *why* the method works.\n\nThe experiments mostly show end results (mAP, qualitative segmentation quality). Hyperparameter studies are useful, though: providing good insight into the role each hyperparameter plays. And there is a minimal ablation study in Table 4.\n\nIt still seems unclear what exactly the network is learning from. For example, to what extent is the smoothness a constraint on the extractor vs. the projector? It seems possible that $E_\\mathrm{smooth}$ is also penalizing the cosine distances, as defined in Equation (2), and pushing feature vectors $X$ together, and not just using it as a weight for spatial smoothness in the labels $Y$, as in the intuitive explanation of Eq (1). I don't see `torch.no_grad` or any equivalent around the weight computations in Algorithm 1. Fig 5 visualizes feature embeddings on Potsdam-3, and this shows that the group similar classes together. Though, if the mapping between feature vectors $X$ and the labels $Y$, in the projector and predictor, is a \"smooth\" function (in the sense of Lipschitz smoothness) so that similar features get similar labels, might we already expect that the smoothness penalty is reduced, even if there is no meaningful spatial smoothness of the labels?\n\nThe relative role/contribution of the within-image and between-image comparisons also seems to complicate the intuitive explanation of the method. There is a large accuracy drop if $E_\\mathrm{smooth}$ only penalizes adjacent pixels within the image, and not across the images. The motivation for $E_\\mathrm{smooth}$ that adjacent pixels are likely to have the same label, the \"natural tendency towards piecewise coherence regarding semantics, texture, or color,\" does not apply to pixels within different images. The smoothness between pixels that are adjacent in a metric space is a property of the construction of that metric space: not a natural property of images, so this part's role in the function of the smoothness perhaps needs more explanation and experiments. Why do we expect with this unsupervised loss that the metric space will be semantically meaningful?\n\nv) Novelty is arguably more limited than is claimed.\n\nThe use, in unsupervised training of semantic segmentation models, of spatial smoothness of the labels does seem to be explored. Though many of the particulars and how it's used within the overall framework is new as of this submission.\n\nSmoothness priors themselves are, of course, quite common in semantic segmentation. Spectral methods also expected to be a form of smoothness prior, but using squared differences in place of the $\\delta$ in Eq (1) of the submission. This kind of prior is used in unsupervised training of semantic segmentation models in:\n\n[a] Xia & Kulis \"W-Net: A Deep Model for Fully Unsupervised Image Segmentation\"\n\n[b] Melas-Kyriazi et. al. \"Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization\"\n\nW-Net also incorporates a fully-connected CRF, in section 4.1, though this is only for preprocessing and not for unsupervised training. I however don't see prior work that uses a fully-connected CRF within the training objective in the way that this submission does. The backbone is also updated relative to previous work: this submission is perhaps in the genre of papers that applies a given vision idea/concept to ViTs instead of CNNs.\n\nvi) There may also be some gaps in the citation of prior work similar to the submission's teacher/student/self-training approach described in section 3.2. Citations on self-supervised learning in the \"Related Work\" in section 2 focus on contrastive learning approaches. More similar approaches might include:\n\n[c] Scudder \"Probability of error of some adaptive pattern-recognition machines\"\n\n[d] Xi et. al. \"Self-training with Noisy Student improves ImageNet classification\"\n\nBased on my read, the submission does *not* modify the inputs or consider different views between teacher and student, as in [19, 20, 21, etc.]. (If my read is incorrect, what exactly is the \"pretext task\" that the authors use?) Though [c.d] and similar papers differ enough from the submission's training scheme, including the use of a projector module to get the prototypes that are compared between teacher & student, that this part is not relevant to judging novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work. \n\n### Why the method works\n**What the network is learning from.** Please refer to the general response.\n\n### The relative role/contribution of the within-image and between-image comparisons.\nWe apologize for any confusion caused. We clarify that our “adjacent” patches are defined in the high-level feature space generated by the frozen pre-trained model, rather than in the spatial coordinate space of images. That is, adjacent patches in our model refer to patches with similar extracted features: they may or may not lie closely in a single image and may belong to different images. Your insight regarding the performance drop when penalizing only adjacent pixels within an image aligns with our observations. The rationale behind this phenomenon lies in two folds: 1) the smoothness term across the images could provide the negative force that prevents the model collapse; 2) the smoothness term across the images helps construct a global and semantically meaningful distribution across the entirely dataset. While it is intuitive that similar patches within the same image should share the same label, the smoothness term across images extends this concept to all patches with the same semantics across all images. This promotes the creation of coherent and meaningful semantic clusters that transcend individual images, as shown in Figure 5 in the main submission.\n\nWe argue that the concept of smoothness between image patches in a metric space is reflective of the natural properties of images. This assertion is grounded on the fact that a pre-trained model can extract semantically consistent features from images. For instance, patches belonging to a common object like a dog would naturally cluster together both in the pixel space and in the metric space when provided with semantically consistent features. This highlights the reason behind our decision to construct the closeness matrices in the metric space, as opposed to the spatial coordinate space.\n\n### Difference with W-Net and Deep Spectral Methods\nWe deeply appreciate the reviewer's kind provision of references and recognition of the originality inherent in our overall framework. While acknowledging the utilization of a fully-connected CRF in W-Net for preprocessing, it's crucial to emphasize that our submission distinguishes itself by incorporating a smoothness loss within the context of unsupervised training, rather than being limited to preprocessing or postprocessing stages. Furthermore, we'd like to clarify that Deep Spectral Methods do not encompass model training; they utilize spectral decomposition for object or foreground segmentation. It's worth noting that Deep Spectral Methods lack the capability of distinguishing the \"stuff\" category. Moreover, we concur with your observation regarding the updated backbone: our method aligns particularly well with ViTs structure.\n\n### Missing citation of prior work\nWe sincerely appreciate the reviewer for pointing out the missing citations. In our revision, we will ensure to include these relevant works and provide a more comprehensive discussion on similar approaches. We also appreciate the reviewer's accurate understanding of our work: SmooSeg does not modify the inputs or consider different views between teacher and student.\n\n### Why focusing on semantic segmentation\nThe focus on semantic segmentation in SmooSeg is mainly due to the way prototypes are learned. In SmooSeg, the prototypes are automatically learned from image patches without any direct constraints, making it challenging to differentiate between foreground and background prototypes or distinguish different instances.\n\nIndeed, smoothness has the potential to be incorporated into various types of semantic segmentation as a smoothness regularization term. However, for tasks requiring finer distinctions, such as instance segmentation or salient foreground segmentation, additional modifications and research may be needed to address the specific challenges associated with those tasks. Its extension to other types of segmentation tasks is an interesting direction for future research.\n\n### The construction of closeness matrices $W^{i,i}$ and $W^{i,i^{\\prime}}$\nWhile early smoothness-prior-based methods often define adjacent pixels for a given pixel based on the 4-connected or 8-connected grid in the coordinate space, we argue that this approach neglects the high-level semantic information of image patches. In contrast, SmooSeg defines the closeness relationship among image patches in a metric space (the feature space of self-supervised representation learning methods) rather than in the coordinate space, by calculating the cosine similarity of high-level features. The resulting closeness matrix is fully connected, representing the adjacency between all image patch pairs. Theoretically, a large element value in the closeness matrix, indicating a high cosine similarity, suggests a high possibility of \"adjacent\" patch pair, and vice versa. From this perspective, when minimizing the smoothness loss, SmooSeg encourages two patches to have similar labels if their relationship in the closeness matrix is positive, and vice versa. SmooSeg enables the utilization of high-level semantic information to guide the smoothness regularization, leading to improved segmentation results that account for semantic coherence and consistency between image patches.\n\nWe apologize for the confusion on the smoothness prior across images. In our smoothness term given by Eq (4), there are two closeness matrices involved—one for within an image and another for across images. Specifically, the closeness matrix across two different images $I_i$ and $I_{i^{\\prime}}$ is computed as the cosine similarities of all across-image patch pairs:\n$W_{pq}^{ii^{\\prime}}=\\frac{X_{i,p} \\cdot X_{i^{\\prime}, q}}{||X_{i, p}|| \\ ||X_{i^{\\prime}, q}||}.$\n\n### Typos and repeated citation\nWe will proofread the paper again and correct all the typos."}, {"review_id": "ezBLdyvty9", "reviewer": "Reviewer_yftt", "summary": "The paper tackles unsupervised semantic segmentation which aims to group pixels into semantic clusters without manual annotation. This paper proposes the smoothness prior by enforcing that adjacent features in metric space share the same semantics. The approach relies on features from a self-supervised representation learning method (ie., DINO). Additionally, the architecture relies on an asymmetric teacher-student style predictor that updates pseudo labels, leading to smoother segmentation predictions. This approach outperforms STEGO (prior s-o-t-a) in terms of pixel accuracy on three datasets: COCOStuff, Cityscapes, and Potsdam with a ViT-S backbone.", "questions": "My most important questions are listed above. \n\nI also have a few additional questions:\n- L265 claims that the proposed method outperforms STEGO at the boundaries. However, are the boundaries not primarily dependent on the CRF? How do these methods compare when we don’t use a CRF? It would also be useful to see a few examples without a CRF in the supplementary (optional).\n- How is the best model selected in the absence of a validation set? Can the loss function be used to select the best model during training?\n- The method is dependent on the pretrained DINO weights. We see that MoCov2 performs worse in Table 1. What happens if we use other weights (MoCov3, Pixpro, DenseCL etc.)? \n- Are the visualized predictions in the supplementary randomly selected? \n", "limitations": "While some of the limitations are mentioned in the paper (see Section 4), I suggest including other limitations (e.g., known CRF weights, and known number of classes). I also couldn't find a paragraph on the societal impact. ", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### Method:\n- Leverages advantages in self-supervised learning (SSL): The proposed method leverages the advancements in self-supervised representation learning in generating dense representations. Notably, it proposes to model the relationships between image patches by relying on high-level features from a pre-trained DINO model.\n- Kmeans is not required: The introduced smoothness prior can be used to directly optimize the generated segmentation map. The paper claims this strategy leads to more coherent and accurate segmentation results than prior work STEGO. In particular, the latter relies on minibatch Kmeans to obtain the final segmentation map.\n- Learnable prior: The approach does not rely on grouping priors as explored in prior works (see below for references). Instead, the approach implicitly uses a learnable grouping prior by relying on the self-supervised representation learning method (i.e., DINO). \n\n### Experiments: \n- Dataset variety: The approach is applicable to different datasets. In particular, the experimental validation considers 3 different datasets: COCOstuff, Cityscapes, and Potsdam. \n- Quantitative results: the approach outperforms prior art (STEGO) on pixel accuracy when using DINO pre-trained weight on Citysapes and COCOstuff dataset.\n\n### Misc.\n- Clarity: The method is well presented by relying on clear figures (e.g., Figure 2). The paper is also easy to follow (for someone who’s familiar with the literature at least).\n", "weaknesses": "### 1. Originality Method - Difference with STEGO / SlotCon:\n\n- The paper can be seen as a combination of STEGO and SlotCon.\nThe method largely follows the methodology of STEGO. The main SmooSeg loss is similar to the correlation loss in STEGO, which is also mentioned in the paper (see L158).  Overall, I don’t think this is a major issue as I haven’t seen this combination being applied to the task of unsupervised semantic segmentation directly.\n\n- (Minor) I also don’t completely follow the reasoning that the proposed method is superior to Kmeans (L165). While this is indeed an advantage, it’s not clear why Kmeans would necessarily perform worse, as used in STEGO (both STEGO and SmooSeg rely on the features from DINO anyway). Kmeans also has to rely on a distance metric (e.g., cosine distance between features), resulting in similar features/patches being assigned to the same cluster. \n\n\n### 2. Robustness -  setting hyperparameters:\nThere are many parameters that require supervision during training:\n- CRF weights: The weights of the CRF are determined using the annotated validation set. How can these be determined without annotations? It’s not clear from the supplementary if the weights are dataset-specific. (I assume that https://github.com/lucasb-eyer/pydensecrf was used.)\n- Number of classes: The number of classes is not always known a priori. In particular, it’s not clear how much the approach relies on this information. As a result, the introduced setup (section 3) is relatively artificial. In most practical applications, this information won’t be available. As a result, there are important experiments missing, where the sensitivity to this parameter is ablated (ideally comparing the mean and variance over multiple runs with other methods (e.g., STEGO, PiCIE). \n- Dataset-specific parameters: Overall, there are many training parameters that differ across datasets and require finetuning (i.e., loss parameters). Ideally, the same set of parameters is applicable to new/unseen images, especially for an ‘unsupervised’ method. Now, it’s hard to judge the robustness of the approach as it relies on finetuned parameters for each dataset. To my knowledge, this is not the case for prior works and baselines (IIC, PiCIE, DeepCluster). Also, TransFGU keeps most of its parameters constant. While this issue is somewhat tackled in the supplementary (see Section B), it’s still not clear if this strategy was actually used to set these parameters during training.\n\n### 3. Scope of the experiments:\n- The experiments are limited to results on 27 classes  (COCOStuff and Cityscapes). TransFGU also includes COCO-80 and COCOStuff-171. It would be interesting to see the performance for these setups as well.\n- A more practical setting is the semi-supervised setup, especially as the ‘unsupervised’ results are relatively poor and not immediately useful for practical applications. Does the method improve the representations by finetuning the learned representation compared to DINO? Finetuning the complete model or simply training a linear probe on top of the model are interesting experiments [f]. I would also expect that the learned representation can be efficiently fine-tuned with only a few samples [b].\n\n### 4. Quantitative results - different backbones:\nI noticed that only a ViT-S is used for COCO-stuff and Cityscapes, while a ViT-B is used for the Potsdam dataset. However, STEGO reports its final results with a ViT-B and outperforms the numbers in this paper (i.e., STEGO obtains 28mIoU on COCOstuff with a ViT-B). So, why is a ViT-B not used for COCO-stuff and Cityscapes as well, to make sure that the observations transfer to stronger backbones? This would furthermore make the main claim of the paper stronger.\n\n### 5. Related work\nThere are also a few related works missing. Interestingly, some of these rely on priors, such as superpixels, edge, or saliency estimators [d, e, f], which the paper under review does not require. More suggestions can be found below:\n\n[a] Wang et al, Dense Contrastive Learning for Self-Supervised Visual Pre-Training, CVPR.\n\n[b] Wang et al, FreeSOLO: Learning to Segment Objects without Annotations, CVPR.\n\n[c] Ziegler et al, Self-Supervised Learning of Object Parts for Semantic Segmentation CVPR.\n\n[d] Hwang et al, Segsort: Segmentation by discriminative sorting of segments CVPR.\n\n[e] VanGansbeke et. al, Unsupervised Semantic Segmentation by Contrasting object mask proposals, ICCV.\n\n[f] Zhang et al., Self-Supervised Visual Representation Learning from Hierarchical Grouping, NeurIPS.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work.\n\n### Why SmooSeg is superior to Kmeans used in STEGO\nThe performance of Kmeans heavily relies on the quality and discriminative power of the features it operates on. In the case of DINO features, as shown in the 2nd column of Figure 5, the Kmeans centers do not possess much semantic meaning as they almost stand at the same point. This phenomenon indicates that the DINO features lack sufficiently discriminative information to learn distinct semantic centers through Kmeans. Additionally, STEGO aims to distill DINO features into a more compact and structured representation using a distillation loss, making the class centers of Kmeans in this learned embedding space more semantically meaningful (3rd column of Figure 5). However, it is worth noting that although the DINO features and STEGO embeddings may exhibit many compact sub-clusters, they are not continuous or coherent in the semantic sense. In contrast, our SmooSeg, empowered by the smoothness prior, can produce highly semantically compact and coherent semantic clusters, as evidenced by the last column of Figure 5.\n\n### Setting hyperparameters\n- **CRF weights:**\nPlease see the general response.\n\n- **Number of classes:**\nPredefining the number of classes appears to be a common practice in existing studies that address the problem of unsupervised semantic segmentation. PiCIE adopts the number of classes as the number of cluster centroids in its algorithm; TransFGU adopts this number as the number of class embeddings in its Segmenter module; and STEGO requires a predefined number of classes when performing Kmeans. We follow this problem setting in our work for fair comparison but agree that this deserves further studies.\n\n- **Dataset-specific parameters:**\nWe agree that tuning parameters without a validation set can be challenging for unsupervised methods. While some prior works (e.g., IIC and PiCIE) do not explicitly have dataset-specific parameters, they generally perform significantly worse than those with (e.g., TransFGU and STEGO). In view of the parameter setting challenge, we strive to find a practical solution, which leads to the analysis in Appendix B. Through experimentation, we found that stable results can be achieved when b1 is set at 0.5 and b2 is nearby 0.0, which may to some extent enhance the practicality and robustness of SmooSeg in real-world applications.\n\n### Scope of the experiments\n- **Additional experiment**\nThanks for your advice. COCOStuff-171 and COCOStuff-80 require much more computations and we are not able to obtain their results before the rebuttal deadline. We will add their results in the revision.\n\n- **Semi-supervised setup and few-shot setup**\nWe appreciate the reviewer for providing these constructive suggestions. As can be observed in Figure 5, the embedding of SmooSeg exhibits significantly higher semantic consistency compared to the features of DINO, which confirms that SmooSeg is able to enhance representations over DINO. We are also curious if the representations learnt by SmooSeg can be further fine-tuned with a few samples. As the current work focuses on the unsupervised setting, we would like to reserve this exploration to future studies.\n\n### The choice of ViT-S\nThe choice of using ViT-S as the backbone on the COCO-stuff and Cityscapes datasets was made to ensure a fair comparison with both STEGO and TransFGU, as TransFGU only adopts ViT-S as its backbone. Accordingly, the results of TransFGU are directly cited from their original paper.\n\n### Missing references\nWe appreciate the reviewer for providing these valuable references. In our revision, we will include a thorough discussion with these literatures to highlight the distinctions in approaches.\n\n### The performance and examples without CRF\nPlease see the general response.\n\n### Select the best model without a validation set\nIn practice, the loss from the pretext task may not always be a reliable indicator of the model's performance on actual downstream tasks. Basically, it is impossible to select the “best” model, but there are some potential approaches to choose a good model without a validation set:\n- Setting a global training step and selecting the last saved mode. As the training progresses, the model tends to become more stable, and the performance may also stabilize.  In our experiments, we have found that the checkpoint at the end of training often shows good results.\n- Making visualizations of segmentation maps for some training samples during training. By comparing these results, one can qualitatively assess the performance of the model and select a model that shows promising segmentation results.\n\n### The performance with different backbones\nFollowing you suggestion, we further conduct experiments on COCOStuff by using Pixpro and DenseCL weights for SmooSeg.\n\n| Methods            | Acc  | mIoU |\n| --------           | ---- | ---- |\n| SmooSeg + DINO     | 63.2 | 26.7 |\n| SmooSeg + MoCoV2   | 52.4 | 18.8 |\n| SmooSeg + pixpro   | 48.3 | 18.1 |\n| SmooSeg + DenseCL  | 54.6 | 19.2 |\n\nDINO features indeed yield the best performance of SmooSeg when compared to its partnership with other SSL representations. Its superority on dense prediction tasks is also evident from its widespread adoption across the literature (in STEGO, TransFGU, and Deep spectral methods [12]). On the other hand, the ResNet architectures employed in models like Pixpro, DenseCL, and MoCov2 have certain limitations in generating low-resolution coarse features, rendering its challenges used for fine-grained semantic map prediction, especially under unsupervised setup.\n\n### How to select the visualized predictions\nTo ensure a comprehensive representation of the model's performance, we manually selected a diverse and representative set of images for visualization. To provide further evaluation, we have included some difficult samples predicted by our SmooSeg in the attached PDF file."}]

Author rebuttal:
# General Response
We appreciate all reviewers for the highly constructive comments that help improve the paper quality. We also thank the reviewers for the recognition of our work being well structured (**1NSo**, **gyDm**) and well presented (**iJZU**, **yftt**), the proposed solution being simple (**gyDm**), novel (**HQSk**), experimentally convincing (**HQSk**, **R6c1n**, **RceoA**), insightful (**HQSk**), of sufficient scope (**gyDm**), and covering various datasets (**yftt**). 

*We respond to some general comments as follows. We hope our rebuttal address the reviewers' questions and concerns. We would be more than happy to discuss with all reviewers if they still have any unresolved concerns or additional questions about the paper or our rebuttal.*

## Why our method works (@1NSo, iJZU) 
**@1NSo** **The difference between our method and previous smoothness-based methods.** Thank you for the thoughtful feedback. As mentioned in the introduction, one of the key challenges in applying smoothness priors to unsupervised semantic segmentation is to define a good closeness relationship among image patches. The main difference of our proposed smoothness prior from those defined in other methods lies in the definition of “adjacent” patches/pixels. Early image segmentation methods, as well as CRF models, define adjacency in the coordinate space, e.g., using a 4-connected or 8-connected grid in the coordinate space to define the adjacent pixels of a given pixel. These methods primarily relied on low-level appearance information and fell short in capturing high-level semantic information in images. Consequently, these approaches alone may not yield good results for high-level semantic segmentation tasks, as it may struggle to effectively capture the complex relationship of images patches necessary for semantic understanding. Our smoothness prior, on the contrary, imposes the closeness in the high-level feature space generated by self-supervised representation learning methods. This definition better suits the task of semantic segmentation as it better captures high-level semantic similarity. Such a closeness design provides strong supervision signals to guide the label learning effectively. Through our ablation study, we demonstrate that the proposed smoothness prior alone (w/o $E_{data}$ in Table 4) can still achieve promising results. We remark that the design of “across images” smoothness term ($E_{smooth}^{across}$) further extends the definition of "smoothness" from local patches to cross-image levels and is also novel and effective (see the ablation in Table 4). We will add these discussions in the revision.


**@iJZU** **What the network is learning from.** (1) We apologize for not explicitly mentioning the use of `torch.no_grad` in Algorithm 1. We will include this missing detail in the revision. We did, however, state that we utilize a frozen pre-trained model as an extractor. (2) Our core learning mechanism revolves around employing the closeness matrices (by Eq (2)) as smoothness signals to facilitate the projector and predictor learning. These closeness matrices act as strong supervision cues to guide the label learning (projector + predictor). (3) In relation to Fig 5, your understanding aligns accurately. After learning the smooth labeling function, similar features are already assigned similar labels, and the smoothness penalty is reduced.

## CRF

- **Why need to apply a CRF (@1NSo):** We would like to emphasize that CRF postprocessing is a common practice in both supervised and unsupervised semantic segmentation (line 23, p6 in [8]), and that the use of CRF does not overshadow the contribution of the smoothness term in our work. The smoothness prior in this work performs on the high-level feature maps and mainly contributes to semantic smoothness, while CRF operates on pixels to refine the fine details and remedy the resolution loss caused by the final upsample operation that exists in most semantic segmentation models (a normal upsample rate is 8x8). Therefore, the application of a CRF serves as a supplement to our smoothness prior to further refine low-level smoothness. 

- **The ablation of CRF (@1NSo, @yftt)**. The following table demonstrates that SmooSeg still achieves state-of-the-art without CRF, as CRF only accounts for $1.3$ mIoU out of the $2.2$ mIoU performance gain over STEGO. Moreover, we also present qualitative visualizations with and without CRF in the appended PDF file. It is found that CRF is able to refine the quality of fine details on both STEGO and SmooSeg. However, SmooSeg is consistently more semantically coherent than STEGO either with or without CRF. 

    |                 | COCOStuff27 | Cityscapes  |  Potsdam-3  |    Avg.     |
    | --------------- |:-----------:|:-----------:|:-----------:|:-----------:|
    |                 | Acc / mIoU  | Acc / mIoU  | Acc / mIoU  | Acc / mIoU  |
    | STEGO w/o CRF   | 46.5 / 22.4 | 63.5 / 16.8 | 74.1 / 58.9 | 61.4 / 32.7 |
    | STEGO           | 48.3 / 24.5 | 69.8 / 17.6 | 77.0 / 62.6 | 65.0 / 34.9 |
    | SmooSeg w/o CRF | 60.6 / 25.2 | 79.8 / 18.0 | 81.4 / 68.4 | 73.9 / 37.2 |
    | SmooSeg         | 63.2 / 26.7 | 82.8 / 18.4 | 82.7 / 70.3 | 76.2 / 38.5 |
    
- **The use of CRF not mentioned in the main submission (@HQSk):** As previous studies (e.g., STEGO) reported their performance with CRF by default, we compare the performance with CRF applied during test by default as well. Thanks for pointing it out, and we will move the postprocessing details from the appendix to the main manuscript in the revised version.
- **CRF weight (@yftt)**. We utilized *pydensecrf* for our CRF refinement, with all parameters set to default values to make a fair comparison with previous studies.

Meta-review: While the initial reviews were mixed, the rebuttal and discussion cleared up the majority of the remaining concerns and extensively discussed STEGO, which was one of the main remaining concerns. Overall, while the paper reuses some ideas from STEGO (dimensionality reduction + clustering) it adds a smoothness constraint which shows a substantial performance improvement. It is thus of interest to the community and should be accepted.
Decision: Accept (poster)