Title: SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: unsupervised semantic segmentation; self-supervised learning; smoothness prior
Abstract: Unsupervised semantic segmentation is a challenging task that segments images into semantic groups without manual annotation. Prior works have primarily focused on leveraging prior knowledge of semantic consistency or priori concepts from self-supervised learning methods, which often overlook the coherence property of image segments. In this paper, we demonstrate that the smoothness prior, asserting that close features in a metric space share the same semantics, can significantly simplify segmentation by casting unsupervised semantic segmentation as an energy minimization problem. Under this paradigm, we propose a novel approach called SmooSeg that harnesses self-supervised learning methods to model the closeness relationships among observations as smoothness signals. To effectively discover coherent semantic segments, we introduce a novel smoothness loss that promotes piecewise smoothness within segments while preserving discontinuities across different segments. Additionally, to further enhance segmentation quality, we design an asymmetric teacher-student style predictor that generates smoothly updated pseudo labels, facilitating an optimal fit between observations and labeling outputs. Thanks to the rich supervision cues of the smoothness prior, our SmooSeg significantly outperforms STEGO in terms of pixel accuracy on three datasets: COCOStuff (+14.9\%), Cityscapes (+13.0\%), and Potsdam-3 (+5.7\%).

SmooSeg: Smoothness Prior for Unsupervised Semantic Segmentation

Abstract

Unsupervised semantic segmentation is a challenging task that segments images into semantic groups without manual annotation. Prior works have primarily focused on leveraging prior knowledge of semantic consistency or priori concepts from self-supervised learning methods, which often overlook the coherence property of image segments. In this paper, we demonstrate that the smoothness prior, asserting that close features in a metric space share the same semantics, can significantly simplify segmentation by casting unsupervised semantic segmentation as an energy minimization problem. Under this paradigm, we propose a novel approach called SmooSeg that harnesses self-supervised learning methods to model the closeness relationships among observations as smoothness signals. To effectively discover coherent semantic segments, we introduce a novel smoothness loss that promotes piecewise smoothness within segments while preserving discontinuities across different segments. Additionally, to further enhance segmentation quality, we design an asymmetric teacher-student style predictor that generates smoothly updated pseudo labels, facilitating an optimal fit between observations and labeling outputs. Thanks to the rich supervision cues of the smoothness prior, our SmooSeg significantly outperforms STEGO in terms of pixel accuracy on three datasets: COCOStuff (+14.9%), Cityscapes (+13.0%), and Potsdam-3 (+5.7%).

1 Introduction

Semantic segmentation is a crucial task in computer vision that allows for a better understanding of the visual content and has numerous applications, including autonomous driving [1] and remote sensing imagery [2]. Despite advancements in the field, most traditional semantic segmentation models heavily rely on vast amounts of annotated data, which can be both arduous and costly to acquire. Consequently, unsupervised semantic segmentation [3; 4; 5; 6; 7; 8] has emerged as a promising alternative. Prior knowledge is fundamental to the success of unsupervised semantic segmentation models. One key prior knowledge is the principle of semantic consistency , which stipulates that an object's semantic label should remain consistent despite photometric or geometric transformations. Recent advances [3; 9; 4; 10] use contrastive learning to achieve consistent features or class assignments. Another essential prior knowledge is the priori concepts implicitly provided by self-supervised learning techniques, e.g. , DINO [11] and precedent arts [12; 8; 6] whose learned features can be employed to partition each image into different segments. Despite their effectiveness,

Image

Label

SmooSeg

STEGO

TransFGU

Sky

Sports

Plant

Water

Building

Ground

Person

Animal

Solid

Outdoor

Structural

these methods often overlook the coherence property of image segments, resulting in predicted segments that are incomplete and lacking in coherence, as shown in Fig. 1.

Real-world images often demonstrate a natural tendency towards piecewise coherence regarding semantics, texture, or color. Observations close to each other, either in the form of adjacent pixels in the coordinate space or close features in a metric space, are expected to share similar semantic labels, and vice versa. This essential property, known as the smoothness prior , plays a crucial role in various computer vision tasks [13; 14; 15]. Surprisingly, it is still under-explored in the field of unsupervised semantic segmentation.

In this paper, we attempt to tackle unsupervised semantic segmentation from the perspective of smoothness prior . As a dense prediction task, semantic segmentation aims at finding a labeling f ∈ F that assigns each observation (pixel, patch, features) p ∈ P a semantic category f ( p ) , which could be formulated within an energy minimization framework [16]: E ( f ) = E smooth ( f ) + E data ( f ) . E smooth is a pairwise smoothness term that promotes the coherence between observations, and E data represents a pointwise data term that measures how well f ( p ) fits the observation p . However, directly applying smoothness prior to unsupervised semantic segmentation faces several obstacles. 1) Due to the large intra-class variations in appearances within an image, it is difficult to define a well-suited similarity (dissimilarity) relationship among low-level observations. This makes it challenging to discover groups of complex observations as coherent segments. 2) E smooth can lead to a trivial solution where f becomes smooth everywhere, a phenomenon known as model collapse. 3) Optimizing E data without any observed label can be challenging.

In this study, we propose a novel approach called SmooSeg for unsupervised semantic segmentation to address the aforementioned challenges. By leveraging the advantages of self-supervised representation learning in generating dense discriminate representations for images, we propose to model the closeness relationships among observations by using high-level features extracted from a frozen pre-trained model. This helps capture the underlying smoothness signals among observations. Furthermore, we implement a novel pairwise smoothness loss that encourages piecewise smoothness within segments while preserving discontinuities across image segments to effectively discover various semantic groups. Finally, we design an asymmetric teacher-student style predictor, where the teacher predictor generates smooth pseudo labels to optimize the data term, facilitating a good fit between the observations and labeling outputs.

Specifically, our model comprises a frozen feature extractor, a lightweight projector, and a predictor. The projector serves to project the high-dimensional features onto a more compact, low-dimensional embedding space, and the predictor employs two sets of learnable prototypes to generate the final segmentation results. We optimize our model using a novel energy minimization objective function. Despite its simplicity, our method has demonstrated remarkable improvements over state-of-the-art approaches. In particular, our method significantly outperforms STEGO [8] in terms of pixel accuracy on three widely used segmentation benchmarks: COCOStuff ( +14.9% ), Cityscapes ( +13.0% ), and Potsdam-3 ( +5.7% ).

2 Related work

Unsupervised semantic segmentation has gained increasing attention for automatically partitioning images into semantically meaningful regions without any annotated data. Early CRF models [17; 18] incorporate smoothness terms that maximize label agreement between similar pixels. They define adjacency for a given pixel in the coordinate space, e.g., using 4-connected or 8 connected grid, which relies heavily on the low-level appearance information and falls short in capturing high-level semantic information in images. Recently, many methods [3; 4; 10] have attempted to learn semantic relationships at the pixel level with semantic consistency as a supervision signal. For example, IIC [3] is a clustering method that discovers clusters by maximizing mutual information between the class assignments of each pair of images. PiCIE [4] enforces semantic consistency between an image and its photometric and geometric augmented versions. HSG [10] achieves semantic and spatial consistency of grouping among multiple views of an image and from multiple levels of granularity. Recent advances [19; 12; 6; 8; 20] have benefited from self-supervised learning techniques, which provide priori concepts as supervision cues. For instance, InfoSeg [19] segments images by maximizing the mutual information between local pixel features and high-level class features obtained from a self-supervised learning model. The work in [12] directly employs spectral clustering on an affinity matrix constructed from the pre-trained features. TransFGU [6] generates pixel-wise pseudo labels by leveraging high-level semantic concepts discovered from DINO [11]. Additionally, STEGO [8] utilizes knowledge distillation to learn a compact representation from the features extracted from DINO based on a correspondence distillation loss, which also implies a smoothness regularization through the dimension reduction process. However, the utilization of smoothness prior in STEGO is implicit and entails separate post-process, such as min-batch K-Means, for the final semantic clustering. Besides, MaskContrast [21] and FreeSOLO [22] leverage mask priors and primarily focus on foreground object segmentation. In contrast, we propose to leverage the smoothness prior as a supervision cue to directly optimize the generated semantic map, achieving more coherent and accurate segmentation results.

Self-supervised representation learning (SSL) aims to learn general representations for images without additional labels, which has offered significant benefits to various downstream tasks, including detection and segmentation [6; 8]. One main paradigm of SSL is based on contrastive learning [23; 24; 25; 26; 27; 11; 28; 29], which seeks to maximize the feature similarity between an image and its augmented pairs while minimizing similarity between negative pairs. For example, MoCo [24] trains a contrastive model by using a memory bank that stores and updates negative samples in a queue-based fashion. SimCLR [23] proposes to learn a nonlinear transformation, i.e. , a projection head, before the contrastive loss, to improve performance. Notably, DINO [11], built upon Vision Transformer (ViT) [30], has a nice property of focusing on the semantic structure of images, such as scene layout and object boundaries. Features extracted by DINO exhibit strong semantic consistency and have demonstrated significant benefits for downstream tasks [12; 6; 8]. Another mainstream belongs to the generative learning approach [31; 32; 33]. MAE [32] and SimMIM [31] propose to predict the raw masked patches, while MaskFeat [32] proposes to predict the masked features of images. Our work also leverages recent progress in SSL for unsupervised semantic segmentation.

3 Method

Problem setting. Given a set of unannotated images I = [ I 1 , . . . , I B ] ∈ R B × 3 × H × W , where B denotes the number of images, and 3 , H, W represent the channel, height, and width dimensions respectively, the objective of unsupervised semantic segmentation is to learn a labeling function f ∈ F that predicts the semantic label for each pixel in each image. We represent the predicted semantic maps as Y = [ Y 1 , . . . , Y B ] ∈ { 1 , · · · , K } B × H × W , where K refers to the number of predefined categories.

Architecture. To achieve this goal, we introduce the SmooSeg approach, which capitalizes on selfsupervised representation learning and smoothness prior within an energy minimization framework, as illustrated in Fig. 2. SmooSeg comprises three primary components: a feature extractor f θ , a projector h θ , and a predictor g θ . Initially, for each image I i , we employ a pre-trained backbone network, such as a frozen version of DINO, to acquire feature representations X i = f θ ( I i ) ∈ R C × N , where C and N denote the number of feature channels and image patches, respectively. Subsequently, the projector h θ maps these features onto a low-dimensional embedding space, resulting in a set

EMA

Extractor

Projector

Predictor

argmax

sg

sg

Scaled Dot

Product

of compact features Z i = h θ ( X i ) ∈ R D × N , where D denotes the reduced feature dimensionality. Finally, the predictor g θ generates the label assignments A { s,t } i ∈ R K × N by computing the similarity scores between the compact features Z i and the prototypes P { s,t } . Here, P s and P t represent student and teacher prototypes, respectively. The semantic map Y i for image I i can be obtained by reshaping the output Y t i of the teacher branch.

3.1 Smoothness Prior

Real-world images typically exhibit inherent continuity and coherence in terms of semantics, texture, and color. Within a single object, semantic labels tend to demonstrate smoothness and consistency, ensuring a cohesive representation of the object. In contrast, labels between distinct objects manifest discontinuity and divergence, facilitating the separation of different object instances. This essential property, known as the smoothness prior, is expected to play a critical role in guiding unsupervised semantic segmentation tasks toward more accurate and meaningful segmentation results. We therefore consider the following pairwise smoothness term:

where W ii ∈ R N × N is the closeness matrix of image I i . δ ( Y i,p , Y i,q ) is the penalty that takes the value of 1 if Y i,p = Y i,q , and 0 otherwise. By minimizing this smoothness term, two close patches with different labels will be penalized. In other words, the segmentation model is encouraged to assign similar labels to close patches, thereby promoting the coherence within objects.

̸

Closeness matrix. It is worth noting that the large intra-class variation in appearances within the raw pixel space renders the discovery of well-suited closeness relationships among low-level observations challenging. We therefore propose to model the closeness relationships by the cosine distance in the high-level feature space. Specifically, W ii can be calculated by:

where X i,p and X i,q represent the feature vectors for patches p and q of image I i , respectively. Theoretically, a large element value in the closeness matrix, i.e., a high cosine similarity, suggests a high possibility of a close patch pair, and vice versa. We apply a zero-mean normalization to this matrix: ¯ W ii p = W ii p -1 N ∑ q W ii pq . This normalization balances the negative and positive forces during optimization, which prevents excessive influence from either the negative or positive components of the closeness matrix and ensures that the optimization process is more stable.

Label penalty. Directly minimizing Eq. 1 to optimize our segmentation model is not feasible due to the non-differentiable property of δ ( · , · ) and the hard label assignment Y . As a result, we have to

resort to another form of penalty cost. Suppose we have the soft label assignment A t i ∈ R K × N of image I i (which will be introduced later), by which we can redefine the penalty cost function as:

Because the non-negative property of the softmax output, i.e. , 0 ≤ A t , 0 ≤ δ ( · , · ) ≤ 1 always holds. A larger value of δ ( · , · ) denotes a greater dissimilarity between two labels, thereby indicating a higher penalty cost, and vice versa.

Smoothness prior within and across images. To prevent the model from converging to a trivial solution where the labeling function becomes smooth everywhere, we also apply the smoothness prior across images, acting as a strong negative force, by introducing another image I i ′ that is randomly selected from the current batch. We then obtain the final smoothness term:

Here, we introduce a scalar b 1 to adjust the threshold for applying the penalty. That is, when ¯ W ii pq -b 1 > 0 , indicating that two patches p, q with a high closeness degree are nearby patches in the embedding space, patches p, q with different labels will be penalized, encouraging the piecewise smoothness within segments; otherwise, they are rewarded to assign different labels, leading to the discontinuities across segments. By doing so, SmooSeg is capable of finding globally coherent semantic segmentation maps.

Discussion with CRF and STEGO. CRF methods [17; 18] model the closeness relationship of pixels using their spatial coordinates, emphasizing the local smoothness within each image. On the contrary, our SmooSeg encodes the global closeness relationship of image patches based on the cosine distance in the feature space, which can discover the high-level semantic groups of images. Our smoothness term appears to be similar to the correlation loss in STEGO: L corr = -∑ ( F -b ) max ( S, 0) , but essentially the two losses model different things. In STEGO, S denotes the feature correlation, by which STEGO aims to learn low-dimensional compact representations for images through a learnable projection head. A separate clustering algorithm, e.g., k-means, is required to obtain the final segmentation maps. However, even with the learned compact representations, the coherence of image segments is not guaranteed in STEGO as slight differences in features may lead to inconsistent labels in the clustering stage. In contrast, our SmooSeg aims to directly learn a labeling function (projector + predictor) based on the smoothness prior, which encourages piecewise smoothness within segments and preserves disparities across segments, leading to more coherent and semantically meaningful segmentation maps. Additionally, the negative part of S contradicts the learning intention of STEGO and therefore requires a 0-clamp via max ( S, 0) , which however, represents discontinuities between image patches and should be preserved. In contrast, our label penalty 0 ≤ δ ( · , · ) ≤ 1 has a desirable property compared to S .

3.2 Asymmetric Predictor

A desirable labeling function learnt through energy minimization should on the one hand produce piecewise smooth results, and on the other hand be well fit between the observations and labeling outputs. For semantic segmentation, we expect the labeling output of an image to align well with its semantic map. In other words, the labeling output should accurately predict a category for each individual pixel with high confidence or low entropy. However, this goal is a nutshell in unsupervised semantic segmentation as there is no observed semantic map.

Self-training [34; 29] emerges as a promising solution for tasks involving unlabeled data. To address the above challenge, we design an asymmetric student-teacher style predictor to learn the labeling function through a stable self-training strategy. The student branch employs a set of K learnable prototypes (class centers) P s = [ p s 1 , · · · , p s K ] ∈ R K × D to predict the semantic maps of images. The teacher branch holds the same number of prototypes P t as the student, and P t is updated as an exponential moving average of P s . We then compute the soft assignment A { s,t } i of the embeddings Z i with the prototypes P { s,t } by computing their cosine similarity. With ℓ 2 -normalized embeddings

Algorithm 1 SmooSeg: PyTorch-like Pseudocode

where temperature parameter τ > 0 controls the sharpness of the output distribution of the teacher branch. The teacher branch is responsible for generating smoothly updated pseudo labels to supervise the student prototypes' learning. By using a patch-wise cross-entropy loss, we have the data term as

where ■ · is an indicator that outputs 1 if the argument is true, and 0 otherwise. Y t i = argmax A t i is the hard pseudo label for patch p of image I i . By minimizing E data, the segmentation model is expected to generate label assignments for each patch with high confidence, thus ensuring a better fit between the observations and their predicted labels.

3.3 Overall Optimization Objective

Our final optimization objective function for training SmooSeg is obtained by incorporating the smoothness term and the data term as follows:

In practice, L could be approximately minimized using Stochastic Gradient Descent (SGD). During each training iteration, the projector is optimized using gradients from the smoothness loss, while the student prototypes are optimized using gradients from the data loss. The teacher prototypes are updated as an exponential moving average of the student prototypes: P t = αP t +(1 -α ) P s , with α denoting the momentum value. After training, we use the output from the teacher branch as the segmentation results. The overall procedure in pytorch-like pseudocode of SmooSeg is summarized in Algorithm 1.

4 Experiments

4.1 Experimental Setup

Datasets. Our experimental setup mainly follows that in previous works [8; 4] in datasets and evaluation protocols. We test on three datasets. COCOStuff [35] is a scene-centric dataset with a total of 80 things and 91 stuff categories. Classes are merged into 27 categories for evaluation, including 15 stuff and 12 things. Cityscapes [36] is a collection of street scene images from 50 cities, with classes merged into 27 classes by excluding the "void" class. Potsdam-3 [3] is a remote sensing dataset with 8550 images belonging to 3 classes, in which 4545 images are used for training and 855 for testing.

Evaluation metrics. For all models, we utilize the Hungarian matching algorithm to align the prediction and the ground-truth semantic map for all images. We also use a CRF [17; 8] as the postprocessing to refine the predicted semantic maps. Two quality metrics including mean Intersection over Union ( mIoU ) and Accuracy ( Acc ) over all the semantic categories are used in the evaluation.

Implementation details. Our experiments were conducted using PyTorch [37] on an RTX 3090 GPU. To ensure a fair comparison with previous works [6; 8], we use DINO [11] with a ViT-small 8 × 8 backbone pre-trained on ImageNet as our default feature extractor, which is frozen during model training. Our projector consists of a linear layer and a two-layer SiLU MLP whose outputs are summed together. The predictor contains two sets of prototypes with the same initialization. The exponential moving average (EMA) hyper-parameter is set to α = 0 . 998 . The dimension of the embedding space is D = 64 . The temperature is set to τ = 0 . 1 . We use the Adam optimizer [38] with a learning rate of 1 × 10 -4 and 5 × 10 -4 for the projector and predictor, respectively.

We set a batch size of 32 for all datasets. For Cityscapes and COCOStuff datasets, we employ a five-crop technique to augment the training set size. We train our model with a total of 3000 iterations for Cityscapes and Potsdam-3 datasets, and 8000 iterations for the COCOStuff dataset.

4.2 Comparison with State-of-the-Arts

Quantitative results. Wesummarise the quantitative results on three datasets in Tables 1, 2 and 3, respectively. Results of baselines, ResNet50[39], MoCoV2[40] and DINO[11] are directly cited from the paper [8], while the results of DINOV2 [41] (Table 3) are obtained by our implementation. For these baselines, we first extracted dense features for all images, then utilized a minibatch k-means algorithm to perform patches grouping, which resulted in the final segmentation maps. Our SmooSeg significantly outperforms all the state-of-the-art methods in terms of both pixel accuracy and mIoU on all datasets. In particular, on the COCOStuff dataset in Table 1, with DINO ViT-S/8 as backbone, SmooSeg gains a 14.9% improvement in pixel accuracy and a 2.2% improvement in mIoU over the best-performing baseline STEGO.

Weobserve that TransFGU outperforms STEGO in terms of accuracy, but is inferior in mIoU on both COCOStuff and Cityscapes. This is due to

Image

Label

SmooSeg

STEGO

TransFGU

0069

0371

0158

0233

Image

Label

SmooSeg

STEGO

the fact that TransFGU adopts a pixel-wise cross-entropy loss, which focuses more on the overall accuracy of pixels, while STEGO achieves better class-balanced segmentation results through minibatch k-means. Our SmooSeg significantly outperforms both TransFGU and STEGO in both accuracy and mIoU, We attribute this superiority to our energy minimization loss, which optimizes both the smoothness term and the data term simultaneously.

Same conclusions can be drawn on the Postdam-3 dataset, as shown in Table 3. We can see that SmooSeg, with DINO ViT-B/8 as the backbone, significantly outperforms STEGO, with gains of 5.7% in accuracy and 7.7% in mIoU. The improvement is particularly significant in terms of mIoU. This is not surprising as Potsdam-3 is a remote sensing image dataset that contains only 3 classes, so segments on the Potsdam-3 are often relatively large. In such a scenario, the smoothness prior becomes even more important in ensuring coherent segmentation maps.

Qualitative results. We present qualitative examples of SmooSeg, STEGO and TransFGU on three datasets in Figs. 3 and 4. Additional qualitative results, along with color maps, can be found in Appendix C. As shown in Fig. 3, SmooSeg produces high-quality fine-grained segmentation maps that outperform those obtained by STEGO and TransFGU. Though STEGO uses a feature correspondence loss to encourage features to form compact clusters, its segmentation maps still suffer from incoherence problems. It can be observed that, slight differences in features can result in inconsistent labels in STEGO. The second image from the left column of Fig. 3 shows such an example: the slight variations in light on the sea surface results in differences in features, leading to an

Centers

：

Centers

：

Centers

：

Image

Features of DINO, Acc=69.6%, mloU=48.6%

Embedding of STEGO, Acc=80.0%, mloU=65.8%

Embedding of SmooSeg, Acc=87.4%, mloU=77.8%

★★

incomplete and incoherent water segment. Similar phenomenon can be observed in the segmentation maps on Cityscapes and Potsdam-3 too. Besides, although TransFGU is a top-down approach, it still overlooks the relationship between image patches in its top-down approach, and therefore achieves much worse segmentation results. In contrast, SmooSeg with the aim of generating smooth label assignments within segments while preserving differences across different segments by leveraging the smoothness prior, the semantic maps produced by SmooSeg show more coherent and semantically meaningful results. In Fig. 4, we can see that SmooSeg outperforms the other methods in terms of accurate boundaries.

4.3 Analyses

Visualization. Feature visualizations of DINO, STEGO and SmooSeg are illustrated in Fig. 5. We can see that the feature distribution of DINO with ViT-base/8 as the backbone exhibits some semantic consistency, with compact clusters within each image but disperse across images. The embeddings of STEGO, which are distilled from DINO features using feature correspondence loss, show higher semantic consistency than DINO, with more compact clusters across images, such as the yellow markers, and improved performance. However, STEGO still suffers from the label incoherence problem due to the large intra-class variation of embeddings, indicating that feature distillation alone is insufficient to capture the high-level semantic coherence of segments. Our SmooSeg leverages the smoothness prior to encourage smooth label assignments, measured by the cosine distance between patch embeddings and prototypes (centers), and achieves remarkable improvement in the semantic consistency of feature embeddings. As shown in the right part of Fig. 5, SmooSeg produces highly semantically compact and coherent clusters with clear class boundaries for all images, and the performance, at 87.4% Acc and 77.8% mIoU, significantly higher than STEGO. These results further prove the effectiveness of our SmooSeg in using smoothness prior for unsupervised semantic segmentation.

Objective function. To assess the effectiveness of our energy minimization objective function, we conduct an ablation study on the COCOStuff dataset by comparing SmooSeg with four variants of the objective function, each with a different term removed. For the variant of w/o E data, we only keep the prototypes in the teacher branch to generate the label map for the smoothness term. The results are shown in Table 4, where E across smooth denotes the smoothness term across different images. We can see that the performance drops by 10.2% on Acc and 1.2% on mIoU when removing the data term E data, which highlights the importance of the data term in promoting the fitting of the labeling function. Besides, removing E across smooth results in a much larger drop in performance, with a decrease of 27.1% on Acc and 16.3% on mIoU. Moreover, our segmentation model fails when removing the entire E smooth . The smoothness term utilizes a closeness matrix constructed from high-level features of a pre-trained model, acting as strong supervision signals to guide the label learning for all

80

ACC

mloU

60

%40

20

0

0.07

0.1

0.2

0.40.6

0.8

1

Temperaturet

80

60

%40

20

AcC

mloU

0.1

0.5

0.9

0.99

0.9950.998

Momentuma

image patches. Therefore, it is reasonable to see that E smooth contributes significantly to the overall performance. On the contrary, the data term operates in a self-training fashion with pseudo labels derived from the teacher branch, which alone cannot generate accurate segmentation maps. These findings demonstrate the crucial role of both the data and smoothness terms for optimal performance of SmooSeg in unsupervised semantic segmentation.

Temperature parameter τ . We investigate the effect of the temperature parameter τ on the performance of SmooSeg on the COCOStuff dataset, and report the results in Fig. 6. Theoretically, a smaller τ sharpens the softmax output, providing greater gradients and supervision signals for model training. Fig. 6 shows that τ plays a critical factor in the success of SmooSeg. Specifically, SmooSeg achieves good results when τ ≤ 0 . 1 , while performance drops considerably when τ ≥ 0 . 2 because the softmax output tends to become uniformly distributed.

Momentum parameter α . We also study the impact of the α on SmooSeg. α controls the smoothness of the update of the teacher predictor from the student predictor. We plot the performance on the COCOStuff dataset as α changes from 0.1 to 1 in Fig. 7. The performance of SmooSeg gradually improves as α increases, and reaches stable when 0 . 99 ≤ α .

Limitation. Setting hyper-parameters without cross-validation is always a challenge for unsupervised learning methods. The main limitation of our method is that it involves two dataset-specific hyper-parameters in the smoothness term. We present a feasible strategy in Appendix A to alleviate this issue.

5 Conclusions

In this paper, we propose SmooSeg, a simple yet effective unsupervised semantic segmentation approach that delves into the potential of the smoothness prior, emphasizing the coherence property of image segments. In particular, we implement a pairwise smoothness loss to effectively discover semantically meaningful groups. We also design an asymmetric teacher-student style predictor to generate high-quality segmentation maps. SmooSeg comprises a frozen extractor, as well as a lightweight projector and a predictor which could be optimized using our energy minimization objective function. Experimental results show that SmooSeg outperforms state-of-the-art approaches on three widely used segmentation benchmarks by large margins.

Acknowledgement. This research is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s), by the National Research Foundation, Singapore under its Industry Alignment Fund - Pre-positioning (IAF-PP) Funding Initiative, and by the Ministry of Education, Singapore under its MOE Academic Research Fund Tier 2 (STEM RIE2025 Award MOE-T2EP202200006). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore, and the Ministry of Education, Singapore.

References

Mennatullah Siam, Mostafa Gamal, Moemen Abdel-Razek, Senthil Yogamani, Martin Jagersand, and Hong Zhang. A comparative study of real-time semantic segmentation for autonomous driving. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 587-597, 2018.

Xiaohui Yuan, Jianfang Shi, and Lichuan Gu. A review of deep learning methods for semantic segmentation of remote sensing imagery. Expert Systems with Applications , 169:114417, 2021.

Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9865-9874, 2019.

Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16794-16804, 2021.

Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. Labels4free: Unsupervised segmentation using stylegan. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13970-13979, 2021.

Zhaoyuan Yin, Pichao Wang, Fan Wang, Xianzhe Xu, Hanling Zhang, Hao Li, and Rong Jin. Transfgu: a top-down approach to fine-grained unsupervised semantic segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX , pages 73-89. Springer, 2022.

Shanghua Gao, Zhong-Yu Li, Ming-Hsuan Yang, Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint arXiv:2203.08414 , 2022.

Yassine Ouali, Céline Hudelot, and Myriam Tami. Autoregressive unsupervised image segmentation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16 , pages 142-158. Springer, 2020.

Tsung-Wei Ke, Jyh-Jing Hwang, Yunhui Guo, Xudong Wang, and Stella X Yu. Unsupervised hierarchical semantic segmentation with multiview cosegmentation and clustering transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2571-2581, 2022.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8364-8375, 2022.

Meng Tang, Federico Perazzi, Abdelaziz Djelouah, Ismail Ben Ayed, Christopher Schroers, and Yuri Boykov. On regularized losses for weakly-supervised cnn segmentation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 507-522, 2018.

Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia. Underexposed photo enhancement using deep illumination estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6849-6857, 2019.

Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2613-2622, 2021.

Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. IEEE Transactions on pattern analysis and machine intelligence , 23(11):1222-1239, 2001.

Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. Advances in neural information processing systems , 24, 2011.

Xide Xia and Brian Kulis. W-net: A deep model for fully unsupervised image segmentation. arXiv preprint arXiv:1711.08506 , 2017.

Robert Harb and Patrick Knöbelreiter. Infoseg: Unsupervised semantic image segmentation with mutual information maximization. In Pattern Recognition: 43rd DAGM German Conference, DAGM GCPR 2021, Bonn, Germany, September 28-October 1, 2021, Proceedings , pages 18-32. Springer, 2022.

Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, CarlJohann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap to real-world object-centric learning. arXiv preprint arXiv:2209.14860 , 2022.

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool. Unsupervised semantic segmentation by contrasting object mask proposals. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10052-10062, 2021.

Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. Freesolo: Learning to segment objects without annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14176-14186, 2022.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6707-6717, 2020.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3024-3033, 2021.

Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. arXiv preprint arXiv:2205.15288 , 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9653-9663, 2022.

Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14668-14678, 2022.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10687-10698, 2020.

Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1209-1218, 2018.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikitlearn: Machine learning in python. the Journal of machine Learning research , 12:2825-2830, 2011.

David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision , volume 2, pages 1150-1157. Ieee, 1999.

Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV) , pages 132-149, 2018.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Appendix

A Setting Hyper-parameters

Tuning hyperparameters without cross-validation on labels is particularly challenging for unsupervised learning methods. For example, STEGO [8] contains six parameters that are both dataset- and network-specific. Inspired by their hyperparameter tuning methods, we introduce a similar strategy for manual hyperparameter tuning. Recall that our SmooSeg introduces two hyperparameters b 1 and b 2 in the smoothness loss, which are used to adjust the threshold for applying the penalty. Ideally, a segmentation model should promote smoothness within segments while maintaining discontinuity between different segments. To achieve this, we can monitor the distribution of δ , referred to as the smoothness degree, during the model training (see Fig. 8). In a balanced setting, the smoothness degree distribution should exhibit bimodality, as demonstrated in the first column of Fig. 8. Otherwise, the segmentation model will generate semantic maps that are either too smooth or too discontinuous. Specifically, The hyperparameters used in SmooSeg with DINO as the backbone are summaried in Table 5.

step

step

step

mIoU

mIoU

mIoU

Balanced

Too smooth

Too discontinuous

23

8.4

21

8

13

19

7.6

12

7.2

17

6.8

11

15

6.4

13

6

10

0

500

1k1.5k2k2.5k3k

0

5001k1.5k2k2.5k3k

0

5001k1.5k2k2.5k3k

1000

1000

1000

2000

2000

2000

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

b. Smoothness degree distribution

B The impact of CRF

CRF postprocessing is a common practice in both supervised and unsupervised semantic segmentation [8], and that the use of CRF does not overshadow the contribution of the smoothness term in our work. The smoothness prior in this work performs on the high-level feature maps and mainly contributes to semantic smoothness, while CRF operates on pixels to refine the fine details and remedy the resolution loss caused by the final upsample operation that exists in most semantic segmentation models (a normal upsample rate is 8x8). Therefore, the application of a CRF serves as a supplement to our smoothness prior to further refine low-level smoothness.

Table 6 demonstrates that SmooSeg still achieves state-of-the-art without CRF. Overall, the performance degradation in STEGO is notably more pronounced compared to SmooSeg. Importantly, the application of a CRF serves as an effective supplement to our smoothness prior. Moreover, we also present qualitative visualizations with and without CRF in Figures 9 and 10. It is found that CRF is able to refine the quality of fine details on both STEGO and SmooSeg. However, SmooSeg is consistently more semantically coherent than STEGO either with or without CRF.

C Additional Qualitative Results

To provide further evaluation, we have included some difficult samples from the COCOStuff dataset that were predicted by our Smooseg and STEGO in Figure 11. In these cases, Smooseg and STEGO tend to generate inaccurate semantic maps. However, it is noteworthy that even in challenging scenarios, SmooSeg consistently generates more semantically coherent segmentation maps compared to STEGO. This observation underscores the advantages of incorporating our smoothness prior in semantic segmentation tasks.

We provide the visualization of more results in Fig. 12 for the COCOStuff dataset, Fig 13 for the Cityscapes dataset, Figs 14 and 15 for the Potsdam-3 datasets. Overall, SmooSeg consistently produces more coherent segmentation maps when compared to both STEGO and TransFGU. These visual results provide further evidence of the efficacy of the smoothness prior in enhancing the label coherence and the overall segmentation quality in unsupervised settings.

Image

Label

SmooSeg

STEGO

STEGO

w/o CRF

SmooSeg

w/o CRF

Image

Label

SmooSeg

STEGO

STEGO

w/o CRF

SmooSeg

w/o CRF

K

Image

Label

SmooSeg

STEGO

Image

SmooSeg

STEGO

TransFGU

Label

Sky

Sports

Plant

Water

Building

Ground

Person

Animal

Solid

Outdoor

Rawmaterial

Foo

Furniture

Indoor

Kitchen

Accessory

Wall

Vehicle

Structural

Ceiling

Electronic

Appliance

Floor

Textile

Window

Image

SmooSeg

STEGO

TransFGU

Label

d

Sidewalk

Vegetation

Wall

Rider

Person

Bicycle

Parking

Road

Building

Traffic sign

Train

Traffic light

Sky

Trailer

Car

Fence

Terrain

Ignore

Image

SmooSeg

STEGO

TransFGU

Label

Buildings + clutter

Roads + cars

Low vegetation + trees

Image

Label

STEGO

SmooSeg

Buildings + clutter

Roads + cars

Low vegetation + trees

Image

Label

STEGO

SmooSeg

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles unsupervised semantic segmentation which aims to group pixels into semantic clusters without manual annotation. This paper proposes the smoothness prior by enforcing that adjacent features in metric space share the same semantics. The approach relies on features from a self-supervised representation learning method (ie., DINO). Additionally, the architecture relies on an asymmetric teacher-student style predictor that updates pseudo labels, leading to smoother segmentation predictions. This approach outperforms STEGO (prior s-o-t-a) in terms of pixel accuracy on three datasets: COCOStuff, Cityscapes, and Potsdam with a ViT-S backbone.\n\nSTRENGTHS:\n### Method:\n- Leverages advantages in self-supervised learning (SSL): The proposed method leverages the advancements in self-supervised representation learning in generating dense representations. Notably, it proposes to model the relationships between image patches by relying on high-level features from a pre-trained DINO model.\n- Kmeans is not required: The introduced smoothness prior can be used to directly optimize the generated segmentation map. The paper claims this strategy leads to more coherent and accurate segmentation results than prior work STEGO. In particular, the latter relies on minibatch Kmeans to obtain the final segmentation map.\n- Learnable prior: The approach does not rely on grouping priors as explored in prior works (see below for references). Instead, the approach implicitly uses a learnable grouping prior by relying on the self-supervised representation learning method (i.e., DINO). \n\n### Experiments: \n- Dataset variety: The approach is applicable to different datasets. In particular, the experimental validation considers 3 different datasets: COCOstuff, Cityscapes, and Potsdam. \n- Quantitative results: the approach outperforms prior art (STEGO) on pixel accuracy when using DINO pre-trained weight on Citysapes and COCOstuff dataset.\n\n### Misc.\n- Clarity: The method is well presented by relying on clear figures (e.g., Figure 2). The paper is also easy to follow (for someone who’s familiar with the literature at least).\n\nWEAKNESSES:\n### 1. Originality Method - Difference with STEGO / SlotCon:\n\n- The paper can be seen as a combination of STEGO and SlotCon.\nThe method largely follows the methodology of STEGO. The main SmooSeg loss is similar to the correlation loss in STEGO, which is also mentioned in the paper (see L158).  Overall, I don’t think this is a major issue as I haven’t seen this combination being applied to the task of unsupervised semantic segmentation directly.\n\n- (Minor) I also don’t completely follow the reasoning that the proposed method is superior to Kmeans (L165). While this is indeed an advantage, it’s not clear why Kmeans would necessarily perform worse, as used in STEGO (both STEGO and SmooSeg rely on the features from DINO anyway). Kmeans also has to rely on a distance metric (e.g., cosine distance between features), resulting in similar features/patches being assigned to the same cluster. \n\n\n### 2. Robustness -  setting hyperparameters:\nThere are many parameters that require supervision during training:\n- CRF weights: The weights of the CRF are determined using the annotated validation set. How can these be determined without annotations? It’s not clear from the supplementary if the weights are dataset-specific. (I assume that https://github.com/lucasb-eyer/pydensecrf was used.)\n- Number of classes: The number of classes is not always known a priori. In particular, it’s not clear how much the approach relies on this information. As a result, the introduced setup (section 3) is relatively artificial. In most practical applications, this information won’t be available. As a result, there are important experiments missing, where the sensitivity to this parameter is ablated (ideally comparing the mean and variance over multiple runs with other methods (e.g., STEGO, PiCIE). \n- Dataset-specific parameters: Overall, there are many training parameters that differ across datasets and require finetuning (i.e., loss parameters). Ideally, the same set of parameters is applicable to new/unseen images, especially for an ‘unsupervised’ method. Now, it’s hard to judge the robustness of the approach as it relies on finetuned parameters for each dataset. To my knowledge, this is not the case for prior works and baselines (IIC, PiCIE, DeepCluster). Also, TransFGU keeps most of its parameters constant. While this issue is somewhat tackled in the supplementary (see Section B), it’s still not clear if this strategy was actually used to set these parameters during training.\n\n### 3. Scope of the experiments:\n- The experiments are limited to results on 27 classes  (COCOStuff and Cityscapes). TransFGU also includes COCO-80 and COCOStuff-171. It would be interesting to see the performance for these setups as well.\n- A more practical setting is the semi-supervised setup, especially as the ‘unsupervised’ results are relatively poor and not immediately useful for practical applications. Does the method improve the representations by finetuning the learned representation compared to DINO? Finetuning the complete model or simply training a linear probe on top of the model are interesting experiments [f]. I would also expect that the learned representation can be efficiently fine-tuned with only a few samples [b].\n\n### 4. Quantitative results - different backbones:\nI noticed that only a ViT-S is used for COCO-stuff and Cityscapes, while a ViT-B is used for the Potsdam dataset. However, STEGO reports its final results with a ViT-B and outperforms the numbers in this paper (i.e., STEGO obtains 28mIoU on COCOstuff with a ViT-B). So, why is a ViT-B not used for COCO-stuff and Cityscapes as well, to make sure that the observations transfer to stronger backbones? This would furthermore make the main claim of the paper stronger.\n\n### 5. Related work\nThere are also a few related works missing. Interestingly, some of these rely on priors, such as superpixels, edge, or saliency estimators [d, e, f], which the paper under review does not require. More suggestions can be found below:\n\n[a] Wang et al, Dense Contrastive Learning for Self-Supervised Visual Pre-Training, CVPR.\n\n[b] Wang et al, FreeSOLO: Learning to Segment Objects without Annotations, CVPR.\n\n[c] Ziegler et al, Self-Supervised Learning of Object Parts for Semantic Segmentation CVPR.\n\n[d] Hwang et al, Segsort: Segmentation by discriminative sorting of segments CVPR.\n\n[e] VanGansbeke et. al, Unsupervised Semantic Segmentation by Contrasting object mask proposals, ICCV.\n\n[f] Zhang et al., Self-Supervised Visual Representation Learning from Hierarchical Grouping, NeurIPS.\n\nQUESTIONS:\nMy most important questions are listed above. \n\nI also have a few additional questions:\n- L265 claims that the proposed method outperforms STEGO at the boundaries. However, are the boundaries not primarily dependent on the CRF? How do these methods compare when we don’t use a CRF? It would also be useful to see a few examples without a CRF in the supplementary (optional).\n- How is the best model selected in the absence of a validation set? Can the loss function be used to select the best model during training?\n- The method is dependent on the pretrained DINO weights. We see that MoCov2 performs worse in Table 1. What happens if we use other weights (MoCov3, Pixpro, DenseCL etc.)? \n- Are the visualized predictions in the supplementary randomly selected?\n\nLIMITATIONS:\nWhile some of the limitations are mentioned in the paper (see Section 4), I suggest including other limitations (e.g., known CRF weights, and known number of classes). I also couldn't find a paragraph on the societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an unsupervised training method for semantic segmentation models. It trains the network on a loss with two terms: one encouraging smoothness in the segmentation labels, and a \"data\" term based on self-training. It assumes a neural net architecture with a) an extractor that produces per-pixel features $X$, b) a projector that produces lower-dimensional features, to be compared against \"prototypes\" and used in self-training, and c) the \"predictor\" that outputs the per-pixel segmentation labels from the prototype/feature comparisons. The backbone feature extractors are primarily based on DINO [11], though some experiments appear to use MoCo [24] in its place. This is evaluated in terms of its accuracy in semantic segmentation tasks including COCOStuff, Cityscapes, and Potsdam-3.\n\nSTRENGTHS:\ni) Experiments consider variety of semantic segmentation tasks: street, general, and aerial image datasets.\n\nThis is a good demonstration that the method can be applied generally, even if it is sensitive to some choice of hyperparameters.\n\nii) Uses a backbone (DINO) also trained in an unsupervised (self-supervised) way.\n\nUses DINO ViT-S/8 [11] or MoCo [20] as backbones in different experiments on the segmentation head & training. These were also trained without hand labelling, wtih a self- or unsupervised method; as opposed to using, for example, a standard ImageNet backbone from supervised training. This is a correct and principled way to make sure the proposed segmentation models are truly and completely unsupervised.\n\niii) Precise presentation\n\nUses both an algorithm, written in plain code, along with diagrams and the math to describe the method quite completely. Those parts of the method that are unclear from the definition of the loss can be deduced quite unambiguously from Algorithm 1, and vice versa.\n\nWEAKNESSES:\niv) Not a lot of experiments or analyses explaining *why* the method works.\n\nThe experiments mostly show end results (mAP, qualitative segmentation quality). Hyperparameter studies are useful, though: providing good insight into the role each hyperparameter plays. And there is a minimal ablation study in Table 4.\n\nIt still seems unclear what exactly the network is learning from. For example, to what extent is the smoothness a constraint on the extractor vs. the projector? It seems possible that $E_\\mathrm{smooth}$ is also penalizing the cosine distances, as defined in Equation (2), and pushing feature vectors $X$ together, and not just using it as a weight for spatial smoothness in the labels $Y$, as in the intuitive explanation of Eq (1). I don't see `torch.no_grad` or any equivalent around the weight computations in Algorithm 1. Fig 5 visualizes feature embeddings on Potsdam-3, and this shows that the group similar classes together. Though, if the mapping between feature vectors $X$ and the labels $Y$, in the projector and predictor, is a \"smooth\" function (in the sense of Lipschitz smoothness) so that similar features get similar labels, might we already expect that the smoothness penalty is reduced, even if there is no meaningful spatial smoothness of the labels?\n\nThe relative role/contribution of the within-image and between-image comparisons also seems to complicate the intuitive explanation of the method. There is a large accuracy drop if $E_\\mathrm{smooth}$ only penalizes adjacent pixels within the image, and not across the images. The motivation for $E_\\mathrm{smooth}$ that adjacent pixels are likely to have the same label, the \"natural tendency towards piecewise coherence regarding semantics, texture, or color,\" does not apply to pixels within different images. The smoothness between pixels that are adjacent in a metric space is a property of the construction of that metric space: not a natural property of images, so this part's role in the function of the smoothness perhaps needs more explanation and experiments. Why do we expect with this unsupervised loss that the metric space will be semantically meaningful?\n\nv) Novelty is arguably more limited than is claimed.\n\nThe use, in unsupervised training of semantic segmentation models, of spatial smoothness of the labels does seem to be explored. Though many of the particulars and how it's used within the overall framework is new as of this submission.\n\nSmoothness priors themselves are, of course, quite common in semantic segmentation. Spectral methods also expected to be a form of smoothness prior, but using squared differences in place of the $\\delta$ in Eq (1) of the submission. This kind of prior is used in unsupervised training of semantic segmentation models in:\n\n[a] Xia & Kulis \"W-Net: A Deep Model for Fully Unsupervised Image Segmentation\"\n\n[b] Melas-Kyriazi et. al. \"Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization\"\n\nW-Net also incorporates a fully-connected CRF, in section 4.1, though this is only for preprocessing and not for unsupervised training. I however don't see prior work that uses a fully-connected CRF within the training objective in the way that this submission does. The backbone is also updated relative to previous work: this submission is perhaps in the genre of papers that applies a given vision idea/concept to ViTs instead of CNNs.\n\nvi) There may also be some gaps in the citation of prior work similar to the submission's teacher/student/self-training approach described in section 3.2. Citations on self-supervised learning in the \"Related Work\" in section 2 focus on contrastive learning approaches. More similar approaches might include:\n\n[c] Scudder \"Probability of error of some adaptive pattern-recognition machines\"\n\n[d] Xi et. al. \"Self-training with Noisy Student improves ImageNet classification\"\n\nBased on my read, the submission does *not* modify the inputs or consider different views between teacher and student, as in [19, 20, 21, etc.]. (If my read is incorrect, what exactly is the \"pretext task\" that the authors use?) Though [c.d] and similar papers differ enough from the submission's training scheme, including the use of a projector module to get the prototypes that are compared between teacher & student, that this part is not relevant to judging novelty.\n\nQUESTIONS:\nWhy the focus on semantic segmentation? If the labels aren't used, then I'd assume the method should equally apply to instance, panoptic, or salient-foreground segmentation. Especially given it does seem to generalize across different kinds of semantic segmentation. Is it because the prototypes $P$ are expected to somehow relate to semantic classes of \"stuff?\"\n\nThe stated goal of the smoothness prior is that \"the segmentation model is encouraged to assign similar labels to adjacent patches, thereby promoting spatial coherence within objects.\" What determines which pixels are \"adjacent\" when constructing E_smooth? I.e. is it 4-connected, 8-connected, or fully connected? The pairwise weights don't seem to include spatial distance. Toward the end of Section 3.1 it's stated that the authors \"also apply the smoothness prior across images.\" Which pairs of pixels across pixels are included as terms in the sum in Eq (1)?\n\nMinor comments:\n\n  * Typos in Algorithm 1 comments: \"updata\" and \"prototyeps\"\n  * Citation to [20] is repeated on line 94\n\nLIMITATIONS:\nThe sensitivity of the results to hyperparameter choices is mentioned as a limitation. This does seem likely to be the major drawback, with bad results like mode collapse being a concern. In Appendix B of the supplement, it is described in qualitative terms that one can monitor the distribution of the differences in soft label assignments (the \"smoothness degree\") to adjust the parameters. They don't give a fully reproducible method or algorithm for determining b1 and b2 to achieve the desired $\\delta$ distribution, but it does seem clear what one would be looking for to tune this manually.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the problem of unsupervised semantic segmentation. In contrast to STEGO (baseline), the approach learns semantic embeddings directly in a student-teacher regime without the need for the K-means. The objective is entirely unsupervised and is reminiscent of a CRF energy formulation with data and a smoothness term. The pseudo labels from the teacher embeddings provide the signal for the data term. The work achieves improved segmentation accuracy on standard benchmarks over the state of the art and offers an interesting analysis. Overall, the work is compelling in quality and contribution.\n\nSTRENGTHS:\n- The application of the smoothness constraint is natural and simple.\n- The learning problem is well-designed and executed. The text is generally well-written (albeit not without minor typos).\n- The experiments have sufficient scope and provide valuable insights. The empirical results are compelling.\n\nWEAKNESSES:\nThere are technical similarities to STEGO, such as in the loss formulation in Eq. (4). There are differences, though, as discussed in 158-174. However, it does take away a bit from the novelty.\n\nEmpirically, the approach appears at its best with DINO features using ViT architecture; the improvement over prior art may not translate to other SSL representations and model architectures (MoCo, ResNet, see Tab. 1). Nevertheless, the approach remains competitive.\n\nThe method introduces a number of hyperparameters, which can be challenging to fine-tune in an unsupervised setup. I like that this is acknowledged in the work, however, and Appendix B provides some interesting analysis. \n\nTypos:\ne.g. l. 178, 307\n\nQUESTIONS:\n- l. 140-141: Why is optimisation only stable with such normalisation?\n- Eq. 5: Why is there a need to stop the gradient flow in this fashion?\n- l. 212: I would be curious to understand how the predictions and the ground truth are aligned in a bit more detail.\n- How were the other parameters (e.g. the momentum, the temperature) chosen/fine-tuned?\n- I do not quite follow the sentence in l. 172: “which represents discontinuities between image patches that should be preserved.”\n\nLIMITATIONS:\nThe limitations are discussed in the main text. The supp. material provides further analysis.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new approach called SmooSeg for unsupervised semantic segmentation, which aims to segment images into semantic groups without manual annotation. SmooSeg is based on the idea of smoothness: adjacent features in a metric space should share the same semantics. Specifically, it formulates unsupervised semantic segmentation as an energy minimization problem (Eq. 1,4). Training is performed using teacher-student style self-training.\n\nSTRENGTHS:\n### The paper is well-structured.\n* The introduction is well-written and motivates the problem well. The methods section breaks down the contribution into provides helpful preliminaries (S3.1) in a good amount of detail. It also describes the contribution (S3.2-3.4) precisely without unnecessary complications.\n\n### The visualizations provided are nicely done\n* The t-SNE plot in Figure 5 is quite illustrative, and the qualitative examples in Figure 3 are also helpful.\n\n### Algorithm 1 does a good job of communicating the proposed method.\n * It is very helpful to be able to refer to the PyTorch pseudocode alongside the equations in the paper.\n\n### The “Discussion with CRF and STEGO” section is very helpful\n* My natural first question upon reading the introduction and the beginning of the methods section was about the relationship between SmooSeg and CRFs. This section answered some of my questions, as it gave an analysis of their relationship.\n\nWEAKNESSES:\n### It would help to have further analysis of why the method works.\n* As discussed in the paper’s introduction, the idea of smoothness is image segmentation has a long history, having been explored since well before deep learning. This is especially true of energy minimization approaches, for which many different types of smoothness losses have been proposed over the years. Empirically, it seems that the performance of the smoothness loss proposed in this paper is good. However, I do not really understand _why_ it should be better than any other approach to enforcing image smoothness, such as the approach applied in CRFs. What would make this paper really useful to the community would be if it tried to understand why this particular formulation works well, so that the community can learn some generalizable lessons/insights (which could then be applied to other problem domains as well). \n* Is it something particular about the combination of your smoothness term and the student-teacher approach that works well, or is it each of them individually? \n* What would happen if you took STEGO (exactly as it is) and added your smoothness loss to their method? Would it be good, or does your smoothness loss work particularly well when combined with your student-teacher method? \n\n### Why do you also need to apply a CRF? \n* The supplement states: “We also use a CRF as the post-processing to refine the predicted semantic maps.” This confuses me. If you already have a smoothness loss (which is your contribution), why do you need a CRF? \n* How does your performance compare without a CRF? \n\n_Meta note on the score (because there is nowhere else to put this in the review)_: I am between 4 and 5. I gave a preliminary score of 4, but I am certainly willing to raise my score when the above questions/points are answered.\n\nQUESTIONS:\n### Removing E_{data} in Table 4\n* For Table 4, I’m surprised that the method still works when the data term is removed in Table 4. In that experiment, are there any losses apart from the smoothness term? Without the data term, should there not be degenerate solutions?\n\nLIMITATIONS:\nLimitations are discussed adequately in the paper. One main limitation (how to set hyperparameters) is mentioned clearly at the end of the paper and addressed in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the difficult task of unsupervised semantic segmentation at the level of the scene (dense prediction). The authors exploit the piecewise coherence regarding the semantics, texture, or color that similar objects naturally have (termed as smoothness prior). The problem formulation raises some challenges the author solves by using a pre-trained (frozen) feature extractor to model the closeness relationships among observations and introduce a novel pairwise smoothness loss, and a teacher-student style predictor. Results are showcased on popular benchmarks: COCOSttuff, Cityscapes, and Potsdam, on which the authors report state-of-the-art performance.\n\nSTRENGTHS:\n1. Originality - the problem formulation (energy minimization objective function) and the addition of the smoothness prior counts as novel work.\n2. Quality - the paper is decently structured, method and experimental analysis are sound and convincing. \n3. Clarity - some aspects of the paper could be improved. Some details are left out, making it be challenging to reproduce the results based on the information provided in the paper.\n4. Significance - the topic is indeed relevant to the research community.\n\nWEAKNESSES:\n* L112 - The architecture description paragraph is tough to follow and the dedicated figure (Figure 1 in the main submission) is overcomplicated. Also, the use of the term \"prototypes\" for the teacher-student paradigm is confusing. Please consider rephrasing this part.\n* L62 vs. L312 - minor contradiction.\n* L307 - Type \"We\" - no capital letter\n* The main submission has an appendix document but there is no reference in the main submission regarding the contents of the supplementary material.\n* Important details are in the supplementary material and not mentioned in the current submission - such as the use of CRF for further refining the final segmentation maps.\n* In the experiments section there is no mention of how the authors produced the segmentation maps from SSL feature extractors such as ResNet50, MoCoV2, DINO (Table 1), DINO (Table 2), or DINO, DINOV2 (Table 3).\n\nQUESTIONS:\nPlease see weaknesses above.\n\nLIMITATIONS:\nThe authors have not properly addressed the limitations of their proposed method. I highly encourage them to do so in a dedicated section in the main submission. Ideas for future work are a plus.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}