SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. Our experimental setup mainly follows that in previous works [8; 4] in datasets and evaluation protocols. We test on three datasets. COCOStuff [35] is a scene-centric dataset with a total of 80 things and 91 stuff categories. Classes are merged into 27 categories for evaluation, including 15 stuff and 12 things. Cityscapes [36] is a collection of street scene images from 50 cities, with classes merged into 27 classes by excluding the "void" class. Potsdam-3 [3] is a remote sensing dataset with 8550 images belonging to 3 classes, in which 4545 images are used for training and 855 for testing.
Evaluation metrics. For all models, we utilize the Hungarian matching algorithm to align the prediction and the ground-truth semantic map for all images. We also use a CRF [17; 8] as the post-processing to refine the predicted semantic maps. Two quality metrics including mean Intersection over Union (mIoU) and Accuracy (Acc) over all the semantic categories are used in the evaluation.
Implementation details. Our experiments were conducted using PyTorch [37] on an RTX 3090 GPU. To ensure a fair comparison with previous works [6; 8], we use DINO [11] with a ViT-small  $8 \times 8$  backbone pre-trained on ImageNet as our default feature extractor, which is frozen during model training. Our projector consists of a linear layer and a two-layer SiLU MLP whose outputs are summed together. The predictor contains two sets of prototypes with the same initialization. The exponential moving average (EMA) hyper-parameter is set to  $\alpha = 0.998$ . The dimension of the embedding space is  $D = 64$ . The temperature is set to  $\tau = 0.1$ . We use the Adam optimizer [38] with a learning rate of  $1 \times 10^{-4}$  and  $5 \times 10^{-4}$  for the projector and predictor, respectively.
We set a batch size of 32 for all datasets. For Cityscapes and COCOStuff datasets, we employ a five-crop technique to augment the training set size. We train our model with a total of 3000 iterations for Cityscapes and Potsdam-3 datasets, and 8000 iterations for the COCOStuff dataset.
4.2 Comparison with State-of-the-Arts
Quantitative results. We summarise the quantitative results on three datasets in Tables 1, 2 and 3, respectively. Results of baselines, ResNet50[39], MoCoV2[40] and DINO[11] are directly cited from the paper [8], while the results of DINOv2 [41] (Table 3) are obtained by our implementation. For these baselines, we first extracted dense features for all images, then utilized a minibatch k-means algorithm to perform patches grouping, which resulted in the final segmentation maps. Our SmooSeg significantly outperforms all the state-of-the-art methods in terms of both pixel accuracy and mIoU on all datasets. In particular, on the CO-COStuff dataset in Table 1, with DINO ViT-S/8 as backbone, SmooSeg gains a  $14.9\%$  improvement in pixel accuracy and a  $2.2\%$  improvement in mIoU over the best-performing baseline STEGO.
We observe that TransFGU outperforms STEGO in terms of accuracy, but is inferior in mIoU on both COCOStuff and Cityscapes. This is due to
Table 1: Performance on the COCOStuff dataset (27 classes).
Methods	backbone	Acc.	mIoU
ResNet50 [39]	ResNet50	24.6	8.9
IIC [3]	R18+FPN	21.8	6.7
MDC [4]	R18+FPN	32.2	9.8
PiCIE [4]	R18+FPN	48.1	13.8
PiCIE+H [4]	R18+FPN	50.0	14.4
SlotCon [29]	ResNet50	42.4	18.3
MoCoV2 [40]	ResNet50	25.2	10.4
+ STEGO [8]	ResNet50	43.1	19.6
+ SmooSeg	ResNet50	52.4	18.8
DINO [11]	ViT-S/8	29.6	10.8
+ TransFGU [6]	ViT-S/8	52.7	17.5
+ STEGO [8]	ViT-S/8	48.3	24.5
+ SmooSeg	ViT-S/8	63.2	26.7
Table 2: Performance on the Cityscapes Dataset (27 Classes).
Methods	backbone	Acc.	mIoU
IIC [3]	R18+FPN	47.9	6.4
MDC [4]	R18+FPN	40.7	7.1
PiCIE [4]	R18+FPN	65.5	12.3
DINO [11]	ViT-S/8	40.5	13.7
+ TransFGU [6]	ViT-S/8	77.9	16.8
+ STEGO [8]	ViT-S/8	69.8	17.6
+ SmooSeg	ViT-S/8	82.8	18.4
Figure 3: Qualitative results on COCOStuff (left) and Cityscapes (right) datasets.
Table 3: Performance on the Potsdam-3 Dataset.
Methods	backbone	Acc.	mIoU
Random CNN [3]	VGG11	38.2	-
K-means [42]	VGG11	45.7	-
SIFT [43]	VGG11	38.2	-
IIC [3]	R18+FPN	65.1	-
Deep Cluster [44]	R18+FPN	41.7	-
InfoSeg [19]	CNN	71.6	-
DINO [11]	ViT-B/8	62.2	43.3
+ STEGO [8]	ViT-B/8	77.0	62.6
+ SmooSeg	ViT-B/8	82.7	70.3
DINOV2 [41]	ViT-B/14	81.8	69.0
+ SmooSeg	ViT-B/14	86.3	75.7
Figure 4: Qualitative results on Postdam-3 dataset.
the fact that TransFGU adopts a pixel-wise cross-entropy loss, which focuses more on the overall accuracy of pixels, while STEGO achieves better class-balanced segmentation results through minibatch k-means. Our SmooSeg significantly outperforms both TransFGU and STEGO in both accuracy and mIoU. We attribute this superiority to our energy minimization loss, which optimizes both the smoothness term and the data term simultaneously.
Same conclusions can be drawn on the Postdam-3 dataset, as shown in Table 3. We can see that SmooSeg, with DINO ViT-B/8 as the backbone, significantly outperforms STEGO, with gains of  $5.7\%$  in accuracy and  $7.7\%$  in mIoU. The improvement is particularly significant in terms of mIoU. This is not surprising as Potsdam-3 is a remote sensing image dataset that contains only 3 classes, so segments on the Potsdam-3 are often relatively large. In such a scenario, the smoothness prior becomes even more important in ensuring coherent segmentation maps.
Qualitative results. We present qualitative examples of SmooSeg, STEGO and TransFGU on three datasets in Figs. 3 and 4. Additional qualitative results, along with color maps, can be found in Appendix C. As shown in Fig. 3, SmooSeg produces high-quality fine-grained segmentation maps that outperform those obtained by STEGO and TransFGU. Though STEGO uses a feature correspondence loss to encourage features to form compact clusters, its segmentation maps still suffer from incoherence problems. It can be observed that, slight differences in features can result in inconsistent labels in STEGO. The second image from the left column of Fig. 3 shows such an example: the slight variations in light on the sea surface results in differences in features, leading to an
Figure 5: t-SNE [45] visualization of feature embeddings for DINO, STEGO, and our SmooSeg on the Potsdam-3 dataset. The large-scale image is partitioned into 25 sub-images with a resolution of  $224 \times 224$  before being input into each algorithm. We highlight the distribution of three sub-images using square, star and triangle_up markers, respectively, and plot the class centers of each algorithm using three pentagon markers with different colors, where teal, yellow and purple represent buildings + clutter, roads + cars, vegetation + trees, respectively. Best viewed in color and zoomed in.
incomplete and incoherent water segment. Similar phenomenon can be observed in the segmentation maps on Cityscapes and Potsdam-3 too. Besides, although TransFGU is a top-down approach, it still overlooks the relationship between image patches in its top-down approach, and therefore achieves much worse segmentation results. In contrast, SmooSeg with the aim of generating smooth label assignments within segments while preserving differences across different segments by leveraging the smoothness prior, the semantic maps produced by SmooSeg show more coherent and semantically meaningful results. In Fig. 4, we can see that SmooSeg outperforms the other methods in terms of accurate boundaries.
4.3 Analyses
Visualization. Feature visualizations of DINO, STEGO and SmooSeg are illustrated in Fig. 5. We can see that the feature distribution of DINO with ViT-base/8 as the backbone exhibits some semantic consistency, with compact clusters within each image but disperse across images. The embeddings of STEGO, which are distilled from DINO features using feature correspondence loss, show higher semantic consistency than DINO, with more compact clusters across images, such as the yellow markers, and improved performance. However, STEGO still suffers from the label incoherence problem due to the large intra-class variation of embeddings, indicating that feature distillation alone is insufficient to capture the high-level semantic coherence of segments. Our SmooSeg leverages the smoothness prior to encourage smooth label assignments, measured by the cosine distance between patch embeddings and prototypes (centers), and achieves remarkable improvement in the semantic consistency of feature embeddings. As shown in the right part of Fig. 5, SmooSeg produces highly semantically compact and coherent clusters with clear class boundaries for all images, and the performance, at  $87.4\%$  Acc and  $77.8\%$  mIoU, significantly higher than STEGO. These results further prove the effectiveness of our SmooSeg in using smoothness prior for unsupervised semantic segmentation.
Objective function. To assess the effectiveness of our energy minimization objective function, we conduct an ablation study on the COCOStuff dataset by comparing SmooSeg with four variants of the objective function, each with a different term removed. For the variant of w/o  $E_{\mathrm{data}}$ , we only keep the prototypes in the teacher branch to generate the label map for the smoothness term. The results are shown in Table 4, where  $E_{\mathrm{smooth}}^{\mathrm{across}}$  denotes the smoothness term across different images. We can see that the performance drops by  $10.2\%$  on Acc and  $1.2\%$  on mIoU when removing the data term  $E_{\mathrm{data}}$ , which highlights the importance of the data term in promoting the fitting of the labeling function. Besides, removing  $E_{\mathrm{smooth}}^{\mathrm{across}}$  results in a much larger drop in performance, with a decrease of  $27.1\%$  on Acc and  $16.3\%$  on mIoU. Moreover, our segmentation model fails when removing the entire  $E_{\mathrm{smooth}}$ . The smoothness term utilizes a closeness matrix constructed from high-level features of a pre-trained model, acting as strong supervision signals to guide the label learning for all
Table 4: Analysis of objective function on the COCOStuff dataset.
Methods	Acc.	mIoU
SmooSeg	63.2	26.7
w/o Edata	53.0	25.5
w/o Eacrosssmooth	36.1	10.4
w/o Esmooth	16.8	0.6
Figure 6: Sensitivity study on the temperature parameter  $\tau$ .
Figure 7: Sensitivity study on the momentum parameter  $\alpha$ .
image patches. Therefore, it is reasonable to see that  $E_{\mathrm{smooth}}$  contributes significantly to the overall performance. On the contrary, the data term operates in a self-training fashion with pseudo labels derived from the teacher branch, which alone cannot generate accurate segmentation maps. These findings demonstrate the crucial role of both the data and smoothness terms for optimal performance of SmooSeg in unsupervised semantic segmentation.
Temperature parameter  $\tau$ . We investigate the effect of the temperature parameter  $\tau$  on the performance of SmooSeg on the COCOStuff dataset, and report the results in Fig. 6. Theoretically, a smaller  $\tau$  sharpens the softmax output, providing greater gradients and supervision signals for model training. Fig. 6 shows that  $\tau$  plays a critical factor in the success of SmooSeg. Specifically, SmooSeg achieves good results when  $\tau \leq 0.1$ , while performance drops considerably when  $\tau \geq 0.2$  because the softmax output tends to become uniformly distributed.
Momentum parameter  $\alpha$ . We also study the impact of the  $\alpha$  on SmooSeg.  $\alpha$  controls the smoothness of the update of the teacher predictor from the student predictor. We plot the performance on the COCOStuff dataset as  $\alpha$  changes from 0.1 to 1 in Fig. 7. The performance of SmooSeg gradually improves as  $\alpha$  increases, and reaches stable when  $0.99 \leq \alpha$ .
Limitation. Setting hyper-parameters without cross-validation is always a challenge for unsupervised learning methods. The main limitation of our method is that it involves two dataset-specific hyper-parameters in the smoothness term. We present a feasible strategy in Appendix A to alleviate this issue.
In this paper, we propose SmooSeg, a simple yet effective unsupervised semantic segmentation approach that delves into the potential of the smoothness prior, emphasizing the coherence property of image segments. In particular, we implement a pairwise smoothness loss to effectively discover semantically meaningful groups. We also design an asymmetric teacher-student style predictor to generate high-quality segmentation maps. SmooSeg comprises a frozen extractor, as well as a lightweight projector and a predictor which could be optimized using our energy minimization objective function. Experimental results show that SmooSeg outperforms state-of-the-art approaches on three widely used segmentation benchmarks by large margins.
Acknowledgement. This research is supported under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s), by the National Research Foundation, Singapore under its Industry Alignment Fund - Pre-positioning (IAF-PP) Funding Initiative, and by the Ministry of Education, Singapore under its MOE Academic Research Fund Tier 2 (STEM RIE2025 Award MOE-T2EP20220-0006). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore, and the Ministry of Education, Singapore.
A Setting Hyper-parameters
Tuning hyperparameters without cross-validation on labels is particularly challenging for unsupervised learning methods. For example, STEGO [8] contains six parameters that are both dataset- and network-specific. Inspired by their hyperparameter tuning methods, we introduce a similar strategy for manual hyperparameter tuning. Recall that our SmooSeg introduces two hyperparameters  $b_{1}$  and  $b_{2}$  in the smoothness loss, which are used to adjust the threshold for applying the penalty. Ideally, a segmentation model should promote smoothness within segments while maintaining discontinuity between different segments. To achieve this, we can monitor the distribution of  $\delta$ , referred to as the smoothness degree, during the model training (see Fig. 8). In a balanced setting, the smoothness degree distribution should exhibit bimodality, as demonstrated in the first column of Fig. 8. Otherwise, the segmentation model will generate semantic maps that are either too smooth or too discontinuous. Specifically, The hyperparameters used in SmooSeg with DINO as the backbone are summarized in Table 5.
b. Smoothness degree distribution
Figure 8: Distributions of the label penalty  $\delta = 1 - \cos(A, A)$  within an image and their corresponding performance under three different parameter settings. If the distribution of  $\delta$  tends toward single-modal with peaks at 0.0 (or 1.0), the semantic maps will be overly smooth (or discontinuous).
Table 5: Hyperparameters used in SmooSeg.
Parameter	COCOSTuff	Cityscapes	Potsdam-3
b1	0.5	0.5	0.5
b2	-0.02	-0.02	0.1
B The impact of CRF
CRF postprocessing is a common practice in both supervised and unsupervised semantic segmentation [8], and that the use of CRF does not overshadow the contribution of the smoothness term in our work. The smoothness prior in this work performs on the high-level feature maps and mainly contributes to semantic smoothness, while CRF operates on pixels to refine the fine details and remedy the resolution loss caused by the final upsample operation that exists in most semantic segmentation models (a normal upsample rate is  $8 \times 8$ ). Therefore, the application of a CRF serves as a supplement to our smoothness prior to further refine low-level smoothness.
Table 6: Experimental results of the impact of CRF on SmooSeg and STEGO.
COCOStuff	Cityscapes	Potsdam-3	Avg.
Acc	mIoU	Acc	mIoU	Acc	mIoU	Acc	mIoU
STEGO w/o CRF	46.5	22.4	63.5	16.8	74.1	58.9	61.4	32.7
STEGO w CRF	48.3	24.5	69.8	17.6	77.0	62.6	65.0 (+3.6)	34.9 (+2.2)
SmooSeg w/o CRF	60.6	25.2	79.8	18.0	81.4	68.4	73.9	37.2
SmooSeg w CRF	63.2	26.7	82.8	18.4	82.7	70.3	76.2 (+2.3)	38.5 (+1.3)
Table 6 demonstrates that SmooSeg still achieves state-of-the-art without CRF. Overall, the performance degradation in STEGO is notably more pronounced compared to SmooSeg. Importantly, the application of a CRF serves as an effective supplement to our smoothness prior. Moreover, we also present qualitative visualizations with and without CRF in Figures 9 and 10. It is found that CRF is able to refine the quality of fine details on both STEGO and SmooSeg. However, SmooSeg is consistently more semantically coherent than STEGO either with or without CRF.
To provide further evaluation, we have included some difficult samples from the COCOStuff dataset that were predicted by our Smooseg and STEGO in Figure 11. In these cases, Smooseg and STEGO tend to generate inaccurate semantic maps. However, it is noteworthy that even in challenging scenarios, SmooSeg consistently generates more semantically coherent segmentation maps compared to STEGO. This observation underscores the advantages of incorporating our smoothness prior in semantic segmentation tasks.
We provide the visualization of more results in Fig. 12 for the COCOStuff dataset, Fig 13 for the Cityscapes dataset, Figs 14 and 15 for the Potsdam-3 datasets. Overall, SmooSeg consistently produces more coherent segmentation maps when compared to both STEGO and TransFGU. These visual results provide further evidence of the efficacy of the smoothness prior in enhancing the label coherence and the overall segmentation quality in unsupervised settings.
Figure 9: Qualitative results of STEGO and SmooSeg with and without CRF on the COCOStuff dataset.
Figure 10: Qualitative results of STEGO and SmooSeg with and without CRF on the Cityscapes and Potsdam-3 datasets.
Figure 11: Difficult examples predicted by SmooSeg and STEGO on the COCOStuff dataset.
Figure 12: Additional qualitative results on the COCOStuff dataset. It's evident that SmooSeg produces higher-quality segmentation maps compared to STEGO and TransFGU.
Figure 13: Additional qualitative results on the Cityscapes dataset.
STEGO
Figure 14: Additional qualitative results on the Potsdam-3 dataset. The large-scale image has a resolution of  $4800 \times 4800$ , which will be partitioned into 225 sub-images with a resolution of  $320 \times 320$  before being input into each algorithm.
SmooSeg
Buildings + clutter
Roads + cars
Low vegetation + trees
STEGO
Figure 15: Additional qualitative results on the Potsdam-3 dataset.