SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets and Metrics. We extensively evaluate the proposed approach on two popular synthetic-to-real benchmarks, i.e., GTA5  $\rightarrow$  Cityscapes and SYNTHIA  $\rightarrow$  Cityscapes. GTA5 [37] is a large-scale driving-game dataset containing 24,966 images with 19 classes for pre-training. Synthia dataset [38] is rendered from a virtual city and contains 9,400 synthetic images with 16 classes for pre-training. The realistic dataset Cityscapes [10] collects street view scenes from 50 different cities with 19 classes, including 2,975 training images and 500 validation images. Following previous methods [30, 19], the semantic segmentation performance is evaluated by calculating the Mean Intersection-over-Union (mIoU) over 19 categories in GTA5  $\rightarrow$  Cityscapes, as well as 13 and 16 (with 3 small-scale categories) categories in SYNTHIA  $\rightarrow$  Cityscapes.
Implementation Details. Our approach is implemented based on the MMSegmentation framework [9] and one training task requires one NVIDIA Tesla V100 GPU. We deploy Swin-B [31] and MiT-B5 [43] as backbones and DAFformer [17] as decode head. During pre-training in the source domain, the backbone is initialized with weights pre-trained with the ImageNet dataset [12]. We utilize AdamW for optimization. Specifically, the learning rate of Swin-B encoder is set as  $6 \times 10^{-6}$  and  $4 \times 10^{-6}$  for the MiT-B5 encoder, while the learning rates of the segmentation head and prompt adapter are respectively set as ten and five times of backbone. Our Uni-UVPT framework typically needs 40k-80k iterations with a batch size of 1 until convergence. More details could be found in the appendices.
Table 1: Quantitative evaluations on GTA5  $\rightarrow$  Cityscapes and SYNTHIA  $\rightarrow$  Cityscapes tasks. Different segmentation architectures: F (FCN8s VGG-16), D (DeepLabv2 ResNet-101), S (Swin-B), M (MiT-B5). FB: whether the backbone is frozen. Params (M): number of trainable parameters. Bold: the best results based on different source pre-trained models. (+x.x): mIoU gains over the corresponding source pre-trained models where the best are in red. Underline: the state-of-the-art results. The full table with per-class IoUs is available in the appendices.
Methods	Arch	FB	Params (M)	GTA5 → Cityscapes	SYNTHIA → Cityscapes
mIoU19(%)	mIoU16(%)	mIoU13(%)
SFDA [30]	F	X	-	35.8	-	-
GtA [19]	F	X	134.5	45.9	41.3	48.9
URMA [14]	D	X	47.4	45.1	39.6	45.0
SRDA [5]	D	X	-	45.8	-	-
SFUDA [46]	D	X	-	49.4	-	51.9
BDT [20]	D	X	43.8	52.6	-	56.7
GtA [19]	D	X	43.8	53.4	52.0	60.1
Standard Single Source	S	X	90.7	50.5	44.6	49.8
CPSL [24]	S	✓	3.9	51.1 (+0.6)	46.4 (+1.8)	52.3 (+2.5)
VPT [18] + ELR [47]	S	✓	7.0	53.5 (+2.0)	47.7 (+3.1)	53.2 (+3.4)
Ours	S	✓	28.6	56.2 (+5.7)	52.6 (+8.0)	59.4 (+9.6)
Standard Single Source	M	X	85.2	52.5	48.6	55.0
CPSL [24]	M	✓	3.7	52.5 (+0.0)	50.5 (+1.9)	57.2 (+2.1)
VPT [18] + ELR [47]	M	✓	7.6	54.1 (+1.6)	51.6 (+3.0)	58.0 (+3.0)
Ours	M	✓	12.3	54.2 (+1.7)	52.6 (+4.0)	59.3 (+4.3)
Source-GtA [19]	S	X	110.4	52.8	48.8	55.0
CPSL [24]	S	✓	3.9	53.5 (+0.7)	49.6 (+0.8)	56.2 (+1.2)
VPT [18] + ELR[47]	S	✓	7.0	55.1 (+2.3)	51.6 (+2.8)	58.2 (+3.2)
GtA [19]	S	✓	23.6	56.1 (+3.3)	52.5 (+3.7)	58.7 (+3.7)
Ours	S	✓	28.6	56.9 (+4.1)	53.8 (+5.0)	60.4 (+5.4)
Source-GtA [19]	M	X	103.7	53.0	50.0	56.2
CPSL [24]	M	✓	3.7	53.2 (+0.2)	52.2 (+2.2)	58.7 (+2.5)
VPT [18] + ELR [47]	M	✓	7.6	54.4 (+1.4)	53.0 (+3.0)	59.5 (+3.3)
GtA [19]	M	✓	22.3	55.2 (+2.2)	53.6 (+3.6)	59.7 (+3.5)
Ours	M	✓	12.3	56.1 (+3.1)	53.8 (+3.8)	60.1 (+3.9)
3.2 Comparative Results
Baselines. We compare the proposed method with state-of-the-art source-free domain adaptive semantic segmentation methods, namely, SRDA [5], SFUDA [46], SFDA [30], URMA [14], BDT [20], and GtA [19]. We directly cite their results from their original papers. All the above methods deploy CNN backbones, not transformer ones. For a fair comparison, we implement GtA with transformer backbones based on official code  $^{2}$ . Besides, we utilize representative self-training with pseudo-label methods for unsupervised DA or source-free UDA, i.e., CPSL [24] and ELR [47], and visual prompt tuning method, i.e., VPT [18], to construct several competitors. Our baseline methods, CPSL [24], VPT [18] + ELR [47], GtA [19] are implemented transferring the authors' official releases  $^{3,4}$  to MMSegmentation framework [9]. We employ two different methods for source pre-training, i.e., standard single-source and Source-GtA [19]. The former directly utilizes source images and annotations for pre-training. The latter augments source data with five augmentations, and leverages auxiliary heads to learn powerful representations [19].
Results Analysis. We compare the proposed approach and the state-of-the-art SFUDA methods on GTA5  $\rightarrow$  Cityscapes and SYNTHIA  $\rightarrow$  Cityscapes tasks. As shown in Table 1, the proposed method with small amounts of learnable parameters performs better than comparative methods in two transfer tasks. For example, our method with 28.6M (31.5% of the source pre-trained model) trainable parameters increases mIoU by 8.0% (16 classes) and 9.6% (13 classes) in SYNTHIA  $\rightarrow$  Cityscapes task. The impressive performance demonstrates that the proposed unsupervised visual prompt tuning method could efficiently and effectively adapt large-scale source pre-trained model to the unlabeled target domain. Besides, we could conclude several interesting conclusions. (1)
$^{2}$ GtA: https://sites.google.com/view/sfdaseg
3VPT: https://github.com/KMnP/vpt
$^{4}$ CPSL: https://github.com/lslrh/CPSL
Figure 4: Qualitative results on GTA5  $\rightarrow$  Cityscapes.
Table 2: Ablation study on the prompt adapter. PG, PI and LE respectively denote prompt generator, prompt interactor and level embedding. The performance drop is over our complete approach.
PG	PI	mIoU (%)
Stem	LE
Multiscale	✓	✓	56.24 (Ours)
Multiscale	X	✓	55.58 ↓0.66
Singlescale	✓	✓	55.52 ↓0.72
X	✓	✓	55.50 ↓0.74
Multiscale	✓	PI 1	55.34 ↓0.90
X	X	X	55.07 ↓1.17
Table 3: Analysis on pseudo-label strategies.
Methods	mIoU (%)
Ours	56.24
ELR [47]	55.60
Ours + offline	55.47
Table 4: Analysis on consistency loss.
Feature	Prediction	mIoU (%)
✓	✓	56.24 (Ours)
X	✓	54.26
✓	X	56.01
X	X	53.81
Compared with SFUDA methods with CNN backbones, SFUDA methods with frozen transformer backbones significantly improve the performance of the target domain, illustrating that it is necessary to adapt the large-scale pre-trained model in domain adaptive semantic segmentation tasks. (2) CPSL underperforms other competitors, illustrating that simply applying pseudo-label correction is not effective in SFUDA. A similar conclusion has been claimed in ELR [47]. (3) VPT + ELR is a strong baseline by combining the visual prompt tuning and SFUDA pseudo-label strategies, which performs worse than our approach in all experimental settings. The possible reasons are as follows: First, visual prompts in VPT are designed as input tokens that have limited capacities to encode informative knowledge for semantic segmentation. Differently, our approach could capture rich information including multiscale spatial information and task-shared knowledge. Second, ELR roughly encourages model predictions to stick to labels generated by the frozen source model without any label correction. We argue that pseudo-label correction is still working but should be performed at suitable moments before the model memorizes noisy pseudo labels, which is supported by our superior performances. More analyses about the pseudo-label strategy refer to the ablation study. As shown in Figure 4, we provide some qualitative results of comparative methods in  $\mathrm{GTA5\rightarrow}$  Cityscapes task. The backbone of none-CNN-based methods is Swin-B, which is pre-trained via Soruce-GtA. Remarkably, our method performs favorably in comparison to other competitors. More visualization results are available in the appendices.
We have conducted extensive experiments on GTA5  $\rightarrow$  Cityscapes to study the contributions of each component in the proposed method. Please refer to the appendices for more details.
Whether can the prompt adapter learn reasonable prompts and facilitate feature generalization? The prompt adapter consists of a prompt generator (PG) and several prompt interactors (PI). The former can be further divided into stem and level embedding (LE) modules. As shown in Table 2, we have several interesting findings: (1) Without the level embedding, the mIoU drops  $0.66\%$ , illustrating that the level embedding could capture task-shared knowledge beneficial for the target domain. (2) To analyze the impact of the stem module, we deploy two variants. One is to replace multiscale feature maps with one feature map. The other is obtained by removing the stem module and repeating level embeddings to the same shape of visual prompts. Compared with single-scale feature map, multiscale feature maps bring  $0.72\%$  mIoU improvements. Besides, the mIoU drops  $0.74\%$  without the stem module, proving that multiscale spatial knowledge is helpful for prompt learning. (3) To testify whether multiple prompt interactors are necessary, we remove PI  $i, 2 \leq i \leq N$
and only preserve PI 1 for transforming the image token sequences. The  $0.9\%$  performance drop illustrates that it is necessary to perform prompt refinement and feature interaction between adjacent stages of the backbone. (4) Without the proposed prompt adapter, the performance decreases by  $1.17\%$ . The performance drop is smaller than other variants because head tuning also benefits feature adaptation. Despite that, it proves that the prompt adapter could capture informative knowledge for reasonable prompt learning and enhancing the generalization of target features.
Whether it is necessary to perform pseudo-label correction in source-free UDA? Li et al. [47] claim that pseudo-label correction strategies perform poorly due to the early-learning phenomenon and even no better than directly supervising the target model with pseudo labels predicted by the frozen source model. We argue that the pseudo-label correction should be conducted at suitable moments before the model memorizes noisy pseudo-labels. As shown in Table 3, compared with ELR [47], the proposed adaptive pseudo label correction strategy improves mIoU by  $0.64\%$ , demonstrating that correcting pseudo-label at suitable moments helps to learn more trustable pre-trained knowledge. As for Ours + offline, we first train the model to get the training IoU curve for each category and then correct pseudo labels with our strategy only once. The mIoU of Ours + offline decreases  $0.77\%$ , illustrating that pseudo-label correction should be performed adaptively otherwise the noisy pseudo labels are memorized again after several training epochs.
Effectiveness of multiscale consistency loss. The multiscale consistency loss performs feature and prediction consistency of rescaled inputs. As shown in Table 4, without feature consistency, the mIoU drops dramatically as visual prompts can not guarantee that features of different scales of one image are similar. In other words, the feature consistency enhances the robustness of visual prompts to spatial perturbations. Similarly, prediction consistency benefits generating scale-robust predictions. The overall consistency loss increases mIoU by  $2.43\%$ .
3.4 Further Remark
Comparative Results of Different Augmentations. To analyze the influences of other augmented samples, we replace scale augmentation with weather augmentation, e.g., snow and frost. That is, we replace downscale/original/upscale images with snow/original/frost images for unsupervised visual prompt tuning. We summarize the performance of Ours and Ours-weather in Table 5. Ours refers to our approach trained with scale-augmented samples. Ours-weather is our approach trained with weather-augmented samples. Compared with Ours, Ours-weather improves mIoU approximately  $-1.5\%$  to  $-0.1\%$  in GTA  $\rightarrow$  Cityscapes task and  $-0.1\%$  to  $+1.6\%$  in SYNTHIA  $\rightarrow$  Cityscapes task. The experimental results illustrate that other augmented samples also work well for unsupervised visual prompt tuning and the SOTA results in SYNTHIA  $\rightarrow$  Cityscapes tasks are updated by weather augmentation.
Table 5: Comparative results of different augmentations on GTA5  $\rightarrow$  Cityscapes and SYNTHIA  $\rightarrow$  Cityscapes tasks. Different segmentation architectures: S (Swin-B), M (MiT-B5). FB: whether the backbone is frozen. Params (M): number of trainable parameters. (+x.x): mIoU gains over the corresponding source pre-trained models.
Methods	Arch	FB	Params (M)	GTA5 → Cityscapes	SYNTHIA → Cityscapes
mIoU19(%)	mIoU16(%)
Ours	S	✓	28.6	56.2 (+5.7)	52.6 (+8.0)
Ours-weather	S	✓	28.6	54.7 (+4.2)	52.9 (+8.3)
Ours	M	✓	12.3	54.2 (+1.7)	52.6 (+4.0)
Ours-weather	M	✓	12.3	54.1 (+1.6)	53.0 (+4.4)
Ours	S	✓	28.6	56.9 (+4.1)	53.8 (+5.0)
Ours-weather	S	✓	28.6	54.1 (+1.6)	53.0 (+4.4)
Ours	M	✓	12.3	56.1 (+3.1)	53.8 (+3.8)
Ours-weather	M	✓	12.3	55.2 (+2.2)	54.5 (+4.5)
Visualization Results of Adaptive Pseudo-Label Correction Strategy. To emphasize the effectiveness of the proposed adaptive pseudo-label correction strategy, we draw the  $\mathrm{IoU}_{el}$  and  $\mathrm{IoU}_m$  curves for each category as shown in Figure 5. We observe that the  $\mathrm{IoU}_{el}$  curves of all categories increase until the model converges and the phenomenon of  $\mathrm{IoU}_{el}$  decreasing due to the noise memorization are disappeared. The above interesting observations demonstrate that the proposed adaptive pseudo-label
correction strategy could effectively alleviate the negative effect caused by pseudo-label noises. It also proves that pseudo-label correction works in source-free unsupervised domain adaptation when it is performed at suitable moments.
Figure 5: Analysis on early learning  $(\mathrm{IoU}_{el}$ , blue curves) and memorization  $(\mathrm{IoU}_m$ , red curves) with our proposed adaptive pseudo-label correction strategy for different categories in Cityscapes dataset. The LCM (Label Correction Moment) denotes the moment when the algorithm performs pseudo-label correction.
Source-Free Unsupervised Domain Adaptation for Semantic Segmentation. Source-free unsupervised domain adaptation has attracted much attention when source data is not available during model adaptation. Most SFUDA methods [21, 23, 26] are proposed for classification task, which is suboptimal for cross-domain semantic segmentation. Only a few methods [30, 19, 20, 46] focus on source-free domain adaptive semantic segmentation. For example, SFDA [30] not only recovers and preserves the source domain knowledge via knowledge transfer, but also distills valuable information from the target domain for self-supervised learning. Kundu et al. [19] design a multi-head framework to extract reliable target pseudo labels for self-training. Kundu et al. [20] further introduce an intermediate mixup domain between the original and the realizable generic domain, which could enhance the trade-off between discriminability and transferability. Ye et al. [46] improve the target domain performance by reconstructing the source domain data and performing distribution-aware adversarial learning and self-training. However, current source-free methods adopt fine-tuning for model adaptation, which is inefficient for large-scale transformer-based architectures.
Visual Prompt Tuning. Recently, visual prompt tuning has been introduced into the vision area [3, 32, 40] to generalize the large-scale pre-trained model to downstream tasks. Previous methods design visual prompts from the view of token-level, pixel-level and module-level. Token-level prompt tuning methods [18, 15, 33, 40] govern learnable visual tokens to the embedding layer or several encoder layers. Pixel-level prompt tuning methods [4, 6, 36] introduce learnable pixel perturbations on the original images. However, token-level or pixel-level prompts are black boxes, which cannot capture convincing knowledge beneficial for downstream tasks. Module-level prompt tuning methods [49, 48, 7] design auxiliary layers or blocks for the pre-trained model. Despite learning rich knowledge, they are specifically designed for the pre-trained model by modifying its basic units, which can not be applied to various architectures. As far as we know, only [15] has applied visual prompts for online unlabeled adaption in image classification, and none of the work has explored unsupervised visual prompt tuning for source-free domain adaptive semantic segmentation.
In this paper, we propose a universal unsupervised visual prompt tuning (Uni-UVPT) framework, which adapts a large-scale pre-trained source model for source-free domain adaptive semantic segmentation. A novel prompt adapter is proposed to progressively encode informative knowledge into prompts and enable target features to match the pre-trained model. Cooperatively, an adaptive pseudo-label correction strategy with a multiscale consistency loss is proposed to correct pseudo-label at suitable moments and enhance visual prompts' spatial robustness. Extensive experiments prove that our approach could effectively and efficiently perform large-model unsupervised knowledge transfer. Our approach also has its limitations, for example, further reducing the trainable parameters by delicately designing prompt modules and exploring more powerful learning constraints for visual prompts, e.g., pixel-wise contrastive loss. We hope this paper could inspire researchers to further investigate unsupervised visual prompt tuning for other vision applications.
We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
[41] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415-424, 2022.
[42] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In ICCV, pages 22-31, 2021.
[43] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, NeurIPS, volume 34, pages 12077-12090. Curran Associates, Inc., 2021.
[44] Shiqi Yang, Yaxing Wang, Joost Van De Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In ICCV, pages 8978-8987, 2021.
[45] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In CVPR, pages 4085-4095, 2020.
[46] Mucong Ye, Jing Zhang, Jinpeng Ouyang, and Ding Yuan. Source data-free unsupervised domain adaptation for semantic segmentation. In ACM MM, pages 2233-2242, 2021.
[47] Li Yi, Gezheng Xu, Pengcheng Xu, Jiaqi Li, Ruizhi Pu, Charles Ling, Ian McLeod, and Boyu Wang. When source-free domain adaptation meets learning with noisy labels. In ICLR.
[48] Bruce XB Yu, Jianlong Chang, Lingbo Liu, Qi Tian, and Chang Wen Chen. Towards a unified view on visual parameter-efficient transfer learning. arXiv preprint arXiv:2210.00788, 2022.
[49] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint arXiv:2206.04673, 2022.