When Visual Prompt Tuning Meets Source-Free Domain Adaptive Semantic Segmentation
Authors: Xinhong Ma
Yiming Wang
Hao Liu
Tianyu Guo
Yunhe Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: unsupervised domain adaptation
semantic segmentation
visual prompt tuning

Abstract: Source-free domain adaptive semantic segmentation aims to adapt a pre-trained source model to the unlabeled target domain 
without accessing the private source data. Previous methods usually fine-tune the entire network, which suffers from expensive parameter tuning. To avoid this problem, we propose to utilize visual prompt tuning for parameter-efficient adaptation. However, the existing visual prompt tuning methods are unsuitable for source-free domain adaptive semantic segmentation due to the following two reasons: (1) Commonly used visual prompts like input tokens or pixel-level perturbations cannot reliably learn informative knowledge beneficial for semantic segmentation. (2) Visual prompts require sufficient labeled data to fill the gap between the pre-trained model and downstream tasks. To alleviate these problems, we propose a universal unsupervised visual prompt tuning  (Uni-UVPT) framework, which is applicable to various transformer-based backbones. Specifically, we first divide the source pre-trained backbone with frozen parameters into multiple stages, and propose a lightweight prompt adapter for progressively encoding informative knowledge into prompts and enhancing the generalization of target features between adjacent backbone stages. Cooperatively, a novel adaptive pseudo-label correction strategy with a multiscale consistency loss is designed to alleviate the negative effect of target samples with noisy pseudo labels and raise the capacity of visual prompts to spatial perturbations. Extensive experiments demonstrate that Uni-UVPT achieves state-of-the-art performance on GTA5 $\to$ Cityscapes and SYNTHIA $\to$ Cityscapes tasks and can serve as a universal and parameter-efficient framework for large-model unsupervised knowledge transfer. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/uni-uvpt and https://github.com/huawei-noah/noah-research/tree/master/uni-uvpt.

When Visual Prompt Tuning Meets Source-Free Domain Adaptive Semantic Segmentation
Xinhong Ma, Yiming Wang, Hao Liu, Tianyu Guo, Yunhe Wang"
Huawei Noah's 's Ark Lab
(maxinhong, wangyiming22, 1iuhao296, tianyu.guo, yunhe. wan..wangn)) @haawe..co
Abstract
Source-free domain adaptive semantic segmentation aims to adapt a pre-trained source model to the unlabeled target domain without accessing the private source data. Previous methods usually fine-tune the entire network, which suffers from expensive parameter tuning. To avoid this problem, propose to utilize visual prompt tuning for parameter-efficient adaptation. However, the existing visual prompt tuning methods are unsuitable for source-free domain adaptive semantic segmentation due to the following two reasons: (1) Commonly used visual prompts like input tokens pixel-level perturbations cannot reliably learn informative knowledge beneficial for semantic segmentation. (2) Visual prompts require suffi- cient labeled data to fill the gap between the pre-trained model and downstream tasks. To alleviate these problems, we propose a universal unsupervised visual prompt tuning (Uni-UVPT)) framework, which is applicable various transformer- based backbones. Specifically, we first divide the source pre-trained backbone with frozen parameters into multiple stages, and propose lightweight prompt adapter for progressively encoding informative knowledge into prompts and en- hancing the generalization of target features between adjacent backbone stages. Cooperatively, a novel adaptive pseudo-label correction strategy with a multiscale consistency loss is designed to alleviate the negative effect of target samples with noisy pseudo labels and raise the capacity of visual prompts to spatial perturbations. Extensive experiments demonstrate that Uni-UVPT achieves state-of-the-a perfor- mance on GTA5 + Cityscapes and SYNTHIA - Cityscapes tasks and can serve as universal and parameter-efficin knowledge transfer. Code will be available at / /eteee...eeeeeeeeeee/ com monm/mindsorre////o hndeel/i/heeeeasteeee//rrrr////r and https: //gittub//tttttttg/ com/
framework for large-model unsupervised
huawei hhaaaa/rrarrrcccaarrreeee/////es
Introduction
Semantic segmentation a critical computer vision task, which aims to segment and parse scene image into different image regions associated with semantic categories. The success of these semantic segmentation techniques relies on large-scale densely-labeled datasets. However, it prohibitively expensive to collect high-quality annotations. Besides, most semantic segmentation methods ignore the distribution shift between training and testing data, making them fail to generalize when deployed in conditions different from training such as cross-city [8] cross-weather [39] scenarios. Unsupervised Domain Adaptation (UDA) [ is an intuitive direction to address the above two issues by transferring knowledge from existing models trained source datasets to the unlabeled target domain. Typical UDA approaches require joint access to both labeled source and unlabeled target data during training, making them unsuitable for Source- Free Unsupervised Domain
"Correspondng author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Adaptation (SFUDA) where source and target data are not concurrently accessible. This SFUDA setting with restricted data-sharing is privacy-oriented and thus holds immense practical value in autonomous driving and other commercial visual applications.
With the above considerations, this paper proposes to solve source-free domain adaptive semantic segmentation task, where only source pre-trained models and unlabeled target data are available for model adaptation. Recently, few source-free UDA methods have been developed to tackle
semantic segmentation [30
19.
20 46] and image classification [21 23 26 Most of them either
perform self-training [30
or synthesize target-style training data (35 fine-tune all model
parameters. However, line-tuning requires many computation resources, which is an expensive and
often infeasible proposition for previous SFUDA methods, especially for modern transformer- based architectures. Recently, prompt tuning has been proposed to explore the knowledge of frozen language
models 22 25 28 29], which could be an alternative strategy to efficiently adapt the pre-trained source model to the target domain.
Prompt Tuning aims design trainable lightweight block as supplementary input (prompt) for a frozen model, which could guide or direct the generalization of powerful representations to achieve desirable performances. Inspired by its great success in natural language processing (NLP) 228. few methods in computer vision propose to add small amounts of learnable parameters as tokens adapt large pre-traines trained models to downstream tasks. However, the existing visual prompt tuning methods have two limitations when applied to source-free domain adaptive semantic segmentation: (1) The learned visual prompts are unreasonable. Specifically, prompts like input tokens or pixel-level perturbations are black boxes, which cannot reliably explore convineing knowledge beneficial for pixel-wiss predictions. Besides, most methods utilize task- oriented loss for optimization, e.g., cross-entropy loss, which limits the learning capacities of visual prompts. (2) Previous methods cannot be directly applied to the unlabeled target domain because most of them rely sufficient labeled data to deal with the gap between pre-trained models and downstream tasks. There is a lack of methods addressing unsupervised visual prompt tuning where only massive unlabeled data is available for training.
[8 or pixel-level perturbations
To address the above problems, we propose a Universal Unsupervised Visual Prompt Tuning (Uni- UVPT) framework for source-free domain adaptive semantic segmentation. Specifically, given a source pre-trained model, we partition its backbone with frozen parameters into multiple stages and design lightweight prompt adapter for stage-wise prompt tuning. The prompt adapter contains prompt generator and several prompt interactors. The former aims to generate reasonable prompts that capture multiscale spatial information and task-shared knowledge. The latter progressively refines prompts between adjacent stages and transforms target features to match the source pre-trained knowledge in the backbone. To learn prompts with massive unlabeled target samples, we utilize target instances with pseudo labels for self-training, and propose an adaptive pseudo-label correction strategy to alleviate pscudo-label noises. The proposed adaptive pscudo-label correction strategy could determine suitable moments to rectify pseudo labels and guide the model to train on instances with corrected pseudo labels, not noisy ones. Mcanwhile, multiscale consistency loss is proposed to impose multiscale consistency of the features and predictions so that the learned visual prompts robust to arbitrary semantic-preservinn spatial perturbations. It is noticed that our method is applicable to various transformer architectures without modifying their basic units.
The contributions of this work are summarized follows: (1) We first highlight the low-efficiency problem of fine-tuning large-scale backbones in source-free domain adaptive semantic segmentation, and propose universal unsupervised visual prompt tuning framework for parameter-efficien model adaptation. (2) A lightweight prompt adapter is introduced to learn reasonable visual prompts and enhance feature generalization in a progressive manner. Cooperatively, novel adaptive pscudo-label correction strategy is proposed to rectify target pscudo labels at suitable moments and improve the learning capacity of visual prompts. Extensive experimental results demonstrate that our method with a few trainable parameters could achieve state-of-the-an performances on GTA5 -> Cityscapes and SYNTHIA -> Cityscapes tasks.
2 Universal Unsupervised Visual Prompt Tuning
In unsupervised domain adaptation, labeled source dataset D, = ((xx,, : ax E Xs, yy E C} and an unlabeled target dataset D, {Ir 2t E X3 are available for training, where X is the input space
2

---

and C denotes the label set. F, is drawn from the marginal distribution while 2, is drawn from another marginal distribution Pt. The goal is learn mapping f X - C that can generalize well for Fz. We focus on the source-free constraint (3), where we do not have access to the original source domain data D,.. Instead, merely source pre-trained model f. and unlabeled target data Dr are provided for adaptation. The source pre-trained semantic segmentation model usually contains a backbone fo (containing some embedding layers, e.g., the patch embedding layer in Swin (31)) and a customized head fr. To efficiently adapt the source pre-trained model fa to the unlabeled target domain D., we propose to perform unsupervised visual prompt tuning. When designing our algorithm, we consider the following two questions: (1) how to design informative visual prompts for source-free domain adaptive semantic segmentation and (2) how to learn visual prompts with unlabeled samples for model adaptation. To this end, we propose a universal unsupervised visual prompt tuning (Uni-UVPT) framework as shown in Figure Specifically, we design a novel prompt adapter for generating informative visual prompts and improving the generalization of target features. Besides, an effective adaptive pseudo-label correction strategy with a multiscale consistency loss is proposed for learning visual prompts with massive unlabeled target data and enhancing visual prompts' capacity for spatial perturbations. As the backbone fo is frozen and only the parameters of the proposed prompt adapter and segmentation head are optimized during training, Uni-UVPT is parameter-efficient for model adaptation. Besides, Uni-UVPT could be directly deployed to various transformer- based architectures without modifying the basic units.
2.1 Prompt Adapter
As shown in Figure the proposed lightweight prompt adapter contains a prompt generator and several prompt interactors. The former aims to generate informative prompts that capture multiscale spatial information and task-shared knowledge. The latter is designed to refine prompts with pre- trained knowledge and transform target features match the pre-trained knowledge in the backbone. To interact with target features via visual prompts, we manually partition the backbone fú into N (usually N 4) stages, each of which composed of multiple basic units. Given a target image Iy- it is first fed into the embedding layer to obtain patch sequences Fout. Foou is then regarded as an input of the frozen backbone. Mcanwhile, the target image Ie is also fed into the prompt generator to output an initial prompt Co that contains L dimensional spatial features of multiple resolutions. Then, the initial prompt is flattened and concatenated as an input of the prompt interactor. For stage of the backbone, its input feature and output feature are respectively denoted as Fi and Fout, Before stage i, the prompt interactor incorporates the output feature F and the interim prompt C,-1 of previous stage to obtain the refined prompts C2 and a suitable input Fr for the stage i. Finally, the output feature Fxot of the last stage is fed into head fn. for segmentation prediction. In the following sections, we introduce the details of the proposed prompt generator and prompt interactor.
Prompt Generator. Previous methods 4 18 have proved that learning prompts brings flexibility to the pre-trained model. However, their prompts like input tokens or lixel-level perturbations are black boxes with limited learning capacity, which cannot reliably explore convincing knowledge beneficial for semantic segmentation. Therefore, we propose to design informative prompts for each image to capture multiscale spatial prior and the task-d knowledge. To achieve this goal, we utilize a standard convolutional stem Jstem borrowed from ResNet [66 to extract multiscale spatial information, because convolution could help transformers better capture the local spatial information [41 42 [ 34). fatem consists of three convolutions, a maxxpoooing layer, and a stack of stride-2 3 3 convolutions to double the channels and reduce the size of feature maps. Several x 1 convolutions are applied at the end to project the feature maps to L dimensions. Finally, we obtain feature pyramid S (s1, 82, 83). }. which contains L-dimensional feature maps with 1/2, 1/4, and 1/8 resolutions of the original image. Besides, following the spirits of previous methods [8,88, we leverage group of trainable vectors Q = {91. . 9s } named level embedding, to learn thar-d knowledge, which contains three L-dimensional vectors initialed by the Gaussian function. In this way, the prompt pyramid Co = {ci, 22, ca of the input image T₁ E |x could be obtained by
G = k (81, q:)
(I)
where x (+. :) could be any feature fusion operations. Here, we first repeat s; several times so that the vectors can be resized as the same shape of q. and then perform element-wise addition. Finally, we flatten and concatenate the prompt pyramid Co into prompt tokens Co E TR. (HW, HW Hy )X) as the input of prompt interactor for prompt refinement and feature adaptation.
3

---

Target Sample
Frozen
Data Flow
Element
wise Addition
Learnable
Prompt
Embeddingg.g Layer
A
Target Sample
Training loU+
Co
Memory Queue
G
Stage 1
Augmentations
Eq
prouc
Eq (5)
Prompt Generator (PG)
Segmentation Model
False
True
Cj
Stage:
query
y,=
Eq (6)
key/value
Cross Attention
Backbone
Cy-t
Features
1
Pseudo Label 9,
Predictions
P
key/value
Stage
Cross Attention
PUBLE
1-1
Head
Multiscale Consistency Loss
Self-Training
Predictions
F
Loss
(a) Pre-trained Model (b) Prompt Adapter (d) Prompt Interactor / (P)
(e) Adaptive Pseudo-Label Correction
The architecture of Uni-UVPT. (a) The source pre-trained segmentation model, whose encoder layers are divided into N stages, (b) The proposed prompt adapter, which contains two designs, including (c) a prompt generator for encoding rich knowledge into prompts, and (d) prompt interactor for prompt refinement and feature interaction.
Figure 1:
(e) The proposed adaptive pseudo-label correction strategy, which rectifies pseudo-labed at suitable moments.
Prompt Interactor. To progressively refine prompts and improve feature generalization via the refined prompts, we design novel prompt interactor based on cross-attention. As shown in Figure [ for the i-th stage of the backbone, we use the interim prompt C,-1 of the previous stage as the query, and the output feature F the key and value for cross-attention that the pre- pre-trained knowledge contained in Fort are injected into prompts. This process be formulated as:
C C + Actention(norm(( norm( (F?r)))
where the norm(-) is LayerNorm attention. After that, we utilize the refined prompt C, to generate adapted input F7"" for the next stage of the backbone. This process can formulated as:
and the attention layer Attention(-) suggests using sparse
F FPut % Attention(norm norm(C;)),
where sparse attention is adopted to reduce computational costs. Besides, we apply learnable vector %% E _ balance the attention layer's output and the feature Fost, which initialized with This initialization strategy ensures that the feature distribution Fo will not modified drastically due to the injection of the prompt C, thus making better use of the pre-trained knowledge of backbone.
2.2 Adaptive Pseudo-Label Correction
To learn visual prompts with massive unlabeled target data, we propose utilize high-quality pseudo labels of target samples for self-training. The pipline of the adaptive pseudo-label cor- rection is shown in Figure (e). Previous SFUDA methods generate pseudo labels by learning meaningful cluster structures in the feature space and the quality of the learned cluster structures hinges on the reliability of pseudo labels generated by the source model. Therefore, the pseudo labels noisy due to the domain shift. Recently, Li al. [47] have formulated SFUDA as

---

learning with label noise problem, and observed early-learning phenomenon that deep neural net- works tend to first fit the training data with correct pseudo labels during an early-learning phase, before eventually memorizing the instance with incorrect/noisy pseudo labels. Not only in im- age classification, we also observe a similar phenomenon in semantic segmentation, where the pseudo-label noise is ubiquitous across samples and distributed in a pixel-wise manner. As shown in Figure 2 we analyze the learning curves on the pseudo labels predicted by the source model during the training process. As the model learning is supervised by noisy pseudo labels, the IoUm curves for all categories increase substantially as training proceeds. The IoU follows a com- pletely different trajectory: it first increases during an early-learning stage where the model learns to correctly segment the incorrectly-labelee pixels, but eventually decreases as memorization occurs.
Road
Motorcycle
Fence
94
84
50
40
ToUn 1%)
93
totl 199
IoU.. (%)
Jobl 190
IaUm
83
loUm m
92
IoUm
49
loUm
39
IoUel el
68
ToUer
56
IoU
91
67
48
55
38
6
Training Process (x (x10* iters)
Training Process (x 10f iters)
Training Process (x10f iters)
Effect of early learning (IoUel, blue curves) and memorization (IoUm- red curves) for different categories in Cityscapes dataset. The IoUet is the IoU between the model output and ground-trun -ruth labels. The IoU,, represents the IoU between the model output and pseudo labels.
Figure 2:
To alleviate the memorization of noisy pseudo la- bels in SFUDA, Li et al. 47 roughly encourage model predictions to stick to labels generated by the frozen source model without any label correction. Differently, we design an adaptive pseudo-label correction strategy based on the early-time training phenomenon, which rectifies target pscudo labels for each category before the model memorizes the noisy pscudo labels. Naturally, it raises question that how to deter- mine when to correct the noisy pseudo labels. Fortunately, shown in Figure 2 we observe that the segmentation performance on the train- ing set (IoU,,) improves rapidly during early learning, and then much more slowly during memorization. In other words, the performance deceleration indicates whether the model over- fits noisy pseudo labels. To estimate the deceleration, we first fit the following exponential parametric model to the training IoU using the least squares:
Xx1
Downscale
Upscale
Shared paramebers
Shared parameters
Segmantation
Segmentation Modely
Segrenntatio Model
Model
FM
Y2
F
y's
Upscale
1
Doumscale]
PIN
P
Fa
FM
Figure 3: Multiscale consistency loss for rescaled images. The rescaled features of F. of backbone utilized for feature consistency and the rescaled model predictions il for prediction consistency.
g(t) = at' + + +t
where t represents training time, and o, b, and d are fitting parameters. However, it is time- consuming to calculate the training IoU on all training samples. We propose to utilize a memory queue with samples as an alternative, which is updating with batch data. Then, compute the derivative g' (4). and the pseudo labels for each category are corrected when the relative change of g'(t) is above certain threshold ;:
lg' (to) - (0) [g' (to)
> T.
(5)
where to is the start time of the carly-learning stage in correction loop. The image pixels with predictive confidence over 7% per class on the entire target training set are chosen for psbudo-label correction. For all experiments, we set r = 0.9 and 7 = 0.65. After the pseudo-label correction, the model learning steps into a new early-learning stage. It noticed that the pseudo-label correction loop will perform several times until achieving the max training iterations.
5

---

To form the final pseudo labels, we average the model outputs corresponding to multiple rescaled copies of inputs. Specifically, let m be the number of scaling operations. We m 3 (downscaling x0.5, no scaling, and upscaling x 2) in our experiments. For an input rescaled according to these scaling operations in Figure(3) 3 yk: ; 1 I k S m denotes the corresponding model predictions and ti obtained by rescaling Vv, as the same size of dig- The corrected pseudo label Dr can be obtained by averaging predictions of multiple rescaled Zy:
Dr
MA k=1
where only categories satisfying Eq (5) require correction while others remain unchanged.
Once the pseudo labels all training samples are corrected, the model steps into a new carly-learning stage via minimizing self-training loss Let shown in Eq (7 and waits for the next correction process. St = >D 1- log / (x+)² + Euy,D, [- [-log] f (a+)),
(7)
where ( is dot product, and (zz) denotes the model output. The first term is cross-entropy loss based on image and pseudo-label pair (z,, (t). The second term is predictive entropy that could reduce model uncertainty target samples during self-training [19]
Besides, we incorporate multiscale consistency loss Lime promote feature consistency and prediction consistency, as shown in Eq
122 KL. (ūk (t)
L.me BerYNDD
Σ
Frin
+ E
B
(8)
m
feature consistency Lyc where a and 3 are balance parameters, and KL denotes the Kullback-Leiberr divergence. As shown in <is N represents the rescaled features of backbone stages and their Figure Fir < k < m, <
prediction consistency Zye
En= fiin The first term the feature consistency regularization Lic. which constrains the visual prompis preserve the semanties of features and meanwhile eliminate the spatial perturbations so that the prompt adapter could generate scale-robust prompts. Similarly, the rest prediction consistency term Le could encourage the model to produce predictions that are robust to arbitrary semantic-preserve spatial perturbations. The final objective is defined as the sum of the self-training loss and multiscale consistency loss:
average denoted as Frim
L Cst + Lme-
More details about the training process introduced in the appendices.
3 Experiment
3.1 Evaluation Setup
Datasets and Metrics. We extensively evaluate the proposed approach on two popular synthetic-to- real benchmarks, iee., GTA5 -> Cityscapes and SYNTHIA - Cityscapes. GTAS [37 is large-scale driving-gan gaag-ggameea dataset containing 24,966 images with 19 classes for pre-training. Synthia dataset [38] rendered from virtual city and contains 9,400 synthetic images with 16 classes for pre- pre-training. The realistic dataset Cityscapes [10 collects street view scenes from 50 different cities with 19 classes, including 2.975 training images and 500 validation images. Following previous methods (30 19), the semantic segmentation performance is evaluated by calculating the Mean Intersection-overs Union (mIoU) over 19 categories in GTA5 - Cityscapes, as well as 13 and 16 (with small-scale categories) categories in SYNTHIA - Cityscapes.
Implementation Details. Our approach is implemented based on the MMSegmentation framework 9 and one training task requires one NVIDIA Tesla V100 GPU. We deploy Swin-B (31 and MiT-B5
49 as backbones and DAFormer [17] as decode head. During pre-training in the source domain, the backbone is initialized with weights pre-trained with the ImageNet dataset [12.. We utilize AdamW
for optimization. Specifically, the learning rate of Swin-B encoder is set as 10-6 and 4 x 10-6 for the MīT-B5 encoder, while the learning rates of the segmentation head and prompt adapter are respectively set as ten and five times of backbone. Our Uni-UVPT framework typically needs 40k-80k iterations with batch size of I until convergence. More details could be found in the appendices.

---

Table 1: Quantitative evaluations on GTAS - Cityscapes and SYNTHIA - Cityscapes tasks. Different segmentation architectures: (FCN8s VGG-16), D (DeepLabv2 ResNet-101), (Swin-B), M (MiT-B5). FB: whether the backbone is frozen. Params (M): number of trainable parameters. Bold: the best results based on different source pre-trained models. (+x.x): mloU gains over the corresponding source pre-trained models where the best are in red. Underline: the state-of satte-of-ther of-the-art results. The full table with per-class IoUs available in the appendices.
GTAS - Cityscapes
SYNTHIA Cityscapes
Methods
Arch
FB
Params (M)
mloU19(%)
mloU1a(% (%)
mloU1s(%)
SFDA 330 GtA 19
X
35.8
X
134.5
45.9
41.3
48.9
URMA 14
D
X
47.4
45.1
39.6
45.0
SRDA 15
X
45.8
SFUDA [46]
X
49.4
51.9
BDT 20
X
43.8
$2.6
56.7
GtA 19
X
43.8
53.4
52.0
60.1
Standard Single Source
x
90.7
50.5
44.6
49.8
CPSL 224
3.9
51.1 (+0.6)
46.4 (+1.8)
52.3 (+2.5)
VPT [18]
ELR 47
7.0
53.5 (+2.0)
47.7 (+3.1)
$3.2 (43.4)
Ours
28.6
56.2 (+5.7)
528.0)
59.4 (49.6)
Standard Single Source
M
X
85.2
52.5
48.6
55.0
CPSL [24
3.7
$2.5 (+0.0)
(01.9))
$7.2 (+2.1)
VPT [18] ELR
7.6
54.1 (+1.6)
(1360)
58.0 (+3.0)
Ours
M
12.3
54.2 (+177
52.6
59.3 (+4.3)
Source- GLA
19
S
110.4
$2.8
48.8
55.0
CPSL 224
3.9
53.5 (+0.7)
19,6
56.2
VPT [8] + ELR
7.0
55.1 (+2.3)
51.6
58.2 (43.2)
GtA [19
23.6
56.1 (+3.3)
52.57)
58.7 (+3.7)
Ours
28.6
56.9 (44.1)
53.8 (+5.0)
60.4 (+5.4)
Source-Gin GIA 19
X
103.7
53.0
50.0
56.2
CPSL 24
3.7
$3.2 (+0.2)
52.2(+..)
58.7 (42.5)
VPT [18]
ELR
M
7.6
54.4 (+1.4)
53.0 (+3.0)
59.5 (43.3)
GtA 19
M
22.3
55.2 (+2.2)
53.6 (+3.6)
59.7 (43.5)
Ours
12.3
56.1 (+3.1)
53.8 (+3.8)
60.1 (+3.9)
3.2 Comparative Results
Baselines. We compare the proposed method with state-of-the-ard source-free domain adaptive se- mantic segmentation methods, namely, SRDA [5), SFUDA (46), SFDA [30], URMA [14,, BDT (20), and GtA 19I. We directly cite their results from their original papers. All the above methods deploy CNN backbones, not transformer ones. For a fair comparison, we implement GtA with transformer backbones based official code Besides, we utilize representative self self-training with pseudo-label methods for unsupervised DA or source-free UDA, i.e., CPSL (24 and ELR [47], and visual prompt tuning method, i.e., VPT (18), to construct several competitors. Our bascline methods,
CPSL 224, VPT [ [47), GtA 19 are implemented transferring the authors' official releases F to MMSegmentttion framework 9 We employ different methods for source pre-training, Le., standard single-source and Source-GtA [9]. The former directly utilizes source images and annotations for pre-training. The latter augments source data with five augmentations, and leverages auxiliary heads to learn powerful representations [19].
Results Analysis. We compare the proposed approach and the state-of-the-art SFUDA methods on GTAS - Cityscapes and SYNTHIA ~ Cityscapes tasks. As shown Table T the proposed method with small amounts of learnable parameters performs better than comparative methods in two transfer tasks. For example, our method with 28.6M (31.5% the source pre-trained model) trainable parameters increases mloU by 8.0% (16 classes) and 9.6 % (13 classes) in SYNTHIA Cityscapes task. The impressive performance demonstrates that the proposed unsupervised visual prompt tuning method could efficiently and effectively adapt large-scale source pre-trained model to the unlabeled target domain. Besides, we could conclude several interesting conclusions. (1)
GtA:
PVPT: http://githubcom/KMMiiyy 'CPSL: http:://gituub.com/////cccC

---

Image
Deeplabb2)
Source-GiA
Ground
Figure 4: Qualitative results on GTAS + Cityscapes.
Table 2: Ablation study on the prompt adapter. PG, PI and LE respectively denote prompt generator. prompt interactor and level embedding. The per- formance drop is over our complete approach.
Table 3: Analysis on pseudo-label strategies.
Methods
mloU (%)
Ours ELR (47
$6 26.24
55.60
PG
mloU (%)
Ours offline
55.47
Stem
LE
Table 4: Analysis on consistency loss.
Multiscalc Multiscale
56.24 (Ours)
X
/
55.58 10.66
Feature
Prediction
mloU (%)
Singlescale
55.52 J0.72
56.24 (Ours)
X Multiscale
55.50
X
$4.26
PI
55.34 J0.90
56.01
X
X
55.07 J1.17
x
53.81
Compared with SFUDA methods with CNN backbones, SFUDA methods with frozen transformer backbones significantly improve the performance of the target domain, illustrating that necessary to adapt the large-scale pre-trained model in domain adaptive semantic segmentation tasks. (2) CPSL underperforms other competitors, illustrating that simply applying pseudo-label correction is not effective in SFUDA. A similar conclusion has been claimed in ELR [47]. (3) VPT ELR strong baseline by combining the visual prompt tuning and SFUDA pseudo-label strategies, which performs worse than our approach in all experimental settings. The possible reasons are as follows: First, visual prompts VPT are designed as input tokens that have limited capacities to encode informative knowledge for semantic segmentation. Differently, our approach could capture rich information including multiscale spatial information and task-shared knowledge. Second, ELR roughly encourages model predictions to stick to labels generated by the frozen source model without any label correction. We argue that pseudo-label correction is still working but should be performed suitable moments before the model memorizes noisy pseudo labels, which is supported by superior performances. More analyses about the pseudo-label strategy refer to the ablation study. As shown in Figure we provide some qualitative results of comparative methods in GTA5 Cityscapes task. The backbone of none-CNN-bal Soruce-GtA. Remarkably, our method performs favorably in comparison other competitors. More visualization results are available in the appendices.
methods is Swin-B, which is pre-trained via
3.3
Ablation Study
We have conducted extensive experiments GTAS - Cityscapes to study the contributions of each component in the proposed method. Please refer to the appendices for more details.
Whether can the prompt adapter learn reasonable prompts and facilitate feature generalization? The prompt adapter consists of a prompt generator (PG) and several prompt interactors (PI). The former can be further divided into stem and level embedding (LE) modules. As shown in Table we have several interesting findings: (1) Without the level embedding, the mloU drops 0.66%, illustrating that the level embedding could capture task-shared knowledge beneficial for the target domain. (2) To analyze the impact of the stem module, we deploy two variants. One to replace multiscale feature maps with one feature map. The other is obtained by removing the stem module and repeating level embeddings to the same shape visual prompts. Compared with single-scale feature map, multiscale feature maps bring 0.72% mloU improvements. Besides, the mloU drops 0.74% without the stem module, proving that multiscale spatial knowledge is helpful for prompt learning, (3) To testify whether multiple prompt interactors necessary, remove P1 < N

---

and only preserve PI 1 for transforming the image token sequences. The 0.9% performance drop illustrates that it is necessary to perform prompt refinement and feature interaction between adjacent stages of the backbone. (4) Without the proposed prompt adapter, the performance decreases by 1.17%. The performance drop smaller than other variants because head tuning also benefits feature adaptation. Despite that, it proves that the prompt adapter could capture informative knowledge for reasonable prompt learning and enhancing the generalization of target features.
Whether is it necessary to perform pseudo-label correction in source-free UDA? Li al. claim that pseudo-label correction strategies perform poorly due to the early-leariing phenomenon and even no better than directly supervising the target model with pseudo labels predicted by the frozen source model. We argue that the pseudo-labed correction should be conducted at suitable moments before the model memorizes noisy pseudo-labels. As shown in Table compared with ELR 47 the proposed adaptive pseudo label correction strategy improves mloU by 0.64%, demonstrating that correcting pseudo-label at suitable moments helps to learn more trustable pre-trained knowledge. As for Ours + offline, we first train the model to get the training IoU curve for each category and then correct pscudo labels with our strategy only once. The mloU of Ours offline decreases 0.77%, illustrating that pseudo-label correction should be performed adaptively otherwise the noisy pseudo labels are memorized again after several training epochs.
Effectiveness of multiscale consistency loss. prediction consistency of rescaled imputs. As shown in Table without feature consistency, the mloU drops dramatically as visual prompts can not guarantee that features of different scales of one image are similar. In other words, the feature consistency enhances the robustness of visual prompts to spatial perturbations. Similarly, prediction consistency benefits generating scale-robust predictions. The overall consistency loss increases mIoU by 2.43%.
The multiscale consistency loss performs feature and
3.4 Further Remark
Comparative Results of Different Augmentations. To analyze the influences of other augmented samples, we replace scale augmentation with weather augmentation, e.g.. snow and frost. That is, we replace dovansaaeeorrginnalusscaal images with snow/original/foot images for unsupervised visual prompt tuning. We summarize the performance of Ours and Durs-weathe in Table Ours refers to our approach trained with augle-au samples. Ours-weather is our approach trained with weather-augmenter samples. Compared with Ours, Ours-weather improves mloU approximately -1.5% to -0.1%% 1% in GTA - Cityscapes task and -0.1 % to +1.6% in SYNTHIA + Cityscapes task. The experimental results illustrate that other augmented samples also work well for unsupervised visual prompt tuning and the SOTA results in SYNTHIA ->ityscapes tasks are updated by weather augmentation.
Table 5: Comparative results of different augmentations on GTA5 + Cityscapes and SYNTHIA -> Cityscapes tasks. Different segmentation architectures: (Swin-B), M (MiT-B5). FB: whether the backbone is frozen. Params (M): number of trainable parameters. (+x.x): mloU gains over the corresponding source pre-trained models.
GTAS - Cityscapes
SYNTHIA + Cityscapes
Methods
Arch
FB
Params (M)
1loU1a(56
mloU1e(%)
Ours
S
28.6
56.2 (+5.7)
52.6 (+8.0) 52.9 (+8.3)
Ours-weather
28.6
54.7 (+4.2)
Ours
M
/
12.3
54.2 (+1.7)
$2.6 (44.0) 53.0 (+4.4)
Ours-weather
12.3
54.1 (+1.6)
Ours
S
/
28.6
(+4.1)
53.8 (+5.0) 53.0 (+4.4)
Ours-weather
S
28.6
54.1 (+1.6)
Ours
M
12.3
56.1 (43.1)
53.8 (43.8)
Ours-weather
M
12.3
55.2 (+2.2)
54.5 (44.5)
Visualization Results of Adaptive Pseudo-Label Correction Strategy. To emphasize the effective- ness of the proposed adaptive pseudo-label correction strategy, we draw the IoU,t and IoUm curves for each category as shown in Figure 5 We observe that the IoU,, curves of all categories increase until the model converges and the phenomenon of ToU decreasing due to the noise memorization are disappeared. The above interesting observations demonstrate that the proposed adaptive pscudo-label

---

correction strategy could effectively alleviate the negative effect caused by pseudo-labe noises. It also proves that pseudo-label correction works in source-free unsupervised domain adaptation when it is performed at suitable moments.
Road
Motorcycle
Fence
85
93
80
51
74
41
78
50
(%)
92
74
76
49
06
40
J0U (%)
e83
91
48 2
64
72
1ol)
90
70
Um
47
39
LCM
68
LCM
46
LCM
81
89
66
45
2
Training Process (x10" iters)
Training Process (x10* iters)
Training Process (x10* iters)
Figure 5: Analysis on early learning (loUeI. blue curves) and memorization (IoUm. red curves) with our proposed adaptive pseudo-label correction strategy for different categories in Cityscapes dataset. The LCM (Label Correction Moment) denotes the moment when the algorithm performs pseudo-label correction.
Related work
Source-Free Unsupervised Domain Adaptation for Semantic Segmentation. Source-free unsu- pervised domain adaptation has attracted much attention when source data is not available during model adaptation. Most SFUDA methods [21 23 26 are proposed for classification task, which is
suboptimal for cross-domain semantic segmentation. Only a few methods [30 19 20. 46) focus on source-free domain adaptive semantic segmentation. For example, SFDA [30] not only recovers and preserves the source domain knowledge via knowledge transfer, but also distills valuable information from the target domain for sulf--uper learning. Kundu et al. [19] design multi-heat framework to extract reliable target pseudo labels for self-training. Kundu et al. (20) further introduce an intermediate mixup domain between the original and the realizable generic domain, which could enhance the trade-off between discrininability and transferability. Ye et al. [46) improve the target domain performance by reconstructing the source domain data and performing distribution-aware adversarial learning and self-training. However, current source-free methods adopt fine-tuning for model adaptation, which is inefficient for large-scale transformer- architectures.
Visual Prompt Tuning. Recently, visual prompt tuning has been introduced into the vision area to generalize the large-scale pre-trained model to downstream tasks. Previous methods design visual prompts from the view of token-level, pixel-level and module-level. Teken-level prompt tuning methods [18 or several encoder layers. Pixel-level prompt tuning methods 44] 6 36] introduce learnable pixel perturbations on the original images. However, token-level pixel-level prompts are black boxes, which cannot capture convincing knowledge beneficial for downstream tasks. Module-level prompt design auxiliary layers or blocks for the pre-trained model. Despite learning rich knowledge, they are specifically designed for the pre-trained model by modifying its basic units, which can not be applied to various architectures. As far as we know, only (15) has applied visual prompts for online unlabeled adaption in image classification, and none of the work has explored unsupervised visual prompt tuning for source-free domain adaptive semantic segmentation.
3 32 40
15
33 40
govern learnable visual tokens the embedding layer
tuning methods 49
48 7
Conclusion and Discussion
In this paper, we propose a universal unsupervised visual prompt tuning (Uni-UVPT) framework, which adapts a large-scale pre-trained source model for source-free domain adaptive semantic segmentation. A novel prompt adapter is proposed to progressively encode informative knowledge into prompts and enable target features to match the pre-trained model. Cooperatively, an adaptive pseudo-label correction strategy with multiscale consistency loss is proposed to correct pseudo-label at suitable moments and enhance visual prompts' spatial robustness. Extensive experiments prove that our approach could effectively and efficiently perform large-model unsupervised knowledge transfer. Our approach also has its limitations, for example, further reducing the trainable parameters by delicately designing prompt modules and exploring more powerful learning constraints for visual prompts, e.g., pixel-wise contrastive loss. We hope this paper could inspire researchers to further investigate unsupervised visual prompt tuning for other vision applications.
10

---

Acknowledgments
We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend Al Processor used for this research.
References
[1] Sk Miraj Ahmed, Dripta S Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K Roy- Roy-Chowdhury. Unsupervised multi-source domain adaptation without access source data. In CVPR. pages 10103- 10112, 2021.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 0.4500 2016.
[3] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203. 1(3)4, 2022.
[4] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modifying pixel space to adapt pre-trained models. arXiv preprint arXiv:2203. 17274. 2022.
[5] Mathilde Bateson, Hoel Kervadec, Jose Dolz, Hervé Lombaert, and Ismail Ben Ayed. Source-relaxed domain adaptation for image segmentation. MICCAI, pages
2020.
[6] Aochuan Chen, Yuguang Yao, Pin- Yu Chen, Yihua Zhang, and Sijia Liu. Understanding and improving visual prompting: A label-mapping perspective. arXiv preprint vrXiv:2211.1163: 2022.
[7] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:2205. 13535, 2022.
[8] Yi- Yi-Hsin Chen, Wei Yu Chen, Yu- Ting Chen, Bo-Cheng Tsai, Yu-Chiang Frank Wang. and Min Sun. No more discrimination: Cross city adaptation road scene segmenters. ICCV, 2017.
MMSegmentation Contributors. MMScgmentation: Openmmlab semantic segmentation toolbox and benchmark. /uttps://github.rr com/ com/open - mm1 wumlaa/mmmeementtiioon 2020.
[10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 321 3213-3223, 2016.
[11] Gabriela Csurka. A Comprehensive Survey on Domain Adaptation for Visual Applications, pages 1-35. Springer International Publishing, Cham, 2017.
[12] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, pages 248-255. Ieee, 2009.
[13] Yuxin Fang, Shusheng Yang, Shijie Wang, Yixiao Ge, Ying Shan, and Xinggang Wang. Unleashing vanilla vision transformer with masked image modeling for object detection. arXiv preprint arXiv:2204.0294 2022.
[14] Francois Fleuret et al. Uncertainty reduction for model adaptation in semantic segmentation. In CVPR, pages 191339923,,211 2021.
[15] Yulu Gan, Xianzheng Ma, Yihang Lou, Yan Bai, Renrui Zhang, Nian Shi, and Lin Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. arXiv preprint 4rX1v:2212.04145. 2022.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[17] Lukas Hoyer, Dengxin Dai, and Lue Van Gool. Daformer: Improving network architectures and training strategies for domain-adee adaptive semantic segmentation. In CVPR, pages 9924 9935, June 2022.
[18] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan. and Nem-Nam Lim. Visual prompt tuning. In ECCV, pages 709-727, 2022.
[19] Jogendra Nath Kundu, Akshay Kulkamni, Amit Singh, Varun Jampani, and R Venkatesh Babu. Generalize then adapt: Source- free domain adaptive semantic segmentation. In ICCV, pages 7046-7055 2021.
11

---

[20] Jogendra Nath Kundu, Akshay R Kulkarni. Suvaansh Bhambri, Deepesh Mehta, Shreyas Anand Kulkarni. Varun Jampani, and Venkatesh Babu Radhakrishnan. Balancing discriminability and transferability for source-free domain adaptation. In ICML, pages 11710-11728, 2022.
[21] Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source- free domain adaptation. In CVPR, pages 4544-4553, 2020.
[22] Brian Lester, Rami Al-Rfou, and Noah Constant, The power of scale for parameter-officient prompt tuning. In EMNLP, pages 3045-3059, 2021.
[23] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In CVPR, pages 9641-9650, 2020.
[24] Ruihuang Li, Shuai Li. Chenhang He, Yabin Zhang, Xu Jia, and Lei Zhang. Class-balanced pixel-level self-labeling for domain adaptive semantic segmentation. In CVPR, pages 11593-11603, 2022.
[25] Xiang Lisa Li and Percy Liang. Prefix teeiixtuuning: Optimizing continuous prompts for generation. In ACL, pages 4582-4597, 2021.
[26] Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, pages 6028-6039, 2020.
[27] Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. TPAMI, 44(11):8602-86)) 8617, 2021.
[28] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 1-35) 2023.
[29] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.0766 2021.
[30] Yuang Liu, Wei Zhang, and Jun Wang. Source-free domain adaptation for semantic segmentation. In CVPR, pages 1215 1224, 2021.
[31] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[32] Jochem Loedeman, Maarten C Stol, Tengda Han, and Yuki M Asano. Prompt generation networks for efficient adaptation of frozen vision transformers. arXiv preprint arX1v:22100064666, 2022.
[33] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In CVPR, pages 2215-5215,.00 2022.
[34] Namuk Park and Songkuk Kim. How do vision transformers work? arXiv preprint a7Xiv:2202.06709 2022.
[35] Zhen Qiu, Yifan Zhang, Honghin Lin, Shuaicheng Niu, Yanxia Liu, Qing Du, and Mingkui Tan. Source- free domain adaptation via avatar prototype generation and adaptation. arXiv preprint arXiv: 106. 15326, 2021.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748 8763, 2021.
[37] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV, pages 102-118. 2016
[38] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: large collection of synthetic images for semantic segmentation of urban scenes. CVPR, pages 3234-3243, 2016.
[39] Christos Sakaridis, Dengxin Dai, Simon Hecker. and Luc Van Gool. Model adaptation with synthetic and real data for semantic dense foggy scene understanding. In ECCV, 2018
[40] Kihyuk Sohn. Yuan Hao, José Lezama, Luisa Polania, Huiwen Chang. Han Zhang, Irfan Essa, and Lu Jiang, Visual prompt tuning for generative transfer learning, arXiv preprint 2000092 2022.
12

---

[41] Wenhai Wang, Enze Xie, Xiang Li. Deng-Ping Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 424, 2022.
[42] Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers In ICCV, pages 22-31, 2021.
[43] Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and efficient design semantic segmentation transformers. In M. Ranzato, Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, NeurIPS, volume 34, pages Curran Associates, Inc., 2021.
[44] Shiqi Yang. Yaxing Wang, Joost Van De Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In ICCV, pages 8978-899899 8987, 2021.
[45] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In CVPR, pages 4085 4095, 2020.
[46] Mucong Ye, Jing Zhang, Jinpeng Ouyang, and Ding Yuan. Source data-free unsupervised domain adaptation for semantic segmentation. In ACM MM, pages 2233-2223 2242, 2021.
[47] Li Yi, Gezheng Xu, Pengcheng Xu, Jiaqi Li, Ruizhi Pu, Charles Ling. Ian McLeod, and Boyu Wang. When source-Iree free domain adaptation meets learning with noisy labels. In ICLR.
[48] Bruce XB Yu, Jianlong Chang, Lingbo Liu, Qi Tian, and Chang Wen Chen. Towards unified view on visual marameter-efficiee transfer learning. arXiv preprint arXiv:2210.0011 2022.
[49] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint vrXiv2200.006673. 2022.
13


Reviews:
[{"review_id": "MUBeCQIUH0", "reviewer": "Reviewer_zvBk", "summary": "The authors present Uni-UVPT (Universal Unsupervised Visual Prompt Tuning), a framework designed to enable the adaptation of a pre-trained source model, originally trained on a large dataset, for source-free domain adaptive semantic segmentation. Their approach involves introducing a unique prompt adapter that gradually integrates relevant knowledge into prompts, facilitating alignment between the target features and the pre-trained model. They also propose an adaptive strategy for correcting pseudo-labels, along with a multi-scale consistency loss, which enhances the spatial reliability of visual prompts by rectifying pseudo-labels at suitable instances.", "questions": "Based on the findings presented in Table 1, the experimental results for the MiT-B5 (M) backbone do not exhibit a noticeable distinction in performance when compared to the results of Ours and GtA.", "limitations": "They mentioned limitation of proposed method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's structure is well-organized, and it effectively addresses the problem at hand while clearly outlining its contributions.\nThe inclusion of experiments using models with different backbones is valuable for analyzing diversity and understanding the results obtained through the experiments.", "weaknesses": "It would be beneficial to include a parameter comparison not only with the previous method but also among the proposed methods in the ablation study.  This would allow us to determine which specific component methods have the most significant impact on memory efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: It would be beneficial to include a parameter comparison not only with the previous method but also among the proposed methods in the ablation study. This would allow us to determine which specific component methods have the most significant impact on memory efficiency.**\n\n**A1**:\nThanks for the reviewer's suggestions.\nWe have provided parameter comparisons of previous methods in Table 1.\nBesides, the parameter comparisons for the ablation study\nare shown in the following table (the revised Table 2).\n\n| Stem        | LE | PI   | Params(M) | mIoU(%)      |\n|:-------------:|:----:|:------:|:-----------:|:--------------:|\n| Multiscale  | &check;  | &check; | 28.6081  | 56.24 (Ours) |\n| Multiscale  | &cross;  | &check; | 28.6077  | 55.58 $_{\\downarrow 0.66}$ |\n| Singlescale | &check;  | &check; | 15.4827  | 55.52 $_{\\downarrow 0.72}$ |\n| &cross;     | &check;  | &check; | 27.5138  | 55.50 $_{\\downarrow 0.74}$ |\n| Multiscale  | &check;  | PI 1    | 6.0624   | 55.34 $_{\\downarrow 0.90}$ |\n| &cross;     | &cross;  | &cross; | 3.9417   | 55.07 $_{\\downarrow 1.17}$ |\n\nFrom the results, we can calculate the parameters of all modules.\nSpecifically, Level Embedding (LE) and Stem are respectively 4e-4M and 1.09M.\nThe head contains less than 4M parameters.\nMost of the trainable parameters in the full model belong to prompt interactors (PG), i.e., 23.57M.\nFurthermore, we calculate the parameters of four PG modules, namely, 1.03M, 3.90M, 15.21M, and 3.43M,\nwhich are positively correlated with the middle features' dimensions of backbone stages.\n\n\n\n\n**Q2: Based on the findings presented in Table 1, the experimental results for the MiT-B5 (M) backbone do not exhibit a noticeable distinction in performance when compared to the results of Ours and GtA.**\n\n**A2**:\nComparing the results for the MiT-B5 (M) backbone in Table 1, our approach outperforms GtA by 0.9% in GtA5 $\\to$ Cityscapes, and 0.2% and 0.4% in SYNTHIA $\\to$ Cityscapes.\nCompared with the noticeable performance of the Swin-B backbone,\nthe performance improvements for MiT-B5 are smaller due to the following two reasons:\\\n(1) To match the middle features' shape of Mit-B5 backbone stages,\nthe prompts' dimensions in Mit-B5 are smaller than those in Swin-B,\nwhich weakens the visual prompts' learning capacity. \\\n(2) Following most previous methods, we apply single one segmentation head for predictions,\nwhile GtA utilizes five auxiliary heads (with more trainable parameters) for consistent predictions.\nIf our approach cooperates with the tricks of GtA, the performance of our approach could be further improved but the trainable parameters increase.\\\nIn summary, our approach is theoretically applicable for all transformer backbones and obtains different performance improvements."}, {"review_id": "rPQJBxtXNc", "reviewer": "Reviewer_bcKo", "summary": "The paper presents an unsupervised visual prompting tuning solution for source-free semantic segmentation, where a pretrained model from a source domain is applied to an unlabeled target domain without accessing source data. The authors design a light-weight prompt adapter module to generate knowledgeable visual prompts and adapt target features via cross-attention. Meanwhile, an adaptive pseudo-label correction is proposed to generate reliable pseudo-labels at suitable moments. Experiments on GTA5 $\\to$ Cityscapes and SYNTHIA $\\to$ Cityscapes tasks validate the efficacy of their method across multiple pretrained models.", "questions": "Please refer to the weaknesses.", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "In general, I like the motivation and technical aspects of this paper. I feel the problem is important and the approach seems to make a step towards the right direction. This paper has a simple and effective idea of the important problem and has great application values. All the technical contributions are well motivated and well ablated. The state-of-the-art comparisons to existing methods are currently a little bit beyond expectation. This paper would be stronger in improving the weakness. Besides, I think the idea of the proposed methods can be extended to CNN backbones and other unlabeled applications except for semantic segmentation. To support my rating, I have listed the strength of this paper,\n1. Applying large-scale pre-trained models to unlabeled target domains (with distribution shift) at low cost is an important and meaningful topic. The motivation is well explained and makes sense. The proposed technique introducing visual prompt tuning into source-free DA, is simple yet effective.\n2. The overall design of the prompt adapter architecture is quite general and can be used in different applications. Especially, using conv-based prompt generator modules seems to be effective and plays an important role. Unlike the previous visual prompt models, the adapter seems to also make use of the input image which can be effectively leveraged along with intermediate extracted features. In addition, the idea of prompt and features bi-direction refinement between layers in backbones seems to be more optimal than VPT which directly prepends prompts in the input token sequences.\n3. The design of the adaptive pseudo-label correction is novel and interesting. Previous methods focus on mining confident pseudo labels via learned feature structures while this paper proposes to correct pseudo labels at several important time moments that are determined by the training IoU curves. Besides, this strategy is simple and universal, which can cooperate with existing pseudo-label strategies and can be applied to other unsupervised and semi-supervised applications.\n4. The method has been thoroughly tested on many different source models (different transformer-based backbones pre-trained with different methods) and achieves SOTA performance on GTA5 $\\to$ Cityscapes and SYNTHIA $\\to$ Cityscapes tasks.\nMoreover, the method is conceptually simple and the code has been provided in the supplemental materials for better reproducibility.", "weaknesses": "1) Multiscale samples and prompt interactors are important according to Table 2. I wonder whether other augmented samples also work well in the proposed solution.\n\n2) I suggest authors add a figure to show the differences between the proposed method and previous solutions, making the motivations more clear to readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: I wonder whether other augmented samples also work well in the proposed solution.**\n\n**A1**:\nTo analyze the influences of other augmented samples, we replace scale augmentation with weather augmentation, e.g., snow and frost.\nThat is, we replace downscale/original/upscale images with snow/original/frost images for unsupervised visual prompt tuning.\nWe summarize the performance of source pre-trained models, Ours and Ours-weather in the following table.\nOurs refers to our approach trained with scale-augmented samples.\nOurs-weather is our approach trained with weather-augmented samples.\\\nCompared with Ours, \nOurs-weather improves mIoU approximately -1.5% to -0.1% in GTA $\\to$ Cityscapes task \nand -0.1 % to +1.6% in SYNTHIA $\\to$ Cityscapes task.\nThe experimental results illustrate that \nother augmented samples also work well for unsupervised visual prompt tuning\nand the SOTA results in SYNTHIA $\\to$ Cityscapes tasks are updated by weather augmentation. \n\n|         Methods        | Arch | FB | Params (M) | GTA $\\to$ Cityscapes (mIoU19) | SYNTHIA $\\to$ Cityscapes (mIoU16)|\n|:----------------------:|:---:|:--:|:---------:|:--------:|:---------:|\n| Standard Single Source |   S  |  N |    90.7   | 50.5          | 44.6            |  \n|          Ours          |   S  |  Y |    28.6   | 56.2 (+5.7)   | 52.6 (+8.0)     |\n|        Ours-weather    |   S  |  Y |    28.6   | 54.7 (+4.2)   | 52.9 (+8.3)     |\n| Standard Single Source |   M  |  N |    85.2   | 52.5          | 48.6            | \n|          Ours          |   M  |  Y |    12.3   | 54.2 (+1.7)   | 52.6 (+4.0)     |\n|        Ours-weather    |   M  |  Y |    12.3   | 54.1 (+1.6)   | 53.0 (+4.4)     | \n|       Source-GtA       |   S  |  N |   110.4   | 52.8          | 48.8            | \n|          Ours          |   S  |  Y |    28.6   | 56.9 (+4.1)   | 53.8 (+5.0)     |\n|        Ours-weather    |   S  |  Y |    28.6   | 56.3 (+3.5)   | 53.7 (+4.9)     | \n|       Source-GtA       |   M  |  N |   103.7   | 53.0          | 50.0            | \n|          Ours          |   M  |  Y |    12.3   | 56.1 (+3.1)   | 53.8 (+3.8)     | \n|        Ours-weather    |   M  |  Y |    12.3   | 55.2 (+2.2)   | 54.5 (+4.5)     | \n\n\n\n**Q2: I suggest authors add a figure to show the differences between the proposed method and previous solutions, making the motivations more clear to readers.**\n\n**A2**:\nThanks for the reviewer's suggestion.\nIn the revised manuscript, we will add a figure to show the key motivations and differences from previous work."}, {"review_id": "IVjcsrinzS", "reviewer": "Reviewer_Ytkx", "summary": "This paper considers the problem of source-free domain adaptive semantic segmentation. To solve this problem, the authors propose a visual prompt tuning approach, which keeps the parameters of the networks while adapts the model to target domain only by learning visual prompts. Specifically, a lightweight prompt adapter proposed to learn informative knowledge for segmentation and a label-correction approach is proposed to generate more accurate pseudo-labels. Experiments on two settings show the benefit of the proposed method.", "questions": "Please see the weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper is well-written and easy to follow. The reviewer can easily understand the motivation of this paper and the method. Experiments are easy to check the benefit of the proposed method.\n\n+ A novel visual prompt tuning approach is proposed. To my knowledge, this is the first prompt tuning work for source-free domain adaptation.\n\n+ A simple label correction is proposed, which is inspired by the phenomenon.\n\n+ Experiments show that the proposed method achieves the best results when fixing the backbone.", "weaknesses": "- The benefit of fixing backbone during adaptation is not well verified. 1. The authors state that fine tuning the backbone will cause knowledge forgetting. However, the authors do not conduct experiments that the proposed method could avoid this problem. 2. The computation cost is not discussed. As my understanding, although the proposed method does not need to fune-tune the parameters of the backbone, the computation cost is mostly the same as the backbone fune-tuning methods. This is because the network needs to propagate the gradients to all layers.\n\n- From Table 2, it seems that the proposed PG, PI and LE are not very important. For example, removing all of them only reduces a 0.53 lower mIoU. I thus question that is it required to include them during adaptation which largely increase the complexity.\n\n- The improvement on one setting is very limited, i.e., G-C with M backbone.\n\n- The results on the source domain should be compared as the authors indicate the knowledge forgetting is important.\n\n- As knowledge forgetting is important, it is also required to show the results on continual domain adaptive semantic segmentation task, where there are several target domains sequentially appeared.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The computation cost is not discussed. As my understanding, although the proposed method does not need to fune-tune the parameters of the backbone, the computation cost is mostly the same as the backbone fune-tuning methods. This is because the network needs to propagate the gradients to all layers.**\n\n**A1**:\nOur approach is not claimed to be training-efficient but parameter-efficient. \nTherefore, we do not report the computation cost (FLOPS) but the trainable parameters of all compared methods. \n\nWe agree that the computation cost of our approach is equal to fine-tuning the backbone, \nbut our approach requires less GPU memory and reduces training resources, making it possible for large-scale model adaptation with a single GPU.\n\n\n**Q2: From Table 2, it seems that the proposed PG, PI and LE are not very important. For example, removing all of them only reduces a 0.53 lower mIoU. I thus question that is it required to include them during adaptation which largely increase the complexity.**\n\n**A2**:\nWe have revised Table 2 as the table of **@Reviewer zvBk: A1**.\nWe are sorry for the performance error of the last line in the table (removing PG, PI, and PE),\nwhich is revised as 55.07$_{\\downarrow 1.17}$.\n\nIt is crucial to include PG, PI, and LE modules due to two reasons:\\\n(1) As shown in Table 1, the performance gains of previous SOTA methods over source pre-trained models are less than 4%, illustrating these transfer tasks are very challenging.\nTherefore, the performance drops in Table 2 are significant, approximately from 0.66% to 1.17%. \\\n(2) The prompt adapter is not complex and is easy to implement, mainly containing convolution layers, trainable vectors, and sparse cross-attention operations.\n\n\n**Q3: The improvement on one setting is very limited, i.e., G-C with M backbone.**\n\n**A3**:\nIn the setting of G-C with M backbone,\nour approach achieves comparable performance in the standard single setting\nand outperforms previous SOTA by 0.9% in the Source-GtA setting.\nPlease refer to **@Reviewer zvBk: Q2 A2** and **@Reviewer HKNc: Q2 A2** for more discussions.\nIt is worth noting that our approach outperforms previous SOTA in most tasks or achieves comparable performance.\n\n**Q4: The authors state that fine-tuning the backbone will cause knowledge forgetting. However, the authors do not conduct experiments that the proposed method could avoid this problem. The results on the source domain should be compared as the authors indicate the knowledge forgetting is important.**\n\n**A4**:\nDealing with knowledge forgetting is not claimed as our contribution.\nDiscussing knowledge forgetting in the paper aims to emphasize that prompt tuning is a better way for model adaptation than fine-tuning the backbone.\nSpecifically, prompt tuning methods in NLP/CV usually freeze backbone parameters so that\nthe pre-trained knowledge/representations can be utilized for downstream tasks\nwhile fine-tuning may discard beneficial properties stemming from source pre-training.\nSimilar to previous prompt tuning methods, our approach also freezes the backbone to preserve source domain knowledge.\nFinally, we kindly remind the reviewer that our technical contributions are to learn reasonable visual prompts and rectify target pseudo-labels when designing a prompt tuning solution for source-free domain adaptive semantic segmentation. \n\nIt is not reasonable to evaluate source domain performance for our approach or other visual prompt tuning methods, due to the following two reasons:\\\n(1) Prompts in these methods are designed to reduce the domain gap between pre-trained models and downstream tasks. \nThe prompt-related parameters are learned with target data, not source data.\nTherefore, the trained model cannot be used for the source domain.\\\n(2) The parameters of the pre-trained backbone are frozen, illustrating source domain knowledge is not forgotten.\\\nTherefore, the source domain results of prompt-tuning methods cannot provide valuable insights.\n\n\n\n**Q5: As knowledge forgetting is important, it is also required to show the results on continual domain adaptive semantic segmentation task, where there are several target domains sequentially appeared.**\n\n**A5**:\nKnowledge forgetting is important but not the contribution of this paper.\nIn **A4**, we have discussed the reasons in detail.\n\nContinual domain adaptive semantic segmentation is an interesting topic\nbut quite different from source-free domain adaptive semantic segmentation.\nThe former needs to jointly address knowledge forgetting and domain gap problems while the latter addresses the domain gap.\nOur approach is designed to reduce the domain gap with prompt tuning.\nTherefore, it is unreasonable to apply our approach to the continual domain adaptive semantic segmentation.\n\nThanks to the reviewer for inspiring us to design prompt tuning approaches for continual domain adaptive semantic segmentation. We will address this interesting topic in our future work."}, {"review_id": "Sfo4JIsd44", "reviewer": "Reviewer_HKNc", "summary": "This paper introduces a visual prompt tuning method to tackle source-free domain adaptation in semantic segmentation. The approach involves partitioning the frozen pre-trained model into multiple stages and proposing a prompt adapter for stage-wise prompt tuning.\n", "questions": "How and what is the optimal approach for selecting 'r' samples in the memory queue to facilitate adaptive pseudo-label correction?\n\nThe learnable vector γ in Eq. (3) is initialized with 0, how does it change during training? How does it affect the training of the prompt interactor?\n\nWhat is the performance of the proposed prompt tuning method when applied to convolutional neural networks, such as deeplabv2 and fcn, in order to provide a fair comparison with other state-of-the-art methods?\n", "limitations": "How do the proposed prompt tuning methods perform on other SFDA tasks, such as classification? \n\nThe comparison experiments are not sufficient, for example the experiment using the same convnets backbone is missing. When using the same vision transformer backbone, the improvement over previous prompt tuning methods is limited.\n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The motivation is clear to apply a prompt tuning method to address the source-free domain adaptation.\n\nThe adaptive pixel pruning method is proposed to compute the KL divergence of each pixel and incorporate it into the modified KD loss to get an adaptive threshold which is used to select valuable pixels in each image during training.\n", "weaknesses": "The proposed prompt tuning method is supposed to be parameter-efficient, however, as compared to previous methods, such as CPSL and VPT, the proposed method requires much more leanable parameters. When the backbone is frozen, the prompt adapter requires an additional 2 to 4 times the number of parameters compared to previous methods.\n\nThe performance gains in this method can be attributed to the utilization of a stronger vision transformer model. However, a comparison with commonly used convnets is missing. \n\nIn the standard single source setting, the enhancement achieved over VPT is minor, ranging approximately from 0.1% to 1% with the MiT backbone, while also entailing nearly twice the number of learnable parameters.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The proposed prompt tuning method is supposed to be parameter-efficient, however, as compared to previous methods, such as CPSL and VPT, the proposed method requires much more learnable parameters. When the backbone is frozen, the prompt adapter requires an additional 2 to 4 times the number of parameters compared to previous methods.**\n\n**A1**:\nThe reviewer misunderstood the concept of parameter-efficient in our paper. We don't claim our approach is more parameter-efficient than previous visual prompt tuning methods. Instead, we propose that prompt tuning is more parameter-efficient for model adaptation in comparison to previous SFDA methods that fine-tune the entire network. \n\nAll three methods (CPSL, VPT, and our approach) freeze the backbone and learn a few parameters. Our approach learns more parameters than the others, but the trainable parameters are less than 1/4 of the backbone, demonstrating that our model is more parameter-efficient than backbone fine-tuning.\n\n**Q2: The performance gains in this method can be attributed to the utilization of a stronger vision transformer model. When using the same vision transformer backbone, the improvement over previous prompt tuning methods is limited. In the standard single source setting, the enhancement achieved over VPT is minor, ranging approximately from 0.1% to 1% with the MiT backbone, while also entailing nearly twice the number of learnable parameters.**\n\n**A2**:\nVision transformer is not the only reason for performance improvements over CNN-based methods. \nAs shown in the following table,\nwhen the performances of pre-trained CNN and transformer models are similar,\nour performance gains are greater than GtA, illustrating that our approach is important.\n\n|Methods| Arch | GTA5 $\\to$ Cityscapes (mIoU19)|\n|:------:|:----:|:----------:|\n|Source-GtA (cited from GtA paper)|D|51.6|\n|GtA |   D  |  53.4(+1.8) |\n|Source-GtA |   S  |  52.8  |\n|Ours|   S  |  56.9 (+4.1) |\n|Source-GtA |   M  |   53.0|\n|Ours|   M  |56.1 (+3.1) |\n\nWe respectfully disagree that our improvements are limited/minor.\nAs shown in the following table, we calculate our performance improvements over VPT according to Table 1.\nOur approach significantly outperforms VPT in most settings. In the single-source standard setting with Mit backbone, our approach performs comparably.\n\nSetting| Arch |  GTA5 $\\to$ Cityscapes (+mIoU19) | SYNTHIA $\\to$ Cityscapes (+mIoU16) | SYNTHIA $\\to$ Cityscapes (+mIoU13)     |\n|:----:|:----:|:----:|:-------:|------|\n|Standard Single Source|S| +2.7 |   +4.9  | +6.2 |\n|Standard Single Source| M  | +0.1 |   +1.0  | +1.3 |\n|GtA-Source| S | +1.8 |   +2.2  | +2.2 |\n|GtA-Source| M | +1.7 |   +0.8  | +0.6 |\n\nCompared with the source pre-trained model (MiT-B5) containing 85.2M trainable parameters,\nVPT requires 7.6M and ours requires 12.3M, demonstrating that prompt tuning methods significantly reduce learnable parameters. \nTherefore, a few more parameters (less than 5M) in our approach are acceptable for superior performance.\n\n**Q3: A comparison with commonly used convnets is missing. What is the performance of the proposed prompt tuning method when applied to convolutional neural networks, such as deeplabv2 and fcn, in order to provide a fair comparison with other state-of-the-art methods?**\n\n**A3**:\nWe have claimed that transferring large-scale source pre-trained models is one of our motivations\nso transformer-based backbones are naturally chosen for our experiments.\nBesides, previous methods do not release the source pre-trained CNN models, making it difficult for fair comparisons. \n\nFor a fair comparison with previous CNN-based SFDA methods, we have extended the GtA method with transformer-based backbones. \nOur approach outperforms GtA in both GTA5 $\\to$ Cityscape and SYNTHIA $\\to$ Cityscape tasks, with a performance improvement ranging from 0.8% to 0.9% and from 0.2% to 1.3% (0.4% to 1.7%), respectively. This shows that our approach is superior to CNN-based SFDA methods.\n\nDue to the limited time, the experiments on CNN models are not finished. The full CNN experiments will be added in the revised paper.\n\n\n**Q4: How and what is the optimal approach for selecting 'r' samples in the memory queue to facilitate adaptive pseudo-label correction?**\n\n**A4**:\nIn lines 80-83 of the supplemental material, we have discussed how to choose the suitable memory queue length $r$.\nSpecifically, \nwe set $r$ as different values and perform complete training.\nThe results are shown in Figure 2(c) of the supplemental material.\nWe find that higher $r$ leads to better performance but brings more memory cost. \nTo balance the memory efficiency and performance, we set r = 1000 in our experiments.\n\n**Q5: The learnable vector γ in Eq. (3) is initialized with 0, how does it change during training? How does it affect the training of the prompt interactor?**\n\n**A5**:\n$\\gamma$ is optimized by Adam optimizer with back-propagation gradients, the same as other model parameters.\nIts effects on the prompt interactor are mainly to balance the backbone features $F_{i-1}^{out}$ and the attention layer's output (prompt knowledge), which has been discussed in lines 191-194 of the main manuscript.\nSpecifically,\nat the early stage of training, the prompt knowledge is not learned well, and directly injecting prompt knowledge with backbone features may be harmful.\nTherefore, $\\gamma$ is initialized with 0 and changes slightly to make $F_i^{in}$ similar to $F_{i-1}^{out}$.\nWhen the prompt knowledge is well learned, $\\gamma$ is learned to adaptively inject prompt knowledge into backbone features.\n\n**Q6: How do the proposed prompt tuning methods perform on other SFDA tasks, such as classification?**\n\n**A6**:\nTheoretically, the proposed prompt tuning methods can be applied to other SFDA tasks. \nDue to the limited time, we cannot report results on the source-free cross-domain classification task. We will add the results in the revised manuscript.\n"}]

Author rebuttal:
We thank all reviewers for their valuable comments. 
We are encouraged that they find our idea to be unique / simple and effective / great application values / important and meaningful / novel and interesting / the first prompt tuning work for source-free domain adaptation (**Reviewer zvBk, bcKo, Ytkx**), 
and our paper is well-organized / well motivated / well-written and easy to follow / clear (**all reviewers**). 
We are glad that they think our experiments are valuable for analyzing diversity and understanding the results / validate the efficacy / are well-ablated / are a little bit beyond expectation / achieve the best results when fixing the backbone (**Reviewer zvBk, bcKo, Ytkx**).

We have provided separate responses to all reviewers’ questions.

Meta-review: Source free domain adaptation is a  challenging problem, and authors are trying to solve it using visual prompt tuning. 
Ytkx has raised valid point regarding experiment lacking w.r.t. "catastrophic forgetting problems". Authors stated that they do not claim contribution around "knowledge forgetting". Meta-reviewers believe that authors should update draft to make this clear. Especially update following lines 
"Previous methods usually fine-tune the entire network, which suffers from expensive parameter tuning and unpredictable catastrophic forgetting problems.  To avoid these issues, we propose to utilize visual prompt tuning for parameter efficient adaptation. "

Please incorporate other points raised by the reviewers too.
Decision: Accept (poster)