SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xinhong Ma, Yiming Wang, Hao Liu, Tianyu Guo, Yunhe Wang*
{maxinhong, wangyiming22, liuhao296, tianyu.guo, yunhe.wang}@huawei.com
Source-free domain adaptive semantic segmentation aims to adapt a pre-trained source model to the unlabeled target domain without accessing the private source data. Previous methods usually fine-tune the entire network, which suffers from expensive parameter tuning. To avoid this problem, we propose to utilize visual prompt tuning for parameter-efficient adaptation. However, the existing visual prompt tuning methods are unsuitable for source-free domain adaptive semantic segmentation due to the following two reasons: (1) Commonly used visual prompts like input tokens or pixel-level perturbations cannot reliably learn informative knowledge beneficial for semantic segmentation. (2) Visual prompts require sufficient labeled data to fill the gap between the pre-trained model and downstream tasks. To alleviate these problems, we propose a universal unsupervised visual prompt tuning (Uni-UVPT) framework, which is applicable to various transformer-based backbones. Specifically, we first divide the source pre-trained backbone with frozen parameters into multiple stages, and propose a lightweight prompt adapter for progressively encoding informative knowledge into prompts and enhancing the generalization of target features between adjacent backbone stages. Cooperatively, a novel adaptive pseudo-label correction strategy with a multiscale consistency loss is designed to alleviate the negative effect of target samples with noisy pseudo labels and raise the capacity of visual prompts to spatial perturbations. Extensive experiments demonstrate that Uni-UVPT achieves state-of-the-art performance on  $\mathrm{GTA5}\rightarrow$  Cityscapes and SYNTHIA  $\rightarrow$  Cityscapes tasks and can serve as a universal and parameter-efficient framework for large-model unsupervised knowledge transfer. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/uni-uvpt and https://github.com/huawei-noah/noah-research/tree/master/uni-uvpt.
Semantic segmentation is a critical computer vision task, which aims to segment and parse a scene image into different image regions associated with semantic categories. The success of these semantic segmentation techniques relies on large-scale densely-labeled datasets. However, it is prohibitively expensive to collect high-quality annotations. Besides, most semantic segmentation methods ignore the distribution shift between training and testing data, making them fail to generalize when deployed in conditions different from training such as cross-city [8] or cross-weather [39] scenarios. Unsupervised Domain Adaptation (UDA) [11] is an intuitive direction to address the above two issues by transferring knowledge from existing models trained on source datasets to the unlabeled target domain. Typical UDA approaches require joint access to both labeled source and unlabeled target data during training, making them unsuitable for Source-Free Unsupervised Domain
Adaptation (SFUDA) where source and target data are not concurrently accessible. This SFUDA setting with restricted data-sharing is privacy-oriented and thus holds immense practical value in autonomous driving and other commercial visual applications.
With the above considerations, this paper proposes to solve source-free domain adaptive semantic segmentation task, where only source pre-trained models and unlabeled target data are available for model adaptation. Recently, a few source-free UDA methods have been developed to tackle semantic segmentation [30, 19, 20, 46] and image classification [21, 23, 26]. Most of them either perform self-training [30, 1, 27, 44] or synthesize target-style training data [35] to fine-tune all model parameters. However, fine-tuning requires many computation resources, which is an expensive and often infeasible proposition for previous SFUDA methods, especially for modern transformer-based architectures. Recently, prompt tuning has been proposed to explore the knowledge of frozen language models [22, 25, 28, 29], which could be an alternative strategy to efficiently adapt the pre-trained source model to the target domain.
Prompt Tuning aims to design a trainable lightweight block as a supplementary input (prompt) for a frozen model, which could guide or direct the generalization of powerful representations to achieve desirable performances. Inspired by its great success in natural language processing (NLP) [28], a few methods in computer vision propose to add small amounts of learnable parameters as tokens [18] or pixel-level perturbations [4] to adapt large pre-trained models to downstream tasks. However, the existing visual prompt tuning methods have two limitations when applied to source-free domain adaptive semantic segmentation: (1) The learned visual prompts are unreasonable. Specifically, prompts like input tokens or pixel-level perturbations are black boxes, which cannot reliably explore convincing knowledge beneficial for pixel-wise predictions. Besides, most methods utilize task-oriented loss for optimization, e.g., cross-entropy loss, which limits the learning capacities of visual prompts. (2) Previous methods cannot be directly applied to the unlabeled target domain because most of them rely on sufficient labeled data to deal with the gap between pre-trained models and downstream tasks. There is a lack of methods addressing unsupervised visual prompt tuning where only massive unlabeled data is available for training.
To address the above problems, we propose a Universal Unsupervised Visual Prompt Tuning (UniUVPT) framework for source-free domain adaptive semantic segmentation. Specifically, given a source pre-trained model, we partition its backbone with frozen parameters into multiple stages and design a lightweight prompt adapter for stage-wise prompt tuning. The prompt adapter contains a prompt generator and several prompt interactors. The former aims to generate reasonable prompts that capture multiscale spatial information and task-shared knowledge. The latter progressively refines prompts between adjacent stages and transforms target features to match the source pre-trained knowledge in the backbone. To learn prompts with massive unlabeled target samples, we utilize target instances with pseudo labels for self-training, and propose an adaptive pseudo-label correction strategy to alleviate pseudo-label noises. The proposed adaptive pseudo-label correction strategy could determine suitable moments to rectify pseudo labels and guide the model to train on instances with corrected pseudo labels, not noisy ones. Meanwhile, a multiscale consistency loss is proposed to impose multiscale consistency of the features and predictions so that the learned visual prompts are robust to arbitrary semantic-preserving spatial perturbations. It is noticed that our method is applicable to various transformer architectures without modifying their basic units.
The contributions of this work are summarized as follows: (1) We first highlight the low-efficiency problem of fine-tuning large-scale backbones in source-free domain adaptive semantic segmentation, and propose a universal unsupervised visual prompt tuning framework for parameter-efficient model adaptation. (2) A lightweight prompt adapter is introduced to learn reasonable visual prompts and enhance feature generalization in a progressive manner. Cooperatively, a novel adaptive pseudo-label correction strategy is proposed to rectify target pseudo labels at suitable moments and improve the learning capacity of visual prompts. Extensive experimental results demonstrate that our method with a few trainable parameters could achieve state-of-the-art performances on  $\mathrm{GTA5} \rightarrow$  Cityscapes and SYNTHIA  $\rightarrow$  Cityscapes tasks.
2 Universal Unsupervised Visual Prompt Tuning
In unsupervised domain adaptation, a labeled source dataset  $\mathcal{D}_s = \{(x_s,y_s):x_s\in \mathcal{X}_s,y_s\in \mathcal{C}\}$  and an unlabeled target dataset  $\mathcal{D}_t = \{x_t:x_t\in \mathcal{X}\}$  are available for training, where  $\mathcal{X}$  is the input space
and  $\mathcal{C}$  denotes the label set.  $x_{s}$  is drawn from the marginal distribution  $p_{s}$  while  $x_{t}$  is drawn from another marginal distribution  $p_{t}$ . The goal is to learn a mapping  $f:\mathcal{X}\rightarrow \mathcal{C}$  that can generalize well for  $x_{t}$ . We focus on the source-free constraint [30], where we do not have access to the original source domain data  $\mathcal{D}_s$ . Instead, merely a source pre-trained model  $f_{s}$  and unlabeled target data  $\mathcal{D}_t$  are provided for adaptation. The source pre-trained semantic segmentation model usually contains a backbone  $f_{b}$  (containing some embedding layers, e.g., the patch embedding layer in Swin [31]) and a customized head  $f_{h}$ . To efficiently adapt the source pre-trained model  $f_{s}$  to the unlabeled target domain  $\mathcal{D}_t$ , we propose to perform unsupervised visual prompt tuning. When designing our algorithm, we consider the following two questions: (1) how to design informative visual prompts for source-free domain adaptive semantic segmentation and (2) how to learn visual prompts with unlabeled samples for model adaptation. To this end, we propose a universal unsupervised visual prompt tuning (Uni-UVPT) framework as shown in Figure 1. Specifically, we design a novel prompt adapter for generating informative visual prompts and improving the generalization of target features. Besides, an effective adaptive pseudo-label correction strategy with a multiscale consistency loss is proposed for learning visual prompts with massive unlabeled target data and enhancing visual prompts' capacity for spatial perturbations. As the backbone  $f_{b}$  is frozen and only the parameters of the proposed prompt adapter and segmentation head are optimized during training, Uni-UVPT is parameter-efficient for model adaptation. Besides, Uni-UVPT could be directly deployed to various transformer-based architectures without modifying the basic units.
2.1 Prompt Adapter
As shown in Figure 1, the proposed lightweight prompt adapter contains a prompt generator and several prompt interactors. The former aims to generate informative prompts that capture multiscale spatial information and task-shared knowledge. The latter is designed to refine prompts with pretrained knowledge and transform target features to match the pre-trained knowledge in the backbone. To interact with target features via visual prompts, we manually partition the backbone  $f_{b}$  into  $N$  (usually  $N = 4$ ) stages, each of which is composed of multiple basic units. Given a target image  $x_{t}$ , it is first fed into the embedding layer to obtain patch sequences  $F_{0}^{out}$ .  $F_{0}^{out}$  is then regarded as an input of the frozen backbone. Meanwhile, the target image  $x_{t}$  is also fed into the prompt generator to output an initial prompt  $C_{0}$  that contains  $L$  dimensional spatial features of multiple resolutions. Then, the initial prompt is flattened and concatenated as an input of the prompt interactor. For stage  $i$  of the backbone, its input feature and output feature are respectively denoted as  $F_{i}^{in}$  and  $F_{i}^{out}$ . Before stage  $i$ , the prompt interactor incorporates the output feature  $F_{i - 1}^{out}$  and the interim prompt  $C_{i - 1}$  of previous stage to obtain the refined prompts  $C_{i}$  and a suitable input  $F_{i}^{in}$  for the stage  $i$ . Finally, the output feature  $F_{N}^{out}$  of the last stage is fed into head  $f_{h}$  for segmentation prediction. In the following sections, we introduce the details of the proposed prompt generator and prompt interactor.
Prompt Generator. Previous methods [4, 18] have proved that learning prompts brings flexibility to the pre-trained model. However, their prompts like input tokens or pixel-level perturbations are black boxes with limited learning capacity, which cannot reliably explore convincing knowledge beneficial for semantic segmentation. Therefore, we propose to design informative prompts for each image to capture multiscale spatial prior and the task-shared knowledge. To achieve this goal, we utilize a standard convolutional stem  $f_{stem}$  borrowed from ResNet [16] to extract multiscale spatial information, because convolution could help transformers better capture the local spatial information [41, 42, 13, 34].  $f_{stem}$  consists of three convolutions, a max-pooling layer, and a stack of stride-2  $3 \times 3$  convolutions to double the channels and reduce the size of feature maps. Several  $1 \times 1$  convolutions are applied at the end to project the feature maps to  $L$  dimensions. Finally, we obtain a feature pyramid  $S = \{s_1, s_2, s_3\}$ , which contains  $L$ -dimensional feature maps with  $1/2$ ,  $1/4$ , and  $1/8$  resolutions of the original image. Besides, following the spirits of previous methods [4, 18], we leverage a group of trainable vectors  $Q = \{q_1, q_2, q_3\}$  named level embedding, to learn task-shared knowledge, which contains three  $L$ -dimensional vectors initialized by the Gaussian function. In this way, the prompt pyramid  $\hat{C}_0 = \{c_1, c_2, c_3\}$  of the input image  $x_t \in \mathbb{R}^{3 \times H \times W}$  could be obtained by
$$
c _ {i} = \kappa \left(s _ {i}, q _ {i}\right), \tag {1}
$$
where  $\kappa (\cdot ,\cdot)$  could be any feature fusion operations. Here, we first repeat  $s_i$  several times so that the vectors can be resized as the same shape of  $q_{i}$ , and then perform element-wise addition. Finally, we flatten and concatenate the prompt pyramid  $\hat{C}_0$  into prompt tokens  $C_0\in \mathbb{R}^{\left(\frac{HW}{2^2} +\frac{HW}{4^2} +\frac{HW}{8^2}\right)\times L}$  as the input of prompt interactor for prompt refinement and feature adaptation.
Figure 1: The architecture of Uni-UVPT. (a) The source pre-trained segmentation model, whose encoder layers are divided into  $N$  stages. (b) The proposed prompt adapter, which contains two designs, including (c) a prompt generator for encoding rich knowledge into prompts, and (d) a prompt interactor for prompt refinement and feature interaction. (e) The proposed adaptive pseudo-label correction strategy, which rectifies pseudo-labels at suitable moments.
Prompt Interactor. To progressively refine prompts and improve feature generalization via the refined prompts, we design a novel prompt interactor based on cross-attention. As shown in Figure 1, for the  $i$ -th stage of the backbone, we use the interim prompt  $C_{i-1}$  of the previous stage as the query, and the output feature  $F_{i-1}^{out}$  as the key and value for cross-attention so that the pre-trained knowledge contained in  $F_{i-1}^{out}$  are injected into prompts. This process can be formulated as:
$$
C _ {i} = C _ {i - 1} + \text {A t t e n t i o n} (\operatorname {n o r m} \left(C _ {i - 1}\right), \operatorname {n o r m} \left(F _ {i - 1} ^ {\text {o u t}}\right)), \tag {2}
$$
where the norm  $(\cdot)$  is LayerNorm [2], and the attention layer Attention  $(\cdot)$  suggests using sparse attention. After that, we utilize the refined prompt  $C_i$  to generate adapted input  $F_{i}^{in}$  for the next stage of the backbone. This process can be formulated as:
$$
F _ {i} ^ {\text {i n}} = F _ {i - 1} ^ {\text {o u t}} + \gamma_ {i} \cdot \text {A t t e n t i o n} (\operatorname {n o r m} \left(F _ {i - 1} ^ {\text {o u t}}\right), \operatorname {n o r m} \left(C _ {i}\right)), \tag {3}
$$
where sparse attention is adopted to reduce computational costs. Besides, we apply a learnable vector  $\gamma_{i}\in \mathbb{R}^{L}$  to balance the attention layer's output and the feature  $F_{i - 1}^{out}$ , which is initialized with 0. This initialization strategy ensures that the feature distribution of  $F_{i - 1}^{out}$  will not be modified drastically due to the injection of the prompt  $C_i$ , thus making better use of the pre-trained knowledge of backbone.
2.2 Adaptive Pseudo-Label Correction
To learn visual prompts with massive unlabeled target data, we propose to utilize high-quality pseudo labels of target samples for self-training. The pipeline of the adaptive pseudo-label correction is shown in Figure 1 (e). Previous SFUDA methods generate pseudo labels by learning meaningful cluster structures in the feature space and the quality of the learned cluster structures hinges on the reliability of pseudo labels generated by the source model. Therefore, the pseudo labels are noisy due to the domain shift. Recently, Li et al. [47] have formulated SFUDA as
learning with label noise problem, and observed early-learning phenomenon that deep neural networks tend to first fit the training data with correct pseudo labels during an early-learning phase, before eventually memorizing the instance with incorrect/noisy pseudo labels. Not only in image classification, we also observe a similar phenomenon in semantic segmentation, where the pseudo-label noise is ubiquitous across samples and distributed in a pixel-wise manner. As shown in Figure 2, we analyze the learning curves on the pseudo labels predicted by the source model during the training process. As the model learning is supervised by noisy pseudo labels, the  $\mathrm{IoU}_m$  curves for all categories increase substantially as training proceeds. The  $\mathrm{IoU}_{el}$  follows a completely different trajectory: it first increases during an early-learning stage where the model learns to correctly segment the incorrectly-labeled pixels, but eventually decreases as memorization occurs.
Figure 2: Effect of early learning  $(\mathrm{IoU}_{el}$ , blue curves) and memorization  $(\mathrm{IoU}_{m}$ , red curves) for different categories in Cityscapes dataset. The  $\mathrm{IoU}_{el}$  is the IoU between the model output and ground-truth labels. The  $\mathrm{IoU}_{m}$  represents the IoU between the model output and pseudo labels.
To alleviate the memorization of noisy pseudo labels in SFUDA, Li et al. [47] roughly encourage model predictions to stick to labels generated by the frozen source model without any label correction. Differently, we design an adaptive pseudo-label correction strategy based on the early-time training phenomenon, which rectifies target pseudo labels for each category before the model memorizes the noisy pseudo labels. Naturally, it raises a question that how to determine when to correct the noisy pseudo labels. Fortunately, as shown in Figure 2, we observe that the segmentation performance on the training set  $(\mathrm{IoU}_m)$  improves rapidly during early learning, and then much more slowly during memorization. In other words, the performance deceleration indicates whether the model over
fits noisy pseudo labels. To estimate the deceleration, we first fit the following exponential parametric model to the training IoU using the least squares:
$$
g (t) = a t ^ {3} + b t ^ {2} + c t + d, \tag {4}
$$
where  $t$  represents training time, and  $a, b, c$  and  $d$  are fitting parameters. However, it is time-consuming to calculate the training IoU on all training samples. We propose to utilize a memory queue with  $r$  samples as an alternative, which is updating with batch data. Then, we compute the derivative  $g'(t)$ , and the pseudo labels for each category are corrected when the relative change of  $g'(t)$  is above a certain threshold  $\tau$ :
$$
\frac {\left| g ^ {\prime} \left(t _ {0}\right) - g ^ {\prime} (t) \right|}{\left| g ^ {\prime} \left(t _ {0}\right) \right|} > \tau . \tag {5}
$$
where  $t_0$  is the start time of the early-learning stage in a correction loop. The image pixels with predictive confidence over  $\gamma \%$  per class on the entire target training set are chosen for pseudo-label correction. For all experiments, we set  $\tau = 0.9$  and  $\gamma = 0.65$ . After the pseudo-label correction, the model learning steps into a new early-learning stage. It is noticed that the pseudo-label correction loop will perform several times until achieving the max training iterations.
Figure 3: Multiscale consistency loss for rescaled images. The rescaled features of  $\tilde{F}_{i,k}^{in}$  of backbone are utilized for feature consistency and the rescaled model predictions  $\tilde{y}_k$  for prediction consistency.
To form the final pseudo labels, we average the model outputs corresponding to multiple rescaled copies of inputs. Specifically, let  $m$  be the number of scaling operations. We set  $m = 3$  (downscaling  $\times 0.5$ , no scaling, and upscaling  $\times 2$ ) in our experiments. For an input  $x_{t}$  rescaled according to these scaling operations in Figure 3,  $y_{k}, 1 \leq k \leq m$  denotes the corresponding model predictions and  $\tilde{y}_k$  is obtained by rescaling  $y_{k}$  as the same size of  $x_{t}$ . The corrected pseudo label  $\hat{y}_{t}$  can be obtained by averaging predictions of multiple rescaled  $x_{t}$ :
$$
\hat {y} _ {t} = \frac {1}{m} \sum_ {k = 1} ^ {m} \tilde {y} _ {k}, \tag {6}
$$
where only categories satisfying Eq (5) require correction while others remain unchanged.
Once the pseudo labels of all training samples are corrected, the model steps into a new early-learning stage via minimizing a self-training loss  $\mathcal{L}_{st}$  shown in Eq (7) and waits for the next correction process.
$$
\mathcal {L} _ {s t} = \mathbb {E} _ {x _ {t} \sim \mathcal {D} _ {t}} [ - \langle \hat {y} _ {t}, \log f (x _ {t}) \rangle ] + \mathbb {E} _ {x _ {t} \sim \mathcal {D} _ {t}} [ - \log f (x _ {t}) ], \tag {7}
$$
where  $\langle \cdot, \cdot \rangle$  is dot product, and  $f(x_{t})$  denotes the model output. The first term is cross-entropy loss based on image and pseudo-label pair  $(x_{t},\hat{y}_{t})$ . The second term is predictive entropy that could reduce model uncertainty on target samples during self-training [19].
Besides, we incorporate a multiscale consistency loss  $\mathcal{L}_{mc}$  to promote feature consistency and prediction consistency, as shown in Eq (8),
$$
\mathcal {L} _ {m c} = \alpha \underbrace {\mathbb {E} _ {x _ {t} \sim \mathcal {D} _ {t}} \left[ \sum_ {i = 1} ^ {N} \frac {1}{m} \sum_ {k = 1} ^ {m} \| \tilde {F} _ {i , k} ^ {i n} - \hat {F} _ {i} ^ {i n} \| _ {2} ^ {2} \right]} _ {\text {f e a t u r e c o n s i s t e n c y} \mathcal {L} _ {f c}} + \beta \underbrace {\mathbb {E} _ {x _ {t} \sim \mathcal {D} _ {t}} \left[ - \frac {1}{m} \sum_ {k = 1} ^ {m} \mathrm {K L} (\tilde {y} _ {k} \| \hat {y} _ {t}) \right]} _ {\text {p r e d i c t i o n c o n s i s t e n c y} \mathcal {L} _ {p c}}, \tag {8}
$$
where  $\alpha$  and  $\beta$  are balance parameters, and KL denotes the Kullback-Leibler divergence. As shown in Figure 3,  $\tilde{F}_{i,k}^{in}$ ,  $1\leq k\leq m$ ,  $1\leq i\leq N$  represents the rescaled features of backbone stages and their average is denoted as  $\hat{F}_i^{in} = \frac{1}{m}\sum_{k = 1}^m\tilde{F}_{i,k}^{in}$ . The first term is the feature consistency regularization  $\mathcal{L}_{fc}$ , which constrains the visual prompts to preserve the semantics of features and meanwhile eliminate the spatial perturbations so that the prompt adapter could generate scale-robust prompts. Similarly, the rest prediction consistency term  $\mathcal{L}_{pc}$  could encourage the model to produce predictions that are robust to arbitrary semantic-preserving spatial perturbations. The final objective is defined as the sum of the self-training loss and multiscale consistency loss:
$$
\mathcal {L} = \mathcal {L} _ {s t} + \mathcal {L} _ {m c}. \tag {9}
$$
More details about the training process are introduced in the appendices.