SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As untrusted pseudo would harm the learning of the source classifier, we use the pseudo labels whose probabilities are higher than  $\tau$ . Similar to Eq. 7, the cross-entropy loss computed with target data is denoted as:
$$
\mathcal {L} _ {t, \mathcal {D}} = \mathbb {E} _ {\mathcal {X} ^ {t}} \left[ - \frac {1}{N _ {r}} \sum_ {j = 1} ^ {N _ {r}} \mathbf {I} \left(p \left(\hat {y} = y _ {j} ^ {t} \mid \mathbf {r} _ {j} ^ {t}, t, \{t \}\right) \geq \tau\right) \log p \left(\hat {y} = y _ {j} ^ {t} \mid \mathbf {r} _ {j} ^ {t}, t, \mathcal {D}\right) \right] \tag {16}
$$
where the  $\mathbf{I}(\cdot)$  is an indicator function. Meanwhile, we minimize the information entropy of the logits to achieve high-confident classification.
$$
\mathcal {L} _ {e n t} = \mathbb {E} _ {\mathcal {X} ^ {t}} \left[ - \frac {1}{N _ {r}} \sum_ {j = 1} ^ {N _ {r}} \mathbf {I} \left(p \left(\hat {y} = y _ {j} ^ {t} \mid \mathbf {r} _ {j} ^ {t}, t, \{t \}\right) \geq \tau\right) p \left(\hat {y} = y _ {j} ^ {t} \mid \mathbf {r} _ {j} ^ {t}, t, \{t \}\right) \log p \left(\hat {y} = y _ {j} ^ {t} \mid \mathbf {r} _ {j} ^ {t}, t, \{t \}\right) \right] \tag {17}
$$
Considering that inaccurate pseudo labels may be detrimental to the training of the source classifier, we only use them to optimize the target classifier. Overall, the objective on the target data  $\mathcal{X}^t$  is:
$$
\mathcal {L} _ {t} = \mathcal {L} _ {t, \{t \}} + \mathcal {L} _ {t, \{s, t \}} + \mathcal {L} _ {e n t} \tag {12}
$$
The overall training objective is:
$$
\mathcal {L} = \mathcal {L} _ {s} + \lambda \mathcal {L} _ {t}. \tag {13}
$$
where  $\lambda$  is used to balance the effect of  $\mathcal{L}_t$  in  $\mathcal{L}$ .
3.4 Prompt Ensemble
During prompt tuning, all the visual and text encoder parameters are frozen. In each iteration, only the learnable prompts are updated via backward gradients. However, data mutation across mini-batches may cause prompt perturbations in the early stage of training, harming the prompt tuning. To alleviate this phenomenon, a prompts ensemble strategy is proposed to improve the stability of prompt tuning by averaging all the history prompts. Concretely, we maintain a prompt buffer  $\mathbb{B} = \{\mathbf{b}_1^s,\mathbf{b}_2^s,\dots,\mathbf{b}_K^s,\mathbf{b}_1^t,\mathbf{b}_2^t,\dots,\mathbf{b}_K^t\}$  about the learnable prompt in DA-Pro. In each iteration iter of training, buffer  $\mathbb{B}_{iter}$  can not be optimized with gradients, and can only be updated by Exponential Moving Average (EMA) according to the weights of itself and the learnable prompt  $\mathbf{t}_{iter}$ $\mathbb{T}_{iter} = \{\mathbf{t}_1^s,\mathbf{t}_2^s,\dots,\mathbf{t}_K^s,\mathbf{t}_1^t,\mathbf{t}_2^t,\dots,\mathbf{t}_K^t\}$ :
$$
\mathbb {B} _ {i t e r} = \alpha \mathbb {B} _ {i t e r - 1} + (1 - \alpha) \mathbb {T} _ {i t e r} \tag {14}
$$
where  $\alpha$  is the ensemble ratio of history prompts. Considering that the buffer needs to quickly forget the inaccurate parameters learned in the early stage, we set a dynamic updating ratio  $\alpha = \min\{1 - \frac{1}{iter + 1}, 0.99\}$ . As EMA reduces the variance of the model, the prompt ensembled strategy infers a more robust prompt  $b$ . After training,  $\mathbb{B}$  is saved and used for inference.
This section evaluates our DA-Pro on mainstream DAOD scenarios, including the Cross-Weather, Cross-Fov, and Sim-to-Real. Further ablation studies are conducted to validate the effectiveness of the proposed domain-adaptive prompt, training scheme, and prompt ensemble strategy.
Cross-Weather Cityscapes [4] is a large-scale dataset that contains diverse images recorded in street scenes. It is divided into 2,975 training and 500 validation images, annotated with 8 classes. Foggy Cityscapes [32] is a synthetic foggy dataset that simulates fog with three distinct densities on Cityscapes, containing 8,925 training images and 1,500 validation images. We take the training set of Cityscapes as the source domain and the training set of foggy Cityscapes as the target domain, evaluating Cross-Weather adaptation performance on the 1500-sized validation set in all 8 categories.
Cross-Fov KITTI [10] is a vital dataset in self-driving, containing 7,481 images with car annotations. Images are captured by driving in rural areas and on highways, yielding the Field of View (FoV) disparity with Cityscapes. For the Cross-Fov adaptation scenario, we migrate KITTI to Cityscapes on the car category and validate performance on Cityscapes.
Sim-to-Real SIM10k [18] is a synthetic dataset containing 10,000 images with car annotations rendered from the video game Grand Theft Auto V. We perform simulated environment to real-world adaptation on SIM10K and Cityscapes datasets.
We adapt RegionCLIP with a domain classifier [3] as baseline, and initialize the default prompt with "A photo of a [class][domain]" . We use ResNet-50 [12] as the visual encoder and Transformer [34]) as the text encoder, and initialize with the pre-trained CLIP model. Configurations of the detector and image pre-processing follow the default settings in [31, 3]. Following [3], we first pre-train the baseline model with classification loss, regression loss, and adversarial loss. Then for adaptation, one batch of source images with ground truth and one batch of target domain images are forwarded to the proposed DA-Pro in each iteration to calculate the supervising and self-training loss. We fix the length of learnable tokens  $M,N$  to 8, 8, respectively. The hyperparameter  $\lambda$  is set to 1.0. We set the batch size of each domain to 2 and use the SGD optimizer with a warm-up learning rate for training. We take the mean Average Precision (mAP) with a threshold of 0.5 as the evaluation metric. All experiments are deployed on a Tesla V100 GPU.
4.3 Comparison with existing methods
We introduce various state-of-the-art DAOD approaches for comparison, including DA-Faster [3], VDD [38], DSS [37], MeGA [36], SCAN [21], TIA [44], SIGMA [22], and AT [24].
Table 1: Comparison with existing methods on three adaptation tasks, for Cross-Weather adaptation Cityscapes  $\rightarrow$  Foggy Cityscapes (C  $\rightarrow$  F), Cross-Fov adaptation KITTI  $\rightarrow$  Cityscapes (K  $\rightarrow$  C) and Sim-to-Real adaptation SIM10K  $\rightarrow$  Cityscapes (S  $\rightarrow$  C). mAP: mean Average Precision (\%).
Methods	C→F	K→C	S→C
Person	Rider	Car	Truck	Bus	Train	Motor	Bicycle	mAP	mAP	mAP
DA-Faster [3]	29.2	40.4	43.4	19.7	38.3	28.5	23.7	32.7	32.0	41.9	38.2
VDD [38]	33.4	44.0	51.7	33.9	52.0	34.7	34.2	36.8	40.0	-	-
DSS [37]	42.9	51.2	53.6	33.6	49.2	18.9	36.2	41.8	40.9	42.7	44.5
MeGA [36]	37.7	49.0	52.4	25.4	49.2	46.9	34.5	39.0	41.8	43.0	44.8
SCAN [21]	41.7	43.9	57.3	28.7	48.6	48.7	31.0	37.3	42.1	45.8	52.6
TIA [44]	52.1	38.1	49.7	37.7	34.8	46.3	48.6	31.1	42.3	44.0	-
SIGMA [22]	44.0	43.9	60.3	31.6	50.4	51.5	31.7	40.6	44.2	45.8	53.7
AT [24]	56.3	51.9	64.2	38.5	45.5	55.1	54.3	35.0	50.9	-	-
Baseline	51.8	59.0	67.4	36.8	59.5	50.6	39.7	55.9	52.6	59.5	60.8
DA-Pro	55.4	62.9	70.9	40.3	63.4	54.0	42.3	58.0	55.9	61.4	62.9
Table 2: Ablation studies (%) on Cross-Weather adaptation scenario Cityscapes  $\rightarrow$  Foggy Cityscapes. AP50 evaluates mAP on detection boxes with IoU  $\geq 0.5$ , and  $\geq 0.75$  for AP75. AP averages AP50 to AP95 with step 5.
Prompt Design	M	N	Prompt Ensemble	AP	AP50	AP75
A photo of a [class][domain]	28.5	52.6	28.7
[v1c][v2c]...[vm][ci]	16	0	28.9	53.0	28.5
[v1c][v2c]...[vm][ci][dld]	16	0	29.2	53.8	29.3
[v1d][v2d]...[vN][ci][dld]	0	16	28.9	53.1	28.7
[v1c][v2c]...[vm][d][v2d]...[vN][ci][dld]	8	8	31.2	55.5	30.5
[v1c][v2c]...[vm][d][v2d]...[vN][ci][dld]	8	8	✓	31.9	55.9	32.0
[v1c][v2c]...[vm][d][v2d]...[vN][ci][dld]	4	4	✓	31.1	55.0	30.2
[v1c][v2c]...[vm][d][v2d]...[vN][ci][dld]	12	12	✓	31.4	55.4	31.3
[v1c][v2c]...[vm][d][v2d]...[vN][ci][dld]	16	16	✓	31.3	55.3	30.6
Cross-Weather Adaptation Scenario As shown in Table 1 (C→F), the proposed DA-Pro achieves the highest mAP over 8 classes of  $55.9\%$ , outperforming SOTA AT [24] by a remarkable margin of  $5.0\%$ . Compared with existing methods, our method significantly improves 6 categories (i.e. rider, car, truck, bus, train and bicycle) ranging from  $1.8\%$  to  $16.2\%$ . The improvement is particularly significant on the previously poor categories, e.g.  $11.0\%$  on Rider and  $16.2\%$  on Bicycle, which may have harder semantics for adaptation. Though its promising mAP of  $52.6\%$ , DA-Pro still improves Baseline by  $3.3\%$ . Concretely, DA-Pro promotes  $2.1 \sim 3.9\%$  on all 8 categories via learning domain-adaptive prompt. The superior performance demonstrates the proposed DA-Pro can improve the generalization ability of the detector on the target domain.
Cross-Fov Adaptation Scenario Table 1 ( $\mathrm{K} \rightarrow \mathrm{C}$ ) also shows that the DA-Pro improves hand-crafted prompt Baseline by  $1.9\%$ . And the proposed method reaches the best remarkable  $61.4\%$  mAP, outperforming SOTA methods in the Cross-Fov detection task. According to [37],  $\mathrm{K} \rightarrow \mathrm{C}$  adaptation faces more complicated shape confusion than  $\mathrm{C} \rightarrow \mathrm{F}$ , which requests higher discriminability of the model. The considerable performance improvement confirms that the proposed method can efficiently generate domain-aware detectors with high discrimination.
Sim-to-Real Adaptation Scenario Unlike the Cross-Weather and Cross-Fov adaptation, the Sim-to-Real task has a wider domain gap at semantic level. As presented in Table 1 (S→C), the proposed method gains  $2.1\%$  on Baseline and peaks at  $62.9\%$ , surpassing SOTA by  $9.2\%$ . It further demonstrates that our strategy is robust in not only appearance but also harder semantics adaptation tasks.
We report detailed ablation studies (Table 2) conducted on the Cross-Weather adaptation scenario, to validate the effectiveness of DA-Pro under different configurations.
Table 3: The influence  $(\%)$  of loss design on Cross-Weather adaptation scenario Cityscapes  $\rightarrow$  Foggy Cityscapes. Total number of tokens is set to 16. - stands for the prompt is not compatible with the loss. Without the historical prompt ensemble strategy.
Prompt Design	Ls,{s} + Lt,{t}	Ls,{t}	Ls,{s,t} + Lt,{s,t}	Lent	Lt,{s}	mAP
A photo of a [class][domain]	-	-	-	-	-	52.6
[v1c][v2c]...[vm][ci]	✓	-	-	-	-	53.0
[v1c][v2c]...[vm][ci][dld]	✓	53.5
[v1c][v2c]...[vm][ci][dld]	✓	✓	53.8
[v1c][v2c]...[vm][ci][dld]	✓	✓	✓	53.8
[v1c][v2c]...[vm][ci][dld]	✓	✓	✓	✓	53.7
[v1d][v2d]...[vN][ci][dld]	✓	52.9
[v1d][v2d]...[vN][ci][dld]	✓	✓	53.1
[v1d][v2d]...[vN][ci][dld]	✓	✓	✓	52.7
[v1d][v2d]...[vN][ci][dld]	✓	✓	✓	✓	52.9
[v1c][v2c]...[vm][1d][v2d]...[vm][ci][dld]	✓	54.4
[v1c][v2c]...[vm][1d][v2d]...[vm][ci][dld]	✓	✓	54.8
[v1c][v2c]...[vm][1d][v2d]...[vm][ci][dld]	✓	✓	✓	55.2
[v1c][v2c]...[vm][1d][v2d]...[vm][ci][dld]	✓	✓	✓	✓	55.5
[v1c][v2c]...[vm][1d][v2d]...[vm][ci][dld]	✓	✓	✓	✓	✓	53.4
Comparison on Prompt Design In this section, we analyze the performance of different prompts. As shown in line  $1\sim 5$  of Table 2, we compare the performances on the following four prompt designs: the pre-defined prompt "A photo of a [class][domain]", the CoOp-style learnable prompt, and the domain-invariant tokens or domain-specific tokens with domain-related textual description. The CoOp-style prompt with 16 learnable domain-invariant tokens improves  $0.4\%$  on AP and AP50 over the hand-crafted prompt. Introducing domain-related textual description further boosts  $0.3\sim 0.8\%$  on three metrics. This exhibits that learning domain-invariant tokens improves the discrimination of the target detection head. When directly applying domain-specific tokens, it only gains a limited improvement over the pre-defined prompt. This shows that the source domain information cannot be transferred to the target domain only using domain-specific tokens. Diving the 16 tokens into 8 domain-invariant and 8 domain-specific tokens further gains extra  $0.3\sim 1.0\%$  improvements. We suppose that based on shared knowledge across domains, learning domain-specific knowledge leads to more discriminative detection head on each domain. These results reveal that compared with the fixed or the domain-agnostic prompt, the domain-adaptive prompt has a higher ability to embed domain information for prediction. Moreover, both domain-invariant tokens and domain-specific tokens bring improvements.
Comparison on Loss Function Table 3 shows that each term of loss function contributes to improvement in the adaption performance. To reduce the influence of other factors, we do not use the historical prompt ensemble strategy. With  $\mathcal{L}_{s,\{s\}} + \mathcal{L}_{t,\{t\}}$ , four types of learnable prompts have improved  $0.3 \sim 1.8\%$  on the hand-crafted prompt. Combined with  $\mathcal{L}_{s,\{t\}}$ , which commands the target prompt to classify the source image accurately, further gains  $0.2 \sim 0.4\%$  on three prompts distinguished across domains. This reveals that by constraining the detection heads in the source and target domains to classify the input image as accurately as possible, the prompt can learn more domain-invariant knowledge. Further, though introduce  $\mathcal{L}_{s,\{s,t\}} + \mathcal{L}_{t,\{s,t\}}$  has few effects on prompt with only domain-invariant/domain-specific tokens, it improves  $0.4\%$  on the domain-adaptive prompt. We conclude that in two respect. On one hand, just using domain-invariant/domain-specific tokens cannot model shared knowledge between domains and domain-specific knowledge at the same time. On the other hand, constraining the detection head generated by one domain to output higher confidence in its own domain than the other domain contributes to learning domain-specific knowledge. Moreover, adding  $\mathcal{L}_{ent}$  brings extra  $0.3\%$  improvement. However, introducing  $\mathcal{L}_{t,\{s\}}$  suffers  $2.1\%$  degradation, showing that directly optimizing the source classifier with pseudo target labels could do harm to the learning process.
Comparison on Prompt Ensemble To verify the effectiveness of the proposed historical prompt ensemble strategy, we introduce it to the domain-adaptive prompt with 8 domain-invariant tokens and 8 domain-specific tokens. Line 6 of Table 2 shows that it gains improvements of  $0.7\%$ ,  $0.4\%$  and
(a.1)
(c.1)
(c) [V][c][dl]
(b) "A photo of [class][domain]" mAP: 54.6
(b.1)
Figure 3: Detection comparison of different prompts on the Cross-Weather adaptation scenario. We visualize the ground truth (a) and the detection results of three prompts: (b) "A photo of a [class][domain]" (c)  $[\mathbb{V}^c ][\mathbf{c}][\mathbf{d}\mathbf{l}_d]$  (d)  $[\mathbb{V}^c ][\mathbb{V}^d ][\mathbf{c}][\mathbf{d}\mathbf{l}_d]$ . (a.1) (b.1) (c.1) (d.1) are roomed from the same region of the image (a) (b) (c) (d). mAP: mean Average Precision on the example image
(d.1)
mAP:63.9
(d)  $\left[\mathbb{V}^c\right]\left[\mathbb{V}^d\right]\left[\mathbf{c}\right]\left[\mathbf{d}\mathbf{l}_d\right]$
mAP:67.5
$1.5\%$  on AP, AP50 and AP75. This reveals ensembling the history prompt can reduce the effect of prompt disturbance and infer a robust prompt, thus improving detection performance.
Comparison on Prompt Length Further, we explore the impact of prompt length, where  $M$  for domain-invariant tokens and  $N$  for domain-specific tokens on DA-Pro with full configuration in line  $6 \sim 9$  of Table 2. We vary the length from (4, 4) to (8, 8) to (12, 12) to (16, 16). DetPro[6] has shown that a longer prompt may cause over-fitting to base categories in open-set object detection. Similarly, we conclude that a shorter prompt is less discriminative, while a too-long prompt is prone to overfitting to the source domain in DAOD. Thus, we set the token length as (8, 8) in DA-Pro.
In Fig. 3, we present the comparison on the target domain among the ground truth boxes (a) and the detection boxes using different prompts. (a.1)(b.1)(c.1)(d.1) are zoomed from the same region of images (a)(b)(c)(d) for a better view. Fig. 3(a.1) indicates four objects in the cropped region: a car and a bicycle on the left, and two overlapping persons on the right. Both hand-crafted and learnable prompt models detect the car correctly. However, it is difficult for the hand-crafted prompt to describe domain information, like weather conditions, accurately. Thus, the hand-crafted prompt misses some objects on the right side of Fig. 3(b), and the bicycle and one of the persons in Fig. 3(b.1). In Fig. 3(c), the domain-invariant prompt improves  $9.3\%$  mAP compared with the hand-crafted prompt. But it still suffers worse discrimination due to insufficient domain representation learning. In Fig. 3(c.1), the domain-invariant prompt misclassifies the bicycle as a rider even confident with  $93\%$ , and also ignores one person. The proposed domain-adaptive prompt correctly detects all objects in the cropped region Fig 3(d.1). By learning domain information, the domain-specific tokens enable the model to perform confident predictions on the bicycle  $(79\%)$  and 2 persons  $(100\%, 89\%)$ . These comparison results reveal the effectiveness of the proposed domain-adaptive prompt in DA-Pro.
In this paper, we propose a novel DAOD framework named Domain-Aware detection head with Prompt tuning (DA-Pro). Serving a pre-trained visual-language model as the robust detection backbone, it applies the learnable domain-adaptive prompt to generate the discriminative detection head for each domain. As the learnable part is designed to capture the domain-invariant and domain-specific knowledge, we proposed two constraints over the source and target domains to guide the optimization. Moreover, a prompt ensemble strategy is also proposed to reduce the effect of prompt disturbance. Comprehensive experiments over multiple cross-domain adaptation tasks demonstrate that using the domain-adaptive prompt can produce a domain-aware detection head with more discrimination for domain-adaptive object detection.
This work is partially supported by the NSF of China (under Grants 92364202, 61925208, 62102399, 62102398), Beijing Natural Science Foundation (4222039), CAS Project for Young Scientists in Basic Research(YSBR-029) and Xplore Prize.
[43] Yixin Zhang, Zilei Wang, and Yushi Mao. Rpn prototype alignment for domain adaptive object detector. In CVPR, pages 12425-12434, 2021.
[44] Liang Zhao and Limin Wang. Task-specific inconsistency alignment for domain adaptive object detection. In CVPR, pages 14217-14226, 2022.
[45] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In CVPR, pages 16793-16803, 2022.
[46] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, pages 16816-16825, 2022.
[47] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 130(9):2337-2348, 2022.
[48] Qianyu Zhou, Qiqi Gu, Jiangmiao Pang, Zhengyang Feng, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma. Self-adversarial disentangling for specific domain adaptation. TPAMI, 45(7):8954-8968, 2023.
5.1 Additional Experimental Results
In order to further verify the effectiveness of DA-Pro, we evaluate performance on three harder benchmarks: Pascal to Clipart, Pascal to Watercolor and Pascal to Comic. Pascal VOC [7] is a large-scale real-world dataset annotated with 20 classes, which contains 2007 and 2012 subsets. Clipart [15] is collected from the website with 1000 comical images, providing bounding box annotations with the same 20 classes as Pascal VOC. Watercolor and Comic [15] both contain 1000 training images and 1000 test images in art style, sharing 6 categories with Pascal VOC. Three benchmarks enable the method to be evaluated under more challenging domain shifts and in multi-class problem scenarios. The comparison is shown in Table 4 Our DA-Pro surpasses the SOTA method (SIGMA++ with ResNet-101) with a weak backbone (ResNet-50) on all three additional benchmarks, showing the effectiveness of DA-Pro.
Table 4: mAP(%) comparison with existing methods on Pascal→Watercolor, Pascal→Clipart and Pascal→Comic tasks.
Methods	Pascal→Watercolor	Pascal→Clipart	Pascal→Comic
DBGL[1] (ResNet-101)	53.8	41.6	29.7
Baseline (ResNet-50)	54.8	43.4	40.6
FGRR[2] (ResNet-101)	55.7	43.3	32.7
SIGMA++[23] (ResNet-101)	57.1	46.7	37.1
DA-Pro (ResNet-50)	58.1	46.9	44.6
5.2 Hyperparameter Search
To select hyperparameters for our loss functions, we perform experiments of different choices of the weight values  $\lambda (\mathcal{L}_t)$ . We conduct the experiment on DA-Pro on both Cityscapes  $\rightarrow$  FoggyCityscapes and Sim10K  $\rightarrow$  Cityscapes adaptation scenarios. Table 5 shows that our DA-Pro is robust with different settings of  $\lambda$ . Considering that the best result is achieved under 1.0, we take  $\lambda = 1.0$  as the default.
Table 5: The influence  $(\%)$  of  $\lambda$  for the loss with target data  $\mathcal{L}_t$ .
Cityscapes→FoggyCityscapes
λ	0.25	0.5	1.0	5.0	10.0
mAP	55.6	55.8	55.9	55.7	55.4
Sim10K→Cityscapes
λ	0.25	0.5	1.0	5.0	10.0
mAP	62.5	62.9	62.9	62.4	62.8
5.3 Options of the pseudo labels
We optimize the domain-adaptive prompt with annotated source data and further distill CLIP's remarkable zero-shot classification ability to the detection head of the target domain. A naive way is to generate classification probabilities, i.e. soft labels, with the fixed prompt "A photo of [class]. However, aligning the model's predictions with these probabilities will result in the learnable prompt converging to the hand-crafted prompt. Unlike probabilities, generating pseudo labels with a threshold, i.e. hard labels, does not demand learning the relative distances to each category provided by hand-crafted prompts. Instead, they require the prompt to be as close to the correct category and as far from the incorrect categories as possible, thereby learning a more discriminative prompt. To verify this, we conduct experiments on the three benchmarks  $(\mathrm{C} \rightarrow \mathrm{F}, \mathrm{K} \rightarrow \mathrm{C}, \mathrm{S} \rightarrow \mathrm{C})$ . As shown in Table 6, replacing pseudo labels with probability supervision suffers  $0.61.6\%$  degradation on performance.
Table 6: The comparison (\%) of different options of pesudo labels on Cross-Weather adaptation scenario Cityscapes  $\rightarrow$  Foggy Cityscapes.
Options	Cityscapes→Foggy Cityscapes	KITTI→Cityscapes	Sim10K→Cityscapes
probabilities	54.3	60.8	62.1
pseudo label (ours)	55.9	61.4	62.9
5.4 Effect of the domain-related textual tokens  $\mathrm{d}\mathbf{l}_d$
The domain-related textual tokens aim to leverage textual descriptions of domains and introduce hand-crafted prior information to facilitate more efficient learning of domain-specific tokens. The hand-crafted token offers a solid initialization which is discriminative. On this foundation, domain-specific tokens further learn the bias between the two domains, further enhancing the prompt's discrimination. The combination of both hand-crafted and learnable tokens yields superior results. We have conducted additional experiments on three different scenarios. The results are shown in Table 7. Without  $\mathbf{d}\mathbf{l}_d$  token, using prompt  $[\mathbf{v}_1^c][\mathbf{v}_2^c]\dots [\mathbf{v}_M^c][\mathbf{v}_1^d][\mathbf{v}_2^d]\dots [\mathbf{v}_N^d][\mathbf{c}_i]$  suffering  $0.7\sim 1.7\%$  mAP. Experimental results demonstrate that  $\mathbf{d}\mathbf{l}_d$  assists in the convergence of domain-specific tokens and enhances the discrimination of the prompt.
Table 7: The influence  $(\%)$  of the domain-related textual tokens  $\mathbf{d}\mathbf{l}_d$  on Cityscapes  $\rightarrow$  Foggy Cityscapes, KITTI  $\rightarrow$  Cityscapes and Sim10K  $\rightarrow$  Cityscapes.
Prompt Design	C→F	K→C	S→C
[v1c][v2c]...[vm][v1d][v2d]...[vnd][ci]	54.3	60.8	62.1
[v1c][v2c]...[vm][v1d][v2d]...[vnd][ci][dld]	55.9	61.4	62.9
5.5 Effect of the Historical Prompt Ensemble Strategy