SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We use top-1 accuracy as our evaluation metric and report results of the following settings: (1) CLIP: zero-shot CLIP on the target domain, which can be regarded as a baseline of our method. (2) Source Combined: all source domains are combined into one single domain and applied with popular single-source UDA methods. Specifically in this setting, we adopt the prompting method from Zhou et al. [50] to serve as another baseline named as "Simple Prompting". (3) Multi-Source: results reported from other multi-source UDA methods.
Implementation Details For fair comparisons, we adopt a ResNet50 as our backbone on ImageCLEF and Office-Home and a ResNet101 on DomainNet. The weights are from CLIP and frozen throughout the experiments. Prompts and auto-encoder of MPA are trained using the mini-batch SGD optimizer with a learning rate of 0.003 and 0.005 while the learned subspace is tuned with a 0.0005 learning rate in LST. We use a batch size of 32 and adopt a cosine learning rate scheduler. For hyper-parameters, token lengths  $M_{1}$  and  $M_2$  are both set to 16. Pseudo-label threshold  $\tau$  is set to 0.4 for producing reliable labels.  $\alpha$  in Equation 9 is set to 500. The weight matrix  $\mathbf{W}_2$  of the back projection function in Equation 7 has a size of  $\mathbb{R}^{384\times d_I}$ , where  $d_I$  is 100 for ImageCLEF, 150 for OfficeHome and 250 for DomainNet.
4.2 Comparison to State-of-the-Art
Multi-Prompt Alignment The results on ImageCLEF and Office-Home are shown in Table 1. For ImageCLEF, it is obvious that MPA outperforms other methods with an average accuracy of  $91.7\%$ , where there is at least a  $3\%$  increase when adapting to domain C and I. For Office-Home, MPA achieves the best results except when adapting to the domain Clipart. Nevertheless, we achieve an accuracy of  $75.4\%$  on average, which is  $1.3\%$  higher than the second best method MFSAN. It is worth noting that compared to state-of-the-art method MFSAN on both datasets, MPA only trains 0.78M and 2.36M parameters, while MFSAN has a total of 51.75M and 51.80M parameters needed for optimizing (66.3 and 21.9 times larger than ours).
ImageCLEF	Office-Home
→C	→I	→P	Avg	→Ar	→Cl	→Pr	→Rw	Avg
Zero-Shot
CLIP [28]	95.1	87.3	74.0	85.5	71.5	50.2	81.3	82.4	71.4
Source Combined
DAN [22]	93.3	92.2	77.6	87.7	68.5	59.4	79.0	82.5	72.4
DANN [9]	93.7	91.8	77.9	87.8	68.4	59.1	79.5	82.7	72.4
D-CORAL [33]	93.6	91.7	77.1	87.5	68.1	58.6	79.5	82.7	72.2
DAPL* [10]	96.0	89.2	76.0	87.1	72.8	51.9	82.6	83.7	72.8
Simple Prompt*	93.6	90.6	80.9	88.4	70.7	52.9	82.9	83.9	72.4
Multi-Source
DCTN [43]	95.7	90.3	75.0	87.0	N.A.	N.A.	N.A.	N.A.	N.A.
MDDA [49]	N.A.	N.A.	N.A.	N.A.	66.7	62.3	79.5	79.6	71.0
SImpAI50[37]	93.3	91.0	77.5	87.3	70.8	56.3	80.2	81.5	72.2
MFSAN [51]	95.4	93.6	79.1	89.4	72.1	62.0	80.3	81.8	74.1
MPA (ours)	98.6	96.2	80.4	91.7	74.8	54.9	86.2	85.7	75.4
Table 1: Accuracy (%) on ImageCLEF and Office-Home. * implies that the method is based on our implementation
DomainNet
→ Clp	→ Inf	→ Pnt	→ Qdr	→ Rel	→ Skt	Avg
Zero-Shot
CLIP [28]	61.3	42.0	56.1	10.3	79.3	54.1	50.5
Source Combined
DANN [9]	45.5	13.1	37.0	13.2	48.9	31.8	32.6
MCD [32]	54.3	22.1	45.7	7.6	58.4	43.5	38.5
DAPL* [10]	62.4	43.8	59.3	10.6	81.5	54.6	52.0
Simple Prompt*	63.1	41.2	57.7	10.0	75.8	55.8	50.6
Multi-Source
M3SDA-β [26]	58.6	26.0	52.3	6.3	62.7	49.5	42.6
SImpAI101 [37]	66.4	26.5	56.6	18.9	68.0	55.5	48.6
LtC-MSDA [38]	63.1	28.7	56.1	16.3	66.1	53.8	47.4
T-SVDNet [17]	66.1	25.0	54.3	16.5	65.4	54.6	47.0
PFSA [8]	64.5	29.2	57.6	17.2	67.2	55.1	48.5
PTMDA [30]	66.0	28.5	58.4	13.0	63.0	54.1	47.2
MPA (ours)	65.2	47.3	62.0	10.2	82.0	57.9	54.1
Table 2 shows that for DomainNet, MPA exceeds other multi-source UDA methods by more than  $5\%$ . To the best of our knowledge, this is the highest reported accuracy on this dataset so far with less than one third parameters optimized compared with most state-of-the-arts methods. Regarding individual adaptations, MPA achieves best results on most of the adaptation tasks but performs mediocre on the Quickdraw domain. Interestingly, the result is even a little worse than CLIP. We hypothesize that this is because of the large domain gap between Quickdraw and other domains.
Latent Subspace Tuning Results from Table 1 and Table 2 are already strong indications of the success of MPA under conventional multi-source UDA settings. Now, we would like to investigate a more practical scenario, where adaptation to a streamlined set of target domains is required, by using the LST strategy as mentioned in Section 3.3.
Results of LST on the DomainNet dataset are shown in Table 3, where compared with re-applying MPA, performance only dropped by  $0.3\%$ . Nevertheless, LST achieves higher accuracy compared to most baseline methods of Table 2, including CLIP with an increase of  $3.2\%$ . Significantly, the adaptation process with LST requires only 11 hours of GPU time on a NVIDIA RTX 3090, whereas adapting using MPA takes 54 hours—a speed increase of approximately 5 times.
Table 2: Accuracy (%) on DomainNet. * implies that the method is based on our implementation
DomainNet
→Inf, Clp	→Clp,Inf	→Inf, Pnt	→Pnt,Qdr	→Qdr,Rel	→Rel, Skt	Avg
CLIP	61.3	42.0	56.1	10.3	79.3	54.1	50.5
MPA	64.9	46.6	61.8	10.2	82.5	57.5	53.9
LST	64.6	46.7	61.6	9.8	81.2	57.6	53.6
Table 3: Results (\%) of continuous adaptation on DomainNet. Here  $\rightarrow \mathcal{D}_1$ ,  $\mathcal{D}_2$  means that adaptation is performed on two target domains and we report the performance on the highlighted second one.
CLIP backbone While we are the first to apply prompt learning to multi-source UDA with the help of a pre-trained CLIP model, most existing approaches that we compare with utilize ImageNet pre-trained backbones, which are generally considered to be inferior compared to CLIP. Therefore, we would like to examine whether MPA's performance gain is simply from CLIP. For such purpose, two additional sets of experiments are conducted: (1) we apply a simple prompt learning method [50] to the source combined scenario, dubbed as the Simple Prompting baseline. As is demonstrated in Table 1 and 2, while it is  $1.33\%$  on average better than zero shot CLIP, MPA still outperforms it with a significant margin; (2) we directly swap MFSAN's ResNet50 backbone pre-trained on ImageNet to CLIP's image encoder. Surprisingly, when tested on ImageCLEF, Table 4 shows that the performance
even drops by a small margin of  $0.3\%$ . From these results, we infer that while CLIP indeed provides a strong backbone, it is not necessarily universally superior, as also observed on the basis of studies Devillers et al. [6], Yang et al. [45]. Moreover, despite CLIP's strong zero-shot performance as stated in Radford et al. [28], our method consistently outperforms CLIP in the domain adaptation setting, especially in cases on the ImageCLEF and OfficeHome datasets where CLIP's zero-shot performance is limited. On these two datasets, MPA consistently delivers excellent results that surpass CLIP by  $6.2\%$  and  $4.0\%$ , respectively. Therefore, we believe while CLIP's strong backbone indeed contributes to MPA's good results, it alone is insufficient for such superior performance and another important piece is MPA's strong domain adaptation ability.
ImageCLEF
→C	→I	→P	Avg
CLIP [28]	95.1	87.3	74.0	85.5
MFSAN [51]	95.4	93.6	79.1	89.4
MFSAN+CLIP*	96.7	93.0	77.7	89.1
MPA (ours)	98.6	96.2	80.4	91.7
Prompt engineering Apparently, the performance of MPA depends on the quality of pseudo-labels generated by CLIP. As such, it is worth investigating whether MPA has been pushed to its maximum extent by using more sophisticated prompts than "A photo of [CLS]". Specifically, we tested two additional sets of prompts: (1) "a [Dom] of [CLS]" (e.g., a painting of dog) and (2) "a photo of [CLS] in domain [Dom]" (e.g., a photo of dog in domain painting) with the intention of incorporating domain information, and as is presented in Tab 5, while each of the three prompts shows certain advantage on some specific domains, in general, the naive "A photo of [CLS]" still performs the best.
Table 4: Exploration of MFSAN equipped with CLIP's backbone on ImageCLEF dataset. * implies that the method is based on our implementation
Zero-shot	MPA
→Ar	→Cl	→Pr	→Rw	Avg	→Ar	→Cl	→Pr	→Rw	Avg
a [Dom] of [CLS]	71.7	52.4	74.9	81.0	70.0	74.1	55.1	82.2	85.0	74.1
a photo of [CLS] in domain [Dom]	68.1	53.1	81.3	82.0	71.1	72.7	55.7	87.1	85.2	75.2
a photo of [CLS]	71.5	50.2	81.3	82.4	71.4	74.8	54.9	86.2	85.7	75.4
Table 5: Ablation studies on different choices of manual prompt on the Office-Home dataset.
Hyper-parameter selection Experimental results for ablation studies on hyper-parameter selections are reported in Table 6. For both pseudo-label threshold  $\tau$  and prompt token length  $M_{1}, M_{2}$ , three different choices  $\tau \in \{0.3, 0.6, 0.8\}$  and  $M_{1}, M_{2} \in \{8, 12, 20\}$  are examined. For simplicity, we are setting  $M_{1}$  and  $M_{2}$  to be equal. As  $\tau$  increases, while the quality of the pseudo-labels gets higher, fewer images will be fed into the model, and Table 6a suggests that doing so hurts the overall performance. On the contrary, shown in Table 6b, the general trend for prompt token length is that the longer the prompt, the better the performance. As a result, we choose  $\tau = 0.4$  and  $M_{1} = M_{2} = 16$  to balance the trade-off between performance and efficiency. For  $\alpha$  in Equation 9, we chose it to be 500 to balance all losses (in this case  $\mathcal{L}_{1}$ ) to be of the same order of magnitude. Our experimental results from Table 6c also support such motivation.
Alignment strategy, prompt denoising and prompt design To validate the effectiveness of our alignment strategy, we evaluate our method without the  $\mathcal{L}_1$  loss, and table 7 shows that this will result in a performance degradation of about  $0.7\%$ . In particular, the  $\mathcal{L}_1$  constraint exhibits a greater impact on the more difficult Clipart domain, where the accuracy dropped by a large margin of  $1.5\%$  without it, demonstrating the effectiveness of constraining consistent predictions on different prompts. Subsequently, we tested the effectiveness of denoising the prompts through auto-encoder reconstruction. Again, results show that the incorporation of an auto-encoder structure with the  $\mathcal{L}_{AE}$  loss is beneficial to the overall performance. Finally, we examined the necessity of using class and domain-specific tokens for learning prompts and results from Table 7 support such design. Specifically,  $\checkmark$  implies that both the class-specific and domain-specific prompts are used while  $\times$  implies that only the class-specific prompts are used.
τ	→Ar → Cl → Pr → Rw Avg
0.3	74.5	55.0	86.1	85.9	75.4
0.6	74.6	54.9	85.9	85.3	75.2
0.8	74.0	54.2	85.2	85.5	74.7
0.4 (reported)	74.8	54.9	86.2	85.7	75.4
(a) Ablation on pseudo-label threshold  $\tau$
Token length	→Ar → Cl → Pr → Rw Avg
M1=M2=8	74.3	54.9	85.8	85.3	75.1
M1=M2=12	74.6	54.3	85.9	85.6	75.2
M1=M2=20	74.8	55.2	86.3	86.0	75.6
M1=M2=16 (reported)	74.8	54.9	86.2	85.7	75.4
(b) Ablation on token lengths  $M_{1}$  and  $M_2$
α	→Ar → Cl → Pr → Rw Avg
1	74.4	53.7	84.9	85.6	74.7
10	74.5	54.1	85.7	86.0	75.1
100	74.7	54.5	85.5	85.6	75.1
1000	74.4	55.0	86.3	86.0	75.4
500 (reported)	74.8	54.9	86.2	85.7	75.4
(c) Ablation on  $\alpha$
Table 6: Ablation studies on hyper-parameter selection on the Office-Home dataset.
L1	LAE	Prompt Design	→Ar	→Cl	→Pr	→Rw	Avg
×	✓	✓	74.8	53.4	85.2	85.3	74.7
✓	×	✓	74.0	53.8	85.5	85.4	74.9
✓	✓	×	74.2	53.0	85.2	85.3	74.4
✓	✓	✓	74.8	54.9	86.2	85.7	75.4
Table 7: Ablation studies on various modules of MPA on the Office-Home dataset.
In this paper, we introduced prompt learning to multi-source UDA and proposed a simple MPA scheme to align the source and target domains. MPA is composed of two steps. The first step is to train individual prompts for each source and target domain pair and the next one to align them after reconstruction using an auto-encoder structure. Extensive experiments showed that MPA achieved better results than state-of-the-art methods on various multi-source UDA tasks with substantially fewer parameters tuned. Moreover, an LST strategy was introduced for efficient adaptation to a streamlined set of target domains. As the model contains useful clues of multiple domains, one potential limitation is that it faces more risks in terms of information leakage where adversaries might produce attacks to steal information from the model.
Acknowledgement This project was supported by NSFC under Grant No. 62102092.
[42] Zejia Weng, Xitong Yang, Ang Li, Zuxuan Wu, and Yu-Gang Jiang. Open-vclip: Transforming clip to an open-vocabulary video model via interpolated weight optimization. 2023.
[43] Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In CVPR, 2018.
[44] Yuanyuan Xu, Meina Kan, Shiguang Shan, and Xilin Chen. Mutual learning of joint and separate domain alignments for multi-source domain adaptation. In WACV, 2022.
[45] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In CVPR, 2022.
[46] Jun Yang, Rong Yan, and Alexander G Hauptmann. Cross-domain video concept detection using adaptive svms. In ACM MM, 2007.
[47] Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In ICML, 2013.
[48] Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. In NeurIPS, 2018.
[49] Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-source distilling domain adaptation. In AAAI, 2020.
[50] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.
[51] Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang. Aligning domain-specific distribution and classifier for cross-domain classification from multiple sources. In AAAI, 2019.
[52] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In ECCV, 2018.