Multi-Prompt Alignment for Multi-Source Unsupervised Domain Adaptation
Authors: Haoran Chen
Xintong Han
Zuxuan Wu
Yu-Gang Jiang
Venue: NeurIPS 2023 poster (2023)
Keywords: multi source unsupervised domain adaptation; transfer learning; computer vision

Abstract: Most existing methods for unsupervised domain adaptation (UDA) rely on a shared network to extract domain-invariant features. However, when facing multiple source domains, optimizing such a network involves updating the parameters of the entire network, making it both computationally expensive and challenging, particularly when coupled with min-max objectives. Inspired by recent advances in prompt learning that adapts high-capacity models for downstream tasks in a computationally economic way, we introduce Multi-Prompt Alignment (MPA), a simple yet efficient framework for multi-source UDA. Given a source and target domain pair, MPA first trains an individual prompt to minimize the domain gap through a contrastive loss. Then, MPA denoises the learned prompts through an auto-encoding process and aligns them by maximizing the agreement of all the reconstructed prompts. Moreover, we show that the resulting subspace acquired from the auto-encoding process can easily generalize to a streamlined set of target domains, making our method more efficient for practical usage. Extensive experiments show that MPA achieves state-of-the-art results on three popular datasets with an impressive average accuracy of 54.1% on DomainNet.

Multi-Prompt Alignment for Multi-Source Unsupervised Domain Adaptation
Haoran Chen
Xintong Han Zuxuan Wu 1.2f Yu-Gang Jiang!
Shanghai Key Lab of Intell. Info. Processing, School CS. Fudan University Shanghai Collaborative Innovation Center of Intelligent Visual Computing
'Huya Inc
Abstract
Most existing methods for unsupervised domain adaptation (UDA) rely shared network to extract commiin-ivvarian features. However, when facing multiple source domains, optimizing such network involves updating the parameters of the entire network, making it both computationally expensive and challenging, particularly when coupled with min-max objectives. Inspired by recent advances in prompt learning that adapts high-capacity models for downstream tasks in a computationally economic way, we introduce Multi-Prompt Alignment (MPA), a simple yet efficient framework for multi-source UDA. Given a source and target domain pair, MPA first trains an individual prompt to minimize the domain gap through a contrastive loss. Then, MPA denoises the learned prompts through an auto-encoding process and aligns them by maximizing the agreement of all the reconstructed prompts. Moreover, we show that the resulting subspace acquired from the auto-encoding process can easily generalize to streamlined set of target domains, making our method more efficient for practical usage. Extensive experi- ments show that MPA achieves state-of-the-an results on three popular datasets with an impressive average accuracy of 54. 1 1% on NomainNet.
1 Introduction
Deep learning has achieved remarkable progress in various computer vision tasks [15, 11, 31, 24]. However, the success usually relies on supervised training using a massive amount of manually labeled data, which are often expensive and consumin collect. Furthermore, current deep models are brittle to the presence of domain shift [27, 34, 47] in the forms of different image styles, varied lighting conditions, diverse viewpoints. etc., between training and testing distributions.
Unsupervised domain adaptation (UDA) is popular strategy that mitigates domain discrepancies by transferring knowledge learned from well-labeled source domain [25, 4, 39, 42, 23]. While significant advances have been achieved, current approaches focus on the single source setting, where all the labeled training data share the same distribution. In practice, however, it is more common for the labeled data to be collected from multiple sources that are diverse in distribution. Naturally, one could still tackle this problem by straightforwardly combining all the data into one single source and applying off-the-shelf UDA methods. However, directly applying single-source UDA methods often results in limited performance, domain shift also exists among different source domains.
unlabeled target domain
The integration of multiple source domains for improved adaptation results the unlabeled target domain is generally known as multi-source unsupervised domain adaptation. Inspired by the theoreti- cal analysis of Ben-David et al. [2], learning domain- invariant feature representations has become
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Conventional Methods
Our Method
1-th
Promot
Prompt
classifier
classiferr
SOWNEN!
Common Feature
Accuraty
Teat
42.0
Contrastive pairing
target images
NBA (Bun))
(Oun)
IPA Encoderr
mages
1x
2x1of
Number Tunable parameters (ogesrale)
(a) Method comparison
(b) Performance comparison DomainNet
Figure 1: (a) Most conventional multi-source UDA methods use a common feature extractor with domain-specific classifier heads, while we introduce prompt learning to multi-source UDA. (b) MPA outperforms all other multi-source UDA methods by a large margin on the DomainNet dataset with roughly one-third of tunable parameters. We also introduce an LST strategy for continuous adaptation to streamlined set of target domains that further reduces the number of tunable parameters and still achieves high accuracy compared with MPA. See texts for more details.
prevailing paradigm for multi-source UDA. One typical approach is to jointly learn common feature extractor together with domain domain-speciff specific classifier heads. Various feature distance metrics [22, 33, 14] or domain adversarial training [35] can serve
preliminary alignment between source and target domains, followed by different auxiliary losses carefully designed to further reduce domain shift. Although these methods offer decent results in the single source setting, as the number of source domains increases, using only a single shared feature extractor to obtain domain-invarian features is inevitably difficult to optimize [44]. Such problem is further amplified in the modern large model era if we wish to apply more advanced backbones for improved performance.
In this paper, we introduce prompt learning [16, 41], which has been designed to transfer knowledge learned from large pre-trained vision language models like CLIP [28], to multi-source UDA. In prompt learning, image representations are learned contrastively with piece of language text termed "prompt". Consequently, prompts are tuned with the backbone network fixed for more efficient adaptation to downstream tasks (See Figure 1 for a comparison). While recent studies [10, 1] suggest that learnable prompts can be used for UDA, they are restricted the single source scenario and directly generalizing them to the multi-source setting produces limited results.
In light of this, we present surprisingly simple framework. Multi Aulti-Prompt Alignment (MPA), for multi-source UDA. MPA is composed of two one to learn an individual prompt by tuning a small set of parameters for each source and target domain pair, while the next to mine the relationships among the learned prompts by deriving a shared embedding space that is domain-invariant More specifically, given a source domain and a target domain, we use CLIP as our backbone and learn one prompt tailored for such pair. Then, inspired by the intrinsic dimensionality of data [40], we further reconstruct the learned prompts using simple auto-encoder network, aiming at removing redundant information potentially stemmed from the discrepancies among all the source domains. Finally, given the denoised prompts, an C1 constraint is incorporated as our alignment strategy so that the prompts agree the classification of target images.
We conduct extensive experiments on multiple benchmarks and the results clearly show that our ap- proach outperforms state-of-te of-t-oo--hhe-rr the--at art methods in the multi-source setting. In particular, on DomainNet [26], the most challenging dataset for multi-source UDA date, MPA surpasses all state-of-the the-art methods. Additionally, as the latent space is optimized with prompts from multiple sources, it encodes knowledge shared by different domains and could potentially generalize to multiple target domains by traversing the space. Consequently, we show how to tune the learned low-dimensional embedding for efficient deployment to a streamlined set of target domains, strategy we name Latent Subspace Tuning (LST). In summary, our contributions three-fold:

---

We introduce Multi- Prompt Alignment (MPA) for multi-souree UDA. MPA takes advantage of prompt learning and thus is capable of achieving a balance between performance and efficiency compared with alternative methods.
Multi-Prompt
MPA learns latent space by maximizing the consensus of multiple learned prompts. Based this, we introduce Latent Subspace Tuning (LST) that is able to continuously adapt to streamlined set of target domains.
MPA achieves state-of-the-ar results on several popular benchmarks while LST offers comparable results with tunable parameters that are one order of magnitude less.
2 Related Work
2.1
Multi-Source Unsupervised Domain Adaptation
First studied by Yang al. [46], multi-source UDA has drawn increasing attention in the community. Throughout the years, various methods have been studied. For example, in MDAN [48] and DCTN [43], a discriminator applied with adversarial losses is trained so that the features from source and target domains are aligned. MFSAN [51] calculates and aligns the maximum mean discrepancy for each source and target pair. Similarly, M SDA [26] aligns the moment distance for both target and source domains. All these methods require a shared feature extractor to obtain
features. On the contrary, Rakshit et al. [29] adopt one domain-specific encoder for each source and target pair while Zhao et al. [49] pre-trains a classifier for each source domain and then adversarially maps the target images into each trained feature space. The better alignment of these methods is at the cost of a significantly increased number of parameters needed for training. More recently, Xu al. [44] propose combine joint alignment and separate alignment for better adaptation, and Deng et al. [5] propose a method that detects and alleviates the negative impact from pscudo-labels for better self-training scheme.
2.2 Prompt Learning
Traditionally, given pre-trained language model, a common approach in deep learning fine-tuning the whole model or its task-specific heads to adjust to downstream tasks. While effective, however, two main drawbacks exist. First of all, the model size keeps increasing, pre-training and fine-tuning are becoming more and more expensive. Secondly, for each new task, fine-tuning needs to be repeatedly conducted. Recently, researchers have shown that pre-trained large-scale language models can handle wide range of downstream tasks with only a few or even samples by pre-pending instructions to the input text [20]. Such instruction texts are called prompts. Consequently, prompts can be tuned instead of the entire network for more efficient adaptation to downstream tasks. Originally, prompts are essentially sequences of manually designed language tokens that are mapped to embedding space. To date, extensive research has demonstrated that training soft prompts, iee., prompts with their own parameters learned by deep models, are more effective [18, 16]. The success of prompt learning in NLP has also garnered attention in the vision community which motivated the establishment of many related works. To name a few, Zhou et al. [50] are the first to apply soft prompt learning to the image recognition task. Ju et al. [13] explore prompt learning for efficient and lightweight video understanding. While promoting in these studies is limited to the input of text encoders, Jia et al. [12] prepend prompt tokens directly the image patches.
3 Method
Our goal is use multiple labeled source domains for improved performance on target domains. To this end, we leverage prompt learning, which is an effective strategy by learning small set of parameters to adapt a pretrained model to different downstream tasks. the following, we first review prompt learning in CLIP in Sec. 3.1 and then elaborate in Sec. 3.2 and Sec. 3.3 our proposed MPA and LST method respectively.
3.1 An Overview of Prompt Learning in CLIP
CLIP consists of an image encoder and text encoder that are jointly trained with a contrastive loss on 400M image and text pairs. The image encoder which can either be ResNet [11] or Vision
3

---

Domain-Invariant feature
Domain-specific featuree
Text Encoder
+01
Source
Prompt
Source Image
Target Prompt
Image Encoder
0.82
0.00
0.011
0.12
0.04
0.00
0:11
0.02
0.001
0.75
0.05
0.000
Target
Figure 2: Each source and target pair prompt P. is the concatenation of a "source prompt" segment and a "target prompt" segment, both composed of domain-invarian and Nomain--specife features. Therefore, the size of P, is R22 x (M, M2MM+MM))XMMM M2)x512, During our prompt training step, the text encoder and the image encoder of CLIP are both frozen.
Transformer [7], maps raw images to an embedding space, and the text encoder is Transformer [36] that projects an input text sequence to the same embedding space. prompt in CLIP usually exists in the form of "a photo of [CLS)" where [CLS] is a class token that can be replaced by certain class name. This sequence of tokens first converted into lower-cased byte pair encoding (BPE) representation, which is essentially a unique numeric ID) [50]. Then the numeric IDss are embedded in a 512 dimension vector that further passed to the Transformer text encoder. In our work, instead of using manually crafted prompts, we train soft prompts that are directly embedded by the text encoder. Given an image v and a text embedding wy for class {1, 2.... K) }, where is the total number of categories, CLIP aligns them in a contrastive manner that
plu = k//)> = exp(< Wk. f(x)) > (T)
ΣK exp( < wi, f(x) > (T)
is maximized when the input image x indeed belongs to class k. Here <..>> denotes the cosine similarity and 7 is a learnable temperature parameter.
3.2 Multi-Prompt Alignment
Let N denote the total number domains, where the first N domains are source domains and the N-th domain is the target domain. For multi-source UDA, our goal is to learn comainn-nvariian latent space so that the domain shift among different source domains as well as the discrepancies between all the source and target domain pairs can be minimized. To achieve such a goal, we introduce the framework MPA. Specifically, we design prompts to contain domain-invariant and domain-specifis feature following Ge et al. [10] and train such a prompt for each source and target domain pair. Then, an auto-encoder structure is introduced to our method to denoise the acquired prompts, followed by L1 constraint for further alignment.
Prompt Design. Following Ge et al. [10], our prompt for multi-source UDA includes a set of class-specific context tokens oft i 21,2,... Mi). k E (1. (1.2.... K) and another set of domain- specific tokens shared across all classes d E 21, M2),d d E {s, See Figure for an overview. Here, Mi and M2 represent the number of tokens, K is the number of classes, is short for source and t short for target, resulting in total of 2K categories for training with contrastive loss. Therefore, the prompt for each source and target pair can be derived as:
P. p== 1
th IT (1,2,.., N -
These prompts serve as learnable parameters that bridge the domain gap between a source domain and target domain through contrastive loss, will be introduced next.

---

Trainable Parameters
MPA
Parameters
LST
Real World
Target Domain
Clipart
Alignment
D
Product
Tuning latent subspace
Domain invariant latent subspaac
(a) Alignment in MPA on the Office-Home Home datasct with Rw as target domain
(b) Illustration of LST
Figure 3: (a) Example of prompt alignment on the Office-Home dataset. Here, P1, 22 P3 are prompts for domain Ar-Rw, CI-Rw and Pr-Rw respectively. All prompts are projected into the same latent space for alignment by an auto-encoder structure. (b) When facing a new target domain, tuning the latent subspace learned by the auto-encoder in MPA can allow quick adaptation that more computationally efficient.
Prompt Learning for Multi-source UDA. To apply prompt learning to multi-source UDA, we first train individual prompts for each source and target pair using the image and text encoders of CLIP. Given an image x" sampled from the source domain D, whose label is y², we optimize the prompts that the outputs from the image and text encoder are aligned. For an image xf from the target domain D; whose label is unknown, we first leverage the strong zero-shot ability of CLIP to generate static pseudo-label y for image-text alignment. Pseudo-labels are only generated for images whose maximum probability is larger than a fixed threshold 7 using Equation 1. While more sophisticated approaches like self-training could be leveraged generate pseudo-labels [52, 19], we find that pseudo-labels from CLIP are simple and effective. Finally, prompts are trained with cross-entropy loss functions and Figure gives overview of the process. More formally, for prompt P., 21.,...., N- - 1}, the objective function for optimization follows:
min P.
logP(y = y= P.)
Σ
logP(y ( = ÿ x'
(3)
The
Thg
ae rD
Here, the probability Ply ki d P.) of image sample belonging the k-th class is derived from a contrastive loss:
exp( < g(( f(t)), f(x)))()))))))))))))))))) > TT)
P(y = k) Ede(o.)) ΣK exp(< g(t)), f(x²) IT)
(4)
where d E {s,t) is domain identifier indicating where the image comes from, T is learnalle temperature parameter, and f and represents the image and text encoder in CLIP respectively, which are kept frozen during training. This specific design can push the prompts to learn disentangled representation of both class-invariant and class-specific semantie information boost the performance of domain adaptation methods [3, 21].
Even though Eqn 3 allows to obtain a prompt for each source and target domain pair, the noise level the learned prompts differs due to variations in the amount of images and the domain gap between each source and target domain. One obvious consequence is that they might produce inconsistent results even for the same target domain image. Therefore, an L2 constraint (Eqn 5) can be applied as strategy for prompt alignment.
N-2 N-1
2
Σ #1i=11
Σ P(y W|aK, P.) P(y = = P3)1
L1 =
(5)
- 1 -
Better Alignment through Reconstructed Prompts. While directly aligning the learned prompts produce decent results for multi-source UDA, the prompts are high-dimensional and might contain redundant information. Motivated by the theory that high dimensional data usually lies in lower
5

---

dimensional manifold [40], we build upon auto-encoders for better alignment. By using such an architecture, we hope learn a domain-invariane "latent subspace" of denoised prompts so that redundant information can be removed through the reconstruction of the learned prompts. More formally, we use an auto-encoder consisting of a projection function Proj(.) and back-projection function (.) The learned prompts P. are first projected into a latent subspace of a lower dimension d, by Proj(-), followed by (r)j,() projecting the vectors back into soft prompts P, The Proj(-) function is implemented by a one-layer feed-forward network while (roj.()) is two-layer nonlinear perceptron:
Proj(P:) (P.P)) + b
(6)
Proj,(vr) = (tann( W20r + b1)) + b
(7)
where 82 Pr)j((.)) and optimize reconstruction loss:
Σ LAE N 1=1
P. - P:
(8)
We then perform prompt alignment, i.e. the Li constraint, on the reconstructed prompts. Therefore, the overall objective function can be written as:
L LOLs + LAE aL1,
where LCLS is the cross entropy loss calculated using the static pseudo-labels that helps with the reconstruction process. Here is a hyper-parameter controlling the weight of the L1 loss. The whole alignment procedure is depicted in Figure 3a. Finally, for predicting the labels of target samples, we compute the average of the output logits using all P.s.
3.3 Latent Subspace Tuning
In real-life applications, it is more practical when adaptation to streamlined set of targets is needed. While we can repeatedly apply MPA each target domain, computationally inefficient especially when equipped with a large-scale backbone model. To mitigate this issue, introduce an LST strategy that explores the latent space derived by autoencoders for fast adaptation. The key idea is that since the latent space of the learned auto-encoder from MPA is optimized with prompts from multiple source domains, it alone capable of encoding domain-invann invariant knowledge. Therefore, we adopt MPA the first target domain, and traverse the subspace learned by MPA to generalize to the following ones.
More formally, given streamlined set of target domains DT, Dr...... Dr. to continuously adapt to each one of them in computationally efficient manner, we first conduct MPA on domain Dr,- After successfully applying MPA, a low-dimensional embedding space that captures the relation- ships among different domains is derived and can be leveraged for quick adaptation to domains Dr....., DTz, Specifically, for each following target domain Dr, domain-invarian feature vector Vuunc E RN M, xd, together with domain-spectie initialized and passed to the learned back projection function Proj,(.) from MPA. As a result, an entirely new prompt pT; = Concat[Proj, ( Proju (dGie)) can be constructed. Again with the help of pseudo-labels, can tune this new prompt by minimizing the following objective function:
feature vector di R xd, is randomly
min
Σ Dr
logP(y 1/" (2; drine- V
(10)
TK::
which is essentially LCLS Eqn and inference on Dr; can be conducted by only using PT. illustration of L.T is shown in Fig 3b. Since the prompt for CL.P is of size K x token length x embedding size, adopting the LST strategy can both reduce the need of training individual prompts embedding size
and decrease the number of tunable parameters by factor of at least (N - 1) x dy times compared with MPA. This is fact non-trivial when confronting large-scale datasets such as NomainNet. where N 6, dj 250, and embedding size 512.

---

Experiments
4.1 Experimental Setup
Experiments are conducted on three popular benchmark datasets UDA to Office-Home, Home, and DomainNet. ImageCLEF a small-scaled dataset consisting of 1,800 images from object categories in 3 different domains: ImageNet ILSVRC 2012 (I), Pascal VOC 2012 (P), and Caltech-250 (C). Office-Home is medium- scaled dataset consisting about 15,500 images from 65 categories in different domains: Art, Clipart, Product, and Real World. DomainNet is the largest dataset date, consisting of about 0.6 million images from 345 categories in 6 different domains: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch.
Datasets and Metries.
evaluate the effectiveness of MPA, namely ImageCLEF, Office
We use top-1 accuracy as our evaluation metric and report results of the following settings: (1) CLIP: zero-shot CLIP on the target domain, which can be regarded (2) Source Combined: source domains are combined into one single domain and applied with popular single-source UDA methods. Specifically in this setting, we adopt the prompting method from Zhou al. [50] to serve as another baseline named as "Simple Prompting". (3) Multi-Source: results reported from other multi-source UDA methods.
baseline of our method.
Implementation Details For fair comparisons, we adopt ResNet50 as our backbone on Image- CLEF and Hooece-Home and ResNet101 on DomainNet. The weights from CLIP and frozen throughout the experiments. Prompts and auto-encoder of MPA are trained using the mini-batch SGD optimizer with learning rate of 0.003 and 0.005 while the learned subspace is tuned with 0.0005 learning rate in LST. We use batch size 32 and adopt cosine learning rate scheduler. For hyper-parameters, token lengths M, and M2 are both 0.4 for producing reliable labels. or in Equation 9 is set to 500. The weight matrix W2 of the back projection function in Equation 7 has size of R384xd1 where d, is 100 for ImageCIERP 150 for OfficeHome and 250 for DemainNet.
16. Pseudo-label threshold is set
4.2 Comparison to State-of-the-Art
Multi-Prompt Alignment The results ImageCLEF and Office-Home shown in Table 1. For ImageCLEF, obvious that MPA outperforms other methods with an average accuracy of 91. 1,,, where there is least 3% increase when adapting to domain C and For Office-Home, MPA achieves the best results except when adapting to the domain Clipart, Nevertheless, we achieve an accuracy of 75.4% on average, which 1.3% higher than the second best method MFSAN. is worth noting that compared to state-of-the-as method MFSAN on both datasets, MPA only trains 0.78M and 2.36M parameters, while MFSAN has total of 51.75M and 51,80M parameters needed for optimizing (66.3 and 21.9 times larger than ours).
ImageCLEF
Office-Home
C
P
Avg
+ Ar
-
CI
Pr
Rw
Avg
Zero-Shot
CLIP [28]
95.1
87.3
74.0
85.5
71.5
50.2
81.3
82.4
71.4
Source Combined
DAN [22]
93.3
92.2
77.6
87.7
68.5
59.4
79.0
82.5
72.4
DANN [9]
93.7
91.8
77.9
87.8
68.4
59.1
79.5
82.7
72.4
D-CORAL [33]
93.6
91.7
77.1
87.5
68.1
58.6
79.5
72.2
DAPL" [10]
96.0
89.2
76.0
87.1
72.8
51.9
82.6
83.7
72.8
Simple Prompt
93.6
90.6
80.9
88.4
70.7
52.9
82.9
83.9
72.4
Multi-Source
DCTN [43]
95.7
90.3
75.0
87.0
N.A.
N.A.
N.A.
N.A.
N.A.
MDDA [49]
N.A.
N.A.
N.A.
N.A.
66.7
62.3
79.5
79.6
SImpAl [37]
93.3
91.0
87.3
70.8
56.3
80.2
72.2
MESAN [51]
95.4
93.6
79.1
89.4
72.1
62.0
80.3
81.8
74.1
MPA (ours)
98.6
96.2
80.4
91.7
74.8
54.9
86.2
85.7
75.4
Table Accuracy (%) on ImageCLEF and Office-Home. 'implies that the method is based on our implementation
7

---

DomainNet
Clp
Inf
Pnt
Qdr
Rel
Skt
Avg
Zero-Shot
CLIP [28]
61.3
42.0
56.1
10.3
79.3
54.1
50.5
Source Combined
DANN [9]
45.5
13.1
37.0
13.2
48.9
31.8
32.6
MCD [32]
54.3
22.1
45.7
58.4
43.5
38.5
DAPL [10]
62,4
43.8
59.3
10.6
81.5
54.6
52.0
Simple Prompt"
63.1
41.2
57.7
10.0
75.8
55.8
50.6
Multi-Source
S [26]
58.6
26.0
52.3
62.7
49.5
42.6
10 [37]
66.4
26.5
56.6
18.9
68.0
55.5
48.6
LIC-MSDA [38]
63.1
28.7
56.1
16.3
66.1
53.8
47.4
T-SVDNet [17]
25.0
54.3
16.5
65.4
54.6
47.0
PFSA [8]
64.5
29.2
57.6
17.2
67.2
55.1
48.5
PTMDA [30]
66.0
28.5
58.4
13.0
63.0
54.1
47.2
MPA (ours)
65.2
47.3
62.0
10.2
82.0
57.9
54.1
Table 2: Accuracy (%) on DomainNet. implies that the method is based on our implementation
Table 2 shows that for DomainNet, MPA exceeds other multi-source UDA methods by more than 5%. To the best of our knowledge, this is the highest reported accuracy on this dataset so far with less than one third parameters optimized compared with most state-of-the thet methods. Regarding individual adaptations, MPA achieves best results on most of the adaptation tasks but performs mediocre on the Quickdraw domain. Interestingly, the result is even little worse than CLIP. We hypothesize that this because of the large domain gap between Quickdraw and other domains.
Latent Subspace Tuning Results from Table I and Table 2 are already strong indications of the success of MPA under conventional multi-source UDA settings. Now, we would like investigate more practical scenario, where adaptation streamlined set of target domains is required, by using the LST strategy as mentioned in Section 3.3.
Results of LST on the DomainNet dataset are shown in Table 3, where compared with re-applying MPA, performance only dropped by 0.3%. Nevertheless, L.ST achieves higher accuracy compared to most baseline methods of Table 2, including CLIP with increase of 3.2%. Significantly. the adaptation process with LST requires only 11 hours of GPU time on NVIDIA RTX 3090, whereas adapting using MPA takes 54 hours-a speed increase of approximately 5 times.
DomainNet
Inf, Clp
Clp.Inf
Inf. Pnt
Pnt, drr
Qdr,Rel
Rel, Skt
Avg
CLIP
61.3
42.0
56.1
10.3
79.3
54.1
50.5
MPA
64.9
46.6
61.8
10.2
82.5
57.5
53.9
LST
64.6
46.7
61.6
9.8
81.2
57.6
53.6
Table Results (%) continuous adaptation on DomainNet. Here + D1, D2 means that adaptation is performed on two target domains and we report the performance on the highlighted second one.
4.3 Ablation Study
CLIP backbone While we are the first to apply prompt learning to multi-source UDA with the help of a pre-trained CLIP model, most existing approaches that we compare with utilize ImageNet pre-trained backbones, which are generally considered to be inferior compared to CLIP. Therefore, we would like examine whether MPA's performance gain is simply from CLIP. For such purpose, two additional sets of experiments are conducted: (1) we apply simple prompt learning method [50] to the source combined scenario, dubbed as the Simple Prompting baseline. As demonstrated in Table I and 2, while it is 1.33% average better than zero shot CLIP. MPA still outperforms it with a significant margin; (2) we directly swap MESAN's ResNet50 backbone pre-trained on ImageNet to CLIP's image encoder. Surprisingly, when tested ImageCLEF, Table 4 shows that the performance

---

even drops by small margin of 0.3%. From these results, we infer that while CLIP indeed provides strong backbone, is not necessarily universally superior, as also observed on the basis of studies Devillers et al. [6]. Yang et al. [45]. Moreover, despite CLIPPs strong zero-shot performance as stated in Radford al. [28], our method consistently outperforms CLIP the domain adaptation setting, especially in cases on the ImageCIEF and OfficeHome datasets where CLIP's zero-shot performance limited. On these two datasets, MPA consistently delivers excellent results that surpass CLIP by 6.2% and 4.0%, respectively. Therefore, we believe while CLIP's strong backbone indeed contributes to MPA's good results, alone is insufficient for such superior performance and another important piece is MPA's strong domain adaptation ability.
ImageCLEF
I
P
Avg
CLIP [28]
95.1
87.3
74.0
85.5
MFSAN [51]
95.4
93.6
79.1
89.4
MFSAN+CLIP
96.7
93.0
77.7
89.1
MPA (ours)
98.6
96.2
80.4
91.7
Table 4: Exploration of MFSAN equipped with CLIP's backbone on ImageCLEF dataset. 'implies that the method is based on our implementation
Prompt engineering Apparently, the performance of MPA depends on the quality of lsbuls-labels generated by CLIP. As such, it is worth investigating whether MPA has been pushed to its maximum extent by using more sophisticated prompts than "A photo of [CLS)". Specifically, we tested two
additional sets of prompts: (1) "a [Dom] of [CLS]" (e.g., a painting of dog) and (2) "a photo of
[CLS] in domain [Dom]" (e-g., a photo of dog in domain painting) with the intention of incorporating domain information, and as is presented in Tab 5, while each of the three prompts shows certain advantage on some specific domains, in general, the naive "A photo of [CLS]" still performs the best.
Zero-shot
MPA
++ Ar
+
CI
->
- Rw
Avg
+
-
+ Pr
+ Rw
Avg
a [Dom] of (CLS) a photo of [CLS] in domain [Dom]
71.7
52.4
74.9
81.0
70.0
74.1
55.1
82.2
85.0
74.1
68.
53.1
81.3
82.0
71.1
72.7
55.7
87.1
85.2
75.2
a photo of [CLS]
71.5
50.2
81.3
82.4
71.4
74.8
54.9
86.2
85.7
75.4
Table 5: Ablation studies different choices of manual prompt the Office-Home dataset.
hyyper-parameter selection Experimental results for ablation studies on
selections
reported in Table 6. For both pseudo-label threshold and prompt token length M,, M2, three different choices 7 E (0.3, 0.6, 0.8) and M,, M2 E (8, 12, 20) are examined. For simplicity, we setting M1 and M2 be equal. As r increases, while the quality of the pseudo-labels gets higher, fewer images will be fed into the model, and Table 6a suggests that doing so hurts the overall performance. On the contrary, shown in Table 6b, the general trend for prompt token length is that the longer the prompt, the better the performance. As result, we choose T = 0.4 and Mi = M2 = 16 to balance the trade-off between performance and efficiency. For a in Equation 9, we chose to be 500 to balance all losses (in this case L1) to be of the same order of magnitude. Our experimental results from Table 6c also support such motivation.
Alignment strategy, prompt denoising and prompt design To validate the effectiveness of our alignment strategy, we evaluate our method without the &1 loss, and table 7 shows that this will result in a performance degradation of about 0.7%. In particular, the Li constraint exhibits a greater impact on the more difficult Clipart domain, where the accuracy dropped by a large margin of 1.5% without it, demonstrating the effectiveness of constraining consistent predictions on different prompts. Subsequently, we tested the effectiveness of denoising the prompts through auto-encoder reconstruction. Again, results show that the incorporation of an auto-encoder structure with the LAF loss is beneficial to the overall performance. Finally, we examined the necessity of using class and domain-specift tokens for learning prompts and results from Table 7 support such design. Specifically, implies that both the class-specific and domain-specift, prompts are used while X implies that only the class- prompts are used.

---

| Ar
Pr
Avg
Token length
Ar
CI
+
Rw
Avg
0.3
74.5
55.0
86.1
85,9
75.4
M1 = M2 =
74.3
54.9
85,8
85.3
75.1
0.6
74.6
54.9
85.9
85.3
75.2
M1 M2 12
74.6
54.3
85.9
85.6
75.2
0.8
74.0
54.2
85.2
85.5
74.7
M1 M2 20
74.8
55.2
86.3
86.0
75.6
0.4 (reported)
74.8
54.9
86.2
85.7
75.4
M, M2 = 16 (reported)
74.8
54.9
86.2
85.7
75.4
(a) Ablation pseudo-label threshold T.
(b) Ablation token lengths M1 and M2.
Ar
CI
Pr
Rw
Avg
53.7
84.9
85.6
74.7
74.5
54.1
85.7
86.0
75.1
100
74.7
54.5
85.5
85.6
75.1
1000
74.4
55.0
86.3
86.0
75.4
500 (reported)
74.8
54.9
86.2
85.7
75.4
(c) Ablation on
Table 6: Ablation studies on hyper-parameter selection on the Office- Home dataset.
L1
LAE
Prompt Design
+ Ar
+
CI
+
Pr
Rw
Avg
X
/
74.8
53.4
85.2
85.3
74.7
x
/
74.0
53.8
85.5
85.4
74.9
74.2
53.0
85.2
85.3
74.4
74.8
54.9
86.2
85.7
75.4
Table 7: Ablation studies various modules of MPA on the Office-Homme dataset.
Conclusion
In this paper, we introduced prompt learning multi-source UDA and proposed simple MPA scheme to align the source and target domains. MPA is composed of two steps. The first step is to train individual prompts for each source and target domain pair and the next one to align them after reconstruction using an auto unto-enccderr structure. Extensive experiments showed that MPA achieved better results than state-of-the-an art methods on various multi-source UDA tasks with substantially fewer parameters tuned. Moreover, an LST strategy was introduced for efficient adaptation to streamlined set of target domains. As the model contains useful clues of multiple domains, one potential limitation is that faces more risks in terms of information leakage where adversaries might produce attacks steal information from the model.
Acknowledgement This project was supported by NSFC under Grant No. 62102092.
10

---

References
[1] Eyal Ben-David, Nadav Oved, and Roi Reichart. Pada: Example-based prompt learning for on-the-fly adaptation to unseen domains. TACL, 2022.
[2] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NeurlPS, 2006.
[3] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In NeurIPS, 2016.
[4] Gabriela Csurka. Domain adaptation for visual applications: comprehensive survey, arXiv preprint arXiv: 2017.
[5] Zhongying Deng, Da Li, Yi-Zhe Song. and Tao Xiang. Robust target training for multi-source domain adaptation. arXiv preprint arXiv:2210.0167,, 2022.
[6] Benjamin Devillers, Bhavin Choksi, Romain Bielawski, and Rufin VanRullen. Does language help generalization in vision models? arXiv preprint arXiv: 104.08343 2021.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[8] Yangye Fu, Ming Zhang, Xing Xu, Zuo Cao, Chao Ma. Yanli Ji, Kai Zuo, and Huimin Lu. Partial feature selection and alignment for multi-source domain adaptation. CVPR, 2021.
[9] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversaria training of neural networks. JMLR, 2016.
[10] Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning, arXiv preprint wwXiv:2202.06687 2022.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[12) Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint urXiv:2203. /2119, 2022.
[13] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual- Sanaul-a models for efficient video understanding. arXiv preprint arXiv:2112.04478 2021.
[14] Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In CVPR, 2019.
[15] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, 2012.
[16] Brian Lester, Rami AI-Rfou, and Noah Constant. The power scale for parameter-efficient prompt tuning. arXiv preprint (00.:::2144.000000 2021.
[17] Ruihuang Li, Xu Jia, Jianzhong He, Shuaijun Chen, and Qinghua Hu. T-svdnet: Exploring high-order prototypical correlations for multi-source domain adaptation. In ICCV, 2021.
[18] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 2021.
[19] Hong Liu, Jianmin Wang, and Mingsheng Long, Cycle self-training for domain adaptation. In NeurIPS, 2021.
[20] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing, arXiv preprint 210.v:210..13586, 2021.

---

[21] CennCheng Liu, Yu- Ying Ych, Tzu-Chien Fu, Sheng- De Wang, Wei-Chen Chiu, and Yu- Chiang Frank Wang. Detach and adapt: Learning cross-domain disentangled deep representation. In CVPR, 2018.
[22]
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015.
[23] Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In CVPR, 2022.
[24] Lingchen Meng, Xiyang Dai, Yinpeng Chen, Pengchuan Zhang, Dongdong Chen, Mengchen Liu, Jianfeng Wang, Zuxuan Wu, Lu Yuan, and Yu-Gang Jiang. Detection hub: Unifying object detection datasets via query adaptation on language embedding. In CVPR, 2023.
[25] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. TKDE, 2010.
[26] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV, 2019.
[27] Joaquin Quinonero- Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. 2008.
[28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., 2021.
[29] Sayan Rakshit, Biplab Banerjee, Gemma Roig, and Subhasis Chaudhuri. Unsupervised multi- source domain adaptation driven by deep adversarial ensemble learning. In GCPR, 2019.
[30] Chuan-Xian Ren, Yong- Hui Liu, Xi-Wen Zhang, and Ke-Kun Huang. Multi-source unsupervised domain adaptation via pseudo target domain. 77P, 2022.
[31] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[32] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR, 2018
[33] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, 2016.
[34] Antonio Torralba and Alexci A Efros. Unbiased look at dataset bias. In CVPR, 2011.
[35] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[37] Naveen Venkat, Jogendra Nath Kundu, Durgesh Singh, Ambareesh Revanur, et al. Your classifier can secretly suffice multi-source domain adaptation. In NeurlPS, 2020.
[38] Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang. Learning to combine: Knowledge aggregation for mouri- domain adaptation. In ECCV, 2020.
[39] Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing, 2018.
[40] Wei Wang, Yan Huang, Yizhou Wang, and Liang Wang. Generalized autoencoder: A neural network framework for dimensionality reduction. In CVPR workshops, 2014.
[41] Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. In NeurIPS, 2021.
12

---

[42] Zejia Weng, Xitong Yang, Ang Li. Zuxuan Wu, and Yu-Gang Jiang. Open-velip: Transforming clip an open- ahen-vocabulary video model via interpolated weight optimization. 2023.
[43] Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In CVPR, 2018.
[44] Yuanyuan Xu, Meina Kan, Shiguang Shan, and Xilin Chen. Mutual learning of joint and separate domain alignments for multi-source domain adaptation. In WACV, 2022.
[45] Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao.
Unified contrastive learning in image-text-label space. CVPR, 2022.
[46] Jun Yang, Rong Yan, and Alexander G Hauptmann. Cross-domain video concept detection using adaptive svms. In ACM MM, 2007.
[47] Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. ICML, 2013.
[48] Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey Gordon. Adversarial multiple source domain adaptation. In NeurlPS, 2018.
[49] Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-source distilling domain adaptation. In AAAI, 2020.
[50] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.
[51] Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang, Aligning domain-specific distribution and classifier for cross-domain classification from multiple sources. AAAI, 2019.
[52] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class balanced self self-training. In ECCV, 2018.
13


Reviews:
[{"review_id": "BCsvFzBvZ8", "reviewer": "Reviewer_tSjx", "summary": "The paper proposes an extension of [10] (Domain Adaptation via Prompt Learning Ge et al., 2022) to the multi-source UDA set-up.\n(i) Distinct soft prompts are learnt via contrastive loss for each source-target pair;  each source-target prompt is composed of class-wise source- and target-prompts. In the target domain, learning is achieved by utilizing pseudo-labels from the pretrained CLIP model, which possesses strong zero-shot ability.\nTo encourage consistent target outputs across all learned prompts, a consistency L1 loss is applied to the soft outputs of all target samples.\n(ii) Prompt reconstruction: to eliminate redundant information that may impede performance, an autoencoder (AE) is learned to \"denoise\" the acquired prompts.\n(iii) Latent Subspace Tuning (LST): efficient adaptation to a new target domain, after learning from the first target domain, is accomplished by optimizing on the latent space of the learned AE using pseudo-labels of the new target.\n\nThe proposed framework demonstrates superior results compared to previous State-of-the-Art (SOTA) methods in multi-source UDA.", "questions": "Given that CLIP alone produces results close to the state-of-the-art (SOTA) as shown in Table 1, and even outperforms the SOTA as shown in Table 2, my main concern is whether the authors have made adequate efforts to create decent baselines using CLIP-pretrained models. I appreciate their effort in Section 4.3 when they ``swap MFSAN’s ResNet50 backbone pre-trained on ImageNet to CLIP’s image encoder''. However, I would love to know more details to know as if careful considerations were taken into account for MFSAN + CLIP or simply the authors swap the backbone's weight. For example, did the authors preserve the text classifier and the contrastive los of CLIP ? As the proposed framework is based on self-training with pseudo-labels, I wonder if how the CLIP + self-training and Single Prompt + self-training baselines perform.\n\nFurthermore, I am curious to know if CLIP with fixed prompts has been pushed to its maximum extent. In other words, how far can this naïve baseline reach with better prompt engineering instead of using the simple prompt \"a photo of [CLS]\"? One can imagine incorporating domain-specific information into the prompt or defining a set of templates rather than relying on just one template.\n\nIn the LST experiment, how good is MPA (on the first domain) + self-training? For efficiency, one can try test time prompt tuning (TPT).", "limitations": "This work lacks significant technical novelty, and the experiment could be improved by using stronger baselines. Based on the current state of the submission, my recommendation leans towards the negative side.\n\n---- After rebuttal ----\nThe rebuttal is convincing and has helped clarify most of my technical concerns. I believe that this work is indeed interesting and help advocate the usage of prompts in domain adaptation. While the novelty limitation still pertains, the original set of experiments, along with the new ones provided during the rebuttal, is sufficient. I believe the paper after revision would certainly pass the NeurIPS' bar.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method leverages the powerful zero-shot capability of the pretrained CLIP model for the multi-source domain adaptation (DA) task. Empirical results demonstrate its effectiveness across various benchmarking setups. The presentation is good, enabling easy comprehension of the method.", "weaknesses": "The main concern of this work is the lack of technical novelty and a more rigorous evaluation. The proposed framework is a straightforward extension of the work by Get et al. (2022) [10] to the multi-source setting. In terms of method evaluation, it is expected to compare against stronger CLIP-based baselines, both in the multi-source UDA experiments (Tables 1 and 2) and in the LST experiment (Table 3). Further details are provided in the following section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *Lack of technical novelty. The proposed framework is a straightforward extension of the work by Ge et al. to the multi-source setting.*\n\nWe respectfully disagree. As shown in Table 1 and 2, directly applying Ge's method to MDA produces limited performance. This is because their method lacks strategies on dealing with domain gap among the source domains. On the contrary, Table 6 shows that with our proposed alignment strategy, domain shift among source domains is effectively reduced, resulting in a significant performance boost. \n\n> *Lack of a more rigorous evaluation. It is expected to compare against stronger CLIP-based baselines.*\n\nTo the best of our knowledge, there are **no** other CLIP-based MDA baselines. To make up for such shortcomings, we proposed **3** CLIP-based baselines, i.e., the zero-shot CLIP baseline, the DAPL in the source combined scenario baseline, and a Simple Prompt baseline based on the paper of CoOp. Results from Table 1 and 2 show that MPA consistently achieves better results than these methods. \n\n> *Given that CLIP alone produces good results, have the authors made adequate efforts to create decent baselines using CLIP-pretrained models.*\n\nWe would like to highlight again that there are no other CLIP-pretrained baselines. Therefore, we believe **CLIP itself forms a decent baseline**. In addition to CLIP, we have also proposed a Simple Prompt baseline, and tested a SOTA method with CLIP-pretrained backbone. More details on these baselines are in the following answers. \n\nWe understand that concerns might be raised by using CLIP and really have tried our best to present a fair comparison. If you feel that these are insufficient, would you please suggest us what other attempts we can do?\n\n> *Whether careful considerations were taken into account for MFSAN + CLIP?*\n\nWe've actually tried keeping the text classifier with the contrastive loss, but finally decided to not do so. The reasons are as follows.\n\nIn MFSAN, a common feature extractor together with domain specific feature extractors and domain specific classifier heads are used. In order to keep the text classifier, two main technical difficulties need to be overcomed: (1) Text classifier generated by CLIP is dependent on the input text prompt and would be the same for all domains if we use the naive \"a photo of a [CLS]\" prompt, which contradicts the design of MFSAN where domain specific classifier heads are used; (2) In MFSAN, the features extracted from the common feature extractor are downsampled from 2048-d to 256-d, whereas text classifier from CLIP is 1024-d. \n\nTo solve the above issues, we could apply soft prompt methods and train a prompt for each domain to generate the domain specific heads, followed by either applying another linear layer to reduce their dimension to 252-d or only downsample the features to 1024-d. However, both settings produced unsatisfactory results:\n\n|           | C   | I   | P   | Avg   |\n|:-|:-:|:-:|:-:|:-:|\n|Linear layer to 252-d      | 21.0|37.8 |19.7 | 26.2  |\n|Downsample to 1024-d     | 88.7|81.7 |74.2 |  81.5 |\n|No text classifier |96.7 |93.0 |77.7 |89.1   |\n\nBased on the above results, further techniques might be needed to acquire a good performance, which however, would most likely result in a complete new method and is beyond the scope of analyzing whether the performance gain of MPA is from CLIP's pretrained visual backbone. Therefore, we chose not to keep the text classifier and only replaced the weights of the common feature extractor with CLIP's visual backbone weights.\n\n> *How the CLIP + self-training and Single Prompt + self-training baselines perform.*\n\nWhen using naive CLIP with manually designed prompts, there are no parameters needed for training (usually the image backbone is frozen). Therefore it is rather unclear to us what CLIP + self-training refers to. Did you mean CLIP with soft prompts + self-training? If so, this is exactly what the Simple Prompt baseline refers to in Table 1 and 2. In Simple Prompt we train a soft prompt using pseudolabels generated by CLIP and results show that simply doing so is ineffective in achieving good MDA ability. \n\nWe are also confused about the \"Single Prompt + self-training baseline\". Did you mean \"Simple Prompt + self-training\"? If so, in Simple Prompt, self-training is already used.\n\n> *If CLIP with fixed prompts has been pushed to its maximum extent.*\n\nThank you for the suggestion! Based on your comment, we tested two sets of prompts: (1) \"a [Domain] of [CLS]\" (e.g., a painting of dog) and (2) \"a photo of [CLS] in domain [Domain]\" (e.g., a photo of dog in domain painting). Their zero-shot results on Office-Home are shown in the following table:\n\n|             | Art   | Clipart   | Product   | Real World   | Avg |\n|:-|:-:|:-:|:-:|:-:|:-: |\n|a [Domain] of [CLS]         |71.7 |52.4 |74.9 |81.0   | 70 |\n|a photo of [CLS] in domain [Domain]|68.1 |53.1 |81.3 |82.0   | 71.1|\n|a photo of [CLS]    |71.5 |50.2 |81.3 |82.4   |71.4|\n\nWe also tested MPA's performance with pseudolabels generated from these two new prompts:\n\n|             | Art   | Clipart   | Product   | Real World   | Avg |\n|:-|:-:|:-:|:-:|:-:|:- |\n|a [Domain] of [CLS]         |74.1 |55.1 |82.2 |85.0   |74.1|\n|a photo of [CLS] in domain [Domain]|72.7 |55.7 |87.1 |85.2   |75.2|\n|a photo of [CLS]    |74.8 |54.9 |86.2 |85.7   |75.4|\n\nIn general, the default \"a photo of [CLS]\" still performs the best. While better handcrafted prompts might exist, for now due to time limitations we are unable to find significantly superior ones. \n\n> *In the LST experiment, how good is MPA (on the first domain) + self-training?*\n\nMPA on the first domain already uses self-training. The results are slightly worse than those in Table 2, as less source domains are used:\n\n|           | &rarr; Inf*,Clp   | &rarr; Clp*,Inf    | &rarr; Skt*, Pnt     | &rarr; Pnt*, Qdr     | &rarr; Qdr*, Rel   | &rarr; Rel*, Skt  |\n|:-|:-:|:-:|:-:|:-:|-:|:-:|\n|MPA         |46.7 |64.6 |57.5 |62.2   |10.1|81.8 |"}, {"review_id": "jfNJh7GkmC", "reviewer": "Reviewer_tF9m", "summary": "This paper deals with the multi-source domain adaptation problem. It proposes to tune the designed domain-invariant prompts and domain-specific prompts to enable the domain adaptation ability. Generally, the training consists of two objectives, i.e., the individual prompt learning objective and the de-noising objective via prompt auto-encoders. To enable a test-time domain adaptation ability and reduce the amount of learnable parameters, the authors propose LST. Experiments on various multi-source benchmarks verify the effectiveness of proposed method.", "questions": "See the weakness part.", "limitations": "I don't see any serious potential negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea is simple and generally reasonable.\n\n- The experiment results are good.", "weaknesses": "- In LST, I don't think it is reasonable to use randomly initialized representations as the input of the back-projector to perform prompt reconstruction, as the latent representations don't subject to the same distribution. And there is no empirical evidence to show such a way really works.\n\n- Missing comparisons to strong baselines. E.g., on DomainNet, [1] achieves a multi-source domain adaptation performance at 53.2 with a ResNet-101 backbone. The authors should compare to the strong baselines.\n\n  [1] Contrastive adaptation network for single- and multi-source domain adaptation, TPAMI 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *In LST, I don't think it is reasonable to use randomly initialized representations as the input of the back-projector to perform prompt reconstruction, as the latent representations don't subject to the same distribution. And there is no empirical evidence to show such a way really works.*\n\nAs pointed out in L130, the learned latent space is supposed to be domain-invariant. Besides, this representation is further tuned by pseudolabels of the target data. Therefore even though the representations subject to different distribution, the final result shouldn't be affected.\n\nAs for empirical evidence, there are actually a few research papers that would use a random initialization when tuning the latent space. For example, in [1], 1000 latent vectors are randomly initialized to find the best performing one. In [2], it is found that random initialized representations are better at generating pictures other than faces. Here the representations are also of a different distribution (faces v.s. cat, dog etc.). \n\n[1] Wen et al., Diamond in the rough: Improving image realism by traversing the GAN latent space\n\n[2] Abdal et al., Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?\n\n> *Missing comparisons to strong baselines. E.g., on DomainNet, [1] achieves a multi-source domain adaptation performance at 53.2 with a ResNet-101 backbone. The authors should compare to the strong baselines.*\n\nThank you for pointing this out! We will include [1] in our paper for a more comprehensive comparison. We would also like to note that since [1] performs a clustering on all source domains for each iteration, our method is much more efficient.\n\n|           |  Clp |   Inf   | Pnt    |  Qdr   | Rel  | Skt | Avg|\n|:------------|:-----:|:---:|:---:|:-----:|---:|---:|---:|\n|      MSCAN   |69.3 |28.0 |58.6 |  30.3 |73.3|59.5 | 53.2 |\n|MPA         |65.2 |47.3 |62.0 |10.2   |82.0 |57.9 | 54.1 |\n\n[1] Kang et al., Contrastive adaptation network for single- and multi-source domain adaptation."}, {"review_id": "bZMf7yBiV9", "reviewer": "Reviewer_UUa1", "summary": "This paper introduces prompt learning to multi-source unsupervised domain adaptation (UDA). Firstly, individual prompts for each source and target pair are learned using a contrastive loss. Then, MPA aligns the learned prompt by an autoencoder-based step with an L_1 constraint to generate consistent results for the same target domain image. In addition, the LST strategy delivers the first target domain adaptation to the subsequent target domains efficiently. Experiments on ImageCLEF, Office-Home, and DomainNet validate the effectiveness.", "questions": "In addition to the questions in the above weaknesses that need to be answered, there are a few things that need to be clarified.\n\n+  The generalization of CLIP is a huge advantage. Although the author discussed the impact of CLIP in 4.3, I hope that the author will discuss the design experiment of the prompt. In addition to using the original \"a photo of [CLS],\" the number of channels of the trainable prompt params can also be discussed.\n\n+ Please state the problems and experimental results faced when the approach of Ge [10] is transferred to the tasks addressed in this paper. And for the problem and results, describe the differences of this paper.\n\n+ For now, it is unknown why prompt, as a trained parameter, has class-specific and domain-specific attributes.", "limitations": "Not applicable", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ This paper's application of prompt learning to multi-source UDA problems is groundbreaking.\n+ This paper is well-written and easy to understand.", "weaknesses": "+  Regarding the prompt design part in sec3.2, can it only be realized by following Ge[10]? There is a lack of novelty and contribution due to the lack of original works in the whole prompt design.\n+  Regarding the problem of reducing the dimensionality of the high-dimensional learned prompt, is this step introduced because the prompt design is not optimal? From the perspective of Tab. 6, the growth brought by AE is very limited, so are other methods of dimensionality reduction effective, or even the step of AE can be removed by considering redundant information in prompt design?\n3 The LST in 3 sec 3.3 is a further extended use of MPA, which is somewhat insufficient as an independent innovation point.\n4 This paper needs to weaken the sense of the existence of CLIP and the dependence on the previous prompt design method, otherwise very much like a simple application essay.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *Regarding the prompt design part in sec3.2, can it only be realized by following Ge[10]?*\n\nThere are other ways of implementing the prompt structure. For example, we had considered training just a naive soft prompt for each source-target domain pair, followed by concatenating them and apply an additional linear/convolution layer to extract a \"common\" prompt for the target domain. However, emperically we find that such way of realizing performs slightly worse than using Ge's:\n\n|             | Art    | Clipart| Product |Real World   | Avg|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|convolutional fusion   |74.1 |54.0|85.3|85.2   |74.7|\n|Ge's |74.8 |54.9 |86.2 |85.7   |75.4|\n\nYet, we would like to highlight that the success of MPA is not only from such prompt design. As presented in Table 1 and 2, directly extending Ge's work to MDA would result in limited performance.\n\nFurthermore, we had also considered using both text and image prompts. However, since most of the SOTA's are using resnet based backbones, it is hard to incorporate image prompts to such architecture. Therefore for a fair comparison, we chose to not use such strategy.\n\n> *Regarding the problem of reducing the dimensionality of the high-dimensional learned prompt, is this step introduced because the prompt design is not optimal? From the perspective of Tab. 6, the growth brought by AE is very limited, so are other methods of dimensionality reduction effective, or even the step of AE can be removed by considering redundant information in prompt design?*\n\nThank you for the question! We would like to note that our intention is **reconstruction** rather than **dimension reduction**, and the reason is not because of the prompt design, but to \"remove redundant information potentially stemmed from the discrepancies among all the source domains.\", as stated in L55-57. This is one major reason why we kept the AE structure. Another reason is that by leveraging the auto-encoder structure, especially its decoder, we are able to adopt our LST strategy that allows efficient and effective adaptation to multiple target domains, which we believe is of practical importance.\n\n> *The LST in 3 sec 3.3 is a further extended use of MPA, which is somewhat insufficient as an independent innovation point.*\n\nWe respectfully disagree that LST doesn't serve as an innovation point. While LST is based on the domain-invariant latent space found by MPA, it surves different purpose and has a complete different design principle compared with MPA. Specifically, LST is most suitable in situation where adaptation to multiple target domains is needed. In such scenarios, MPA would require **all** source-target domain prompts to be repeatedly trained. On the contrary, for LST, all we need is to tune **one** prompt with only the target domain data, which significantly reduces the computational cost. As stated in L249, LST would boost the speed of adaptation on the DomainNet dataset of approximately 5 times. Such improvement is far beyond marginal and we humbly believe that it serves as a solid contribution.\n\n> *This paper needs to weaken the sense of the existence of CLIP and the dependence on the previous prompt design method, otherwise very much like a simple application essay.*\n\nThank you for the suggestion! We will revise accordingly.\n\n> *I hope that the author will discuss the design experiment of the prompt. In addition to using the original \"a photo of [CLS],\" the number of channels of the trainable prompt params can also be discussed.*\n\nAs a matter of fact, the channels of prompt parameters are fixed to 512-d in CLIP, as stated in L122. What could be changed is the length of the prompt token, i.e., the $M_1$ and $M_2$ in L140, and ablation study on these parameters are discussed in Section 4.3 (Table 5).\n\n> *Please state the problems and experimental results faced when the approach of Ge [10] is transferred to the tasks addressed in this paper. And for the problem and results, describe the differences of this paper.*\n\nThe usual way of extending single source method to multiple sources (the task in this paper) is to apply to the source combined scenario, where all source domains are combined into one single domain, and this is what we have done in the paper (DAPL in Source Combined). However, by doing so, one major problem is that no strategy is taken on dealing with domain gap among the source domains, and often the times will produce unsatisfactory performances. The experimental results for doing so are already shown in Table 1 and 2, and we have put them on the tables below for reference:\n\nImageCLEF:\n|           | C   | I   | P   | Avg   |\n|:-|:-:|:-:|:-:|:-:|\n|DAPL in Source Combined      | 96.0|89.2 |76.0.7 | 87.1  |\n|MPA     | 98.6|96.2 |80.4 |  91.7 |\n\nOfficeHome:\n|             | Art   | Clipart   | Product   | Real World   | Avg |\n|:-|:-:|:-:|:-:|:-:|:-: |\n|DAPL in Source Combined   |72.8 |51.9 |82.6 |83.7   | 72.8 |\n|MPA|74.8 |54.9 |86.2 |85.7   | 75.4|\n\nDomainNet:\n|           |  Clp |   Inf   | Pnt    |  Qdr   | Rel  | Skt | Avg|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|DAPL in Source Combined         |62.4 |43.8 |59.3 |10.6   |81.5|54.6 | 52.0 |\n|MPA         |65.2 |47.3 |62.0 |10.2   |82.0 |57.9 | 54.1 |\n\nHere we can see MPA surpasses DAPL in Source Combined setting by an average of 3.1%, indicating the efficacy of our approach. The reason for such performance gain is that unlike DAPL, in MPA we treated each source domain and target domain pair independently, and further incorporated our alignment strategy for dealing with the domain gap among the source domains.\n\n> *For now, it is unknown why prompt, as a trained parameter, has class-specific and domain-specific attributes.*\n\nGreat question! The intuition is that if we use a hard prompt like \"a [Domain] of [CLS]\", then for each class, the embedding of [Domain] will remain the same while that of [CLS] will change. We hope this is what the soft prompt will learn through training."}, {"review_id": "VABcTMhOLZ", "reviewer": "Reviewer_34CN", "summary": "This paper proposes an interesting pipeline for large pre-trained model-based UDA, and realizes a latent subspace tuning for continuous adaption.\n", "questions": "Please see the weakness part.", "limitations": "The authors should provide the limitation of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation behind this article is very meaningful, and the proposed method supports the motivation well.\n2. In terms of methodology, the design of this paper is clear and innovative for the combination of existing technologies.\n3. This easy but effective baseline and latent subspace tuning ability may provide new paradigm for future UDA.\n4. Good performances.", "weaknesses": "Using CLIP as the backbone may limit the ability of MPA on only classification tasks, how to extend MPA to other tasks, like segmentation, referring, etc? And how to utilize other large pre-trained models like GPT-3? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *Using CLIP as the backbone may limit the ability of MPA on only classification tasks, how to extend MPA to other tasks, like segmentation, referring, etc?*\n\nAs a matter of fact, there are many other CLIP-based segmentation/referring works [1] [2] [3], all of which used CLIP as backbones. Therefore we believe that CLIP backbone won't be a problem of transfering MPA to other tasks. \n\n[1] Lüddecke1 et al., Image Segmentation Using Text and Image Prompts\n\n[2] Liang et al., Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP.\n\n[3] Wang et al., CRIS: CLIP-Driven Referring Image Segmentation\n\n> *And how to utilize other large pre-trained models like GPT-3?*\n\nOne possible way to use other large pre-trained models is to use them for better text prompt generation. However, since CLIP is trained with its own GPT-2 like text encoder, further engineering techniques might be needed for a better adaptation.\n\n> *The authors should provide the limitation of their work.*\n\nThank you for pointing this out! As a matter of fact, we have provided certain limitation of our work in the Conclusion section: \"As the model contains useful clues of multiple domains, one potential limitation is that it faces more risks in terms of information leakage where adversaries might produce attacks to steal information from the model.\"\n\n"}]

Author rebuttal:

Meta-review: The reviewers all agreed that the problem is important and that the method is sound. Several reviewers expressed reservations about the novelty of the method, namely the differences to [10]. They also raised several questions of detail. The authors provided a rebuttal that addressed most of the reviewer concerns. After discussion, all reviewers were positive towards publication, although none was strongly supportive. It was nevertheless considered that the paper will make an interesting contribution.
Decision: Accept (poster)