Title: Multi-Prompt Alignment for Multi-Source Unsupervised Domain Adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi source unsupervised domain adaptation; transfer learning; computer vision
Abstract: Most existing methods for unsupervised domain adaptation (UDA) rely on a shared network to extract domain-invariant features. However, when facing multiple source domains, optimizing such a network involves updating the parameters of the entire network, making it both computationally expensive and challenging, particularly when coupled with min-max objectives. Inspired by recent advances in prompt learning that adapts high-capacity models for downstream tasks in a computationally economic way, we introduce Multi-Prompt Alignment (MPA), a simple yet efficient framework for multi-source UDA. Given a source and target domain pair, MPA first trains an individual prompt to minimize the domain gap through a contrastive loss. Then, MPA denoises the learned prompts through an auto-encoding process and aligns them by maximizing the agreement of all the reconstructed prompts. Moreover, we show that the resulting subspace acquired from the auto-encoding process can easily generalize to a streamlined set of target domains, making our method more efficient for practical usage. Extensive experiments show that MPA achieves state-of-the-art results on three popular datasets with an impressive average accuracy of 54.1% on DomainNet.

Multi-Prompt Alignment for Multi-Source Unsupervised Domain Adaptation

Abstract

Most existing methods for unsupervised domain adaptation (UDA) rely on a shared network to extract domain-invariant features. However, when facing multiple source domains, optimizing such a network involves updating the parameters of the entire network, making it both computationally expensive and challenging, particularly when coupled with min-max objectives. Inspired by recent advances in prompt learning that adapts high-capacity models for downstream tasks in a computationally economic way, we introduce M ultiP rompt A lignment (MPA), a simple yet efficient framework for multi-source UDA. Given a source and target domain pair, MPA first trains an individual prompt to minimize the domain gap through a contrastive loss. Then, MPA denoises the learned prompts through an auto-encoding process and aligns them by maximizing the agreement of all the reconstructed prompts. Moreover, we show that the resulting subspace acquired from the auto-encoding process can easily generalize to a streamlined set of target domains, making our method more efficient for practical usage. Extensive experiments show that MPA achieves state-of-the-art results on three popular datasets with an impressive average accuracy of 54.1% on DomainNet.

1 Introduction

Deep learning has achieved remarkable progress in various computer vision tasks [15, 11, 31, 24]. However, the success usually relies on supervised training using a massive amount of manually labeled data, which are often expensive and time-consuming to collect. Furthermore, current deep models are brittle to the presence of domain shift [27, 34, 47] in the forms of different image styles, varied lighting conditions, diverse viewpoints, etc. , between training and testing distributions.

Unsupervised domain adaptation (UDA) is a popular strategy that mitigates domain discrepancies by transferring knowledge learned from a well-labeled source domain to an unlabeled target domain [25, 4, 39, 42, 23]. While significant advances have been achieved, current approaches focus on the single source setting, where all the labeled training data share the same distribution. In practice, however, it is more common for the labeled data to be collected from multiple sources that are diverse in distribution. Naturally, one could still tackle this problem by straightforwardly combining all the data into one single source and applying off-the-shelf UDA methods. However, directly applying single-source UDA methods often results in limited performance, as domain shift also exists among different source domains.

The integration of multiple source domains for improved adaptation results on the unlabeled target domain is generally known as multi-source unsupervised domain adaptation. Inspired by the theoretical analysis of Ben-David et al. [2], learning domain-invariant feature representations has become a

CLIP's Image Encoder

target images

source j images

source i images

Contrastive

pairing

Prompt i

i-th

classifier

CLIP's Text Encoder

j-th

classifier

Prompt j

Feed

Forward

Add & Norm

N

×

Add & Norm

Multi-Head

Attention

(a) Method comparison



×10 6



×10 6



×10 6



×10 6



×10 6























MPA (Ours)

LST (Ours)

SImpAI

LtcMSDA

M 3 DSA

DCTN

(b) Performance comparison on DomainNet

prevailing paradigm for multi-source UDA. One typical approach is to jointly learn a common feature extractor together with domain-specific classifier heads. Various feature distance metrics [22, 33, 14] or domain adversarial training [35] can serve as a preliminary alignment between source and target domains, followed by different auxiliary losses carefully designed to further reduce the domain shift. Although these methods offer decent results in the single source setting, as the number of source domains increases, using only a single shared feature extractor to obtain domain-invariant features is inevitably difficult to optimize [44]. Such a problem is further amplified in the modern large model era if we wish to apply more advanced backbones for improved performance.

In this paper, we introduce prompt learning [16, 41], which has been designed to transfer knowledge learned from large pre-trained vision language models like CLIP [28], to multi-source UDA. In prompt learning, image representations are learned contrastively with a piece of language text termed 'prompt'. Consequently, prompts are tuned with the backbone network fixed for a more efficient adaptation to downstream tasks (See Figure 1 for a comparison). While recent studies [10, 1] suggest that learnable prompts can be used for UDA, they are restricted to the single source scenario and directly generalizing them to the multi-source setting produces limited results.

In light of this, we present a surprisingly simple framework, M ultiP rompt A lignment (MPA), for multi-source UDA. MPA is composed of two steps-one to learn an individual prompt by tuning a small set of parameters for each source and target domain pair, while the next to mine the relationships among the learned prompts by deriving a shared embedding space that is domain-invariant. More specifically, given a source domain and a target domain, we use CLIP as our backbone and learn one prompt tailored for such a pair. Then, inspired by the intrinsic dimensionality of data [40], we further reconstruct all the learned prompts using a simple auto-encoder network, aiming at removing redundant information potentially stemmed from the discrepancies among all the source domains. Finally, given the denoised prompts, an L 1 constraint is incorporated as our alignment strategy so that the prompts agree on the classification of target images.

We conduct extensive experiments on multiple benchmarks and the results clearly show that our approach outperforms state-of-the-art methods in the multi-source setting. In particular, on DomainNet [26], the most challenging dataset for multi-source UDA to date, MPA surpasses all state-of-the-art methods. Additionally, as the latent space is optimized with prompts from multiple sources, it encodes knowledge shared by different domains and could potentially generalize to multiple target domains by traversing the space. Consequently, we show how to tune the learned low-dimensional embedding for efficient deployment to a streamlined set of target domains, a strategy we name L atent S ubspace T uning (LST). In summary, our contributions are three-fold:

We introduce M ultiP rompt A lignment (MPA) for multi-source UDA. MPA takes advantage of prompt learning and thus is capable of achieving a balance between performance and efficiency compared with alternative methods.

MPA learns a latent space by maximizing the consensus of multiple learned prompts. Based on this, we introduce L atent S ubspace T uning (LST) that is able to continuously adapt to a streamlined set of target domains.

MPA achieves state-of-the-art results on several popular benchmarks while LST offers comparable results with tunable parameters that are one order of magnitude less.

2 Related Work

2.1 Multi-Source Unsupervised Domain Adaptation

First studied by Yang et al. [46], multi-source UDA has drawn increasing attention in the community. Throughout the years, various methods have been studied. For example, in MDAN [48] and DCTN [43], a discriminator applied with adversarial losses is trained so that the features from source and target domains are aligned. MFSAN [51] calculates and aligns the maximum mean discrepancy for each source and target pair. Similarly, M 3 SDA [26] aligns the moment distance for both target and source domains. All these methods require a shared feature extractor to obtain domain-invariant features. On the contrary, Rakshit et al. [29] adopt one domain-specific encoder for each source and target pair while Zhao et al. [49] pre-trains a classifier for each source domain and then adversarially maps the target images into each trained feature space. The better alignment of these methods is at the cost of a significantly increased number of parameters needed for training. More recently, Xu et al. [44] propose to combine joint alignment and separate alignment for a better adaptation, and Deng et al. [5] propose a method that detects and alleviates the negative impact from pseudo-labels for a better self-training scheme.

2.2 Prompt Learning

Traditionally, given a pre-trained language model, a common approach in deep learning is fine-tuning the whole model or its task-specific heads to adjust to downstream tasks. While effective, however, two main drawbacks exist. First of all, as the model size keeps increasing, pre-training and fine-tuning are becoming more and more expensive. Secondly, for each new task, fine-tuning needs to be repeatedly conducted. Recently, researchers have shown that pre-trained large-scale language models can handle a wide range of downstream tasks with only a few or even no samples by pre-pending instructions to the input text [20]. Such instruction texts are called prompts. Consequently, prompts can be tuned instead of the entire network for a more efficient adaptation to downstream tasks. Originally, prompts are essentially sequences of manually designed language tokens that are mapped to an embedding space. To date, extensive research has demonstrated that training soft prompts, i.e. , prompts with their own parameters learned by deep models, are more effective [18, 16]. The success of prompt learning in NLP has also garnered attention in the vision community which motivated the establishment of many related works. To name a few, Zhou et al. [50] are the first to apply soft prompt learning to the image recognition task. Ju et al. [13] explore prompt learning for efficient and lightweight video understanding. While promoting in these studies is limited to the input of text encoders, Jia et al. [12] prepend prompt tokens directly to the image patches.

3 Method

Our goal is to use multiple labeled source domains for improved performance on target domains. To this end, we leverage prompt learning, which is an effective strategy by learning a small set of parameters to adapt a pretrained model to different downstream tasks. In the following, we first review prompt learning in CLIP in Sec. 3.1 and then elaborate in Sec. 3.2 and Sec. 3.3 our proposed MPA and LST method respectively.

3.1 An Overview of Prompt Learning in CLIP

CLIP consists of an image encoder and a text encoder that are jointly trained with a contrastive loss on 400M image and text pairs. The image encoder f , which can either be a ResNet [11] or a Vision

Source

Prompt

Target

Prompt

Domain-invariant

feature

Domain-specific

feature

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

Text

Encoder

...

...

Image

Encoder

Source

Image

Target

Image

...

...

0.82

0.00

0.01

0.12

0.04

0.00

0.11

0.02

0.01

0.75

0.05

0.00

...

...

...

...

Transformer [7], maps raw images to an embedding space, and the text encoder g is a Transformer [36] that projects an input text sequence to the same embedding space. A prompt in CLIP usually exists in the form of 'a photo of [CLS]' where [CLS] is a class token that can be replaced by a certain class name. This sequence of tokens is first converted into a lower-cased byte pair encoding (BPE) representation, which is essentially a unique numeric ID [50]. Then the numeric IDs are embedded in a 512 dimension vector that is further passed to the Transformer text encoder. In our work, instead of using manually crafted prompts, we train soft prompts that are directly embedded by the text encoder. Given an image x and a text embedding w k for class k ∈ { 1 , 2 , ..., K } , where K is the total number of categories, CLIP aligns them in a contrastive manner so that

is maximized when the input image x indeed belongs to class k . Here < · , · > denotes the cosine similarity and T is a learnable temperature parameter.

3.2 Multi-Prompt Alignment

Let N denote the total number of domains, where the first N -1 domains are source domains and the N -th domain is the target domain. For multi-source UDA, our goal is to learn a domain-invariant latent space so that the domain shift among different source domains as well as the discrepancies between all the source and target domain pairs can be minimized. To achieve such a goal, we introduce the framework MPA. Specifically, we design prompts to contain domain-invariant and domain-specific feature following Ge et al. [10] and train such a prompt for each source and target domain pair. Then, an auto-encoder structure is introduced to our method to denoise the acquired prompts, followed by an L 1 constraint for further alignment.

Prompt Design. Following Ge et al. [10], our prompt for multi-source UDA includes a set of class-specific context tokens v k i , i ∈ { 1 , 2 , ..., M 1 } , k ∈ { 1 , 2 , ..., K } and another set of domainspecific tokens shared across all classes d d j , j ∈ { 1 , 2 , ..., M 2 } , d ∈ { s, t } . See Figure 2 for an overview. Here, M 1 and M 2 represent the number of tokens, K is the number of classes, s is short for source and t is short for target, resulting in a total of 2 K categories for training with contrastive loss. Therefore, the prompt for each source and target pair can be derived as:

These prompts serve as learnable parameters that bridge the domain gap between a source domain and a target domain through a contrastive loss, as will be introduced next.

Domain invariant latent subspace

MPA

Trainable Parameters

Art

Product

Real World

Alignment

Domain invariant latent subspace

MPA

Trainable Parameters

Art

Product

Real World

Alignment

(a) Alignment in MPA on the Office-Home dataset with Rw as target domain

Tuning the

latent subspace

LST

New Target Domain

Tuning the

latent subspace

LST

New Target Domain

(b) Illustration of LST

Prompt Learning for Multi-source UDA. To apply prompt learning to multi-source UDA, we first train individual prompts for each source and target pair using the image and text encoders of CLIP. Given an image x s sampled from the source domain D s whose label is y ∗ , we optimize the prompts so that the outputs from the image and text encoder are aligned. For an image x t from the target domain D t whose label is unknown, we first leverage the strong zero-shot ability of CLIP to generate a static pseudo-label ˆ y ∗ for image-text alignment. Pseudo-labels are only generated for images whose maximum probability is larger than a fixed threshold τ using Equation 1. While more sophisticated approaches like self-training could be leveraged to generate pseudo-labels [52, 19], we find that pseudo-labels from CLIP are simple and effective. Finally, prompts are trained with cross-entropy loss functions and Figure 2 gives an overview of the process. More formally, for a prompt P i , i ∈ { 1 , 2 , ..., N -1 } , the objective function for optimization follows:

Here, the probability P ( y = k | x d ; P i ) of an image sample belonging to the k -th class is derived from a contrastive loss:

where d ∈ { s, t } is a domain identifier indicating where the image comes from, T is a learnable temperature parameter, and f and g represents the image and text encoder in CLIP respectively, which are kept frozen during training. This specific design can push the prompts to learn disentangled representation of both class-invariant and class-specific semantic information to boost the performance of domain adaptation methods [3, 21].

Even though Eqn 3 allows us to obtain a prompt for each source and target domain pair, the noise level in the learned prompts differs due to variations in the amount of images and the domain gap between each source and target domain. One obvious consequence is that they might produce inconsistent results even for the same target domain image. Therefore, an L 1 constraint (Eqn 5) can be applied as a strategy for prompt alignment.

Better Alignment through Reconstructed Prompts. While directly aligning the learned prompts produce decent results for multi-source UDA, the prompts are high-dimensional and might contain redundant information. Motivated by the theory that high dimensional data usually lies in a lower

dimensional manifold [40], we build upon auto-encoders for better alignment. By using such an architecture, we hope to learn a domain-invariant 'latent subspace' of denoised prompts so that redundant information can be removed through the reconstruction of the learned prompts. More formally, we use an auto-encoder consisting of a projection function Proj ( · ) and a back-projection function Proj b ( · ). The learned prompts P i are first projected into a latent subspace of a lower dimension d I by Proj ( · ) , followed by Proj b ( · ) projecting the vectors back into soft prompts ˆ P i . The Proj ( · ) function is implemented by a one-layer feed-forward network while Proj b ( · ) is a two-layer nonlinear perceptron:

where v I = Proj ( P i ) and we optimize a reconstruction loss:

We then perform prompt alignment, i.e. the L 1 constraint, on the reconstructed prompts. Therefore, the overall objective function can be written as:

where L CLS is the cross entropy loss calculated using the static pseudo-labels that helps with the reconstruction process. Here α is a hyper-parameter controlling the weight of the L 1 loss. The whole alignment procedure is depicted in Figure 3a. Finally, for predicting the labels of target samples, we compute the average of the output logits using all ˆ P i s.

3.3 Latent Subspace Tuning

In real-life applications, it is more practical when adaptation to a streamlined set of targets is needed. While we can repeatedly apply MPA to each target domain, it is computationally inefficient especially when equipped with a large-scale backbone model. To mitigate this issue, we introduce an LST strategy that explores the latent space derived by autoencoders for fast adaptation. The key idea is that since the latent space of the learned auto-encoder from MPA is optimized with prompts from multiple source domains, it alone is capable of encoding domain-invariant knowledge. Therefore, we can adopt MPA on the first target domain, and traverse the subspace learned by MPA to generalize to the following ones.

More formally, given a streamlined set of target domains D T 1 , D T 2 , ..., D T L , to continuously adapt to each one of them in a computationally efficient manner, we first conduct MPA on domain D T 1 . After successfully applying MPA, a low-dimensional embedding space that captures the relationships among different domains is derived and can be leveraged for quick adaptation to domains D T 2 , ..., D T L . Specifically, for each following target domain D T i , a domain-invariant feature vector v T i tune ∈ R N × M 1 × d I together with a domain-specific feature vector d T i tune ∈ R 1 × M 2 × d I is randomly initialized and passed to the learned back projection function Proj b ( · ) from MPA. As a result, an entirely new prompt ˜ P T i = Concat [ Proj b ( v T i tune ) , Proj b ( d T i tune )] can be constructed. Again with the help of pseudo-labels, we can tune this new prompt by minimizing the following objective function:

which is essentially L CLS in Eqn 9 and inference on D T i can be conducted by only using ˜ P T i . An illustration of LST is shown in Fig 3b. Since the prompt for CLIP is of size K × token length × embedding size, adopting the LST strategy can both reduce the need of training individual prompts and decrease the number of tunable parameters by a factor of at least ( N -1) × embedding size d I times compared with MPA. This is in fact non-trivial when confronting large-scale datasets such as DomainNet, where N = 6 , d I = 250 , and embedding size = 512 .

4 Experiments

4.1 Experimental Setup

Datasets and Metrics. Experiments are conducted on three popular benchmark datasets of UDA to evaluate the effectiveness of MPA, namely ImageCLEF, Office-Home, and DomainNet. ImageCLEF is a small-scaled dataset consisting of 1,800 images from 12 object categories in 3 different domains: ImageNet ILSVRC 2012 (I), Pascal VOC 2012 (P), and Caltech-256 (C). Office-Home is a mediumscaled dataset consisting of about 15,500 images from 65 categories in 4 different domains: Art, Clipart, Product, and Real World. DomainNet is the largest dataset to date, consisting of about 0.6 million images from 345 categories in 6 different domains: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch.

We use top-1 accuracy as our evaluation metric and report results of the following settings: (1) CLIP: zero-shot CLIP on the target domain, which can be regarded as a baseline of our method. (2) Source Combined: all source domains are combined into one single domain and applied with popular single-source UDA methods. Specifically in this setting, we adopt the prompting method from Zhou et al. [50] to serve as another baseline named as 'Simple Prompting'. (3) Multi-Source: results reported from other multi-source UDA methods.

Implementation Details For fair comparisons, we adopt a ResNet50 as our backbone on ImageCLEF and Office-Home and a ResNet101 on DomainNet. The weights are from CLIP and frozen throughout the experiments. Prompts and auto-encoder of MPA are trained using the mini-batch SGD optimizer with a learning rate of 0.003 and 0.005 while the learned subspace is tuned with a 0.0005 learning rate in LST. We use a batch size of 32 and adopt a cosine learning rate scheduler. For hyper-parameters, token lengths M 1 and M 2 are both set to 16. Pseudo-label threshold τ is set to 0.4 for producing reliable labels. α in Equation 9 is set to 500. The weight matrix W 2 of the back projection function in Equation 7 has a size of R 384 × d I , where d I is 100 for ImageCLEF, 150 for OfficeHome and 250 for DomainNet.

4.2 Comparison to State-of-the-Art

Multi-Prompt Alignment The results on ImageCLEF and Office-Home are shown in Table 1. For ImageCLEF, it is obvious that MPA outperforms other methods with an average accuracy of 91.7%, where there is at least a 3% increase when adapting to domain C and I. For Office-Home, MPA achieves the best results except when adapting to the domain Clipart. Nevertheless, we achieve an accuracy of 75.4% on average, which is 1.3% higher than the second best method MFSAN. It is worth noting that compared to state-of-the-art method MFSAN on both datasets, MPA only trains 0.78M and 2.36M parameters, while MFSAN has a total of 51.75M and 51.80M parameters needed for optimizing (66.3 and 21.9 times larger than ours).

Table 2 shows that for DomainNet, MPA exceeds other multi-source UDA methods by more than 5%. To the best of our knowledge, this is the highest reported accuracy on this dataset so far with less than one third parameters optimized compared with most state-of-the-arts methods. Regarding individual adaptations, MPA achieves best results on most of the adaptation tasks but performs mediocre on the Quickdraw domain. Interestingly, the result is even a little worse than CLIP. We hypothesize that this is because of the large domain gap between Quickdraw and other domains.

Latent Subspace Tuning Results from Table 1 and Table 2 are already strong indications of the success of MPA under conventional multi-source UDA settings. Now, we would like to investigate a more practical scenario, where adaptation to a streamlined set of target domains is required, by using the LST strategy as mentioned in Section 3.3.

Results of LST on the DomainNet dataset are shown in Table 3, where compared with re-applying MPA, performance only dropped by 0.3%. Nevertheless, LST achieves higher accuracy compared to most baseline methods of Table 2, including CLIP with an increase of 3.2%. Significantly, the adaptation process with LST requires only 11 hours of GPU time on a NVIDIA RTX 3090, whereas adapting using MPA takes 54 hours-a speed increase of approximately 5 times.

4.3 Ablation Study

CLIP backbone While we are the first to apply prompt learning to multi-source UDA with the help of a pre-trained CLIP model, most existing approaches that we compare with utilize ImageNet pre-trained backbones, which are generally considered to be inferior compared to CLIP. Therefore, we would like to examine whether MPA's performance gain is simply from CLIP. For such purpose, two additional sets of experiments are conducted: (1) we apply a simple prompt learning method [50] to the source combined scenario, dubbed as the Simple Prompting baseline. As is demonstrated in Table 1 and 2, while it is 1.33% on average better than zero shot CLIP, MPA still outperforms it with a significant margin; (2) we directly swap MFSAN's ResNet50 backbone pre-trained on ImageNet to CLIP's image encoder. Surprisingly, when tested on ImageCLEF, Table 4 shows that the performance

even drops by a small margin of 0.3%. From these results, we infer that while CLIP indeed provides a strong backbone, it is not necessarily universally superior, as also observed on the basis of studies Devillers et al. [6], Yang et al. [45]. Moreover, despite CLIP's strong zero-shot performance as stated in Radford et al. [28], our method consistently outperforms CLIP in the domain adaptation setting, especially in cases on the ImageCLEF and OfficeHome datasets where CLIP's zero-shot performance is limited. On these two datasets, MPA consistently delivers excellent results that surpass CLIP by 6.2% and 4.0%, respectively. Therefore, we believe while CLIP's strong backbone indeed contributes to MPA's good results, it alone is insufficient for such superior performance and another important piece is MPA's strong domain adaptation ability.

Prompt engineering Apparently, the performance of MPA depends on the quality of pseudo-labels generated by CLIP. As such, it is worth investigating whether MPA has been pushed to its maximum extent by using more sophisticated prompts than 'A photo of [CLS]'. Specifically, we tested two additional sets of prompts: (1) 'a [Dom] of [CLS]' (e.g., a painting of dog) and (2) 'a photo of [CLS] in domain [Dom]' (e.g., a photo of dog in domain painting) with the intention of incorporating domain information, and as is presented in Tab 5, while each of the three prompts shows certain advantage on some specific domains, in general, the naive 'A photo of [CLS]' still performs the best.

Hyper-parameter selection Experimental results for ablation studies on hyper-parameter selections are reported in Table 6. For both pseudo-label threshold τ and prompt token length M 1 , M 2 , three different choices τ ∈ { 0 . 3 , 0 . 6 , 0 . 8 } and M 1 , M 2 ∈ { 8 , 12 , 20 } are examined. For simplicity, we are setting M 1 and M 2 to be equal. As τ increases, while the quality of the pseudo-labels gets higher, fewer images will be fed into the model, and Table 6a suggests that doing so hurts the overall performance. On the contrary, shown in Table 6b, the general trend for prompt token length is that the longer the prompt, the better the performance. As a result, we choose τ = 0 . 4 and M 1 = M 2 = 16 to balance the trade-off between performance and efficiency. For α in Equation 9, we chose it to be 500 to balance all losses (in this case L 1 ) to be of the same order of magnitude. Our experimental results from Table 6c also support such motivation.

Alignment strategy, prompt denoising and prompt design To validate the effectiveness of our alignment strategy, we evaluate our method without the L 1 loss, and table 7 shows that this will result in a performance degradation of about 0.7%. In particular, the L 1 constraint exhibits a greater impact on the more difficult Clipart domain, where the accuracy dropped by a large margin of 1.5% without it, demonstrating the effectiveness of constraining consistent predictions on different prompts. Subsequently, we tested the effectiveness of denoising the prompts through auto-encoder reconstruction. Again, results show that the incorporation of an auto-encoder structure with the L AE loss is beneficial to the overall performance. Finally, we examined the necessity of using class and domain-specific tokens for learning prompts and results from Table 7 support such design. Specifically, ✓ implies that both the class-specific and domain-specific prompts are used while ✕ implies that only the class-specific prompts are used.

(a) Ablation on pseudo-label threshold τ .

(c) Ablation on

α

.

5 Conclusion

In this paper, we introduced prompt learning to multi-source UDA and proposed a simple MPA scheme to align the source and target domains. MPA is composed of two steps. The first step is to train individual prompts for each source and target domain pair and the next one to align them after reconstruction using an auto-encoder structure. Extensive experiments showed that MPA achieved better results than state-of-the-art methods on various multi-source UDA tasks with substantially fewer parameters tuned. Moreover, an LST strategy was introduced for efficient adaptation to a streamlined set of target domains. As the model contains useful clues of multiple domains, one potential limitation is that it faces more risks in terms of information leakage where adversaries might produce attacks to steal information from the model.

Acknowledgement This project was supported by NSFC under Grant No. 62102092.

References

Eyal Ben-David, Nadav Oved, and Roi Reichart. Pada: Example-based prompt learning for on-the-fly adaptation to unseen domains. TACL , 2022.

Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NeurIPS , 2006.

Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In NeurIPS , 2016.

Gabriela Csurka. Domain adaptation for visual applications: A comprehensive survey. arXiv preprint arXiv:1702.05374 , 2017.

Zhongying Deng, Da Li, Yi-Zhe Song, and Tao Xiang. Robust target training for multi-source domain adaptation. arXiv preprint arXiv:2210.01676 , 2022.

Benjamin Devillers, Bhavin Choksi, Romain Bielawski, and Rufin VanRullen. Does language help generalization in vision models? arXiv preprint arXiv:2104.08313 , 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Yangye Fu, Ming Zhang, Xing Xu, Zuo Cao, Chao Ma, Yanli Ji, Kai Zuo, and Huimin Lu. Partial feature selection and alignment for multi-source domain adaptation. In CVPR , 2021.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR , 2016.

Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. arXiv preprint arXiv:2202.06687 , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.12119 , 2022.

Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. arXiv preprint arXiv:2112.04478 , 2021.

Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In CVPR , 2019.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS , 2012.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 , 2021.

Ruihuang Li, Xu Jia, Jianzhong He, Shuaijun Chen, and Qinghua Hu. T-svdnet: Exploring high-order prototypical correlations for multi-source domain adaptation. In ICCV , 2021.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 , 2021.

Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. In NeurIPS , 2021.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 , 2021.

Yen-Cheng Liu, Yu-Ying Yeh, Tzu-Chien Fu, Sheng-De Wang, Wei-Chen Chiu, and YuChiang Frank Wang. Detach and adapt: Learning cross-domain disentangled deep representation. In CVPR , 2018.

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICML , 2015.

Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In CVPR , 2022.

Lingchen Meng, Xiyang Dai, Yinpeng Chen, Pengchuan Zhang, Dongdong Chen, Mengchen Liu, Jianfeng Wang, Zuxuan Wu, Lu Yuan, and Yu-Gang Jiang. Detection hub: Unifying object detection datasets via query adaptation on language embedding. In CVPR , 2023.

Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. TKDE , 2010.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV , 2019.

Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning . 2008.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Sayan Rakshit, Biplab Banerjee, Gemma Roig, and Subhasis Chaudhuri. Unsupervised multisource domain adaptation driven by deep adversarial ensemble learning. In GCPR , 2019.

Chuan-Xian Ren, Yong-Hui Liu, Xi-Wen Zhang, and Ke-Kun Huang. Multi-source unsupervised domain adaptation via pseudo target domain. TIP , 2022.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS , 2015.

Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR , 2018.

Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV , 2016.

Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR , 2011.

Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR , 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Naveen Venkat, Jogendra Nath Kundu, Durgesh Singh, Ambareesh Revanur, et al. Your classifier can secretly suffice multi-source domain adaptation. In NeurIPS , 2020.

Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang. Learning to combine: Knowledge aggregation for multi-source domain adaptation. In ECCV , 2020.

Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing , 2018.

Wei Wang, Yan Huang, Yizhou Wang, and Liang Wang. Generalized autoencoder: A neural network framework for dimensionality reduction. In CVPR workshops , 2014.

Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. In NeurIPS , 2021.

Zejia Weng, Xitong Yang, Ang Li, Zuxuan Wu, and Yu-Gang Jiang. Open-vclip: Transforming clip to an open-vocabulary video model via interpolated weight optimization. 2023.

Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In CVPR , 2018.

Yuanyuan Xu, Meina Kan, Shiguang Shan, and Xilin Chen. Mutual learning of joint and separate domain alignments for multi-source domain adaptation. In WACV , 2022.

Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In CVPR , 2022.

Jun Yang, Rong Yan, and Alexander G Hauptmann. Cross-domain video concept detection using adaptive svms. In ACM MM , 2007.

Kun Zhang, Bernhard Sch¨ olkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In ICML , 2013.

Han Zhao, Shanghang Zhang, Guanhang Wu, Jos´ e MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. In NeurIPS , 2018.

Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-source distilling domain adaptation. In AAAI , 2020.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV , 2022.

Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang. Aligning domain-specific distribution and classifier for cross-domain classification from multiple sources. In AAAI , 2019.

Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In ECCV , 2018.

Peer reviews:
comments: SUMMARY:
This paper proposes an interesting pipeline for large pre-trained model-based UDA, and realizes a latent subspace tuning for continuous adaption.

STRENGTHS:
1. The motivation behind this article is very meaningful, and the proposed method supports the motivation well.
2. In terms of methodology, the design of this paper is clear and innovative for the combination of existing technologies.
3. This easy but effective baseline and latent subspace tuning ability may provide new paradigm for future UDA.
4. Good performances.

WEAKNESSES:
Using CLIP as the backbone may limit the ability of MPA on only classification tasks, how to extend MPA to other tasks, like segmentation, referring, etc? And how to utilize other large pre-trained models like GPT-3?

QUESTIONS:
Please see the weakness part.

LIMITATIONS:
The authors should provide the limitation of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces prompt learning to multi-source unsupervised domain adaptation (UDA). Firstly, individual prompts for each source and target pair are learned using a contrastive loss. Then, MPA aligns the learned prompt by an autoencoder-based step with an L_1 constraint to generate consistent results for the same target domain image. In addition, the LST strategy delivers the first target domain adaptation to the subsequent target domains efficiently. Experiments on ImageCLEF, Office-Home, and DomainNet validate the effectiveness.\n\nSTRENGTHS:\n+ This paper's application of prompt learning to multi-source UDA problems is groundbreaking.\n+ This paper is well-written and easy to understand.\n\nWEAKNESSES:\n+  Regarding the prompt design part in sec3.2, can it only be realized by following Ge[10]? There is a lack of novelty and contribution due to the lack of original works in the whole prompt design.\n+  Regarding the problem of reducing the dimensionality of the high-dimensional learned prompt, is this step introduced because the prompt design is not optimal? From the perspective of Tab. 6, the growth brought by AE is very limited, so are other methods of dimensionality reduction effective, or even the step of AE can be removed by considering redundant information in prompt design?\n3 The LST in 3 sec 3.3 is a further extended use of MPA, which is somewhat insufficient as an independent innovation point.\n4 This paper needs to weaken the sense of the existence of CLIP and the dependence on the previous prompt design method, otherwise very much like a simple application essay.\n\nQUESTIONS:\nIn addition to the questions in the above weaknesses that need to be answered, there are a few things that need to be clarified.\n\n+  The generalization of CLIP is a huge advantage. Although the author discussed the impact of CLIP in 4.3, I hope that the author will discuss the design experiment of the prompt. In addition to using the original \"a photo of [CLS],\" the number of channels of the trainable prompt params can also be discussed.\n\n+ Please state the problems and experimental results faced when the approach of Ge [10] is transferred to the tasks addressed in this paper. And for the problem and results, describe the differences of this paper.\n\n+ For now, it is unknown why prompt, as a trained parameter, has class-specific and domain-specific attributes.\n\nLIMITATIONS:\nNot applicable", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper deals with the multi-source domain adaptation problem. It proposes to tune the designed domain-invariant prompts and domain-specific prompts to enable the domain adaptation ability. Generally, the training consists of two objectives, i.e., the individual prompt learning objective and the de-noising objective via prompt auto-encoders. To enable a test-time domain adaptation ability and reduce the amount of learnable parameters, the authors propose LST. Experiments on various multi-source benchmarks verify the effectiveness of proposed method.

STRENGTHS:
- The idea is simple and generally reasonable.

- The experiment results are good.

WEAKNESSES:
- In LST, I don't think it is reasonable to use randomly initialized representations as the input of the back-projector to perform prompt reconstruction, as the latent representations don't subject to the same distribution. And there is no empirical evidence to show such a way really works.

- Missing comparisons to strong baselines. E.g., on DomainNet, [1] achieves a multi-source domain adaptation performance at 53.2 with a ResNet-101 backbone. The authors should compare to the strong baselines.

  [1] Contrastive adaptation network for single- and multi-source domain adaptation, TPAMI 2020.

QUESTIONS:
See the weakness part.

LIMITATIONS:
I don't see any serious potential negative societal impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an extension of [10] (Domain Adaptation via Prompt Learning Ge et al., 2022) to the multi-source UDA set-up.\n(i) Distinct soft prompts are learnt via contrastive loss for each source-target pair;  each source-target prompt is composed of class-wise source- and target-prompts. In the target domain, learning is achieved by utilizing pseudo-labels from the pretrained CLIP model, which possesses strong zero-shot ability.\nTo encourage consistent target outputs across all learned prompts, a consistency L1 loss is applied to the soft outputs of all target samples.\n(ii) Prompt reconstruction: to eliminate redundant information that may impede performance, an autoencoder (AE) is learned to \"denoise\" the acquired prompts.\n(iii) Latent Subspace Tuning (LST): efficient adaptation to a new target domain, after learning from the first target domain, is accomplished by optimizing on the latent space of the learned AE using pseudo-labels of the new target.\n\nThe proposed framework demonstrates superior results compared to previous State-of-the-Art (SOTA) methods in multi-source UDA.\n\nSTRENGTHS:\nThe proposed method leverages the powerful zero-shot capability of the pretrained CLIP model for the multi-source domain adaptation (DA) task. Empirical results demonstrate its effectiveness across various benchmarking setups. The presentation is good, enabling easy comprehension of the method.\n\nWEAKNESSES:\nThe main concern of this work is the lack of technical novelty and a more rigorous evaluation. The proposed framework is a straightforward extension of the work by Get et al. (2022) [10] to the multi-source setting. In terms of method evaluation, it is expected to compare against stronger CLIP-based baselines, both in the multi-source UDA experiments (Tables 1 and 2) and in the LST experiment (Table 3). Further details are provided in the following section.\n\nQUESTIONS:\nGiven that CLIP alone produces results close to the state-of-the-art (SOTA) as shown in Table 1, and even outperforms the SOTA as shown in Table 2, my main concern is whether the authors have made adequate efforts to create decent baselines using CLIP-pretrained models. I appreciate their effort in Section 4.3 when they ``swap MFSAN’s ResNet50 backbone pre-trained on ImageNet to CLIP’s image encoder''. However, I would love to know more details to know as if careful considerations were taken into account for MFSAN + CLIP or simply the authors swap the backbone's weight. For example, did the authors preserve the text classifier and the contrastive los of CLIP ? As the proposed framework is based on self-training with pseudo-labels, I wonder if how the CLIP + self-training and Single Prompt + self-training baselines perform.\n\nFurthermore, I am curious to know if CLIP with fixed prompts has been pushed to its maximum extent. In other words, how far can this naïve baseline reach with better prompt engineering instead of using the simple prompt \"a photo of [CLS]\"? One can imagine incorporating domain-specific information into the prompt or defining a set of templates rather than relying on just one template.\n\nIn the LST experiment, how good is MPA (on the first domain) + self-training? For efficiency, one can try test time prompt tuning (TPT).\n\nLIMITATIONS:\nThis work lacks significant technical novelty, and the experiment could be improved by using stronger baselines. Based on the current state of the submission, my recommendation leans towards the negative side.\n\n---- After rebuttal ----\nThe rebuttal is convincing and has helped clarify most of my technical concerns. I believe that this work is indeed interesting and help advocate the usage of prompts in domain adaptation. While the novelty limitation still pertains, the original set of experiments, along with the new ones provided during the rebuttal, is sufficient. I believe the paper after revision would certainly pass the NeurIPS' bar.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}