Multi-Prompt Alignment for Multi-source Unsupervised Domain Adaptation
Authors: Haoran Chen
Zuxuan Wu
Yu-Gang Jiang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Most existing methods for multi-source unsupervised domain adaptation (UDA) rely on a common feature encoder to extract domain-invariant features. However, learning such an encoder involves updating the parameters of the entire network, which makes the optimization computationally expensive, particularly when coupled with min-max objectives. Inspired by recent advances in prompt learning that adapts high-capacity deep models for downstream tasks in a computationally economic way, we introduce Multi-Prompt Alignment (MPA), a simple yet efficient two-stage framework for multi-source UDA. Given a source and target domain pair, MPA first trains an individual prompt to minimize the domain gap through a contrastive loss, while tuning only a small set of parameters. Then, MPA derives a low-dimensional latent space through an auto-encoding process that maximizes the agreement of multiple learned prompts. The resulting embedding further facilitates generalization to unseen domains. Extensive experiments show that our method achieves state-of-the-art results on popular benchmark datasets while requiring substantially fewer tunable parameters. To the best of our knowledge, we are the first to apply prompt learning to the multi-source UDA problem and our method achieves the highest reported average accuracy of 54.1% on DomainNet, the most challenging UDA dataset to date, with only 15.9M parameters trained. More importantly, we demonstrate that the learned embedding space can be easily adapted to novel unseen domains.

Under review as a conference paper at ICLR 2023
MULTI-PROMPT ALIGNMENT FOR MULTI-SOURCE UNSUPERVISED DOMAIN ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Most existing methods for multi-source unsupervised domain adaptation (UDA) rely on a common feature encoder to extract domain-invariant features. However, learning such an encoder involves updating the parameters of the entire network, which makes the optimization computationally expensive, particularly when cou- pled with min-max objectives. Inspired by recent advances in prompt learning that adapts high-capacity deep models for downstream tasks in a computation- ally economic way, introduce Multi-Prompt Alignment (MPA), simple yet efficient two-stage framework for multi-source UDA. Given a source and target domain pair, MPA first trains an individual prompt to minimize the domain gap through a contrastive loss, while tuning only small set of parameters. Then, MPA derives a low-dimensional latent space through an auto-encoding process that maximizes the agreement of multiple learned prompts. The resulting embed- ding further facilitates generalization to unseen domains, making MPA naturally suitable for test time adaptation. Extensive experiments show that our method achieves state-of-the-art results on popular benchmark datasets while requiring substantially fewer tunable parameters. To the best of our knowledge, we are the first to apply prompt learning to the multi-source UDA problem and our method achieves the highest reported average accuracy of 54. % on DomainNet, the most challenging UDA dataset to date, with only 15.9M parameters trained. More im- portantly, demonstrate that the learned embedding space can be easily adapted to novel unseen domains with even fewer tuned parameters.
INTRODUCTION
Deep learning has achieved remarkable progress in various computer vision tasks such as image classification (Krizhevsky et al., 2012; He et al., 2016), object detection (Ren et al., 2015; Redmon et al., 2016; Liu et al., 2016) and image segmentation (Long et al., 2015a; Chen et al., 2017). How- ever, these success relies high capacity models trained in supervised manner using massive amount of manually labeled data, which are oftentimes expensive and time-consuming to collect. Furthermore, current deep models are brittle to the presence of domain shift (qainonero-Cnnell, et al., 2008; Torralba & Efros, 2011; Zhang et al., 2013) in the forms of different image styles, varied lighting conditions, diverse viewpoints, etc., between training and testing distributions.
Unsupervised domain adaptation (UDA) is a popular strategy that mitigates domain discrepancies through transferring knowledge learned from a well-labeled source domain to an unlabeled target domain (Pan & Yang, 2010; Csurka, 2017; Wang & Deng, 2018). While significant advances have been achieved, current approaches focus on the single source setting, where all the labeled training data share the same distribution. In practice, however, it is more common for the labeled data to be collected from multiple sources that are diverse in distribution. Naturally, one could still tackle this problem by straightforwardly combining all the data into single source and apply off-the-sher UDA methods. However, directly applying single source UDA methods often results in a limited performance, domain shift also exists among different source domains.
The integration of multiple source domains for improved adaptation results on the unlabeled tar- get domain is generally known multi-source unsupervised domain adaptation. Inspired by the theoretical analysis of Ben-David et al. (2006), learning domain-invarian feature representations has become prevailing paradigm for multi-source UDA. One typical approach is jointly learn

---

Under review as a conference paper at ICLR 2023
Conventional
th classiBer
j-lb clossifer
prompt
prompi
$5.9
$2.9
50.9
T Common Fenture Enceder
Encoder
47.5
42.5
MPA (Ours)
LST (Duns)
target Image
413
SlmaAl
LOCATEDA
30,5
MPCSA
DCTN
SouFcE IIADE
CLIP's Imagge Uncuder
Parumetes (logan)
(a) Method comparison
(b) Tunable parameters vs accuracy on DomainNet
Figure 1: (a) Most conventional multi-source UDA methods use common feature extractor with domain-specific classifier heads, while we introduce prompt learning to multi-source UDA and omit the repeated need for classifier heads. (b) MPA outperforms all other multi-source UDA methods by large margin on the DomainNet dataset with roughly one third tunable parameters. We also introduce Latent Space Tuning (LST) strategy that further reduces the trainable parameters from 15.9M (MPA) to 1.47M while still capable of achieving high accuracy. See texts for more details.
common feature extractor together with domain-specific feature extractors and classifier heads. Various feature distance metrics (Long et al., 201 5b; Sun & Saenko, 2016; Kang et al., 2019) or domain adversarial training (Tzeng et al., 2017) can leveraged that serves as preliminary align- ment between source and target domains, followed by different auxiliary losses carefully designed to further reduce the domain shift. While these methods offer decent results, they require optimizing the entire set of parameters in the network and hence poses significant challenge for optimization even without the widely used min-max objective (Zhao et al., 2018; Li et al., 201 8; Hoffman et al., advanced backbones such as
2018). Such problem is further amplified if wish to apply Vision Transformer (Dosovitskiy et al., 2021) for improved performance.
In this paper, introduce simple yet efficient approach for multi-source UDA without the need of retraining the entire network and the use of complicated min-max loss functions (See Figure 1 for a comparison). In particular, we build upon prompt learning (Lester al., 2021; Wei et al., 2021) that has been designed to transfer knowledge learned from large pre-trained vision language models like CLIP (Radford et al., 2021). In prompt learning, image representations are learned contrastively with piece of language text termed as "prompt" that describes the class of the image. While recent studies (Ge et al., 2022; Ben-David et al., 2022) suggest that learnable prompts can be used for UDA, they are restricted to the single source scenario and directly generalizing them to the multi- source setting produces limited results. Furthermore, while prompt learning provides an efficient alternative, argue that the number of parameters needed be tuned could be further reduced.
In light of this, we present a surprisingly easy framework, Multi-Prompt Alignment (MPA), for multi-source UDA. MPA is composed of two stages, one to learn an individual prompt by tuning a small set of parameters for each source and target domain pair and one to mine the relationships among learned prompts through deriving a shared embedding space. The resulting embedding is expected to be domain-invariant and can generalize to unseen domains. More specifically, for the CLIP our backbone and learn one
first stage, given a source domain and target domain, prompt tailored for such a pair. We then align all the learned prompts in a latent space of lower dimension dy. This is accomplished by simple auto-encoder network with reconstruction loss. Additionally, we incorporate an C₁ loss so that the reconstructed prompts agrec on the classification of target images. This is beneficial for prompts handle situations in which the target data lies near the decision boundary. We conduct extensive experiments on multiple benchmarks and the results clearly show that our method outperforms state-of-the-art methods in the multi-source setting. In particular, on DomainNet (Peng al., 2019), the most challenging dataset for multi-source UDA so far, MPA surpasses all state-of-the ofwtoff---atttt arts methods. More importantly, as the latent space is optimized with prompts from multiple source domains, it encodes knowledge shared by different domains and could potentially generalize to unseen domains by traversing the space. Consequently, we show how surprisingly easy it is to tune the learned low-dimensional embedding for deployment unseen
2

---

Under review as a conference paper at ICLR 2023
domains. Since such tuning involves source domain data, this can be naturally extended to test time adaptation problems. In summary, our contributions are three-fold:
We introduce Multi-Prompt Aligmment (MPA) for multi-source UDA. MPA takes advantage of prompt learning, thus greatly reducing the number of parameters needed for training compared with alternative methods.
MPA learns a latent embedding space by maximizing the consensus of multiple learned prompts. Consequently we provide methodology that is able to solve test time adaptation problems by facilitating the adaptation of the resulting low-dimensional embedding to novel unseen domains.
MPA achieves state-of- of- results on several popular benchmarks. Specifically, on the large- scale DomainNet dataset, MPA achieves the best reported average accuracy. Moreover, tuning the learned latent subspace offers comparable results with even fewer parameters when generalizing unseen domains.
2
RELATED WORK
2.1 MOLTI-SOURCE UNSUPERVISED DOMAIN ADAPTATION
First studied by Yang et al. (2007), multi-source UDA has drawn increasing attention in the commu- nity. Throughout the years, various methods have been studied. For example, in MDAN (Zhao et al., 2018) and DCTN (Xu et al., 2018), discriminator applied with adversarial losses is trained so that the features from source and target domains are aligned. MFSAN (Zhu et al., 2019) calculates and aligns the maximum mean discrepancy for each source and target pair. Similarly, SDA (Peng et al., 2019) aligns the moment distance for both target and source domains. All these methods require a shared feature extractor to obtain domain-invariant features, which inevitably difficult to optimize and at the risk of losing semantic information as the number of domains increases. On the contrary, Rakshit et al. (2019) adopt one domain-specific encoder for each source and target pair while Zhao et al. (2020) pretrain a classifier for each source domain and then adversarially map the target images into each trained feature space. The better aligmment of these methods is at the cost of significantly increased number of parameters needed for training. To overcome such trade-off be- tween performance and efficiency, we introduce prompt learning to multi-source UDA. Since each prompt contains substantially fewer parameters compared to a feature extractor, learning distinct prompts for each source and target pair is affordable.
2.2 PROMPT LEARNING
Traditionally, given a pre-trained language model, a common approach in deep learning is fine- tuning the whole model or its task-specific heads to adjust to downstream tasks. While effective, however, two main drawbacks exist. First of all, as the model size keeps increasing, pre-training and fine-tuning is becoming more and more expensive. Secondly, for each diverse task, fine-tuning needs to be repeatedly conducted. Recently, researchers have shown that learned large-scale language models can handle wide range of downstream tasks with only few or even no samples by pre- pending instructions to the input text (Liu et al., 2021b). Such instruction texts are called prompts. Consequently, prompts can be tuned instead of the entire network for a more efficient adaptation to downstream tasks. Originally, prompts are essentially sequences of manually designed language tokens that are mapped to an embedding space. To date, extensive research has demonstrated that training soft prompts, i.e., prompts with their own parameters learned by deep models, is more effective (Li & Liang, 2021; Lester et al., 2021). The success of prompt learning in NLP has also garnered attention in the vision community that motivated the establishment of many related work. To name a few, Zhou et al. (2022) are the first to apply soft prompt learning to the image recognition task. Ju et al. (2021) explore prompt learning for efficient and lightweight video understanding. While prompting in these studies are limited to the input of text encoders, Jia et al. (2022) prepend token embeddings directly the image patches.
3

---

Under review as a conference paper at ICLR 2023
3 METHOD
Our goal is to use multiple labeled source domains for improved performance on a target domain while only tuning a small set of parameters. To this end, we leverage prompt learning, which is an effective strategy by learning a small set of parameters to adapt pretrained model to different down- stream tasks. In the following, we first review prompt learning CLIP in Sec. 3.1 and then elaborate in Sec. 3.2 our proposed MPA method, which is two-stage framework that seeks agreement from multiple learned prompts and derives a joint latent space with Gomain-invarian knowledge.
3.1 AN OVERVIEW OF PROMPT LEARNING IN CLIP
CLIP consists of an image encoder and text encoder that are jointly trained with a contrastive loss on 400M image and text pairs. The image encoder J. which can either be a ResNet (He et al., 2016) or a Vision Transformer (Dosovitskiy et al., 2021), maps raw images to an embedding space, and the text encoder g is Transformer (Vaswani al., 2017) that maps input text sequence to the same embedding space. A prompt in CLIP usually exists in the form of "a photo of [CLS]" where [CLS] is a class token that can be replaced by certain class name. This sequence of tokens is first converted into a lower cased byte pair encoding (BPE) representation, which is essentially a unique numeric ID (Zhou et al., 2022). Then the numeric IDs embedded to 512 dimension vector that is further passed to the Transformer text encoder. In our work, instead of using manually crafted prompts, we train soft prompts that directly embedded by the text encoder. Given an image and a text embedding wy for class k E {1, 2.., K}, where K is the total number of categories, CLIP aligns them in a contrastive manner so that:
exp(< Wk. f(x) (T)
ply klz) Σ exp(<< wi f(x) > /T)
is maximized when the input image æ indeed belongs to class k. Here <,, denotes the cosine similarity and T' is a learnable temperature parameter.
3.2 MULTI-PROMPT ALIGNMENT
Let N denote the total number of domains, where the first N domains are source domains and the N-th domain is the target domain. For all N - 1 source domains, both images and their labels are provided, while for the target domain, we only assume access to their images, as in the standard UDA setting. For multi-source UDA, we wish to learn a domain-invariant latent space so that the domain shift among different source domains as well as the discrepancies between all the source and target domain pairs can be minimized. Unlike current multi multi-source UDA methods that require a significant amount of parameters for adaption, we leverage prompt learning for alignment among different domains. In the following, learning individual prompts and multi-prompt alignment. Finally we describe how to learn a joint embedding space that has the potential of generalizing to unseen domains.
begin by introducing the prompt design, followed by
Prompt Design. Following (Ge et al., 2022), prompt for multi- UDA includes set of class-specific context vectors of, i E {1, M1). k E f1,2...., K} and another set of domain- specific vectors shared across all classes j E {1.2...., .M2, E t). See Figure for an overview. Here, Mi and M2 represent the number of tokens, K is the number of classes, is short for source and is short for target, resulting in a total of 2K categories for training. In other words, each class prompt tf E IRI X2)x212 prompt". Therefore, the prompt for each source and target pair can be derived as:
concatenation of a "source prompt" and a "target
P. [t f2,t tx t tg t......I]]
1T i E (1.
(2)
These prompts serve as learnable parameters that help bridging the domain gap between source domain and the target domain through a contrastive loss, as will be introduced below.
Learning Individual Prompts. To apply prompt learning to multi- UDA, we first train individual prompts for each source and target pair using the image and text encoders of CLIP. Given an image x sampled from the source domain D, whose label is y", we optimize the prompts so that the outputs from the image and text encoder aligned. For an image xf from the target
4

---

Under review as a conference paper at ICLR 2023
Domain-invarian
Domain-specific Feature
Feature
Text Encoder
CLS
Source Prompi
M
CLS
M
CLS
Source Image
Image Encoder
0.82
0.00
0.01
0.12
0.04
Target Prompt
d
d
CLS
0.02
0.01
0.75
0.05
0.00
C
Targel Image
Figure 2: Each source and target pair prompt P, the concatenation of "source prompt" segment and domain-specifee features. There-
and "target prompt" segment composed of fore, the size of P. is R2 x512 During our prompt training stage, the text encoder and the image encoder of CLIP are both frozen.
domain D. whose label is unknown, we first leverage the strong zero-shot ability of CLIP to generate static pseudo-label y for image-text alignment. Pseudo-labels only generated for images whose maximum probability is larger than a fixed threshold r using Equation 1. While more sophisticated approaches like self-training could be leveraged to generate pseudo labels (Zou et al., 2018; Liu et al., 2021a), we find that pseudo labels from CLIP are simple and effective. Finally, prompts trained with cross-entropy loss functions and Figure 2 gives an overview of the process. More formally, for prompt P, f1,2...., N 1}, the objective function for optimization follows:
min P.
logP(y- y²1="; P.)
logP(y ÿ x²; P.).
(3)
The xp" TD
w'
Here, the probability P(-1x= P:) of an image sample belonging to the k-th class is derived from a contrastive loss:
exp(<g((#)). g(()), f(x (T)
P-y= = k|o
Ede(e.e) Σ exp(< [(x) f(xrd) > /T)
where d E {s, of a domain identifier indicating where the image comes from, T is learnable tem- perature parameter, and f and g represents the image and text encoder in CLIP respectively, which are kept frozen during training. This specific design can push the prompts to learn disentangled rep- resentation of both class-invariant and class-speciff semantic information boost the performance of domain adaptation methods (Bousmalis et al., 2016; Liu et al., 2018). Once prompts are learned, the predieted label of an image x' can be computed
exp(<< f()), (T)
arg max k
(5)
E f(t)), /T)
Multi-Prompt Alignment. Thus far, we have obtained prompt for each source and target domain pair. However, the number of images as well as the noise level in each source domain varies, and hence these learned prompts might produce inconsistent results even for the same image. In the second stage, we aim to align predictions from different prompts and more importantly, we wish to find conainninvariiat latent space that minimizes the noise in learned prompts well can potentially generalize to unseen domains. To this end, we leverage auto-encoders that are trained to reconstruct the learned prompts. More formally, we of a projection function Proj(-) and a back-projection function Proj. The learned prompts P. are first projected into a latent subspace of lower dimension d; by Proj(-), followed by Proja(-) projecting the vectors back into soft prompts P.. Instead of reconstructing the whole prompt, we adjust P. to only contain the target token segment and feed its domain-speciff and domain-invariant pieces to the two auto-encoders respectively. We posit that since these two feature vectors serve different purposes, using two separate auto-encoders will help with the alignment process. The Proj(-) function is implemented by a one-layer feed forward network while Proj,()) is implemented by two-layer nonlinear perceptron:
separate auto-encoders, each consisting
Proj(P.) = (P.)) + b and Proj Wa(tanh( W2d, + b.)) b2
5

---

Under review as a conference paper at ICLR 2023
M x-12
KM, xd,
K x512
Frauure
Fixed parameters
parameters
Kx(M, M,
repeae
Kx x512
1x122
PIK
ryporr
1x x512
1x x512
KM, xd,
Prompe
M, xd,
M, xd,
(Taning
(a) Reconstruction of prompts
(b) Generalize novel domain
Figure 3: (a) All prompts are first sliced to the size + containing only the target token segment. Then they are further partitioned into Nomain-invariane and domain-specinfe pieces that are projected into the same latent space and aligned by an auto-encoder structure. (b) The latent subspace learned by the auto-encoder
be utilized for generalizing novel domain.
Here, d, represents vector in the embedding space and we optimize reconstruction loss
1 N LAE N Σ P. P.12 where P. Proj, (Proj(P:)).
Intuitively, would expect certain target domain image to be classified the same category for all the reconstructed prompts P.. Inspired by this, align the reconstructed prompts by introducing an additional Li loss to the objective function
N-2 N
L (N 1) (N 2)
2
Σ
Σ
Ply = k P.) Py = k2 P3)
(8)
j=1 1=1+1
The overall loss function now becomes:
f= + aL1 + LAE:
where LCLS is the cross entropy loss calculated using the reconstructed prompts P. and static pseudo labels. Here a is a hyper-parameter controlling the weight of the Lı loss. The whole training procedure is depicted in Figure 3a. Finally, for predicting the labels of target samples, compute the average of the output logits using each P..
Generalizing to Unseen Domains. With multi pollipp alignment, we are able to derive a low- dimensional embedding space that potentially captures the relationships among different domains. With this, we introduce a Latent Subspace Tuning (LST) strategy. Given novel domain, domain-invariin invariant feature vector Ptune E R M, dd; with domain-speciff feature vector drane R Ma xdj is randomly initialized and passed to the two learned back projection function Proj,
from MPA. Consequently, entirely new prompt P can be constructed for predicting image labels on the novel domain. This is achieved by minimizing the following loss function:
min
logP(y 1² a new, dranes Utune)
(10)
deune Ptune
Oner
de (ee Drew
By doing so, this further decreases the number of tunable parameters by a factor of at least (N 1) de times (for example, the Office-Home dataset, d, = 150, d, = 512 and N 4) as against to MPA when adapting to novel domains and is thus much more computationally efficient. Since data from the source domains are not required in the training of LST,, this is in fact a problem test time adaptation (Kundu et al., 2020; Liang et al., 2020; Wang et al., 2021). Considering that the domain of interest is not involved in the training of the latent subspace throughout the whole process, we dub it "unseen".
dy
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets and metrics. Experiments conducted on three popular benchmark datasets of UDA to evaluate the effectiveness of MPA, including ImageCLEF, Office-Homm and DomainNet. Image- CLEF is small-scaled dataset consisting of 1,800 images from 12 object categories in different

---

Under review as a conference paper at ICLR 2023
ImageNet ILSVRC 2012(D,, Pascal VOC 2012(P), and Caltech-256 (C). Office-Home is a medium scaled dataset consisting of about 15,500 images from 65 categories in 4 different do- mains: Art, Clipart, Product and Real World. DomainNet is the largest dataset to date, consisting of about 0.6 million images from 345 categories in 6 different domains: Clipart, Infograph, Painting, Quickdraw, Real and Sketch.
domains:
We use top-1 accuracy as our evaluation metric and report results of the following settings: (1) CLIP: zero-shot CLIP on the target domain, which can be regarded as baseline of our method. (2) Source Combined: all source domains are combined into one single domain and applied with popular single-source UDA methods. Specially in this setting, we adopt the prompting method from Zhou et al. (2022) to serve as another baseline named as "Simple Prompting".
(3) Multi-Son Multi-Source:: results reported from other multi-source UDA methods. Note that we tried to re-implement state-of- the-art methods using backhone networks from CLIP, yet most of the results are unsatisfactory and thus we only report one such attempt. Similar trends are also found in (Devillers et al., 2021; Yang et al., 2022) when transferring CLIP to pure vision tasks.
4.2 COMPARISON TO STATE-OF-THE-ART
The results on ImageCIEl and Office-Home are shown Table 1. For ImageCIEER, is obvious that MPA outperforms other methods every task with average accuracy of 91.7%, where there is at least a 3% increase when adapting to domain C and I. For Office-Home, MPA achieves the best results except when adapting the domain Clipart. Nevertheless. on average, which is 1.3% higher than the second best method MFSAN. It is worth noting that compared state-of-the-aas method MFSAN on both datasets, MPA only trains 0.78M and 2.36M parameters, while MFSAN has total of 51.75M and 51.80M parameters needed for optimizing (66.3 and 21.9 times larger than ours). Furthermore, from more powerful backbone, we conduct two different sets of experiments. One is applying simple prompt learning method (Zhou et al., 2022) to the source combined scenario; and again, as Table suggests, while the Simple Prompting baseline is I 1% on average better than zero shot CLIP, MPA still outperforms with a significant margin. The other experiment is by directly switching MESAN's ResNet50 backbone pretrained ImageNet CLIP's image encoder. Surprisingly, when tested on the ImageCLEF dataset, the performance even drops by a small margin 0.3%. To conclude, both results demonstrate that CLIP's backbone not universally better and the majority of the performance gain of our method is from its domain adaptation ability.
achieve an accuracy of 75.4%
justify whether the performance gain
ImageCLEF
Office- Home
C
+I
+P Avg + Ar + - - Rw Avg
Zero-Shot
CLIP (Radford et al., 2021)
95.1
87.3
74.0
85.5
71.5
50.2
81.3
82,4
71.4
Source Combine
DAN (Long ct al., 2015b)
93.3
92.2
77.6
87.7
68.5
59.4
79.0
82.5
72.4
DANN (Ganin ct al., 2016)
93.7
91.8
77.9
87.8
68.4
59.1
79.5
82.7
72.4
D-CORAL (Sun & Saenko, 2016)
93.6
91.7
77.1
87.5
68.1
58.6
79.5
82.7
72.2
DAPL* (Ge et al., 2022)
96.0
89.2
76.0
87.1
72.8
51.9
82.6
83.7
72.8
Simple Prompt"
93.6
90.6
80.9
88.4
70.7
52.9
82.9
83.9
72.4
Multi-Source
DCTN (Xu et al., 2018)
95.7
90.3
75.0
87.0
N.A.
N.A.
N.A.
N.A.
N.A.
MDDA (Zhao al., 2020)
N.A.
N.A.
N.A.
N.A.
66.7
62.3
79.5
79.6
71.0
SImpAIço (Venkat al., 2020)
93.3
91.0
77.5
87.3
70.8
56.3
80.2
81.5
72.2
MESAN (7hu et al., 2019)
95.4
93.6
79.1
89.4
72.1
62.0
80.3
81.8
74.1
MFSAN+CI.IP
96.7
93.0
77.7
89.
N.A.
N.A.
N.A.
N.A.
N.A.
MPA (ours)
98.6
96.2
80.4
91.7
74.8
54.9
86.2
85.7
75.4
Table Accuracy (%) on ImageCI.F and Office-Home. 'implies that the method is based on our implementation
Table 2 shows that for DomainNet. MPA exceeds other multi-source UDA methods by more than
5%. To the best of our knowledge, this the highest reported accuracy on this dataset so far with less than third parameters optimized compared with most otate-of-the-arts methods. Regard-
7

---

Under review as a conference paper at ICLR 2023
DomainNet
Clp
Inf
Pnt
Qdr
Rel
Skt
Avg
Zero-Shot
CLIP (Radford et al., 2021)
61.3
42.0
56.1
10.3
79.3
54.1
50.5
Source Combined
DANN (Ganin al., 2016)
45.5
13.1
37.0
13.2
48.9
31.8
32.6
MCD (Saito et al., 2018)
54.3
22.1
45.7
7.6
58.4
43.5
38.5
DAPL (Ge et al., 2022)
62.4
43.8
59.3
10.6
81.5
54.6
52.0
Simple Prompt*
63.1
41.2
57.7
10.0
75.8
55.8
50.6
Multi-Source
DCTN (Xu et al., 2018)
48.6
23.5
48.4
7.2
53.5
47.3
38.2
SImpATon (Venkat ct al., 2020)
66.4
26.5
56.6
18,9
68.0
55.5
48.6
S'A-3 (Peng et al., 2019)
58.6
26.0
52.3
6.3
62.7
49.5
42.6
LIC-MSDA (Wang ct al., 2020)
63.1
28.7
56.1
16.3
66.1
53.8
47.4
T-SVDNet (Li et al., 2021)
66.1
25.0
54.3
16.5
65.4
54.6
47.0
PFSA (Fu et al., 2021)
64.5
29.2
57.6
17.2
67.2
55.1
48.5
PTMDA (Ren ct al., 2022)
66.0
28.5
58.4
13.0
63.0
54.1
47.2
MPA (ours)
65.2
47.3
62.0
10.2
82.0
57.9
54.1
Table 2: Accuracy (%) DomainNet. implies that the method is based on implementation
ing individual adaptations, MPA achieves best results on most of the adaptation tasks but performs mediocre on the Quickdraw domain. Interestingly, the result is even little worse than CLIP. We hy- pothesize that this is because of the large domain gap between Quickdraw and other domains, While CLIP exhibits strong performance on the DomainNet dataset, its performance on ImageCLEF and OfficeHome dataset is limited. On the contrary, MPA consistently achieves decent results regardless of the dataset assessed on. As result, for all three datasets, MPA surpasses CLIP by 6.2%, 4.0% and 3.6% respectively. More importantly, the Simple Prompt baseline reached an average accuracy of 50.6%, 3.5% lower than MPA. All above results further demonstrate the success of our strategy.
Effectiveness of Generalizing Unseen Domains Inspired by recent research that shows the generalization ability of large language models when adapting to unseen tasks presented with lan- guage prompts (Sanh et al., 2022), we are interested in whether MPA also possess such ability. Results from Table and Table 2 are already strong indication of the success of MPA under conven- tional multi-source UDA setting. Now, we would like investigate more efficient framework, the LST strategy mentioned in Section 3.2, for test-time adaptation tasks.
We first present a concrete example that simulates a test-time adaptation scenario. On the Office- Home dataset, LST can be conducted by training MPA using Clipart and Product as source domains and Art as target domain, while one-tuning the latent subspace on the unseen Real World domain. Applying this to all domains from the Office-Home and DomainNet dataset produces results shown in Table 3, where compared with MPA, the performance only dropped by 0.4% on DomainNet and 1.6% on Office-Home. Nevertheless, LST achieves higher accuracy compared methods of Table and Table 2. In particular, LST is still better than CLIP with an increase of 2.4% and 3.2% respectively for the tested datasets. Notably, only a total of 0.17M and 1.47M parameters are tuned, which is further drop in the number of parameters (0. 17M v.s. 2.36M, 1.47M v.s. 15.9M in MPA). All of these are convincing evidence showing the generalization ability of MPA.
most baseline
OfficeHHome Pr
DomainNet
Ar
Cl
Rw
Avg
Clp
Inf
-
- Qdr
Rel
Skt
Avg
CLIP
71.5
50.5
81.3
82,4
71.4
61.3
42.0
56.1
10.3
79.3
54.1
50.5
MPA
74.8
54.9
86.2
85.7
75.4
65.2
47.3
62.0
10.2
82.0
57.9
54.1
LST
72.9
52.2
84.9
85.0
73.8
64.6
46.7
61.6
9.8
81.2
57.6
53.6
Table 3: Results (%) of different approaches when generalizing unseen domains.
Qualitative Results To obtain further insights into MPA, visualize, in Figure 4, the distribu- tions of prediction confidence of MPA and CLIP for the four target domains on the Office-Homa dataset. We see that for Art. Prouct and Real World, MPA produces highly confident predictions for the majority of samples. As for the Clipart domain where the recognition accuracy is lower than the other domains, MPA is clearly better than CLIP that produces distribution skewed to the bottom.
8

---

Under review as a conference paper at ICLR 2023
MPA CLIP
16gh
Confidence
Low
Art
Product Figure 4: Visualization of the confidence density.
Clipart
Real World
4.3 ABLATION STUDY
Effectiveness of Prompt Alignment The second stage of MPA be seen refinement of the prompt learned during stage one. Thus, we verify how the quality of those prompts has changed and the results accuracy of CLIP by an average of 1.2%, while the second stage improves the first stage by an average of 2.8%, highlighting that the prompt has indeed been refined.
shown in Table 4. It is worth pointing out that stage one of MPA improves the
A CA-
PA-
R C-
+ PC- RP-
P
CP
R
RR-
A
R
CR-
P Avg
CLIP
50.2
81.3
82.4
71.5
81.3
82.4
71.5
50.2
82.4
71.5
50.2
81.3
71.4
Stage One
52.2
83.5
82.1
72.8
83.6
82.8
73.3
52.7
82,4
72.0
51.6
82.1
72.6
Stage Two
54.1
85.9
85.2
74,3
86.0
85.3
74.6
54.1
85.3
74.3
54.2
86.0
75.4
Table 4: Performance (%) of individual prompts on Office-Home dataset.
Effectiveness loss function To validate the effectiveness of our objective function Equation 9, two variants are evaluated: one without the A loss and one without the L1 loss. Table 5a shows that the removal of either piece will result a performance degradation of about 0.7%. particular, f loss exhibits a larger impact the more difficult Clipart domain, where the accuracy dropped by a large margin of 1.5% without the &1 loss, demonstrating the effectiveness of enforcing consistent predictions on different prompts.
L1
CAN
Ar
CI
Pr
Rw
Avg
# of AE
Ar
CI
Pr
Rw
Avg
74.8
53.4
85.2
85.3
74.7
Zero
75.0
53.4
85.7
85.4
74.9
/
x
74.0
53.8
85,5
85.4
74.7
One
74.6
54.3
85,9
85.6
75.1
/
/
74.8
54,9
86.2
85.7
75.4
Two
74.8
54.9
86.2
85.7
75.4
(a) Effectiveness of loss function.
(b) Effectiveness of auto-encoders.
Table 5: Ablation studies.
Effectiveness of auto-encoders In addition to the objective function, we also tested the effec- tiveness of utilizing the auto-encoder structure in MPA. For details, auto-encoders means completely discarded the auto-encoder structure and one auto-encoder for testing the necessity of using two separate auto-encoders for reconstructing domain-specific and domain-invariant prompt. Results from Table 5b show that in general, our design of the auto-encoder structure is beneficial the overall performance. Furthermore, find that another benefit of applying the auto-encoder structure is that it helps stabilizing the training process.
5 CONCLUSION
In this paper, we introduced prompt learning to multi-source UDA and proposed a simple MPA scheme to align the source and target domains. MPA is composed of two stages. The first stage is to train individual source and target pair prompts and the second stage is to align them by auto-encoder structures. Extensive experiments showed that MPA achieved better results
various multi-source UDA tasks with substantially fewer parameters tuned. Moreover, a latent subspace tuning strategy was introduced for generalization to novel unseen domains that further dropped the number of pa- rameters needed for training and can be applied to test-time adaptation tasks. Hopefully, our work inspire future work applying prompt learning to transfer learning.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sk Miraj Ahmed, Dripta S Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K. Roy-Chowdhury. Unsupervised multi-source domain adaptation without access to source data. In CVPR, 2021.
Eyal Ben-David, Nadav Oved, and Roi Reichart. Pada: Example- baadlee-asedd prompt learning for on-the- fly adaptation to unseen domains. TACI., 2022.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. NeurIPS, 2006.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In NeurlPS, 2016.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017.
Gabriela Csurka. Domain adaptation for visual applications: A comprehensive survey. arXiv preprint arXiv:/702.053: 2017.
Benjamin Devillers, Bhavin Choksi, Romain Bielawski, and Rufin VanRullen. Does language help generalization in vision models? arXiv preprint arXiv:2104.08313. 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Yangye Fu, Ming Zhang, Xing Xu, Zuo Cao, Chao Ma. Yanli Ji, Kai Zuo, and Huimin Lu. Partial feature selection and alignment for multi-source domain adaptation. In CVPR, 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. JMLR, 2016.
Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang, Domain adaptation via prompt learning. arXiv preprint arXiv:2202.06687 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Judy Hoffman, Mehryar Mohri, and Ningshan Zhang. Algorithms and theory for multiple-soun source adaptation. In NeurlPS, 2018
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint htt::2203.1211112 2022.
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding, arXiv preprint arXiv:2112.04478 2021.
Guoliang Kang. Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In CVPR, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Jogendra Nath Kundu, Naveen Venkat, Venkatesh Babu, al. Universal source-free domain adaptation. In CVPR, 2020.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffiient prompt tuning. arXiv preprint arXiv:2104.08691 2021.
Ruihuang Li, Xu Jia, Jianzhong He, Shuaijun Chen, and Qinghua Hu. T-svdnet: Exploring high- order prototypical correlations for multi-source domain adaptation. In ICCV, 2021.
10

---

Under review as a conference paper at ICLR 2023
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint orXiv:2101.00190 2021.
Yitong Li, David E Carlson, et al. Extracting relationships by multi-domain matching. In NeurIPS, 2018.
Jian Liang, Dapeng Hu, and Jiashi Feng. we really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
Hong Liu, Jianmin Wang, and Mingsheng Long. Cycle self-training for domain adaptation. In NewrlPS, 2021a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.13588 2021b.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. ECCV,221..
Yen-Cheng Liu, Yu-Ying Yeh, Tzu-Chien Fu, Sheng-De Wang. Wei-Chen Chiu, and Yu- Chiang Frank Wang. Detach and adapt: Learning cross-domain disentangled deep representation. In CVPR, 2018.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015a.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015b.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. TKDE, 2010.
Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang. Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV, 2019.
Joaquin Quinonero- Cainonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. 2008
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Sayan Rakshit, Biplab Banerjee, Gemma Roig, and Subhasis Chaudhuri. Unsupervised multi-source domain adaptation driven by deep adversarial ensemble learning. In GCPR, 2019.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
Chuan-Xians Ren, Yong-Hur Liu, Xi-Wen Zhang, and Ke-Kun Huang. Molti-source unsupervised domain adaptation via pseudo target domain. TIP, 2022.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. CVPR, 2018.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In ICLR, 2022.

---

Under review as a conference paper at ICLR 2023
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, 2016.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, 2011.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Naveen Venkat, Jogendra Nath Kundu, Durgesh Singh, Ambareesh Revanur, et al. Your classifier can secretly suffice multi-source domain adaptation. In NeurlPS, 2020.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang. Learning to combine:
Knowledge
aggregation for multi-source domain adaptation. In ECCV, 2020.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputing. 2018.
Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in down- stream tasks? an analysis of head and prompt tuning. In NeurIPS, 2021.
Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi- source unsupervised domain adaptation with category shift. In CVPR, 2018.
Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In CVPR, 2022.
Jun Yang, Rong Yan, and Alexander G Hauptmann. Cross-domain video concept detection using adaptive svms. In ACM MM, 2007.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. ICML, 2013.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey Gordon. Adversarial multiple source domain adaptation. In NeurIPS, 2018.
Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-source distilling domain adaptation. In AAAI, 2020.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. IJCV, 2022.
Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang. Aligning domain-specific distribution and clas- sifier for cross-domain classification from multiple sources. In AAAI, 2019.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class- balanced self-training. In ECCV, 2018.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 IMPLEMENTATION DETAILS
For fair comparisons, we adopt ResNet50 as our backbone on ImageCLEF and Office-HHoee and a ResNet101 on DomainNet. The weights for ResNet50 and ResNet101 are from CLIP and frozen through our experiments. The prompts and auto-encoders are trained using the mini-batch SGD optimizer with a learning rate of 0.003 and 0.005 while the learned subspace is tuned with 0.0005 learning rate. We use a batch size of 32 and adopt cosine learning rate scheduler. For the hyper- parameters of stage one, token length M1 and M2 are both set to 16. Pseudo-label threshold 7 is set 0.4 for producing reliable labels. As for stage two, a in Equation is set to 500. Ablation studies on hyper-parameter selection is presented in the following section. The weight matrix W2 of the back projection function in Equation has a size of 183xxx, where d, is 100 for ImageCLEF, 150 for OfficeHome and 250 for DomainNet. Therefore, for generalizing novel domains, only 0.02M, 0. 1.M and 1.47M parameters are tuned respectively.
A.2 PAPER-PARAMETER SELECTION
Experimental results for ablation studies on hyper-paramene selections are reported in Table 6.
+ Ar + CI + Pr + Rw
Avg
Token length
+ Ar - CI + Pr + Rw Avg
0.3
74.5
55.0
86.1
85.9
75.4
Mi M2 =
74.3
54.9
85.8
85.3
75.1
0.6
74.6
54.9
85.9
85.3
75.2
M, -M2
74.6
54.3
85.9
85.6
75.2
0.8
74.0
54.2
85.2
85.5
74.7
M = M2 = 20
74.8
55.2
86.3
86.0
75.6
0.4 (reported)
74.8
54.9
86.2
85.7
75.4
M = M2 = 16 (reported)
74.8
54.9
86.2
85.7
75.4
(a) Ablation on pseudo-label threshold T.
(b) Ablation on token lengths M1 and M2.
Ar
CI
Pr
Rw
Avg
74.4
53.7
84.9
85.6
74.7
10
74.5
54.1
85.7
86.0
75.1
100
74.7
54.5
85.5
85.6
75.1
1000
74.4
55.0
86.3
86.0
75.4
500 (reported)
74.8
54.9
86.2
85.7
75.4
(c) Ablation on
Table 6: Ablation studies hyper-parameter selection.
For both pseudo-label threshold and prompt token length M1, M2, three different choices т (0.3, 0.6, 0.8) and M1, M2 E {8, 12, 20} (for simplicity we are setting M, and M, to be equal) are examined. As t increases, while the quality of the pseudo labels gets higher, fewer images will be fed into the model, and Table 6a suggests that doing so hurts the overall performance. On the contrary, shown in Table 6b, the general trend for prompt token length is that the longer the prompt, the better the performance. Consequently, we choose т 0.4 and M, the off between performance and efficiency. For a in Equation 9, we examined four different choices {1, 10, 100, 1000} and the main reason chose 500 for is to balance all losses (in this case L1) to be of the same order of magnitude. Our experimental results from Table 6c also support such motivation.
M, = 16 to balance
A.3 ABLATION ON PROMPT DESIGN
We also conduct experiments ablating the necessity of using class and domain specific tokens in our prompt design.
+ Ar
-
Cl
- Pr
Rw
Avg
Only class specific tokens Class specific and domain specific tokens (reported)
74.2
53.0
85.2
85.3
74.4
74,8
54,9
86.2
85.7
75.4
Table 7: Ablation on prompt design.
13

---

Under review as a conference paper at ICLR 2023
Note that for a fair comparison, when excluding domain specific tokens in the prompt design, the total number of tokens remains 32, f.e., M1 = 32, M2 O. Still, results from Table demonstrates that the reported prompt design exhibits significant performance gain.
A.4 SIMILARITY BETWEEN RECONSTRUCTED PROMPTS
As is observed in Table 4, the reconstructed prompts of different domains achieve almost the same results on the target domain and this is further indication of the success of our alignment strategy. To justify such statement, the reconstructed prompts are tested without incorporating L1 loss in the objective function and the results are shown in Table
Source
Source
Ar
Cl
Pr
Rw
Std
Ar
Cl
Pr
Rw
Std
Target
Target
Ar
73.5
74.1
74.3
0.34
Ar
74.3
74.6
74.3
0.14
53.4
52.6
52.9
0.65
54.1
54.1
54.2
0.05
Pr
85,4
82.6
83.9
1.14
Pr
85.9
86.0
86.0
0.05
Rw
84.2
84.6
83.5
0.45
Rw
85.2
85.3
85.3
0.05
(a) Without &1 loss
(b) With Ln loss
Table 8: Comparison of performance (%) of individual prompts with and without Li loss.
By comparing the standard deviation on the same target domain, it clear that, without the L loss, the reconstructed prompts achieve different results on the target domain.
A.5 MPA WITH LIMITED DATA
Since prompt learning is also well-known for its strong few-shot ability, a limited data setting is designed where only small portion of the data used for training MPA while still testing it on the full dataset to assess whether our method has inherited such ability. We think that this framework also practically meaningful access to data might be limited in real-life scenario due privacy, security and storage concerns (Ahmed et al., 2021). Specifically, no more than 50 images of each category are sampled using two distinct sampling strategies. The most direct one is by complete random sampling. On the contrary, strategy two selects high quality images by utilizing CLIP's zero shot inference ability similar our pseudo label generating process.
Such setting is only experimented on the DomainNet dataset due to its large data scale. Despite the fact that the total number of data was reduced from 0.6M to roughly 0. results from Table show that MPA is still capable of achieving surprisingly good result where the recognition accuracy only dropped by an average of 0.7% compared with full data training and is still 2.9% better than our baseline CLIP. We also want to point out that the quality of input images seems to be irrelevant to the performance, as results from the two sampling strategies don't 't vary much.
- Clp
+
Inf
+
Pnt
Qdr
+ Rel
Skt
Avg
CLIP Random Sampled Data
61.3
42.0
56.1
10.3
79.3
54.1
50.5
64.1
46.9
60.5
9.6
81.6
57.3
53.3
CLIP Sampled Data
64.2
47.3
60.9
9.6
81.6
57.6
53.5
All Data
65.2
47.3
62.0
10.2
82.0
57.9
54.1
Table 9: MPA with limited data on the DomainNet dataset
14


Reviews:
[{"review_id": "sknscmsW5j", "reviewer": "Reviewer_Zi5U", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-prompt alignment method for multi-source unsupervised domain adaptation. The main idea is to first learn an individual prompt for each source-target domain pair and then mine the relationships among learned prompts through deriving a shared embedding space. The resulting embedding is expected to be domain-invariant and can be generalize to unseen domains. ", "summary_of_the_review": "In summary, the overall novelty is limited. Moreover, the design of the model is not very well justified, which requires further clarification.", "strength_and_weaknesses": "- Strength\n    1. The introduction of prompt learning into multi-source domain adaptation is interesting. \n    2. The performance of the proposed method on MSDA is good compared to previous methods.\n- Weaknesses\n    1. The idea of prompt learning for domain adaptation is not new, and the prompt design in this paper is highly based on the method proposed by Ge et al., 2022. So the main contribution is to extend the prompt learning for domain adaptation from the single source domain to multiple source domains. The novelty is limited.\n    2. How to ensure the two-stage training will achieve the desired solutions? Since the individual prompts are learned to reduce domain shift between each individual source and the target domain while the multi-prompt alignment stage aims to align multiple source domains. The learning objective is changing in the two stages. Is it possible to combine the two stages into a single one to achieve the alignment of all the domains simultaneously?\n    3. Why is the auto-encoder necessary? If the two stages of prompt learning and alignment strategy can be combined into a single stage with multiple objectives for alignment and consistent prediction, is it still require the auto-encoder for reconstruction?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "l0L8koVnkKh", "reviewer": "Reviewer_xddt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method based on prompt learning for multi-source domain adaptation. By adopting a pre-trained CLIP based text and image encoders, the authors design prompts that are learnable to adapt them for source and target domains. Specifically, there are two stages. First, prompts are designed with class-specific and domain-specific ones, and are learned for each source and target pairs individually via a contrastive loss. Then, to further align between target prompts, the authors apply the autoencoder-based method for prompt reconstruction, with a loss function to ensure output predictions are similar from different prompts. In addition, the authors also show a finetuning scheme that can adapt to new domains with small finetuning parameters. Experiments are conducted on three benchmark settings, including ImageCLEF, Office-Home, and DomainNet.", "summary_of_the_review": "Overall, using prompt learning is an interesting direction for domain adaptation. However, there can be issues by using pre-trained CLIP that already has strong generalization ability for DA. In addition, some design choices in the proposed method are not well motivated nor well validated. The authors should consider the above comments and address them carefully in the rebuttal.", "strength_and_weaknesses": "**Strength**\n\n* The idea of using prompt learning for domain adaptation is interesting, as prompt learning has the potential for transfer learning in the new domain under a cheaper cost\n* Experimental results show improvements over the CLIP baseline\n* Ablation study shows some gains for the designed modules\n\n**Weakness**\n\nTechnical motivation and novelty\n* Although applying prompt learning for domain adaptation is interesting, the motivation is not very intuitive as prompt learning is based on a highly generalized CLIP model, which already suffers less in the domain adaptation setting. This may raise two questions: 1) whether the experimental comparisons are fair (see comments below), and 2) whether the alignment really happens as CLIP already performs very well in target domains (see comments below).\n\n* The proposed techniques in this paper are not new, e.g., extending the prompt design to multi-domains is straightforward and the loss functions (e.g., contrastive loss) are also not new. More importantly, some designs are not well motivated and lacks experimental validations (discussed below).\n\n* For multi-prompt alignment, it's not clear how the autoencoder is needed to achieve alignment between prompts. In experimental results (Table 5 (a)), performance gain is also marginal.\n\n* It seems the performance of using prompt learning would highly depends on the CLIP model (e.g., Qdr results in Table 2). This should be discussed in the paper.\n\nTechnical clarity\n* For eq(8), it's not clear which prompts are aligned, e.g., between P_i and P_j\n* d_I appears many times in the paper but the definitions are not clear, e.g., eq(6), dimensions in v_tune and d_tune.\n\nExperimental results\n* Since CLIP already has a strong generalization ability, e.g., many results in Table 1 and 2 (Zero-Shot) are already better than other existing DA methods, it's not clear whether the comparisons are fair, as CLIP has been trained with millions of images. While I understand it's not easy to perform experiments using the CLIP backbone with existing methods, the proposed training scheme also cannot validate it's DA ability but only show the transfer learning ability.\n* Prompts are designed with class-specific and domain-specific ones. However, there are no experiments to validate whether it is necessary.\n* Lots of sensitivity experiments are not provided, e.g., length of prompts, thresholds for pseudo-labels, /alpha in eq(9)\n* The explanation of using LST for experiments is not clear (paragraph above Table 3)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XpFy-3Bcn1", "reviewer": "Reviewer_iJo5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the multi-source UDA problems by proposing the method Multi-Prompt Alignment (MPA). MPA is built on the pre-trained CLIP which can effectively encode images and texts. Compared to other existing approaches to mutli-source UDA, MPA only needs to train a small number of parameters by prompt learning.\n", "summary_of_the_review": "The proposed approach is based on the pre-trained CLIP and utilises the prompt learning strategy to learn useful prompts for mutli-source UDA. However, the main concern is that it is not justified where the performance gains mainly come from, the proposed approach MPA or the pre-trained CLIP?", "strength_and_weaknesses": "\n++ The proposed method based on CLIP can address the multi-source UDA problems by prompt learning which is more efficient during model training.\n++ The proposed method can generalise to unseen domains and the parameters to learn can be further reduced.\n++ The experimental results on three benchmark datasets show competitive performance compared with others. The ablation study also validates the effectiveness of different components of the proposed approach.\n\n-- It seems the threshold for pseudo-label selection is a very important hyper-parameter; the author should discuss how the value affects the performance.\n-- The proposed method uses pre-trained CLIP as the backbone. Since CLIP is pre-trained on a different dataset from other comparative models pre-trained on ImageNet. How to justify the performance gain is not from a more powerful pre-trained model when compared with other multi-source UDA approaches? A simple baseline can be \"source combine\" + \"simple prompt learning\".\n-- The authors claim the proposed approach can generalise to unseen domains by the introduced Latent Subspace Tuning (LST) strategy. Does it mean it can solve the domain generalisation problem? It seems pseudo labels are required in Eq.(10) which means the \"unseen\" domain is actually seen during training, please clarify this.\n-- Although the proposed method aims for multi-source UDA, it seems that it can also solve single-source UDA problems. How it performs when compared with SOTA UDA approaches?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UFD1FcAVixE", "reviewer": "Reviewer_rNJs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores how CLIP can be used effectively for multi-source domain adaptation. In this problem setting there is a single labeled source domain dataset and multiple target domain datasets that are not labeled. Several methods have been proposed in past to address this problem, but they all have limited accuracy owing the limited abilities of ImageNet trained model (in comparison to CLIP dataset). Authors propose a simple idea to learn prompts using pseudo-labels from CLIP and propose an auto-encoder network to learn a latent space that generalizes to new domains. While not surprising, the proposed CLIP based method outperforms the existing methods. It is interesting to note that ResNet models initialized from the CLIP's image encoder do not perform as well.", "summary_of_the_review": "Based on the strengths and weaknesses of the paper, I vote to accept the paper.", "strength_and_weaknesses": "**Strengths**\n\n* The paper demonstrates how prompt engineering/learning can be performed in context of multi-source domain adaptation.\n* The paper shows that the the text encoder plays an important role in obtaining higher accuracy.\n* The results on the challenging domain-net dataset is impressive. The proposed method achieves state-of-the-art results with fewer trainable parameters.\n\n**Weaknesses**\n\n* While the paper includes ablation studies on other methods with image encoder, it is not clear (unfair comparison) if this enough to benchmark the proposed methods against the previous method. \n* It is not clear how to go about deciding the hyperparams $M_1$ and $M_2$. 16 seems a little too big for the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4TsJaSoogV", "reviewer": "Reviewer_iJht", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces prompt learning to multi-source UDA. A simple two-stage framework is proposed. In the first stage, individual prompts for each source and target pair are learned. In the second stage, Multi-Prompt Alignment (MPA) is proposed to align the learned prompts. The experimental results show the effectiveness of the proposed framework.", "summary_of_the_review": "Overall, this is a good submission with strong results. I hope the author could provide more detailed explanations. ", "strength_and_weaknesses": "Strengths:\n1.\tThis paper is first to apply prompt learning to multi-source UDA problem.\n2.\tThe proposed framework is simple and achieves state-of-the-art results on multi-source UDA benchmark.\n3.\tThe paper is well written, easy to understand.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject