Enhancing Domain Adaptation through Prompt Gradient Alignment
Authors: Hoang Phan
Tung Lam Tran
Quyen Tran
Trung Le
Venue: NeurIPS 2024 poster (2024)
Keywords: transfer learning; domain adaptation; prompt-tuning
Primary area: other
TL;DR: We propose a novel training method for prompt-based unsupervised domain adaptation by effectively leveraging and manipulating domain-specific gradients.

Abstract: Prior Unsupervised Domain Adaptation (UDA) methods often aim to train a domain-invariant feature extractor, which may hinder the model from learning sufficiently discriminative features. To tackle this, a line of works based on prompt learning leverages the power of large-scale pre-trained vision-language models to learn both domain-invariant and specific features through a set of domain-agnostic and domain-specific learnable prompts. Those studies typically enforce invariant constraints on representation, output, or prompt space to learn such prompts. Differently, we cast UDA as a multiple-objective optimization problem in which each objective is represented by a domain loss. Under this new framework, we propose aligning per-objective gradients to foster consensus between them. Additionally, to prevent potential overfitting when fine-tuning this deep learning architecture, we penalize the norm of these gradients. To achieve these goals, we devise a practical gradient update procedure that can work under both single-source and multi-source UDA. Empirically, our method consistently surpasses other vision language model adaptation methods by a large margin on a wide range of benchmarks. The implementation is available at https://github.com/VietHoang1512/PGA.

Enhancing Domain Adaptation through
Prompt Gradient Alignment
Hoang Phan" New York University hvp2011 edu
Lam Tran" VinAI Research 1amtt120vinai...
Quyen Tran' VinAI Research quyentt15@vinai. 10
Trung Le
Monash University trunglm@monash. edu
Abstract
Prior Unsupervised Domain Adaptation (UDA) methods often aim train a domain-invariant feature extractor, which may hinder the model from learning sufficiently discriminative features. To tackle this, line of works based on prompt learning leverages the power of large-scale pre-trained vision-language models to Iearn both domain-invariant and specific features through a set domain-agnostic and domain-speciffe learnable prompts. Those studies typically enforce invariant constraints on representation, output, or prompt space learn such prompts. Dif- ferently, we cast UDA multiple-objective optimization problem in which each objective represented by a domain loss. Under this new framework, we propose aligning per-objective gradients to foster consensus between them. Additionally, prevent potential overfitting when fine-tuning this deep learning architecture, we penalize the norm of these gradients. To achieve these goals, devise practical gradient update procedure that can work under both single-source and multi-source UDA. Empirically, our method consistently surpasses other vision language model adaptation methods by large margin on a wide range of benchmarks. The imple- mentation is available at https: //tt:////tttt...t/ http//tttu..om/iiiittiee http.///ieeeoooagg//////000
Introduction
Deep learning has significantly advanced the field of computer vision, achieving remarkable perfor- mance in tasks such as image classification 51 object detection [6-9), and semantic segmentation
13. However, the effectiveness of these deep learning models heavily relies on large amounts of labeled training data, which is often labor-intensive and expensive to collect. Moreover, the discrepancy between training data and real-world testing data can lead to substantial performance drops when models are deployed in practical settings [4H
To address these challenges, Unsupervised Domain Adaptation (UDA) has emerged pivotal solution. UDA aims to transfer knowledge from labeled source domain to an unlabeled target domain the presence of domain shift, thereby enabling models generalize well across different domains without requiring extensive labeled data for the target domain. This is often achieved by optimizing objective function on source domains and other auxiliary terms that encourage learning domain- invariant feature representations [7|20] or enhance model robustness 241 which mitigates the domain shift and improve the performance on unseen data. Nevertheless, aligning representations could potentially hurt the model performance due to the loss of discriminative features 25 26
Equal contributions.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Conceptually, our proposed method is orthogonal to these invariant feature learning methods, and they could complement each other.
Recent works leveraging pre-trained models like CLIP 27 for UDA can significantly bridge domain gaps and improve generalization by utilizing rich semantic information and robust visual represen- tations through extensive pre-training on diverse image-text datasets. Following this vein, DAPL 25 first introduces domain-specific and domain-agnostic prompts to efficiently adapt pre-trained vision-language models without fine-tuning the entire model. Furthermore, MPA (28] aligns multiple prompts from different sources using an auto-encoder. While these methods could obtain superior performance on different benchmarks, we find that the main improvement comes from the strong zero-shot performance and self self-training mechanism. In particular, prior works often generate pseudo- label for unlabeled images and then train the model on those samples. Consequently, finetuning pretrained CLIPP model on this dataset alone without leveraging source datasets can help refine model prediction significantly, boosting the performance from 88. 1% to 90.1 %, yielding competitive result compared against MPA, as presented in Table
x
DAP
Dataset
C
P
Avg
Coeeh
Zero-shot
87.9
88.2
78.7
88.1
HTSSM
Simple Prompt
93.6
90.6
80.9
88.4
Accuraty
PGA tours)
HIGAA burs!
Self-training
92.9
94.3
83.2
90.1
MPA
97.2
96.2
80.4
91.3
Table I: Self-training on pscudo-labeled target data already strong baseline.
Number of trainable parameters
Figure 1: Baselines performance on office-Home
Motivated by this observation, we directly optimize the main objective function not only on source domains but also the target data, instead of only using them for auxiliary objectives as previous work 29 30). We thus cast the original UDA problem as multi-objective optimization (MOO) problem. Specifically, we minimize vector-valued loss function, which includes the objectives of multiple source domains and the target domain. This formulation allows us to apply existing results from MOO literature for finding Pareto solutions, from which we can not optimize an objective without hurting another (3133] or encourage positive inter-task transfer between objectives [34 37). Note that in the context of UDA, we focus more on learning the target task, thus motivating us to apply prioritized MOO algorithms [8 40 or to incorporate predefined preferences [32 43). While those methods allow practitioners to focus more or less on the objectives at hand, they come with
the cost of extensive hyperparameter tuning. Besides, recent works [6 argue that simple loss functions reweighting can match the performance of gradient-based MOO methods 35 47 Those findings suggest we focus more the inherent conflict nature of per-objective gradients instead of attempting to remove the conflict between them [35 34).
In this paper, we propose casting the problem of UDA as multi-objective optimization by leveraging powerful pre-trained models. However, while obtaining impressive results on various downstream tasks, over-paree parameterization is still a crucial problem for transformer-bas -based models [8] 49, which
potentially causes overfitting [50 54 more severely than small parammetr architectures 55 espe-
cially in the multi-task learning context. For that reason, we propose to (i) fine-tune pre-trained model via prompt learning, which known for being more robust (56458) and especially more light- than full fine-tuning, (ii) and to leverage the gradient norm penalty to encourage model generalization [59 24 60 61]. Furthermore, we introduce a gradient alignment algorithm to foster inherent con- sensus between per-objective gradients without modifying the gradient itself. Our proposed method, termed Prompt Gradient Alignment (PGA), and its variant for multi-source UDA, Multi-Prompt Gradient Alignment (MPGA), achieve state-of-the-ar performance on different UDA benchmarks. As shown in Figure I PGA and MPGA outperform traditional UDA methods like MESAN (62 and recent prompt- based UDA methods such as MPA (28) and DAMP [63] while requiring fewer trainable parameters. We also provide a generalization bound for UDA and show how theoretical insights motivate the design of our proposed method.

---

2 Related work
Unsupervised Domain Adaptation. A dominant approach solving the UDA problem is to reduce the distribution shift between source and target domains. Following the foundational theory outlined in [64], one group of methods seeks minimize the H-divergence between the marginal distributions of these domains [6567]] Alternatively, other methods aim to align the moments of these distributions, as suggested in 668 (0). Additionally, adversarial learning has been employed to learn innvariat features. For instance, methods such as those in [19 71 use a domain discriminator to differentiate between source and target samples, training a feature extractor to deceive this discriminator. However, as (25] highlights, these methods often struggle with trade-off between domain alignment and classification performance, particularly in multi souti- scenarios where only a single model is used.
Prompt learning-basse based domain adaptation a novel approach introduced in [25], leverages the generalization capabilities of CLIP to learn both domain-agnostic and domain-specific prompts. This method effectively addresses the trade-off between domain alignment and classification performance by employing a contrastive loss. This loss aligns the representation of an image with the prompt corresponding its ground truth class and domain, thereby encouraging the learning of domain- invariant features. Building on this foundation, MPA (28) advances the concept of multi-source UDA. adapts the prompt learning strategy to each source-target domain pair. The prompts are aligned through denoising auto-encoder using Fuclidean distance. However, prompting is known as brittle process where a small shift to the prompt can cause large variations in the model predictions (72 Therefore, in this work, we propose to intervene in the training on the gradient space as it offers a more interpretable and controllable effect during training. Furthermore, PGA is trained in an end-to-end fashion, avoiding the sequential training for each source-target pair as in MPA.
Gradient-based multi-task learning. Due to the multi-objective nature of the multi-source domain adaptation problem, one can leverage recent methods from the multi-tase task learning literature [34 35| [5] to derive an optimization procedure that benefits the learning across domains or put more weight on some specific domains via incorporating preference [3 41 42). While those techniques are readily applicable our context, we directly re-weight per-task gradients, similar to scalarization, instead of adopting multi-task learning methods for simplicity. Furthermore, our work is orthogonal to those gradient-basee based multi-task learning methods where we encourage the consensus among objects instead of directly manipulating their gradients to remove inherent conflicts among them.
Gradient matching is commonly used in continual learning (66-78) to measure conflict and trans- ferability between tasks. A positive dot product between two tasks' gradients implies updating the models with one task can benefit the other. This principle also applied in domain generalization 80) to focus on invariant features. However, our approach aligns in the space of prompt gradient, significantly smaller parameter set than the full model gradients used in previous works. Besides, to avoid the computation of costly second-order derivatives, (79 linearly approximate the inner product between gradients, which underperforms on datasets with larger number of domains due to cumulative approximation error. Meanwhile, our method does not face this problem since we implicitly compute this term without using any approximation. More works sharing the same intuition of gradient alignment are provided in Appendix
3 Background
3.1 Unsupervised Domain Adaptation
Given a set of N 2 1 source domains { each of which is a collection of data-label pairs of domain i, i.c. Ds,; = {zz,, y3 Ns.. and one unlabelled target domain Dr {x, 1 where Ng.; and Ny are respectively the number of data points in source domain i and target domain T, the goal is to learn model that can perform well on the unlabelled target domain. In this paper, we focus on classification problems and denote K as the number of categories.
3.2 Prompt Learning on CLIP-based models
CLIP 27 is a vision- languag model that consists of an image encoder fa and text encoder fr. which is trained to align the visual representation fi()) of image æ with the textual representation
3

---

fe(y) of the corresponding label. The textual representation is derived from a manually crafted prompt p5: in the form "A photo of [CLASS]&= ", where (CLASS): is the class k's name. With great generalization capability, pre-trained CLIP models are often used for variety of downstream tasks through prompt learning.
For zero-shot inference, K class names are forwarded through the text encoder, and the one with the highest representation similarity with the image is the predicted class:
(I) Upred argmax, P(u kk/)), where Ply k//=) Σ (xp(((((),
and (...) measures the cosine similarity and is the temperature.
For fine-tuning, set of learnable class-shared prompts are added to the class token to form Pk [11 [22] ... a |[CLAS]]<> where D; is a vector with the same size as the word embedding, and M is the number of added prompts. These prompts are learnt by maximizing log-likelihood on downstream data, i.e. max E, log P(y = Ys x P). Note that in this predictive probability, we abuse symbol P refer to the learnable tokens va. and when drop the symbol in I we refer the zero-shot prediction using CLIP. As a result, additional information about the downstream task can be encoded in the prompts, and this design will enable knowledge transfer from pre-trained datasets.
4 Proposed method
In this section, we describe our proposed prompt gradient alignment method. Motivated by the lightweight and effective nature of prompt learning in adapting pre-trained knowledge to downstream tasks, we cast UDA as a multi-objective optimization (MOO) problem, from which we propose aligning gradients of different objectives and minimizing their norms simultancously. Additionally, we derive a UDA generalization bound to justify the intuition of our method. The full details of our proposed method in the generalized case where we have more than source domain provided in Appendix B
4.1 Prompt design
A common assumption in domain adaptation literature is that each domain can be represented by domain -specfc features and those that are shared with others. To reflect this, we employ two sets of prompts for each domain: domain-agnon
prompt (or shared prompt, interchangeably) Peh- and -ppeiii prompts Ps., and Pr. Here, Ps., refers to prompt used for source domain i, and Pr is that for target one. In particular, following DAPL, we use K x M1 tokens to construct sh [of| -... . is class-specific shared tokens. For source- and S uu 1 ... | Pr = [u]' I... I. Pr] as the whole prompts used in our method. Based on this, to compute the predictive distribution of source
P s 1K where P
target-specific prompts, we use M2 tokens: Ps.+ = [u And denote P = [Prh, [Ps.) we use prompt of the form i sample belonging to class k; and similarly [ [[r][[[[AAASSS for a target sample.
4.2 Empirical risk minimization: simple baseline
As we introduced, learn those prompts, we consider the cross-entropy losses applied to source data and target data with pseudo labels as a set of objectives to optimize simultaneously:
Liotal ()) = [[Cs,, (P) NT
[[Cs,( Ps.:)] LT (P PT)].
No.1
1
(s Ps.)) = CB(PPt sh, P Ps.; X Ys,s) = Ns.1
log P(y = yزトæi, Pah, Ps.), (2)
jml
CT sh. PT) = CE, (P sh, Pr; XT, YT)
I(P(y 5g( T) log P(y 10 2e, Pah, PT),
(3)
j=1
0 = arg max P(y = k
(4)
&

---

In summary, the total loss consists of N + 1 objectives. The target objective is applied to target samples whose zero-shot predictions made by CLIP are larger than a threshold т.
Given these objectives, source- and target-specific prompts can be updated by minimizing source and target losses, respectively. Regarding domain-agnostic prompt, can put weighting term on the signal from source losses to compute the gradient. Formally, for Vi = 1 + N, we have:
Jsh, 9
V SpL. sh: P PsA)s
I IT VpCT(P.s) PT).
P 3.i
P 8, 195
PT PT 19T
(5)
ssh ssh sh 19 Σ gons).
(6)
where 17 is the learning rate, and X is the weighting term to control how much emphasis we want to put on the target domain. Note that we treat gradient signals from source domains equally as we assume no prior preference knowledge about them. Nevertheless, one can measure the domain similarity between each source and target domain devise a better way to reweight source domains' objectives. However, as will be shown in the experiments, taking the average is simple yet yields superior results, hence we will leave this for future work.
4.3 Prompt gradient aligmment for UDA
For simplicity, we first consider the single-source UDA setting and will present the extension to the multi-source one later in Appendix B One problem with the method above is we ignored the potential inherent gradient conflict between objectives when updating the shared prompt. To mitigate this, one can follow gradient-basee tbaseentt-badddddeee methods, such [35
47
to manipulate the gradients so that conflict
reduced. However, it has been shown in 44 46 that comparable performance can be obtained without such complex manipulations, but with simple re-weighting the loss functions. Therefore, to encourage consensus between these gradients without modifying them, we propose aligning gradients between source and target domains during training. Specifically, similarity, ahh, S s If this goal is achieved, one can expect the shared prompt to capture useful features for classes regardless of domains. Indeed, -9 denotes the direction that moves the shared prompt towards low-loss region of source data, and similar for -hh Hence, when they point to the same direction, i.e., (9 9 > O, updating the shared prompt as in Eq. 6 can reduce loss of both domains, because the aggregated gradient 9*h = >gæn. + I will create acute angles with both s and I.h.T- As result, the shared prompt can learn knowledge that benefits both domains, which is its ultimate goal.
aim maximize their cosine
However, there remain two important questions when implementing this gradient alignment constrain: (i) How incorporate the cosine similarity maximization term regularization in the framework described in Sec. 4.2]; and (ii) How to reduce training time and space when explicitly maximizing it, as it involves the computation of Hessian matrix w.r. the shared prompt? Our method will address these two concerns.
Consider the following loss applied target data with ||-I1 denoting no-m of vector:
PP) LT
aalign
Cr
Jeh.S
Pr)
11g |
(g P.
(P Pr)
Cr sh, PT)
llg. 119.7T1
Lr PT) p(g s 9
where Eq. is obtained by applying first-order Taylor expansion with is a small value, and T is the vector transpose. It can be seen that minimizing this loss also maximizes cosine similarity between
gradients of the two domains. In order to achieve this, let denote lla..s 11
9rh.8
and consider
the loss's gradient w.r.t Pak:
align g shh
= Vp,, CT pa, PT)
d(P ppa)
1 p,,, LT Pr)
d(Psh)
P.A=Pa. Pap-pa
CT PT)
P.s PPa--appa PT)IP.=PA-p
(8)
5

---

In the approximation of Eq. 8 we avoid the Hessian computation by dropping the derivative of a w.r. P Now we can practically apply deep learning optimizers, such as SGD, to minimize LT (P). Specifically, we first compute gradients of the source and target losses w.r.t the shared prompt to get vector then move the current shared prompt to the new stage: sh sh pa. Finally, at this new stage, co-pote the loss on target data then calculate the new gradient.
Caliun (P) on source data and then compute its new gradient w.r. In a similar way, we can derive LS
the shared prompt, i.e. align Given these two new gradients, we can combine them to get the final update direction of the shared prompt, which will navigate it to common low-valued regions in the loss landscapes of both domains.
b
IThTT
align
V. Ps) ss Pon-pb"
1 all.slllllgah...
9
align
align Th.
+ >g sh
align
4.4 Prompt gradient-norm penalization for UDA
So far, we have proposed casting each domain loss as an objective in multiple-objective optimization framework, and have suggested maximizing congruence between gradients of these objectives to reduce their inherent conflict. However, the domain loss in the empirical form, which has been shown to be easily stuck in sharp minima and thus limiting generalization ability [81 [2), especially under distribution shifts [83]. Therefore, we argue that explicit control over the generalization of these prompts can be beneficial. Moreover, inspired by the finding in [59] that gradient
penalization can help model favor flat minima, and by the effectiveness of such minima in the context of multi-task learning (81), we propose minimizing prompt gradient norm of each objective enhance prompt generalization.
By following the same analysis as in Eq. 7 we can seamlessly fuse the gradient norm penalty term with the cosine similarity maximization with the loss below:
CHOA(P P) = CPh Pga |lgh,
J
+ Pgn
IThTT
Pr + Pyn
gT
|grrl
sh,T
Pgo (g T. V P. CT(P
sho P
~ (T(Pshn PT) -
+ Pan Ill + l1
|g [1ghh,
Cr sh. PT) Pga ahh Ioh,T) Pgn (1lg + |1grl)),
where 9T is the gradient of the target loss w.r.t target-specific Pr. and gn stands for gradient norm.
We then follow the derivation of Eq. Nooo come up with practical approximation of the gradient of CPGA P))
PGA 9 T 9T
PGA
V PPCAA (P)
2sh.T VPL sho PT)) P..=Pan PPon-Paaat a lla.k.rl
.Pr=Prt RT lorill
Similarly, we obtain the gradient of the source objective
PGA Sah,5 gs ~ VpLs(P sh: Ps) P.,. Pan PPcb Taak. . Pa=PstPon Twsll
PGA
Rah.5
8
Following the same update rules in Eq. and Eq. 6 the prompts can be leamt to achieve both of our two goals: inter-domain gradient alignment and flat minima enforcement. which can lead improved performance for UDA. We will recap this with generalization bound in the next part, and provide details for the final loss function in Appendix
4.5 Theoretical Analysis of PGA
We informally present information-theont
bound to explain why PGA works. Refer to Appendix for the formal version. For simplicity, we will consider the single-source UDA setting and abuse N as the number of source samples. Let Z, P be the input-label space and prompt space (or hypothesis

---

space), respectively. Assume the loss function P 2 - j is R-subgaussian Denote μ. f as the two underlying distributions from which the source and target data is sampled, and K (.Il)) the KL-diivegeence. The generalization erro defined as the difference between the target population loss and the source empirical loss
Err = Dy RDs (P) - Ep. B. (
E (P.
N
Theorem 4.1. Under the assumption R-subgaussianity, the generalization error can be upper- bounded by:
4R T. P. Dr [19 |1 llg 9: 11 V (R=KL()))
Errl S
N
where T is the total number of training iterations, tie is the learning rate at iteration scaled by tgt VpLr(Pt-1)) are the gradients w.rl Pt-1 of source loss Eq. scalar, gi = (P-t)P- 9t
2 and larget loss Eq where Pt the prompt iteration
As our method tries to minimize source empirical loss, gradient norms and gradient mis alignment, from the first term in the R.H.S of Eq. 4.1 its benefit to the performance target domain can be justified. Furthermore, the second term shows that the generalization error can be reduced by bridging the gap between the two domain distributions, which is the of many UDA methods, such as [70 [8). However, as stated earlier, our work orthogonal to this line of method as we do explicitly attempt close such gap. Hence, interesting future development could be taking the second term into account, Refer to Appendix A.5 for more discussion about this bound.
Experiments
In this section, evaluate the efficacy of proposed method different UDA benchmarks, following the same protocol of recent prompt-based UDA studies [25] 28 Before that, we start with simple maltii-bjeetiveeoptiiizzze method compared to conventional empirical risk minimization (ERM).
setup derive insights into the effectiveness of our proposed
5.1 Illustrative example
Let y E {-1.1} be the true label, be the environmental feature and e be Gaussian noise, x E 3:00 and p (0, 1), C be predefined scalar constants. The data-generating process is given by:
Pp(e y 1 v) Pp(e = y = y) = (1-p) e~ N C~, (2.5.%) x [C CC
~~ 1x-1(111} e
ERM
Alignment
PGA
In-domain performance
Out-of- adt-of-domain performance
100
90
80
80
Acursty
70
70
60
30
70
10
Epochs
Epochs
Figure 2: Performance of ERM and PGA on the in-domain data (validation set) and cittof-diitribbution data (test set). Average results and shaded standard errors are obtained from 10 random seeds.
The environmental feature e correlates with the true label y according to p. Similar to [5 85] set p 0.9 for the training and validation set (in-distribution) and 0.1 for the test set
random variable X is R-subgaussian for any P. Eeg(eex(p(X EX)) R2/2 Refer to the Appendix why the expectation taken over P, Ds. Dr.
7

---

Table Accuracy (%) on ImageCLEF and Office-Home, We use bold to denote the best method overall and underscore to denote the best method using source combined. Overall, PGA and MPGA consistently obtain the best results among source combined and multi-source scenarios, respectively.
ImageCLEF
Office-Home
C
+1
+ P
Avg
Ar + CI Pr
Rw
Avg
Zero-Shot
CLIP
87.9
88.2
78.7
88.1
71.2
50.4
81.4
82.6
71.4
Source Combined
DAN 19
93.3
92.2
77.6
87.7
68.5
59.4
79.0
82.5
72.4
DANN 18
93.7
91.8
77.9
87.8
68.4
59.1
79.5
82.7
72.4
D-CORAL 69
93.6
91.7
77.1
87.5
68.1
58.6
79.5
82.7
72.2
DAPL 25
96.0
89.2
76.0
87.1
72.8
51.9
82.6
83.7
72.8
Simple Prompt
28
93.6
90.6
80.9
88.4
70.7
52.9
82.9
83.9
72.4
PGA (Ours)
96.8
95.7
84.6
92.4
75.2
59.7
86.2
86.2
76.8
Multi-Source
DCTN 888
95.7
90.3
75.0
87.0
N.A.
N.A.
N.A.
N.A.
N.A.
MDDA 89
N.A.
N.A.
N.A.
N.A.
66.7
62.3
79.5
79.6
71.0
SIMpIDA 96
93.3
91.0
77.5
87.3
70.8
56.3
80.2
81.5
72.2
MFSAN 62
95,4
93.6
79.1
89.4
72.1
62.0
80.3
81.8
74.1
MPA 28
97.2
96.2
80.4
91.3
74.8
54.9
86.2
85.7
75.4
MPGA (Ours)
97.4
96.5
84.7
92.9
76.3
63.8
90.0
87.4
79.4
(out-of-distributt Figure presents the performance of three linear classifiers trained by ERM, gradient alignment method only and PGA. In summary, while ERM learns non-predictive features and fails generalize beyond in-distribution data, gradient alignment algorithm leverage the gradient information from multiple environments to learn the core feature that helps perform well on OOD data. Besides, incorporating the gradient norm penalty term further enhances stability and robustness at convergence.
5.2 Experimental setup
Datasets. We conduct experiments using three well-established UDA datasets of varying scales: ImageCLEF 17), Office-Home [86), and DomainNet 87 respectively. Detailed descriptions of these datasets available in Appendix C.I
Metrics. We evaluate our model by reporting the top-1 accuracy for each target domain and average accuracy across all domains. To further validate the effectiveness of our proposed method, we conduct experiments in two distinct settings: source-combined setting, where data from source domains merged, and multi-source setting, which utilizes individual domain identifications. Additionally, we provide pair-wise single-source domain adaptation results for the Office-Home dataset.
Baselines. Regarding prompt-based baselines, we compare our method with MPA 28 DAPL Simple Prompt (28), and Zero-shot CLIP [27]. To ensure comprehensive evaluation, we also include comparisons with various non- con-prompt methods such as DCTN 88). MDDA [8), MFSAN 622.
T-SVDNet 190 and PFSA [91 As we follow the same settings as in 28 and 25 results for
baselines are taken from those studies for consistency. Note that while DAPL 25 MPA 28 and our methods employ CoOp [92] with text-end soft-prompt, other methods finetune the transformer block [63] or both image and text-end soft-prompts [93] or the whole encoders 944 955. Since those methods typically fine-tune many more parameters, thus do not include them in the experimental results for the sake of fair comparison.
5.3 Experimental results
Table presents the results for the ImageCLEF and Office-Home datasets. Under the source-combined scenario, PGA significantly outperforms nearly other baseline methods on both datasets, with the exception of its own multi-source variant, MPGA. For instance, PGA surpasses the second-

---

best source combined method by notable 4% in average accuracy and exceeds MPA by over 1%. Notably, in the Office-Home domain Clipart, while two prompt baselines, DAPL and Simple Prompt, lag behind their non-prompt counterparts, PGA still manages to achieve slightly better results than these non-prompt methods. In the multi-source setting, MPGA consistently delivers the highest performance across all domains, notably outperforming MPA, the state-of-the-an (SOTA) prompt-based method for multi-source UDA, by a substantial margin 4% on Office-Home.
Table 3: Accuracy (%) on NemainNet. We use bold to denote the best method overall and underscore denote the best method using source combine.
DomainNet
Clp
Inf
Pnt
Qdr
Rel
Skt
Avg
Zero-Shot
CLIP
227
61.3
42.0
56.1
10.3
79.3
54.1
50.5
Source Combined
DANN 18
45.5
13.1
37.0
13.2
48.9
31.8
32.6
MCD
54.3
22.1
45.7
7.6
58.4
43.5
38.5
DAPL 25
62.4
43.8
59.3
10.6
81.5
54.6
52.0
Simple Prompt
63.1
41.2
57.7
10.0
75.8
55.8
50.6
PGA (Ours)
66.3
49.2
63.3
11.1
81.8
60.6
55,4
Multi-Souree
M:SDA-3 98
58.6
26.0
52.3
6.3
62.7
49.5
42,6
SImpA1101
96
66.4
26.5
56.6
18.9
68.0
55.5
48.6
LIC-MSDA
63.1
28.7
56.1
16.3
66.1
53.8
47.4
T-SVDNet 90
66.1
25.0
54.3
16.5
65.4
54.6
47.0
PFSA [91
64.5
29.2
57.6
17.2
67.2
55.1
48.5
PTMDA [000]
66.0
28.5
58.4
13.0
63.0
54.1
47.2
MPA (28
65.2
47.3
62.0
10.2
82.0
57.9
54.1
MPGA (Ours)
67.9
50.5
63.8
11.6
82.2
61.0
56.2
On DomainNet, Table presents, method still obtains superior average accuracy under both source combined and multi-source higher than the runner-up by 3.4% and 2.1 %, respectively. Overall, in the domain where CLIP brings significant results compared with non non-prompt baselines, leads to better performance, except for the difficult QuickDraw domain, as remarked by a relatively low zero-shot accuracy for CLIP-based methods, where it seems that prompt learning fails to beat non-prompt counterparts. Even though, both PGA and MPGA still outperform other prompt-baa based counterparts while fine-tuning fewer parameters (e.g. 500k versus 2M of MPA).
method
addition, we also demonstrate our method's effectiveness under 12 pair-wise source- target settings on Office-Home in Table[4] Again, PGA acquires the highest average DAPL under settings while using the same parameter-efficee www.nntte-effiiint--te seneciient-iintunnngg method [92].
and consistently beats
Table 4: Accuracy (%) on Office-Home) [01] for unsupervised domain adaptation (ResNet-
The best accuracy indicated in bold.
Method
Ar-CCI
Ar-Pt
Ar-Rw
Cl-Ar
Cl-PPr
Cl-RR
Pr-AAr Pr-ECl
Pr-RW
Ru-tAr Rww+CC
Rw-1th
76.8
JAN
45.9
61.2
68.9
%0.
39.7
61.0
45.8
43.4
70.3
63.9
52.4
76.8
58.3
CDAN-E [1]
50.7
70.5
76.0
57.6
70,0
70,0
S7.4
50.9
77.3
70.9
56.7
81.6
65.8
RSP4CDAN [03)]
52.0
68.6
76.1
5K0
703
70.2
58.6
50.2
77.6
72.2
59.3
81.9
66.3
SymnNes T04
79.4
73.8
73.9
749
49.5
79.7
70.5
$3.6
82.2
ESDA 1077
61.3
76.1
79.4
55.4
71.
74.3
650
53.2
80.0
72.2
6066
89.1
70.3
GVB- GVB-GD [08
$7.0
78.7
79.8
51.6
T4.1
7466
65.2
55.1
81.0
74,8
39.7
81.3
70.1
RSDA-MSTN (b9
53.2
77.7
B1.3
66.4
74.0
76.5
67.9
$3.0
82/0
75.8
$7.8
85.4
70.9
[11
DisCluseer
58.8
77.0
80.8
74.6
77.1
B1,4
74.2
60.5
83.6
71.4
CLIP
X2.6
75.2
PGA (Ours)
85.8
86.1
75.4
$6.7
85.8

---

5.4 Ablation study
From Table we can see that (i) learning prompts using solely the target loss, the aceuracy across all settings already surpasses that of Zero-shot CLIP. This confirms the reliability of pseudo labels generated by CLIP. (ii) When adding source loss and grad-norm penalization, the results improve slightly. (iii) Importantly, adding gradient alignment, the scores increase more clearly. These observations verify each of our contributions.
Cosine similer by belwwen gradients
LT
Ls
GN
GA
C
P
Avg
Aunt
X
87.9
88.2
78.7
X
92.9
94.3
83.2
90.1
X 93.3
95.0
83.3
90.6
X 94.3
95.3
90.9
32
96.8
95.7
84.6
92.4
Table 5: Ablation studies on various modules of PGA on the ImageCLEF, Each of the proposed components shows its effectiveness while combin- Figure 3: Evolution of the gradient similarity ing them helps obtain the best performance.
during training.
Furthermore, to show that gradient alignment indeed increases consensus between gradients, we plot cosine similarity along the training process with three different values of Pga in Figure 3 First, during early training stages, there seems to be less agreement between gradients when no alignment is enforced, c.f. Pga = O. When Pao 0, we can see the similarity increase. Noticeably, there exists a point where similarity starts plummeting. This reasonable when the model starts to converge to Pareto solution where source and target gradients cancel each other. This is depicted more clearly Figure 4 in the appendix where the closer the model is to the Pareto front, the more confict the gradients
6 Conclusion
In this work, we have proposed a framework for UDA inspired by Multi-objective optimization thanks the generalizability of CLIP and the lightweight nature of prompt learning. We have then devised a practical method to align per-objective gradients, which aims to encourage inherent consensus between objectives. We have further fused gradient norm penalization the method enhance prompt generalization. Finally, a UDA generalization bound is presented to justify the benefits of our method.
Acknowledgemeets:: Trung Le was supported by ARC DP23 grant DP230101176 and by the Air Force Office of Scientific Research under award number FA2386-23-1-4044,
10

---

References
[1] Siddharth Srivastava and Gaurav Sharma. Omnivec: Learning robust representations with cross modal sharing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages
2024.
[2] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[3] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korbblihh, al. Model soups: averaging weights of multiple fine-tuned models improves aceuracy without increasing inference time. In International conference on machine learning, pages 23965-23998, 23998. PMLR, 2022.
[4] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. In The Eleventh International Conference on Learning Representations, 2022.
[5] Zihang Dai, Hanxiao Liu, Quoc Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in neural information processing systems, 34:3965-3977, 2021.
[6] Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. In Proceedings of the IEEF/CVF international conference on computer vision, pages 6748-6758. 2023.
[7] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14408-14419, 2023.
[8] Weijie Su, Xizhou Zhu, Chenxin Tao, Lewei Lu, Bin Li, Gao Huang, Yu Qiao, Xiaogang Wang, Jie Zhou, and Jifeng Dai. Towards all-in-one pre-training via maximizing multi-modal mutual information. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15888-15899, 2023.
[9] Kemal Oksuz, Selim Kuzucu, Tom Joy, and Puncet K Dokania. Mocae: Mixture of calibrated experts significantly improves object detection. arXiv preprint arXiv:2309.14:: 14976, 2023.
[10] Xudong Wang, Shufan Li, Konstantinos Kallidromitis, Yusuke Kato, Kazuki Kozuka, and Trevor Darrell. Hicrarchical open-vocabulary universal image segmentation. Advances in Neural Information Processing Systems, 36, 2024.
[11] Peng Wang, Shijie Wang, Junyang Lin, Shuai Bai, Xiaohuan Zhou, Jingren Zhou, Xinggang Wang, and Chang Zhou. One-peace: Exploring general representation model toward unlimited modalities. arXiv preprint 3rXiv:2305.11172 2023.
[12] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
[13] Serdar Erisen. Sernet-former: Semantic segmentation by efficient residual network with attention-boosting gates and attention-fusion networks. arXiv preprint 15X1v 2024.
[14] Michael Lones. How to avoid machine learning pitfalls: guide for academic researchers. arXiv preprint arXiv:2108.02443 2021.

---

[15] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips. Irena Gao, al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, pages 5637-5664. PMLR, 2021.
[16] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2019.
[17]
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adaptation networks. In ICML, pages 2208 2000-2217,22 2017.
[18] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Nomain-adverssan training of neural networks. Journal of machine learning research, 17(59):1-35, 2016.
[19] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, pages 1 1180-1189, 2015.
[20] Mingsheng Long, Yue Cao, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Transferable representation learning with deep adaptation networks. JEEE transactions pattern analysis and machine intelligence,
(112):3771-085, 2018.
[21] Boxiao Pan, Zhangjie Cao, Ehsan Adeli, and Juan Carlos Niebles, Adversarial cross-domain action recognition with co-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1181211122, 2020.
[22] Zhongyi Han, Xian-Jin Gui, Chaoran Cui, and Yilong Yin. Towards accurate and robust domain adaptation under noisy environments. In Proceedings the Twenty-Ninth International Conference International Joint Conferences on Artificial Intelligence, pages 2276, 2021.
[23] Jinyu Yang, Chunyuan Li, Weizhi An, Hehuan Ma, Yuzhi Guo, Yu Rong, Peilin Zhao, and Junzhou Huang. Exploring robustness of unsupervised domain adaptation in semantic segmen- tation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9194-9203, 2021.
[24] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2020.
[25] Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[26] Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering. In CVPR, pages 8725-8735, 2020.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, volume 139, pages 8748-8763, 2021.
[28] Haoran Chen, Zuxuan Wu, and Yu-Gang Jiang. Multi-prompt alignment for multi-source unsupervised domain adaptation. Neural Information Processing Systems, 2022.
[29] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979 2018
[30] Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. In International Conference on Learning Representations, 2018.
[31] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
12

---

[32] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. Advances in neural information processing systems, 32, 2019.
[33] Michinari Momma, Chaosheng Dong, and Jia Liu. A multi-objjectiv/multittaaa learning framework induced by pareto stationarity. In International Conference on Machine Learning, pages 15895-15907. PMLR, 2022.
[34] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836 2020.
[35] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 14:18878-18890, 2021.
[36] Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In International Conference Learning Representations, 2021.
[37] Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039-20550 2020.
[38] Xiaobo Xia, Jiale Liu, Shaokun Zhang, Qingyun Wu, and Tongliang Liu. Coreset selection with prioritized multiple objectives. arXiv preprint arXiv:231 11222311.000 1.08675, 2023.
[39] Yuru Song, Zan Lou, Shan You, Erkun Yang, Fei Wang, Chen Qian, Changshui Zhang, and Xiaogang Wang. Learning with privileged tasks. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 10685-10694, 2021.
[40] Aviv Shamsian, Aviv Navon, Neta Glazer, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Auxiliary learning as asymmetric bargaining game. In International Conference on Machine Learning, pages 30689-30705 PMLR, 2023.
[41] Pingchuan Ma, Tao Du, and Wojeiech Matusik. Efficient continuous pareto exploration in multi-task learning. In International Conference on Machine Learning, pages 6522-6531. PMLR, 2020.
[42] Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik. Learning the pareto front with hypernetworks. In International Conference on Learning Representations, 2020.
[43] Hoang Phan, Andrew Gordon Wilson, and Qi Lei. Controllable prompt tuning for balancing group distributional robustness. In Forty-first International Conference on Machine Learning, 2024.
[44] Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov, Shimon Whiteson, and Pawan K Mudigonda. In defense of the unitary scalarization for deep multi-task learning, Advances in 2022.
Neural Information Processing Systems,
[45] Derrick Xin, Behrooz Ghorbani, Justin Gilmer, Ankush Garg, and Orhan Firat. Do current multi-task optimization methods in deep learning even help? Advances in neural information processing systems,
2022.
[46] Yuzheng Hu, Ruicheng Xian, Qilong Wu, Qiuling Fan, Lang Yin, and Han Zhao. Revisiting scalarization in multi-task learning: A theoretical perspective. Advances in Neural Information Processing Systems, 36, 2024.
[47] Shiji Zhou, Wenpeng Zhang, Jiyan Jiang, Wenliang Zhong, Jinjie Gu, and Wenwu Zhu. On the convergence of stochastic multi-objective gradient manipulation and beyond. Advances in Neural Information Processing Systems, 15:38103-38114 2022.
[48] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations, 2019.
13

---

[49] Aliakbar Panahi, Seyran Saeedi, and Tom Arodz. Shapeshifter: a parameter-efficient trans- former using factorized reshaped matrices. Advances in Neural Information Processing Systems, 34:1337-1350, 2021.
[50] Bonan Li, Yinhan Hu, Xuecheng Nie, Congying Han, Xiangjian Jiang, Tiande Guo, and Luoqi Liu. Dropkey for vision transformer. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 22700-22709, 2023.
[51] Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. Evaluating clip: towards characterization of broader capabilities and downstream implications. arXiv preprint wrXiv:21080028111 2021.
[52] Jialu Wang, Yang Liu, and Xin Wang. Are neutral queries really gender-neutral? mitigating gender bias in image search. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1995-2008, 2021.
[53] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi. Yue Gao, and Guogi Li. Learning to prompt for open- vecabuaayy object detection with languala model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14084-14093, 2022.
[54] Michael Zhang and Christopher Ré. Contrastive adapters for foundation model group robust- ness. Advances in Neural Information Processing Systems, 11-2216822269999 2022.
[55] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning, pages 8346-8356. PMLR, 2020.
[56] Lifu Tu, Caiming Xiong, and Yingbo Zhou Prompt-tuning can be much better than fine- tuning on cross-lingual understanding with multilingual language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 8478-5485 2022.
[57] Cheng-En Wu, Yu Tian, Haichao Yu, Heng Wang. Pedro Morgado, Yu Hen Hu, and Linjie Yang. Why is prompt tuning for vision-language models robust to noisy labels? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15488-15497, 2023.
[58) Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 16816-16825, 2022.
[59] Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. arXiv preprint arXiv: 2209.03599, 2022.
[60] Devansh Bisla, Jing Wang, and Anna Choromanska. Low-pass filtering sgd for recovering flat optima in the deep learning optimization landscape. In International Conference on Artificial Intelligence and Statistics, pages 8299 8339. PMLR, 2022.
[61] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances in neural information processing systems, 33:2958-2966, 2020.
[62] Yongchun Zhu, Fuzhen Zhuang, and Deqing Wang. Aligning domain distribution and classifier for cross-domain classification from multiple sources. In The Thirty-Thie Third AAAI Conference on Artificial Intelligence. AAAI 2019, The Thirty-Firs Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February 1, 2019, pages 5989-5996. AAAI Press, 2019.
[63] Zhekai Du, Xinyao Li, Fengling I.,, Ke Lu, Lei Zhu, and Jingjing Li. Domain-agnostic mutual prompting for unsupervised domain adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, 2024.
[64] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. theory of learning from different domains. Mach. Learn., 79(1- 1):151-175, 2010.
14

---

[65] Han Zhao, Shanghang Zhang, Guanhang Wu, José M. F. Moura, Joao P Costeira, and Geof- frey J Gordon. Adversarial multiple source domain adaptation. In S. Bengio, H. Wallach. H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[66] Han Zhao, Shanghang Zhang, Guanhang Wu, Jo ao P. Costeira, José M. F, Moura, and Geoffrey J. Gordon. Multiple source domain adaptation with adversarial learning, 2018.
[67] Hoang Phan, Trung Le, Trung Phung. Anh Tuan Bui, Nhat Ho, and Dinh Phung. Global- local regularization via distributional robustness. In International Conference on Artificial Intelligence and Statistics, pages 7644-7664. PMLR, 2023.
[68] Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Scholkopf, and Alexander J. Smola. A kernel method for the two-sample problem. arXiv preprint arXiv: 0805.2368, 2008.
[69] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation.
In ECCV, pages 443 450. Springer, 2016.
[70] Trung Quoe Phung, Trung Le, Long Tung Vuong, Toan Tran, Anh Tuan Tran, Hung Bui, and Dinh Phung. On learning domain-invariant representations for transfer learning with multiple sources. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[71] Mingsheng Long, Zhangjic Cao, Jianmin Wang, and Michael I. Jordan. Conditional adversarial domain adaptation. In NeurlPS, pages 1647-1657, 2018.
[72] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 2697-12706. PMLR, 2021.
[73] Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competition: Why the highest probability answer isn't 't always right. Proceedings of the 202 Conference on Empirical Methods in Natural Language Processing, pages 7038-7051, 2021.
[74] Simran Arora. Avanika Narayan, Mayee Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re. Ask me anything: A simple strategy for prompting language models. In The Eleventh International Conference on Learning Representations, 2022.
[75] Hoang Phan, Ngoc Tran, Trung Le, Toan Tran, Nhat Ho, and Dinh Phung. Stochastic multiple target sampling gradient descent. Advances in neural information processing systems, 35:22643-22655, 2022.
[76] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference Learning Representations, 2019.
[77] David Lopez-Paz. and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Neural Information Processing Systems, 2017.
[78] Arslan Chaudhry, Mare' Aurelio Ranzato. Mareus Rohrbach, and Mohamed Elhoseiny. Effi- cient lifelong learning with a- a-GEM. In International Conference on Learning Representations, 2019.
[79] Yuge Shi, Jeffrey Seely, Philip Torr, Siddharth N, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In International Conference on Learning Representations, 2022.
[80] Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang, Sharpness-awar gradient matching for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3769-3778. 2023.
[81] Hoang Phan, Lam Tran, Ngoc N. Tran, Nhat Ho, Dinh Phung, and Trung Le. Improving multi-task learning via seeking task-based flat regions. arXiv preprint arXiv: 2211. 137223 2022.
15

---

[82] Yaowei Zheng, Richong Zhang, and Yongyi Mao. Regularizing neural networks via adversarial model perturbation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8156-8165, 2021.
[83] Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, Arihant Jain, and R. Venkatesh Babu. A closer look at smoothness in domain adversarial training. International Conference Machine Learning, 2022.
[84] Shuang Li, Chi Liu, Qiuxia Lin, Binhui Xie. Zhengming Ding, Gao Huang, and Jian Tang. Domain conditioned adaptation network. In AAAI, volume 34, pages 11386-11393, 2020.
[85] Aahlad Manas Puli, Lily Zhang, Yoav Wald, and Rajesh Ranganath. Don't blame dataset shift! shortcut learning due to gradients and cross entropy. Advances in Neural Information Processing Systems, 36:71874-71910, 2023.
[86] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 5018-5027, 2017.
[87] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference compuler vision, pages 1 1406-1415, 2019.
[88] Ruijia Xu, Ziliang Chen, W. Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Multi- source unsupervised domain adaptation with category shift. IEEF/CVF Conference on Com- puter Vision and Pattern Recognition, 2018.
[89]
Sicheng Zhao, Guangzhi Wang. Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-source distilling domain adaptation. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secone Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 12975-12983, AAAI Press, 2020.
[90] Ruihuang Li, Xu Jia, Jianzhong He, Shuaijun Chen, and Qinghua Hu. T-svdnet: Exploring high-order prototypical correlations for multi-source domain adaptation. ICCV, 2021.
[91] Yangye Fu, Ming Zhang, Xing Xu, Zuo Cao, Chao Ma, Yanli Ji, Kai Zuo, and Huimin Lu. Partial feature selection and alignment for multi-source domain adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16654-16663, June 2021.
[92] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lision-language models. International Journal of Computer Vision, 130(9))2337-2348, 2022.
[93] Zhengfeng Lai, Haoping Bai, Haotian Zhang, Xianzhi Du, Jiulong Shan, Yinfei Yang. Chen- Nee Chuah, and Meng Cao. Empowering unsupervised domain adaptation with large-scale pre-trained vision-language models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 29011, 2024.
[94] Zhengfeng Lai, Noranart Vesdapunt, Ning Zhou, Jun Wu, Cong Phuoc Huynh, Xuelu Li, Kah Kuen Fu, and Chen-Nee Chuah. Padclip: Pseudo-labeling with adaptive debiasing in clip for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16155-16165, 2023.
[95] Wenlve Zhou and Zhiheng Zhou. Unsupervised domain adaption harnessing vision-language pre-training, IEEE Transactions on Circuits and Systems for Video Technology, 2024.
[96] Naveen Venkat, Jogendra Nath Kundu, D. K. Singh, Ambareesh Revanur, and R. Venkatesh- Babu. Your classifier can secretly suffice multi-souree domain adaptation. Neural Information Processing Systems, 2021.
16

---

[97] Kuniaki Saito, Kohei Watanabe, Y. Ushiku, and T. Harada. Maximum classifier discrepancy for unsupervised domain adaptation. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
[98] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In 2019 IEEF/CVE International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 -Noemmerr 2, 2019, pages 1406-1415, IEEE, 2019.
[99] Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang. Learning to combine: Knowledge aggregation for multi-source domain adaptation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII, volume 12353 of Lecture Notes in Computer Science, pages 727-744. Springer, 2020.
[100] Chuan-Xian Ren, Yong Liu, Xiwen Zhang, and Ke-Kun Huang. Multi-source unsupervised domain adaptation via pseudo target domain. IEEE Transactions Image Processing, 2022.
[101] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, pages 5385-5394, 2017.
[102] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[103] Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability discrim- inability: Batch spectral penalization for adversarial domain adaptation. In ICML, volume 97, pages 1081-1090, 2019.
[104] Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan. Domain-symmetric networks for adversarial domain adaptation. In CVPR, pages 5031-5040, 2019.
[105] Mengxue Li, Yiming Zhai, You-Wei Luo, Pengfei Ge, and Chuan-Xian Ren. Enhanced transport distance for unsupervised domain adaptation. In CVPR, 2020.
[106] Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In CVPR, pages 3940-3949, 2020.
[107] Lanqing Hu, Meina Kan, Shiguang Shan, and Xilin Chen. Unsupervised domain adaptation with hierarchical gradient synchronization. In CVPR, pages 4043-4052, 2020.
[108] Shuhao Cui, Shuhui Wang, Junbao Zhuo, Chi Su, Qingming Huang, and Qi Tian. Gradually vanishing bridge for adversarial domain adaptation. In CVPR, pages 1-1544, 2020.
[109] Xiang Gu, Jian Sun, and Zongben Xu. Spherical space domain adaptation with robust pseudo- label loss. In CVPR, pages 100 2020.
[110] Qian Wang and Toby P. Breckon. Unsupervised domain adaptation via structured prediction based selective pscudo-labeling. In AAAI, pages 6243-6250, 2020.
[111] Hui Tang, Yaowei Wang, and Kui Jia. Unsupervised domain adaptation via distilled discrimi- native clustering. Pattern Recognition, 127:108638, 2022.
[112] Ziqiao Wang and Yongyi Mao. Informatioo--theortticc analysis of unsupervised domain adapta- tion. In The Eleventh International Conference on Learning Representations, 2023.
[113] Gergely Neu. Information -toommatin-ttheortiie generalization bounds for stochastic gradient descent. In Annual Conference Computational Learning Theory, 2021.
[114] Jonas Geiping, Micah Goldblum, Phillip Pope, Michael Moeller, and Tom Goldstein. Stochas- tic training is not necessary for generalization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
17

---

[115] Lingfeng Shen. Weiting Tan, Boyuan Zheng, and Daniel Khashabi. Flatness-ave aware prompt selection improves accuracy and sample efficiency. Conference on Empirical Methods in Natural Language Processing, 2023.
[116] Liangchen Liu, Nannan Wang, Dawei Zhou, Xinbo Gao, Decheng Liu, Xi Yang, and Tongliang Liu. Gradient constrained sharpness-awarr aware prompt learning for vision- visiunelanguage models, 2024.
[117] Ziqiao Wang and Yongyi Mao. Two facets of sde under an cntommation-theoreee lens: Gen- eralization of sgd via training trajectories and via terminal states. arXiv preprint arXiv: 221 2261,1 2022.
[118] Yury Polyanskiy and Yihong Wu. Lecture notes on information theory. Technical report, 2019.
[119] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012. 13255, 2020.
[120] Eckart Zitzler, Kalyanmoy Deb, and Lothar Thiele. Comparison of multiobjective evolutionary algorithms: Empirical results. Evolutionary computation, 8(2):173-195, 2000.
[121] Hui Tang and Kui Jia. A new benchmark: On the utility of synthetic data with blender for bare supervised learning and downstream domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15954-15964, 2023.
[122] Zhenbin Wang, Lei Zhang, Lituan Wang, and Minjuan Zhu. Landa: Language-guided multi- source domain adaptation. arXiv preprint arXiv:2401.14148 2024.
[123] Mainak Singha, Harsh Pal, Ankit Jha, and Biplab Banerjee. Ad-clip: Adapting domains in prompt space using clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4364, 2023.
[124] Korawat Tanwisuth, Xinjie Fan, Huangjie Zheng, Shujian Zhang, Hao Zhang, Bo Chen, and Mingyuan Zhou, A prototype-oriented framework for unsupervised domain adaptation. Advances in Neural Information Processing Systems, 34:17194- 2021.
[125] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. International Conference on Computer Vision, 2023.
[126] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta- learning for domain generalization. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Seconl AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelligence (FAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 3490-3497. AAAI Press, 2018.
[127] Da Li, Yongxin Yang, Yi-Zhe Song. and Timothy M. Hospedales. Sequential learning for domain generalization. ECCV Workshops, 2020.
[128] Juncheng Li, Minghe Gao, Longhui Wei, Siliang Tang, Wenqiao Zhang, Meng Li, Wei Ji, Qi Tian, Tat seng Chua, and Yueting Zhuang, Gradient-regulated meta-prompt learning for generalizable vision-language models. IEEE International Conference Computer Vision, 2023.
18

---

Supplement to "Enhancing Domain Adaptation through Prompt Gradient Alignment*
Due to space constraints, some details were omitted from the main paper. We therefore include additional theoretical developments (section [A], the detailed algorithm description (section B and experimental results (section Cp in this appendix.
A UDA generalization bound
Here, we provide an information- -hteoretc generalization bound for UDA, which can reduced by our gradient alignment and gradient norm penalization. For simplicity, we will consider the single-source UDA setting.
To begin, we first define some additional notations: let X V/ P be the input space, output space, and prompt space (or hypothesis space), respectively. Denote the input-label space as X' V, and the loss function as & P x 2 + R (Cross entropy in our case). Finally, denote μr, p² as the two underlying distributions from which the source and target domains are sampled. The training data for source domain D5 { is drawn i.i.d from ON and that for target domain Dr = (x) M
from
@M xxx".
For each prompt parameter, the population risk in the target domain is defined as
Ry (P) = Ez u [e(
This risk is the ultimate goal that UDA algorithm aims to minimize. However, since μí' is unknown, and only a finite number of training data is given, we define the empirical risk in the source domain as
N
RD P)) = ((P
(10)
In the iiformationtthorette analysis framework, model parameter, P P in our case, is a random variable that is outputted from learning algorithm A characterized by some conditional distribution
Pr.Ds.Dr Then the generalization error, measuring how close these two risks can be, has the form
Err = Ep.DD. Ry (P) - RDs (P)),
(11)
where the expectation is taken over P, Ds, Dr ~ PpIDs.Dr / S P
'
To derive the bound, need the following assumption the loss function, which is commonly adopted in many cntematiio--heoretii bounds such as those in [112
Assumption A.1. (Subgaussianity). (() is R-subgaussian under Pr.z'/x/=a), Vay E X, for any P P.
We also present the definitions of Mutual Information, Disintegrated Mutual Information, and Conditional Mutual Information:
Definition A.2. (Mutual Information). I(; = (L ||x @ Py), where KL is the KL- divergence and @ denote the product of marginal distributions.
Definition A.3. (Disintegrated Mutual Information). The disintegrated mutual information between two random variables and given a realization a variable Z is
1 (X;Y) = KL KL((Xx.12==:II2x12== @ Py z=z)
Definition A.4. (Conditional Mutual Information). [(X,Y 12
"A random variable X is R-subgaussian if for any p. log Eexp(p(X EX)) 2p² 12122
19

---

Theorem A.5. Under assumption A.1
the generalization enror can be upper- bounded by
4R T
|Err|
1/1 oi
Ep
P P
0 + +1 1|2 "
11 9:.0132 /2R2KL(ul()")
N
(12)
where T is the total number of training iterations, Ttt is the learning rate at iteration Pt is the Agt P (Pt-a) are the gradients w.E. P prompt at iteration t, gt P P 1t-1), 9t of source loss En[ target loss Eq and oe is the standard deviation of the isotropic Gaussian noise added to the update of P..
Remark A.6. For the purpose of simplicity, here we consider 'noisy' update version of prompts: P. P "/g + N,, N₁ ~ NN, However, note that the bound still holds for the conventional SGD update, i.e., no added noise, by following techniques in (113).
Remark A.7. Our methods align gradients of Shared-prompe but here we can omil its subscript in 196 1|2 by noting that gi = =gg, sroc gi 0] and tgt ||2 = | 2 +
the inter-domain gradient matching term, l|g
igt = gt. sh.ttt 0, ) Indeed. !|g, 9t
2gt 1|2 2g ) g:
sh.tgt where T
denotes the vector transpose. In addition, this bound suggests maximizing the dot product between gradients; however to stabilize training, we aim maxâmize the cosine similarity instead.
This theorem suggests that penalizing gradient norm and matching gradients across domains can improve generalization on the target domain, i.e., the first term in the R.H.S of [f.5 is minimized. Note that minimizing gradient norm has been widely used in [59 81 [14 to control the sharpness of the loss landscape, which is strongly related to the generalization capability of the model. In this work,
empirically and theoretically verify the effectiveness of this technique in the gradient space of prompt, consistent with results previous works [15 [16].
Regarding the second term, we do not aim for method that can explicitly reduce the gap between source and target distributions, because we do not want to remove any domain-specific features that may be helpful for prediction. Instead, we want to capture domain-agnostic features in the shared prompt, and specific features in the domain domain-specific ones so that at inference, a more meaningful representation can be obtained by using these prompts. Hence, one possible direction for future work to design and learn prompts such that domain distribution alignment can also be achieved.
Finally, this bound can grow the number of training iterations increases unless gradient norms and the difference between source and target gradients are extremely small at final iterations. Future work could be overcoming this limitation by considering other bounds, such as ones suggested in [117).
Proof. Our bound inspired from the bound in Theorem 5.1 following lemma
112 which is restated as the
Lemma A.8. Under assumption A.l
the generalization error can be upper-bounded by
M N
Errl S
E Ex
+ (L
(13)
NM
2R² D(P:Ds (222KL)ll
(14)
Now consider the 'noisy' update of the prompt as presented in Eqs. and
P = Pt- 1/1 ( p Corel P.-1) + P Pt-1)) + N₂
(15)
P 7.9.""
T..." tgt + Ni.
(16)
Assume that we obtain the final prompts after T iterations, then following the chain rule of mutual information and data processing incquality, we have
20

---

(PT:Ds|DD) 1(PT-1 T TTT NT; Dr)
(17)
(I(--- - 91
nre 78r6 + Nri Dsi:s
(18)
1[P 1( -7 7g igt + Ns:
PT
(19)
(20)
(11 er nigtt lgt + NNg Dr, Pt-1))
(21)
I(-9i
gi
are
gt
NN/m: Ds Dr, Pt-1)
(22)
N₂
N₂
are
tot
+
Dsl Dr. Pt-1
(23)
211
2716
N₂
9.
+
Ds Dr. DslDDr, Pa-1)
2nt
N₂ Ds Dr, P.-1, are +
N₂
+I
tgt
(24)
2n
2m)
Eq. 21 is due to the assumption of independence of Po w.r. Dg and Dr, and Eq. 22 is because mutual information is scale-invariant.
Consider the first term Eq. 24] for all I, inspired by the proof of Lemma
118 we have
N
Dr, P,-1)
9i
EDs,Dr, KL (P
- 1 DD.,P 1.Dg |P -g p
Ny NrD N
(25)
-BDg.Dr.P KI.
(P
(P-gree Ni IDr (PP KI. (P
(P-
-g
Mi 2m
Dr. Pt-
(26)
< EDs KL (P
-D Pl PG.
(27)
where Pr.,Dr.P.-I some random distribution, every choice which results upper bound for the MI, and the equality holds when PG P.. = gore
No. Tne
Dr.Pi 1'
Therefore, if we choose P..,Dy.P.. = NO, of I). the R.H.S of Eq. 27 will be upper-bounded 4n/
by 2nd oi Eng.Dr.P. [[l9gre[1]] which derived from the KL-divergence between two Gaussian
distributions.
Similarly for the second term in Eq. choosing PG
NOO, I) gives Ang
us the upper bound Dr. D...PrPP-. Pa-.
2n E
o [13] Furthermore, letting G P.... [
of
P --ggro which is also Gaussian distribution due to the effect of the added noise, we reach the gradient matching term,
2n DDDs.DD.P..1 [llg."e -1
3.55112]
of
we suppose the weicht term wishes to keep the impact A, they can change g are
simplify proof. However, if one [>g. sh,src ge 0]. In this case, the terms under the expectation in the bound will become: (X² -X=-1)ller sre
Note that in Eq. 15
| 22 g
ggt
-11g?"" Agt - -Ag 2
Combining everything together, the proof is done.
21

---

B Algorithm
B.1 Final objectives
As we cast UDA as MOO problem, the ideal final objectives, in the case of single-souree UDA, would be
[CS
PGA
(P))
CT
CPGA
(P),,
where
CT
CPGA (P) == (P
Pga
s
S
||. 9sh.T
+ Pgn
9.h.T
Pr + Pyn
99
9T
(P)( Cs (P) == Cs(Pah Pga
Pyn
Ps + Pan
9s
g
|g
As aforementioned, we use scalarization method, i.c. reweighting loss functions with put on the PGA source objective. As result, the PGA gradient updates for prompts are
PGA PGA s TT = Pp
PGA (P),
PGA
PGA = V VpL3 9s
PGA (P).
Ssh.st
Ps Ps 119
PGA
PT PT 19
PGA
ssh sh ghh
Aghh.s)
However, computing these PGA gradients will trigger the computation of the Hessian matrix. Hence, we approximate them with practical version:
9 9T
PGA PGA
vpCT (P)
V (P PT) Pan -Pe
soa.n PPnn 19ah.
8
Rch.7 Pr= Pr Pan gr Twy
PGA PGA 9 S
Vp (P)
PPLs(Pan) sh, Ps) P.n PPa Taash. 1Rahh
BoheT
Buh. Tiahh
Ps Pst Tlaa
B.2 Extension to Multi-source UDA
Our method can be easily extended to work with multi-source domains by noting that the target gradient is aligned with each of the source gradients.
PGA PGA I' 91
V P
PGA (P);
g shij I8.1
CPPAA PP).
Vi =
Ps. Ps.i 198.
Vi = 1==1-4 1
Pr Pr 199
PGA
Peh. Pah gg
PGA
I JT
PGA
VpLr sh: Pry
Pa -aa-Pan Poa E. lloahal 1oo... Pan TTl"
Pah. Rsh.T
RT Pr= Pr==Prtpa Pr (lrr!
(28)
PGA 93. s
PGA
Rah,T Pus PPo TRsh, +Pg TMah,
Rahe
Pe., Ps. + Tl
0..
(29)
The details of our proposed method for the general case of N source domains are presented in Algorithm When N 1, our method degrades to PGA.
C Experimental details
In this section, we provide additional information for our experimental settings Section C.1
and
C.2 then include detailed ablation studies and other empirical results in Section C
22

---

Algorithm Prompt gradient alignment for unsupervised domain adaptation
Phs ( IN Prl. gradient norm penalization trade-off Pans alignment strength Peus Source-gradienn trade-off X, learning rate 17.
Input: Prompt P =
Output: Updated prompt P
1: Compute target loss (r(Psh: Pr) as in Eq.
2: Compute gradients shared and target-specifie prompts wir. target loss
Th 9T + VpLr (Psh. sh, Pr)
3: Compute source losses (s.i(Pst, Ps,s) as in Eq 2
4: Compute gradient of shared and source-specific prompts w.r.t each source loss
sh Ss f- SpCs. P P ss)), Vi =
5: Compute 9sh,T' 9T as in Eq. 28
PGA
6: Compute gsh, Ss
PGA as Eq.
29
1
PGA 7: Compute combined gradient of shared prompt goh = Ihh
+ À E, g
8: Update prompt
P Pahi (Ps.) Pr] nlghh. (gs. Ji=1 9T
PGAAN PGA
C.1 Datasets
ImageCI.FF is a small-scaled dataset with 1,800 images across 12 object categories from three domains: ImageNet ILSVRC 2012 (I), Pascal VOC 2012 (P), and Caltech-256 (C). Office-Home is medium-scalle dataset containing approximately 15,500 images from 65 categories in four domains: Art, Clipart, Product, and Real World. DomainNet is the largest dataset, comprising around 600,000 images from 345 categories across six domains: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch.
C.2 Implementation details
For fair comparisons, we use ResNet50 as our backbone on Image-CLEF and Office-Home and ResNet101 on DomainNet. Their weights are taken from pretrained-CLIP and kept frozen during training. Prompts are trained with the mini-batch SGD optimizer with learning rate of 0.003 and 0.005. We use batch size of 32 and adopt a cosine learning rate scheduler. For token lengths Mj and M2 are both set to 16. Pseudo-label threshold is set to 0.4 for producing reliable labels. Pgn- Pya and À are found using grid-search. Details are provided the public source code.
During inference, we average the prediction of both source Ps and target Pr prompts, which empirically yield the best performance. Please note that the inference cost remains almost the same as using a pretrained CLIP as computing class embeddings is an one-time-cost. The complexity grows linearly with the number of prompts during training (= 2 with PGA and N + 1 in the case of MPGA), which is typically not big issue in practice since the model training can quickly converge by fine-tuning under intrinsic dimension [119). We further confirm this in the computation complexity ablation study below.
C.3 Additional experiments
C.3.1 Illustrative example
We small multi-bbjective-ppiiizzaoon problem the ZDT-1 1 problem [120] The ZDT-1 problems have a 30-dimensional variable and two differentiable objective functions f1, f2:
min fi min fe(x) = g()))
The function g(x) can be considered the function for convergence, their formulas given by:
23

---

fi(2) = 21
g(x) 1 π1
24
(fig)) = 1 Vh1
0 0 $5
with the Pareto solutions are given by:
0
and
1 = for = ,,...,
Iteration: 0/500
Iteration: 100/500
Iteration: 300/500
Iteration: 400/5 D0
3.0
1.5
2.5
3-
2.0
1.0
0.0
1.0
0.5
0.5
0.0
0.00
0.50
0.75
0.50
0.75
1.00
0.25
0.00
0.25
0.50
0.75
1.00
Reration: 450/500
Reeatiin: 480/500
Iteration: 490/500
Iteration: 500/500
1.0
1.0
1.0
1.0
9.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
f1
Figure 4: ZDT-1 task-specific gradient directions different iterations. Red curve represents the Pareto front while the blue and green arrows indicate the updating directions for minimizing fi and f2, respectively.
As can be seen from Figure 4 the cosine similarity increases at the beginning of the training and then decreases when the obtained solution reach the region near the Pareto front. This behavior aligns with the gradient similarity evolution experiment the main paper.
C.3.2 Large-scale single-source unsupervised domain adaptation
Apart from those experiments in the main paper, expand the single-source unsupervised domain adaptation setup by including the empirical results on two large-scale synthetic-to-red benchmark for classification adaptation S2RDA-49 and S2RDA-MS--3 [21] For each task, synthetic samples are created by rendering 3D models from ShapeNet, matching the label space of the real/target domain, with 12K RGB images per class. The S2RDA-49 real domain contains 60.535 images across 49 classes from various sources including the ImageNet validation set. The S2RDA-MS-39 real domain includes 41,735 natural images for 39 classes from MetaShift, featuring complex contexts like object co-occurrence and attributes, which adds to the task's 's difficulty.
Table 6: Unsupervised domain adaptation results on S2RDA. The best accuracy is indicated bold.
Transteer Tank
No Adaptation
DANN
MCD
RCA
SRDC
DicClusterIA
DAPL
PGA (Ours)
Mean
Mean
Mean
Mean
Mean
Avc.
Mean
Mean
Mean
SIRDA
47.8
47.1
67.8
36.9
Table 6 illustrates accuracy and mean score over classes, where utilizing pretrained vision-language models still shows their impressive performance. Using pretrained CLIP standalone outperforms other traditional DA methods and PGA further boosts the performance by large margins, 4% on S2RDA-49 and 1.5% on S2RDA-MS-39, respectively.

---

C.3.3
Ablation studies
Similar to previous work on CLIP adaptation 25 28 we vary the pseudo label threshold value to study its sensitivity. As can be seen in Figure 7 both PGA and MPGA's performance is relatively stable across different values of т, indicating that methods not sensitive to and the best result obtained at a reasonable trade-off between the quantity and quality of pseudo data.
Table 7: Accuracy (%) of different threshold on ImageCLEF,
0.1
0.2
0.3
0.4
0.5
0.6
0.8
0.9
PGA
91.5
92.0
92.1
92.4
92.3
92.1
92.0
92.0
MPGA
92.4
92.6
92.9
92.7
92.7
92.7
92.6
92.5
In Figure 5 we provide the complexity for some comparative baselines, Accuracy curve (left): While DANN and CDAN obtain their best performance at approximately 77% after more than 1000s, PGA and MPGA achieve 84% within 100s. Besides, the first stage of pairwise source-target training of MPA takes 159s, followed by 35s for the second stage actually train the final model. Number of Trainable Parameters (middle): PGA and MPGA, with fewer than 140k parameters, require significantly fewer parameters than MPA, DANN and CDAN, which have around 1M, 48.9M and 51.7M parameters, respectively. GPU Memory Usage (right) PGA, MPGA, and MPA exhibit substantially lower memory footprints, around 1300MB compared to 7000MB of DANN and CDAN throughout training.
DAHH
40.0M
GPU Mommrry Miagee (Pyitss
COAN
51.77
CEAM
1H
NIS
PCA
NIGA
PGA
MPG1
230
620
:ike
3200
16:00
Relative Time (Seconda)
Figure 5: Computational complexity: accuracy curve (left), number of trainable parameters (middle).
and GPU memory (right).
Figure 6 shows that PGA is generally not sensitive to Pga and Pyn within their acceptable range, i.c. le-2 to 10 for Pga and le-5 to 0.1 for Pyn. Specifically, (i) too large value of Pyn is less effective than smaller ones; (ii) ImageCLEF prefers larger values of Pga while OfficeHome prefers smaller ones, suggesting that source and target domains in the former dataset may be more similar than those in the latter, hence over-matching gradients in the latter dataset may be adverse.
ImageCLEF
Office-Home
91.7
91.9
91.8
91.9
91.7
76.8
76.5
76.6
76.7
76.1
77.0
91.9
92.2
92.4
92.1
91.4
77.0
77.2
77.0
76.9
76.4
91.5
76.8
Pge
90.8
90.7
91.2
91.4
91.0
Oge
77.0
77.0
76.9
76.7
76.7
76.6
0.1
90.6
90.6
91.1
91.0
90.9
77.0
77.0
76.9
76.8
76.6
76.4
90.5
90.1
90.1
90.7
90.7
90.5
77.0
77.0
77.0
76.7
76.4
76.2
0.001
0.1
0.01
Pgn
Pgn
Figure 6: Parameter sensitivity analysis on Pyn and Pya of PGA on ImageCLEF and Office-Home with CLIP-RN50 backbone.
25

---

We present results of our methods using ViT-B/16, VIT-L/14 backbones on OfficeHome in Tables and following experimental setups [22 [23]. We can observe the superiority of methods among all baselines while finetuning small portion of the backhones using prompt tuning. Especially, PGA outperforms the second-best method on ViT-B/16 backbone by 1% accuracy score.
Table 8: Accuracy (%) on Office-Home of ViT-based vision encoder CLIP backbones (except CDTrans* uses DeiT). The overall best accuracy and best within per backbone are indicated in bold and underscore, respectively.
Method
Ar-PP
Ar-Rw
Avg
probe
C.CoOp
70.4
21.8
YPT-shillow
31.6
VPT-doop
VIT-B/16
71.6
00.3
91.0
82.0
91.3
MaPLe
12.2
91.5
90.3
$2.6
90.9
$2.4
90.0
85.
$4.2
CLIP
67.5
89.0
89.8
82.9
39.0
89.8
$2.9
57.8
39.8
82.9
67.8
89.0
DAPL
70.5
91.2
91.0
$4.9
89.2
91.99
$4.8
70.5
90.5
84.5
70.1
9008
84,0
PGA (Ouns)
21.5
91.5
91.
34.8
91.5
90.9
34.9
71.5
91.1
85.9
22.1
92.4
35.1
CLII
74.2
93.1
93.3
87.3
93.1
93.3
87.3
74.2
93.3
87.3
74.2
95.1
87.0
DAPI
VIT- /14
77.3
94.6
94.1
83.6
94.6
94.0
88.8
76.8
94.0
89.0
77.6
94.4
88.7
POIA (Cuos)
79.0
95.1
94.1
$8.9
95.1
94.2
83.9
78.8
94.2
88.9
79.0
95.3
89.4
Following a different protocol, Table the results of ViT-L/14 backbones on Office-Home but with three source domains category on Art, Clipart, Realworld and Product domain. In this setup, MPGA and PGA still consistently yield the best and second-best scores among all categories.
Table 9: Three-source domain adaptation the Office-Home dataset on ViT-L/14.
Method
> Ar
Rw
+ Pr
Avg
CLIP ZS(G)
84.97
91.94
90.96
89.29
CLIP ZS(A)
86.34
92.10
87.73
88.73
CLIP LP
87.02
92.55
92.70
90.76
LADS
87.71
93.86
93.00
91.52
LanDA
88.83
94.09
93.22
92.05
PGA (Ours)
89.17
95.37
94.34
92.96
MPGA (Ours)
89.88
95.49
94.97
93.45
C.3.4 Domain adaptation with label shift
This section to study how does the method performs when there are extreme label distribution shifts between source and target domains. We test PGA the setting of label shift following [124] where the source or target domains are down-sampled with only 30% of data from the first-half the classes are taken (indicated by prefix).
Table 10: Accuracy (%) on the sub-sampled Office-Home Home for unsupervised domain adaptation. The prefix denotes the domain where we sample only 30% of the images from the first half of its classes, following the label shift setting from prior work.
Method
-P:
sPr-iCI
sRt
<Rw
52.5
33.0
65.2
56.4
56.8
59.0
37.3
70.0
72.0
CDAN
38.0
56.8
64.8
48.0
6000
61.2
49.7
41.4
20.2
62.4
47.0
74.7
56.3
IWDANN
39.8
83.0
68.7
47.4
61.1
60.4
50.4
41.6
72.5
61.0
.9.4
76.1
57.6
IWJAN
36.2
61.0
66.3
48.7
59.9
61.9
529
37.7
70.9
00.3
41.5
73.3
55.9
IWCDAN
43.0
65.0
71.3
$2.9
64.7
66.5
44.3
75.9
67.0
78.6
(Ous)
I Label-shift results presented in Table 10 and 11 effectiveness of PGA on different levels label shift. PGA consistently yields superior performance on every sub-experiment under these two setups.
below and Table 5 in the main text show the
26

---

Table 11: Accuracy (%) on the sub sub-samples sampled (target) Office-Home for unsupervised domain adaptation.
An-isUI
Ar-tsPt
Cl~AAAr
Cl-isPr
Cl-ssWw
Pr-ssAr
Pr-raR
Rw-rsAr
Arg
ResNett-0
39.5
63.6
71.3
42.3
75.
DANN
47.8
53.9
06.11
45.3
34.8
56.3
49.4
48.0
70.2
63.4
55.3
72.7
50.3
JAN
45.8
64.7
74.9
53.9
03.2
653
56
42.5
63.9
47.4
78.8
61.4
CDAN
51.1
69.7
74.6
56.9
60.4
64.6
57.2
45.5
75.6
68.5
52.7
79.8
63.0
IWDANN
48.7
62.0
71.6
50.4
57.0
60.3
514
41.1
69.9
62,6
$1.0
77.2
58.6
IWJAN
44.0
71.9
75.1
55.2
05.0
67.7
57.1
42.4
74.9
66.1
46.1
78.5
62.0
TWCDAN
12.3
72.2
76.3
$6.9
67.3
67.7
57.2
44.8
77.8
67.3
53.3
80.6
64.6
PCT Uniform
55.8
77.6
72.3
76.7
67.0
90.9
72.6
57.0
PCT-Lesraable
C.4 Training Resources
All experiments are run Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz and NVIDIA A100- SXM4-80GB GPU.
D Additional related work
Another work sharing the same intuition of gradient alignment is ProGrad [25]] which manipulates gradient of the fine-tuned loss to preserve general knowledge of the pretrained model. Similar to other gradient-based MTI. methods 34 35
attempts to remove conflicts between per-ohjective gradients at each time step, thus is orthogonal to our approach. In contrast, we aim to stimulate their inherent consensus throughout training by encouraging the same training trajectory for both domains, hence, the model can find commonly good regions for them. Another concept that relates to gradient alignment is meta-learning. This has been introduced to Domain generalization in [126] [27]. Their intuition is training procedure that enables the model achieve low loss a subset of training domains after having learned the other ones, and they work on the full model space.
recent
work about Vision-Language Models [128], veta-leerning
used to deal with the problem of few-shot prompt learning by meta-learning prompt initialization. The gradient
the inner loop is modified with learnable regulating function, and data for the support and query sets are found by hierarchically clustering an auxiliary large-scale image-text dataset. This method also has the impact of aligning gradient between support and query data result of meta-leaming. However, its computation and space complexity rather large as it requires the computation of Hessian matrix, web-scale of image-text pairs, and meta-learns the soft initialization for prompts.
E
Limitations and Future works
First, our work relies on pretrained-CLIP. meaning that if UDA data is too different from pretrained knowledge, our method may fail to learn adequately. Therefore adapting our method seratch- training scenarios without heavy computation and space complexity should be investigated. Second, derived bound can be potentially loose as number of training iterations increases. Thus studying other types of bounds could be an interesting work. Finally, as we mentioned, strategy to explicitly align feature distribution across domains worth looking into.
27

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
28

---

Justification:
We reported our computational complexity and also have discussion section in Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We provided full assumption and proof of our theory in appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We reported detailed descriptions and hyperparameters for all experiments.
Guidelines:
29

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Our reproducible codebase is published in https://githu: com/
VietHoang1612/PGA
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nis.cc/public//uuse http://i/cgiiieee/eeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
30

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed settings and hyperparameters for experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conducted the illustrative experiment ten times independently with different random seeds and reported the mean and standard derivation of the result. The other experiment setups follow the protocol of prior well-known work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mention about compute resources in Appendix.
Guidelines:
31

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper has no negative social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
32

---

Justification:
All datasets used in our paper are public.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets used in our paper are public.
Guidelines:
The answer NA means that the paper does not existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: There is no new assets introduced in the paper.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
33

---

Answer: [NA]
Justification: There are no crowdsourcing experiments and research with human subjects in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "A9RuTSGk2o", "reviewer": "Reviewer_YFTB", "summary": "To enhance both transferability and discriminability for prompt learning based domain adaptation, this paper proposes a Prompt Gradient Alignment (PGA) method. PGA encompasses multiple domain-wise classification objectives, cosine similarity maximization regularizers between prompt gradients of different domains, and prompt gradient norm penalty for each classification objective. It can thus achieve both inter-domain gradient alignment and flat minima enforcement. To efficiently and effectively update the designed prompts, a practical gradient update procedure is devised and works under both single-source and multi-source UDA. Empirical results shown the superiority of the proposed method.", "questions": "See Weaknesses.", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A novel method with an information-theoretic explanation. Aligning prompt gradients across domains moves the shared prompt towards the low-loss region of both domains, such that domain-invariant class-discriminative features can be captured, thus benifiting both domains.\n2. A practical, flexible, and efficient gradient update procedure. It has a weighting term on the source signal to control how much emphasis should be put on the target domain and avoids the costly Hessian computation. \n3. A well-written article. The paper has a smooth logic and is easy to understand.", "weaknesses": "1. The abstract should reflect the technical highlights. For example, UDA is essentially a multiple-objective optimization problem, which is not the technical novelty of this paper.\n\n2. As indicated by Theorem 4.1, the proposed method minimizing source empirical loss, gradient norms, and gradient misalignment, model and reduce the upper bound of the established generalization error. Nevertheless, it is not intuitive why minimizing gradient norms and gradient misalignment are beneficial for the performance on the target domain. \n\n3. In Theorem 4.1, g_t^src and g_t^T are the gradients w.r.t. P_t of source loss and target loss respectively. The two gradients should be derived from the same prompt or different prompts? Can the gradient alignment theory be extended to other spaces like image, feature, and output spaces?\n\n4. Due to use of pseudo labels for the target domain, though they are improved by confidence filtering, the accuracy of pseudo labels may greatly affect the success of inter-domain gradient alignment. It is encouraged to investigate the relationship between them.\n\n5. Some important related works are missing in the discussion of related works, e.g., [a].\n\n[a] Tang et al. Unsupervised domain adaptation via distilled discriminative clustering. PR, 2022.\n\n6. Validation on larger datasets like S2RDA [b] is important for three reasons. First, it is a more large-scale, realistic, challenging multi-domain benchmark dataset. Second, it is tailored for synthetic-to-real transfer, more in line with industrial needs. Third, this paper leverages the powerful generalization capability of large-scale pre-trained vision-language models and fine-tuning with a large amount of training data can avoid the potential risk of overfitting.\n\n[b] Tang and Jia. A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptation. CVPR, 2023.\n\n7. What circumstances does the overfitting issue appear? Why does penalizing gradient norms alleviate the overfitting issue? Are there any theoretical reasons?\n\n8. In Eq. (6), L_tgt should be L_T? In Eq. (20), L_s should be L_s^PGA? The authors may re-write the final overall objective for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Thanks for the suggestion, we will revise the abstract to reflect our technical contributions more clearly. Regarding the multiple objective optimization viewpoint, our MOO problem consists of per-domain objectives, motivated by the strong performance of the self-training baseline in Section 1. This empirical results show that optimizing target loss using pseudo data alone is already a strong baseline, which is not typically found in traditional UDA methods. They instead often combine source loss with auxiliary domain alignment objectives.\n2. \n- As can be seen from the first term on the R.H.S of Theorem 4.1, minimizing these two terms will reduce the generalization error, thereby closing the gap between target population risk and source empirical risk. Furthermore, as we also minimize the latter through the cross entropy loss on labeled source data, the former will be reduced. This is the ultimate goal an UDA method aims to achieve, as low population target risk indicates a model having a good generalizability on target data. \n- Intuitively, maximizing gradient alignment encourages the optimization trajectories to be the same for all domains, and minimizing gradient norm helps find flat regions for both shared and domain-specific prompts, which alleviates overfitting and generally leads to better performance on source/target data [4, 18, 60]. Altogether, performance on target domain can be improved.\n3.  \n- ${g_t^{src};g_t^{tgt}}$  should be derived from the same prompt, and in the theorem, to ease the proof, they refer to the whole prompt set $[P_{sh}, P_S, P_T]$ at time step $t$. Note that this does not contradict our gradient alignment strategy which is applied on the shared prompt only. Indeed, as we showed in Remark A.7, since $P_T$ is not involved when computing the source loss, we can write $g^{src}_t = [g^{sh,src}_t, g^S_t, 0]$. Similarly, $g^{tgt}_t = [g^{sh, tgt}_t, 0, g^T_t]$. Therefore, the inner product between $g^{src}_t$ and $g^{tgt}_t$ is equal to the inner product between $g^{sh,src}_t$ and $g^{sh,tgt}_T$.\n - Extension of the theory to other spaces like image, feature, and output spaces: Please note that the second term in Theorem 4.1, which contains the KL-divergence between the two underlying source and target distributions $\\mu, \\mu'$, is the motivation for aligning these two distributions in image, feature or output space. Normally, this is done in feature space due to the rich representation captured by the feature extractor. For example, marginal feature alignment is adopted in [R1] and conditional feature alignment in [R2]. In the case of conditional alignment, one has to assign pseudo labels to target data using the learned source model [R3], which may lead to accumulation error. In our case, as we use zero-shot CLIP to predict pseudo labels, the error could be alleviated. Last but not least,  as mentioned in the paper, incorporating our gradient matching and norm penalization into those methods fully reflects the two terms from the R.H.S of the bound, hence could further boost performance.\n4. We provide an ablation varying the value of threshold $\\tau$ in Table 2 of the attached PDF. Our methods are not sensitive to $\\tau$, and the best result is obtained at a reasonable trade-off between the quantity and quality of pseudo data.\n5. Thank you for your suggestion, we will add this paper to the related works and its results to the Office-Home experiment, where they obtain an average accuracy of 71.4 compared to 75.8 (ours).\n6. We include the performance of CLIP-based models on two Synthetic-to-Real datasets S2RDA-49 and S2RDA-MS-39 in Table 1 of the attached PDF. PGA achieves the best performance among the baselines.\n7. \n- When does the overfitting issue appear: Since we are optimizing multiple objectives across domains, naively minimizing them might lead to overfitting on some particularly easy-to-optimize or limited-data objectives, while forcibly minimizing them could harm other objectives.\n- Why does penalizing gradient norms help: Briefly, in standard supervised training, minimizing gradnorm will lower the population risk, hence reduce overfitting. Similarly, in UDA, applying this on source and target data will lead to lower population risks on respective domains. Specifically, followed Theorem 1 in [55], we can upper bound these risks as:\n$[L_{\\mu}(P), L_{\\mu'}(P)] \\leq \\max_{||\\epsilon_{sh}||\\leq\\rho_{gn}}[\\max_{||\\epsilon_S||\\leq\\rho_{gn}}L_S(P_{sh}+\\epsilon_{sh},P_{S}+\\epsilon_S)+f_S(||P||), \\max_{||\\epsilon_T||\\leq\\rho_{gn}}L_T(P_{sh}+\\epsilon_{sh},P_{T}+\\epsilon_T)+f_T(||P||)],$\n    where $f_S,f_T$ are strictly increasing functions. \nFurthermore, the worst-case source loss can be approximated as\n$\\max_{||\\epsilon_{sh}||\\leq\\rho_{gn}}\\max_{||\\epsilon_S||\\leq\\rho_{gn}}L_S(P_{sh}+\\epsilon_{sh},P_{S}+\\epsilon_S) \\approx L_S(P)+\\rho_{gn}(||g_{sh,S}||+||g_S||),$ and similarly for the worst-case target loss.\n\n8.\n- \"In Eq. (6)\". Thanks for pointing this out. Yes, $L_T$ is the correct notation. Also in the definition of the gradient w.r.t target loss (in line 224), the correct notation should be $g_t^{tgt}$. We will fix these in the revision.\n- In Eq. (20): In fact, it should be $L_S$, which is similar to $L_T$ in the approximation of Eq.(19). \nInformally, the PGA gradients corresponding to the source objective are the derivative of $L_S^{\\text{PGA}}$ w.r.t $P$, which are then approximated by the derivative of the source loss at  $L_S$ the perturbed prompts: $P_{sh}=P_{sh}-\\rho_{ga}{b}+\\rho_{gn}\\frac{g_{sh,S}}{||g_{sh,S}||}, P_S =P_S+\\rho_{gn}\\frac{g_S}{||g_S||}$. (This approximation follows the derivation of Eq.(13)).\n- Overall objective: please refer to the general response.\n\n[R1] Nguyen, A. Tuan, et al. \"KL guided domain adaptation.\", ICLR 22\n\n[R2] Nguyen, A. Tuan, et al. \"Domain invariant representation learning with domain density transformations.\", NeurIPS 21\n\n[R3] Long, M., et al. \"Conditional adversarial domain adaptation\", NeurIPS 18", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HAP1rIwQ35", "reviewer": "Reviewer_JYym", "summary": "This paper proposes a novel approach called Prompt Gradient Alignment (PGA) for unsupervised domain adaptation (UDA). The key contributions are: (1) Formulating UDA as a multi-objective optimization problem with objectives for source and target domains; (2) Aligning gradients between objectives to encourage consensus; (3) Penalizing gradient norms to improve generalization; (4) Leveraging pre-trained vision-language models through prompt learning. The method is evaluated on standard UDA benchmarks and demonstrates consistent improvements over other prompt-based baselines. A theoretical generalization bound is also provided to justify the approach.", "questions": "1. How does PGA's computational complexity compare to baseline methods? Are there significant differences in training time?\n2. Have you explored performance on data types beyond image classification? How generalizable do you expect the method to be?\n3. How sensitive is the method to hyperparameter choices, particularly for gradient alignment and norm penalization?\n4. Given the reliance on CLIP pre-training, how well do you expect the method to perform in domains very different from CLIP's training data?\n5. How does the method's performance change as the degree of domain shift varies? Is there a point where simpler approaches become competitive?", "limitations": "The reliance on CLIP pre-training potentially limits applicability to domains very different from CLIP's training data. It's unclear how well the method would perform on specialized domains (e.g., medical imaging, satellite imagery) that are not well-represented in CLIP's training set.\nThe paper does not thoroughly explore potential failure cases or scenarios where the method might struggle. For instance, how does the method perform when there are extreme label distribution shifts between source and target domains? Or when the domain shift is particularly severe?\nThe scalability of the method to very large datasets or high-dimensional data is not addressed. It's unclear how the computational requirements scale with dataset size or feature dimensionality.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The strong empirical results across multiple UDA benchmarks are particularly impressive. The consistent outperformance of other prompt-based methods on datasets like ImageCLEF, Office-Home, and DomainNet (as shown in Tables 3-5) provides robust evidence for the method's effectiveness. The performance gains are substantial in many cases, with improvements of up to 4% on average accuracy compared to state-of-the-art methods.\nThe theoretical analysis offering a generalization bound (Section 4.5) adds credibility to the approach, providing insights into why the method works and under what conditions it can be expected to perform well. This combination of empirical success and theoretical grounding is a significant strength of the paper.\nThe ablation studies (Section 5.4, Table 6) effectively demonstrate the contribution of each component of the proposed method, providing a clear understanding of how different elements (like gradient alignment and norm penalization) contribute to the overall performance. The visualization of gradient similarity evolution (Figure 7) offers additional insights into the method's behavior during training.", "weaknesses": "The computational complexity and training time comparisons to baseline methods are notably absent. Without this information, it's difficult to assess the practical trade-offs of implementing PGA compared to existing methods. This is particularly important given the method's use of gradient manipulation, which could potentially increase computational requirements.\nThe paper's heavy reliance on pre-trained CLIP models raises questions about the method's applicability in scenarios where such pre-training is not available or suitable. While the use of pre-trained models is a strength in many cases, it could also be a limitation in domains significantly different from CLIP's training data.\nThe discussion of hyperparameter sensitivity is limited. Given the importance of hyperparameters like ρga and ρgn in the gradient alignment and norm penalization processes, a more thorough exploration of their impact on performance would be valuable.\nWhile the method shows improvements over existing approaches, the degree of novelty is somewhat incremental. The core ideas build heavily on existing prompt-based and gradient manipulation techniques, which may limit the paper's impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. In Figure 1 of the attached PDF we provide the complexity for some comparative baselines. Accuracy curve (left): While DANN and CDAN obtain their best performance at approximately 77% after more than 1000s, PGA and MPGA  achieve 84% within 100s. Besides, the first stage of pairwise source-target training of MPA takes 159s, followed by 35s for the second stage to actually train the final model. Number of Trainable Parameters (middle): PGA and MPGA, with fewer than 140k parameters, require significantly fewer parameters than MPA, DANN and CDAN, which have around 1M, 48.9M and 51.7M parameters, respectively. GPU Memory Usage (right)} PGA, MPGA, and MPA exhibit substantially lower memory footprints, around 1300MB compared to 7000MB of DANN and CDAN throughout training.\n\n2. Thank you for the suggestion. Given the task-agnostic nature of our method, where PGA/MPGA utilizes only gradient information, and the successful application of CLIP-based methods in diverse tasks like image segmentation [R2] and object detection [R3], we believe our approach could be similarly effective in other tasks. For instance, applying our method to image segmentation, which requires predictions for each pixel, would necessitate training an additional decoder. Due to constraints in the rebuttal period, this extension is not straightforward, and is considered beyond the scope of our current experimental setup. We leave this for future work. Nonetheless, we believe that our method's superior results in image classification are sufficient to verify its benefits, as we adhere closely to the standard protocols of related UDA methods.\n\n3. Figure 2 in the attached PDF shows that PGA is generally not sensitive to $\\rho_{ga}$ and $\\rho_{gn}$ within their acceptable range, i.e. 1e-2 to 10 for $\\rho_{ga}$ and 1e-5 to 0.1 for $\\rho_{gn}$. Specifically, (i) a too large value of $\\rho_{gn}$ is less effective than smaller ones; (ii) ImageCLEF prefers larger values of $\\rho_{ga}$ while OfficeHome prefers smaller ones, suggesting that source and target domains in the former dataset may be more similar than those in the latter, hence over-matching gradients in the latter dataset may be adverse.\n4. It is true that when source and target domains are very different  from CLIP's training data, prompt learning alone may not be sufficient to adapt well to the target domain. Yet, our method is still expected to perform better than prompt-based baselines. A failure case is discussed in our adaptation to the QuickDraw domain of DomainNet, which contains black and white sketches. Since this domain differs significantly from CLIP's training data, all prompt-based methods fail to achieve good accuracy and perform worse than some full-finetuning methods. This observation suggests integrating our methods with other invariant feature learning methods (lines 32-33 in our manuscript) or finetuning more parameters can better adapt to the target domain. Nonetheless, our PGA and MPGA still yield the highest results among prompt-based ones, indicating that our gradient alignment and norm penalization procedure is more helpful in closing domain gap than previous DAPL and MPA. Another example where prompt-based cannot surpass traditional UDA baselines is Rw→Cl from OfficeHome in Table 5. \n\n    To conclude, for specialized domains not well-represented by the pretrained CLIP model, a more effective strategy might involve updating additional parameters, such as the layers close to the output in both vision and text encoders, beyond just the prompts. This could lead to better domain adaptation at the expense of computational complexity increase. We acknowledge this limitation in Appendix D and leave it for future work.\n\n5. We test PGA on the setting of label shift following [R1], where the source or target domains are down-sampled with only 30% of data from the first-half of the classes are taken (indicated by **s-** prefix). Results presented in Table 4 in the attached PDF, table below and Table 5 in main text show the effectiveness of PGA on different levels of label shift.\n| Method      | Ar→sCl | Ar→sPr | Ar→sRw | Cl→sAr | Cl→sPr | Cl→sRw | Pr→sAr | Pr→sCl | Pr→sRw | Rw→sAr | Rw→sCl | Rw→sPr | Avg  |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| ResNet-50   | 41.5  | 65.8  | 73.6  | 52.2  | 59.5  | 63.6  | 51.5  | 36.4  | 71.3  | 65.2  | 42.8  | 75.4  | 58.2 |\n| DANN        | 47.8  | 55.9  | 66.0  | 45.3  | 54.8  | 56.8  | 49.4  | 48.0  | 70.2  | 65.4  | 55.5  | 72.7  | 58.3 |\n| JAN         | 45.8  | 69.7  | 74.9  | 53.9  | 63.2  | 65.0  | 56    | 42.5  | 74    | 65.9  | 47.4  | 78.8  | 61.4 |\n| CDAN        | 51.1  | 69.7  | 74.6  | 56.9  | 60.4  | 64.6  | 57.2  | 45.5  | 75.6  | 68.5  | 52.7  | 79.8  | 63.0 |\n| IWDANN      | 48.7  | 62.0  | 71.6  | 50.4  | 57.0  | 60.3  | 51.4  | 41.1  | 69.9  | 62.6  | 51.0  | 77.2  | 58.6 |\n| IWJAN       | 44.0  | 71.9  | 75.1  | 55.2  | 65.0  | 67.7  | 57.1  | 42.4  | 74.9  | 66.1  | 46.1  | 78.5  | 62.0 |\n| IWCDAN      | 52.3  | 72.2  | 76.3  | 56.9  | 67.3  | 67.7  | 57.2  | 44.8  | 77.8  | 67.3  | 53.3  | 80.6  | 64.6 |\n| PCT| 57.5 | 78.2 | 80.5  | 66.7  | 74.3  | 75.4  | 64.6  | 50.7  | 81.3  | 72.9  | 57.3  | 83.5  | 70.2 |\n| PGA (Ours)  | 57.4  | 84.8  | 86.4  | 76.0  | 84.6  | 85.6  | 74.5  | 57.1  | 86.1  | 75.9  | 57.4  | 85.3  | 75.9 |\n6. Scalability w.r.t dataset-size: Our experiments already included small (ImageCLEF), medium (OfficeHome), and large-scale (DomainNet, S2RDA) datasets. Furthermore, our proposed optimization procedure is not data-dependent, hence the increase in data size or dimension would not cause computational overhead for PGA and MPGA compared to other baselines.\n\n[R1] Tanwisuth, Korawat, et al. \"A prototype-oriented framework for unsupervised domain adaptation.\" NeurIPS 21\n\n[R2] Wang, Zhaoqing, et al. \"Cris: Clip-driven referring image segmentation.\" CVPR 22.\n\n[R3] Wu, Xiaoshi, et al. \"Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching.\" ICCV 23.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5KpBI0B771", "reviewer": "Reviewer_dhjM", "summary": "This paper proposed a novel domain adaptation method that tunes the text prompts based on self-training with pseudo labels. The proposed method treats the source training and target training as multi-objective optimization problems, and it introduces to alignment of the gradients from both training (Prompt Gradient Alignment) so as not to cause conflict between the source and target training. The authors validated the proposed method empirically and theoretically.", "questions": "+ As mentioned in the third bullet of weaknesses, I could not fully understand the formulations. Could you explain which is the actual loss function?", "limitations": "+ I did not found any additional limitation for this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ This paper is rich in technical novelty.\n  + There are many methods in domain generalization research that aim to align gradients from multiple domains. However, requires heavy hessian calculations. The proposed method successfully avoids this issue by approximation with Taylor expansion.\n+ The proposed method is empirically and theoretically justified.", "weaknesses": "+ It is questionable whether the comparison with existing methods is fair.\n  + The old domain adaptation methods fail to show their effectiveness when they are applied to the CLIP model. I suspect that those domain adaptation methods were not properly applied to CLIP. Is there any reason why they fail?\n  + Conversely, would it not be possible to evaluate the proposed method in accordance with existing domain adaptation benchmarks? For example, would it be possible to evaluate the proposed method by updating the model parameters instead of the prompts?\n+ There are some additional related works that should be discussed in the paper.\n  + Bose, Shirsha, et al. \"Stylip: Multi-scale style-conditioned prompt learning for clip-based domain generalization.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024. This work proposes a prompt training method for the domain generalization tasks.\n  + Zhu, Beier, et al. \"Prompt-aligned gradient for prompt tuning.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. This introduces a gradient matching idea into prompt tuning, which has a similar mind to this paper.\n+ Some of the formulas are a little difficult to follow.\n  + It is very difficult to understand which of eq 9 or eq 11 is the actual loss function. This goes for eq 16 and eq 18 as well. I recommend separating the formulations for easier understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> whether the comparison with existing methods is fair\n\nAs we follow the experimental settings in [6, 15, 22, R4, R5, R6], we use their reported results which we believe have been verified to be fair when comparing old UDA methods and prompt-based methods. Specifically, they share the same vision backbone as prompt-based ones, e.g. ResNet, and are optimized to reach their full potential. Only one advantage of prompt-based methods is the use of additional textual information about class names when making predictions. However, this comparison setup is not only used in UDA, but also adopted in many works in other domains such as spurious debiasing [R7], continual learning [R1, R2]. Therefore, we believe that old UDA baselines are fairly compared. Additionally, we include comparisons with more recent CLIP-based baselines in Table 5 of the attached PDF. Here we finetune only a small portion of the model compared to other baselines, but still obtain competitive performance.\n>is there any reason why they fail?\n\nTheir underperformance could be attributed to the difficulty of using a single backbone to simultaneously learn domain-invariant and class discriminative features [22, 75]. This difficulty usually arises from the inherent entanglement between domain and semantic information [R3], which can be further amplified as more source domains are involved [6]. In contrast, we not only devote a set of shared prompts to learn domain-invariant features, but also domain-specific prompts to capture useful information for classification. By leveraging these prompts, we create a more meaningful representation, leading to more precise predictions for the target domain\n> would it be possible to evaluate the proposed method by updating the model\n\nYes, it is possible. Our proposed algorithm and theory development are generic and could be extended to finetuning the entire model. However, this would still entail a large computation and space complexity due to the size of the whole model, despite the avoidance of Hessian matrix calculation. \nEven so, adapting the full model on source/target domain can easily lead to overfitting and potentially inferior results in the presence of domain shift, even in the case of transformer models [1, 14, 34]. Therefore we opt for prompt learning which is a better way to adapt CLIP to downstream tasks (similar to other CLIP-based baselines) and is less likely to overfit. \n\n> Related works \n\nThanks for the suggestion, we will add them in the revision. Briefly, the second work, ProGrad, manipulates gradient of the fine-tuned loss to preserve general knowledge of the pretrained model. Although it shares the intuition of gradient alignment with our work, there is a significant difference: ProGrad attempts to remove conflicts between per-objective gradients at each time step, similar other gradient-based MTL methods such as [96, 39]. In contrast, we aim to stimulate their inherent consensus throughout training by encouraging the same training trajectory for both domains, hence, the model can find commonly good regions for them. The first work appears unrelated to ours as it approaches UDA from a architecture-view, i.e. introducing additional modules to generate content- and style-aware prompts, whereas we tackle it from the view of model optimization.\n\n> Difficulty of some formulas and the actual loss function\n\n - Ideally, Eq.(9) and Eq.(16) would be the actual loss functions that we want to take derivative. The reason we approximate them to get Eq.(11) and Eq.(18) is to show that minimizing them can fulfill our purposes of (i) aligning prompt gradients and (ii) penalizing prompt gradient norms.\n - Note that Eq.(16) indicates our full algorithm while Eq.(9) is only about gradient alignment. \n - However, one difficulty in taking derivative of Eq.(9) or Eq.(16) is the computation of Hessian matrix, hence we devise a practical approximation as shown in Eq.(13) and Eq.(19), which allows us to only need to compute gradient of the original source/target loss, i.e. Eqs.(3,4), at the perturbed prompts.\n  - Please refer to the general response to see the actual loss function.\n\n[R1] Wang, Yabin, et al. \"S-prompts learning with pre-trained transformers: An occam’s razor for domain incremental learning.\", NeurIPS 22\n\n[R2] Yu, Jiazuo, et al. \"Boosting continual learning of vision-language models via mixture-of-experts adapters.\", CVPR 24\n\n[R3] Cai, Ruichu, et al. \"Learning disentangled semantic representation for domain adaptation.\" IJCAI 19\n\n[R4] Bai, Shuanghao, et al. \"Prompt-based distribution alignment for unsupervised domain adaptation.\" AAAI 24\n\n[R5] Li, Xinyao, et al. \"Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation.\" CVPR 24\n\n[R6] Du, Zhekai, et al. \"Domain-agnostic mutual prompting for unsupervised domain adaptation.\" CVPR 24\n\n[R7] Phan, Hoang, et al. \"Controllable Prompt Tuning For Balancing Group Distributional Robustness\", ICML 24", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Un6PMgLXC", "reviewer": "Reviewer_aPJr", "summary": "This paper aims to leverage prompt tuning of vision-language models for Unsupervised Domain Adaptation (UDA) tasks. The authors formulate UDA as a multi-objective problem where each objective is modeled by a domain loss. To resolve conflicts between domains, they manipulate gradients by maximizing their cosine similarity. Additionally, to stabilize the training procedure, they propose using the norm of the gradients as a regularization term. They also provide a generalization error bound for their method. Empirical results demonstrate the effectiveness of the proposed method.", "questions": "Please refer to the weaknesses, and I am willing to raise the score if the major weaknesses are addressed.", "limitations": "Limitations have been discussed and no negative societal impact has been identified.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-organized and the presentation is clear. To my knowledge, the idea of aligning gradients of different objectives by maximizing cosine similarity and regularizing the gradient norm is intuitive and novel. The authors also provide a generalization bound for the proposed method, which is a valuable contribution.", "weaknesses": "Major:\n\n(1) Recent empirical and theoretical studies [1] show that simply reweighting the loss of different objectives can match the performance of gradient surgery. When the model is under-parameterized (which may be the case for prompt learning, as the prompt parameters are relatively small compared to the CLIP parameters), simple reweighting is sufficient to fully explore the Pareto front. My question is why the proposed gradient manipulation is better. Can the authors provide theoretical analysis or empirical validation?\n\n[1] Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspective\n\n(2) The backbone CLIP-ResNet-50 is a weak backbone. Please consider conducting some experiments on a CLIP-ViT based model.\n\nMinor:\n\n(1) Missing references. There have been some studies that use gradient alignment for prompt learning, e.g., [3,4]. Please consider adding them to the related works section.\n\n[3] Prompt-aligned Gradient for Prompt Tuning\n[4] Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models\n\n(2) Table 1 in the Introduction is not well explained. It does not specify which dataset and CLIP backbone are used, and the symbols $\\rightarrow C, I, P$ are not explained. In addition, Table 2 should be referenced as Figure 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">1. why the proposed gradient manipulation is better?: \n\nThank you for your insightful question. We would appreciate further clarification on your reference to the paper discussing the limitations of scalarization, particularly in under-parametrized setups, as it seems to suggest scalarization’s inherent inability to fully explore the solution space.\n- Regarding your point of the efficiency of scalarization over multi-task learning (MTL) methods, we discuss this in lines 53-59 of the main text. In particular, we simply re-weight per-task gradients (similar to scalarization) instead of adopting multi task learning methods. In addition, the MTL methods in [33,92] are generic methods and are designed to optimize all objectives simultaneously. In contrast, we focus more on adapting our model to the target task rather than training a model that has universal performance across all domains or profiling the entire Pareto front. This further motivates our choice to use re-weighting source and target objectives' gradients. \n - While studies [33, 92] demonstrate scalarization’s advantages over certain MTL methods like IMTL, MGDA, GradDrop, and PCGrad (ProGrad), they do not establish scalarization’s equivalence with all gradient manipulation techniques. In contrast, gradient alignment has been proven effective in learning invariant features [69], and penalizing gradient norms is widely recognized for enhancing model generalization [102, 18, 4, 90]. Empirically, we conduct some ablation studies on these two components in Table 6, the illustrative example, and gradient similarity experiments, to show that solely using scalarization is not enough.\n\n2. We provide results of our methods using ViT-B/16, ViT-L/14 backbones on OfficeHome in Tables 3 and 5 in the attached PDF, following experimental setups in [R1, R2]. We can observe the superiority of our methods among all baselines while finetuning a small portion of the backbones.\n\n3. Thank you for the suggested references. We will add them in the revision.\n\n4. Caption for Table 1 and reference typos: Thank you for pointing these out, we will fix them in the revision. The experiment in Table 1 is on dataset ImageCLEF with I, P, C representing ImageNet ILSVRC 2012, Pascal VOC 2012, and Caltech-256, respectively, and ResNet50 backbone is adopted. The results in Table 1 in fact are from Table 3, we will add detailed discription for this experiment in the later revision.\n\nWe hope our response could help address your concerns, we are very open to further discussion.\n\n[R1] Wang, Zhenbin, et al. \"Landa: Language-guided multi-source domain adaptation.\", arXiv 24\n\n[R2] Singha, Mainak, et al. \"Ad-clip: Adapting domains in prompt space using clip.\", ICCV 23", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for the valuable and supportive feedback. We appreciate that our paper is recognized for having strong **empirical results** (reviewers JYym, dhjM), a **generalization bound** which adds credibility to our approach (all reviewers), a **novel** method with **clear intuitions** (reviewer aPJr, dhjM, YFTB) and efficiency (reviewer YFTB, dhjM), and a **clear and coherent structure** (reviewer aPJr, YFTB).

In this paper, we tackle the unsupervised domain adaptation (UDA) problem by framing it as a multi-objective optimization task, which is motivated by the strong performance of self-training on the target domain, i.e. minimizing target domain loss on pseudo-labeled data. Furthermore, to promote inherent consensus between domains and encourage model flatness, we propose to align their gradients and penalize their gradnorms with an efficient gradient update procedure, supported by a backup generalization error bound.

Inspired by the reviewers' comments, we have enhanced our manuscript by including additional experimental results, responses, and clarifications, which are briefly summarized as follows:
- Results on S2RDA datasets (reviewer YFTB), ViT backbones (reviewer aPJr), and label-shift setting (reviewer JYym)
- Analysis on hyper-parameter sensitivity (reviewers JYym, YFTB) & computational complexity (reviewer JYym)
- Clarification on the final loss function (reviewers YFTB, dhjM)
- Discussion on failure cases (reviewer JYym) and on the difference between our method and gradient-surgery ones (reviewer aPJr).

We also attach a **PDF** file that contains our main additional results, which we will incorporate in the revision. Except for the complexity benchmark, results for all baselines are directly reported from prior work for a fair comparison. We hope that the new results will help address reviewers' concerns.

Here we would like to clarify the formula for final loss functions, and then present detailed responses separately for each reviewer. 

**Final loss function**
As we cast UDA as a MOO problem, the ideal final objectives, in the case of single UDA, would be  $[L_S^{PGA}(P), L_T^{PGA}(P)]$, where:

$L_T^{PGA}(P) := L_T(P_{sh} - \rho_{ga}\frac{g_{sh,S}}{||g_{sh,S}||.||g_{sh,T}||} +  \rho_{gn}\frac{g_{sh,T}}{||g_{sh,T}||}, P_T + \rho_{gn}\frac{g_T}{||g_T||}),$

$L_S^{PGA}(P) := L_{S}(P_{sh} - \rho_{ga}\frac{g_{sh,T}}{||g_{sh,S}||.||g_{sh,T}||} +  \rho_{gn}\frac{g_{sh,S}}{||g_{sh,S}||}, P_S + \rho_{gn}\frac{g_S}{||g_S||}).$

As mentioned in the paper (e.g. line 58-59, 172-174), we use scalarization method [30], i.e. simply reweighting loss functions with $\lambda$ put on the PGA source objective. As a result, the gradient updates for prompts are 

$g_{sh,T}^{PGA},g_T^{PGA} = \nabla_P L_T^{PGA}(P), \quad g_{sh,S}^{PGA},  g_S^{PGA} = \nabla_P L_S^{PGA}(P),$

$P_S = P_S-\eta g ^{PGA}_S, P_T = P_T-\eta g^{PGA}_T,$

 $P_{sh} = P_{sh} - \eta (g_{sh,T}^{PGA}+\lambda g_{sh,S}^{PGA}).$

However, computing these PGA gradients will trigger the computation of Hessian matrix. Hence, we implicitly calculate them via a practical algorithm that is shown in Eq.(19, 20) in the main text.

Meta-review: This paper proposes an approach for enhancing domain generalization by aligning prompt gradient. While reviewers believe that the paper is well-written and the approach is novely, concerns are raised in terms of the experiments and comparisons. The rebuttal successfully addressed most of the concerns, while there are still doubts in terms of whether the approach can be used in traditional DA approaches. Overall, the AC believes the paper provides a new pespective for DA research and thus recommends acceptance.
Decision: Accept (poster)