Understanding and Robustifying Sub-domain Alignment for Domain Adaptation
Authors: Yiling Liu
Juncheng Dong
Ziyang Jiang
Ahmed Aloui
Keyu Li
Michael Hunter Klein
Vahid Tarokh
David Carlson
Venue: Accepted by TMLR ()

Abstract: In unsupervised domain adaptation (UDA), aligning source and target domains improves the predictive performance of learned models on the target domain. A common methodological improvement in alignment methods is to divide the domains and align sub-domains instead. These sub-domain-based algorithms have demonstrated great empirical success but lack theoretical support. In this work, we establish a rigorous theoretical understanding of the advantages of these methods that have the potential to enhance their overall impact on the field. Our theory uncovers that sub-domain-based methods optimize an error bound that is at least as strong as non-sub-domain-based error bounds and is empirically verified to be much stronger. Furthermore, our analysis indicates that when the marginal weights of sub-domains shift between source and target tasks, the performance of these methods may be compromised. We therefore implement an algorithm to robustify sub-domain alignment for domain adaptation under sub-domain shift, offering a valuable adaptation strategy for future sub-domain-based methods. Empirical experiments across various benchmarks validate our theoretical insights, prove the necessity for the proposed adaptation strategy, and demonstrate the algorithm's competitiveness in handling label shift.

Published in Transactions on Machine Learning Research (03/2025)
Understanding and Robustifying Sub-domain Alignment for Domain Adaptation
Yiling Liu"
yilkngg liu@duke. edu
Program in Computational Biology and Bioinformatics, Duke University
Juncheng Dong"
dunchenggdonng@tdhuke.
Department of Electrical and Computer Engineering, Duke University
Ziyang Jiang
32g95310@mettn
Meta Platforms, Inc.
Ahmed Aloui
ahmed, almniedueeuu@duuue.. edu
Department of Electrical and Computer Engineering, Duke University
Keyu Li
keyu.li@duke. edu
Deportment of Electricol and Computer Engineering, Duke University
Michael Hunter Klein
michaee..leinn,138ddbka edu
Department Electrical and Computer Engineering, Duke University
Vahid Tarokh
wahid. wor.kkttoooh@kkke..kkk edu
Department of Electrical and Computer Engineering, Duke University
David Carlson!
david. Kawid.ccarlso@@duue..
Department of Civil and Envirenamnnall Engineering, Duke University Department of Biostatistics and Bioinformaties, Duke University
Reviewed on OpenReview: https: optp//vveee......./mm///// net/ forun? id= oAzu0gzUUb
Abstract
In unsupervised domain adaptation (UDA), aligning source and target domains improves the predictive performance of learned models the target domain. A common method- ological improvement in alignment methods is to divide the domains and align sub-domains instead. These sub-domain-baset algorithms have demonstrated great empirical success but lack theoretical support. In this work, we establish rigorous theoretical understanding of the advantages of these methods that have the potential to enhance their overall impact on the field. Our theory uncovers that sub-domain-based methods optimize an error bound that at least as strong as non-sub-domain based error bounds and empirically verified to be much stronger. Furthermore, our analysis indicates that when the marginal weights of sub-domains shift between source and target tasks, the performance of these methods may be compromised. We therefore implement an algorithm to robustily sub-domain alignment for domain adaptation under sub-domain shift, offering a valuable adaptation strategy for future sub-domain-based methods. Empirical experiments across various benchmarks vali- date our theoretical insights, prove the necessity for the proposed adaptation strategy, and demonstrate the algorithm's competitiveness in handling label shift.
"Equal contribution.
i "Correpponding author.

---

Published in Transactions on Machine Learning Research (03/2025)
1 Introduction
Supervised deep learning has achieved unprecedented success in a wide range of real-world applications. How- ever, obtaining labeled data may be costly, labor-intensive, and/or time-consuming in certain applications, particularly in medical and biological domains (Lu et al. 2017 Li et al. 20200. To this end, unsupervised domain adaptation (UDA) transfers knowledge from a labeled source domain to a different but related un- labeled target domain Farahani et al.) 2021]. However, efficient UDA is challenging due to the statistical discrepancies wetw two domains, herealter referred to as domain shift (Wang & Deng 2018 Sankara- narayanan et al. 2018 Deng et al. 2019). To address this challenge, much of the UDA research has focused
on reducing the distributional gap between the source and target domains (Shen ct al. 2018 Liu ct al. 2016:
Isola el al. 2017: Tzeng et al.
2015
2017
2020 Ganin & Lempilsky 2015 Ganin et al. 2016 Peng el al.
2018 Recent methods further partition the data into sub-domains and align the sub-domains instead PPin-
heiro 2018 Long et al. 2018 Deng et al. 2019). In the literature, a domain refers to the data distribution of the covariates X (e.g., images from a specific camera), characterized by its feature space X' E X and a marginal probability distribution. Within domain, class refers to specific category (e-g., 'cat', 'dog'). ) Tn this work, mainly use the term sub-domain to refer to the overall domain distribution of each class, such that the k-th sub-domain represents the domain conditioned on class label k, i.e., X|Y k ~ P(X k). This allows us to perform fine-grained alignment based on different classes, viewed through their conditional distributions.
Notably, while one straightforward definition of the sub-domainn the conditional distributions based on the classification label, other strategies for defining sub-domains include cross-domain adaptive clustering Li ct al.] 2021b), classificr-based backprop-inducee weighting (Westfechtel et al. 20230. domain consensus clustering (Li et al. 2021ah, joint learning of domain-invariant features and classiliers (Shi & Sha 2012 and
the use of deep clustering (Gao et al., 2020). These sub-domain-based algorithms have shown substantial empirical success. However, the benefits of sub-domain alignments have not been rigorously justified.
In this work, we present a theoretical analysis to establish that the sub-domain based methods are in fact optimizing a generalization bound that is at least as strong as (and empirically much stronger than) the full-domain--asee objective functions. Our analysis further reveals that when the marginal weights of the sub-domains shift between source and target, the sub-domain based methods can fail. We then present novel CDA algorithm, Domain Adaptation via Rebalanced Sub-domein Alignment (DARSA), that is motivated by our analysis and addresses the case when sub-domain marginal weights shift. DARSA optimizes reweighted classification error and discrepancy between sub-domains of the source and larget lasks. The reweighting scheme follows a simple intuition: important nub-domains in the target domain need more attention. To illustrate the concept visually, Figure highlights the strengths of sub-domain alignment, providing insight into how our method operates and the benefits it brings.
The contribution of our work is two-fold:
Theoretical Contribution:
Our work analyzes and provides a theoretical foundation for sub- domain basod methods in domain adaptation, addressing their previous lack of rigorous understand- ing. Our theoretical framework not only supports our algorithm but can be extended to other methods, contributing to broader impact and value in the field.
Algorithmic Contribution: Our theoretical analysis leads to our algorithm DARSA. DARSA addresses shifted marginal sub-domain weights, which adversely impact existing sub-domain-basee methods. We empirically verify its competitive performance under label shifting on various bench- marks, confirming our theoretical insights and validating the proposed adaptation strategy.
2
Related Work
Discrepancy-based Domain Adaptation. UDA commonly trics to reduce the distribution gap between the source and target domains, One approach to achieve this is discrepancy-base methods in the extract
feature space Tzeng al. 2014 Long
al. 2015 Sun et al. 2016), which often maximum mean
2

---

Published in Transactions on Machine Learning Research (03/2025)
Distance: 0.63
Distance: 21.2
source
target
source target
300
Dist 5.9
Dist 1: 9.5
250
Frequency
D
D
Dim2
Dist 3: 11.4
200
Sub-domain
Sub-domain
Dist 4: 7.7
Dist 6: 7.0
100
Dist 11.9
D
D
Dist 8.1
50
Dist 9: 9.6
25
Diml
(a) ine-dimensional space
(d) Visualization of high-dimensional space
Distance: 0.0018
Distance: 0.0018
W
source target
target
8
WT
Frequency
Frequency
3
WT
3
WT
MNIST
MNIST-M
-16 -10
-25
-10
-1s
-12
-80
1.5
2.
2.5
3.0
(b) Sub-domain
(c) Sub-domain 2
(e) DARSA illustration
Figure 1: Concept.ul overview of our motivation for sub-domain alignment. Listed distances are Wasserstein- distances. (a): Domain Alignment. The source domain (coral) consists of two Gaussians centered at 11.5 and 1.5 with weights 0.7 and 0.3, respectively. The target domain (darkblue) a mixture of two Gaussians centered at -1. and 1.6 with inverse weights. For illustrative purposes, we divide the source domain (coral) and target domain (darkblue) into partitions, i.e., sub-domains D and D3 for the source domain and sub- domains DJ and DJ for the target domain. Note that while our work formally defines sub-domains conditional distributions (e.g., based on class labels where applicable), this initial illustration demonstrates the broader concept of aligning finer-gras mher-grained structures within domains by conceptually partitioning the domains into sub-domains at æ = 0. (b-c): Sub-domain Alignments. The distances between these illustrated sub-domain partitions are trivial compared to the overall domain distance (a). This serves as conceptual evidence for the benefits of nub-domain alignment. (d): MNIST to MNIST-M UDA task. The features are projected 2-D with UMAP. The legend indicates distances between corresponding sub-domains (with red sub-domain indices labeled in the figure), and the sub-figure title shows the overall domain distance. Here, the sub-domains are defined based the class (here digits), resulting in the depicted aligument. These sub-domain distances are small compared to the overall distance given at the top. (e): DARSA illustration with w indicating target sub-domain weights, showing DARSA's applicability under label shifting. This demonstrates our approach using sub-domains defined by digits.
discrepancy (MMD) Borgwardt al. 2006 While MMD a well-known Reproducing Kernel Hilbert Space (RKHS) metric, is weaker than the Wasserstein-1 distance (Lu & Lu 2020 Therefore, we use Wasserstein-1 distance in our work. Futhermore, many discrepancy-based methods enforce the sharing of the first few layers of the networks between the source and target domains (HassanPour Zonoozi & Seydi
2023 In contrast, our method allows a more flexible feature space.
Adversarial-based Domain Adaptation. Adversarial-basee domain adaptation methods aim to encour-
age domain similarity through adversarial learning (Shen et al. 2018 Liu et al. 2016: Isola et al. 2017
Tzeng al. 2015 2017
2020) Ganin & Lempitsky 2015 Ganin al. 2016 Peng et al. 2018 Hotfman et al.] 2018). These methods are divided into generative methods, which combine discriminative models with generating process, and non-generative methods, which domain confusion loss to learn domain-
3

---

Published in Transactions on Machine Learning Research (03/2025)
invariant discriminative features (Wang & Deng 2018/. However, many existing algorithms fail to align multi-modal distributions under label shifting scenarios. Additionally, training adversarial networks can be challenging due to mode collapse and oscillations (Liang et al. 2018
Sub-domain-bes based Domain Adaptation. The use of sub-domain adaptation has proven effective in
aligning multi-modal distributions, enhancing performance across various tasks (Deng et al. 2019 Long et al.
2018
Pinheiro 2018
Shi & Shap 2012 Jiang et al. 2020; Snell al., 2017 (Deng et al. 2019) introduces the
Cluster Alignment with a Teacher (CAT) approach that aligns class-conditional structures across domains.
Long et al. 2018 offers conditional adversarial domain adaptation, enhancing alignment through classifier
predictions. Pinhhiro 2018) proposes an unsupervised domain adaptation approach based on similarity learning, wherein classification is conducted by computing similarities between target. domain images and prototype representations of each category. On the other hand, (Shi & Sha, 2012) introduces a method that concurrently learns domain-invariant features and classifiers. (Jiang et al.) 2020) elucidates a sampling- basod implicit alignment technique, addressing concerns of class imbalance. (Snell et al. 2017) presents prototypical networks designed for few-shot. classification, employing distances to class protolype representations for the process. While these methods have demonstrated empirical success, a detailed theoretical perspective on the benefits of incorporating sub-domain structures has yet to be fully explored. Our work aims to complement these existing methodologies by providing a comprehensive theoretical understanding of the advantages inherent in these structures.
Theoretical Analysis of Domain Adaptation. Many existing domain adaptation methods are inspired by generalization bounds based on the H-divergence (Ben-David et al.) 2006]. The H-divergence (Ben- David et al. 20060 is modified version of the total variation distance (Ta) that restricts the hypothesis to given class. These generalization bounds can be estimated by learning domain classifier with finite Hapnik-Chervoneee
(VC) dimension. However, this results in a loose bound for most neural networks Li et al. 2018). In our method, we use the Wasserstein distance for two reasons. First, the Wasserstein- distance is bounded above by the total variation distance (Ben-David et al. 2010). Additionally, the Wasserstein-1 distance is bounded above by the Kullback-Leible divergence (a special case of the Rényi divergence when a goes 1 (Fournier & Guillin 2013)), giving stronger bounds than those presented by Redko et al (Redko et al., 2017) and Mansour et al Mansour et al., 2012). Additionally, the Wasserstein distance has stable gradients even when the compared distributions are far apart (Gulrajani et al. 2017
Expanding Theoretical Insights into Domain Adaptation. Our work contributes to the understanding and improvement of sub-domain aligmment methods, a type of popular but yet to be rigorously investigated domain adaptation method. In contrast to our work, (Mansour et al. 2009 studies the adaptation perfor- mance of various loss functions and models; (Dhouib et al.] 20200 focuses on the margin violation rate; (Wang
et al.
2022)
addresses the problem of learning features that align with human understanding of data; (Zhang
et al.
2019
proposes generalization theory for classifiers with scoring function and margin loss; (Germain
et al.
2016
studies the generalization theory for the weighted majority vote framework; (Bancchad et al.
2021
Albuquerque et al. 2019 Zhao et al. 2018 focus on the setting with multiple source domain.
3
Preliminaries
Assume a labeled source dataset (1rs, Ns from source domain Xs with distribution Ps and an unlabeled target dataset { has N₂ labeled samples, and the target dataset has Ny unlabeled samples. We assume that the samples a' E X C Idd and 2i E X S Rd are independently drawn from Ps and Pr, respectively. The goal is to learn a classifier f(x) that predicts labels {uf N for the target dataset. We further assume that Ps and Pr are probability densities of Borel probability measures in the Wasserstein space Pi (Rd), i.e. the space of probability measures with finite first moment.
Nr from a target domain X2 with distribution Pr. The source dataset
Sub-domains. We assume that both Xs and XT are mixtures of K sub-domains. In other words, we have P3 P = P and Pr = -k= Pf where we use Pš and P to respectively represent the distribution of the k-th sub-domain of the source domain and that of the target domain, and w /w/ correspond to the weights of each sub-domain. Note that ws = [ws...., w and wr = [wf...., .,, w belong to AK (the K-1 probability simplex). It is straightforward to define sub-domains as conditional distributions, such that the
4

---

Published in Transactions on Machine Learning Research (03/2025)
k-th sub-domain is represented as P3 P(XslYs k) and Pf- P(Xr|Yr =)) where Ys and Yr are the source and larget labels, respectively. Note that sub-domain & refers to all the covariates distribution where the class label is k, not just the class itself. However, we note that the framework presented in this work is applicable across various sub-domain methods, that is, any approach assuming that the covariates are mixtures of distributtin with P(X) = Ex w (X) where pk is the component distribution. While this work focuses on the cases where P* (x) P(X = k), all of our theoretical results and algorithms casily extend to other sub-domain methods with different sub-domain definitions.
Probabilistic Classifier Discrepancy. For distribution D, we define the discrepancy between two functions f and g as:
7p(199) = EZ-D -1](=)
We use gr and gs represent the true labeling functions of the target and source domains, respectively, We use (f()) = Yrs (f. gs) and (rr(f) = (f.)) to respectively denote the discrepancies of hypothesis f to the true labeling function for the source and target domains.
dual representation of the Wasserstein-1 dis- tance Villani 2009 between two distributions Ps and Pr is defined as
Wasserstein Distance. The
Wi Wi PT) SuPlisI E2 (=))= He- [f(x)],
where the supremum is over the set of 1-Lipschitz functions (all Lipschitz functions f with Lipschitz constant L S 1. For notational simplicity, we use X2) to denote distance between the distributions of any pair of random variables X, and X2. For instance, (x{)()s,, ((X))) demotes the Wasserstein-1 distance between the distributions of the random variables (Xxs) and ((Xr) for any transformation Φ.
4 Understanding Sub-domain-based Methods
We now present our theoretical analysis of sub-domain- based methods. We first present a generalization bound for domain adaptation that closely related to existing work, and then establish a novel general- ization bound for sub-domain-based methods, aligning with the objectives used by these existing methods. Furthermore, demonstrate that the sub-domain-based generalization bound is at least strong as the non-sub-domaain-besed generalization bound, which establishes rigorous theoretical understanding of the advantages of these methods. Our analysis also uncovers that when the marginal weights of sub-domains shift between the source and the target task, sub-domain methods can potentially fail.
4.1 Generalization Bounds for Domain Adaptation
Before presenting our novel theoretical results about sub-domain- based domain adaptation, we first present an upper bound closely related to Ben-David et al. (2010) and Li et al. (201)) Theorem A.8. It worth
noting that we use the Wasserstein-1 distance in our analysis, as it provides stronger bound than the total
variation distance Redko al. 2017 employed by [Ben-Davd al. (2010) The proof of Theorem is
deferred to the Appendix A.4
Theorem 4.1 (Full Domain Generalization Bound). For a hypothesis x [0, 1].
(f( S 7s()) + (A (Ps, +
(1)
where 7* min rT(), HI a hypothesis class included in the set of XH- Lipschitt functions, and the fEB
true functions gr and gs both X-Lipschitz functions (as defined in Appendix A.1
Remark 4.2. The upper bound in Theorem 4.1 consists of three components: (i) (f( is the performance
of the hypothesis on the source domain, (ii) (Pss Pr) is the distance between the source and the target
domains, and (iii) 7* is a constant related to the difference between the source and the target problems that
cannot be addressed by domain adaptation. For succinctness and clarity of the following analysis, we assume without loss of generality that + Ry < 1, simplifying the bound to
rr( S 7s(s) WPsss Pr) +
(2)
5

---

Published in Transactions on Machine Learning Research (03/2025)
Numerous works attempt to solve the domain adaptation problem by designing algorithms that minimize
similar generalization bounds to the one in equation 2 e.g., Theorem in Ben-David et al. (2010). These
approaches consist of two components: (i) a mapping + : X'' + H. that transforms the original problem by
embedding X3 and Xy into a shared hidden space N. and (ii) a hypothesis h - [0, 1] for prediction. Since
yr o +) = (x())) (h), with Theorem
4.1 we have a generalization bound of the function h o $ X' - [0, 1]
on the original target problem:
rr o b) = (h))) S 7M(xs) (1) + (xx))s). +(X))) + +
(3)
If the distance between & (X5) and ((T)), ie., (Xx) (XT)), is close and the classification error of h on the transformed source problem, i..,, (o()s) (h), remains low, then the performance of the hypothesis h & on the original larget problem can be guaranteed. This motivation has led to variely of domain adaptation frameworks with objectives of the following format:
min - (h) (h) + ((((Xs),0(X)))) (0,1)
(4)
where (h) is the classification error of h on the transformed source problem, D is a distance between distributions and a is the balancing weight. In this work, we use Wasserstein-1 distance.
4.2 Analysis of lub-domain-basse Methods
We first present several results that will be used to build the main theorem. These results themselves may be of interest.
First of all, Theorem 4.1 directly leads to the following proposition:
Proposition 4.3 (Individual Sub-domain Generalization Bound). For k E {1...., (),, where K represents the lotal number of distincl sub-domains, for sub-domain X with distribution Pÿ and X with distribution Pt. it holds any f H. that
*() S +(()+++++ W, (P P
(5)
where ()((//(()) is the performance of the hypothesss on the sub-domain X ( minyer T$(s)+ (f), H. is hypothesis class included in the set of λи -ipschhit functions, the true functions gr and gs are both X-Lipschitz functions, and X + λи 1.
+
The second result below shows that the classification error of any hypothesis f on a domain can be de- composed into weighted sum of the classification errors of f on its sub-domains (proofs deferred to the Appendix A.5
Lemma 4.4 (Decomposition of the Classification Error). For any hypothesis f E EH,,
7s( = ΣK
(f)
(6)
With above results, we present a generalization bound with sub-domain information (proofs deferred to the Appendix A.6 Theorem 4.5 (Sub-domain based Generalization Bound).
rr(f) < EK
W + K w
(7)
In porticular, in a balanced domain adaptation setting where for all k, w = w we have that
Tr(s) S 7s( + ΣK_ W Pb) + K "
(8)
Remark 4.6. Note that the format of the RHS of equation reminiscent of the objectives used by the majority of the sub-domain-based methods.
We next show that, under reasonable assumptions, the weighted sum of distances between corresponding sub-domains of the source and target domains at most as large as the distance between the marginal distribution of the source domain and that of the target domain.

---

Published in Transactions on Machine Learning Research (03/2025)
Theorem 4.7 (Benefits of Sub-domain Alignment). Under the following assumptions:
A1. For all k, Pš P are Gaussian distributions with mean m / /m and coveriance D/ /2r.
A2. Distance between the poired source-taryet sub-domain is less or equal to distance between the non-paired source- source-target sub- domain, i.e., Wi(P5, Pf) I Wi (P(Ps, (Ps, Pf') for l + k..
A3. There exists small constant e > O, such that max (tr(2i)) S and max (tr(2))) S.. Then the ISkSk
1SKŞK
following inequality holds:
Σ W Pf) S Wi(Ps, Pr) + des
(9)
where de is AVE. In particular, when w = w for all k,
Σ W u
(P'. Pf) < Wi Pr) 8c.
(10)
Proof. Note that ws = jws..... w and wT = |wy...... w/i belong to K (the K-1 -1 probability simplex). [[ Wr) represents the simplex A Kxk with marginals wg and WT. With w II(ws, wT), we can write out w as Ex= Wk,k's then based on assumption A.2, we have:
K
K = Σ
WAP
W Wi(PS, Pt)
k= ===
Wk W 1 (P), (P PK
==
Thus we have (Ps, Pr) defined in Appendix A.7
K
K
Σ w WP S
wkek W1 (P's PK
min ww..(w..TT)
(11)
(Ps
Also we prove in Theorem A.10 that:
& 4VE.
Then we conclude our proof and show that:
K
w WP pt) &
(Ps, - +6.
(12)
Remark 4.8. In Appendix B we provide empirical evidence to verify that these assumptions are satisfied on real-world datasets. We note that the assumption of a Gaussian distribution for X is not unreasonable since it is often the result of complex transformation, 4 and the Central Limit Theorem indicates that the outcome of such transformation is approximately normally distributed under regularity assumptions (please see Appendix B.1 for empirical evidence).
Remark 4.9. & a constant dependent only on the variance of the features but not the covariance bet weew features in different dimensions. Moreover, the inequality holds empirically without d, as demonstrated in Figure g as well Figure and Figure in Appendix F.2
4.3 Challenges of Imbalanced UDA
Theorem shows that the objective function of sub-domain methods at least as strong as the objective function of domain alignment methods, explaining its improved performance. However, if the marginal
7

---

Published in Transactions on Machine Learning Research (03/2025)
weights of the sub-domain shifts, i.e., w + wh, the inequality in equation 10 is not likely to hold and the framework can collapse. One such example is the scenario of shifted label distributions where w and w (class weights for target and source domains) can be vastly different.
To overcome this, we propose to minimize an objective with the simple intuition that important sub- rub-domains the target domain need more atlention. With this motivation, we propose the following objective function for UDA with shifted label distribution:
C()= K E
(13)
In particular, L reweighs the losses of sub-domains so that the sub-domain with more weight the target domain can be emphasized more. We next prove that through the proposed approach (proofs deferred
the Appendix A.12 we can again obtain a sub-domain- based generalization bound that is at least as strong the full domain generalization bound without the sub-domain information.
Theorem 4.10. Let H = {flf X -
10, denote hypothesis space. Under the assumptions in Theo-
rem 7 for any I E H. such that:
w S Ek= wg)((().
(14)
then we have Eizn w ( S Further, let
ealf) = Ekn, w + E W Pt) + Ekm wh (7*)*
denote the tab-dommin--basee generalization bound and let
{f( = 7s( + Wi Pr) 7
denote the generalization bound without any sub-domain information, we have,
<c( S {f( 6
Remark 4.11. In Section 6.1 and Appendix P2 we provide extensive empirical evidence to establish that equation 14 can easily hold, as the left. hand side is the optimization objective. Moreover, in these sections, we offer empirical evidence to further verify the value of this theoretical result by showing that our proposed bound empirically much stronger than the existing one.
Inspired by our analysis, we propose a framework, Domain Adaptation with Rebalanced Sub-domain Align- ment (DARSA), for imbalanced UDA, a special case of the sub-domain weight shifting scenario where the class weights of the target domain shifts from that of the source domain.
5 Methods
In DARSA, we divide the source domains into sub-domains based class labels, and divide target domains into sub-domains using predicted class labels (serving pseudo labels, which have shown success in previous
rescarch (Deng et al. 2019 Lee et al. 2013 for unlabeled target domains. Motivated by Theorem 4.10 the framework of DARSA, shown in Figure is composed of a source encoder f parameterized by 0 a target encoder f parameterized by of, and a classifier fy parameterized by 0y. The pseudo-code for DARSA can
be found Appendix D
The objective function of DARSA is defined as follows:
mingy og Xy + ApLD Lc.
(15)
where Ly, LD. Lo are losses described below with relative weights given by Xy and Ap.
Weighted source domain classification error Ly. The weighted source domain classification error in Theorem 4.10 be further expressed
Ek k= u+++t(()= = ΣK w Ps(x/c= b)]l(()))) - (z)
K
K
(16)
= -E w/ (Pol==kle)Psaa) f(() - gs(x))dz = -k w Ps(c=k)
Egnn, wz(x))f(x)) gg)(
wg
8

---

Published in Transactions on Machine Learning Research (03/2025)
his
C
Encoder
Xs
Ys
fE
Classifier fy
As
P
Encoder
X
fi
Ly
hT
The DARSA framework. Orange lines representing the clustering loss Lc, green lines indicating domain discrepancy CD, and purple lincs indicating source classification loss LY.
Figure 2:
where variable e represents class, why = Pr(c = k), S = Ps(c = wz()) = Ps(c klz). We set Ps(c = k|z) = 1 only when data point x in class k, otherwise Ps(c = k|z) = w can be set to the marginal source label distribution, and w can be estimated from the target predictions. From equation 16 C0 0 is defined as:
Cr NE Exexxs 11' 1 w (l'
where fy the predicted label and can be any non non-negative loss function (e.g., cross-entropy loss for classification tasks).
Weighted source-target subdomain discrepancy Lp. The weighted source-target domain discrepancy
in Theorem 4.10 can be further expressed as:
LD 07, By) EK W P+)= E (F
(17)
where r are source samples with labels ys = k, and o are target samples with predicted labels dr = We leverage the Sinkhorn algorithm (Cuturi 2013
approximate the Wasscrstein metric.
Clustering loss Lc. The clustering loss Lc XcCintra Cinter is comprised of two components: the intra-clustering loss, Lintro, and the inter-clustering loss, Linter. The role of Cintra is to satisfy the assumption A.3 in Thcorem 4.7 It encourages embeddings of the same label to cluster tightly together, while also pushing embeddings of different labels to separate by at least a user-specified distance, m [uo et al. 2018). The inter-clustering loss Linter further enhances sub-domain alignment by aligning the centroids of source sub-domains with those their corresponding target sub-domains in the representation space. We define Lintra and Cinter as follows:
Lintra Sintra(05, 08, 0y) Lintrooff
(fgss + Lintra (FT
(18)
Lintra (x())) N/2 Lij== Dig + (1 8,,) max (0, m - D.y)
Linter (0s OT, 0r) 1 (C( C((E(())))
(19)
where N represents the number of samples in the domain X' and C(-) calculates the centroids of the sub- domains, & = only if x, and zj have the same label; otherwise, dy = O. We use the ground truth label or the predicted label if x is in source domain or target domain, respectively. m pro-defined distance controlling how separated each sub-domain should be. Di, = (lf - ff,(z, represents distance between 2; and x---

---

Published in Transactions on Machine Learning Research (03/2025)
6 Experiments
In this section, we verify our thcoretical results and assess DARSA's efficacy through rcal-world experiments. We begin by empirically confirming the superiority of the sub-domain-based generalization bound (Theo- rem 4.10 in Section 6.1 Then, we verify that the assumptions for Theorem 4.10 are empirically satisfied on real-world datasets (details in Appendix B Next, we demonstrate the vital role of subdomain weight re- bolaancning in Section 6.2 and show DARSA's robustness to minor weight estimation discrepancies. Lastly, given that, our theoretical analysis guarantees that DARSA should have competitive performance in scenar- ios where the number of classes is not overwhelming, we evaluate DARSA on real-world datasets with this property. Comparing with other stale-of-hhe-rr, UDA baselines, we validate our theoretical analysis and demonstrate DARSA's effectiveness in real-world applications, including those in medical settings. Notably, on the Tail Suspension Test (TST) dataset [Gallagher et al. 2017) dataset for multi-class classífication of behavioral states (a medical//nuueee neeroscince use case), DARSA ellectively handles label distribution shifts and learns robust embeddings across genotypes. Meanwhile, our experiments on the Digit datasets and VisDA-
2017 dataset (Peng et al.
2017 further confirm DARSA's robustness across diverse domains (see Sections H for detailed results).
6.3 and Appendix F
G
Experiments the Digits Datasets. In our Digits datasets experiments, we evaluate our performance
across four datasets: MNIST (M) (LeCun et al. 1998 MNIST-M (MM) (Ganin et al. 2016 USPS (U), and SVHN (S), all modified to induce label distribution shifts. Here, the parameter a denotes the class imbalance rate, representing a ratio such as 1:a and a:1 for the oddevvn distribution in the source and target datasets, respectively. Weak and strong imbalance correspond to a = 3 and a = 8. For comprchensive details, refer to Appendix
Experiments on the TST Dataset. We use the Tail Suspension Test (TST) dataset GGallaher et al, 2017] of local field potentials (LFPs) from 26 mice with two genetic backgrounds: Clock-A19 (a bipolar disorder model) and wildtype. This dalaset is publicly available (Carlson et al. 2023 Our study involves two domain adaptation tasks, predicting the current condition- - home cage (HC), open field (OF), or tail- suspension (TS) from one genotype to the other. We subsample datasets to induce label distribution shifts with imbalance rate- = 2. For comprehensive details, refer to Appendix C
Experiments on the VisDA-2011 Dataset. We further evaluate DARSA on the large-scale VisDA-2017 dataset (Pnng et al. 2017). a challenging syuthetic-to-res benchmark with 12 categories. We impose a strong label distribution shift (a 8), i..., eightfold class imbalance between source and target. For comprehensive
details, refer to Appendix H
6.1 Empirical Analysis of our Proposed Generalization Bound
We first verify the pivotal result in Theorem 4.10 that the sub-domain based generalization bound is at least as light as the the non-sub-domain bound. We empirically evaluate the proposed bound on the Digits datasets under weak imbalance. As shown in Figure our empirical results demonstrate that the addomminnaaase generalization bound in Theorem 1.5 is empirically much stronger than the non-sub-dommain-based bound in Theorem corroborating experiments on the other UDA lasks in the Digits datasets under weak and strong imbalance also support this claim, and full results are in Appendix
insights for the effectiveness of sub-domain based methods. Additional
6.2 Importance of Re-weighting
Here, we experiment on the Digits datasets under weak imbalance to demonstrate the importance of (1) weights re-weighting and (ii) the accuracy of target sub-domain weights estimation. We compare DARSA with one variation of DARSA which employs uniform weights for all sub-domains and another variation which swaps sub-domain weights estimation of source with target. We also include two other basclines where the weights of the targel domain are chosen to be deviating from the truth. Specifically, we compare DARSA with the following configurations:
The code to replicate all experiments is available at: http:://gthaab.cc///lllggggaauip
10

---

Published in Transactions on Machine Learning Research (03/2025)
Sub-domain - Ey
0.25
Reweighted by
30
Full-domain Lp
Unweighhtd br
0.20
Losses
Losses
0.15
20
0.10
0.05
0
1000
2000
3000
1000
2000
3000
Epoches
Epoches
(a) Domain Discrepancy
(b) Source Classification Loss
Figure 3: For MNIST to MNIST-M task under weak imbalance. (a) Compare the domain discrepancy term (Cp) in our proposed bound to that in Theorem our proposed bound to that in Theorem
(b) Compare the source classification term (Ly) in
DARSA: Full algorithm where weights are inferred.
DARSA Oracle: Utilizing true values of w
20% divergent from true values. DARSA Small Divergence: Setting w
DARSA Large Divergence: Setting w be 50% divergent from true values.
Swapping w with w effectively flipping importance weighting. DARSA Uniform: Assigning uniform weights for all sub-domains.
DARSA Flip:
The results of these experiments are in Table 1 We verify the importance of subdomain weights re-balancing by showing that the performance of DARSA degrades significantly without the weights re-balancing or wrong sub-domain weights, further corroborating the value of our insights. Aditionally, while the oracle case provides the best performance, inferring the weights the DARSA algorithm provides nearly the same quality of predictions. In addition, we found our method, DARSA robust to minor divergence in weights estimation and varying imbalance rates.
Table I: Evaluation the importance of re-weighting on Digits datasets under weak imbalance. Performance is measured by prediction accuracy (%) on the target domain.
M MM
MM
M
U U
M
S + M
DARSA Oracle
96.2
98.4
92.7
92.6
DARSA Uniform
67.9
96.6
75.9
71.7
DARSA Small Divergence
95.6
98.3
91.4
92.4
DARSA Large Divergence
85.0
98.2
85.2
DARSA Flip
55.7
65.7
57.4
65.7
DARSA
96.0
98.8
92.6
90.1
6.3 DARSA on Real-world Datasets
We now compare DARSA with many competing algorithms across three datasets. Full details on the experi- ments, the rationale behind the choice of competing algorithms, and their settings can be found in Appendix and for the Digits, TST, and FisDa-2017 datasets, respectively.
Digits. Results shown in Table 2 demonstrates DARSA's 's competitiveness in handling label shifting. Addi- tionally, DARSA performs well with varying imbalance rates (Appendix Table and competes favorably in scenarios without. label distribution shifts (Appendix Table 7
TST. As demonstrated Table 3 DARSA achieves competitive performance on this biologically relevant task. To ensure the robustness of our results, we conduct runs with different random sceds, the details of which are provided in Table T0 We report both the mean and standard deviation of these runs, offering
11

---

Published in Transactions on Machine Learning Research (03/2025)
Table 2: Summary UDA results on the Digits datasets with shifted label distribution, measured terms of prediction accuracy (%) on the target domain.
M
MM
MM
M
M
M
MM
MM
M
U
M
S
a
a
=
a =
==
8
a=
a
a
DANN Ganin et al.
2016
63.1
93.0
59.8
61.9
61.1
90.2
49.1
57.3
DSN [Bousmass et
22016
62.3
98.4
59.9
15.2
57.5
95.3
30.3
17.8
ADDA Uzeng
2017
88.2
90.7
44.8
42.4
47.9
89.4
45.7
45.3
PixcIDA Bousmalis
al.
72.0
68.0
81.0
95.6
29.2
CDAN
Long et al.
2018
58.7
96.0
42.0
38.3
37.1
90.6
34.8
32.5
WDGRL
Shen et a. 2018
60.4
93.6
63.9
64.3
22.3
91.4
46.7
52.2
MCD
Saito al.
2018
58.1
98.2
74.6
75.5
37.4
97.5
76.1
66.7
CAT
Deng et
2019
54.1
95.4
81.0
65.8
48.9
93.8
61.3
62.2
MDD
hann et
2019
97.7
82.3
62.4
47.6
93.6
83.2
DRANet
(Lee el al.
2021
95.2
97.8
86.5
40.2
63.3
96.1
54.2
31.3
Source Only
47.9
91.5
40.8
53.7
39.6
88.4
27.8
47.2
DARSA
96.0
98.8
92.6
90.1
78.8
97.3
87.9
83.5
Table 3: Summary of UDA results the TST datasets with shifted label distribution, measured in terms of prediction accuracy (%) on the target domain.
DANN
WDGRL
DSN
ADDA
CAT
CDAN
Source only
DARSA
Clock A19 to Wildtype Wildtype to Clock-A19
79.9
79.6
79.4
75.1
77.3
75.0
73.8
86.6
81.5
79.5
80.9
72.6
78.6
73.6
70.4
84.8
reliable estimate of the model's performance and its reproducibility. For comprchensive experimental details, refer to Appendix G
VisDA-2017. As shown in Table 12 in Appendix H
DARSA achieves 72.2% average accuracy, outperform-
ing prior methods under this strong imbalance setup (a = 8). These results confirm DARSA's robustness to more complex domains.
Ablation. To assess the impact of each component within our objective function (Section 5 we conduct ablation study under weak imbalance. As demonstrated in Table T the ablation analysis confirms that each component in our objective function contributes to the overall performance. Therefore, we recommend the use of all components for optimal results. In addition, we have included feature space visualizations in Appendix C and Appendix Figure y which demonstrate that the learned representation of DARSA has improved separation when using all the components, supporting the effectiveness of the proposed objective function.
7 Limitations
However, despite demonstrating empirical improvements over existing methods, DARSA may be less effective in applications with large number of class labels (e.g., 1,000). In such cases, aligning the corresponding sub-domains (classes) becomes significantly more challenging because the alignment relies on pseudo-labels, which tend to be less reliable when the label space is large. Moreover, effective and stable alignment of sub- domains requires an increased batch size for SGD. This can cause difficulty when the computational resource is limited. On the other hand, DARSA may not achieve the best performance when the domain shift is extreme (e.g., when the probability of observing certain sub-domains is extremely low or even zero). In these scenarios, accurately estimating the sub-domain Wasserstein distance and marginal weights is difficult due high variance from the scarcity of samples. However, note that these limitations not unique to DARSA: most UDA methods rolying pscudo-labels face similar challenges under large label spaces or extreme domain shifts. In terms of the theoretical results, our theorems assume that sub-domain embeddings are well-separated and that each sub-domain in the source corresponds to the closest sub-domain in the target. This requirement equivalent to requiring that both the source and target domains themselves must be vell-separated (i.c., classified accurately with high probability). Hence, when either domain is inherently difficult to classify, theoretical guarantees may not hold, and the benefit. of DARSA is less
12

---

Published in Transactions on Machine Learning Research (03/2025)
Table 4: Ablation study results. The table presents different configurations and their corresponding predic- tion accuracy (%) across four experimental setups.
Experiment
Ay
Ap
Ag
Accuracy
0.4
0.35
0.9
1
96.0
0.35
0.9
1
61.3
M MM
0.4
0.9
72.5
0.4
0.35
0
61.9
0.4
0.35
0.9
0
33.5
0.5
98.8
0.5
MM + M
1
98.4
0.5
1
15.0
0.5
92.6
0.5
65.9
U M
85.8
0.5
0
1
76.2
0.5
1
0
58.4
0.95
0.11
0.11
0.3
90.1
0.11
0.11
0.3
77.9
S - M
0.95
0
0.11
0.3
86.1
0.95
0.11
0.11
64.3
significant. Looking ahead, more exlensive empirical validations can deepen our understanding of DARSA's 's performance. Examining larger and more diverse real-world benchmarks (e-g., additional medical imaging repositories with multiple disease types) could reveal how DARSA handles distinct sub-domain structures across varied tasks. Furthermore, performing multi-sced runs and employing bootstrapping techniques obtain error bars and confidence intervals experiments would rigorously quantify the robustness. By addressing these directions, spectrum of domains.
better characterize DARSA's strengths and limitations across broader
8 Conclusion
Sub-domain-based algorithms have demonstrated considerable empirical success across various applications in domain adaptation. However, comprehensive theoretical understanding of their advantages had been elusive. This work addresses this gap and presents a substantial contribution by providing rigorous theo- retical perspective on the benefits of sub-domain-basee mothods, thereby potentially cnhancing their overall impact in the field. Moreover, our analysis leads to an algorithm DARSA with improved robustness to the shift of sub-domain weights and label distributions. Additionally, our framework can be extended to data integration and causal effect estimation by reframing these tasks distribution alignment problems.
9 Reproducibility Statement
Rigorous definitions and complete proofs of our theoretical analysis are included in the Appendix with empirical evidence evidennce to verify assumptions in Appendix B The code to replicate all experiments is available at:
repo Full details of each experiment, including data sampling,
hyperparameter selection, and architecture, are in Appendix F Appendix G and Appendix H The MNIST, BSDS500, USPS, SVHN, and VisDA-2017 datasets are publicly available with an open-access license. The Tail Suspension Test (TST) dataset (Gallagher et al. 2017) is available to download at httts::/researchh repository. duke. cluster equipped with NVIDIA GeForce RTX 2080 Ti that has memory capacity of 11019MiB.
for free under a Creative Commons BY- 4.0 International license. The experiments are conducted on computer
NC
13

---

Published in Transactions on Machine Learning Research (03/2025)
Acknowledgments
Rescarch reported in this publication was supported by the National Institutes of Health under Award Number R01MH125430. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We acknowledge the generous support of Rhodes Family, who provided partial funding for this work, as acknowledged by Ahmed Aloui, Juncheng Dong and Vahid Tarokh.
References
Ali Abodi, QM Wu, Ning Zhang, and Farhad Pourpanah. Euda: An efficient unsupervised domain adaptation via self-supervised vision transformer. arXiv preprinl arXi::2407.21311, 2024.
Isabela Albuquerque, João Monteiro, Tiago H Falk, and Ioannis Mitliagkas. Adversarial target-invariant representation learning for domain generalization. arXio preprint orXi::1911.0080, 8, 2019.
Jason Altschuler, Jonathan Niles- Niles-Weed, and Philippe Rigollct. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. Advances in newral information processing systems, 30, 2017.
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. JEEE transactions on pattern analysis and machine intelligence, 33(5):898-95) 916, 2010.
Eylan Bakshy, Max Balandat, and Kostya Kashin. Open-sourcing ax and botorch: New ai tools for adap- tive experimentation. URL https://ai. facebook. com/blog/ opwn-sourrcngggar ar aa--aa------------------- nootorch-new-a--toooe adimtiiee-cperrimentttio
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulosza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):151 175, 2010.
Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. Domain gener- alization by marginal transfer learning. The Journal of Machine Learnving Research, 22(1):46 100, 2021.
Karsten M Borgwardt, Arthur Gretton, Malte J Rasch, Hans- Kriegel, Bernhard Schölkopf, and Alex J Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22 (14):e49-657, 2006.
Konslantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain scparation networks. Advances in neural information processing systems, 29, 2016.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsuper- vised pixel-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3722-3731, 2017.
David Carlson, Sunil Kumar, and Kafui Dzirasa. Multi-region local field potential recordings during a tall-suspension test. Duke Research Data Repository, 2023.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Julie Delon and Agnes Desolneux. A wasserstein-type distance in the space of gaussian mixture models.
SIAM Journal on Imaging Sciences, 13(2):936-970, 2020.
Zhijic Deng, Yucen Luo, and Jun Zhu. Cluster alignment with teacher for unsupervised domain adaptation. 2019.
In Proceedings of the IEEE/CVI international conference computer vision, pp.
Sofien Dhouib, levgen Redko, and Carole Lartizien. Margin-aware adversarial domain adaptation with optimal transport. In International conference machine learning, pp. 2514-2524 PMLR, 2020.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Abolfazl Farahani, Sahar Voghoci, Khaled Rasheed, and Hamid R Arabnia. A brief review of domain adaptation. Advances in dala science and information engineering, pp. 877-894, 2021.
Nicolas Fournier and Arnaud Guillin. On the rate of convergence in wasserstein distance of the empirical measure. Probability Theory and Related Fields, 162(3):707-74) 2015.
Neil Gallagher, Kyle R Ulrich, Austin Talbot, Kafui Dzirasa, Lawrence Carin, and David E Carlson. Cross- spectral factor analysis. Advances in neural information processing systems, 30, 2017.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pp- 1180-1189. PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Nomain-adversariaa training of neural networks. The journal of machine learning research, 17(1):209-20300 2016.
Boyan Gao, Yongxin Yang, Henry Gouk, and Timothy M Hospedales. Deep clustering for domain adapta- tion. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 4247-4251. IEEE, 2020.
Pascal Germain, Amaury Habrard, François Laviolette, and Emilie Morvant. A new pac-bayesian perspective domain adaptation. In International conference machine learning, pp. 859-865 968.868. PMLR, 2016.
Ishaan Gulrajani, Faruk Ahmod, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. Advances in neural informalion processing systems, 30, 2017.
Mahta, HassanPour Zonoozi and Vahid Scydi. A survey on adversarial domain adaptation. Neural Processing Letters, 55(3):2429 2469, 2023.
Judy Hoffiman, Eric Tzeng, Taesung Park, Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998 Pmlr, 2018.
Phillip Isola, Yun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditiona adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Xiang Jiang, Qicheng Lao, Stan Matwin, and Mohammad Havaci. Implicit class-conditioned domain align- ment for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 4816 4827. PMLR, 2020.
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaplation. In Computer Vision-Foe ECCV 2020: 16th Europeon Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XXI 16, pp. 464 480. Springer, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to docu- ment recognition. Proceedings of the IEFE, 86(11):2278 2272-2244, 1998.
Dong- Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Seunghun Lee, Sunghyun Cho, and Sunghoon Im. Dranet: Disentangling representation and adaptation networks for unsupervised cross-domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15252-152611 2021.
Benjamin Letham, Brian Karrer, Guilherme Ottoni, and Eytan Bakshy. Constrained bayesian optimization with noisy experiments. 2019.
Guangrui Li, Guoliang Kang, Yi Zhu, Yunchao Wei, and Yi Yang. Domain consensus clustering for uni- versal domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9767-9766, 2021a.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Jichang Li, Guanbin Li, Yemin Shi, and Yizhou Yu. Cross-domain adaptive clustering for semi-supervised domain adaptation. In Proceedings of the TEEE/CVF Conference on Compuler Vision and Patlern Recog- nition, pp. 2505 2514, 2021b.
Xiaomeng Li, Lequan Yu, Hao Chen, Chi-Wing Fu, Lei Xing, and Pheng-Ann Heng. Transformation- consistent self-ensembling model for semisupervised medical image segmentation. IEEE Transactions Newral Nelworks und Learning Systems, 32(2):523 534, 2020.
Yitong Li, David E Carlson, et al. Extracting relationships by multi-domain matching, Advances in Neural Information Processing Systems, 31, 2018.
Kevin J Liang, Chunyuan Li, Guoyin Wang, and Lawrence Carin. Generative adversarial network training is a continual learning problem. arXiv preprint arXie:1811. 1108, 2018.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Coupled generative adversarial networks. orXiv preprint arXiv:1666.0765 2016.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. International conference on machine learning, pp. 97 105. PMLR, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
Le Lu, Yefeng Zheng, Gustavo Carneiro, and Lin Yang- Deep learning and convolutional neural networks for medical image computing. Advances in computer vision and pattern recognition, 10:978 3, 2017.
Yulong Lu and Jianfeng Lu. A universal approximation theorem of deep neural networks for expressing probability distributions. Advances in neural information processing systems, 33:3094-3105, 2020.
Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi- supervised learning. In Proceedings of the IEEE conference computer vision and pottern recognition, pp. 8896-8905, 2018.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.3430, 2009.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Multiple source adaptation and the rényi divergence. arXiv preprint 120.::205.2000 2628, 2012.
Khai Nguyen, Tongzheng Ren, Huy Nguyen, Litu Rout, Tan Nguyen, and Nhat Ho Hierarchical sliced wasserstein distance. arv preprint arXiv:2209. 13570, 2022.
Papers with Code. Domain Adaptation. httpp///ppeessttoodddecccooasosr adaptatiio 2023. Online; accessed 24-Sept-2023.
Ofir Pele and Michael Werman. Fast and robust earth mover's distances. In 2009 IEEE 12th international conference on computer vision, pp. 460-467. IEEE, 2009.
Kuan-Chuan Peng, Ziyan Wu, and Jan Ernst. Zero-shot deep domain adaptation. In Proceedings of the Europaan Conference on Computer Vision (ECCV), pp. 764-781, 2018.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiw: 06924, 2017.
Pedro O Pinheiro. Unsupervised domain adaptation with similarity learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8004-8013, 2018.
Tevgen Redko, Amaury Habrard, and Marc Scbban. Thcoretical analysis of domain adaptation with optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 737-753. Springer, 2017.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Tim Sainburg, Leland McInnes, and Timothy Q Gentner. Parametric umap embeddings for representation and semisupervised learning. Neural Compulation, 33(11):2881 29077, 2021.
Kumiaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision. and pattern recognition, pp. 3723-3732, 2018.
Swami Sankaranarayanan, Yogesh Balaji, Arpit Jain, Ser Nam Lim, and Rama Chellappa. Learning from synthetic data: Addressing domain shift for semantic sogmentation. In Proceedings of the IEEE conference on compuler vision and pollern recognilion, pp. 3752-377 3361,, 2018.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Yuan Shi and Fei Sha. Information--heorrtiia learning of discriminative clusters for unsupervised domain adaptation. arXiv preprint arXie:1206. 6438, 2012.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Sacnko, and Trevor Darrell. Deep domain confusion: Maxi- mizing for domain invariance. arXiv preprinl arXiv:1412.3474 2014.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultancous deep transfer across domains and tasks. In Proceedings of the IEEE international conference computer vision, pp.
2015.
Eric Tzeng, Judy Holfman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 7167-7176, 2017.
Eric Tzeng, Coline Devin, Judy Hoffman, Chelsea Finn, Pieter Abbeel, Sergey Levine, Kate Saenko, and Trevor Darrell. Adapting deep visuomotor representations with weak pairwise constraints. In Algorithmic Foundations of Robotics XII, pp. 688 703. Springer, 2020.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Haohan Wang, Zeyi Huang, Hanlin Zhang, Yong Jac Lee, and Eric Xing. Toward learning human-aligned cross-domain robust models by countering misaligned features. In Uncertainly in Artificial Intelligence, pp. 2075-2084 PMLR, 2022.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocompuling, 312:135 153, 2018.
Thomas Westfechtel, Hao- Wei Yeh, Qier Meng, Yusuke Mukuta, and Tatsuya Harada. Backprop induced feature weighting for adversarial domain adaptation with iterative label distribution alignment. In Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 392 401, 2023.
YiFan Zhang, Xue Wang, Jian Liang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. Free lunch for domain adversarial training: Environment label smoothing. arXiv preprint arXiv:2302.0019, 2023.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaplation. In International conference on machine learning, pp. 7404-7413. PMLR, 2019.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. Advances neural information processing systems, 31, 2018.
17


Reviews:
[{"review_id": "wt0znT3WMd", "reviewer": "Reviewer_7vNi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The authors provide a theoretical foundation for sub-domain-based domain adaptation methods, addressing gap where previous methods relied solely on empirical success, strengthening the understanding of why sub-domain alignment outperforms full-domain alignment.\n\nThe authors analyse an important failure case where the marginal weights of sub-domains differ between the source and target domains. They propose a reweighted loss function to address this issue providing a deeper understanding of when and why sub-domain methods can fail and how to fix them.\n\nWeaknesses: While the theoretical contributions are novel, the components of DARSA- weighted classification loss, Wasserstein alignment, and clustering loss have been extensively used in previous works.  The novelty lies more in the theoretical analysis rather than the components themselves.  \n\nThe theoretical analysis relies on strong assumptions such as sub-domains following Gaussian distributions, and distances between paired source-target sub-domains being smaller than non-paired distances. While these assumptions  work on certain datasets, they may not hold in highly complex or noisy real-world domain, thereby weakening the theoretical guarantees.\n\nTarget sub-domains are defined using pseudo-labels generated by the target encoder and classifier, which could be noisy leading to inaccurate sub-domain definitions that may degrade the performance.\n\nComputing the Wasserstein distance for multiple sub-domains, along with intra and inter clustering losses could introduce significant computational overhead. For large-scale datasets with many sub-domains, DARSA may be slower and less scalable compared to simpler domain adaptation methods.\n\nThis work mainly addresses sub-domain weight shifts, however it does not deeply explore other types of domain shift, such as feature shift or covariate shift, so the proposed approach might be less effective in cases where shifts occur at levels other than sub-domain weights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors make a significant theoretical contribution by deriving a sub-domain-based generalization bound that is at least as strong as the full-domain generalization bound, providing a rigorous explanation for why sub-domain alignment methods often outperform full-domain alignment methods. They also further analyse the failure condition of sub-domain-based methods under marginal weight shifts between the source and target domains and finally propose an algorithm called DARSA that combines weighted losses, Wasserstein alignment, and clustering using as a theoretically justified and practical solution to address these shifts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical Adjustments: As mentioned in the weaknesses section, the theoretical results rely on strong assumptions, please provide a thorough justification for these assumptions by discussing their limitations. Include scenarios/challenges where these assumptions might fail and discuss how DARSA would perform under such conditions to ensure that the theoretical contributions are robust and widely applicable.\n\nDARSA relies on pseudo-labels to define target sub-domains, but pseudo-labels may be noisy, particularly in the early training stages. Consider analysing the impact of pseudo-label noise on sub-domain alignment and generalization performance and incorporate techniques for mitigating the noise. \n\nProvide an analysis of the computational cost of DARSA, including time complexity and resource requirements and compare its efficiency with other domain adaptation methods on large-scale datasets.  \n\nWhile the theoretical contributions are significant, the algorithmic components (weighted losses, Wasserstein alignment, clustering loss) are well-known. Can you clarify and highlight how DARSA improves over existing methods by integrating these components.\n\nOther Adjustments: While DARSA focuses on label distribution shifts, exploring other types of domain shift such as feature shift, covariate shift would make the work more comprehensive. \n \nAdd a short discussion on the limitations of DARSA, such as its reliance on pseudo-labels, computational costs, and the assumptions made in the theoretical analysis.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BNiq9Y7o07", "reviewer": "Reviewer_fDR1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n\n1. This work provides  a thorough theoretical study on the generalization bound for sub-domain-based methods for UDA. This analysis, supported by the experimental results, is important for understanding the generalization problems of deep models.\n\n2. Based on the theoretical findings, the authors further propose DARSA to tackle the issue of marginal weight shifts between sub-domains. The algorithm focuses on evaluating the importance of each sub-domain and its relevance to the target domain, which is both practical and well-supported by the paper.\n\n3. This work is supported by extensive experimenrts on various benchmarks, which supports the theoretical claims and shows DARSA’s effectiveness in practical scenarios. Using Wasserstein distances to measure discrepancies adds strength to the analysis.\n\n4. The manuscript is well-written with clear motivations. I find the paper easy to follow, despite its focus on a theoretical bound.\n\n### Weaknesses\n\n1. While the paper discusses the effectiveness of using Wasserstein distances, it lacks a detailed discussion on the computational analysis, especially in high-dimensional spaces where such calculations can become challenging.\n\n2. The paper could benefit from broader comparisons with recent state-of-the-art methods. Including more diverse UDA scenarios and newer methods could help in establishing the relative performance of DARSA more comprehensively. As unsupervised domain adaptation (UDA) is an active research area, more recent baseline methods should be included. The latest baseline cited in the paper dates back to 2021.\n\n3. The discussion on the limitations of the proposed method is somewhat limited. A more detailed exploration of scenarios where DARSA might underperform, such as extreme domain shifts or very high-dimensional data, would provide a more balanced view of the method’s applicability. One limitation I can see is the high-dimensionality challenge that may arise from using Wasserstein distances.\n\n4. The generalization of the proposed method across different types of domain adaptation tasks (beyond those tested) is not extensively discussed. Insights into how DARSA performs with non-Gaussian distributions or with datasets having complex or non-convex structures could enhance the paper. How does DARSA perform on the VisDA-2017 dataset?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript contributes to the field of unsupervised domain adaptation (UDA) by providing a robust theoretical framework and a practical algorithm for improving the alignment of sub-domains between source and target datasets. The contributions are found below:\n\n1.This work focuses on the theoretical basis for sub-domain-alignment-based methods in UDA, which shows that these methods can optimize a generalization bound which is at least as strong as those optimized by full-domain UDA methods. \n\n2. The authors then propose the Rebalanced Sub-domain Alignment (DARSA) algorithm designed to handle domain shifts in the marginal weights of sub-domains effectively. The algorithm adjusts the focus on sub-domains according to their importance in the target domain.\n\n3. The paper provides extensive empirical experiments to support their theoretical findings. The authors show that DARSA outperforms existing methods, particularly in scenarios involving label shifts across various baselines.\n\nOverall, the paper contributes new knowledge by deepening the understanding of how sub-domain-based methods can be effectively utilized in UDA and by introducing a novel algorithm to address the importance weighting of each sub domain in target adaptation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Including a computational analysis on the proposed DARSA.\n\n2. Including more recent baseline methods for comparisons.\n\n3. Including more discussion on the limitations of the proposed theoretical work and DARSA.\n\n4. Running experiments on VisDA-2017 dataset.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hcaKYMH4Af", "reviewer": "Reviewer_o63h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The theoretical analysis of sub-domain-based generalization bound is novel and insightful, which adds value to the theoretical understanding of unsupervised domain adaptation (UDA).\n2. I liked the focus on imbalanced UDA because it is an under-studied area and is closer to real-world situations.\n3. The empirical results compare the proposed DARSA approach with different variants—reweighting mechanisms—and demonstrate the impact of DARSA under the predicted accuracy metric.\n\nWeaknesses:\n1. Confusion between class and domain\n\nI find the paper difficult to follow due to the confusion between the concepts of 'class' and 'domain.' From my understanding, the term 'sub-domain' in the paper refers to individual classes within each domain. However, this usage is misleading, as 'sub-domain' might be interpreted as a hierarchical domain structure where a target domain is comprised of smaller sub-domains. Moreover, the paper uses 'class' and 'domain' interchangeably, contradicting literature on domain adaptation and dataset shift. I recommend referring to relevant literature (e.g., [1, 2]) to clearly define the distinction between 'domain' and 'class.'\n\n\n2. Generalization bound and evaluation metric under class imbalance\n\nIn the analysis of the generalization bound, the empirical target error is defined by the empirical error on the target domain, i.e., $\\epsilon_S(h, f)=\\mathrm{E}_{\\mathbf{x} \\sim \\mathcal{D}_S}[|h(\\mathbf{x})-f(\\mathbf{x})|]$ in [3]. However, this may not be the ideal bound under class imbalance. In the extreme imbalance setup, a trivial classifier that always assigns a class label to the majority class could lead to low empirical target error. Empirical results from the paper only report the prediction accuracy and disregard evaluations under class imbalance, e.g., balanced accuracy. I think more consideration should be given to what bound/metric to optimize.\n\nReferences:\n\n[1] Quiñonero-Candela, Joaquin, et al., eds. Dataset shift in machine learning. Mit Press, 2022.\n\n[2] Pan, Sinno Jialin, and Qiang Yang. \"A survey on transfer learning.\" IEEE Transactions on knowledge and data engineering 22.10 (2009): 1345-1359.\n\n[3] Ben-David, Shai, et al. \"A theory of learning from different domains.\" Machine learning 79 (2010): 151-175.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on analyzing and improving \"sub-domain\" based domain adaptation methods with the following main contributions:\n\n1. Theoretical analysis shows that the generalization bound of sub-domain-based methods is at least as strong as full domain-based generalization bounds from the marginal distribution. This finding is insightful in better understanding sub-domain-based methods and is validated using empirical results in Sec. 6.1.\n2. The paper pointed out a challenge in class distribution shift that compromises the sub-domain-based generalization bound, and proposes a framework, Domain Adaptation with Rebalanced Sub-domain Alignment (DARSA), for imbalanced adaptation.\n3. Empirical results validate the effectiveness of the proposed approach on the Digits and TST datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the issues in the weakness section above.\n\n1. Provide a clear definition of class, domain\n2. Elaborate on considerations for class imbalance in generalization bound, especially on the empirical target error and its suitability under imbalance.\n3. Include class imbalance metrics, e.g., balance accuracy, in empirical results and verify the model does not learn trivial solutions.\n4. Improve the writing and clarity of this paper. I find the introduction section and Fig. 1 very challenging to understand due to the confusion between class and domain.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The initial version of the paper raised some comments from the reviewers about the clarity of the the theoretical results, terminology, real-world applicability and a better discussion of the practical implementation of the theoretical results (specifically, how the assumptions hold up in practice). The authors substantially revised the paper for their rebuttal, after a follow up discussion with the reviewers, no questions remained.

However, based on what I remarked in the discussion of *Claims and Evidence* above, I would like to request the following minor edits for the camera ready:

- Reference the VisDA results from Appendix H in the paper: Right now these results are not mentioned at all, e.g. in the experimental setup or in Sec. 6.3 as the other results. If it is possible to meet the page limit, I would even consider including these results in the main paper or mentioning them in the experimental setup (If e.g. only average accuracy from Table 11 is reported, it should be possible to pull it in the main paper and discuss the results there).
- Please slightly reword/clarify the sentence about medical and operational research in Sec 6.3, and/or cite more results. VisDA could be mentioned in this context, for instance.
- Please include a brief statement about opportunities for further empirical validations of DARSA's performance in the limitations section, and/or follow the suggestion below to include error bars for the main result tables (optional!).

Additional minor stylistic suggestions for the camera ready version:

- Consider re-formatting Figure 1. Some text is very small, and the font sizes between the individual panels differ.
- Improve the formatting of Table 4: There are a lot of vertical bars, and the font sizes differs notably from other tables in the paper. This could also make space to include some of the VisDA results in the main paper.

While not remarked by the reviewers, I want to give the authors the chance to also update the results by improving the empirical investigation: This could be done by reporting error bars derived from repeated experiments on at least some of the results, for DARSA and/or other algorithms. This suggestion and to which extent it is incorporated is strictly **optional**, as it requires additional work on the paper and was not asked for explicitly by the reviewers post-rebuttal.
Decision: Accept (poster)